On-Prem Kubernetes Cluster for Scalable MERN Applications

TL;DR

• On-premises Kubernetes cluster designed for production-grade scalability and control
• Microservices-based MERN stack architecture (React frontend + Node.js backend)
• External traffic secured and routed via Cloudflare and Load Balancer
• NGINX Ingress Controller handles domain and path-based routing (/ and /api)
• Fully containerized using Docker with images stored in a private registry
• Automated CI/CD pipeline powered by Jenkins
• Persistent shared storage implemented using NFS volumes
• Database layer uses MongoDB Replica Set for high availability
• Monitoring and observability with Prometheus + Grafana + Loki
• Designed for high availability, fault tolerance, and real-world production workloads

Project Overview

This architecture represents a self-managed Kubernetes cluster deployed on-premises, built to host a large-scale web application with real-world traffic. The system follows a microservices-based design, separating frontend, backend, and supporting services while ensuring:

• High availability
• Scalable infrastructure
• Secure internal communication
• Reliable data persistence

It is optimized for environments where cloud is not preferred, giving full control over infrastructure.

Goals of the Architecture

• Build a highly available on-prem Kubernetes cluster
• Enable scalable microservices deployment
• Ensure secure internal networking
• Implement persistent shared storage (NFS)
• Establish complete CI/CD pipeline
• Provide real-time monitoring and logging

High-Level Architecture

At a high level, the system works like this:

1. Users access the system via Cloudflare
2. Traffic is routed through a Load Balancer
3. Requests enter Kubernetes via Ingress Controller
4. Microservices run across multiple worker nodes
5. Backend services interact with a replicated database cluster
6. Monitoring and logs are collected centrally

Traffic Flow

User → Cloudflare → Load Balancer → Kubernetes Ingress → Services → Pods (Frontend/Backend) → MongoDB Replica Set → Response → User

Core Components

Kubernetes Cluster (On-Prem)

• Master Node
  • API Server
  • Scheduler
  • Controller Manager
  • ETCD
• Worker Nodes
  • Application Pods (Frontend + Backend + Microservices)
  • Kubelet, Kube Proxy
  • Container Runtime

Application Layer

• Frontend (React + NGINX)
  • Served as optimized static assets
• Backend (Node.js API)
  • Handles authentication, business logic, file uploads
• Microservices
  • Scalable and independently deployable

CI/CD & Container Management

• Jenkins – CI/CD pipeline
• Private Docker Registry – Secure image storage
• Docker – Containerization with multi-stage builds

Storage & Persistence

• NFS Server
  • Shared storage across pods
  • Persistent file handling (PDFs, uploads)

Observability

• Prometheus – Metrics collection
• Grafana – Visualization dashboards
• Loki + Alloy – Log aggregation

Database Layer

• MongoDB Replica Set
  • Primary + Secondary nodes
  • Ensures redundancy and failover
• Backup system for data protection

Networking & Traffic Flow

The architecture uses a secure VPC design with public and private subnets:

• Public Subnet:
  • Load Balancer
  • Bastion Host
• Private Subnet:
  • EKS Cluster
  • Database

Data Layer & Persistence Design

• MongoDB Replica Set
  • High availability with primary-secondary replication
• NFS Storage
  • Shared file storage across pods
  • Ensures persistence beyond pod lifecycle

Key Advantage:
Even if pods restart, data remains intact and accessible.

Key Design Choice:
All application services are stateless, making scaling easy and reliable.

Security Design

• Cloudflare for DDoS protection and edge security
• Private internal cluster networking
• Kubernetes Secrets & ConfigMaps for sensitive data
• Controlled access to private Docker registry
• No direct public exposure of internal services
• Role-based access control (RBAC) in Kubernetes

CI/CD & Deployment Strategy

1. Developer pushes code
2. Jenkins pipeline triggers
3. Docker images are built
4. Images pushed to private registry
5. Kubernetes pulls images securely
6. Deployments updated using manifests

✔ Supports:

• Versioned releases
• Rollbacks
• Controlled production deployments

Key Learnings

• On-prem Kubernetes requires strong networking knowledge
• Persistent storage (NFS) is critical for real-world apps
• CI/CD pipelines must handle private registry authentication
• Monitoring and logging are essential for debugging
• Microservices improve scalability but increase complexity
• Proper ingress configuration is crucial for routing