Kubernetes in Production: What They Don't Tell You

Kubernetes tutorials make container orchestration look effortless: apply a YAML file, watch your pods spin up, and celebrate auto-scaling. But when production traffic spikes at 2:00 AM, the subtleties of Linux cgroups, kernel packet filters, and scheduler decisions quickly turn into high-severity incidents.

1. The CPU Limits Trap: Why Your App is Throttled at 30% Utilization

One of the most counterintuitive traps in Kubernetes is setting strict resources.limits.cpu. Many platform engineers assume that assigning a limit prevents a runaway process from consuming the entire node. While true, how the Linux Completely Fair Scheduler (CFS) enforces this limit is often misunderstood.

CFS operates on quota periods (defaulting to 100ms). If your container has a limit of 500m (0.5 cores), it is allocated 50ms of CPU run-time within each 100ms slice. If a multi-threaded web server spikes across multiple threads and burns through that 50ms quota within the first 15ms, the kernel pauses the entire container process for the remaining 85ms.

Your APM dashboards will report average CPU utilization at only 25%, yet your API response latency will suddenly jump from 20ms to 250ms due to repeated CFS throttling pauses.

2. Surviving OOMKilled: Understanding V8 & JVM Memory in Containers

Unlike CPU, memory cannot be throttled. When a pod exceeds its resources.limits.memory, the Linux Out-Of-Memory (OOM) killer instantly terminates the process with Exit Code 137 (OOMKilled).

Common culprits include runtimes that do not automatically detect container cgroup memory boundaries:

• Node.js V8: Node.js often attempts to allocate heap memory based on total machine RAM rather than the pod's container limit. Always pass --max-old-space-size explicitly in your start script (e.g., set to 75% of your pod memory limit).
• JVM (Java): Ensure you are running Java 17+ with -XX:+UseContainerSupport and tune -XX:MaxRAMPercentage=75.0 to prevent JVM native memory overhead (metaspace, thread stacks) from triggering container eviction.

3. Zero-Downtime Deployments & The preStop Hook

When Kubernetes rolls out a new deployment version, it concurrently sends a SIGTERM signal to the terminating pod while removing the pod's IP address from EndpointSlices and iptables routing rules.

Because iptables updates take several seconds to propagate across all worker nodes, traffic intended for the terminating pod continues to arrive after the application has begun shutting down—resulting in HTTP 502/504 Bad Gateway spikes during routine deployments.

The battle-tested fix is adding a preStop sleep hook to allow inflight requests to complete while routing rules update:

spec:
  containers:
    - name: api-server
      image: registry.internal.net/api:v2.4
      lifecycle:
        preStop:
          exec:
            # 1. Delay shutdown to allow iptables propagation
            command: ["/bin/sh", "-c", "sleep 15"]
      readinessProbe:
        httpGet:
          path: /health/ready
          port: 8080
        initialDelaySeconds: 5
        periodSeconds: 5
      livenessProbe:
        httpGet:
          path: /health/live
          port: 8080
        initialDelaySeconds: 15
        periodSeconds: 10
  # Ensure grace period accounts for the preStop sleep duration
  terminationGracePeriodSeconds: 45

4. Moving Beyond 'kubectl apply' with GitOps

Running manual kubectl apply or executing scripts directly from developer laptops destroys auditability and invites configuration drift. In enterprise production, Git must be the single source of truth for all declarative cluster state.

Using ArgoCD or Flux, changes are merged into a Git repository via pull requests. The GitOps operator continuously synchronizes the live cluster state with the desired state in Git, auto-healing any unauthorized manual cluster modifications.

5. Essential SRE Checklist for Kubernetes

• Define PodDisruptionBudgets (PDB) to prevent node drains and upgrades from taking down critical services simultaneously.
• Set both readinessProbe and livenessProbe with distinct endpoints to prevent pods undergoing heavy GC or warm-up from being prematurely killed.
• Implement NetworkPolicies from Day 1 to prevent lateral movement in the event of a container compromise.