1. Enforcing Pod Anti-Affinity

If an operator controller runs with multiple replicas for active-passive leader election, scheduling both replicas on the same physical Kubernetes worker node creates a single point of failure. Use preferredDuringSchedulingIgnoredDuringExecution to spread replicas across failure domains.

2. Dropping Linux Capabilities

Most operator pods only interact with the Kubernetes apiserver over HTTPS. They do not require elevated Linux capabilities like NET_ADMIN or SYS_PTRACE. Dropping all capabilities and mounting the root filesystem as read-only satisfies the highest compliance tier of the Kubernetes Pod Security Standards.

How the Linux Kernel Triggers SIGKILL (Exit 137)

Every container runtime allocates a cgroup for the container process. When resident set memory (RSS) surpasses resources.limits.memory, the cgroup memory controller dispatches SIGKILL (Signal 9 + 128 = Exit Code 137). Kubernetes records this condition as OOMKilled.

Configuring Memory Headroom for Operator Controllers

Operators frequently maintain in-memory caches (such as controller-runtime client-go informers) of all managed custom resources in the cluster. As the number of CRs grows from 10 to 1,000, informer cache memory grows linearly. Always calculate a 30% to 50% buffer over baseline idle consumption.

1. Inspecting the Previous Container Instance

The single most common mistake when troubleshooting CrashLoopBackOff is running kubectl logs <pod> without flags. Because the container is repeatedly dying and restarting, standard logs only query the newly spawned process, which is often still initializing.

# Inspect the crash traceback of the dying container
kubectl logs <pod-name> -n <namespace> --previous

2. Checking Exit Codes with kubectl describe

Run kubectl describe pod <pod-name> -n <namespace> and scroll to the Last State section:

• Exit Code 137: The Linux kernel Out-Of-Memory (OOM) killer dispatched SIGKILL. The operator exceeded its resources.limits.memory.
• Exit Code 1: An unhandled application panic or missing environment variable caused an abnormal exit.
• Exit Code 143: Graceful SIGTERM received, usually because a liveness probe failed or the node was cordoned.

3. Verifying Operator RBAC Permissions

Kubernetes operators require granular ClusterRole permissions to watch Custom Resource Definitions (CRDs). If an operator attempts to query a CRD without appropriate verbs (get, list, watch, update), the apiserver returns HTTP 403 Forbidden, triggering an application panic on startup.