A pod keeps restarting — kubectl get pods shows CrashLoopBackOff with a climbing restart count, and the backoff between restarts keeps growing.

What is actually happening (the mental model).

CrashLoopBackOff means the container starts, exits, and Kubernetes is restarting it with an exponentially increasing delay — it isn't a failure state itself, it's a description of a pattern (start → exit → wait → repeat). The real cause is always in why the process exited, which the crash-loop label doesn't tell you.

How to work through it.

1. kubectl describe pod — check the Last State block for the exit code (0 = clean exit but the process shouldn't have exited; 1 = generic app error; 137 = SIGKILL, usually OOM; 143 = SIGTERM).
2. kubectl logs --previous — the crashed container's logs, which usually show the actual application error.
3. If logs are empty, the process may be exiting before it can log anything — check the command/entrypoint and confirm the image actually runs standalone (docker run it locally).
4. Common root causes: a missing ConfigMap/Secret key the app expects at startup, a bad DB connection string, a misconfigured liveness probe killing a slow-starting app, or a genuine application bug.

Root causes, and how to tell them apart.

Exit code 137 → OOM (check limits and actual usage). Exit code 1 with a stack trace in --previous logs → application bug or bad config. Empty logs → the process died before logging, often a missing env var/secret the entrypoint reads immediately. Crashes only under load → look at readiness/liveness probes killing it prematurely, not the app itself.

The trap that less-experienced engineers fall into.

Reading the current logs (of the new, still-starting container) instead of --previous, and concluding "there are no errors" when the actual crash information is one restart behind.

🎯 Interviewer follow-up questions you should expect.

• "What does exit code 137 mean?" It's SIGKILL, almost always from an OOM kill — you check the memory limits against actual usage.
• "Why are the logs empty?" Because the process died before it could log anything — you check the entrypoint and the required startup configuration.
• "How do you see the logs of a container that already crashed?" With kubectl logs --previous.

A pod has been sitting in Pending for several minutes — it was never even placed on a node.

What is actually happening (the mental model).

Pending means the scheduler hasn't been able to find a node that satisfies the pod's requirements. This is entirely a placement problem — the pod's own container hasn't even started yet, so logs won't help; describe and its Events are the whole story.

How to work through it.

1. kubectl describe pod — the Events section spells out the scheduler's reason: "Insufficient cpu/memory," "didn't match node selector," "had taints the pod didn't tolerate," or "no nodes available."
2. Check requested resources against real node capacity (kubectl describe nodes, or kubectl top nodes) — over-requesting is the most common cause.
3. Check node affinity/selectors and taints/tolerations if resources look fine — a pod can be unschedulable purely on policy, with plenty of free capacity elsewhere.
4. If using a cluster autoscaler, confirm it's actually configured to add capacity — otherwise the pod waits forever for a node that will never appear.

Root causes, and how to tell them apart.

"Insufficient cpu/memory" → real capacity shortage, add nodes or lower requests. "Didn't match node selector/affinity" or "had taint X the pod didn't tolerate" → a policy mismatch, not a capacity problem — fix the pod spec or the node labels. No autoscaler and no free capacity → the cluster is simply full.

The trap that less-experienced engineers fall into.

Checking pod logs first (there are none — it never started) instead of going straight to describe and reading the scheduler's Events, which almost always states the exact reason in plain text.

🎯 Interviewer follow-up questions you should expect.

• "A pod is Pending — what's the first thing you check?" The describe pod Events, which name the scheduling failure directly.
• "Node has free capacity but the pod still won't schedule — why?" It's usually a taint or toleration mismatch, or an affinity or selector mismatch, not a resource problem.

A pod keeps dying with reason OOMKilled in describe, even though the application "shouldn't" be using that much memory.

What is actually happening (the mental model).

The container hit its memory limit and the kernel's cgroup OOM killer terminated it — this is a hard ceiling, not a soft warning. It can mean the limit is genuinely too low, or that the app has a real memory leak/spike, or — very commonly — that a JVM/Node runtime is sizing its heap against the host's memory rather than the container's actual limit.

How to work through it.

1. Confirm it with describe pod — Last State: Terminated, Reason: OOMKilled, exit code 137.
2. Compare the limit against actual usage over time (metrics-server / Prometheus) — is this a spike, a slow leak, or just an under-sized limit for normal operation?
3. If the runtime is JVM/Node/similar, confirm it's cgroup-aware (recent versions are; older ones need explicit heap flags sized to the container limit, not the host).
4. Fix by right-sizing the limit to real usage with headroom, or fixing an actual leak — don't just keep raising the limit as a reflex.

Root causes, and how to tell them apart.

Usage climbs steadily over the pod's lifetime → a real leak. Usage is fine normally but spikes under specific load → the limit needs headroom for peaks, or the workload needs to be optimised. Usage looks fine in metrics but the pod still OOMs → the runtime isn't respecting the cgroup limit at all.

The trap that less-experienced engineers fall into.

Reflexively doubling the memory limit every time this happens, without ever looking at whether it's a leak (which will eventually hit any limit) or a cgroup-unaware runtime (which no limit increase truly fixes).

🎯 Interviewer follow-up questions you should expect.

• "How do you confirm a pod was OOMKilled?" You run describe pod and look for Last State Terminated, with Reason OOMKilled and exit code 137.
• "You raised the limit and it still OOMs eventually — why?" It's likely a real memory leak, not an under-sized limit.

A pod is stuck with ImagePullBackOff — Kubernetes can't pull the container image, and it's retrying with backoff.

What is actually happening (the mental model).

The kubelet on the node tried to pull the image and failed — this is entirely about the image reference and registry access, not the application. Common causes: a typo in the image name or tag, the image genuinely doesn't exist at that tag, or the node lacks credentials for a private registry.

How to work through it.

1. kubectl describe pod — the Events show the exact pull error: "manifest not found," "unauthorized," "not found," or a network timeout to the registry.
2. Double-check the image name and tag for typos, and confirm that tag actually exists in the registry.
3. If private, confirm an imagePullSecret is attached to the pod (or its ServiceAccount) and that the credentials in it are still valid — expired registry credentials are a very common cause.
4. If it's intermittent, check registry rate limits (Docker Hub's anonymous-pull limits are a classic culprit) or node-to-registry network/firewall issues.

Root causes, and how to tell them apart.

"Manifest not found"/"not found" → wrong name or tag, or the image was never pushed. "Unauthorized"/"denied" → missing or expired imagePullSecret. Works sometimes, fails other times → registry rate limiting or a flaky network path from the node.

The trap that less-experienced engineers fall into.

Assuming the application is broken and digging through app logs — there are no app logs, because the container image was never even pulled.

🎯 Interviewer follow-up questions you should expect.

• "ImagePullBackOff — first thing you check?" The describe pod Events for the exact pull error, then the image name and tag, and the imagePullSecret.
• "It works most of the time but fails intermittently — why?" It's likely registry rate limiting or a flaky network path, not the image itself.

A perfectly healthy application keeps getting restarted by Kubernetes itself, or never shows as "ready" even though it's clearly serving traffic fine.

What is actually happening (the mental model).

Liveness and readiness probes are Kubernetes actively testing the app and acting on the result — a liveness failure gets the container killed and restarted; a readiness failure gets the pod pulled out of Service endpoints (traffic stops, but the pod keeps running). Misconfigured probes cause Kubernetes to "fix" a problem that doesn't exist.

How to work through it.

1. Identify which probe is failing and what it actually checks — describe pod shows probe failure events.
2. The classic mistake: initialDelaySeconds too short for a slow-starting app, so the liveness probe kills it before it's even finished booting.
3. Confirm the probe endpoint/command genuinely reflects health — a liveness probe that checks a downstream dependency (like a database) will kill the app for someone else's outage, which is usually wrong; that belongs in readiness, not liveness, or not at all.
4. Tune periodSeconds, timeoutSeconds, and failureThreshold to match how the app actually behaves, rather than leaving defaults that don't fit a slow or bursty app.

Root causes, and how to tell them apart.

Restarts right after startup, repeatedly → initialDelaySeconds too short for the app's real boot time. Never becomes Ready despite being healthy → the readiness check is testing the wrong thing or the wrong port/path. Restarts correlate with a downstream outage → the liveness probe is checking a dependency it shouldn't.

The trap that less-experienced engineers fall into.

Making the liveness probe check the same deep dependency health as readiness — so one flaky downstream service causes Kubernetes to kill and restart every pod in the fleet, turning a partial outage into a total one.

🎯 Interviewer follow-up questions you should expect.

• "App is healthy but keeps restarting — what do you check?" The liveness probe configuration, especially initialDelaySeconds against the real startup time.
• "Difference between what a liveness and a readiness failure does?" A liveness failure kills and restarts the container; a readiness failure removes it from the Service's endpoints without restarting it.

A pod shows Init:0/1 forever — the main application container never even starts.

What is actually happening (the mental model).

Init containers run to completion, in order, before any main container starts — they're for setup work (waiting for a dependency, running a migration, fetching config). If an init container hangs or keeps failing, the whole pod is stuck, because Kubernetes will never start the main containers until every init container has exited successfully.

How to work through it.

1. kubectl describe pod and kubectl logs <pod> -c <init-container-name> — you must name the specific init container; plain logs defaults to the main container, which hasn't started.
2. Check what the init container is actually waiting on (commonly: a dependency like a database that isn't up yet, or a network policy silently blocking it).
3. If it's crash-looping rather than hanging, check its exit code and logs the same way you would a main-container crash.

The root causes and trade-offs.

Sometimes the init container is simply waiting on a dependency that's slow or down, and the fix there is either fixing that dependency or adding a bounded retry with a timeout to the init logic itself. Other times there's a genuine bug or misconfiguration in the init container. And sometimes a NetworkPolicy is blocking the init container's outbound call even though the main container's own policy would allow it.

The trap that less-experienced engineers fall into.

Running kubectl logs without -c and seeing an error or nothing useful, because it defaulted to a main container that has never started — instead of explicitly targeting the init container by name.

🎯 Interviewer follow-up questions you should expect.

• "Pod stuck at Init:0/1 — how do you see what's wrong?" With kubectl logs <pod> -c <init-container>, naming the init container explicitly.
• "What are init containers for?" Sequential setup work that must complete before the main containers start — waiting on a dependency, running migrations, fetching configuration.

A routine Deployment update — meant to be zero-downtime — caused real request failures for users during the rollout.

What is actually happening (the mental model).

A rolling update is only zero-downtime if Kubernetes actually knows when a new pod is ready to take traffic and when an old pod has finished in-flight work before it's killed. Without a correct readiness probe and a graceful shutdown, the Service sends traffic to a new pod before it's really ready, or kills an old pod mid-request.

Why a rolling update caused downtime (the root causes).

1. No or too-lenient readiness probe — Kubernetes adds the new pod to the Service before it can actually handle requests.
2. No graceful shutdown — the app doesn't handle SIGTERM to stop accepting new work and finish in-flight requests; Kubernetes sends SIGTERM, waits terminationGracePeriodSeconds, then SIGKILLs, possibly mid-request.
3. Aggressive rollout parameters — maxUnavailable too high relative to real capacity, taking down more old pods than the new ones can immediately absorb.

How to fix it, verify it, and prevent it.

Add an accurate readiness probe that only passes once the app can truly serve traffic; implement graceful shutdown (stop accepting new connections on SIGTERM, drain in-flight requests, then exit) and size terminationGracePeriodSeconds to match; tune maxUnavailable/maxSurge conservatively. Verify with a load test during a rollout, watching for error-rate spikes. Prevent recurrence by making a zero-downtime rollout check part of your deployment pipeline, not a one-off fix.

The trap that less-experienced engineers fall into.

Assuming "rolling update" automatically means zero downtime by default — it only removes downtime if readiness and graceful shutdown are both correctly implemented; neither is automatic.

🎯 Interviewer follow-up questions you should expect.

• "Why did a rolling update cause errors?" A missing or weak readiness probe, no graceful shutdown handling for SIGTERM, or a maxUnavailable setting that's too aggressive.
• "How do you achieve a truly zero-downtime deploy?" Accurate readiness gating, graceful shutdown, and conservative rollout parameters, all verified under real load.

A newly rolled-out version is misbehaving in production and you need to get back to the last known-good version, fast.

What is actually happening (the mental model).

A Deployment keeps a revision history of its ReplicaSets, so "rolling back" is really just telling the Deployment controller to make an older ReplicaSet the active one again — the same rolling-update machinery runs in reverse.

How to work through it (the safe sequence).

1. Confirm it's actually the new version at fault (check describe pod/logs/metrics) before rolling back — rolling back a red herring wastes the window while the real cause keeps running.
2. kubectl rollout undo deployment/<name> — reverts to the previous revision; add --to-revision=N to target a specific one from kubectl rollout history.
3. Watch the rollback itself like any other rollout — kubectl rollout status — since it goes through the same readiness-gated process.
4. Afterward, keep the bad image/manifest around long enough to diagnose why it failed, rather than only reverting and moving on.

The root causes and trade-offs.

Manually patching forward under pressure is higher risk and slower, while a clean rollout undo is fast and uses machinery that's already been tested. Rolling back trades away the speed of fixing forward in exchange for safety, which is the right trade whenever the actual cause isn't yet understood.

The trap that less-experienced engineers fall into.

Deleting the revision history or the Deployment itself in a panic instead of using rollout undo — or rolling back without confirming the new version is actually the cause, masking the real problem.

🎯 Interviewer follow-up questions you should expect.

• "How do you roll back a bad deployment?" With kubectl rollout undo deployment/<name>, optionally targeting a specific revision.
• "How does Kubernetes know what the previous version was?" From the Deployment's ReplicaSet revision history.

kubectl rollout status hangs — some new pods are up, some old ones remain, and it's been stuck like that for a while.

What is actually happening (the mental model).

A rollout only proceeds when new pods pass their readiness probe; if the new ReplicaSet's pods never become ready, the Deployment controller simply stops progressing exactly where it is — it won't automatically roll back on its own by default.

How to work through it.

1. kubectl get pods — look at the new ReplicaSet's pods specifically; are they CrashLoopBackOff, Pending, or Running but never Ready?
2. Apply the standard pod-debugging reflex (describe, logs --previous) to those specific new pods to find why they're failing readiness or crashing.
3. If the new version is simply broken, kubectl rollout undo rather than waiting indefinitely.
4. Consider setting progressDeadlineSeconds going forward so a stuck rollout is flagged/failed automatically instead of hanging silently.

Root causes, and how to tell them apart.

New pods crash-looping → an app-level bug in the new version. New pods Pending → a scheduling problem unrelated to the app (resources, affinity). New pods Running but never Ready → a readiness probe/config issue specific to the new version.

The trap that less-experienced engineers fall into.

Watching the overall rollout status and waiting, without drilling into the new ReplicaSet's specific pods to see which failure mode is actually blocking it.

🎯 Interviewer follow-up questions you should expect.

• "Rollout is stuck halfway — what do you check?" The new ReplicaSet's pods specifically, with the standard describe and logs reflex.
• "Does Kubernetes auto-rollback a stuck rollout?" Not by default — you need to run rollout undo yourself, though progressDeadlineSeconds can mark it as failed.

Pods are Running and look healthy, but requests to the Service in front of them time out or fail.

What is actually happening (the mental model).

A Service finds its pods purely through label selectors — if the selector doesn't match the pods' actual labels (a typo, or a label that changed), the Service has zero endpoints and traffic goes nowhere, even though the pods themselves are perfectly healthy.

How to work through it.

1. kubectl get endpoints <service> — if this is empty, the Service isn't matching any pods; that's the whole problem right there.
2. Compare the Service's selector against the actual labels on the pods (kubectl get pods --show-labels) — look for a typo or a mismatch.
3. If endpoints are populated but traffic still fails, the pods may be listed but failing their readiness probe — not-ready pods are excluded from endpoints too.
4. If endpoints are correct and populated, the problem has moved one layer up — check kube-proxy/iptables on the node, or a NetworkPolicy blocking the path.

Root causes, and how to tell them apart.

Empty endpoints with matching-looking labels → check for whitespace/typo mismatches carefully, string-exact. Endpoints populated but requests still fail → readiness gating pods out, or a NetworkPolicy, or a port-name/targetPort mismatch in the Service spec.

The trap that less-experienced engineers fall into.

Debugging the application itself (assuming a code bug) when the pods are perfectly healthy and the actual fault is a label selector that simply doesn't match — get endpoints would have shown this immediately.

🎯 Interviewer follow-up questions you should expect.

• "Service isn't reaching healthy pods — first command?" kubectl get endpoints <service> — an empty result means a selector or label mismatch.
• "Endpoints look right but it still fails — what next?" You check readiness gating, a NetworkPolicy, or a port versus targetPort mismatch.

A pod can't resolve another Service's name (payment.default.svc.cluster.local) — connection attempts fail with a DNS error, not a connection-refused.

What is actually happening (the mental model).

In-cluster DNS is provided by CoreDNS, itself running as pods in the cluster — so cluster DNS can fail for the same reasons any other workload can (CoreDNS pods down, misconfigured, or unreachable), on top of ordinary lookup mistakes.

How to work through it.

1. Test resolution directly from a throwaway pod: kubectl run -it --rm dnsutils --image=... -- nslookup payment.default.svc.cluster.local.
2. Check CoreDNS itself is healthy: kubectl get pods -n kube-system -l k8s-app=kube-dns and its logs.
3. Confirm the name is actually correct — the full form is <service>.<namespace>.svc.cluster.local; a common mistake is omitting the namespace when calling cross-namespace.
4. If CoreDNS is healthy and the name is right, check NetworkPolicies aren't blocking pods from reaching CoreDNS's IP/port (UDP/TCP 53) at all.

Root causes, and how to tell them apart.

CoreDNS pods down/crash-looping → a cluster-wide DNS outage, fix CoreDNS first. Only some pods fail → likely a NetworkPolicy blocking their path to CoreDNS. One specific name fails everywhere → the name itself is wrong (missing namespace, typo) rather than a DNS-system problem.

The trap that less-experienced engineers fall into.

Treating "DNS is a cluster thing that just works" and not realising CoreDNS is a regular workload that can itself be down, resource-starved, or blocked by policy like anything else.

🎯 Interviewer follow-up questions you should expect.

• "In-cluster DNS is failing — what do you check?" Whether the CoreDNS pods themselves are healthy, then the exact service name form, then any NetworkPolicy.
• "What's the full DNS name of a Service?" <service>.<namespace>.svc.cluster.local.

Requests through the Ingress return 404s or hit the wrong backend, even though the Service and pods behind it are fine on their own.

What is actually happening (the mental model).

An Ingress resource does nothing by itself — it's read and implemented by an Ingress controller, and routing bugs live in the gap between "what the YAML says" and "what the controller actually configured." Path-type semantics (Prefix vs Exact), host-header matching, and TLS/annotation quirks are the usual suspects.

How to work through it.

1. Confirm the Ingress controller is actually running and has picked up the resource — check its logs, which usually state what it synced.
2. Check pathType carefully — Prefix vs Exact behave very differently for the same-looking path, and a wrong choice silently misroutes or 404s.
3. Confirm the host field matches the actual request's Host header exactly, and that the backend service/port named in the rule is correct.
4. Check for a second, unexpected Ingress or a default backend catching the request first — rule ordering/precedence across multiple Ingress resources is a common surprise.

The root causes and trade-offs.

A pathType mismatch, confusing Prefix with Exact, is the single most common cause. Host-header mismatches and multiple competing Ingress resources are the next most common causes after that. All of these are configuration precision issues, not capacity or code problems.

The trap that less-experienced engineers fall into.

Debugging the Service and pods (which are fine) instead of the Ingress controller's own logs and its interpretation of the rule — the fault is almost always in how the routing rule was written or how multiple rules interact.

🎯 Interviewer follow-up questions you should expect.

• "Ingress returns 404 even though the Service works directly — where do you look?" The Ingress controller's own logs and the exact pathType and host rules, not the Service.
• "Prefix vs Exact pathType — why does it matter?" Because they match differently for what looks like the same path, and getting it wrong silently misroutes traffic.

After a NetworkPolicy was introduced (by you or someone else), a previously-working connection between two pods now times out.

What is actually happening (the mental model).

The moment any NetworkPolicy selects a pod for a given direction (ingress or egress), that pod's traffic in that direction becomes default-deny for anything not explicitly allowed — NetworkPolicies are additive/whitelist, not a set of exceptions to an otherwise-open default.

How to work through it.

1. Identify every NetworkPolicy that selects either pod involved (source and destination) — a policy on either end can be the blocker.
2. Check whether the blocked traffic is ingress (into the destination) or egress (out of the source) — you need an explicit allow rule in the right direction, on the right pod.
3. Remember that allowing ingress on the destination doesn't help if the source's own egress policy blocks it first, and vice versa — both sides need to permit the connection.
4. Confirm the policy's pod/namespace selectors actually match what you think they match — a subtly wrong label selector can lock out traffic you meant to allow.

Root causes, and how to tell them apart.

Missing ingress rule on the destination, missing egress rule on the source, or a selector that doesn't match the intended pods precisely. Testing from both directions (can the source send, can the destination receive) narrows it quickly.

The trap that less-experienced engineers fall into.

Assuming a NetworkPolicy only restricts what it explicitly lists as denied, when in fact selecting a pod at all flips it to default-deny in that direction — "I didn't deny it" doesn't mean "it's allowed."

🎯 Interviewer follow-up questions you should expect.

• "A NetworkPolicy broke traffic that used to work — what's the mental model?" Selecting a pod makes that direction default-deny, so you need an explicit allow rule, not just the absence of a deny.
• "Traffic still blocked after adding an ingress rule — why?" The source pod's own egress policy may be blocking it independently — both sides need to allow it.

A PersistentVolumeClaim has been sitting in Pending and the pod that needs it can't start.

What is actually happening (the mental model).

A PVC stays Pending until something can provision or bind a matching PersistentVolume — with dynamic provisioning, that "something" is a StorageClass and its provisioner, which can fail for reasons that have nothing to do with your pod spec at all.

How to work through it.

1. kubectl describe pvc — the Events show why binding/provisioning hasn't happened (no matching StorageClass, provisioner errors, quota exceeded).
2. Confirm the storageClassName requested actually exists (kubectl get storageclass) — a typo or a class that was never created is a very common cause.
3. Check the underlying cloud-provider quota/permissions for the provisioner (e.g. an EBS volume limit, or the provisioner's ServiceAccount lacking cloud API permissions).
4. For static provisioning, confirm a matching PersistentVolume actually exists with compatible size/access modes/StorageClass.

The root causes and trade-offs.

Sometimes the StorageClass is missing or misnamed. Sometimes the provisioner itself lacks the cloud permissions it needs, or it's hit a quota. And for static PVs specifically, sometimes there's simply no matching volume available at all. All of these are infrastructure-layer issues that the pod's own spec has no way to fix on its own.

The trap that less-experienced engineers fall into.

Treating a Pending PVC as a pod problem and debugging the Deployment, when the real information is entirely in describe pvc and the storage layer underneath it.

🎯 Interviewer follow-up questions you should expect.

• "PVC stuck Pending — where do you look?" The describe pvc Events first, then whether the named StorageClass actually exists and the provisioner has the quota and permissions it needs.

A StatefulSet pod was rescheduled to a different node and now won't start — it's stuck waiting to attach its volume.

What is actually happening (the mental model).

Many block-storage volumes (like EBS) can only be attached to one node at a time. If a node dies or is drained abruptly, the volume may still show as attached to the old node until it's forcibly detached, and the new pod can't proceed until that's resolved — this is a real safety mechanism (preventing two nodes writing to the same disk), not a bug.

How to work through it.

1. kubectl describe pod — look for a "multi-attach" or volume-attach error naming the old node.
2. Confirm whether the old node is genuinely gone (in which case the attachment needs to be cleaned up at the cloud/CSI level) or just temporarily unreachable (in which case waiting, or fencing it properly, is safer than forcing a detach).
3. Understand this is inherent to ReadWriteOnce volumes tied to StatefulSets — it's the trade-off for strong per-instance identity and storage, versus a stateless Deployment that doesn't care which node it lands on.

The root causes and trade-offs.

The root cause is usually a node that failed or was removed without a clean volume detach, combined with the volume's access mode, ReadWriteOnce, which inherently limits it to a single attachment point at a time. Forcing a detach too early risks real data corruption if the old node isn't actually dead, so the trade-off here is genuinely between availability and data safety.

The trap that less-experienced engineers fall into.

Forcibly deleting the old pod/node record to "fix" it quickly without confirming the old node is genuinely gone — risking two writers on the same volume if it wasn't.

🎯 Interviewer follow-up questions you should expect.

• "StatefulSet pod won't start on a new node — why?" Its RWO volume is still considered attached to the old node, which is a safety mechanism, not a bug.
• "Is it safe to force-detach?" Only once you've confirmed the old node is genuinely gone, not just unreachable — otherwise you risk two writers on one volume.

A kubectl command or an in-cluster application call fails with Error from server (Forbidden): ... is forbidden: User "..." cannot ....

What is actually happening (the mental model).

The API server checked the request against RBAC and found no Role/ClusterRole plus RoleBinding/ClusterRoleBinding that grants the actor (user or ServiceAccount) that specific verb on that specific resource — RBAC is deny-by-default, so any gap in the chain shows up as Forbidden.

How to work through it.

1. Read the error message carefully — it names the exact actor, verb (get/list/create/delete…), and resource that was denied.
2. kubectl auth can-i <verb> <resource> --as=<user-or-serviceaccount> — directly tests the permission and confirms the gap without guessing.
3. Check whether a Role/ClusterRole granting that permission exists at all, and if so, whether a RoleBinding/ClusterRoleBinding actually attaches it to this specific actor (and in the right namespace, for a namespaced Role).
4. Fix by adding the minimum necessary permission — resist the urge to grant a broad ClusterRole just to make the error go away.

The root causes and trade-offs.

Sometimes no Role exists at all with the needed permission. Sometimes a Role does exist but it was never bound to this particular actor. And sometimes a RoleBinding exists but it references the wrong Role, or the wrong namespace. Over-granting access, like handing out cluster-admin just to unblock things fast, trades a real security regression for a quick fix that isn't actually worth it.

The trap that less-experienced engineers fall into.

Reflexively granting cluster-admin or a wildcard Role to make a Forbidden error disappear, instead of granting the specific verb/resource that was actually denied.

🎯 Interviewer follow-up questions you should expect.

• "How do you debug a Forbidden error precisely?" With kubectl auth can-i --as=<actor> to confirm the exact gap, then check the Role and its Binding.
• "Why not just grant cluster-admin to unblock it?" Because it fixes the symptom by removing all least-privilege protection — you should grant only the specific verb and resource that's actually needed.

A security review finds an application's ServiceAccount can do far more than the app needs (or, conversely, the app keeps hitting permission walls it shouldn't).

What is actually happening (the mental model).

Every pod runs as some ServiceAccount — the default one if you didn't specify — and if nobody has deliberately scoped it, it's easy to end up either over-privileged (broad ClusterRole bound out of convenience) or under-privileged (nobody granted the one specific permission the app actually needs).

How to reason about it.

1. Give every application its own dedicated ServiceAccount rather than sharing the namespace default — this makes auditing and scoping possible at all.
2. Enumerate what the app actually calls the API for (read its own ConfigMaps? watch its own CRDs? nothing at all?) and grant exactly that, via a Role scoped to the namespace, not a ClusterRole, unless cluster-wide access is genuinely required.
3. Audit existing bindings with kubectl get rolebindings,clusterrolebindings -A -o wide and cross-reference against what's actually used.
4. Disable auto-mounting the ServiceAccount token (automountServiceAccountToken: false) for pods that don't call the Kubernetes API at all.

The root causes and trade-offs.

A convenience-driven broad grant, the "just give it cluster-admin, we'll fix it later" instinct, trades away real security for the upfront work of scoping a Role precisely never getting done. The trade-off is a bit more initial setup, in exchange for a real reduction in blast radius if that pod is ever compromised.

The trap that less-experienced engineers fall into.

Every application sharing the namespace's default ServiceAccount with a broad Role bound to it — so a vulnerability in any one app hands an attacker the combined permissions of every app in the namespace.

🎯 Interviewer follow-up questions you should expect.

• "How do you scope permissions for an application?" With a dedicated ServiceAccount per app, and a namespaced Role granting exactly the verbs and resources it uses.
• "Why not just use the default ServiceAccount?" Because it's usually shared and over-scoped, so a compromise in one app can reach everything bound to it.

A review finds a Secret's value showing up somewhere it shouldn't — in logs, in a describe output, or readable by a team that shouldn't have access.

What is actually happening (the mental model).

Kubernetes Secrets are base64-encoded, not encrypted, by default — base64 is trivially reversible, so anyone with API read access to the Secret (or etcd access) can read the real value. "It's a Secret object" is not, by itself, a security control.

How to work through it.

1. Enable encryption at rest for Secrets in etcd (an EncryptionConfiguration) so a raw etcd compromise doesn't expose plaintext values.
2. Restrict RBAC read access to Secrets tightly — get/list on secrets is one of the most sensitive permissions in the cluster and should be scoped far more carefully than most other resources.
3. Never echo Secret values into logs — application code and init scripts should read them directly into memory/env, not print them for "debugging."
4. For real secret management maturity, use a dedicated tool (Vault, Sealed Secrets, an external-secrets operator) rather than relying on raw Kubernetes Secrets as the system of record.

The root causes and trade-offs.

One cause is treating base64 encoding as if it were actual encryption. Another is leaving RBAC on Secrets just as loose as on any other resource. A third is debug logging that carelessly prints environment variables or configuration, secret values included. The fix — encryption at rest, tight RBAC, and a real secrets manager — costs real setup effort, in exchange for a genuine security guarantee.

The trap that less-experienced engineers fall into.

Believing a Kubernetes Secret is inherently secure because it's a special object type, and not realising base64 offers no confidentiality at all — it's an encoding, not a cipher.

🎯 Interviewer follow-up questions you should expect.

• "Are Kubernetes Secrets encrypted?" Not by default — they're only base64 encoded, so you have to enable encryption at rest for real protection.
• "How do you restrict who can read Secrets?" With tight, deliberate RBAC specifically on the secrets resource, tighter than most other objects.

The HorizontalPodAutoscaler either never scales up under real load, or scales wildly up and down (flapping) every few minutes.

What is actually happening (the mental model).

HPA makes decisions from metrics (usually CPU/memory via the metrics-server, or custom metrics) compared against a target you set — if the metrics pipeline is broken, or requests/limits are set oddly, or there's no cooldown, the HPA's math either never triggers or triggers constantly.

How to work through it.

1. kubectl describe hpa — shows the current vs target metric value and any errors reaching the metrics source (metrics-server down, or a custom metrics adapter misconfigured).
2. Confirm pods actually have CPU/memory requests set — HPA's percentage-of-request calculation is meaningless without them.
3. If it's flapping, check for missing/short stabilization windows — without them, a brief metric dip or spike triggers an immediate scale event in the other direction.
4. For request-driven or bursty workloads, consider whether CPU/memory are even the right signal — a custom metric (queue depth, request rate) may better reflect real load.

Root causes, and how to tell them apart.

"Unknown" or missing metrics in describe hpa → the metrics pipeline itself is broken (metrics-server down, adapter misconfigured). Never scales despite real load → requests aren't set, so the percentage math never crosses target. Flaps constantly → no stabilization window smoothing decisions.

The trap that less-experienced engineers fall into.

Assuming HPA "just works" off raw CPU usage, without realising it's calculated as a percentage of the pod's requested CPU — pods with no request set, or a request wildly mismatched to real usage, make the whole calculation meaningless.

🎯 Interviewer follow-up questions you should expect.

• "HPA won't scale up under load — what do you check?" describe hpa for metric errors, then whether CPU and memory requests are actually set on the pods.
• "HPA keeps flapping — why?" Because it's missing stabilization windows, so brief metric swings trigger immediate opposite scaling.

One pod on a shared node starts consuming most of its CPU/memory, and every other pod on that node slows down — even ones from a completely different team.

What is actually happening (the mental model).

Without resource requests and limits, pods compete for a node's finite resources with no fairness guarantee, and Kubernetes has no way to prioritise or contain one workload over another — the classic "noisy neighbour" problem, now at cluster scale instead of a single Docker host.

How to reason about it.

1. Set both requests and limits on every workload — requests inform scheduling and fairness, limits cap worst-case consumption.
2. Understand QoS classes follow directly from this: Guaranteed (requests = limits on all resources) is evicted last under pressure; Burstable (requests set, limits higher or unset) is in the middle; BestEffort (neither set) is evicted first.
3. For genuinely latency-sensitive workloads, this is where you'd also reach for node affinity/taints to physically separate them from bursty or untrusted workloads rather than relying on limits alone.
4. Diagnose an active incident with kubectl top pods/top nodes to identify the actual offender before changing anything.

The root causes and trade-offs.

Setting no requests or limits at all gives you maximum flexibility but zero fairness between workloads. Setting them everywhere costs some upfront tuning effort, but it buys real isolation and a predictable, QoS-based eviction order. For shared, multi-tenant nodes, the trade-off strongly favours setting them.

The trap that less-experienced engineers fall into.

Leaving requests/limits unset "to keep things flexible," not realising that's exactly what makes one workload able to starve every neighbour on the node with no mechanism to stop it.

🎯 Interviewer follow-up questions you should expect.

• "How do you prevent one pod starving others on a node?" You set requests and limits on every workload, and understand the resulting QoS class.
• "What are the three QoS classes?" Guaranteed, where requests equal limits; Burstable, where some are set; and BestEffort, where none are set — that's also the eviction order under pressure.

Pods are landing on the wrong nodes (a general workload lands on a specialised GPU node pool), or a dedicated node pool sits empty when it should be in use.

What is actually happening (the mental model).

Taints (on nodes) repel pods unless the pod has a matching toleration; affinity (on pods) attracts pods toward or away from nodes with certain labels. They solve related but different problems — tainting a node stops the wrong workloads from landing there uninvited; affinity expresses a preference from the pod's side — and mixing them up is the most common source of scheduling surprises.

How to work through it.

1. A dedicated node pool should be tainted so ordinary pods (with no toleration) can never land there by accident, and the pods that belong there should have both the toleration (permission to land) and node affinity/selector (actual preference to land there) — a toleration alone doesn't attract a pod, it only permits it.
2. If a node pool sits empty, check whether pods have the toleration but nobody gave them the affinity actually pointing at it — with only a toleration, the scheduler has no reason to prefer that node.
3. If a pod lands somewhere unwanted, check its affinity/selector for being too loose, or check whether the node was ever tainted at all.
4. Use requiredDuringScheduling (hard) vs preferredDuringScheduling (soft) affinity deliberately — a soft preference can be silently ignored if it can't be satisfied.

The root causes and trade-offs.

A toleration without affinity permits a pod to land on the node but doesn't attract it there, so the node often just sits empty. Affinity without a matching taint attracts the pod, but nothing stops other, unrelated pods from landing there too. And a soft affinity preference can silently go unhonoured whenever it can't actually be satisfied. The fix is using a taint and toleration together with affinity, deliberately, for any genuinely dedicated node pool.

The trap that less-experienced engineers fall into.

Adding a toleration and assuming that alone routes the pod to the tainted node — a toleration only removes the repulsion, it creates no attraction; without matching affinity, the pod might land anywhere else entirely.

🎯 Interviewer follow-up questions you should expect.

• "Difference between taints/tolerations and affinity?" Taints repel unless tolerated, which is a permission; affinity attracts or repels based on preference — you usually need both for a truly dedicated node pool.
• "A pod has the toleration but the dedicated node is empty — why?" Because a toleration only permits landing there, it doesn't attract the pod — it needs matching affinity too.

kubectl drain on a node hangs — it won't evict some pods, and the drain command sits there or errors out.

What is actually happening (the mental model).

A PodDisruptionBudget (PDB) tells Kubernetes the minimum number/percentage of a workload's pods that must stay available during voluntary disruptions like a drain — if evicting a pod would violate the PDB (e.g. a Deployment with minAvailable already at its floor), the eviction is refused, and the drain stalls on that pod.

How to work through it.

1. Identify which pod the drain is stuck on and check its PDB (kubectl get pdb) — is minAvailable/maxUnavailable set so tightly that even one eviction would violate it?
2. Check whether the Deployment actually has enough healthy replicas right now to satisfy the PDB during the drain — a PDB that's fine in normal times can block a drain if some replicas are already unhealthy for an unrelated reason.
3. This is a real safety mechanism, not a bug — the fix is usually to ensure enough healthy replicas exist before draining, or to accept a temporarily wider PDB for planned maintenance, rather than forcing the eviction.
4. As a last resort kubectl delete pod --force bypasses the PDB, but that reintroduces exactly the disruption risk the PDB exists to prevent — use it deliberately, not as a default unblock.

The root causes and trade-offs.

Sometimes the PDB is set correctly, but the workload currently has fewer healthy replicas than usual because of some unrelated issue compounding on top. Other times the PDB itself was simply set unrealistically tight for the actual replica count. Forcing past it trades a controlled disruption for an uncontrolled one, which is rarely a good trade.

The trap that less-experienced engineers fall into.

Force-deleting the blocked pod to "just get the drain done," without checking why the PDB is blocking it — sometimes it's correctly protecting against a real availability drop that the force-delete then causes anyway.

🎯 Interviewer follow-up questions you should expect.

• "A drain is stuck — what's likely blocking it?" A PodDisruptionBudget refusing an eviction that would drop availability below its floor.
• "Is it safe to force past a PDB?" Only once you understand why it's blocking — forcing it defeats the exact protection the PDB is providing.

A node shows NotReady in kubectl get nodes, and its pods are either stuck or being rescheduled elsewhere.

What is actually happening (the mental model).

NotReady means the control plane has stopped hearing healthy heartbeats from that node's kubelet — the node itself might be down, the kubelet might have crashed, or it might just be a network partition between the node and the control plane, and Kubernetes can't tell which from the status alone.

How to work through it.

1. kubectl describe node <name> — the Conditions section usually narrows it down: disk pressure, memory pressure, or a kubelet that's stopped reporting entirely.
2. If you have host access, check the kubelet's own status/logs on that node (systemctl status kubelet, its logs) — is it running at all?
3. Check basic node health: disk full, out of memory, a kernel/container-runtime issue, or a network problem isolating it from the control plane.
4. Kubernetes will, after a timeout, start evicting the node's pods to reschedule elsewhere — know your pod-eviction-timeout/tolerations so you understand when that kicks in, especially for StatefulSets with attached volumes (see the reattach scenario above).

Root causes, and how to tell them apart.

Disk/memory pressure conditions shown explicitly → resource exhaustion on the node itself. No conditions updating at all, kubelet unresponsive → the kubelet process or the node itself is down. Node responds to ping/SSH but still NotReady → likely a network path issue specifically between node and control plane, or a certificate/auth problem for the kubelet.

The trap that less-experienced engineers fall into.

Immediately rebooting or replacing the node without checking describe node Conditions first — sometimes it's a fixable resource-pressure issue, and other times replacing the node is exactly right, but you want to know which before acting.

🎯 Interviewer follow-up questions you should expect.

• "A node is NotReady — first thing you check?" describe node Conditions, for disk or memory pressure and kubelet status, then the kubelet's own logs if you have host access.
• "What happens to its pods?" After a timeout, Kubernetes reschedules them elsewhere, with extra care needed for StatefulSet pods that have attached volumes.

You need to take a node out of service (maintenance, decommission, an upgrade) without disrupting the applications running on it.

What is actually happening (the mental model).

kubectl drain combines two steps: cordon (mark the node unschedulable, so no new pods land there) and evict (gracefully remove the pods already there, respecting PodDisruptionBudgets), so that existing workloads move elsewhere in a controlled way rather than being yanked out from under you.

How to work through it (the safe sequence).

1. kubectl cordon <node> first if you want to stop new scheduling without evicting yet (useful to observe before disrupting anything).
2. kubectl drain <node> --ignore-daemonsets --delete-emptydir-data — evicts pods respecting PDBs; DaemonSet pods are excepted by design (they're meant to run on every node) and emptyDir data is expected to be lost.
3. If it stalls, it's very likely a PodDisruptionBudget blocking eviction — see that scenario rather than forcing past it blindly.
4. Confirm the workloads rescheduled cleanly elsewhere before actually decommissioning the node (kubectl delete node, or terminating the instance).

The root causes and trade-offs.

Draining is inherently a trade-off between speed and safety. Cordoning, then draining, then verifying is slower, but it stays controlled the whole way. Forcing an eviction past PDBs or DaemonSet warnings is faster, but it reintroduces exactly the disruption risk that draining exists to avoid.

The trap that less-experienced engineers fall into.

Deleting or terminating the node directly without draining first — pods don't get a graceful eviction, in-flight requests can be cut off, and StatefulSet volume-reattach issues (see above) become far more likely.

🎯 Interviewer follow-up questions you should expect.

• "Walk me through safely taking a node out of service." You cordon it to stop new scheduling, drain it to gracefully evict while respecting PDBs, verify the workloads rescheduled, then decommission it.
• "Why does drain sometimes need --ignore-daemonsets?" Because DaemonSet pods are meant to run on every node by design, and aren't meant to be evicted by a drain.

You need to upgrade the cluster's Kubernetes version — control plane and nodes — without an outage, and without breaking workloads that depend on removed APIs.

What is actually happening (the mental model).

A Kubernetes upgrade is really two upgrades — the control plane and the nodes — that must happen in order (control plane first), plus a separate, often-forgotten risk: each release can deprecate and remove API versions, so manifests using a removed apiVersion will fail to apply on the new version even if nothing else changed.

How to reason about a good setup.

1. Check for deprecated/removed APIs first — tools like kubectl-convert/pluto scan manifests for APIs that won't exist on the target version, so you fix them before upgrading, not after something breaks.
2. Upgrade the control plane before any nodes, one minor version at a time (Kubernetes doesn't support skipping minor versions) — a managed service (EKS/GKE/AKS) or kops handles much of this orchestration for you.
3. Upgrade nodes via cordon → drain → replace (or in-place upgrade) → uncordon, one at a time or in small batches, never all at once.
4. Test in a lower environment first with the same version jump, and have a rollback/redeploy plan for the previous node AMI/image if something goes wrong.

The root causes and trade-offs.

One cause is skipping the deprecated-API check, which can break on apply and sometimes isn't discovered until the upgrade is already underway. Another is upgrading the nodes before the control plane, which creates an unsupported version skew. A third is upgrading everything at once instead of in batches, which creates a large blast radius if anything turns out to be wrong. The safe path costs more time and process, in exchange for a dramatically lower risk of an upgrade-day outage.

The trap that less-experienced engineers fall into.

Upgrading straight to the latest version without checking API deprecations first — discovering only after the upgrade that a Deployment or Ingress manifest uses an apiVersion that no longer exists.

🎯 Interviewer follow-up questions you should expect.

• "How do you plan a cluster upgrade safely?" You check for deprecated APIs first, upgrade the control plane before the nodes, move one minor version at a time, and upgrade nodes in batches with cordon and drain.
• "Why can't you skip minor versions?" Because Kubernetes only supports upgrading one minor version at a time, due to its API and skew compatibility guarantees.

Leadership asks: if etcd were lost or corrupted right now, how would you get the cluster back, and how long would it take?

What is actually happening (the mental model).

etcd holds the entire state of the cluster — every object, every secret, everything kubectl get could ever show you. Losing it without a backup doesn't just lose some data, it loses the cluster's entire memory of what it's supposed to be running; a snapshot-based backup strategy is non-negotiable for anything beyond a disposable dev cluster.

How to reason about setting it up.

1. Take regular etcd snapshots (etcdctl snapshot save) on a schedule, and store them off the control-plane nodes — a backup sitting on the same disk as the thing it backs up isn't a real backup.
2. Test the restore process before you need it for real — a backup you've never restored from is a hope, not a plan; practice on a throwaway cluster.
3. Understand a restore effectively means the control plane comes back to the state at snapshot time — anything created/changed since then is gone, so snapshot frequency directly determines your recovery point.
4. For managed Kubernetes (EKS/GKE/AKS), the cloud provider manages etcd for you — know whether that's the case for your cluster, since it changes who's actually responsible for this.

The root causes and trade-offs.

Having no backup strategy at all means a total loss the moment etcd fails. Regular snapshots with genuinely tested restores give you a recovery point objective you can actually state and defend. The trade-off is maintaining backup infrastructure and running periodic restore drills, in exchange for a real disaster-recovery story instead of a hopeful one.

The trap that less-experienced engineers fall into.

Having a backup script that runs on a schedule but has never once been used to actually restore anything — discovering it's broken or incomplete only during a real incident, when it's too late to fix calmly.

🎯 Interviewer follow-up questions you should expect.

• "How do you back up a self-managed cluster's state?" With a scheduled etcdctl snapshot save, stored off the control-plane nodes, with restores actually tested.
• "What's your recovery point if etcd is lost?" Whatever changed since the last snapshot, which is exactly why snapshot frequency matters and untested backups are a real risk.

A kubectl apply that looks completely correct is rejected with an error pointing at a webhook, or every deployment in the cluster is suddenly failing at once.

What is actually happening (the mental model).

Admission webhooks (validating and mutating) sit in the request path before an object is persisted to etcd — they're how policy tools (OPA/Gatekeeper, Kyverno, cert-manager, service meshes) enforce or modify rules cluster-wide. But that means if the webhook's own backend is down, slow, or misconfigured, it can block every matching request cluster-wide, not just the one you're looking at.

How to work through it (when a pod is rejected).

1. Read the rejection error carefully — it usually names the specific webhook and the policy/reason it failed, if the webhook is behaving correctly and just enforcing a real rule.
2. If the error instead looks like a timeout or connection failure to the webhook itself, the webhook's backing service/pod is the actual problem — check its health, not your manifest.
3. kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations — confirm what's actually registered and its failurePolicy (Fail blocks everything if the webhook is unreachable; Ignore lets requests through, which is its own risk).
4. In a genuine emergency, a webhook can be temporarily disabled/scoped down to unblock the cluster — but treat that as a stopgap, not a fix, and re-enable it once the underlying service is healthy.

The root causes and trade-offs.

Sometimes this is a correctly-enforced policy rejecting a genuinely non-compliant manifest, and the fix is simply fixing the manifest. Other times the webhook's own backend is down or slow while it's configured with failurePolicy: Fail, which blocks the entire cluster until the webhook service itself is fixed. And with failurePolicy: Ignore, a down webhook instead lets bad objects through silently during an outage, which is a different, quieter risk to watch for.

The trap that less-experienced engineers fall into.

Assuming their own manifest must be wrong and endlessly tweaking it, when the real problem is the webhook's backend being unreachable — the fix is restoring the webhook service, not further editing an already-correct manifest.

🎯 Interviewer follow-up questions you should expect.

• "Every deployment in the cluster suddenly fails — what do you suspect?" An admission webhook whose backend is down, with failurePolicy: Fail blocking everything.
• "Fail vs Ignore failurePolicy — what's the trade-off?" Fail is safe, but it can block the whole cluster if the webhook is down; Ignore keeps the cluster moving, but it silently skips enforcement during an outage.

"Make this workload production-ready" — it's deployed and technically Running, but what's actually missing before it can survive a real node failure or a bad day?

What is actually happening (the mental model).

"Deployed and Running" only means the happy path works right now — production-readiness is a checklist, and every item on it exists to remove one specific, realistic failure mode (a node dying, an AZ outage, a bad deploy, silent degradation) rather than being generic best-practice box-ticking.

The checklist (each item removes a specific failure mode).

1. At least 2–3 replicas, so no single pod is a SPOF.
2. Spread across nodes and AZs with topology spread constraints/anti-affinity, so one node or AZ loss doesn't take out every replica at once.
3. Requests and limits set, giving it a real QoS class and preventing it from starving or being starved by neighbours.
4. Accurate readiness and liveness probes, so traffic only reaches instances that can truly serve it, and hung instances actually get restarted.
5. Graceful shutdown (handles SIGTERM, drains in-flight work) so rollouts and node drains don't drop requests.
6. A PodDisruptionBudget, so voluntary disruptions (drains, upgrades) can't take out too many replicas at once.
7. An HPA (or planned capacity) for real load variation, not a fixed replica count sized for an average day.
8. Observability — metrics, logs, and alerts wired up — so degradation is visible before a customer reports it, not discovered after.

The bigger question a senior asks.

Is 3 replicas actually enough, or is it "3 replicas that all happen to be on the same node"? Every item above only counts if it's verified, not just declared in YAML — a topology spread constraint that's never been tested against an actual node failure is a hope, not a guarantee.

The trap that less-experienced engineers fall into.

Equating "deployed and Running" with "production-ready" — missing PDBs, spread, graceful shutdown, and observability, so the workload works fine until the first node failure, AZ event, or rollout, at which point every gap becomes an incident at once.

🎯 Interviewer follow-up questions you should expect.

• "Make this workload production-grade — what do you add?" The checklist above, with each item tied to a specific failure mode.
• "You have 3 replicas — is that enough?" Only if they're actually spread across nodes and AZs — you check the topology, you don't assume it.
• "How do you know it's actually production-ready, not just declared as such?" You verify each control under a simulated failure — kill a node, kill a pod, run a drain — rather than trusting the YAML alone.