A bloated image that's slow to pull, slow to deploy, expensive to store, and a large attack surface. "Why is our image 2 GB?"

What is actually happening (the mental model).

The image bundles compilers, dev dependencies, source, and build cache alongside the runtime — most of which the running app never touches. A multi-stage build separates the concerns: a builder stage with everything needed to compile, and a lean runtime stage that COPY --froms only the compiled artifact.

How to work through it.

1. Multi-stage build — builder stage compiles; final stage FROM a minimal/distroless/alpine base and copies only the artifact.
2. Minimise layers and combine RUN commands where it reduces cache-busting, and clean up package-manager caches in the same layer they were created.
3. Use .dockerignore so the build context (and accidentally, the image) doesn't pick up .git, node_modules, local env files.
4. Measure with docker history/dive to see which layer is actually heavy before guessing.

The root causes and trade-offs.

One cause is a single-stage Dockerfile that ships all its build tooling into production. Another is having no .dockerignore file at all. A third is using a verbose base image, like full Ubuntu or Debian, where something like alpine or distroless would do the job just as well. The trade-off is a slightly less familiar debugging environment, since you lose some shell tools, in exchange for major wins in size and security.

The trap that less-experienced engineers fall into.

Treating image size as unavoidable ("that's just how big the app is") instead of separating build-time and run-time concerns — or switching to alpine and being surprised by subtle glibc/musl compatibility issues without testing.

🎯 Interviewer follow-up questions you should expect.

• "How do you shrink a bloated image?" You use a multi-stage build with a minimal or distroless base, add a .dockerignore, and measure with dive or docker history.
• "Alpine vs distroless vs full OS?" Alpine is small but its musl-libc can subtly break some binaries; distroless has no shell at all, so it's harder to debug but more secure; a full OS is easiest to debug, but largest and least secure.
• "Why not just delete files at the end of a RUN?" Because layers are additive — deleting in a later layer doesn't remove the bytes from the image, so you either must not create them in the final image's layers at all through a multi-stage build, or delete them within the same layer they were added.

Every CI build reinstalls all dependencies from scratch even though only one source file changed. Builds that should take seconds take minutes.

What is actually happening (the mental model).

Docker caches each layer, but invalidates it — and every layer after it — the moment anything in that instruction changes. If COPY . . (the whole source tree, including a one-line change) comes before RUN npm install, then every source change invalidates the dependency-install layer too, even though dependencies didn't change.

How to work through it.

1. Order instructions by change frequency — least-frequently-changed first. Copy dependency manifests (package.json, requirements.txt) and install before copying the rest of the source.
2. Separate dependency install from source copy so the expensive install layer stays cached across ordinary code changes.
3. Use BuildKit cache mounts for package-manager caches so even a cache-miss install doesn't re-download everything from the network.

Root causes, and how to tell them apart.

Wrong instruction order (source copied before dependency install) is the classic cause. If dependency installs are slow even on a genuine cache hit, that's a different problem — a slow registry/mirror, not cache ordering.

The trap that less-experienced engineers fall into.

COPY . . as the very first instruction "to keep it simple" — which silently busts the cache on every single source change, including whitespace and comments.

🎯 Interviewer follow-up questions you should expect.

• "Why does changing one file reinstall all dependencies?" Because the source copy happened before the install layer, so any source change invalidates it.
• "How do you fix it?" You copy the manifest files and install dependencies first, then copy the rest of the source afterward.

A docker build takes ages just to start ("Sending build context to Docker daemon... 900MB"), and you later find a .git folder or local .env file baked into the image.

What is actually happening (the mental model).

Before Docker executes a single instruction, it packages the entire build context (the directory you pointed docker build at) and sends it to the daemon. Without a .dockerignore, that includes .git, node_modules, local secrets, and anything else sitting in the folder — slow to send, and easy to accidentally COPY into the image.

How to work through it.

1. Add a .dockerignore excluding .git, node_modules, local env files, build artifacts — mirror your .gitignore as a starting point.
2. Keep the build context minimal — build from a directory that only contains what the image actually needs, rather than the whole repo.
3. Audit what's actually in the image with docker history/dive if you suspect something leaked in already.

The root causes and trade-offs.

No .dockerignore, or a build context set too broad (the repo root instead of a scoped subfolder). The fix is nearly free — a text file — so there's no real trade-off.

The trap that less-experienced engineers fall into.

Assuming COPY only takes what you explicitly reference — it's easy to COPY . . and forget that "." includes everything in the build context, secrets and all.

🎯 Interviewer follow-up questions you should expect.

• "Why is my build context huge?" Because there's no .dockerignore, so .git, node_modules, and build artifacts are all being sent to the daemon.
• "How did a secret end up in the image?" It was sitting in the build context and got swept up by a broad COPY . ..

An image built and tested fine on an Apple Silicon laptop fails to start (or silently misbehaves) once deployed to a standard x86_64 server.

What is actually happening (the mental model).

Docker images are architecture-specific. A Mac with Apple Silicon builds arm64 images by default; most cloud servers run amd64. If you never specified a target platform, you built an image for your own machine's architecture, not the one it's actually going to run on.

How to work through it.

1. Build for the target platform explicitly — docker buildx build --platform linux/amd64 (or whatever your servers run).
2. For multi-architecture support, use buildx to build a multi-arch manifest so the same tag resolves correctly on both arm64 and amd64 hosts.
3. Make platform an explicit, documented part of the CI pipeline rather than relying on whatever architecture the build machine happens to be.

The root causes and trade-offs.

Building on a different CPU architecture than production, with no explicit --platform flag. The fix costs a slightly slower/cross-compiled build in exchange for correctness.

The trap that less-experienced engineers fall into.

Not realising a laptop's CPU architecture affects the built image at all — chasing a "flaky" runtime bug that's actually just the wrong binary format for the target host.

🎯 Interviewer follow-up questions you should expect.

• "Image works locally, fails on the server — first thing you check?" A CPU architecture mismatch, like an arm64 build laptop versus an amd64 server — you check it with --platform.
• "How do you support both architectures?" You use buildx to produce a multi-arch manifest under one tag.

A security review flags that your application containers run as root inside the container — if the app is compromised, the attacker has root in that container's namespace.

What is actually happening (the mental model).

By default, unless a Dockerfile says otherwise, the process runs as root inside the container. That's not the same as root on the host, but it still means a container-breakout vulnerability hands the attacker root, and a broad set of accidental capabilities (writing anywhere in the container's filesystem, binding low ports, and more) that the app never actually needs.

How to work through it.

1. Create a non-root user in the Dockerfile and switch to it with USER before the app runs.
2. Drop unnecessary Linux capabilities at run time (--cap-drop=ALL, add back only what's needed) and consider --read-only for the root filesystem where the app doesn't need to write to it.
3. In Kubernetes, enforce this centrally with a Pod Security Standard / admission policy rather than trusting every Dockerfile to get it right.

The root causes and trade-offs.

Root is simply the Docker default and nobody opted out. The trade-off is occasionally needing to fix file-permission issues (a non-root user can't write where root could), which is a one-time Dockerfile fix, not an ongoing cost.

The trap that less-experienced engineers fall into.

Assuming container isolation alone is "good enough" security and that root-inside-the-container doesn't matter because "it's just a container" — ignoring that container escapes and shared-kernel vulnerabilities are real.

🎯 Interviewer follow-up questions you should expect.

• "Why does root-in-container matter if it's isolated?" Because container isolation isn't absolute — the kernel is shared, and escape vulnerabilities exist — so root inside the container multiplies the blast radius if that isolation is ever broken.
• "How do you enforce non-root across a whole cluster?" With Pod Security Standards or admission control, rather than relying on every Dockerfile individually.

An API key was set with ENV or copied in during the build, then "removed" in a later layer — but it's still recoverable from the image's history.

What is actually happening (the mental model).

Image layers are additive and permanent. A secret written in one layer still exists in that layer even if a later layer deletes the file — docker history or simply extracting the layer tarballs recovers it. "Removing" a secret in a later RUN only hides it from the final filesystem view, not from the image itself.

How to work through it.

1. Never ENV/COPY a real secret into any layer — treat every layer as public once the image is pushed anywhere.
2. Use BuildKit secret mounts (--mount=type=secret) which make a secret available only during a single RUN and never persist it in a layer.
3. Inject secrets at runtime instead of build time — environment variables or mounted files supplied by the orchestrator (Kubernetes Secrets, a vault), not baked into the image.
4. If a secret has already leaked into a pushed image, rotate the secret — treat the image history as compromised, don't just rebuild without the line.

The root causes and trade-offs.

Treating build-time ARG/ENV as if it were ephemeral, when it's actually baked into a permanent layer. The fix (runtime injection, BuildKit secret mounts) costs a little pipeline setup for a real security guarantee.

The trap that less-experienced engineers fall into.

Deleting a secret file in a later Dockerfile line and believing that's sufficient — not realising layers are cumulative and the earlier layer with the secret is still part of the image.

🎯 Interviewer follow-up questions you should expect.

• "Is deleting a secret in a later layer enough?" No — the earlier layer still contains it, and anyone with the image can extract it.
• "How do you use a secret during build without baking it in?" With BuildKit's --mount=type=secret.
• "A secret leaked into a pushed image — now what?" You rotate the secret immediately, and treat it as compromised, not just fixed by a rebuild.

A scanner flags dozens of CVEs in your base image, or leadership asks how you know an image pulled from a registry is actually what it claims to be.

What is actually happening (the mental model).

Every layer you didn't write yourself — the base image, and anything it pulled in — is part of your supply chain, and any of it can carry known vulnerabilities or be tampered with. Minimal base images reduce the surface; scanning and provenance catch what's left.

How to work through it.

1. Scan images in CI (Trivy, Grype, Snyk) and fail the build on criticals, not just report them after the fact.
2. Prefer minimal/distroless base images to shrink the attack surface in the first place.
3. Pin base image versions/digests rather than floating tags like latest, so a base image can't silently change under you.
4. For provenance, use image signing (Cosign/Notary) so you can verify an image actually came from your pipeline and hasn't been tampered with.

The root causes and trade-offs.

One cause is a large base image with many packages, which increases your CVE exposure simply through surface area. Another is floating tags with no version pinning. A third is having no scanning gate in CI at all. The trade-off is some added friction in CI, from scanning and signing, in exchange for real supply-chain assurance.

The trap that less-experienced engineers fall into.

Scanning only in a dashboard nobody reads, rather than gating the build/deploy pipeline on scan results — or chasing every low-severity CVE while ignoring that the base image itself is needlessly large.

🎯 Interviewer follow-up questions you should expect.

• "How do you catch vulnerable images before they ship?" You scan them in CI, with a tool like Trivy, Grype, or Snyk, and fail the build on anything critical.
• "How do you know an image wasn't tampered with?" Through image signing, with Cosign or Notary, plus pinned digests rather than floating tags.

You run a container and it exits within a second, or in Kubernetes it shows CrashLoopBackOff — the app "should" be running, but the container won't stay up.

What is actually happening (the mental model).

A container's life is tied to its PID 1 — the moment that process exits, the container exits, no matter what else might still be running. The two classic causes: the intended long-running process actually finished or crashed immediately (a config error, a missing dependency, a foreground-vs-background mistake), or the entrypoint script itself exits after doing setup work instead of handing control to the real server process.

How to work through it.

1. Check the exit code and logs — docker logs / docker inspect (the .State.ExitCode) tell you whether it crashed or exited cleanly.
2. Confirm the main process is meant to run in the foreground — a common mistake is starting a service in daemon/background mode inside the container, after which the entrypoint script has nothing left to do and exits.
3. If using an entrypoint script, make sure it ends by exec-ing the real server process (so it becomes PID 1), not launching it in the background and falling off the end of the script.

The root causes and trade-offs.

Sometimes this is a genuine application-level failure, like bad configuration or a missing dependency. Other times it's simply a foreground-versus-background mistake in how the process was started inside the container. The logs and the exit code together tell the two apart quickly.

The trap that less-experienced engineers fall into.

Backgrounding the main process inside the container (service nginx start style) instead of running it in the foreground — the container exits the instant the launcher script finishes, even though the service technically "started."

🎯 Interviewer follow-up questions you should expect.

• "A container exits immediately — first thing you check?" You check docker logs and the exit code, then whether the main process is actually running in the foreground.
• "Why does the container die even though the service 'started'?" Because the service was launched in the background, so PID 1, the launcher, exited once its own work was done.

A container takes ages to stop (always hitting the hard kill timeout), or you notice zombie processes accumulating inside long-running containers.

What is actually happening (the mental model).

PID 1 has special responsibilities in Linux that most application processes were never written to handle: forwarding signals like SIGTERM to child processes, and reaping zombie (exited-but-not-collected) child processes. Many applications ignore SIGTERM when running as PID 1, so docker stop waits out the grace period and then hard-kills with SIGKILL — an ungraceful shutdown.

How to work through it.

1. Use a minimal init process as PID 1 — Docker's built-in --init flag (which runs tini) — so signal forwarding and zombie reaping are handled correctly, and your app runs as a normal child process.
2. Make sure your application actually handles SIGTERM for graceful shutdown (closing connections, finishing in-flight work) rather than relying on the hard-kill timeout.
3. If using a shell-script entrypoint, remember CMD in shell form doesn't forward signals well — prefer exec form or exec the final process.

The root causes and trade-offs.

Running an app directly as PID 1 without an init process, and/or the app not handling SIGTERM. The trade-off of adding --init is essentially free — it's a tiny, purpose-built binary.

The trap that less-experienced engineers fall into.

Not realising PID 1 is special at all — assuming any process can be PID 1 with no consequences, then being confused why docker stop always takes the full grace period before hard-killing.

🎯 Interviewer follow-up questions you should expect.

• "Why does docker stop always hang and hard-kill?" Because the app running as PID 1 isn't handling or forwarding SIGTERM — you add --init and make sure the app handles graceful shutdown.
• "What does PID 1 have to do that other processes don't?" It has to forward signals to its children and reap any zombie processes.

A database or file the app wrote inside the container is simply gone after the container restarted or was recreated — nobody deleted it, it just disappeared.

What is actually happening (the mental model).

A container's writable layer is ephemeral — it exists only as long as that specific container does. Restart, recreate, or reschedule the container (all completely normal operations) and the old writable layer, along with anything written to it, is gone. Containers are meant to be treated as disposable "cattle," not long-lived "pets" with state living inside them.

How to work through it.

1. Identify anything that needs to survive the container's lifetime — data, uploads, a database's files.
2. Move it to a Docker volume (or a bind mount) so it lives outside the container's writable layer and persists across recreation.
3. For anything that must survive the host too (not just the container), use external, durable storage — a managed database, object storage, a network filesystem — rather than any local Docker storage.

The root causes and trade-offs.

Writing state directly into the container's own filesystem instead of a volume. The fix (mount a volume) costs nothing meaningful and should be the default for anything stateful.

The trap that less-experienced engineers fall into.

Treating a container like a small persistent server and being surprised when routine operations (a redeploy, a reschedule, a crash-restart) wipe out data that was never actually persisted outside the container.

🎯 Interviewer follow-up questions you should expect.

• "Why did our data disappear after a restart?" Because it was written to the container's own writable layer, which is ephemeral — it needed to be on a volume instead.
• "How do you persist data across container restarts?" With Docker volumes, or bind mounts, for container-lifetime persistence, and external durable storage if the data must outlive the host too.

One container on a host starts consuming most of the CPU or memory, and every other container on the same host slows down or gets OOM-killed — with no limits set, Docker lets any container use as much as the host has.

What is actually happening (the mental model).

Without explicit resource limits, a container can consume the entire host's CPU and memory, starving its neighbours — the classic "noisy neighbour" problem. Worse, a runtime that isn't cgroup-aware (an older JVM or Node process, for example) may look at the host's total memory/CPU to size its own heap or thread pool, not the much smaller limit actually granted to its container — and get OOM-killed even though it "thinks" it has room.

How to work through it.

1. Set explicit CPU and memory limits (--memory, --cpus, or their Kubernetes equivalents) on every container so one workload can't starve others.
2. Make sure the runtime is cgroup-aware — modern JVM/Node versions read the container's cgroup limits automatically; older ones may need explicit flags (e.g. -Xmx sized to the container limit, not the host).
3. Set requests as well as limits (in Kubernetes) so the scheduler places workloads sanely rather than over-packing a host.

The root causes and trade-offs.

No resource limits at all (the default), or limits set but the runtime inside isn't cgroup-aware and mis-sizes itself anyway. The trade-off of setting limits is occasionally OOM-killing your own container if the limit is set too tight — which is a tuning problem, not a reason to skip limits.

The trap that less-experienced engineers fall into.

Setting a memory limit on the container but leaving the JVM/Node heap sizing on its old defaults, which size against the host's memory — so the app OOMs well before it appears to be near the container's own limit.

🎯 Interviewer follow-up questions you should expect.

• "One container is starving the others on a host — what's missing?" Resource limits — you need to add CPU and memory limits per container.
• "We set a memory limit but the app still OOMs early — why?" Because the runtime isn't cgroup-aware, and it's sizing itself against the host's memory rather than the container's limit.

A frontend container can't reach a backend container that's supposedly "right there" — often because the code tries to reach it on localhost.

What is actually happening (the mental model).

localhost inside a container refers to the container itself, never the host and never a sibling container, no matter how "close" they seem. Two containers only reach each other by name if they're on the same user-defined network, where Docker's embedded DNS resolves container names to their IPs.

How to work through it.

1. Put the containers on the same user-defined bridge network (not just the default bridge, which doesn't do automatic name resolution the same way).
2. Have the frontend reach the backend by container/service name, not localhost and not a hardcoded IP (container IPs change on recreation).
3. Verify with docker network inspect and a quick docker exec ping/curl by name from one container to the other.

The root causes and trade-offs.

Using localhost or a hardcoded IP where a container/service name was needed, or the two containers simply aren't on the same network. The fix is just correct network configuration — no real trade-off.

The trap that less-experienced engineers fall into.

Reaching for localhost out of habit from running processes directly on a machine — forgetting that each container has its own network namespace and its own idea of what localhost means.

🎯 Interviewer follow-up questions you should expect.

• "Frontend can't reach backend — what's the first thing you check?" Whether the code is using localhost instead of the backend's container or service name, and whether they're on the same user-defined network.
• "Why not just use the container's IP?" Because container IPs change whenever the container is recreated, while names resolved through Docker's embedded DNS stay stable.

You need to inspect what's happening inside a running container, but it's a distroless image with no shell — docker exec ... sh just fails.

What is actually happening (the mental model).

Distroless images deliberately have no shell, no package manager, and almost no tooling — that's the whole point, security-wise. So the usual "exec in and poke around" debugging workflow doesn't exist for these images by design, and you need a different approach that doesn't undo the security benefit permanently.

How to work through it.

1. Use docker debug (or kubectl debug with an ephemeral container in Kubernetes) to attach a separate debugging toolbox to the running container's namespaces, without modifying the production image itself.
2. Lean more heavily on logs and metrics emitted by the app itself — structured logging and observability reduce how often you need to get inside a container at all.
3. If you must, keep a separate debug build with a shell for local troubleshooting — but never ship that build to production.

The root causes and trade-offs.

The lack of a shell is intentional (security), not an oversight — the trade-off you're managing is security vs debuggability, and the answer is external debug tooling, not adding a shell back into the production image.

The trap that less-experienced engineers fall into.

"Fixing" the problem by adding a shell and debugging tools back into the production image — quietly reversing the entire security benefit of going distroless in the first place.

🎯 Interviewer follow-up questions you should expect.

• "How do you debug a distroless container with no shell?" With docker debug or an ephemeral debug container that attaches without modifying the image, and you invest in logs and metrics so you need this less often.
• "Why not just add a shell for convenience?" Because that undoes the entire security reason for going distroless in the first place.

A host running Docker for a while gradually runs out of disk space, even though nobody's actively creating huge files — builds and deploys eventually start failing with "no space left on device."

What is actually happening (the mental model).

Docker accumulates artifacts over time that are easy to forget about: old/unused images, stopped containers, dangling (untagged) layers from repeated builds, unused volumes, and build cache. None of it gets removed automatically by default, so a host that builds images regularly just keeps growing.

How to work through it.

1. Diagnose first — docker system df shows exactly where the space is going (images, containers, volumes, or build cache).
2. Clean up routinely — docker system prune (and --volumes if unused volumes are safe to remove) rather than only reactively when the disk is already full.
3. Prevent recurrence — set up a scheduled prune/cleanup job on build hosts, and avoid tagging every build uniquely forever without a retention policy in your registry too.

The root causes and trade-offs.

No routine cleanup policy for images, containers, volumes, and build cache. The trade-off with aggressive pruning is losing cached layers you might have reused — balance cleanup frequency against how much it slows the next build.

The trap that less-experienced engineers fall into.

Only discovering the problem when a build fails from a full disk, then pruning reactively — instead of monitoring disk usage and cleaning up on a schedule before it becomes an incident.

🎯 Interviewer follow-up questions you should expect.

• "A build host slowly fills up its disk — what's accumulating?" Old images, stopped containers, dangling layers, unused volumes, and build cache.
• "How do you find out where the space went?" With docker system df, followed by a targeted or full docker system prune.

A team is running their production application with a single docker compose up on one VM. Is that a reasonable production setup, or a problem waiting to happen?

What is actually happening (the mental model).

Compose is excellent for local dev and simple single-host setups, but it lacks self-healing, rolling updates, multi-host scheduling, and HA. A single-VM Compose production is a single point of failure — the box dies, the service dies. Whether the answer is Kubernetes, ECS, or something lighter depends on scale and team maturity; a senior names the reliability gap and right-sizes the orchestration to the actual availability requirement (not reflexively pushing everyone to K8s).

How to reason about it.

1. Compose is great for local dev and low-stakes single-host services.
2. Its gaps for prod: no self-healing (a crashed container stays down), no rolling deploys, no multi-host scheduling, no HA — a single-VM Compose prod is a SPOF.
3. Right-size the orchestration: Kubernetes/ECS/Nomad (or Swarm for simple needs) — matched to scale and team maturity. Don't cargo-cult K8s (it's its own operational tax), but do name the reliability gap.

The root causes and trade-offs.

Running Compose on a single VM is simple, but it gives you no high availability and no self-healing if that VM goes down. A real orchestrator is resilient, but it comes with genuine operational overhead to run. The right choice is whichever one actually matches the availability requirement for that workload.

The trap that less-experienced engineers fall into.

Either running critical prod on single-VM Compose (a SPOF) or reflexively insisting everything needs Kubernetes (over-engineering for a team that can't operate it).

🎯 Interviewer follow-up questions you should expect.

• "Is Docker Compose OK for production?" It's fine for a low-stakes single host, but it has no self-healing, rolling updates, or high availability — a single-VM production setup is a single point of failure.
• "When would you move to an orchestrator?" When you need high availability, self-healing, rolling deploys, or multi-host scale, right-sizing the choice, whether Kubernetes, ECS, or Swarm, to the actual requirement and team.