Service A can't reach Service B in prod. It works on a laptop, in staging, or between two other things — but this specific path in this specific environment fails. Vague, and under pressure.

What is actually happening (the mental model).

Something in the path between A and B differs in prod — DNS resolves differently, a firewall or security group blocks it, the app is bound to the wrong interface, or TLS fails. The senior move is to not theorise and instead walk the six layers, letting the first failing one tell you exactly where to stop. "Works locally" almost always means an environmental difference in one of those layers — a firewall that is not there locally, an app bound to 127.0.0.1, or a different DNS view.

How to work through it.

Exactly the reflex above:

1. dig +short B — does it resolve, and to the IP you expect? (Half of "it's down" is stale or wrong DNS.)
2. ping / traceroute — is it reachable? (ICMP may be blocked, so its absence is not proof of unreachability.)
3. nc -zv B <port> / curl -v — is the port open? Note refused versus timeout, which localises the problem instantly.
4. The firewall layers — the security group, the NACL, and the host's iptables or nftables.
5. ss -tlnp on B — is the app bound to 0.0.0.0 or the right interface, or only to 127.0.0.1? Binding to localhost is a classic "works locally, not from another host" cause.
6. openssl s_client — does the TLS handshake complete?

Root causes, and how to tell them apart.

It could be wrong or stale DNS (found with dig), a security group or NACL dropping the traffic (a timeout at step 3), the app bound to localhost (which works locally but is refused or unreachable remotely, as ss shows the bind address), or a TLS failure (found with openssl). Each layer's test isolates one cause.

How to fix it, verify it, and prevent it.

Fix the specific failing layer, and verify end-to-end from the actual source. Prevent "works locally" surprises with environment parity — using infrastructure as code so that the production and staging firewalls and DNS match — and by binding services to the correct interface explicitly.

The trap that less-experienced engineers fall into.

Random pings and service restarts instead of the layered walk — which is both faster and teaches you exactly which layer owns the problem.

🎯 Interviewer follow-up questions you should expect.

• "Walk me through debugging A can't reach B." You go through the six layers in order — DNS, reachability, port, firewall, whether the app is listening, then TLS — stopping at the first failure.
• "It works locally but not from another host — common cause?" The app is bound to 127.0.0.1 instead of 0.0.0.0 — ss -tlnp shows it.
• "How does refused vs timeout help?" Refused points to a process or port issue, while a timeout points to a firewall or route dropping the traffic.

A connection attempt fails with one of two specific messages — "connection refused" or "connection timed out" — and knowing what each means tells you which half of the problem you're in before you run anything else.

What is actually happening (the mental model).

These two errors come from fundamentally different failures at the TCP level, and that is why they are diagnostic gold:

• Connection refused means your SYN reached the host, but nothing is listening on that port, so the host sent back a RST (reset) immediately. It is fast. It means you got to the machine, but the app is down, has crashed, or is on a different port.
• Connection timed out means your SYN got no response at all; it vanished, and after the retry timeout you give up. It means a firewall, security group, or NACL is silently dropping the packet, the route is wrong, or the host is down or unreachable.

So the error type immediately splits "go look at the process" from "go look at the network path."

How to work through it.

1. Read the error precisely — is it refused or timed out?
2. Refused means the network path is fine and the problem is at the host: is the app running? On the expected port? Bound to the right interface? Run ss -tlnp on the host.
3. Timed out means the app might be fine and the problem is the path: a security group, NACL, or firewall is dropping the packets, the route is wrong, or the host is down. Check the firewall rules and the routing.

The root causes.

For refused, the app has crashed or not started, is on the wrong port, or is bound to a different interface. For timed out, the security group is not allowing the port, the NACL is missing the ephemeral return rule, the host firewall is dropping the packet, the route table is wrong, or the host is genuinely down.

How to fix it, verify it, and prevent it.

Fix the located side and re-test. This heuristic does not need much prevention — it is a diagnostic accelerator that you apply to the other scenarios.

The trap that less-experienced engineers fall into.

Treating both errors as "it's down" and checking everything, instead of letting the error type immediately halve the search space.

🎯 Interviewer follow-up questions you should expect.

• "Connection refused vs timed out — what does each tell you?" Refused means you reached the host but nothing is listening, a process or port issue; timed out means the packets are dropped silently, a firewall or route.
• "You get a timeout to port 443 — where do you look first?" The firewall, security group, NACL, and routing, not the app.

A service works fine normally but starts throwing intermittent connection resets or timeouts only at peak load — some requests fail, most succeed, and it clears up when traffic drops.

What is actually happening (the mental model).

Peak-only failures point at a resource that only saturates under load. The prime suspects are connection and port limits, and in the cloud the sneaky, senior-signal one is SNAT or ephemeral-port exhaustion on a NAT gateway. When many instances make outbound connections through a NAT gateway to the same destination, they share a limited pool of source ports (about 64,000 per destination IP). Under heavy outbound load to one endpoint, that pool runs out, and new connections get intermittent resets or failures — invisible until you look at the NAT gateway's metrics.

How to work through it.

1. Confirm it is load-correlated (it fails at peak and recovers at the trough).
2. Check socket state: ss -s, and the counts of TIME_WAIT and SYN_SENT — pileups hint at churn or exhaustion.
3. Ephemeral and SNAT ports: are many outbound connections going to a single destination through a NAT gateway? Check the NAT gateway's ErrorPortAllocation metric on AWS — a non-zero value means SNAT exhaustion.
4. A full conntrack table on a stateful firewall or host drops new connections (compare nf_conntrack_count against the max).
5. Too small a backlog (net.core.somaxconn) or too low a load-balancer maximum-connection limit under a burst.

Root causes, and how to tell them apart.

It could be SNAT or ephemeral-port exhaustion on a NAT gateway (shown by the ErrorPortAllocation metric and many outbound connections to one endpoint), a full conntrack table (the conntrack count near its maximum), too small a listen backlog, or connection churn from a keep-alive misconfiguration. Each one has a distinct metric.

How to fix it, verify it, and prevent it.

For SNAT exhaustion, distribute the outbound traffic across multiple NAT gateway IPs, use VPC endpoints to keep AWS-service traffic off the NAT entirely (which is often the real fix), or reduce connection churn with keep-alive and pooling. For a full conntrack table, raise the table size or reduce the connection rate. Prevent it by monitoring the NAT gateway and conntrack metrics.

The trap that less-experienced engineers fall into.

Blaming the application for intermittent resets when the app is fine — the NAT gateway is silently exhausting its SNAT ports, which no amount of app debugging reveals.

🎯 Interviewer follow-up questions you should expect.

• "Intermittent resets only at peak — top suspects?" SNAT or ephemeral-port exhaustion on a NAT gateway, a full conntrack table, or too small a backlog.
• "What's SNAT exhaustion?" A NAT gateway has about 64,000 source ports per destination, and many outbound connections to one endpoint exhaust them, causing intermittent failures.
• "How do you fix or avoid it?" VPC endpoints to bypass the NAT, multiple NAT IPs, and connection pooling to reduce the churn.

The unmistakable fingerprint: small requests work, large payloads hang — a health check or small API call succeeds, but a file upload or large response times out or resets. Common right after introducing a VPN, a Kubernetes overlay (VXLAN), or any tunnel.

What is actually happening (the mental model).

Tunnels and overlays add encapsulation headers to every packet, which shrinks the effective MTU below the standard 1500 bytes. Normally, Path MTU Discovery (PMTUD) handles this: a too-big packet triggers an ICMP "fragmentation needed" message telling the sender to use smaller packets. But if a firewall blocks that ICMP (which is very common, because people block "ICMP" wholesale), PMTUD is broken, and large packets silently blackhole — they are dropped with no feedback, so small requests (which fit) work and large ones (which do not) just hang. This is one of the most time-consuming mysteries in networking, and recognising the fingerprint instantly is a huge senior differentiator.

How to work through it.

1. Recognise the signature: small requests are fine, but large ones (uploads, big responses) hang or reset — especially over a tunnel, VPN, or overlay.
2. Probe the real MTU: ping -M do -s 1472 <dest> (1472 + 28 header = 1500). If it fails ("message too long" or no reply) but a smaller size works, the path MTU is below 1500.
3. Fix it: MSS clamping (have TCP negotiate a smaller maximum segment size — the clean fix, often on the tunnel or router), or lower the interface MTU to match the tunnel. Make sure the ICMP that PMTUD needs (type 3, code 4) is not blocked.

The root causes.

Encapsulation overhead (from a VPN or VXLAN) shrinking the effective MTU, combined with blocked ICMP breaking PMTUD, which blackholes large packets. The "small works, big hangs" fingerprint plus a recently added tunnel is the tell.

How to fix it, verify it, and prevent it.

Apply MSS clamping or a matched MTU. Prevent it by configuring the MTU correctly on tunnels and overlays from the start, and by not blanket-blocking ICMP (PMTUD needs it).

The trap that less-experienced engineers fall into.

Debugging the application or the far-end service for days when it is an MTU blackhole — because "it works for small requests" makes it look like an app-level size limit rather than a network-layer packet problem.

🎯 Interviewer follow-up questions you should expect.

• "Small requests work but large ones hang — what is it?" An MTU or PMTU-discovery problem — tunnels add headers, blocked ICMP breaks PMTUD, and large packets blackhole.
• "How do you confirm and fix it?" You run a do-not-fragment ping sweep to find the real MTU, then fix it with MSS clamping or a lower interface MTU, and you don't block PMTUD's ICMP.

After a migration or IP change, clients keep hitting the old endpoint — "it works for us but customers still see the old behaviour" — or resolution fails entirely.

What is actually happening (the mental model).

DNS answers are cached at many layers, each with its own TTL — the application, the OS or stub resolver, downstream recursive resolvers, and the CDN. "I updated the record" does not mean clients see it until their cached copy expires. And some clients cache far beyond the TTL — notably the JVM, which by default caches DNS for the life of the process. So a migration's "why are clients still hitting the old IP?" is almost always caching, not a DNS server problem.

How to work through it.

1. See the authoritative truth versus what is being served: dig +trace B (which walks from the root to the authoritative server for the real current answer) versus dig B against your resolver (what is cached). A mismatch means you are seeing a cached answer.
2. Check the record's TTL and whether it is high (which means slow propagation).
3. Rule out local overrides: /etc/hosts, split-horizon DNS (a different internal versus external view), or negative caching (a cached failure).
4. Check application caching: long-lived processes, especially the JVM (networkaddress.cache.ttl), caching a resolved IP indefinitely.

Root causes, and how to tell them apart.

It could be a high TTL that has not yet expired (dig shows the old answer with the TTL counting down), a JVM or app caching forever (the app hits the old IP even after the resolvers updated), a stale /etc/hosts entry, or split-horizon DNS serving a different view. Comparing dig +trace against the resolver's answer distinguishes a server-side problem from a caching one.

How to fix it, verify it, and prevent it.

The key prevention is to lower the TTL a day before a planned migration, so that propagation is fast when you flip, and then raise it back afterwards. For JVM apps, set a sane DNS cache TTL. Verify with dig +trace.

The trap that less-experienced engineers fall into.

Assuming an updated record takes effect immediately and being baffled that clients still hit the old IP — not accounting for TTLs and multi-layer caching, and especially not knowing about the JVM's infinite DNS caching.

🎯 Interviewer follow-up questions you should expect.

• "You updated DNS but clients still hit the old IP — why?" Caching at multiple layers, each per its own TTL — some clients, like the JVM, cache forever.
• "How do you make a migration cutover fast?" You lower the TTL a day ahead so propagation is quick when you flip.
• "dig vs dig +trace?" dig shows the resolver's possibly cached answer, whereas +trace walks to the authoritative server for the real current answer.

curl fails with a cert error, browsers warn, or a service-to-service call fails TLS validation. It may work in a browser but fail in curl or Java — a telling detail.

What is actually happening (the mental model).

A TLS error is one of a small, well-defined set, and openssl s_client shows you which in seconds: an expired certificate, an incomplete chain (a missing intermediate certificate), a name mismatch (the SAN or CN does not cover the hostname), the wrong certificate for the SNI, clock skew (the validity window fails on a box with the wrong time), or a protocol or cipher mismatch. The most common real one — and a great senior detail — is the missing intermediate certificate: browsers cache intermediates so it works there, but curl and Java do not, so it fails there.

How to work through it.

1. openssl s_client -connect host:443 -servername host, and read the presented chain, the notAfter (expiry), the subject and SAN, and the Verify return code.
2. Diagnose the specific failure:
   • a missing intermediate works in browsers (which cache intermediates) but fails in curl and Java, so serve the full chain.
   • an expired certificate needs renewal — this should never happen, so it is a process failure.
   • a name mismatch means the SAN does not cover the hostname, or the wrong cert is served for the SNI.
   • clock skew gives "not yet valid" or "expired" on a box with the wrong time (which ties to the Linux clock-skew scenario).
3. Automate renewal (ACM, cert-manager, or Let's Encrypt) with an alert 30 days before expiry.

Root causes, and how to tell them apart.

It could be a missing intermediate (which works in a browser but fails in curl or Java), an expiry (notAfter is in the past), a name mismatch (the SAN versus the hostname), or clock skew (the client's clock is wrong). The "browser is OK, curl fails" split strongly implies a chain problem.

How to fix it, verify it, and prevent it.

Serve the full chain, renew the certificate, or fix the SAN. Prevent it with automated renewal and expiry alerting — a cert-expiry outage is a 100%-preventable unforced error.

The trap that less-experienced engineers fall into.

Only testing in a browser (which caches intermediates and hides a missing-intermediate problem), then being confused when clients and services fail. And letting a certificate simply expire — which should be impossible with automation.

🎯 Interviewer follow-up questions you should expect.

• "Works in a browser but fails in curl — why?" A missing intermediate certificate — browsers cache intermediates, but curl and Java don't, so you need to serve the full chain.
• "How do you debug a TLS error?" You use openssl s_client for the chain, expiry, SAN, and verify code.
• "A cert expired in prod — what's the real problem?" A process failure — certificates must auto-renew with an expiry alert.

Users get 5xx errors from the load balancer. The specific code — 502, 503, or 504 — is the single biggest clue, and treating "5xx" as one undifferentiated thing wastes time.

What is actually happening (the mental model).

Each code points at a different layer, so learning them precisely tells you exactly where to look:

• 502 Bad Gateway means the LB got an invalid or broken response from the backend, or the connection was reset — so the app crashed, returned garbage, or there is an HTTP keep-alive mismatch. Look at the backend process.
• 503 Service Unavailable means there are no healthy targets, or the LB is at capacity — so the health checks are failing and all targets are marked unhealthy, or you have scaled to zero. Look at target health.
• 504 Gateway Timeout means the backend did not respond in time — so the app is slow, or (the classic misconfiguration) the LB idle timeout is shorter than the app's response time. Look at latency and timeout settings.

How to work through it.

1. Read the exact code — 502, 503, or 504.
2. 502 — check the backend health and logs: did it crash, return an invalid response, or reset the connection? Check for an HTTP keep-alive timeout mismatch (where the backend closes a connection the LB reuses).
3. 503 — check the target group health: are the targets passing health checks? Is anything healthy at all? Check the health-check path, port, and thresholds.
4. 504 — check the app's response time against the LB idle timeout and any per-target timeout. A slow endpoint, or a timeout shorter than the response time, causes it.

Root causes, and how to tell them apart.

A 502 is an app crash, an invalid response, or a keep-alive mismatch. A 503 is failed health checks, no healthy targets, or a capacity limit. A 504 is a slow backend, or an LB timeout shorter than the app's response. The code itself is the discriminator.

How to fix it, verify it, and prevent it.

Fix the located layer (the app, the health check, or the timeout). Prevent it with proper health checks, aligned timeouts (the LB idle timeout at least the app's response time plus a margin), and connection draining. Alert on 5xx by code, so that next time you know which layer to look at.

The trap that less-experienced engineers fall into.

Treating all 5xx the same and checking everything, instead of letting the code point at the specific layer — especially missing that a 504 is often just an LB idle timeout set shorter than the app's legitimate response time.

🎯 Interviewer follow-up questions you should expect.

• "502 vs 503 vs 504 — what does each mean?" 502 is a bad or invalid backend response, 503 means no healthy targets, and 504 is the backend timing out.
• "You're getting 504s — likely misconfig?" The LB idle timeout is shorter than the app's response time.
• "All requests 503 after a deploy — where do you look?" The target health checks — the new version is failing them, so nothing is healthy.

Latency spikes, intermittent connection resets between services, or high connection churn — often showing up as occasional 502s (tying to scenario 7) or p99 latency that's mysteriously high.

What is actually happening (the mental model).

There are two related failure modes. First, a client keep-alive timeout that is longer than the server's — the server closes an idle connection the client still thinks is open, so the client's next request on that dead connection gets a reset. Second, no connection pooling — every request does a fresh TCP and TLS handshake, which is brutal latency at scale. The senior tuning insight is that the client's idle timeout should be shorter than the server's (and the LB's), so that the client is the one that closes, which avoids the race where it reuses a connection the server has already dropped.

How to work through it.

1. Check the symptom: intermittent resets or 502s (a keep-alive mismatch) versus high latency with lots of new connections (no pooling).
2. Compare the idle timeouts across the chain — client, LB, server. If the client's is longer than the server's, that is the reset cause.
3. Check pooling: is the client reusing connections, or opening a new one (with a TLS handshake) per request? Connection metrics or a tcpdump showing handshakes reveal it.
4. Check the pool sizing: is the pool exhausted under load, with requests queuing for a free connection?

Root causes, and how to tell them apart.

It could be a client keep-alive that is longer than the server's (causing resets on reused-but-closed connections), no pooling (a handshake per request adding latency), or a pool that is too small (causing queuing). Comparing the timeouts and the connection metrics distinguishes them.

How to fix it, verify it, and prevent it.

Align the idle timeouts so that client is less than server is less than LB; enable and size the connection pools; and reuse connections to amortise the cost of TLS. Prevent it by setting these consistently across the whole call chain.

The trap that less-experienced engineers fall into.

Ignoring keep-alive alignment and connection pooling — then chasing "random" 502s and p99 latency that are actually connection-churn and timeout-mismatch problems.

🎯 Interviewer follow-up questions you should expect.

• "Intermittent resets between two services — a keep-alive cause?" Yes — if the client's idle timeout is longer than the server's, it reuses a connection the server has already closed.
• "Why does no connection pooling hurt latency?" You pay a TLS handshake per request — pooling amortises it.
• "How should timeouts be ordered?" The client idle timeout should be shorter than the server's, and the LB's, so the client closes first.

Users see stale content after a deploy, or the CDN isn't caching at all (every request hits the origin, origin load is high, latency poor).

What is actually happening (the mental model).

A CDN caches based on response headers and the cache key. If everything is a MISS, the origin is usually telling it not to cache — through Cache-Control: no-store or private, a Set-Cookie (which disables caching for that response), or a Vary header that fragments the cache key so every request is treated as unique. If content is stale, the TTL is too long or an invalidation was missed on deploy. So you reason about the cache key and the origin's headers together, never the CDN in isolation.

How to work through it.

1. Read the headers: curl -I <url> for Cache-Control, Age, and the CDN's X-Cache (HIT or MISS) and similar.
2. All MISS means the origin is preventing caching: Cache-Control: no-store or private, a Set-Cookie on cacheable content, or a Vary: * or cookie fragmenting the key so nothing is reused.
3. Stale content means the TTL is too long, or the deploy did not trigger an invalidation or purge.
4. Reason about the cache key: are query strings, cookies, or Vary making each request unique?

Root causes, and how to tell them apart.

No caching means the origin is sending no-store, a Set-Cookie, or a Vary that fragments the key (the headers show it, and everything is a MISS). Stale content means a long TTL or a missed invalidation (the Age is high and the content is old). The response headers are the whole story.

How to fix it, verify it, and prevent it.

Fix the origin headers (a cacheable Cache-Control, no Set-Cookie on static assets, and a tighter Vary), and invalidate on deploy. Prevent it with correct cache headers per content type, and automated cache invalidation in the deploy pipeline.

The trap that less-experienced engineers fall into.

Debugging the CDN configuration in isolation when the origin's headers (a stray Set-Cookie or Vary) are what is disabling caching — or forgetting to invalidate on deploy and blaming the CDN for stale content.

🎯 Interviewer follow-up questions you should expect.

• "Everything's a cache MISS — why?" The origin is sending no-store, a Set-Cookie, or a Vary that fragments the cache key.
• "Users see stale content after a deploy — cause?" The TTL is too long, or a cache invalidation was missed.
• "How do you debug it?" You use curl -I for Cache-Control, Age, and the X-Cache HIT/MISS.

Traffic is blocked in AWS and you need to reason about which control is dropping it — a security group or a network ACL — often after adding a NACL and having "mysterious" breakage.

What is actually happening (the mental model).

The key difference is state:

• A security group is stateful — if you allow inbound 443, the response is automatically allowed back out. It is instance-level and allow-only.
• A NACL is stateless — it evaluates each direction independently, so you must explicitly allow the ephemeral return port range (for example, 1024–65535) for the reply traffic. It is subnet-level, allows and denies, and is evaluated in rule-number order. Forgetting the ephemeral return rule is the number-one NACL mistake.

How to work through it (when something is blocked).

1. The security group first — is the port allowed inbound on the destination's security group (and outbound on the source's, if that is restricted)? Since security groups are stateful, that is usually all you need there.
2. The NACL, in both directions — is the port allowed inbound, and is the ephemeral return range allowed outbound for the response? This is where being stateless bites you.
3. Check the rule order on the NACL (the lowest number wins, so an early deny shadows a later allow).
4. Then the host firewall (iptables).

Root causes, and how to tell them apart.

It could be the security group missing the inbound allow (the simplest case), the NACL missing the ephemeral return range (the classic case — inbound works but responses are dropped), or NACL rule-order shadowing. If the security group is fine but a NACL was recently added and traffic broke, suspect the ephemeral-return rule.

How to fix it, verify it, and prevent it.

Do your real allow-listing at the security-group level (stateful and simpler); use NACLs only for coarse subnet-wide denies, and if you do, remember the ephemeral return rule. Prevent confusion by keeping NACLs minimal.

The trap that less-experienced engineers fall into.

Adding a NACL, allowing the inbound port, and forgetting the ephemeral return range — so requests arrive but responses are dropped, causing a baffling one-way failure.

🎯 Interviewer follow-up questions you should expect.

• "Security group vs NACL — key difference?" A security group is stateful, so return traffic is auto-allowed, while a NACL is stateless, so you must allow the ephemeral return range explicitly.
• "You added a NACL and things broke — likely cause?" You forgot to allow the ephemeral return port range for responses.
• "Where should you do allow-listing?" At the security group, since it's stateful and simpler — use NACLs only for coarse subnet denies.

Traffic mysteriously fails or gets dropped even though both endpoints seem reachable — often with a stateful firewall in the path, or on hosts with multiple NICs or both a VPN and a direct route.

What is actually happening (the mental model).

A stateful firewall must see both directions of a flow — it tracks the outbound connection so that it can allow the matching return traffic. If routing sends the request out one path and the reply comes back a different path, the firewall on the return path never saw the outbound SYN, so it treats the reply as unsolicited and drops it. This "asymmetric routing" breaks stateful devices even when every individual path is technically fine.

How to work through it.

1. Suspect it with multi-NIC hosts (two default routes), a VPN plus a direct path, or complex route tables.
2. Map both directions: ip route get <dest> for the outbound path, and check the return path — do they traverse the same stateful device?
3. If the paths differ and a stateful firewall is involved, that asymmetry is the cause.

The root causes.

A multi-NIC host with two default routes sending replies out the "wrong" interface, a VPN-out but internet-back asymmetry, or misconfigured route tables. The signature is mismatched forward and return paths through a stateful device.

How to fix it, verify it, and prevent it.

Correct the route tables so that the flows are symmetric (both directions traverse the same firewall), use policy routing or source-based routing on multi-NIC hosts, or make the firewall aware of both paths. Prevent it by avoiding accidental multi-path setups and by reviewing the routing whenever you add NICs or VPNs.

The trap that less-experienced engineers fall into.

Not considering routing symmetry — debugging the firewall rules or the app when the actual problem is that the reply takes a different path than the request, so a stateful firewall drops it.

🎯 Interviewer follow-up questions you should expect.

• "A stateful firewall drops traffic even though both ends are reachable — why?" Asymmetric routing — the reply takes a different path, so the firewall never saw the outbound and drops the return.
• "Common cause of asymmetry?" Multi-NIC hosts with two default routes, or a VPN-out-and-internet-back setup.
• "How do you fix it?" You make the routing symmetric, through policy or source routing, or ensure both directions traverse the same firewall.

A data transfer is far slower than the link should allow — a "10 Gbps" link delivering a fraction of that on a single transfer, especially over distance.

What is actually happening (the mental model).

A single TCP flow's throughput is capped by window size divided by RTT — the bandwidth-delay product — regardless of how fat the pipe is. On a high-latency link, a single connection simply cannot fill the pipe unless the TCP window is large enough. And separately, even tiny packet loss collapses TCP throughput (TCP interprets loss as congestion and backs off hard). So a "slow transfer" is usually latency and loss, not pipe size.

How to work through it.

1. Measure honestly: run iperf3 between the endpoints, both a single stream and parallel streams. If the parallel streams are much faster, you were window-limited or single-flow limited.
2. Check for loss: mtr or ping — even 0.1% loss murders throughput over a high-RTT link.
3. On a high-latency link, tune the TCP window scaling and use larger buffers, or parallelise across multiple streams to fill the pipe.
4. Rule out NIC offload issues and MTU (scenario 4).

Root causes, and how to tell them apart.

It could be the bandwidth-delay-product limit on a single flow (parallel streams fix it, which means it was window-limited), packet loss (mtr shows loss and the throughput collapses), an MTU issue, or NIC configuration. The single-versus-parallel iperf3 comparison and the mtr loss localise it.

How to fix it, verify it, and prevent it.

Tune TCP window scaling, parallelise the transfers, and eliminate the loss. Prevent it by sizing TCP buffers for high-bandwidth-delay-product links and by monitoring for loss.

The trap that less-experienced engineers fall into.

Assuming "slow transfer means a small or congested pipe" and asking for more bandwidth — when the real limit is latency (the bandwidth-delay product on a single flow) or a tiny amount of packet loss, neither of which more bandwidth fixes.

🎯 Interviewer follow-up questions you should expect.

• "A single transfer is slow on a fat link — why?" The bandwidth-delay product — a single TCP flow is capped by window divided by RTT, so you parallelise it or tune the window.
• "Why does a little packet loss hurt so much?" TCP treats loss as congestion and backs off hard, collapsing the throughput.
• "How do you diagnose?" You use iperf3 with single versus parallel streams, and mtr for loss.

A design question: lay out the network for a new environment or account. This tests whether you think about tiering, blast radius, and — the permanent mistake — CIDR planning.

What is actually happening (the mental model).

Good network design is tiered by exposure, isolates the blast radius, and plans non-overlapping CIDRs so that future peering or VPN stays possible — overlapping CIDRs are an expensive, near-permanent mistake. It uses least privilege by default, and keeps AWS-service traffic private and cheap with endpoints.

How to reason about it.

• Subnet tiers across multiple AZs: public (load balancers only), private (the app tier), and isolated (the data tier, with no route to the internet). Egress from the private tiers goes through NAT, and ingress comes only through the load balancer. No public IPs on the app or data tiers.
• Security groups (stateful, instance-level) for day-to-day allow-listing, and NACLs (stateless, subnet-level) as a coarse backstop only.
• CIDR planning: non-overlapping with other VPCs and on-premises — or peering and VPN become impossible later. Plan generous space for growth.
• VPC endpoints for S3, DynamoDB, and ECR, so that the traffic stays off the internet and the NAT (which is more secure and cheaper, with no NAT data-processing charges).
• Multi-AZ everything for resilience.

The trap that less-experienced engineers fall into.

Putting everything in one flat subnet with public IPs; picking a CIDR that overlaps another VPC or on-premises (permanently blocking future connectivity); and routing S3 and ECR traffic through the NAT gateway instead of a VPC endpoint (which is costly and less secure).

🎯 Interviewer follow-up questions you should expect.

• "How do you tier subnets?" Public for load balancers, private for the app with NAT egress, and isolated for data with no internet — all spread across AZs.
• "Why does CIDR planning matter?" Overlapping CIDRs permanently block peering and VPN, so you plan non-overlapping ranges with room to grow.
• "Security group vs NACL in the design?" Security groups for allow-listing, since they're stateful, and NACLs as a coarse subnet backstop.
• "Why VPC endpoints?" To keep S3 and ECR traffic private and off the NAT — it's more secure, and it avoids the NAT data charges.

In an EKS cluster, pods fail to schedule with an IP-allocation error, or describe pod shows the CNI unable to assign an address, even though nodes have CPU/memory to spare.

What is actually happening (the mental model).

With the AWS VPC CNI, every pod consumes a real VPC IP address from the subnet (that is how pods get first-class VPC networking). So a busy cluster can exhaust the subnet's CIDR — you run out of IPs before you run out of compute — and pods go Pending with IP-allocation failures. It is a networking constraint masquerading as a scheduling problem, and it surprises people who sized their subnets for nodes, not pods.

How to work through it.

1. The symptom: pods are Pending or ContainerCreating, and describe pod or the CNI (aws-node) logs show no free IPs or an ENI allocation failure.
2. Confirm subnet exhaustion: compare the free IPs in the pod subnets against the pods scheduled.
3. The fixes:
   • Prefix delegation — assign /28 prefixes to ENIs so each node can host far more pods per IP allocation (a big density win).
   • A secondary CIDR — add more address space to the VPC for the pod subnets.
   • Larger pod subnets than intuition suggests, from the start.
   • An overlay CNI (such as Calico overlay) to decouple pod IPs from the VPC space entirely — a real architectural trade-off, since you lose native VPC pod addressing.

The root causes.

The subnet CIDR is too small for the pod count under the VPC CNI's one-IP-per-pod model. The IP-allocation error combined with free compute is the tell.

How to fix it, verify it, and prevent it.

Use prefix delegation, a secondary CIDR, or bigger subnets, or an overlay CNI. Prevent it by sizing the pod subnets for the peak pod count (not the node count) up front.

The trap that less-experienced engineers fall into.

Sizing subnets for the number of nodes and being blindsided when the pods exhaust the IPs — not realising that the VPC CNI gives every pod a real VPC IP.

🎯 Interviewer follow-up questions you should expect.

• "Pods won't schedule with an IP error but there's free CPU — why?" The AWS VPC CNI gives every pod a real VPC IP, and the subnet CIDR is exhausted.
• "How do you fix or avoid it?" Prefix delegation, a secondary CIDR, larger pod subnets, or an overlay CNI to decouple pod IPs from the VPC.

A design question: connect on-prem to the cloud, or region to region, or many VPCs together — with the right trade-offs on cost, consistency, and scale.

What is actually happening (the mental model).

There are two key senior insights: weigh cost against consistency for the link type, and know that VPC peering is not transitive — so a growing mesh of VPCs eventually needs a hub, not more point-to-point links.

How to reason about it.

• On-premises to cloud: a site-to-site VPN (cheap, quick to stand up, but rides the public internet with variable latency) versus Direct Connect or ExpressRoute (dedicated, with consistent bandwidth and latency, but costly and with weeks of lead time). Usually Direct Connect with a VPN backup for production.
• VPC to VPC at scale: point-to-point peering does not scale — it is non-transitive (A↔B and B↔C does not give you A↔C) and grows as O(n²). Move to a Transit Gateway hub-and-spoke, which gives transitive routing through a central hub.
• Plan redundancy (dual Direct Connect or VPN), the routing, and — again — non-overlapping CIDRs.

The trap that less-experienced engineers fall into.

Building an ever-growing mesh of VPC peerings (which is non-transitive and becomes unmanageable), or choosing a VPN for a latency-sensitive production workload that really needs Direct Connect's consistency.

🎯 Interviewer follow-up questions you should expect.

• "VPN vs Direct Connect?" A VPN is cheap and quick but rides the internet with variable latency, while Direct Connect is dedicated, consistent, and costly with lead time — usually you'd pick Direct Connect with a VPN backup.
• "Why not just peer all your VPCs?" Peering is non-transitive and O(n²) — you use a Transit Gateway hub instead, which scales.
• "What breaks all of this?" Overlapping CIDRs — you plan them up front.

Pods can't resolve service names, or external calls from pods are mysteriously slow, or resolution is flaky under load. (This is the Kubernetes DNS scenario, included here because it's fundamentally a networking problem.)

What is actually happening (the mental model).

In-cluster DNS is served by CoreDNS, and pods are configured with options ndots:5 in /etc/resolv.conf. That ndots:5 is the famous trap: for a name with fewer than 5 dots, the cluster search domains are appended and tried first, so resolving an external name like api.stripe.com (2 dots) fires several failed lookups before the real one — adding latency on every external call and hammering CoreDNS, which at scale can CPU-throttle and cause cascading resolution failures.

How to work through it.

1. Test from inside a pod: kubectl exec -it <pod> -- nslookup <service> and an external name — is it internal, external, or both that fail? That split localises it.
2. Check CoreDNS health: kubectl get pods -n kube-system -l k8s-app=kube-dns, its logs, and whether it is CPU-throttled under the query volume the ndots amplification creates.
3. Inspect the pod's /etc/resolv.conf — the ndots:5 and the search list; the amplification is why external lookups are slow.
4. Rule out a NetworkPolicy blocking UDP or TCP 53 to CoreDNS, and confirm the target Service has endpoints.

Root causes, and how to tell them apart.

Slow external lookups mean ndots amplification (and maybe CoreDNS throttling). Total failure means CoreDNS is down or a NetworkPolicy is blocking port 53. An internal-only failure means a Service or endpoints issue. The internal-versus-external split from step 1 points straight at the cause.

How to fix it, verify it, and prevent it.

For the ndots latency, use FQDNs with a trailing dot for external services (which skips the search list), or tune ndots for specific pods. For scale, give CoreDNS adequate resources, autoscale it, and add NodeLocal DNSCache (per-node caching that slashes the CoreDNS load). For blocked port 53, fix the NetworkPolicy to allow DNS egress.

The trap that less-experienced engineers fall into.

Profiling an application for latency that is actually coming from failed DNS lookups on every external call due to ndots — not realising that DNS is the hidden cost.

🎯 Interviewer follow-up questions you should expect.

• "External calls from pods are slow — how could DNS be involved?" ndots:5 appends the search domains first, so external names do several failed lookups before succeeding.
• "Nothing resolves at all — what do you check?" CoreDNS health, and whether a NetworkPolicy is blocking port 53.
• "How do you scale cluster DNS?" Adequate CoreDNS resources, autoscaling, and NodeLocal DNSCache.