A TLS cert on the main API expired. Every client got handshake failures. There was no monitoring on cert expiry and renewal was manual.

How it unfolded.

Pages flooded in with vague "API down" errors. Early responders chased app and network issues for ~40 minutes — restarting services, checking load balancers — before someone ran openssl s_client and saw notAfter was in the past. The obvious cause was invisible because nobody suspected it.

How a senior handled it.

Renewed/rotated the cert to restore service immediately, communicated status, then in the postmortem attacked the real failure — not "the cert expired" but "a manual, time-based task with no monitoring was allowed to exist."

The lesson, and how to prevent it.

A cert expiry is a 100%-preventable, self-inflicted outage — the trigger is the expiry, but the vulnerability is the missing automation and monitoring. Fix: automated renewal (ACM/cert-manager), and alerting 30 days before expiry. The deeper principle: any manual, time-based task that can cause an outage is a latent incident — automate it or monitor it, ideally both.

🎯 Where this story earns you points.

Great for "a preventable failure" or "a time you improved a system's reliability." The senior signal is separating the trigger (expiry) from the vulnerability (no automation/monitoring).

An engineer ran terraform destroy believing they were in dev. They were in prod. Real infrastructure started coming down.

How it unfolded.

They caught it partway and Ctrl-C'd, but several resources were already gone. Recovery took hours — rebuilding from the Terraform code and restoring data from backups — while the service was degraded.

How a senior handled it.

Coordinated the rebuild from IaC + restored data from backups, kept stakeholders informed with honest ETAs, and ran a blameless postmortem — the focus was the system, not the engineer.

The lesson, and how to prevent it.

"Be more careful" is not a control. The fix is structural: isolate prod into a separate account so a dev-context mistake physically can't reach prod, prevent_destroy on critical resources, and plan-then-apply-the-saved-plan so a mass-destroy is visible in review. The person wasn't the problem — the system allowed a single human error to be catastrophic.

🎯 Where this story earns you points.

Perfect for "tell me about a mistake" (if it was yours) or "a time you improved a process." The senior signal is the blameless framing + systemic guardrails over blame.

The monthly AWS bill jumped several thousand dollars. Cause: a new service made huge volumes of calls to S3 through the NAT gateway, racking up data-processing charges.

How it unfolded.

Nobody noticed until finance flagged the bill weeks later — there was no cost anomaly detection. Cost Explorer grouped by usage type finally revealed NAT data-processing as the driver, hiding in plain sight.

How a senior handled it.

Added an S3 VPC Gateway Endpoint so that traffic bypassed the NAT entirely — cost dropped immediately. Then added Cost Anomaly Detection and a tagging policy so a spike is caught in days.

The lesson, and how to prevent it.

Data transfer is the cost that hides in the architecture — and a VPC endpoint for S3 is free and would have prevented the whole thing. Now VPC endpoints are a default in the network baseline, and anomaly detection means a doubling is caught in days, not on the invoice.

🎯 Where this story earns you points.

Great for "a time you saved cost / found an inefficiency" or "improved something nobody asked you to." The senior signal is the specific NAT/VPC-endpoint knowledge plus the FinOps instinct to add anomaly detection so it can't recur silently.

Under heavy load, an app got slow. Kubernetes liveness probes timed out, so K8s killed the pods — right when they were most needed — causing a cascading CrashLoop across the fleet.

How it unfolded.

The more pods got killed, the more load concentrated on the survivors, making them slower, so their liveness probes failed too, and K8s killed them. A self-inflicted death spiral — the more Kubernetes "healed," the worse it got.

How a senior handled it.

Temporarily relaxed the probe to stop the killing, scaled out to absorb the load, let it stabilize, then fixed the probe design.

The lesson, and how to prevent it.

An aggressive liveness probe turns "slow" into "dead" and can cause the exact outage it's meant to prevent. Fix: separate concerns — a startupProbe for slow starts, a readiness probe to shed traffic from busy pods without killing them, and a liveness probe tuned generously to only catch truly hung processes. And keep liveness shallow — a liveness check that pings the database turns a DB blip into a fleet-wide restart storm.

🎯 Where this story earns you points.

Strong for "a tricky incident" or "a time a safety mechanism backfired." The senior signal is the deep K8s judgment — the startup/readiness/liveness distinction under load, and the counterintuitive insight that the "healing" mechanism was the problem.

A schema migration was applied to prod. It locked a large table and, combined with a bug, corrupted data. There was no tested rollback, and the migration was coupled to the code deploy.

How it unfolded.

The app started erroring; the migration couldn't be cleanly reversed because it was a one-way change. Recovery meant restoring from backup and replaying — over an hour of downtime plus data reconciliation.

How a senior handled it.

Made the restore-from-backup call decisively, communicated transparently, and owned the process gap in the postmortem.

The lesson, and how to prevent it.

Two lessons: expand-contract migrations (add new, backfill, dual-write, switch, later remove) so old and new code coexist and every step is reversible — never a one-way breaking migration bundled with a deploy; and online schema-change tooling so big tables don't lock. "No tested rollback" is now a hard stop for any destructive change.

🎯 Where this story earns you points.

Excellent for "a mistake / failure" or "a time you improved reliability." The senior signal is expand-contract and "never bundle a breaking schema change with the deploy" — and the discipline that came from being burned. Ties directly to CI/CD and SDLC.

An AWS access key was committed to a repo that was (or became) public. Within hours, automated scanners found it and spun up expensive GPU instances across regions for crypto-mining.

How it unfolded.

GuardDuty flagged anomalous API activity and a cost spike. CloudTrail showed the key had launched instances in regions the team never used and had created new IAM users as backdoors.

How a senior handled it.

Contained first — deactivated the key immediately, before investigating. Then used CloudTrail to enumerate everything the key did, tore down the attacker's resources, removed the backdoor users, and rotated related secrets.

The lesson, and how to prevent it.

A pushed secret is a leaked secret — rotate before you scrub history. But the whole thing was preventable: there shouldn't have been a long-lived key at all. The org moved CI to OIDC federation, added gitleaks in pre-commit and CI, and least-privilege meant the future blast radius is capped. A leaked key is a symptom of missing controls.

🎯 Where this story earns you points.

Strong for "a security incident" or "a time you handled a crisis." The senior signal is contain-then-investigate + the modern OIDC / no-long-lived-keys posture.

A "small, safe" change shipped at 4pm on a Friday. It had a subtle bug that only manifested under weekend traffic patterns. It paged on-call all weekend.

How it unfolded.

The original author was offline. On-call lacked context and couldn't safely roll back because the deploy also included a migration — so they were stuck mitigating a problem they didn't fully understand.

How a senior handled it.

On-call mitigated by disabling the feature via its feature flag (thankfully it was flagged), and the real fix waited for Monday with full context and the author present.

The lesson, and how to prevent it.

Two things saved it, one hurt it. Saved: feature flags let on-call disable the feature without a risky rollback — which is why risky changes are flagged. Hurt: deploying something non-trivial right before everyone logs off, with no author around. The team didn't ban Friday deploys outright (that's cargo cult) — instead: whoever deploys must be available to support it, and risky changes go out when there's runway to react.

🎯 Where this story earns you points.

Good for "a time you learned a hard lesson about process" or "a deploy that went wrong." The senior signal is the nuance — not dogmatic "never deploy Friday," but "deploy when you can support it" — plus the value of feature flags for safe mitigation.

A downstream service slowed down. Every upstream client retried aggressively with no backoff, multiplying the load and turning a slowdown into a full outage of the dependency — then a cascading failure across everything that depended on it.

How it unfolded.

The dependency, already struggling, got hammered by retries and fell over completely. As it failed, clients retried even harder, so it couldn't recover — the retries were the DDoS.

How a senior handled it.

Shed load (temporarily stopped traffic to let the dependency breathe), let it recover, then re-enabled traffic gradually rather than all at once (which would have re-triggered the storm).

The lesson, and how to prevent it.

Naive retries turn a partial degradation into a total outage. Fix: exponential backoff with jitter, circuit breakers so clients stop hammering a failing dependency, timeouts everywhere, and bulkheads to isolate failure. Resilience patterns aren't optional at scale — without them, one slow service cascades into a company-wide outage.

🎯 Where this story earns you points.

Excellent for "a complex/distributed-systems incident" or "a time you improved resilience." The senior signal is distributed-systems resilience patterns — backoff, jitter, circuit breakers, bulkheads — and the insight that the retries were the attack.

During an unrelated incident, a service logged very verbosely. The extra logging filled /var/log, which then broke the service entirely — turning a minor issue into a hard outage.

How it unfolded.

df showed 100% full; du didn't add up. The confusion was that a process held deleted-but-open log files — space wasn't freed until the process was restarted. Responders wasted time deleting files that didn't reclaim space.

How a senior handled it.

Recognized the deleted-open-files trap via lsof | grep deleted, restarted the offending process to free the space, and restored the service.

The lesson, and how to prevent it.

Verbose logging during an incident can cause a bigger incident. Fix: log rotation with hard caps, disk alerting at 80%, and shipping logs off-host so local disk isn't a failure domain. And the deleted-open-file trap is now in the runbook — it catches everyone who trusts df and du to agree.

🎯 Where this story earns you points.

Great for "a tricky Linux problem" or "an incident that wasn't what it seemed." The senior signal is the deleted-open-files deep Linux knowledge plus the systemic fix (rotation + alerting + off-host logs).

A planned migration cut DNS over to a new endpoint. But the old records had a 24-hour TTL, so clients kept hitting the old (now-decommissioned) endpoint for a full day.

How it unfolded.

"It works for us but customers still report the old behavior" — because caches hadn't expired. Some clients (JVM apps) had cached the DNS essentially forever, regardless of TTL.

How a senior handled it.

Kept the old endpoint alive far longer than planned to drain traffic, and communicated the extended timeline honestly rather than declaring success prematurely.

The lesson, and how to prevent it.

Lower the TTL before a migration, not during — a day ahead, so propagation is fast when you flip. And know that some clients (notably JVMs) cache DNS aggressively regardless of TTL. Now "reduce TTL 24–48h in advance" is a standard step in migration runbooks.

🎯 Where this story earns you points.

Good for "a migration that didn't go as planned" or "a time you learned about a system's hidden behavior." The senior signal is DNS/TTL operational wisdom (pre-lower the TTL) and the JVM-caches-DNS-forever gotcha.

A queue-processing service was set to autoscale on CPU. But the work was I/O-bound — CPU stayed low while the queue backed up to hours of lag. It never scaled up.

How it unfolded.

Customers saw huge processing delays, but the CPU dashboards looked healthy, which sent responders down the wrong path for a while — "the boxes are fine, why is it slow?"

How a senior handled it.

Manually scaled out to burn down the backlog, then re-architected the scaling signal to reflect the actual bottleneck.

The lesson, and how to prevent it.

Autoscaling is only as good as the metric you scale on. CPU is the lazy default and it was wrong here — the fix was to scale on queue depth per worker, which reflects actual load. The broader lesson: always scale on the signal that represents the real bottleneck, not the one that's easiest to configure.

🎯 Where this story earns you points.

Strong for "a subtle scaling/performance issue" or "a time the metrics misled you." The senior signal is reasoning about what actually saturates — scaling on the real bottleneck, not CPU-by-default.

After introducing a VPN/overlay, some requests worked and large ones hung intermittently. Teams blamed the network, the app, and each other for days.

How it unfolded.

Small API calls succeeded; large payloads and file uploads hung. Classic "works sometimes" that resisted diagnosis — every team's component looked fine in isolation.

How a senior handled it.

Recognized the signature — small works, big hangs — as an MTU/fragmentation issue. Confirmed with a do-not-fragment ping sweep; the overlay's encapsulation had shrunk the effective MTU and a firewall was dropping the ICMP that Path MTU Discovery needed.

The lesson, and how to prevent it.

"Small works, big hangs" is the fingerprint of an MTU problem, and tunnels/overlays cause it by adding headers. MSS clamping fixed it. Knowing that one signature saved days — it's the kind of pattern you only recognize after you've been burned once, which is exactly why war stories are worth collecting.

🎯 Where this story earns you points.

Excellent for "the hardest bug you've debugged" or "a time your experience let you solve something others couldn't." The senior signal is the obscure-but-real networking depth — recognizing the MTU fingerprint instantly.

A "highly available" system went fully down. Root cause: every service secretly depended on one legacy component (a shared config service / a single Redis / one person's script) that had no redundancy.

How it unfolded.

The hidden dependency failed and everything cascaded. The HA design was an illusion because of one un-owned, un-redundant chokepoint that wasn't on any architecture diagram.

How a senior handled it.

Restored the component, then mapped the actual dependency graph (not the assumed one) to find other hidden SPOFs.

The lesson, and how to prevent it.

Your architecture's real availability is defined by its weakest hidden dependency, not your diagram. The team ran a dependency-mapping exercise and game-day failure injection to find SPOFs before they find you — chaos engineering in a small dose. Assumed HA is not HA; you have to test it by actually removing things.

🎯 Where this story earns you points.

Strong for "a surprising outage" or "a time you improved system resilience." The senior signal is systems thinking + the chaos/game-day mindset — proactively finding SPOFs rather than discovering them in an outage.

A real, severe incident's alert fired — but it was one of dozens of noisy alerts the team had long since learned to ignore. The signal was lost in the noise; response was badly delayed.

How it unfolded.

By the time someone realized this alert was actually real, the impact had grown significantly. The alert had been "crying wolf" for months, so nobody trusted it.

How a senior handled it.

Handled the incident, then attacked alert quality as a first-class reliability problem — not housekeeping.

The lesson, and how to prevent it.

Alert fatigue kills — a muted alert is worse than no alert because it trains people to ignore the pager. The team ruthlessly cut alerts to actionable, symptom-based, SLO-driven pages, each with a runbook. If a page doesn't require human action, it's a dashboard, not a page. Alert hygiene is a reliability control.

🎯 Where this story earns you points.

Good for "a time process/tooling failed you" or "how do you think about alerting." The senior signal is treating alert quality as a real, prioritized concern — symptom-based, actionable, SLO-driven — and understanding that a muted alert is worse than none.

One engineer was the only person who understood a critical system. During a major incident, they were unreachable (on a plane / on leave), and nobody else could resolve it.

How it unfolded.

The team floundered for hours doing what that one person could have done in minutes, because all the knowledge was in their head and nothing was documented. The "hero" was, in effect, the SPOF.

How a senior handled it.

Muddled through the incident, then made bus-factor a deliberate priority — treating the knowledge concentration as the real risk.

The lesson, and how to prevent it.

Individual indispensability is an organizational risk disguised as a strength. Fix: runbooks for the critical systems, rotate on-call so knowledge spreads through practice, pair on incidents, and run game days. The counterintuitive senior lesson: being the hero who's always needed is a failure mode — your job is to make yourself replaceable so the team is resilient without you.

🎯 Where this story earns you points.

Excellent for "a time you improved the team" or "reducing risk / bus factor," and it pairs with the intra-team "indispensability is a failure" theme. The senior signal is the maturity to see hero-dependence as a bug and to deliberately spread knowledge.

A popular cache entry expired at a moment of high traffic. Suddenly thousands of concurrent requests all missed the cache at once and hit the origin database simultaneously to recompute the same value — flattening the database.

How it unfolded.

Everything was fine until the TTL expired; then latency and errors spiked instantly as the origin was hammered by a stampede of identical requests, and the database couldn't keep up.

How a senior handled it.

Restored by shedding load and warming the cache, then fixed the pattern so a single expiry couldn't stampede the origin again.

The lesson, and how to prevent it.

A synchronized cache expiry under load causes a thundering herd. Fixes: request coalescing / single-flight (only one request recomputes, others wait for it), jittered/staggered TTLs so keys don't expire together, stale-while-revalidate (serve stale while one worker refreshes), and rate-limiting the origin. Caching without stampede protection is a latent outage.

🎯 Where this story earns you points.

Strong for "a scaling/performance incident" or "a subtle failure under load." The senior signal is knowing the thundering-herd pattern and the specific mitigations (single-flight, jittered TTLs, stale-while-revalidate) — distributed-systems depth.

A service called a downstream dependency with no timeout. When the dependency slowed to a crawl (but didn't fail outright), the calling service's threads blocked waiting, the thread pool filled with stuck requests, and the calling service became unresponsive too — a cascade from one slow dependency.

How it unfolded.

The downstream wasn't down, just slow — which is worse, because without a timeout the caller waits indefinitely. Thread after thread blocked until the pool was exhausted and healthy requests couldn't be served either. The failure propagated upward.

How a senior handled it.

Restarted the wedged services to clear the blocked threads, then added timeouts and isolation so a slow dependency couldn't exhaust the caller again.

The lesson, and how to prevent it.

A slow dependency is more dangerous than a dead one — a dead one fails fast, a slow one exhausts your resources. Fix: timeouts on every network call (non-negotiable), circuit breakers to fail fast when a dependency is unhealthy, and bulkheads (separate thread pools) so one slow dependency can't consume all your capacity.

🎯 Where this story earns you points.

Excellent for "a cascading failure" or "a resilience lesson." The senior signal is "a slow dependency is worse than a dead one" + timeouts/circuit-breakers/bulkheads. Pairs with the retry-storm story as the two classic cascade patterns. --- ## How to use these in an interview - Pick the 4–5 that map to your real experience and rehearse them as STAR stories until they're natural. You almost certainly have your own versions of several of these. - Always land the lesson/prevention — that's what makes a war story senior instead of just "a thing that went wrong." Interviewers will follow up with "what would you do differently?" — the lesson is your answer, so have it ready. - Quantify wherever you can — downtime minutes, dollars, % reduction, MTTR before/after. - Keep the blameless framing — the system allowed the failure; the fix is a systemic guardrail. Never make yourself or a teammate the villain. - Map each story to the behavioural questions it answers (I've noted these per story) so when a question comes, you already know which war story to reach for. - Separate the trigger from the vulnerability in every telling — "the cert expired" (trigger) vs "a manual task with no monitoring was allowed to exist" (vulnerability). That distinction is the seniority. --- This completes the Approaches series — all 16 domains, technical and people/process, worked through at depth. Drill them in a lab, map them to your real experience, and say the spoken versions out loud until they're muscle memory. Good luck.