A nightly automation script has been "working fine" for weeks — except it stopped actually doing its job two weeks ago, and nobody noticed because it never crashed or logged anything.

What is actually happening (the mental model).

A bare except: (or except Exception: pass) catches every error and does nothing with it — the script "succeeds" from the outside (exit code 0, no crash) while silently failing to do its actual job on the inside, which is worse than crashing because a crash at least gets noticed.

How to work through it.

1. Find every bare or overly broad except block and check what it's actually catching versus what it should be catching — a network timeout and a typo'd variable name should not be handled identically.
2. Replace silent swallowing with explicit handling: catch the specific exception you expect and know how to recover from, log it with real context, and let unexpected exceptions propagate rather than disappear.
3. Add monitoring for the script's actual output/effect, not just whether the process exited cleanly — "did it run" and "did it actually accomplish its job" are different questions.

The root causes and trade-offs.

The root cause is usually a broad except added early just to be safe, without thinking through what it should actually catch, combined with no verification that the script's real-world effect actually happened, only that the process didn't crash. Catching specific exceptions takes more upfront thought than a catch-all, but it's the difference between a script that fails loudly and one that fails invisibly for weeks.

The trap that less-experienced engineers fall into.

Wrapping a whole script body in a broad try/except specifically to stop it from ever crashing or alerting — which "succeeds" at silencing errors while guaranteeing that a real failure goes completely unnoticed.

🎯 Interviewer follow-up questions you should expect.

• "Why is a bare except worse than letting the script crash?" Because a crash gets noticed; a silently swallowed error looks identical to success from the outside, for as long as nobody checks the actual output.
• "How do you monitor whether a script did its real job, not just that it ran?" You check the effect it was supposed to have, like a file written or a resource updated, not just its exit code.

A script that calls a third-party API is stuck — it's been running for two hours with no error and no output, blocking everything downstream of it.

What is actually happening (the mental model).

By default, requests.get() (and most HTTP clients) will wait indefinitely for a response unless you explicitly set a timeout — a slow or hung remote server doesn't produce an error, it just never returns, and the calling script hangs with it, silently, for as long as the remote side does.

How to work through it.

1. Add an explicit timeout= to every outbound HTTP call — there's rarely a good reason for a production script to wait unboundedly on a remote system it doesn't control.
2. Decide what should happen when the timeout is hit — retry (with backoff, see the retry-loop scenario), fail the run cleanly, or fall back to a cached/default value, depending on how critical this call is.
3. For anything scheduled, also add an overall run-time limit at the orchestration layer (a cron/CI job timeout) as a second line of defence, in case an individual call's timeout logic has its own bug.

The root causes and trade-offs.

The default, and the actual bug here, is simply having no timeout set at all. A related but less common cause is a timeout that was set, but too short for a call that's genuinely just slow sometimes, which causes unnecessary failures. Getting the timeout value right requires knowing the API's normal latency distribution, not guessing a round number.

The trap that less-experienced engineers fall into.

Assuming a hung script must be stuck in an infinite loop in its own logic, and debugging the wrong part of the code, when the actual cause is a blocking network call with no timeout that's simply waiting on a remote server that will never respond.

🎯 Interviewer follow-up questions you should expect.

• "A script is stuck for hours with no error — what's your first suspicion?" An outbound network call with no timeout, waiting indefinitely on an unresponsive remote server.
• "What do you do once the timeout is hit?" It depends on criticality — you retry with backoff, fail the run cleanly, or fall back to a cached value.

During an incident, a script retrying a failing API call with no delay ends up sending hundreds of requests a second at the already-struggling service.

What is actually happening (the mental model).

A naive retry loop (while True: try again immediately) assumes failures are brief blips — but during a real outage, immediate, unlimited retries from every affected client can turn a struggling service into a completely overwhelmed one, actively working against its own recovery, sometimes called a "retry storm."

How to work through it.

1. Add exponential backoff — wait progressively longer between each retry attempt (1s, 2s, 4s, 8s…) instead of retrying immediately, so load on the struggling service actually decreases over time rather than compounding.
2. Add a maximum retry count — an unbounded retry loop that never gives up is itself a bug; eventually the script should fail loudly and let something (an alert, an on-call human) decide the next step.
3. For a script that might run concurrently many times (many hosts, many pods), consider jitter — randomising the exact backoff delay slightly — so retries from many sources don't all land in the same synchronized burst.
4. Distinguish errors worth retrying (a transient timeout, a 503) from ones that never will be (a 401, a malformed request) — retrying the latter is pure wasted load with zero chance of success.

The root causes and trade-offs.

One cause is having no backoff at all, so retries compound the load instead of easing it. Another is having no maximum attempt count, so a failure becomes an infinite loop instead of a clean one. A third is retrying errors that were never retryable in the first place, which just wastes load for zero benefit. Backoff adds latency to eventual success, and that's a reasonable trade against making an outage worse.

The trap that less-experienced engineers fall into.

Adding a retry loop as a reflexive "make it more resilient" fix without backoff, a max attempt count, or a check on which errors are actually worth retrying — turning a script that's supposed to help into one that actively contributes to an outage.

🎯 Interviewer follow-up questions you should expect.

• "Why is a naive retry loop dangerous during a real outage?" Because unlimited, immediate retries from every affected client can overwhelm an already-struggling service — a retry storm.
• "What does a properly built retry look like?" Exponential backoff, a maximum attempt count, jitter across many callers, and only retrying errors that are actually transient.

A debugging print() statement left in a script dumps an entire API request — headers, auth token, and all — into the CI job's log output, which is visible to the whole team.

What is actually happening (the mental model).

Logging is often treated as inherently safe ("it's just for debugging"), but log output frequently ends up far more widely accessible than the code itself — CI logs, log aggregation systems, and shared dashboards — so anything printed there needs the same care as anything committed to source control.

How to work through it.

1. Immediately treat the exposed secret as compromised — rotate/revoke it — the same response as if it had been committed to Git, regardless of how briefly it was visible.
2. Remove the offending log statement, and specifically avoid logging entire request/response objects wholesale, since headers and bodies are exactly where secrets live.
3. Where logging request details is genuinely useful for debugging, log a redacted version explicitly — the URL and status code, not the full headers — rather than an all-or-nothing choice between full detail and no logging at all.
4. If the CI/logging platform supports it, add automated secret-scanning on log output as a backstop, the same instinct as scanning commits for secrets.

The root causes and trade-offs.

The root cause is usually a debug print() of a whole object that was left in after debugging was done, combined with no habit of redacting sensitive fields before logging anything request-related. Redacting takes a small amount of extra code, just stripping specific keys before logging, in exchange for a large reduction in exposure risk.

The trap that less-experienced engineers fall into.

Treating logs as a private, low-stakes space "just for me while debugging," and forgetting that CI output, aggregated logs, and dashboards are often visible to a much wider audience than the source code itself.

🎯 Interviewer follow-up questions you should expect.

• "A secret showed up in a log — what's your first action?" You treat it as compromised and rotate it immediately, exactly as you would for a secret committed to Git.
• "How do you prevent this going forward?" You never log whole request or response objects wholesale — you log a redacted, deliberately chosen subset, and consider automated secret-scanning on log output.

A script builds a shell command using string formatting with a value that ultimately comes from user or external input, and a security review flags it.

What is actually happening (the mental model).

Building a shell command by string-interpolating external input directly (os.system(f"ping {host}")) lets an attacker who controls that input append their own shell syntax (host = "8.8.8.8; rm -rf /") and have it executed with the script's own privileges — the vulnerability is in constructing a shell command from untrusted input, not in any single function being inherently unsafe.

How to work through it.

1. Replace shell-string construction with a subprocess call that takes arguments as a list, not a formatted string (subprocess.run(["ping", "-c", "1", host])) — this passes the value as a literal argument, never interpreted as shell syntax.
2. Avoid shell=True in subprocess calls wherever the command includes any external input — it's what re-introduces exactly the shell-interpretation risk the list form avoids.
3. Where a library exists for the underlying action (a DNS/ping library instead of shelling out to ping), prefer it — avoiding a subprocess call entirely removes this whole category of risk.
4. Treat any input crossing a trust boundary (user input, an API payload, a webhook body) as needing this same care, not just obviously "external" data.

The root causes and trade-offs.

The root cause is string-formatting untrusted input directly into a shell command, often combined with using shell=True out of convenience. The list form of subprocess.run is barely more verbose and closes the vulnerability entirely — there's essentially no real trade-off against just doing it correctly from the start.

The trap that less-experienced engineers fall into.

Treating "it's just an internal script, nobody malicious will use it" as a reason not to fix this — but input that "should" always be trusted (a hostname from a config file, a value from an internal API) can still end up attacker-controlled if any upstream system is ever compromised or misused.

🎯 Interviewer follow-up questions you should expect.

• "How do you avoid command injection when calling a shell command from Python?" You pass arguments as a list to subprocess.run, never build a shell command via string formatting, and avoid shell=True with any external input.
• "Isn't this only a risk for user-facing scripts?" No — any input crossing a trust boundary, including from an internal API or config, deserves the same care.

A script that's run unchanged for a year suddenly fails after a routine pip install on a new machine — nothing in the script itself changed.

What is actually happening (the mental model).

Without a pinned, recorded set of dependency versions, pip install resolves to whatever the latest compatible versions are right now — which can be meaningfully different, with breaking changes, from what was installed when the script was written and last tested, even though the script's own source hasn't changed at all.

How to work through it.

1. Pin exact dependency versions in a requirements.txt (or lockfile) generated from a known-working environment (pip freeze), rather than a loose, unversioned list of package names.
2. Reproduce the failure by installing the exact pinned versions the script was originally tested against, to confirm the diagnosis before chasing the wrong thing.
3. If pinned to old versions, deliberately plan an upgrade — reading the changelog for what changed — rather than being forced into an unplanned one by a fresh install elsewhere.
4. Run the script inside a virtual environment (from L1) consistently, so "what's installed system-wide on this machine" never silently substitutes for "what this script actually needs."

The root causes and trade-offs.

Having no pinned versions at all works fine today, but it may break on any future install. Having pinned versions that are never deliberately revisited avoids that, but it also means missing security fixes and improvements indefinitely. The right middle ground is pinning plus a deliberate, periodic, tested upgrade — not "always latest," and not "frozen forever."

The trap that less-experienced engineers fall into.

Assuming a script that "hasn't changed" can't have a new bug, and looking everywhere except the dependency versions — when the actual change was in what pip install resolved to on a fresh machine, not in the script's own code.

🎯 Interviewer follow-up questions you should expect.

• "A script fails on a new machine with no code changes — what do you suspect?" Unpinned dependencies resolving to newer, possibly breaking versions than what the script was originally tested against.
• "How do you prevent this?" You pin exact versions from a known-working environment, work inside a virtual environment, and upgrade deliberately rather than accidentally.

A script that creates a JIRA ticket from a webhook event gets triggered twice for the same event (a common webhook-delivery quirk), and now there are two identical tickets.

What is actually happening (the mental model).

Many systems (webhooks especially) offer at-least-once delivery, not exactly-once — meaning your script genuinely will be invoked more than once for the same logical event sometimes, and a script that isn't idempotent (running it twice doesn't produce the same end result as running it once) will duplicate whatever side effect it has, every time this happens.

How to work through it.

1. Design the script to check for existing state before acting — before creating a ticket, search for one already linked to this same source event's unique ID, and skip creation if it's already there.
2. Use a natural or explicit idempotency key — some unique identifier from the triggering event — stored alongside the created resource, so a repeat delivery can be recognised and safely ignored.
3. Where the downstream system supports it, prefer an "upsert" (create-or-update) over a blind "create," so a repeat call converges to the same state rather than creating a duplicate.

The root causes and trade-offs.

This usually comes from treating "the webhook fired" as equivalent to "this event has never been processed before," when at-least-once delivery explicitly doesn't guarantee that, combined with having no existing-state check in place before acting. Adding an idempotency check costs a small amount of extra logic, usually just one lookup, for a large reliability gain on anything triggered by an external, at-least-once system.

The trap that less-experienced engineers fall into.

Assuming a script only ever runs once per logical event because "that's how it's supposed to work," instead of designing for the actual, documented at-least-once delivery guarantee that most webhook and queue systems provide.

🎯 Interviewer follow-up questions you should expect.

• "Why did the same webhook event trigger the script twice?" Because most webhook systems are at-least-once delivery, not exactly-once — duplicate invocations for the same event are expected, not a bug in the webhook itself.
• "How do you make a script safe to run more than once for the same event?" You check for existing state, an idempotency key, before acting, or you use an upsert instead of a blind create.

A script updating tags on 200 EC2 instances crashes on instance 140 — leaving the fleet in an inconsistent, partially-updated state.

What is actually happening (the mental model).

A script that processes a batch sequentially with no tracking of progress leaves you, on any mid-run failure, unsure exactly which items succeeded and which didn't — the fix isn't "prevent all crashes" (impossible), it's designing the batch so a partial failure is recoverable and re-runnable, not a mystery to reconstruct by hand.

How to work through it.

1. Make each individual item's operation idempotent (see the previous scenario) — updating a tag that's already set to the desired value should be a harmless no-op, not an error.
2. Log each item's outcome as it happens (succeeded/failed/skipped, with the specific reason), not just an overall pass/fail at the very end — this is what tells you exactly where to resume, rather than guessing.
3. Structure the script so it can simply be re-run against the same full list — idempotent operations mean the already-completed 139 are safely no-ops, and it naturally picks up from where it actually left off.
4. For a large or critical batch, consider processing in smaller chunks with a checkpoint, so a crash loses less progress and the blast radius of a partial failure is smaller.

The root causes and trade-offs.

One cause is having no per-item logging, which makes the actual state of the batch a mystery after a crash. The other is operations that aren't idempotent, which makes a naive re-run risky, since it could double-apply a change, rather than safe. Per-item logging and idempotent operations cost a bit more upfront design, in exchange for a batch script that's dramatically safer to re-run after any failure.

The trap that less-experienced engineers fall into.

Treating a batch script as "either it all works or something's badly wrong," with no per-item tracking — so a crash partway through forces a slow, manual, error-prone reconstruction of what actually got done versus what still needs doing.

🎯 Interviewer follow-up questions you should expect.

• "A batch script crashes partway through — how do you recover?" Per-item logging tells you exactly what succeeded, and idempotent operations mean you can simply re-run the whole batch safely.
• "How do you design a batch script to be safely re-runnable from the start?" You design idempotent per-item operations plus per-item outcome logging, not just an overall pass or fail.

A script that's parsed the same API's JSON response reliably for months suddenly throws a KeyError — the API returned something slightly different this time.

What is actually happening (the mental model).

Code that indexes directly into a dict (data["field"]["nested"]) is implicitly assuming the response always has that exact shape — but real APIs can return an optional field as absent rather than null, change a field's structure in a version bump, or return an error-shaped payload instead of the expected success shape, and direct indexing has no way to handle any of that gracefully.

How to work through it.

1. Use .get("field") with an explicit default instead of direct indexing wherever a field is genuinely optional, so a missing key doesn't crash the whole script.
2. Validate the response's overall shape early — check the HTTP status code and, if useful, whether the payload looks like the expected success shape — before assuming any of its fields are safe to read.
3. Wrap the parsing logic in a try/except that catches the specific parsing exceptions you'd expect (KeyError, TypeError) and logs the actual unexpected payload for debugging, rather than crashing with no context about what was actually returned.
4. If the API is versioned, pin to a specific version where possible so its response shape doesn't shift underneath you unannounced.

The root causes and trade-offs.

This usually comes from direct dict indexing with no defensive checks, combined with no validation of the response's overall shape or status before parsing fields out of it, often on an unversioned or newly-changed API. Defensive parsing, using .get() and explicit validation, adds a little code in exchange for real protection against exactly this kind of external, out-of-your-control change.

The trap that less-experienced engineers fall into.

Writing the happy-path parsing logic once against a response that looked consistent in testing, and never revisiting it to handle the response shapes that are merely likely eventually, not the one shape observed once during development.

🎯 Interviewer follow-up questions you should expect.

• "A script that's worked for months suddenly throws a KeyError — what's your first thought?" That the external API's response shape likely changed — you check what was actually returned before assuming the script's own logic broke.
• "How do you make JSON parsing more resilient to this?" You use .get() with explicit defaults instead of direct indexing, validate the response shape and status before parsing fields, and catch specific parsing exceptions with real logging.

A log-processing script that's always worked fine gets OOMKilled the first time it's pointed at a genuinely large log file.

What is actually happening (the mental model).

Reading an entire file into memory at once (f.read() or iterating a list built from the whole file) works fine until the file's size approaches or exceeds available memory — the script's logic was never wrong, it just implicitly assumed "the file will always be small enough to fully load," an assumption that held until it didn't.

How to work through it.

1. Confirm the failure is genuinely about file size vs. available memory (check the file's size against the process's memory limit) before assuming a different bug.
2. Switch from loading the whole file at once to streaming it — iterating a file object line by line (for line in f:) processes one line at a time without ever holding the entire file in memory simultaneously.
3. If the processing genuinely needs to look at data in aggregate (not just line by line), consider chunked processing — reading and processing fixed-size blocks — or, for very large-scale needs, a tool actually designed for out-of-memory-scale data processing rather than a plain Python script.

The root causes and trade-offs.

One cause is loading a whole file into memory when line-by-line streaming would do the same job, which is an easy fix that usually needs no logic change at all. The other is genuinely needing whole-file context for the processing logic, which is a harder redesign, or an argument for a different tool entirely. Streaming is nearly always worth doing by default for anything that might ever see a large file, rather than only fixing it after the first OOM.

The trap that less-experienced engineers fall into.

Writing and testing a script against a small sample file, where loading everything into memory works fine and feels simpler, and never revisiting that choice until it's run against real production-scale data for the first time.

🎯 Interviewer follow-up questions you should expect.

• "A script gets OOMKilled processing a large file — what's the likely fix?" You switch from reading the whole file into memory to streaming it line by line, which never holds the entire file in memory at once.
• "When would streaming not be enough?" When the processing genuinely needs whole-file context, which points toward chunked processing or a dedicated large-data tool instead of a plain script.

A script written in an afternoon for a one-time migration is, two years later, still running weekly on someone's laptop because it turned out to be needed on an ongoing basis.

What is actually happening (the mental model).

A script's engineering rigor should match how it's actually used, not how it was originally intended — "one-off" scripts routinely become de facto permanent infrastructure without anyone deciding that deliberately, and by the time that's obvious, it's often carrying real operational weight with none of the discipline (error handling, secrets management, monitoring, ownership) that would normally accompany something depended on this heavily.

How to work through it.

1. Recognise the signal explicitly — a script running on a schedule, or repeatedly by hand, for months is no longer "one-off" regardless of its original intent, and deserves a deliberate decision about its future.
2. Retrofit the basics it's likely missing: proper error handling, secrets out of hard-coded values, logging, and — critically — it running somewhere other than one person's laptop (a scheduled job, CI, a small service) so it doesn't depend on that one person's machine being on.
3. Give it a named owner and some documentation, even minimal, since "the person who wrote it" being the only source of institutional knowledge is itself a real risk if they leave or forget the details.
4. Decide deliberately whether it's worth properly re-platforming (into a scheduled job, a small dedicated service) versus continuing as a script, rather than it continuing to grow in importance by default with no such decision ever made.

The root causes and trade-offs.

This usually comes from nobody ever revisiting a script's status as its usage pattern quietly shifts from one-off to load-bearing, combined with the effort to properly harden it competing with other work, so there's rarely a forcing function until something actually breaks. The cost of retrofitting it now is real, but far smaller than the cost of it failing silently later with no owner, no error handling, and no memory of why it exists.

The trap that less-experienced engineers fall into.

Continuing to treat something as a "quick script" indefinitely, based on how it started, rather than periodically reassessing how it's actually being used and whether that usage now demands real operational discipline.

🎯 Interviewer follow-up questions you should expect.

• "How do you know when a one-off script needs to be taken seriously?" When it's running on a schedule or repeatedly relied upon for months, regardless of its original intent — that's the signal, not how it started.
• "What does 'taking it seriously' actually involve?" Proper error handling, secrets out of hard-coded values, logging, a named owner, minimal documentation, and running somewhere other than one person's laptop.

A cron job that's supposed to run nightly hasn't actually run in a week — the cron entry itself was silently removed during a server migration — and nobody noticed until a downstream report was missing data.

What is actually happening (the mental model).

A scheduled script that only logs when it runs has no way to alert anyone when it doesn't run at all — the absence of a log entry looks identical to "nothing to report" and identical to "this hasn't executed in a week," and nothing distinguishes them without an external check specifically watching for the expected execution itself.

How to work through it.

1. Add a "dead man's switch" style check — the script pings a monitoring endpoint (or writes a timestamp somewhere) every time it successfully runs, and a separate, independent alert fires if that expected ping hasn't arrived within the expected window.
2. Don't rely solely on the script's own internal logging to detect its own absence — by definition, if it's not running at all, it can't log anything about that fact itself.
3. Treat the underlying infrastructure (the cron entry, the CI schedule, the orchestrator's job definition) as itself something that can silently disappear during unrelated changes like a migration, and make its continued existence something checked periodically, not just assumed.

The root causes and trade-offs.

The root cause is usually having no independent "did this run" monitoring in place, relying entirely on the script's own logs, which say nothing at all if it never executes. It's made worse by scheduling infrastructure, like a cron entry or a CI schedule, that can be silently lost during a migration or a config change with nothing in place to flag its disappearance. Independent run-monitoring costs a small amount of setup, in exchange for a real guarantee against exactly this kind of silent gap.

The trap that less-experienced engineers fall into.

Assuming that good internal logging inside the script is sufficient monitoring — it's entirely sufficient for diagnosing a failure once you know something ran and had a problem, but useless for detecting that the script silently stopped running at all.

🎯 Interviewer follow-up questions you should expect.

• "How do you detect a scheduled job that silently stopped running entirely?" You use an independent 'did this run' check, a dead man's switch pinging a monitor on success, since the script's own logs say nothing if it never executes.
• "Isn't good internal logging enough?" No — it helps diagnose a failure once you know something ran, but it can't detect the job not running at all.

"It works on my machine" — a script runs perfectly on a developer's laptop but fails immediately when run in CI or in production.

What is actually happening (the mental model).

"Works locally, fails elsewhere" almost always means the local environment has some implicit dependency that isn't actually part of the script's declared requirements — a specific Python version, a package installed globally outside any virtual environment, an environment variable set in the developer's own shell profile, or credentials cached from a previous manual login — none of which travel with the script itself to a fresh environment.

How to work through it (the systematic check).

1. Compare Python versions between the two environments first — subtle syntax or standard-library behaviour differences between versions are a very common, easy-to-check culprit.
2. Check whether the local environment has anything installed outside the project's declared dependencies (a virtual environment, a requirements file) — a globally installed package masking a missing declared dependency is one of the most common causes.
3. Diff environment variables between the two environments — a value quietly set in the developer's shell profile (a token, a config path) that was never actually declared as a requirement for running the script.
4. Check for cached credentials or local state (a config file in the developer's home directory, a previously-authenticated CLI session) that the fresh CI/production environment simply doesn't have.

The trap that less-experienced engineers fall into.

Debugging the script's logic itself first, on the assumption that "it works locally" proves the code is correct — when the actual gap is almost always an environment difference invisible from reading the code, which no amount of staring at the script's logic will reveal.

🎯 Interviewer follow-up questions you should expect.

• "A script works locally but fails in CI — what's your systematic approach?" You compare Python versions, check for dependencies installed outside the declared requirements, diff environment variables, and check for cached local credentials or state.
• "Why doesn't this show up by reading the script's code?" Because the actual gap lives in the environment surrounding the script, not in its logic — the code can be completely correct and still fail purely due to what's different about where it's running.