HomeFoundationsSDLC
Day 2 — the lifecycle every one of your later tools plugs into

SDLC & Organisation Roles

Before any pipeline, container, or cluster, software goes through a lifecycle — an idea becomes a requirement, a requirement becomes a design, a design becomes code, and code becomes something running and maintained. This chapter rebuilds Day 2 into a workshop: first the phases of that lifecycle and how design fits into it, then the roles and where DevOps actually sits among them, then the interview questions the instructor flags, and finally the senior scenarios where the process itself — not a tool — is the whole problem.

① Theory② Interview Q&A③ Scenario drills
Day 2 · The Software Development Life Cycle, explained end to end Prerequisite video · Organisation roles, and where DevOps fits among them
⚡ The whole of SDLC in one mental model

The Software Development Life Cycle is the repeating sequence every piece of software moves through, from "someone has an idea" to "it's running and maintained in production." The classic phases are requirements, design, implementation, testing, deployment, and maintenance — and the reason this matters for DevOps specifically is that nearly every tool in this course automates or accelerates one of these phases: CI/CD automates implementation-to-deployment, monitoring is the feedback loop inside maintenance, and infrastructure as code turns "design" into something version-controlled rather than a diagram nobody updates.

Requirements Design Implementation Testing Deployment Maintenance a cycle,not a line
A cycle, not a straight line. Maintenance feeds new requirements back in — the SDLC repeats for as long as the software is alive, which is exactly why "deployment" was never meant to be the finish line.

Two ideas carry this chapter: (1) the SDLC is a cycle, not a one-way pipeline — maintenance surfaces new requirements, which restarts the loop, which is why a "finished" project is really just one that's paused; and (2) the phases map onto real organisational roles — product/business analysts own requirements, architects own design (both high-level and low-level), developers own implementation, QA owns testing, and DevOps sits across implementation, testing, deployment, and maintenance, which is exactly why it's described as removing a wall rather than being one more silo alongside the others.

01
Phase 1 · Theory

The lifecycle, its design layer, and the roles built around it

First what the SDLC is and why every team follows some version of it, then the phases as an actual repeating cycle, then the split between high-level and low-level design, then the organisational roles mapped onto each phase.

T1

What the SDLC is, and why it exists

Without a shared structure, "building software" becomes a different, ad hoc process every single time — the SDLC is what stops that.

The Software Development Life Cycle is the standard sequence of phases a piece of software moves through: someone identifies a requirement, it gets turned into a design, the design becomes real code through implementation, that code is verified through testing, it's shipped through deployment, and afterward it needs maintenance to keep running and improving. There's no single official SDLC — different teams follow the phases in different orders and with different amounts of ceremony (waterfall does them once, in strict sequence; Agile cycles through them repeatedly in small sprints) — but nearly every team recognises some version of these six phases, because skipping one tends to reappear as pain later: skip design and implementation stalls on unclear direction; skip testing and users find the bugs instead.

T2

The phases, as a cycle

The most common mistake in describing the SDLC is drawing it as a straight line — it's a loop, and that loop is the whole reason "maintenance" matters as much as "deployment."

Once software is deployed, it enters maintenance — bug fixes, performance tuning, and small improvements — and maintenance is where real usage surfaces new requirements: a bug report is a new requirement ("this must not happen"), a user request is a new requirement ("we need this too"). Those new requirements feed straight back into the start of the cycle. This is why treating "deployment" as the finish line is a category error — for any software still in active use, the SDLC is continuously running, not a project with a single end date. DevOps' entire contribution to this loop is making the deployment-to-maintenance-to-new-requirement leg faster and safer, which is exactly what the DORA metrics from the fundamentals chapter measure.

T3

High-level design vs low-level design

"Design" in the SDLC isn't one activity — it splits into two very different altitudes of decision, usually owned by different people.

High-level design (HLD) is the architecture altitude: which major components exist, how they talk to each other, what technology choices anchor the whole system (a monolith or microservices, which database family, which cloud). It's usually owned by an architect or a senior engineer and is deliberately light on implementation detail. Low-level design (LLD) is the implementation altitude: the actual classes, functions, database schemas, and API contracts within one component from the HLD — detailed enough that a developer can start writing code directly from it. The practical reason to separate them: HLD decisions are expensive to reverse (rearchitecting a monolith into microservices is a multi-month project) while LLD decisions are cheap to reverse (renaming a function is a five-minute change) — so HLD deserves more upfront scrutiny and sign-off than LLD does.

T4

Organisation roles, and where DevOps fits

Each SDLC phase tends to have an owner in a real organisation — and DevOps is unusual in that it doesn't own one phase, it spans several.

A typical mapping: a product manager or business analyst owns requirements — talking to stakeholders and users to define what's actually needed. An architect (sometimes a senior engineer wearing that hat) owns high-level design; developers doing their own low-level design as they implement. A software engineer owns implementation — writing the actual code against that design. QA/test engineers own testing, though in a mature DevOps culture much of this is automated into the pipeline rather than performed manually by a separate team. And DevOps engineers sit across implementation, testing, deployment, and maintenance — building and running the pipeline that carries code through all of those phases, and closing the loop with the monitoring that turns production behaviour back into new requirements. This is precisely why DevOps is described as removing a wall rather than being one more silo alongside product, architecture, and QA — its job is the connective tissue between the phases, not ownership of a single one.

02
Phase 2 · Interview Q&A

The SDLC questions you'll be asked

These are the questions the instructor explicitly flags on Day 2 and in the organisation-roles prerequisite video, with answers built from his explanations. Answer each out loud, then reveal.

Q1What are the phases of the SDLC?reveal ▸
Answer

Requirements, design, implementation, testing, deployment, and maintenance. Different methodologies run through them differently — waterfall once, in strict sequence; Agile repeatedly, in small sprints — but nearly every team recognises some version of these six phases.

Q2Why is the SDLC better understood as a cycle than a straight line?reveal ▸
Answer

Because maintenance — bug fixes, performance tuning, user requests — surfaces new requirements that feed straight back into the start of the cycle. For any software still in active use, the SDLC never actually stops at "deployment"; it keeps looping for as long as the software is alive.

Q3What's the difference between high-level design and low-level design?reveal ▸
Answer

High-level design is the architecture altitude — major components, how they communicate, core technology choices — usually owned by an architect. Low-level design is the implementation altitude — classes, functions, schemas, API contracts within one component — detailed enough to code directly from. HLD is expensive to reverse and gets more scrutiny; LLD is cheap to reverse and gets less.

Q4Where does DevOps fit among the SDLC phases and roles?reveal ▸
Answer

DevOps doesn't own one phase — it spans implementation, testing, deployment, and maintenance, building and running the pipeline that carries code through all of them and closing the loop with monitoring that turns production behaviour into new requirements. That's why it's described as removing a wall between phases rather than being one more silo alongside product, architecture, and QA.

Q5What typically happens if a team skips the design phase?reveal ▸
Answer

Implementation stalls on unclear direction — developers make architectural decisions ad hoc, mid-coding, that should have been settled and reviewed upfront, which tends to surface later as inconsistent components or a costly rearchitecture. Skipping a phase doesn't remove its cost; it just moves that cost later and usually makes it larger.

Q6Who typically owns requirements, and why does that matter?reveal ▸
Answer

A product manager or business analyst, whose job is talking to stakeholders and users to define what's actually needed before any design or code happens. It matters because requirements gathered poorly or skipped entirely is the single most common root cause of a team efficiently building the wrong thing.

03
Phase 3 · Scenario drills

The senior layer — where the process itself is the problem

You now know the lifecycle and where the roles sit in it. These 13 scenarios are the senior version: technical debt, hostile code review, schema migrations, and a definition of "done" that half the team doesn't actually meet. Answer out loud, reveal, and mark yourself.

🔄

SDLC

13 scenarios · 1–13

The complete SDLC scenario set, worked through in depth. This is the thinking and the story behind each one, not answers to memorise. The goal is that a hostile code review or an ambiguous "done" is a puzzle you already know how to open, not a page you have to look up.

⚡ The universal process-health reflex

Before treating a process complaint as a personality problem, ask these in order:

  1. Is this actually a tooling/automation gap? Most process friction (style debates, slow environments, unclear "done") is fixable by moving the decision into a machine, not by asking people to try harder.
  2. Is the real cost of this visible to the people who'd fund a fix? Technical debt and process pain are invisible until translated into business terms.
  3. Does the fix close this one instance, or the whole class of problem? A patch that doesn't add a test, a gate, or a rule just guarantees a repeat.
  4. Is the process serving the outcome, or has the outcome been forgotten? Ceremonies and gates exist to serve delivery — the moment they don't, they're the problem, not the fix.

Almost every scenario below is one of these four questions applied to a specific situation. Get the reflex automatic and the rest is detail.

---

📋 The full scenario inventory (distinct — no padding)

A. Velocity & debt

  1. Feature pressure is quietly burying the team in technical debt
  2. Getting leadership to fund a technical-debt paydown
  3. Shifting problems left — catching them earlier in the cycle

B. Review & quality culture

  1. Code review culture & standards

C. Production & delivery risk

  1. Handling a production bug found post-release
  2. Feature flags & progressive delivery

D. Requirements & process fit

  1. Requirements are unclear / changing
  2. Environments & parity (works in staging, broke prod)
  3. Choosing a development methodology

E. Data & multi-team delivery

  1. Database schema changes / migrations safely
  2. Releases across multiple services / teams

F. Testing & done

  1. Testing strategy / the test pyramid
  2. Quality gates & definition of done

---

1Feature pressure is quietly burying the team in technical debtConcept

Every sprint is scoped for new features, with no time ever allocated to the shortcuts and workarounds quietly piling up underneath them. Things are visibly getting slower to build.

The mental model.

Technical debt behaves exactly like financial debt: a shortcut today buys speed now at the cost of paying interest — extra effort — on every future change that touches that area. Taken occasionally and deliberately, it's a legitimate trade-off; taken constantly and invisibly, the interest payments eventually consume more time than the shortcuts ever saved, and nobody notices until velocity has already been quietly falling for months.

The playbook.

  1. Make debt visible. Track it the same way you track a feature backlog — a specific, named item, not a vague shared feeling that "this codebase is messy."
  2. Distinguish deliberate debt from accidental debt. A shortcut taken knowingly to hit a real deadline, with a plan to repay it, is very different from debt that just accumulated from nobody ever revisiting a decision.
  3. Allocate real, recurring capacity to paying it down — a fixed percentage of every sprint, not "whenever there's spare time," because there's never spare time by default.
  4. Tie debt items to the pain they're actually causing (this area takes 3x longer to change safely, this area causes X% of production incidents) so paying it down competes fairly against features on the same terms — impact, not vibes.

The trap that less-experienced engineers fall into.

Treating technical debt as a purely engineering concern to raise only among other engineers, rather than translating it into terms — velocity, incident rate, time-to-change — that make it a legitimate, fundable item in the same conversation as new features.

🎯 Interviewer follow-up questions you should expect.

  • "How do you know if technical debt is actually a problem, not just an engineer's preference?" You tie it to a measurable cost — slower delivery in that area, a disproportionate share of incidents — not a vague sense that the code is ugly.
  • "How much time should go to debt versus features?" A fixed, recurring allocation agreed with leadership is the answer, not an ad hoc "if there's time," which in practice always loses to feature pressure.
Say it like this"Technical debt is just financial debt applied to code — a shortcut buys speed now and charges interest later, and taken constantly without ever repaying it, the interest eventually costs more than the shortcuts ever saved. I make it visible as tracked items, distinguish debt we took on deliberately from debt that just accumulated, and push for a fixed recurring share of every sprint to pay it down rather than 'whenever there's time,' because there's never spare time by default. And I always tie a debt item to its actual cost — slower changes, more incidents — so it competes fairly against features instead of losing every time on vibes."

---

Mark:
2Getting leadership to fund a technical-debt paydownConcept

You want a sprint (or several) dedicated to paying down technical debt, but leadership sees it as engineers wanting to "clean up code" instead of shipping anything customers will notice.

The mental model.

Leadership funds outcomes, and "clean code" isn't one — the debt has to be translated into the same currency as everything else they fund: it's currently costing X extra days per feature in this area, or it caused Y% of last quarter's incidents, or it's the reason the team can't safely take on a specific new opportunity. Framed that way, paying it down isn't a favour to engineers, it's restoring lost delivery capacity — a completely fundable outcome.

The playbook.

  1. Quantify the cost in business terms — extra delivery time, incident frequency, or a concrete opportunity currently blocked by this debt — not "the code is hard to work with."
  2. Propose a bounded, scoped piece of debt work with a clear before/after metric, the same way you'd pitch any other project, rather than an open-ended "we need to clean things up."
  3. Tie it to a near-term business need where possible — "we can't safely add the feature you want next quarter until this is addressed" lands far better than debt paydown as an abstract virtue.
  4. Report the result the same way you'd report a feature's success — the delivery-time or incident-rate improvement — so debt paydown earns the same credibility as feature work for next time.

The trap that less-experienced engineers fall into.

Pitching debt paydown in purely technical language ("this code is a mess," "we need to refactor this") instead of business language (cost, risk, blocked opportunity) — which reads to leadership as an aesthetic preference rather than a funded, measurable investment.

🎯 Interviewer follow-up questions you should expect.

  • "How do you pitch a debt paydown to non-technical leadership?" You quantify its cost in business terms — delivery time, incident rate, a blocked opportunity — and propose it as a bounded, measurable project, not an open-ended cleanup.
  • "What if they still say no?" You keep tracking and reporting the cost until a specific blocked opportunity makes the trade-off undeniable, the same escalation path as any underfunded but real risk.
Say it like this"I never pitch this as 'the code is messy' — leadership funds outcomes, not aesthetics. I quantify the actual cost: this area takes three times longer to change safely, or it's behind a specific percentage of last quarter's incidents, or it's genuinely blocking a feature they want next quarter. Then I propose a bounded, scoped piece of work with a clear before-and-after metric, the same way I'd pitch any feature, and report the result the same way afterward — so debt paydown earns the same credibility as shipping a feature, instead of being treated as a favour to engineers."

---

Mark:
3Shifting problems left — catching them earlier in the cycleConcept

Bugs, security issues, and design flaws keep getting caught at the very end of the cycle — in testing, or worse, in production — where they're most expensive to fix.

The mental model.

"Shift left" means moving a check from later in the SDLC to earlier — the further left (toward requirements and design) a problem is caught, the cheaper it is to fix, because fewer downstream decisions have been built on top of the mistake. A design flaw caught in design review costs a conversation; the same flaw caught in production costs a migration, an incident, and possibly a rollback.

The playbook.

  1. Move security earlier — static analysis and dependency scanning in CI (on every pull request), not a security review bolted on right before release.
  2. Move testing earlier — automated tests run on every commit, not a manual QA pass reserved for the end of a sprint.
  3. Move design review earlier — a lightweight design doc or discussion before implementation starts on anything nontrivial, not a retroactive architecture review after the code already exists.
  4. Measure where problems are actually being caught (a bug tracker tag for "caught in design" vs "caught in testing" vs "caught in production") to see whether shifting left is actually working, rather than assuming it is.

The trap that less-experienced engineers fall into.

Treating "shift left" as a slogan to repeat rather than a specific, measurable change to where in the pipeline a check actually runs — without concretely moving a scan, a test, or a review earlier, nothing has actually shifted.

🎯 Interviewer follow-up questions you should expect.

  • "What does 'shift left' actually mean in practice?" It means moving a specific check — security scanning, testing, design review — earlier in the SDLC, so problems are caught when they're cheaper to fix.
  • "How do you know if shifting left is actually working?" You track where issues are actually being caught over time, comparing design versus testing versus production, and watch that distribution move earlier.
Say it like this"Shift left just means catching a problem earlier in the lifecycle, because the same flaw is cheap to fix in design review and expensive to fix in production, once other decisions have been built on top of it. Concretely that's static analysis and dependency scanning on every pull request instead of a pre-release security review, automated tests on every commit instead of a manual QA pass at sprint end, and a lightweight design discussion before implementation starts on anything nontrivial. And I actually track where problems get caught over time, so I can tell whether shifting left is working rather than just assuming it from the slogan."

---

Mark:
4Code review culture & standardsConcept

Code reviews are slow, inconsistent, or hostile. People argue endlessly about trivial style points, pick at tiny details until the author is worn down, or adopt a gatekeeping tone.

The mental model.

A good review culture starts with small pull requests. A pull request with a thousand lines just gets a rubber-stamp, whereas one with a hundred lines gets real scrutiny. You automate everything mechanical, so formatting and linting are enforced by the CI system and are never argued about by people, because style arguments only breed resentment. You then reserve human review for the things a machine cannot check: the logic, the design, and the security. You separate the issues that must block the change from the minor nitpicks, and you label them clearly. You phrase your concerns as questions, such as "what happens if this value is null?", and you pick your battles, especially with more junior engineers, so that a review does not feel demoralising. Most importantly, the senior engineers set the tone: how you give a review, and how you take one, defines the whole team's culture and its sense of psychological safety.

The playbook.

  1. Keep pull requests small, so that each one can be reviewed in under thirty minutes, and coach people towards making small changes.
  2. Automate the mechanical checks, so that formatting and linting run in CI and are never argued about by people.
  3. Have people review the logic, the design, and the security — the things that a machine cannot check.
  4. Clearly label whether a comment is blocking or just a minor nitpick, phrase your concerns as questions, praise the things that are done well, and pick your battles.
  5. Use a CODEOWNERS file to route each change to the right expert, and set a target turnaround time for reviews so that pull requests do not sit and rot.

The trap that less-experienced engineers fall into.

Arguing about style, which the CI system should enforce automatically; wearing the author down with a thousand tiny nitpicks, which is demoralising, especially for junior engineers; and using a gatekeeping tone that makes the review feel like a power trip rather than a shared effort to improve quality and to learn.

🎯 Interviewer follow-up questions you should expect.

  • "What makes a good review culture?" Small pull requests, automated style checks, people reviewing the logic, design, and security, clear labelling of what is blocking versus a nitpick, comments that are kind and specific, and senior engineers setting the tone.
  • "How do you handle style debates in review?" You automate them in CI, and never argue about style between people.
  • "How do you keep review from demoralising junior engineers?" You separate the blocking issues from the nitpicks, phrase concerns as questions, pick your battles, and praise what is done well.
Say it like this"A good review culture starts with small pull requests. A thousand-line pull request just gets a rubber-stamp, while a hundred-line one gets real scrutiny. I automate everything mechanical: formatting and linting are enforced by CI and never debated by people, because style arguments in review are pure friction. Human review then focuses on the logic, the design, and the security. I separate the blocking issues from the nitpicks and label them, I phrase concerns as questions like 'what happens if this is null?', I pick my battles, especially with junior engineers, and I call out what is good. Reviews should spread knowledge and catch real problems, not be a gatekeeping power trip — and the culture is set by how the senior engineers behave in them."

---

Mark:
5Handling a production bug found post-releaseConcept

A bug has slipped through to production, and you need to respond both immediately and in a way that stops it happening again.

The mental model.

The first thing you do is triage the problem and mitigate it, using whatever restores users fastest — rolling back, turning a feature flag off, or applying a hotfix — and then you communicate. But the senior part is what happens afterwards. You add a regression test that reproduces the bug, so that this exact issue can never ship again. And you run a blameless retrospective that asks not just "what was the bug?" but "how did it escape?" Was it a gap in testing, a gap in code review, or a gap in monitoring? The valuable outcome is not the one-line fix; it is closing the systemic hole so that a whole class of bugs stops getting through. A bug that is fixed without a test is a bug that will come back.

The playbook.

  1. Triage the severity and impact, then mitigate it (by rolling back, turning off a flag, or hotfixing), and then communicate what is happening.
  2. Find the root cause of the bug.
  3. Add a regression test that reproduces it, so that it cannot happen again.
  4. Run a blameless retrospective on how it managed to escape, and close the gap in testing, review, or monitoring that let it through, so that you fix the whole class of problem rather than just this one instance.

The trap that less-experienced engineers fall into.

Fixing just the one-line bug and moving on, without adding a regression test (so it can come back) and without asking how it escaped (so the same kind of bug keeps getting through).

🎯 Interviewer follow-up questions you should expect.

  • "A bug reached production — what do you do?" You mitigate first, by rolling back, toggling a flag, or hotfixing, and then you add a regression test and run a blameless retrospective on how it escaped.
  • "Why a regression test?" A regression test means this exact bug can never ship again — a fix without a test can come back.
  • "What is the real deliverable here?" The real deliverable is closing the gap that let the bug through, which is fixing the whole class of problem, not just this single instance.
Say it like this"First I triage the impact and mitigate — I roll back, toggle the feature flag off, or apply a hotfix, whichever restores users fastest. Then I communicate honestly. But the senior part is what happens afterwards. I add a regression test that reproduces the bug so that this exact issue can never ship again, and I run a blameless retrospective that asks not just 'what was the bug?' but 'how did it escape?' — was it a gap in testing, in review, or in monitoring? The valuable outcome is not the one-line fix; it is closing the systemic hole so that a whole class of bugs stops getting through. A bug fixed without a test is a bug that will return."

---

Mark:
6Feature flags & progressive deliveryConcept

You need to be able to deploy risky features safely, and to separate when the code is shipped from when users actually see it.

The mental model.

Feature flags separate deploying from releasing. You ship the code to production without exposing it to anyone, then turn it on for internal users, then for a small group of customers, watching your metrics as you go, with an instant kill switch if anything goes wrong. This turns a frightening big-bang launch into a controlled dial that you can turn up gradually. It also makes trunk-based development possible, because unfinished work can be merged behind a flag. The discipline that people forget is the lifecycle of the flags themselves. A flag is effectively a branch in your code, and stale flags that are left switched on forever create a nightmare of combinations to test, so you track them and remove them once a feature has been fully rolled out.

The playbook.

  1. Ship the feature "dark", by deploying it behind a flag, and then enable it first for internal users, then for a small group of customers, and then more widely, watching your metrics at each stage.
  2. Have a kill switch, so that you can disable the feature instantly if there is a problem.
  3. Use flags to enable trunk-based development, where unfinished work can be merged behind a flag.
  4. Manage the lifecycle of your flags, and remove stale ones after a feature has fully rolled out, because they are a form of technical debt.

The trap that less-experienced engineers fall into.

Treating flags as permanent configuration and never removing them, so that they accumulate into a nightmare of combinations to test, or coupling deployment and release together so that every launch is a risky big-bang event.

🎯 Interviewer follow-up questions you should expect.

  • "How do feature flags help delivery?" They separate deploying from releasing: you ship the feature dark, roll it out gradually with a kill switch, and they also enable trunk-based development.
  • "What is the discipline people miss?" It's the lifecycle of the flags — you have to remove stale flags after rollout, or they become debt and a testing nightmare.
Say it like this"Feature flags let me deploy code to production without releasing it to users. I ship it dark, then turn it on for internal users, then for a small group of customers, watching the metrics as I go, with an instant kill switch if something is wrong. That separation is powerful: it turns a frightening big-bang launch into a controlled dial, and it enables trunk-based development because unfinished work can be merged behind a flag. The discipline people forget is the lifecycle. A flag is effectively a branch in the code, and stale flags left switched on forever create a nightmare of combinations to test, so I track them and remove them once a feature is fully rolled out."

---

Mark:
7Requirements are unclear / changingConcept

You are asked to build something, but the requirements are vague or keep changing.

The mental model.

Vague requirements are a signal to go and talk to people, not to guess and start building. You dig for the reason behind the request, because very often the thing that was asked for is not the real problem, and once you understand the real problem, solving it turns out to be simpler. You propose building a small first version, a minimum viable product, so that you get real feedback quickly, instead of spending a whole quarter carefully building the wrong thing. You write your assumptions down, so that any misunderstanding surfaces early. And you design the system so that it is easy to change, keeping the parts loosely coupled, because you already know the requirements are going to shift. A senior engineer's job is to stop the team from efficiently building the wrong thing.

The playbook.

  1. Clarify the request by digging for the reason behind it and for the real problem that lies behind what was actually asked for.
  2. Build a small first version to validate the idea, rather than fully building something that might turn out to be wrong.
  3. Write your assumptions down, so that any misalignment comes to light early.
  4. Design for change by keeping the parts loosely coupled, and push back constructively on the scope by laying out the trade-offs.

The trap that less-experienced engineers fall into.

Taking the vague requirement at face value, building in isolation, and delivering exactly what was asked for but not what was actually needed — which is the classic failure of efficiently building the wrong thing.

🎯 Interviewer follow-up questions you should expect.

  • "Requirements are vague — what do you do?" You clarify the reason behind the request, propose a small first version to get feedback, write down your assumptions, and design for change.
  • "Why a minimum viable product?" It validates quickly — it is far better to ship a small correct thing than a big wrong thing.
  • "What is the senior job here?" The senior job is to stop the team from efficiently building the wrong thing.
Say it like this"Vague requirements are a signal to go and talk to people, not to guess and build. I dig for the reason behind the request, because often the thing that was asked for is not the real problem, and solving the real problem is simpler. I propose building a small first version so that we get real feedback quickly, instead of spending a quarter carefully building the wrong thing. I write my assumptions down explicitly so that any misunderstanding surfaces early, and I design the system with loosely coupled parts so that change is cheap, because I know the requirements will shift. And I will constructively push back on the scope, laying out the trade-offs, because a senior engineer's job is to stop the team from efficiently building the wrong thing."

---

Mark:
8Environments & parity (works in staging, broke prod)Concept

A change passed all the tests in the staging environment, but then broke in production.

The mental model.

"It passed in staging but broke in production" means that your staging environment is not representative of production. The data, the scale, the configuration, or the underlying infrastructure is different in some way. You tackle this by building both environments from the same definitions using infrastructure-as-code, rather than configuring them by hand in slightly different ways, by keeping your configuration in code, and by using production-like data in staging (with any sensitive parts anonymised). But you also have to be realistic. You often cannot afford to reproduce production's full scale in staging, so where the gap is unavoidable, you compensate for it with canary releases, feature flags, and strong observability. This lets you test the final stretch safely in production itself, with a very small blast radius, rather than pretending that staging caught everything.

The playbook.

  1. Achieve parity by using infrastructure-as-code, so that the same definitions build every environment, by keeping configuration in code, and by using production-like data that has been anonymised.
  2. Acknowledge that some gap is unavoidable, because you cannot cheaply reproduce production's scale.
  3. Compensate for that gap with canary releases, feature flags, and strong observability, so that you can test the final stretch safely in production.

The trap that less-experienced engineers fall into.

Assuming that perfect parity between staging and production is achievable, so that "it passed in staging" is treated as proof, and configuring the environments by hand in different ways (which causes drift) instead of using infrastructure-as-code.

🎯 Interviewer follow-up questions you should expect.

  • "It passed staging but broke in production — what is the root cause and fix?" The environments have diverged — Fix it with parity through infrastructure-as-code and production-like data, and compensate for the unavoidable gap with canary releases, feature flags, and observability.
  • "You cannot replicate production scale in staging — what then?" You accept the gap and compensate with canary releases and observability, so that you test the final stretch safely in production.
Say it like this"'Passed staging, broke production' means staging is not representative — usually the data, the scale, or a configuration has drifted. I attack it with infrastructure-as-code, so that environments are built from the same definitions rather than configured by hand in different ways, with configuration kept in code, and with production-like anonymised data in staging. But I am realistic: you often cannot afford to reproduce production's scale in staging, so where the gap is unavoidable I compensate with canary releases, feature flags, and strong observability. I test the final stretch safely in production, with a very small blast radius, rather than pretending that staging caught everything."

---

Mark:
9Choosing a development methodologyConcept

You are asked how your team works — Agile or something else — or you find a team that is going through the motions of the ceremonies without getting any of the benefits.

The mental model.

A methodology exists to serve outcomes, not the other way around. Agile, Scrum, and Kanban are simply tools, and you choose between them based on the kind of work you do. Kanban suits work that is interrupt-driven, such as operations or platform work where tasks arrive unpredictably, while Scrum suits planned feature work. The anti-pattern is cargo-cult ceremonies, where a team performs the standups and retrospectives as a ritual while missing the actual point, which is fast feedback, small batches of work, continuous improvement, and delivering value. You adapt the process to fit the team's real work.

The playbook.

  1. Choose the methodology based on the type of work: Kanban for interrupt-driven operations, and Scrum for planned feature work. The point is always the outcome.
  2. Focus on the real goals: fast feedback, small batches, continuous improvement, and delivering value.
  3. Avoid cargo-cult behaviour, and do not perform ceremonies that do not serve the outcome.
  4. Adapt the process to the team.

The trap that less-experienced engineers fall into.

Treating the framework itself as the goal — for example, forcing rigid Scrum ceremonies onto an interrupt-driven operations team — or performing the ceremonies as a ritual without their underlying purpose, which is cargo-cult agile.

🎯 Interviewer follow-up questions you should expect.

  • "Scrum or Kanban?" It depends on the type of work: Kanban for interrupt-driven operations, and Scrum for planned features — the methodology serves the outcome.
  • "What is cargo-cult agile?" Cargo-cult agile means performing the ceremonies while missing the actual point, which is fast feedback, small batches, and continuous improvement.
Say it like this"I care about outcomes — fast feedback, small batches, and continuous learning — far more than which framework's ceremonies we perform. For a DevOps or platform team whose work is largely interrupt-driven, Kanban fits better than Scrum's fixed sprints, because the work arrives unpredictably; for planned product work, Scrum's cadence can help. The anti-pattern I actively push against is cargo-cult agile, where the team does standups and retrospectives as a ritual while missing the point, which is continuous improvement and shipping value. So I adapt the process to the team's actual work and treat the methodology as a means, never as the goal."

---

Mark:
10Database schema changes / migrations safelyConcept

A database schema change risks either downtime or data loss, and it is tied to a code deployment.

The mental model.

Schema changes are the hardest part of deploying safely, because they are often one-way and cannot easily be undone. The safe approach is an "expand and contract" migration, which is designed to stay backward-compatible at every step. First you add the new structure. Then you backfill the existing data into it and write to both the old and new structures at once, so that the old and new versions of the code can run side by side. Then you switch your reads over to the new structure. And only later, once everything is stable, do you remove the old structure. Each of these steps can be deployed on its own and can be reversed, which means you can roll the application back without ending up with a broken schema. You avoid migrations that lock a large table (by using online schema-change tools), and you never bundle a breaking schema change into the same deployment as the code that needs it, because that is exactly how you end up with a change you cannot roll back. You test both the migration and the rollback on production-like data first.

The playbook.

  1. Use expand-and-contract: add the new structure, backfill the data, write to both structures at once, switch the reads over, and only later remove the old structure. Each step is reversible, and the old and new code can coexist throughout.
  2. Avoid migrations that lock large tables, by using online schema-change tools.
  3. Never bundle a breaking schema change into the same deployment as the code that needs it, because then you could not roll back.
  4. Test both the migration and the rollback on production-scale data.

The trap that less-experienced engineers fall into.

Bundling a breaking, one-way schema change together with the code deployment, so that when the code fails they cannot roll back, because the schema has already changed. And running a table-locking migration on a large table, which causes an outage.

🎯 Interviewer follow-up questions you should expect.

  • "How do you do a safe schema migration?" You use expand-and-contract, so that the old and new code can coexist and every step is reversible; avoid locking migrations; and never bundle a breaking change into the deployment.
  • "Why is expand-and-contract important for rollback?" It matters because the old code still works against the new schema, so you can roll the application back safely.
Say it like this"Schema changes are the hardest part of deploying safely, because they are often one-way. I use expand-and-contract: I add the new structure, backfill the data and write to both the old and new structures so that both versions of the application work, migrate the reads over, and only later drop the old column. Each step can be deployed on its own and is reversible, which means I can roll the application back without a broken schema. On large tables I use online schema-change tools, so that the migration does not take a table lock and cause an outage. And I never bundle a breaking schema change into the same deployment as the code that needs it, because that is how you get a change you cannot roll back. I test both the migration and the rollback on production-scale data first."

---

Mark:
11Releases across multiple services / teamsConcept

A feature spans several services that are owned by different teams, and you have to coordinate the release across all of them.

The mental model.

The goal is that no service ever has to be deployed in lockstep with another. Deployments that have to be coordinated across many teams do not scale, and they turn every release into a scheduling nightmare. You reach that goal in three ways. First, you use API contracts that are versioned and backward-compatible, so that a service providing an API can deploy a change without breaking the services that consume it. Second, you use contract tests, so that a breaking change is caught in CI rather than in production. Third, you use feature flags to separate the act of turning on a cross-service feature from the order in which the services are deployed: everyone deploys independently, and then you flip the flag once all the pieces are live. The underlying design principle is independent deployability. If teams have to carefully choreograph their deployments, that is a sign of a deeper architectural problem.

The playbook.

  1. Use versioned, backward-compatible API contracts, so that a provider can evolve without breaking its consumers, and no lockstep is needed.
  2. Use contract tests, so that a breaking change is caught in CI rather than in production.
  3. Use feature flags to separate turning a feature on from the order of deployment: deploy each service independently, and then flip the flag.
  4. Design for independent deployability, and treat lockstep deployments as a smell to be removed.

The trap that less-experienced engineers fall into.

Trying to coordinate simultaneous deployments across many teams, which does not scale and becomes a scheduling nightmare, and breaking API compatibility without versioning or contract tests, which breaks the consumers in production.

🎯 Interviewer follow-up questions you should expect.

  • "A feature spans several teams' services — how do you release it?" You release it with versioned, backward-compatible contracts, contract tests, and feature flags to separate activation from deployment order, so that each service can be deployed independently.
  • "Why avoid lockstep deployments?" You avoid them because they do not scale — they become a scheduling nightmare, and every release turns into a heavy coordination exercise.
Say it like this"The goal is that no service has to deploy in lockstep with another, because coordinated multi-team deployments do not scale and turn every release into a scheduling nightmare. I get there with versioned, backward-compatible API contracts, so that a provider can deploy without breaking its consumers, and with contract tests that catch a breaking change in CI rather than in production. Feature flags then let us separate turning on a cross-service feature from the order of deployment — everyone deploys independently, and we flip the flag once all the pieces are live. The design principle is independent deployability; if teams have to choreograph their deployments, that is an architectural smell I would work to remove."

---

Mark:
12Testing strategy / the test pyramidConcept

The test suite is slow and unreliable, or it has the wrong balance: it is made up mostly of slow end-to-end tests with very few unit tests, which is known as the "ice-cream cone" anti-pattern.

The mental model.

The ideal shape is the test pyramid: a broad base of many fast unit tests, a smaller middle layer of integration tests, and a thin layer of end-to-end tests at the top. When that shape is turned upside down into the "ice-cream cone", with mostly slow and flaky end-to-end tests, the whole suite becomes slow and brittle, and it is bad at telling you where a problem is, because a failing end-to-end test does not point to the specific piece of code that broke. What you want is fast feedback from cheap unit tests, integration tests for the places where components join together, and just enough end-to-end tests to check the critical user journeys. Getting the balance right is the whole game.

The playbook.

  1. Aim for the pyramid shape: many unit tests (which are fast and isolate the logic), fewer integration tests (which cover the joins and contracts between components), and a thin layer of end-to-end tests (for the critical journeys only).
  2. Get your fast feedback from the base of the pyramid; end-to-end tests are slow, flaky, and expensive, so keep that layer thin.
  3. Avoid the ice-cream cone, because too many end-to-end tests make the suite slow and brittle, and poor at localising failures.
  4. Test the right thing at the right level, and do not use an end-to-end test to check something that a unit test could cover.

The trap that less-experienced engineers fall into.

Ending up with an inverted pyramid, the ice-cream cone, which is mostly slow and flaky end-to-end tests, so that the suite is slow, brittle, and unable to localise failures; and testing at the wrong level, such as using an end-to-end test to check pure logic.

🎯 Interviewer follow-up questions you should expect.

  • "What is a good test strategy?" The test pyramid: many fast unit tests, fewer integration tests, and a thin layer of end-to-end tests. Avoid the ice-cream cone.
  • "Why not lots of end-to-end tests?" You avoid them because they are slow, flaky, and expensive, and they do not localise failures — keep the end-to-end layer thin, for the critical journeys.
Say it like this"I aim for the test pyramid: a broad base of fast unit tests that isolate the logic, a smaller layer of integration tests for the joins and contracts between components, and a thin layer of end-to-end tests for the critical user journeys. The anti-pattern is the ice-cream cone, where you have mostly slow, flaky end-to-end tests — that makes the suite slow and brittle, and a failure does not tell you where the problem is. Fast feedback comes from the cheap unit tests at the base, and I keep end-to-end tests thin because they are slow and expensive. The whole game is testing the right thing at the right level."

---

Mark:
13Quality gates & definition of doneConcept

The meaning of "done" is fuzzy, so half-finished work gets called complete — code that compiles but has no tests, no documentation, no monitoring, and no way to roll it back.

The mental model.

"Done" has to mean "safely running in production and operable," not "the code compiles on my branch." A strong definition of done includes the tests passing, the code being peer-reviewed, the security scan coming back clean, and the documentation and runbooks being updated. It also includes the part that people usually skip: observability. The feature has to ship with the metrics, logs, and alerts needed to operate it. A feature that you cannot monitor or roll back is not done; it is a liability. You automate the mechanical parts of this in CI, so that "done" is partly enforced by the pipeline itself, rather than relying only on people's discipline.

The playbook.

  1. Have a clear definition of done: the tests pass, the code is reviewed, the security scan is clean, the documentation and runbooks are updated, and the observability — the metrics, logs, and alerts — has shipped with the feature.
  2. Make operability part of "done": a feature you cannot monitor or roll back is not done.
  3. Automate the mechanical gates in CI, so that the definition of done is enforced rather than merely intended.

The trap that less-experienced engineers fall into.

Treating "the code works and compiles" as the same thing as "done", so that tests, documentation, monitoring, and rollback are all missing, and the team ends up with a pile of features that nobody can support or safely change.

🎯 Interviewer follow-up questions you should expect.

  • "What does 'done' mean?" It means safely running in production and operable: the tests pass, it is reviewed and security-scanned, the documentation and runbooks are updated, and the observability has shipped.
  • "What do people forget in the definition of done?" People forget operability — the metrics, logs, alerts, and rollback needed to run it.
Say it like this"'Done' has to mean 'safely running in production and operable,' not 'the code compiles on my branch.' My definition of done includes the tests passing, the code being peer-reviewed, the security scan being clean, the documentation and runbooks being updated, and — the part people skip — observability: the feature ships with the metrics, logs, and alerts needed to operate it. A feature you cannot monitor or roll back is not done; it is a liability. I automate the mechanical gates in CI so that 'done' is partly enforced by the pipeline rather than relying on discipline. Making operability part of done is what stops us accumulating features that nobody can support at two in the morning."

---

Mark:
🧠 How to turn this into muscle memory
  • Drill the principles first — that small changes are safer, that you should shift problems left, that deploying is not the same as releasing, that "done" includes operability, that you fix the class of problem rather than the individual instance, and that process serves outcomes. Nearly every question is really an application of one of these.
  • Map each scenario to your own real experience. For every one, find a time you lived through it — a bug that escaped to production, an argument for tackling technical debt that you made to leadership, or a migration you did using expand-and-contract. It is the story that makes the answer land.
  • Take each scenario in three passes. Reason it out, say the spoken version out loud without looking, and then explain it to a colleague. Pay particular attention to velocity versus quality, framing technical debt in business terms, the expand-and-contract migration, and the definition of done, because these come up constantly.
  • Keep a log of the process decisions you make and their outcomes. The SDLC calls you make — how you framed a piece of technical debt, or how you handled a schema change — become the concrete stories you can tell in the behavioural round of an interview.

Next in order: the Project Management chapter, at this same depth.

No scenarios match that search. Clear search
SDLC & Organisation Roles — DevOps Zero → Hero. Theory & Interview rebuilt from Day 2 and the organisation-roles prerequisite video (the SDLC phases as a cycle, HLD vs LLD, roles and where DevOps fits); the 13 scenario drills are the Scenario Playbook's SDLC domain, rewritten in plain English. ← All chapters