System Design

Home›Advanced drills · playbook only›System Design

Playbook only — senior scenario drills, no course video behind this one

This is the thinking behind each one, not diagrams to memorise. Classic system design is app-centric (data models, APIs); the Cloud/DevOps lens adds what makes you distinctive — how the system is deployed, scaled, made reliable, observed, secured, and what it costs. That operability lens is your edge in the room.

① Scenario drills

⚡ The universal system design reflex

Every design question follows the same arc — narrate it out loud, in order, and you will never freeze:

1. Clarify requirements   — functional (what it does) + NON-functional (scale, latency, availability, consistency, cost)
2. Estimate               — back-of-envelope: QPS, storage, bandwidth → sizes the design
3. High-level design      — boxes and arrows: clients → LB → services → data stores → async
4. Deep-dive              — the interesting components; data model; the hot path
5. Identify bottlenecks   — where does it break at 10x? SPOFs? hot partitions?
6. Trade-offs             — every choice has a cost; state them explicitly

The mental model: there is no perfect design — only trade-offs for a given set of requirements. So you can't design anything until you know the requirements, especially the non-functional ones (a system for 100 users looks nothing like one for 100M). You start simple and scale the bottleneck, rather than over-engineering up front.

Your DevOps differentiators — bring these and you stand out from pure-software candidates:

Operability: how is it deployed (zero-downtime), observed (metrics/logs/traces), and debugged at 2am?
Reliability: failure domains, SPOFs, blast radius, graceful degradation.
Cost: what does this architecture cost, and where's the waste? (data transfer, over-provisioning).
Security: least privilege, encryption, network segmentation, secrets.

Phase 1 · Scenario drills

The senior layer — system design under real interview conditions

These 20 scenarios cover the Cloud/DevOps take on system design: the approach, the core building blocks, distributed-systems resilience, worked designs, and the DevOps differentiators that set you apart. Answer out loud, reveal, and mark yourself.

🧩

System Design

20 scenarios · 1–20

The complete System Design scenario set, from a Cloud/DevOps perspective, worked through in depth. This is the thinking behind each one, not diagrams to memorise.

📋 The full scenario inventory (distinct — no padding)

A. Approach & fundamentals

How to approach a Cloud/DevOps system design interview
Back-of-envelope capacity estimation
Scalability — vertical vs horizontal & stateless design
High availability — redundancy, failure domains & the nines

B. Core building blocks

Load balancing (L4 vs L7, health checks)
Caching strategies
Database scaling (replication, sharding, SQL vs NoSQL)
Async & message queues / event-driven design
API gateway & API design

C. Distributed-systems resilience

Designing for failure (timeouts, retries, circuit breakers, bulkheads)
Consistency & CAP in practice
Idempotency & delivery semantics

D. Worked designs (Cloud/DevOps flavoured)

A full worked example — design a URL shortener end-to-end
Design a scalable CI/CD platform
Design a multi-region active-active system
Design a scalable observability / logging pipeline
Design an internal developer platform (IDP) / container platform
Multi-tenancy architecture

E. The DevOps differentiators

Disaster recovery & zero-downtime deployment architecture
Designing for operability & observability from day one

1How to approach a Cloud/DevOps system design interviewDesign▸

An open-ended "Design X" (a URL shortener, a rate limiter, a deployment platform, a notification system). The prompt is deliberately vague to see how you structure ambiguity.

What is actually happening (the mental model).

The #1 failure is jumping straight to a solution. A senior drives the process: clarify requirements (especially non-functional), estimate scale, sketch high-level, deep-dive the interesting parts, then discuss bottlenecks and trade-offs. You treat it as a collaborative conversation, not a monologue — surface assumptions, ask about scale, and let the interviewer steer.

How to work through it.

Clarify functional + non-functional requirements — what must it do; and how much scale, what latency, what availability target, what consistency needs, what's the cost sensitivity? Nail the NFRs — they shape everything.
Estimate (scenario 2) — a rough QPS/storage/bandwidth to size the design.
High-level design — clients → load balancer → stateless services → data stores → async where needed.
Deep-dive the interesting components (the data model, the hot path, the tricky part).
Bottlenecks & trade-offs — where it breaks at 10×, SPOFs, and the explicit trade-offs you're making.

Key trade-offs and decisions.

The whole exercise is trade-offs — you're demonstrating that you can reason about them (consistency vs availability, cost vs performance, simplicity vs scale) rather than reciting one "right" architecture.

The trap that less-experienced engineers fall into.

Jumping to components ("I'll use Kafka and Cassandra") before understanding requirements or scale, and designing a monologue instead of a conversation.

🎯 Interviewer follow-up questions you should expect.

"Where would this break at 10× traffic?" Identify the bottleneck (hot partition, LB, DB writes) and how you'd scale it.
"What are you trading off here?" Name it (e.g. eventual consistency for availability).
"How would you deploy/monitor this?" Your DevOps edge — zero-downtime deploys, observability.

Say it like this"I resist jumping to a solution — I drive the process. First I clarify requirements, especially the non-functional ones: scale, latency, availability, consistency, and cost, because those shape everything and a system for a thousand users looks nothing like one for a hundred million. Then a quick back-of-envelope estimate to size it, a high-level design of clients through a load balancer to stateless services and data stores, and I deep-dive the interesting parts. Then I proactively call out where it breaks at 10×, the single points of failure, and the trade-offs I'm making — because there's no perfect design, only trade-offs for these requirements. And as a DevOps engineer I always bring the operability, reliability, and cost lens, which is where I add value beyond a pure app design."

Mark:

2Back-of-envelope capacity estimationDesign▸

"How many servers / how much storage does this need?" — or you need to size a design to justify your choices.

What is actually happening (the mental model).

Estimation isn't about precision — it's about getting the order of magnitude right so your design is appropriately sized and you can spot the bottleneck. You work from users → QPS → storage → bandwidth, using round numbers, and you know a few reference points cold (a day ≈ 86,400s ≈ 10⁵; common read/write ratios; typical latencies).

How to work through it.

Traffic: DAU × actions/day ÷ 86,400 ≈ average QPS; then peak QPS ≈ 2–3× average. Distinguish read vs write QPS (often 100:1 read-heavy → caching/replicas matter).
Storage: items/day × size/item × retention → total; project growth.
Bandwidth: QPS × payload size.
Memory (cache): apply the 80/20 rule — cache the hot 20% that serves 80% of reads.
Reference numbers: ~10⁵ seconds/day, memory access ~100ns, SSD ~0.1ms, network round-trip within a region ~0.5ms, cross-region ~50–150ms.

Key trade-offs and decisions.

The estimate tells you what kind of system you're building — a read-heavy 100:1 ratio pushes you toward caching and read replicas; huge write volume pushes you toward sharding/partitioning or an append-optimised store.

The trap that less-experienced engineers fall into.

Getting lost in false precision, or skipping estimation entirely so the design is un-sized (can't justify why you need a cache, replicas, or sharding).

🎯 Interviewer follow-up questions you should expect.

"What's the read/write ratio and why does it matter?" Read-heavy → caching, replicas; write-heavy → sharding/partitioning.
"How much cache memory?" 80/20 — the hot subset that serves most reads.
"Peak vs average?" Design for peak (2–3× average) with headroom.

Say it like this"Estimation is about order of magnitude, not precision — enough to size the design and find the bottleneck. I go users to QPS to storage to bandwidth: daily active users times actions over about 86,400 seconds gives average QPS, and I take peak as two-to-three times that. I always split read versus write, because a 100:1 read-heavy ratio immediately tells me I need caching and read replicas, whereas heavy writes push me toward sharding. For cache I apply the 80/20 rule — cache the hot 20% that serves 80% of reads. Those rough numbers justify every scaling decision I make in the design."

Mark:

3Scalability — vertical vs horizontal & stateless designDesign▸

"How would you scale this?" — or the design needs to handle growth.

What is actually happening (the mental model).

Vertical scaling (bigger box) is simple but has a ceiling and is a SPOF; horizontal scaling (more boxes) is how you scale indefinitely — but it only works if services are stateless, because any request must be servable by any instance. So the key enabler is externalising state (sessions in a shared store, data in managed DBs) so instances are interchangeable and can be added/removed freely behind a load balancer.

How to work through it.

Stateless services — externalise session/state to a shared store (Redis, DB) so any instance handles any request → add/remove instances freely.
Horizontal over vertical for the app tier — more commodity instances behind a load balancer, auto-scaled on the right metric.
Scale the data tier separately — read replicas, caching, sharding (scenario 7); the database is usually the real bottleneck.
Vertical scaling where horizontal is hard (some databases) — but know its ceiling.

Key trade-offs and decisions.

Vertical: simple, ceiling, SPOF. Horizontal: scales infinitely but needs statelessness + a LB + handles distributed-systems complexity. The app tier scales easily once stateless; the data tier is the hard part.

The trap that less-experienced engineers fall into.

Designing stateful app servers (sticky sessions, in-memory state) that can't scale horizontally, and forgetting the database is usually the real bottleneck (a stateless app tier doesn't help if the single DB can't keep up).

🎯 Interviewer follow-up questions you should expect.

"Why must services be stateless to scale horizontally?" Any instance must serve any request; state in a shared store makes instances interchangeable.
"What's usually the real scaling bottleneck?" The database — scale it with replicas/caching/sharding.
"Vertical vs horizontal trade-off?" Vertical is simple but has a ceiling and is a SPOF; horizontal scales indefinitely but adds complexity.

Say it like this"Vertical scaling — a bigger box — is simple but hits a ceiling and is a single point of failure. Horizontal scaling is how you scale indefinitely, but it only works if the services are stateless, so any request can be served by any instance. So the key enabler is externalising state — sessions and data go to a shared store or managed database — which makes instances interchangeable behind a load balancer, and I auto-scale them on the right metric. The app tier scales easily once it's stateless; the real bottleneck is almost always the data tier, which I scale separately with read replicas, caching, and sharding."

Mark:

4High availability — redundancy, failure domains & the ninesDesign▸

"How do you make this highly available?" — or you need to hit an availability SLA.

What is actually happening (the mental model).

Availability comes from redundancy across failure domains and eliminating single points of failure — a system is only as available as its weakest chokepoint. You reason in failure domains (instance → rack → AZ → region): redundancy must span the domain you need to survive. You know the nines (99.9% ≈ 8.7h/yr down; 99.99% ≈ 52min/yr) and that each nine costs exponentially more, so you match the target to the business need. And you design for graceful degradation (partial function beats total outage).

How to work through it.

Eliminate SPOFs — redundancy at every tier (multiple instances, multi-AZ LB/DB/app).
Failure domains — spread redundancy across the domain you must survive (multi-AZ for AZ failure; multi-region for region failure).
The nines — 99.9% vs 99.99% vs 99.999%; each costs exponentially more; match to the business requirement (don't over-build).
Graceful degradation — shed non-critical features under stress rather than fully failing (e.g. serve stale data, disable recommendations).
Health checks + auto-recovery — detect and replace failed components automatically.

Key trade-offs and decisions.

More nines = exponentially more cost and complexity (multi-region active-active is expensive). Redundancy costs money; match the availability target to what the business actually needs.

The trap that less-experienced engineers fall into.

Claiming "highly available" with a hidden SPOF (a single DB, a single LB, a single shared component), and over-building to five-nines when the business needs three.

🎯 Interviewer follow-up questions you should expect.

"What's a single point of failure here?" Find the un-redundant chokepoint; make it redundant.
"99.9% vs 99.99% — what's the cost?" Each nine is exponentially more expensive; match to the business need.
"What's graceful degradation?" Serve partial/stale function under stress rather than a total outage.

Say it like this"Availability comes from redundancy across failure domains and eliminating single points of failure — a system is only as available as its weakest chokepoint. I reason in failure domains: to survive an AZ failure I spread across AZs, to survive a region failure I go multi-region. I know the nines — 99.9% is nearly nine hours down a year, 99.99% is under an hour — and each nine costs exponentially more, so I match the target to the business need rather than over-building. I design for graceful degradation, shedding non-critical features under stress instead of failing entirely, and I use health checks with auto-recovery so failed components are replaced automatically."

Mark:

5Load balancing (L4 vs L7, health checks)Design▸

The design needs to distribute traffic — and the interviewer probes L4 vs L7, algorithms, and the LB itself as a SPOF.

What is actually happening (the mental model).

A load balancer distributes traffic across instances and is the enabler of horizontal scaling. L4 (transport) balances on IP/port — fast, protocol-agnostic; L7 (application) understands HTTP — can route by path/host/header, do TLS termination, and enable canary/blue-green. Health checks are what make it work (unhealthy instances are removed). And the LB itself must not be a SPOF — cloud LBs are managed and redundant; a self-managed LB needs HA.

How to work through it.

L4 vs L7 — L4 for raw speed/non-HTTP; L7 for content-based routing, TLS termination, and traffic-shifting (canary/blue-green). Most web systems use L7.
Algorithms — round-robin, least-connections, IP-hash (sticky); pick for the workload.
Health checks — remove unhealthy instances; tune thresholds so a slow-starting instance isn't killed.
The LB is not a SPOF — use a managed, redundant cloud LB (or HA pair); it's the front door.

Key trade-offs and decisions.

L4: faster, less flexible. L7: richer routing (canary, path-based) at slightly more overhead. Sticky sessions (IP-hash) help stateful apps but hurt even distribution — prefer stateless + no stickiness.

The trap that less-experienced engineers fall into.

Making the load balancer a hidden SPOF, and relying on sticky sessions (which undermine statelessness and even distribution) instead of externalising state.

🎯 Interviewer follow-up questions you should expect.

"L4 vs L7 — when each?" L4 for speed/non-HTTP; L7 for content routing, TLS, and canary/blue-green.
"Is the LB a SPOF?" It mustn't be — managed redundant LB or an HA pair.
"Why avoid sticky sessions?" They undermine statelessness and even distribution; externalise state instead.

Say it like this"A load balancer distributes traffic and enables horizontal scaling. L4 balances on IP and port — fast and protocol-agnostic — while L7 understands HTTP, so it can route by path or header, terminate TLS, and enable canary and blue-green deploys, which is why most web systems use L7. Health checks are what make it work — unhealthy instances are pulled out — and I tune them so a slow-starting instance isn't killed prematurely. The LB itself must not be a single point of failure, so I use a managed redundant cloud load balancer. And I avoid sticky sessions where I can, because they undermine the statelessness that lets the app tier scale."

Mark:

6Caching strategiesDesign▸

The system is read-heavy and you need to reduce latency and database load — or the interviewer probes cache patterns and invalidation.

What is actually happening (the mental model).

Caching trades staleness for speed and load reduction by keeping hot data closer to the reader. You reason about where to cache (client → CDN → app/in-memory → distributed cache like Redis → DB buffer), the pattern (cache-aside is the default), and the hard parts: invalidation ("there are only two hard things...") and stampedes (scenario overlaps with war stories). The 80/20 rule means a small cache of the hot data yields most of the benefit.

How to work through it.

Cache layers — CDN for static/edge, distributed cache (Redis/Memcached) for hot data, in-process for tiny hot sets, DB buffer pool.
Cache-aside (lazy loading) as the default — app checks cache, on miss reads DB and populates. Understand write-through / write-back trade-offs.
Invalidation — TTLs for simplicity; explicit invalidation on write for freshness; accept the staleness window.
Eviction — LRU/LFU when the cache is full.
Stampede protection — single-flight / jittered TTLs / stale-while-revalidate so a hot key expiring doesn't flatten the origin.

Key trade-offs and decisions.

Caching = faster + less DB load, at the cost of staleness and complexity (invalidation, consistency). Cache-aside is simple but has a miss penalty; write-through keeps cache fresh but adds write latency. TTL vs explicit invalidation is a freshness-vs-simplicity trade.

The trap that less-experienced engineers fall into.

Ignoring invalidation (serving stale data indefinitely) and stampede protection (a hot-key expiry flattens the DB — thundering herd), and caching everything instead of the hot 20%.

🎯 Interviewer follow-up questions you should expect.

"How do you invalidate the cache?" TTL for simplicity, explicit invalidation on write for freshness; accept the staleness window.
"A hot key expires under load and the DB gets hammered — why and fix?" Thundering herd; single-flight, jittered TTLs, and stale-while-revalidate.
"Cache-aside vs write-through?" Cache-aside simple with a miss penalty; write-through fresh but adds write latency.

Say it like this"Caching trades staleness for speed and load reduction by keeping hot data closer to the reader. I reason about the layers — CDN at the edge, a distributed cache like Redis for hot data, in-process for tiny hot sets — and default to cache-aside, where the app reads the cache and on a miss populates it from the DB. The hard parts are invalidation and stampedes: I use TTLs for simplicity or explicit invalidation on write for freshness, and I protect against thundering herd with single-flight and jittered TTLs so a hot key expiring under load doesn't flatten the database. And I cache the hot 20% that serves most reads, not everything."

Mark:

7Database scaling (replication, sharding, SQL vs NoSQL)Design▸

The database is the bottleneck; you need to scale reads, writes, and storage — and choose SQL vs NoSQL.

What is actually happening (the mental model).

The database is usually the real scaling bottleneck, and you scale its three dimensions differently: reads with read replicas + caching; writes/storage with sharding (partitioning data across nodes). SQL gives strong consistency and flexible queries but is harder to scale writes; NoSQL scales horizontally for known access patterns but trades flexible querying/consistency. The choice is driven by access patterns and consistency needs, not fashion.

How to work through it.

Scale reads — read replicas (async → replica lag, eventual consistency for reads) + caching. Handles read-heavy systems.
Scale writes/storage — sharding/partitioning by a good shard key (even distribution, avoids hot partitions). This is the hard part — cross-shard queries and rebalancing are painful.
SQL vs NoSQL — SQL for transactions/flexible queries/strong consistency; NoSQL (DynamoDB/Cassandra) for massive scale with known access patterns and horizontal writes.
Denormalisation for read performance at scale (trade storage/consistency for speed).

Key trade-offs and decisions.

Read replicas: scale reads but add replica lag (eventual consistency). Sharding: scales writes/storage but adds huge complexity (cross-shard joins, rebalancing, hot partitions). SQL: consistency + flexibility, harder write scaling. NoSQL: scale + performance, but access patterns must be modelled up front.

The trap that less-experienced engineers fall into.

Picking a bad shard key (creating hot partitions), forgetting read replicas are async (stale reads), and choosing NoSQL "for scale" without modelling access patterns.

🎯 Interviewer follow-up questions you should expect.

"How do you scale writes?" Sharding by a good shard key (even distribution, no hot partitions); it's the hard part.
"Read replicas — what's the catch?" Async replication → replica lag → eventual consistency for reads.
"SQL or NoSQL and why?" Driven by access patterns and consistency needs; NoSQL needs access patterns modelled up front.

Say it like this"The database is usually the real bottleneck, and I scale its dimensions differently. Reads I scale with read replicas plus caching — remembering replicas are async, so there's replica lag and reads become eventually consistent. Writes and storage I scale with sharding, partitioning data by a shard key chosen for even distribution to avoid hot partitions — that's the hard part, because cross-shard queries and rebalancing are painful. On SQL versus NoSQL, I choose from access patterns and consistency needs: SQL for transactions and flexible queries, NoSQL like DynamoDB for massive scale with known access patterns — but NoSQL only works if you model those patterns up front."

Mark:

8Async & message queues / event-driven designDesign▸

The system has spiky load, slow operations, or needs to decouple services — and the interviewer probes queues, delivery semantics, and backpressure.

What is actually happening (the mental model).

A message queue decouples producers from consumers — the producer drops work and moves on, consumers process at their own pace. This buys you spike absorption (the queue buffers bursts), resilience (work survives a consumer outage), and independent scaling. The trade-offs are eventual consistency (async), delivery semantics (usually at-least-once → need idempotency), and backpressure (what happens when the queue grows unboundedly).

How to work through it.

Decouple with a queue — for slow work (send email, process video), spiky load (buffer bursts), and fan-out (one event, many consumers).
Choose the tool — SQS (simple, managed), Kafka (high-throughput, ordered, replayable log, stream processing), depending on throughput/ordering/replay needs.
Delivery semantics — most are at-least-once, so consumers must be idempotent (scenario 12); exactly-once is expensive/rare.
Backpressure & DLQ — bound the queue, dead-letter queue for poison messages, monitor queue depth (and scale consumers on it).

Key trade-offs and decisions.

Async decouples and absorbs spikes but adds eventual consistency and complexity (ordering, delivery semantics, debugging distributed flows). SQS: simple. Kafka: powerful (ordering, replay, high throughput) but operationally heavier.

The trap that less-experienced engineers fall into.

Non-idempotent consumers under at-least-once delivery (duplicate processing), no dead-letter queue (a poison message blocks the queue), and unbounded queues with no backpressure (they grow forever).

🎯 Interviewer follow-up questions you should expect.

"Why introduce a queue?" Decouple, absorb spikes, resilience, and fan-out; at the cost of eventual consistency.
"SQS vs Kafka?" SQS simple/managed; Kafka for high-throughput, ordering, replay, and stream processing.
"Delivery semantics?" Usually at-least-once → consumers must be idempotent; exactly-once is expensive.

Say it like this"A queue decouples producers from consumers — the producer drops work and moves on, consumers process at their own pace. That buys me spike absorption because the queue buffers bursts, resilience because work survives a consumer outage, and independent scaling. I pick the tool from the needs: SQS when I want simple and managed, Kafka when I need high throughput, ordering, or a replayable log for stream processing. The key gotcha is delivery semantics — most systems are at-least-once, so my consumers must be idempotent to tolerate duplicates. And I bound the queue with backpressure, a dead-letter queue for poison messages, and I scale consumers on queue depth."

Mark:

9API gateway & API designDesign▸

The system exposes APIs (external or between services) and needs a front door — the interviewer probes gateways, rate limiting, versioning, and REST vs gRPC.

What is actually happening (the mental model).

An API gateway is the single entry point that handles cross-cutting concerns — authentication, rate limiting, TLS termination, routing, request aggregation — so individual services don't each reimplement them. Good API design is about stable, versioned, backward-compatible contracts (so clients don't break), appropriate protocols (REST for public/simple, gRPC for high-performance internal), and protecting the backend (rate limiting, pagination).

How to work through it.

API gateway for cross-cutting concerns — auth, rate limiting, TLS, routing, aggregation — centralised instead of per-service.
Versioning & backward compatibility — version the API (URL/header), never break existing clients; deprecate with notice.
Protocol — REST/JSON for public and simple; gRPC for high-performance internal service-to-service (binary, streaming, contracts).
Protect the backend — rate limiting (scenario 13), pagination, request size limits, timeouts.

Key trade-offs and decisions.

A gateway centralises concerns but can become a bottleneck/SPOF (make it scalable + HA). REST: universal, simple, verbose. gRPC: fast, typed, streaming, but less browser-friendly. Versioning adds maintenance but protects clients.

The trap that less-experienced engineers fall into.

Reimplementing auth/rate-limiting in every service (instead of a gateway), breaking API compatibility without versioning, and making the gateway a SPOF.

🎯 Interviewer follow-up questions you should expect.

"What does an API gateway do?" Centralises cross-cutting concerns (auth, rate limiting, TLS, routing) so services don't each reimplement them.
"REST vs gRPC?" REST for public/simple; gRPC for high-performance internal service-to-service.
"How do you version APIs?" Versioned, backward-compatible contracts; deprecate with notice so clients don't break.

Say it like this"An API gateway is the single front door that handles cross-cutting concerns — authentication, rate limiting, TLS termination, routing, request aggregation — so each service doesn't reimplement them. For the APIs themselves, I care most about stable, versioned, backward-compatible contracts so I never break existing clients, and I deprecate with notice. Protocol-wise, REST and JSON for public and simple cases, gRPC for high-performance internal service-to-service where I want binary encoding and streaming. And I protect the backend with rate limiting, pagination, and timeouts. The one caution is that the gateway mustn't become a bottleneck or SPOF, so it's scalable and HA."

Mark:

10Designing for failure (timeouts, retries, circuit breakers, bulkheads)Design▸

"How does this system handle a dependency failing?" — the resilience question, and where DevOps candidates shine.

What is actually happening (the mental model).

In a distributed system, failure is the normal case, not the exception — dependencies will be slow or down. So you design assuming they'll fail: timeouts on every call (a slow dependency is worse than a dead one — it exhausts your resources), retries with backoff + jitter (but bounded, or you cause a retry storm), circuit breakers (stop hammering a failing dependency, fail fast), bulkheads (isolate resources so one failing dependency can't consume everything), and graceful degradation (partial function over total outage).

How to work through it.

Timeouts everywhere — a slow dependency blocks your threads and cascades (scenario overlaps with war stories); never call without a timeout.
Retries with exponential backoff + jitter, bounded — retries help transient failures but naive retries cause storms.
Circuit breakers — after N failures, "open" the circuit and fail fast, giving the dependency time to recover.
Bulkheads — separate thread pools/resources per dependency so one slow one can't exhaust all capacity.
Graceful degradation — serve stale/partial data when a dependency is down rather than failing entirely.

Key trade-offs and decisions.

Resilience patterns add complexity but prevent cascading failures. Retries help transient errors but risk amplifying load — bounded + jittered. Circuit breakers add fail-fast at the cost of some false opens. This is the DevOps differentiator — pure-app candidates often miss it.

The trap that less-experienced engineers fall into.

No timeouts (a slow dependency cascades and takes you down), naive retries (retry storm), and no isolation (one failing dependency exhausts the whole thread pool — cascading failure).

🎯 Interviewer follow-up questions you should expect.

"A dependency slows down — what protects you?" Timeouts (a slow dependency is worse than a dead one), circuit breakers, and bulkheads.
"Why is a slow dependency worse than a dead one?" A dead one fails fast; a slow one exhausts your threads/resources.
"What's a circuit breaker?" After repeated failures it opens and fails fast, letting the dependency recover.

Say it like this"In a distributed system, failure is the normal case — dependencies will be slow or down — so I design assuming it. Every call has a timeout, because a slow dependency is actually worse than a dead one: it blocks your threads and cascades. Retries use exponential backoff with jitter and are bounded, or naive retries cause a storm that turns a slowdown into an outage. Circuit breakers stop hammering a failing dependency and fail fast so it can recover. Bulkheads isolate resources per dependency so one slow one can't exhaust all my capacity. And graceful degradation means serving stale or partial data rather than failing entirely. This resilience layer is where I add a lot of value beyond a pure application design."

Mark:

11Consistency & CAP in practiceDesign▸

"Strong or eventual consistency here?" — or a question probing CAP theorem in a real design.

What is actually happening (the mental model).

CAP says under a network partition you must choose consistency or availability — you can't have both. In practice, partitions will happen, so it's really a C-vs-A choice for distributed data, and most large-scale systems choose availability + eventual consistency because being down is worse than being briefly stale. But it's per-use-case: money/inventory need strong consistency; a like-count or a feed can be eventually consistent. You match the consistency model to what the data actually requires.

How to work through it.

CAP in practice — partitions happen, so choose C or A. AP (available, eventually consistent) for most; CP (consistent, may reject requests) where correctness is critical.
Per-use-case — strong consistency for money/inventory/uniqueness; eventual for counts, feeds, analytics.
Eventual consistency techniques — read replicas, async replication, conflict resolution (last-write-wins, CRDTs, version vectors).
Strong consistency techniques — quorum reads/writes, distributed transactions (costly), a single-writer/leader.

Key trade-offs and decisions.

Strong consistency: correct but lower availability/higher latency (coordination). Eventual: highly available and fast but temporarily stale/conflicting. The art is applying each where appropriate — not one model everywhere.

The trap that less-experienced engineers fall into.

Applying strong consistency everywhere (killing availability/latency), or eventual everywhere (corrupting money/inventory), instead of matching the model to each use case. And misstating CAP as "pick 2 of 3" always (it's about the partition case).

🎯 Interviewer follow-up questions you should expect.

"Strong or eventual here?" Per use case: strong for money/inventory, eventual for counts/feeds.
"What does CAP really say?" Under a partition you choose consistency or availability; partitions happen, so it's C vs A.
"How do you get strong consistency at scale?" Quorums, a single leader, or distributed transactions — at a latency/availability cost.

Say it like this"CAP says under a network partition you have to choose consistency or availability, and since partitions do happen, it's really a C-versus-A choice for distributed data. Most large systems choose availability with eventual consistency, because being briefly stale beats being down. But it's per-use-case, not one size fits all: money, inventory, and uniqueness need strong consistency, while a like-count or a feed can be eventually consistent. So I match the model to what the data actually requires — quorums or a single leader where correctness is critical, async replication and conflict resolution where availability matters more. Applying the right model per use case is the skill."

Mark:

12Idempotency & delivery semanticsDesign▸

The system processes payments, messages, or events and must not double-process — the interviewer probes exactly-once vs at-least-once.

What is actually happening (the mental model).

In distributed systems, exactly-once delivery is effectively impossible end-to-end (networks retry, clients time out and re-send). So the practical pattern is at-least-once delivery + idempotent processing = effectively exactly-once. Idempotency means processing the same request twice has the same effect as once — achieved with an idempotency key and a dedupe/process-once check. This is essential wherever duplicates would cause harm (double charge, double order).

How to work through it.

Accept at-least-once — networks and retries mean duplicates happen; don't rely on exactly-once delivery.
Make processing idempotent — an idempotency key (client-supplied or derived), stored, so a repeat is detected and returns the prior result instead of re-doing the action.
Where it matters — payments (idempotency key on the charge), order creation, message consumers, webhooks.
Dedup window / storage — how long you remember processed keys (trade storage for the dedup guarantee).

Key trade-offs and decisions.

Idempotency adds a storage/lookup cost per request but is the only reliable way to get correctness under retries. At-least-once + idempotent is simpler and more reliable than chasing true exactly-once.

The trap that less-experienced engineers fall into.

Assuming exactly-once delivery exists (building on a false guarantee), and non-idempotent handlers that double-process on a retry (double charge).

🎯 Interviewer follow-up questions you should expect.

"How do you guarantee a payment isn't charged twice?" At-least-once delivery, idempotent processing with an idempotency key; not "exactly-once delivery."
"Is exactly-once delivery possible?" Not reliably end-to-end; achieve effectively-once via at-least-once, idempotency.
"How does an idempotency key work?" Store it; a repeat request with the same key returns the prior result rather than re-processing.

Say it like this"Exactly-once delivery is effectively impossible in a distributed system — networks retry, clients time out and resend — so the practical pattern is at-least-once delivery plus idempotent processing, which gives you effectively exactly-once. Idempotency means processing the same request twice has the same effect as once, which I implement with an idempotency key that I store, so a repeat is detected and returns the prior result instead of, say, charging the card again. It's essential anywhere a duplicate would cause harm — payments, order creation, webhooks. So rather than chase a delivery guarantee that doesn't exist, I make the processing safe to repeat."

Mark:

13A full worked example — design a URL shortener end-to-endDesign▸

The canonical warm-up ("design TinyURL/bit.ly") — deceptively simple, and a great vehicle to show the whole method: requirements → estimate → design → deep-dive → scale.

What is actually happening (the mental model).

It looks trivial but tests everything: read-heavy scaling (100:1+ reads), unique ID generation at scale (the crux), caching, and key-value storage. You walk the method and go deep on the interesting part — how to generate short, unique, collision-free codes without a bottleneck.

How to work through it.

Requirements — shorten a URL → short code; redirect code → original. NFRs: read-heavy (redirects ≫ creates), low-latency redirects, high availability, codes never collide.
Estimate — e.g. 100M new URLs/day, 100:1 read:write → massive redirect QPS → caching + read scaling are essential; storage = URLs × size × retention.
Core design — API + service behind an LB; a key-value store (URL DB) mapping code → URL; a cache (Redis) for hot redirects; a CDN optionally.
The crux — ID generation: options: (a) hash + encode (MD5/base62, handle collisions); (b) a counter / ID-allocation service (base62-encode a globally unique auto-increment — needs a distributed counter, e.g. a range-allocation service or Snowflake-style IDs to avoid a single-counter bottleneck); (c) pre-generated key pool. Discuss trade-offs (collision handling vs coordination).
Scale — cache the hot redirects (80/20), read replicas, shard the KV store by code; the redirect path is a cache hit → very fast.

Key trade-offs and decisions.

Hash-based: simple, stateless, but collisions to handle. Counter-based: no collisions but needs distributed coordination (single counter is a bottleneck → range allocation / Snowflake IDs). Redirect is 301 (cached, less analytics) vs 302 (not cached, full analytics) — a real trade-off.

The trap that less-experienced engineers fall into.

A single auto-increment counter (a write bottleneck/SPOF), ignoring collision handling in hash approaches, and forgetting the read-heavy nature (no caching).

🎯 Interviewer follow-up questions you should expect.

"How do you generate unique short codes at scale?" Hash+base62 with collision handling, or a distributed ID allocator (range allocation / Snowflake) to avoid a single-counter bottleneck.
"301 vs 302 redirect?" 301 is cached (fast, less analytics), 302 isn't (full analytics per hit).
"How do you make redirects fast?" Cache the hot codes (80/20), so the redirect path is a cache hit.

Say it like this"It looks trivial but it tests the whole method. Requirements: shorten and redirect, and it's extremely read-heavy — redirects vastly outnumber creates — so caching and read scaling are essential. The core is a key-value store mapping short code to URL, a Redis cache for hot redirects, all behind a load balancer. The interesting part is ID generation: I can hash the URL and base62-encode it, handling collisions, or use a distributed ID allocator — but a single auto-increment counter is a bottleneck and SPOF, so I'd use range allocation or Snowflake-style IDs to generate unique codes without coordination on every write. To scale reads I cache the hot 20% and shard the KV store by code, so the redirect path is a fast cache hit. I'd also note 301 versus 302 — 301 caches at the client for speed, 302 gives full per-hit analytics."

Mark:

14Design a scalable CI/CD platformDesign▸

"Design a CI/CD system for a large org" — a DevOps-native design prompt where your domain knowledge shines.

What is actually happening (the mental model).

A CI/CD platform is a distributed job-execution system with strict requirements: isolation (one build can't poison another — ephemeral runners), scalability (autoscale runners to demand), security (untrusted code, secrets, supply chain), speed (caching, parallelism), and reliability (the platform itself is production — if it's down, nobody ships). You design the control plane (orchestration, queue), the data plane (ephemeral runners), artifact storage, and the secrets/security model.

How to work through it.

Control plane — receives triggers (webhooks), queues jobs, orchestrates pipelines, tracks state. Must be HA (scenario: CI as a SPOF).
Data plane — ephemeral, autoscaling runners — spun up per job (Kubernetes/VMs), destroyed after, so jobs are isolated and can't poison each other; autoscale on queue depth.
Artifact & cache storage — object storage for artifacts (build-once-promote-many), a shared cache (deps, layers) for speed.
Security — OIDC for cloud creds (no static keys), scoped secrets, supply-chain scanning, never run untrusted PR code on privileged runners.
Speed — caching, parallelism/sharding, affected-only builds (monorepo).

Key trade-offs and decisions.

Ephemeral runners: secure/clean but slower startup (mitigate with warm pools). Self-hosted vs managed runners (control/cost vs ops). The platform is production-severity — HA and a break-glass path.

The trap that less-experienced engineers fall into.

Reused (non-ephemeral) runners (one job poisons the next), treating the platform as "just tooling" (no HA), and static cloud keys instead of OIDC.

🎯 Interviewer follow-up questions you should expect.

"How do you isolate builds?" Ephemeral, single-use runners destroyed after each job; autoscale on queue depth.
"How do you secure it?" OIDC for cloud creds, scoped secrets, supply-chain scanning, and never run untrusted code on privileged runners.
"CI is down — what's the impact and mitigation?" Nobody ships; treat as production (HA, a break-glass manual deploy path).

Say it like this"A CI/CD platform is really a distributed job-execution system with strict requirements. The control plane receives triggers, queues jobs, and orchestrates pipelines, and it has to be HA because if it's down nobody ships — it's production. The data plane is ephemeral, autoscaling runners, spun up per job and destroyed after, so builds are isolated and one can't poison the next, and I scale them on queue depth. I add object storage for artifacts to enable build-once-promote-many, a shared cache for speed, and a security model built on OIDC so there are no static cloud keys, scoped secrets, and supply-chain scanning — and I never run untrusted PR code on privileged runners. Speed comes from caching, parallelism, and affected-only builds in a monorepo."

Mark:

15Design a multi-region active-active systemDesign▸

"Design a system that survives a full region failure with low global latency" — the advanced availability prompt.

What is actually happening (the mental model).

Multi-region active-active (both regions serving traffic) gives the best availability (survive a region loss) and lowest latency (serve users from the nearest region) — but the hard problem is data: keeping data consistent across regions is where the complexity and trade-offs live. You route users (GeoDNS/anycast), replicate data, and confront cross-region consistency (async replication → eventual consistency, or the latency cost of synchronous). Most go active-active for stateless tiers + carefully-designed data replication.

How to work through it.

Global routing — GeoDNS / anycast / a global load balancer routes users to the nearest healthy region; failover on region outage.
Stateless tiers active-active — app tier runs in both regions, easy.
The hard part — data: async cross-region replication (eventual consistency, conflict resolution) is common; synchronous is consistent but adds cross-region latency (~50–150ms/write). Options: single-writer-per-region with partitioning, active-active with conflict resolution (CRDTs/LWW), or a globally-distributed DB (Spanner/DynamoDB global tables).
Handle conflicts — concurrent writes in two regions; resolution strategy needed.
DR framing — active-active is your DR (RTO≈0) but is the most expensive pattern.

Key trade-offs and decisions.

Active-active: best availability + latency, but data consistency is hard and it's expensive (double infra, replication, complexity). Active-passive is simpler/cheaper but has failover time and idle capacity. Sync replication: consistent but slow cross-region; async: fast but eventually consistent with conflicts.

The trap that less-experienced engineers fall into.

Hand-waving the data layer (the actual hard part), assuming you can just "replicate the DB" without confronting cross-region consistency/conflicts, and proposing active-active when active-passive meets the RTO at a fraction of the cost.

🎯 Interviewer follow-up questions you should expect.

"What's the hard part of multi-region?" Data consistency across regions; the stateless tier is easy.
"Sync vs async cross-region replication?" Sync is consistent but adds cross-region latency per write; async is fast but eventually consistent with conflicts.
"Do you need active-active?" Only if RTO≈0 and global low latency are required; active-passive is cheaper if the RTO allows.

Say it like this"Active-active — both regions serving — gives the best availability, surviving a full region loss, and the lowest latency by serving users from the nearest region. Global routing with GeoDNS or anycast sends users to the closest healthy region. The stateless tiers are easy to run in both. The hard problem is data: synchronous cross-region replication is consistent but adds 50-to-150 milliseconds per write, so most systems use async replication and accept eventual consistency with a conflict-resolution strategy, or use a globally-distributed database like Spanner or DynamoDB global tables. Active-active is effectively zero-RTO DR but it's the most expensive pattern, so I'd only choose it if the business genuinely needs that availability and global latency — otherwise active-passive is far cheaper."

Mark:

16Design a scalable observability / logging pipelineDesign▸

"Design a system to collect, store, and query logs/metrics/traces for a large fleet" — a DevOps-native design prompt.

What is actually happening (the mental model).

An observability pipeline is a high-volume data ingestion and query system with a brutal cost/cardinality problem. You design collection (agents/OTel), ingestion (a buffer/queue to absorb spikes), processing (parse, sample, enrich), storage (tiered, by signal type), and query — while controlling cardinality (the cost killer) and volume (sampling). The three signals have different storage needs: metrics (time-series DB), logs (indexed store), traces (sampled).

How to work through it.

Collection — agents/OpenTelemetry on each host/pod; decouple instrumentation from backend (OTel).
Ingestion buffer — a queue (Kafka) absorbs spikes so a traffic surge doesn't drop telemetry or overwhelm storage.
Processing — parse/enrich, and sample (tail-based for traces — keep errors/slow, drop the boring majority).
Storage, tiered by signal — metrics in a TSDB (Prometheus/Mimir), logs in an indexed store (Loki/ELK) with hot/warm/cold tiers, traces sampled (Tempo).
Control cardinality & cost — cap label cardinality (the #1 cost driver), retention tiers, sampling; correlate via trace IDs.

Key trade-offs and decisions.

Fidelity vs cost — you can't store everything at full resolution affordably, so sampling and tiering are essential. Push vs pull collection. Managed (Datadog) vs self-hosted (Prometheus/Grafana/Loki) — cost vs ops. Cardinality is the make-or-break constraint.

The trap that less-experienced engineers fall into.

Ignoring cardinality (cost explosion), no ingestion buffer (telemetry dropped under load), and storing all traces at full fidelity (unaffordable — needs tail-based sampling).

🎯 Interviewer follow-up questions you should expect.

"What's the main cost/scale problem?" Cardinality (unbounded labels) and volume; control with cardinality caps, sampling, tiering.
"How do you not drop telemetry under a traffic spike?" An ingestion buffer (Kafka) absorbs the burst.
"How do you store the three signals?" TSDB for metrics, indexed tiered store for logs, and sampled store for traces; correlate by trace ID.

Say it like this"An observability pipeline is a high-volume ingestion and query system with a brutal cardinality and cost problem. Collection is agents or OpenTelemetry on each host, decoupling instrumentation from the backend. I put a buffer like Kafka at ingestion so a traffic spike doesn't drop telemetry or overwhelm storage, then a processing stage that parses, enriches, and samples — tail-based sampling for traces so I keep the errors and slow requests and drop the boring majority. Storage is tiered by signal: a time-series DB for metrics, an indexed store with hot-warm-cold tiers for logs, sampled storage for traces, all correlated by trace ID. The make-or-break constraint is controlling cardinality, because an unbounded label explodes cost — that's the thing I'd design around most carefully."

Mark:

17Design an internal developer platform (IDP) / container platformDesign▸

"Design a platform that lets 500 developers deploy services safely and quickly" — the platform-engineering prompt, increasingly common at senior/staff level.

What is actually happening (the mental model).

An IDP's goal is a paved road / golden path — make the right way (secure, observable, reliable deploys) the easy way, so developers self-serve without deep infra knowledge and without each team reinventing CI/CD, monitoring, and security. You provide self-service (a developer portal / templates), abstraction (hide K8s complexity behind a simple interface), guardrails (policy-as-code, secure defaults baked in), and golden paths for the common cases — while keeping an escape hatch for the unusual.

How to work through it.

Golden paths / templates — a new service scaffolds with CI/CD, observability, security, and deploy config built in (cookiecutter/Backstage templates).
Self-service — a developer portal (Backstage) or CLI/GitOps so devs deploy without filing tickets or knowing Kubernetes internals.
Abstraction — hide the platform complexity (K8s, networking) behind a simple contract (e.g. "here's my container + resource needs").
Guardrails baked in — policy-as-code (OPA/Kyverno), secure defaults, cost controls — the paved road is compliant.
Escape hatch — power users can drop to the lower level for the unusual case; don't force everything through the abstraction.

Key trade-offs and decisions.

Abstraction vs flexibility — a good IDP hides complexity for the 80% common case but must not trap the 20% who need more (leaky-abstraction risk). Build vs buy (Backstage vs commercial). The platform is a product — adoption and DX are the success metrics.

The trap that less-experienced engineers fall into.

Building a rigid abstraction with no escape hatch (power users route around it), and treating it as infra rather than a product (poor DX → nobody adopts it).

🎯 Interviewer follow-up questions you should expect.

"What's the goal of an IDP?" A golden path — make the secure/observable/reliable way the easy way, self-service, so devs don't reinvent or need deep infra knowledge.
"How do you balance abstraction vs flexibility?" Paved road for the common 80%, an escape hatch for the 20% who need more.
"What's the success metric?" Adoption and developer experience — it's a product with users.

Say it like this"An internal developer platform is about a golden path — making the right way, secure, observable, reliable deploys, the easy way, so 500 developers can self-serve without each team reinventing CI/CD, monitoring, and security or needing to know Kubernetes internals. I provide templates that scaffold a new service with all of that built in, a self-service portal like Backstage or a GitOps flow so they deploy without filing tickets, and an abstraction that hides the platform complexity behind a simple contract. The guardrails — policy-as-code, secure defaults, cost controls — are baked into the paved road, so the easy path is the compliant path. But I always keep an escape hatch for power users, because a rigid abstraction with no way out gets routed around. And I treat the platform as a product — adoption and developer experience are the real success metrics."

Mark:

18Multi-tenancy architectureDesign▸

"Design a system serving many customers/tenants on shared infrastructure" — the isolation-vs-efficiency prompt.

What is actually happening (the mental model).

Multi-tenancy trades resource efficiency (shared infra is cheaper) against isolation (tenants shouldn't affect each other's security, performance, or data). The core decision is the isolation model on a spectrum: shared everything (one DB, tenant_id column — cheapest, weakest isolation) → shared app, isolated data (schema/DB per tenant) → fully isolated (separate infra/account per tenant — strongest, most expensive). You choose based on the tenant trust level, compliance needs, and the noisy-neighbour risk.

How to work through it.

Isolation spectrum — pooled (shared DB + tenant_id), bridge (schema/DB per tenant), silo (dedicated infra per tenant). Cheaper ↔ more isolated.
Noisy-neighbour control — per-tenant quotas/rate limits so one tenant can't starve others (the #1 real multi-tenancy failure).
Data isolation — enforce tenant scoping rigorously (a missing tenant_id filter = a cross-tenant data leak); consider row-level security.
Security — a tenant must never access another's data; defense in depth.
Tiering — offer silo isolation for premium/enterprise tenants, pooled for the rest.

Key trade-offs and decisions.

Pooled: cheap, efficient, but weak isolation and a leak is catastrophic (bad tenant filter). Silo: strong isolation and compliance-friendly, but expensive and operationally heavy (per-tenant infra). Often a hybrid: pooled for small tenants, silo for enterprise.

The trap that less-experienced engineers fall into.

Weak data isolation (a forgotten tenant filter → cross-tenant leak, a severe breach), no per-tenant quotas (noisy neighbour starves everyone), and forcing one isolation model for all tenants.

🎯 Interviewer follow-up questions you should expect.

"What are the isolation models?" Pooled (shared DB, tenant_id), bridge (schema/DB per tenant), silo (dedicated infra) — efficiency vs isolation.
"Biggest multi-tenancy risk?" A cross-tenant data leak from a missing tenant filter, and noisy neighbours without quotas.
"How do you serve both small and enterprise tenants?" Hybrid — pooled for small, silo isolation for enterprise/compliance.

Say it like this"Multi-tenancy trades resource efficiency against isolation. The core decision is the isolation model on a spectrum: pooled, where tenants share a database with a tenant_id, is cheapest but weakest; then schema-or-database-per-tenant; then fully siloed, dedicated infra per tenant, which is strongest but most expensive. I choose based on tenant trust and compliance. The two things I'm most careful about are data isolation — a forgotten tenant filter is a catastrophic cross-tenant leak, so I enforce scoping rigorously, maybe with row-level security — and noisy neighbours, so per-tenant quotas and rate limits stop one tenant starving the rest, which is the most common real failure. Often the answer is hybrid: pooled for small tenants, siloed isolation for enterprise ones."

Mark:

19Disaster recovery & zero-downtime deployment architectureDesign▸

"How do you design for disaster recovery and deploy without downtime?" — two operability concerns DevOps candidates own.

What is actually happening (the mental model).

DR starts from the business RTO/RPO (how fast back, how much data loss tolerable) and maps to a pattern (backup/restore → pilot light → warm standby → active-active) — cheapest that meets the requirement, with tested and isolated backups. Zero-downtime deployment is an architectural property: it requires stateless app tiers, backward-compatible changes (API + schema via expand-contract), health/readiness gating, and a strategy (rolling/blue-green/canary) so old and new coexist and traffic only hits ready instances.

How to work through it.

DR from RTO/RPO — pick the cheapest pattern that meets them; test restores (an untested backup is a hope); isolate backups (separate account/region) so the event can't destroy them; run game days.
Zero-downtime = an architectural property — stateless app tier, backward-compatible API + schema (expand-contract), readiness gating so traffic only hits ready pods, graceful shutdown (SIGTERM drain).
Deploy strategy — rolling (default), blue/green (instant rollback), canary (lowest blast radius) — matched to risk.
Database migrations — expand-contract so old and new code coexist and rollback is possible (the hardest part).

Key trade-offs and decisions.

DR: more resilience = more cost (active-active vs backup/restore); match to RTO/RPO. Zero-downtime: requires design discipline (backward compatibility, statelessness) — you can't bolt it on. Blue/green = instant rollback + double resources; canary = safe + needs observability.

The trap that less-experienced engineers fall into.

Untested/co-located backups (fail when needed), coupling breaking schema changes to deploys (can't roll back), and assuming rolling updates are zero-downtime without readiness gating and backward compatibility.

🎯 Interviewer follow-up questions you should expect.

"How do you pick a DR strategy?" From RTO/RPO — cheapest pattern that meets them; test restores; isolate backups.
"What makes zero-downtime deploys possible?" Stateless tiers, backward-compatible changes (expand-contract), readiness gating, and graceful shutdown.
"The hardest part of zero-downtime?" Database migrations — expand-contract so old and new coexist and you can roll back.

Say it like this"DR starts from the business RTO and RPO — how fast we must be back and how much data loss is tolerable — and I pick the cheapest pattern that meets them, from backup-restore up to active-active. Non-negotiables: test the restores because an untested backup is a hope, and isolate backups in a separate account or region so the same event or a compromised account can't destroy them. Zero-downtime deployment is an architectural property, not a switch — it needs a stateless app tier, backward-compatible API and schema changes via expand-contract so old and new versions coexist, readiness gating so traffic only reaches ready instances, and graceful shutdown so in-flight requests drain. The strategy — rolling, blue/green, or canary — I match to the risk, and the hardest part is always the database migration, which expand-contract makes reversible."

Mark:

20Designing for operability & observability from day oneDesign▸

Often the closing question ("anything else?") or a probe on what you'd add — and it's your single biggest differentiator from pure-SWE candidates.

What is actually happening (the mental model).

Most candidates design the happy path and stop. A senior DevOps engineer bakes in how the system is operated from the start: observability (metrics/logs/traces, SLOs, alerting on symptoms), deployability (zero-downtime, rollback, IaC), debuggability (can you diagnose it at 2am?), security (least privilege, encryption, secrets), and cost (what does it cost, where's the waste). A system you can't observe, deploy safely, or afford isn't done — operability is a first-class design concern, not an afterthought.

How to work through it.

Observability built in — the golden signals, SLOs/error budgets, distributed tracing across services, alerting on user-facing symptoms not causes.
Deployability — zero-downtime deploys, easy rollback, everything as code (IaC), feature flags.
Debuggability — correlation IDs, structured logs, runbooks; can on-call diagnose it fast?
Security — least privilege, encryption at rest/in transit, secrets management, network segmentation.
Cost — a rough cost model, identify waste (data transfer, over-provisioning), tagging for attribution.

Key trade-offs and decisions.

Operability adds upfront design effort but is what makes a system runnable in production — the difference between a demo and a system that survives a year of 2am pages. It's the lens pure-app designs miss.

The trap that less-experienced engineers fall into.

Designing only the functional happy path — no thought to how it's monitored, deployed, secured, or what it costs — so it looks complete but is unrunnable in production.

🎯 Interviewer follow-up questions you should expect.

"How would you operate/monitor this?" Golden signals, SLOs, tracing, symptom-based alerting; deployability and rollback; runbooks.
"What does this cost and where's the waste?" A rough cost model; data-transfer and over-provisioning are the usual culprits.
"How do you secure it?" Least privilege, encryption, secrets management, and network segmentation.

Say it like this"This is where I add the most value beyond a pure application design. Most designs cover the happy path; I bake in how the system is operated from day one. Observability — the golden signals, SLOs and error budgets, distributed tracing across services, and alerting on user-facing symptoms not internal causes. Deployability — zero-downtime deploys, easy rollback, everything as code. Debuggability — correlation IDs and runbooks so on-call can diagnose it fast at 2am. Security — least privilege, encryption, secrets management. And cost — a rough cost model and where the waste is, usually data transfer or over-provisioning. A system you can't observe, deploy safely, or afford isn't finished — operability is a first-class design concern for me, not an afterthought." --- ## How to turn this into muscle memory - Drill the universal reflex first (clarify requirements → estimate → high-level → deep-dive → bottlenecks → trade-offs) until narrating it is automatic — it's the skeleton for every question. - Practise on a whiteboard, out loud, timed. Take a prompt (URL shortener, notification system, CI/CD platform) and talk through the whole arc in ~30 minutes. Record yourself once. - Always bring the DevOps lens — after the functional design, proactively cover operability, reliability, security, and cost. That's your differentiator; make it a habit, not an afterthought. - Learn the building blocks cold (LB, cache, DB scaling, queues, resilience patterns) so you can assemble them fluidly — every design is a combination of these. - Have real trade-offs ready — for every choice, know the alternative and why you chose it. "I'd use X, though Y would give us Z at the cost of W" is the senior signal. Next (of the three you asked for): 18 — DevSecOps, at this same depth.

Mark:

🧠 How to turn this into muscle memory

No scenarios match that search. Clear search

System Design — DevOps Zero → Hero. Playbook-only chapter — the 20 scenario drills are worked through in full depth; there is no course video behind this domain. ← All chapters