A service's dashboard panels have gone flat — not showing an outage, just showing nothing new, and nobody noticed for two days.

What is actually happening (the mental model).

Prometheus only has data for targets it can successfully scrape — if a target becomes unreachable (a pod restarted with a new IP that Prometheus's service discovery hasn't picked up, a NetworkPolicy blocking it, a port change), Prometheus doesn't error loudly; the metric just stops updating, and a dashboard panel with no fresh data can look deceptively like "flat and stable" rather than "broken."

How to work through it.

1. Check Status → Targets in the Prometheus UI — every target shows its last scrape time and error, if any; a target stuck in the past or marked down is the whole answer.
2. If the target is missing entirely (not even listed as down), the problem is service discovery, not the scrape itself — check the relevant labels/annotations the Prometheus config relies on to discover it.
3. Confirm the target's /metrics endpoint actually still exists and responds, independent of Prometheus, with a direct curl from inside the cluster.

The root causes and trade-offs.

One cause is the pod's IP changing while service discovery either lagged behind or was misconfigured to begin with. Another is a NetworkPolicy introduced after the scrape configuration was already set up, quietly blocking it. A third is the application changing its metrics port without anyone updating the scrape job to match. All three are silent by nature — nothing crashes, the data just quietly stops arriving.

The trap that less-experienced engineers fall into.

Reading "the graph looks flat" as "the system is stable," instead of checking whether the flat line means healthy or means no new data arrived at all — the two look identical on most dashboards.

🎯 Interviewer follow-up questions you should expect.

• "How do you know if a target has actually stopped reporting, vs. genuinely being flat?" You check Status → Targets for the last-scrape time and any errors, not just the dashboard graph.
• "How would you alert on this happening again?" You'd add an alert on the up metric being 0, or on scrape staleness, for critical targets.

A PromQL query that worked last month now returns nothing, and a dashboard panel built on it is empty.

What is actually happening (the mental model).

Metrics in Prometheus are identified by name and labels — a metric "disappearing" almost always means either the exporter/library version changed and renamed it, or a label value changed so your exact query no longer matches anything, even though a very similar series still exists under a slightly different name or label set.

How to work through it.

1. Search for the metric family by partial name in the Prometheus UI's metric browser — renamed metrics are usually a close variant of the old name.
2. Check the exporter's or client library's changelog around the time it disappeared — a version bump is a very common trigger for metric renames.
3. If the name is unchanged, check whether a label value the query filters on (a version tag, an environment label) changed, silently breaking the exact match.

The root causes and trade-offs.

One cause is an exporter upgrade that renamed or restructured its metrics. Another is a label's value changing, for example a deploy that relabels environment or version differently than before. A third is the exporter itself being removed or replaced entirely. Pinning exporter versions trades away newer metrics in exchange for query stability, and that's a real trade-off worth making deliberately rather than discovering by accident.

The trap that less-experienced engineers fall into.

Assuming the underlying system stopped producing the data entirely, and investigating the application, when the data is still there under a slightly different name or label — a broader, unfiltered search would have found it immediately.

🎯 Interviewer follow-up questions you should expect.

• "A query that used to work now returns nothing — first thing you check?" You search for the metric family more broadly before assuming the data is gone.
• "How do you prevent this from breaking dashboards silently?" You pin exporter and library versions deliberately and review changelogs before upgrading anything metrics-related.

A nightly batch job runs for 90 seconds and finishes — its metrics never appear in Prometheus at all, even though the job logs say it emitted them.

What is actually happening (the mental model).

Prometheus's pull model assumes a target stays up long enough to be scraped on the next interval — a job that starts, runs, and exits inside 90 seconds may finish and disappear entirely between one scrape cycle and the next, so Prometheus never gets the chance to pull anything from it.

How to work through it.

1. Recognise this as a mismatch between the pull model and short-lived workloads, not a configuration bug in the usual sense.
2. Use the Pushgateway — a component built exactly for this — where the short-lived job pushes its final metrics before exiting, and Prometheus then scrapes the Pushgateway itself (which holds the value until explicitly cleared) instead of the job directly.
3. Confirm the job is instrumented to push (not just expose /metrics, which is useless if nothing scrapes it in time) and that stale values are cleared appropriately so old runs don't linger and mislead.

The root causes and trade-offs.

The fundamental cause is that pull-based scraping simply doesn't fit workloads that run for less time than the scrape interval itself. The Pushgateway fixes the visibility problem, but it adds a component with its own trade-off to manage: because it just keeps serving the last pushed value, it can itself mask a job that has stopped running entirely, so that trade-off needs active management rather than being ignored.

The trap that less-experienced engineers fall into.

Trying to fix this by shortening the scrape interval — even a very short interval can still race a genuinely fast job, and it doesn't address the real mismatch between pull-based scraping and ephemeral workloads.

🎯 Interviewer follow-up questions you should expect.

• "Why don't short batch jobs show up in Prometheus?" They can finish and disappear between scrape cycles, so pull never gets a chance to reach them.
• "How do you monitor them properly?" You use the Pushgateway — the job pushes before exiting, and Prometheus scrapes the gateway instead.

The Prometheus server pod itself gets OOMKilled or its disk fills up, and monitoring for the whole cluster goes down with it.

What is actually happening (the mental model).

Prometheus keeps recent time-series data in memory before it's compacted to disk, and its resource footprint scales directly with how many distinct time series it's tracking (its cardinality) and how long it retains data — this isn't a fixed cost, it grows with every new label combination and every additional day of retention you configure.

How to work through it.

1. Check prometheus_tsdb_head_series (or similar internal metrics Prometheus exposes about itself) to see the actual current series count and whether it's been growing.
2. Check the retention window (--storage.tsdb.retention.time) and disk size against how much data that retention actually requires — a retention bump without a matching disk/memory bump is a common cause.
3. Look for a recent cardinality spike (see the next scenario) as the trigger, rather than assuming it's simply "grown over time" from ordinary usage.
4. Fix with more resources for genuine growth, or by reducing retention/cardinality if the growth is a symptom of something that shouldn't be happening.

The root causes and trade-offs.

This can come from real, expected growth as more services get onboarded, from a cardinality explosion like the one in the next scenario, or from a retention setting that was increased without a matching increase in resources. More retention and more granular labels are both genuinely useful, but the trade-off is that both cost real memory and disk, and that cost needs to be planned for in advance rather than discovered through an outage.

The trap that less-experienced engineers fall into.

Treating Prometheus's own resource limits as something to just keep raising, without ever checking why usage is growing — sometimes it's healthy growth, but sometimes it's a single bad label choice multiplying series count unnecessarily.

🎯 Interviewer follow-up questions you should expect.

• "Prometheus keeps running out of memory — how do you investigate?" You check the current series count and its trend, then correlate it with retention settings and any recent cardinality spike.
• "How does retention affect resource usage?" More retention means more data held on disk, and touched in memory during compaction, so it must scale with actual capacity, not be bumped in isolation.

Prometheus's memory usage suddenly spikes hard after what looked like a routine application change.

What is actually happening (the mental model).

Every unique combination of a metric name plus its label values is a separate time series — a label with a genuinely unbounded set of values (a user ID, a raw URL path with IDs in it, a request ID) multiplies the number of series by however many distinct values that label ever takes, which for something like a user ID can mean millions of new series from a single bad instrumentation choice.

How to work through it.

1. Identify the metric responsible via Prometheus's own prometheus_tsdb_head_series breakdown, or a "top series by metric name" query if your setup supports it.
2. Inspect that metric's label set for anything with unbounded cardinality — user IDs, full URL paths, request/session IDs are the classic offenders.
3. Fix at the source: remove or bucket the offending label in the application's instrumentation (a URL template like /users/:id instead of the literal path with the real ID in it) — Prometheus-side mitigation (dropping the label via relabeling) is a stopgap, not the real fix.

The root causes and trade-offs.

The root cause is instrumenting a metric with a label that was convenient to add but turns out to have effectively unlimited values. The underlying trade-off is between very granular labels, which are genuinely useful for debugging one specific request, and the very real cost that granularity has at scale. The fix is almost always to bucket or template the label, not to eliminate the metric entirely.

The trap that less-experienced engineers fall into.

Reflexively giving Prometheus more memory/disk to absorb the spike instead of finding and fixing the specific label causing it — the underlying problem keeps growing unbounded, so more resources only delays the same outage.

🎯 Interviewer follow-up questions you should expect.

• "What causes a cardinality explosion?" A label with effectively unbounded values — a user ID, a raw URL path, a request ID — multiplies the number of distinct time series.
• "How do you fix it properly, not just patch it?" You remove or bucket the offending label at the instrumentation source, not just add more Prometheus resources.

Someone asks for a year-over-year comparison of error rates, but Prometheus only retains 15 days of data.

What is actually happening (the mental model).

Prometheus's local TSDB is deliberately optimised for recent data at high resolution, not indefinite long-term storage — its retention is a knob you set, and raising it substantially costs real disk and query performance on the single server, which is why long-term/historical needs are usually solved by a separate remote-storage layer, not by cranking local retention arbitrarily high.

How to work through it.

1. Distinguish the actual need: is this a one-off report, or an ongoing requirement for long-range queries?
2. For genuine long-term retention, look at Prometheus's remote_write integration to a long-term storage backend (Thanos, Cortex, Mimir, or a managed equivalent) that's designed for exactly this instead of the local TSDB.
3. Understand these systems typically downsample older data (keep full resolution recently, coarser resolution further back), which is a reasonable trade-off for year-scale queries that don't need per-second granularity anyway.

The root causes and trade-offs.

Sometimes local retention is genuinely set too low for a real, ongoing long-term need, and the fix there is remote long-term storage. Other times it's really just a one-off report that doesn't justify standing up new infrastructure, and the right response is simply to note the gap for future retention planning rather than over-engineering a solution for a single ask. Downsampling trades away exact historical precision in exchange for a query that stays fast at long time ranges.

The trap that less-experienced engineers fall into.

Simply cranking up local retention to a very long window without recognising the disk and query-performance cost that comes with keeping full-resolution data indefinitely on a single Prometheus server — long-term storage is a different problem with different tools, not just a bigger number in a config flag.

🎯 Interviewer follow-up questions you should expect.

• "How do you handle year-scale historical queries when Prometheus only keeps 15 days?" You use remote_write into a long-term storage system like Thanos, Cortex, or Mimir, rather than just raising local retention.
• "What's the trade-off with long-term storage systems?" They typically downsample older data, keeping full resolution recently and coarser resolution further back, trading precision for sustainable query performance.

The on-call Slack channel is so full of alerts that people have started muting it — and a genuine incident got missed in the noise.

What is actually happening (the mental model).

Every alert that fires and doesn't require action trains the on-call engineer to ignore alerts — this is a real, well-documented failure mode, not a discipline problem with the humans involved; the fix is treating alert quality as a metric to actively manage, not something that accumulates passively as thresholds get added over time.

How to work through it.

1. Audit existing alerts for ones that fire often but rarely require action — these are the highest-value candidates to delete, raise the threshold on, or convert into a dashboard-only signal instead of a page.
2. Distinguish symptom-based alerts (user-facing impact — error rate, latency) from cause-based alerts (an internal metric that might contribute to a symptom) — page on symptoms, and let cause-based signals live on dashboards for diagnosis after a symptom alert has already fired.
3. Use Alertmanager's grouping and inhibition features so one root cause doesn't fan out into a dozen separate pages for every downstream symptom.
4. Track a "percentage of alerts that led to action" metric over time, so alert quality is visibly improving or degrading, not just assumed.

The root causes and trade-offs.

This usually comes from alerting on every internal metric just in case, instead of on genuine user-facing symptoms, combined with not using Alertmanager's grouping and inhibition features, so that one incident turns into many separate pages. It's made worse by nobody ever going back to remove an alert once it's been added. Tightening alerts does carry a real, if small, risk of missing something genuinely rare, but that's worth accepting against the much larger cost of training the whole team into alert-blindness.

The trap that less-experienced engineers fall into.

Responding to alert fatigue by adding more alerting rules to "catch things better," which makes the noise worse — the fix is almost always deleting or consolidating alerts, not adding new ones.

🎯 Interviewer follow-up questions you should expect.

• "How do you fix alert fatigue?" You audit for low-value alerts and remove or consolidate them, favour symptom-based paging over cause-based, and use Alertmanager's grouping and inhibition features.
• "Symptom-based vs cause-based alerting — what's the difference?" Symptom-based alerting pages on user-facing impact, like error rate or latency; cause-based alerting flags internal metrics that might contribute — you page on symptoms and diagnose with causes.

After an outage, the postmortem finds there was a metric that clearly showed the problem building — but no alert ever fired.

What is actually happening (the mental model).

"No alert fired" can mean the system was healthy, or it can mean nobody wrote a rule for this failure mode, or the rule existed but had a bug (wrong threshold, wrong PromQL, wrong duration window) — these are three very different findings with three very different fixes, and a postmortem needs to identify which one actually happened.

How to work through it.

1. Check whether an alerting rule for this condition existed at all — if not, this is a coverage gap, not a bug.
2. If a rule existed, re-run its exact PromQL against the historical data from the incident window to see whether it actually would have fired — sometimes the logic is subtly wrong (an "or" that should be "and," a threshold set from stale assumptions).
3. Check the rule's for duration — a rule requiring a condition to hold for 10 minutes won't fire for a spike that resolves in 3, which might be correct behaviour or might be exactly the gap.
4. Add the missing rule (or fix the buggy one) and, ideally, replay it against the historical data to confirm it would now catch the same incident.

The root causes and trade-offs.

Sometimes no rule was ever written for this specific failure mode at all, which is a genuine blind spot. Other times a rule did exist but had incorrect logic or thresholds, a latent bug that stays invisible until it's actually tested against a real incident. And sometimes the rule's duration window was a legitimate noise-reduction trade-off that simply happened to miss this specific case.

The trap that less-experienced engineers fall into.

Treating "no alert fired" as proof the monitoring is broken in general, and reflexively adding a pile of new alerts, rather than precisely diagnosing whether this was a missing rule, a buggy rule, or a legitimate trade-off that just happened to bite this once.

🎯 Interviewer follow-up questions you should expect.

• "How do you investigate a missed alert after an incident?" You check whether a rule existed at all, then replay its exact logic against historical data to see if it should have fired.
• "What's a common reason a real alert rule doesn't fire in time?" A common reason is a for duration window tuned for noise reduction that happens to be longer than the incident took to resolve.

The same alert fires, resolves, fires again, resolves again — dozens of times over an hour, each one paging on-call.

What is actually happening (the mental model).

Flapping means the underlying metric is oscillating right around the alert's threshold — every small crossing re-triggers the alert, which is a config/tuning problem (the threshold or duration doesn't match how the metric actually behaves), not necessarily a sign the underlying issue itself is getting better or worse each time.

How to work through it.

1. Graph the raw metric over the flapping window — is it hovering right at the threshold, genuinely unstable, or is the threshold simply set at a noisy point in the metric's normal range?
2. Add or lengthen the alert's for duration, so a brief crossing doesn't immediately fire — this smooths out noise without hiding a sustained problem.
3. Use Alertmanager's grouping so repeated fire/resolve cycles for the same alert don't each generate a separate page.
4. If the metric is inherently noisy, consider alerting on a smoothed/averaged version of it (a rolling average over PromQL) rather than the raw instantaneous value.

The root causes and trade-offs.

The root cause is usually a threshold set exactly where the metric naturally hovers, combined with no for duration in place to smooth over brief crossings, and often alerting on the raw noisy value instead of a rolling average. Adding a duration window and smoothing trades a little detection latency for a lot less noise, and that's almost always worth it for anything that isn't safety-critical enough to need an instant response.

The trap that less-experienced engineers fall into.

Reflexively silencing or disabling a flapping alert to stop the noise, without fixing the underlying threshold/duration tuning — which either hides a real recurring problem or just defers the same noise to reappear later.

🎯 Interviewer follow-up questions you should expect.

• "An alert keeps flapping — what do you do?" You check whether the metric is hovering right at the threshold, then fix it with a longer for duration or a smoothed metric, rather than silencing it.
• "How does Alertmanager help with repeated fire/resolve cycles?" Its grouping feature consolidates related notifications so the same flapping alert doesn't generate a fresh page every cycle.

An alert fires at 3am, pages a generic on-call rotation, and the responder has no idea what team or system it's even about.

What is actually happening (the mental model).

An alert that doesn't clearly identify which team owns it, what it means, and what to do transfers the cognitive load of figuring that out onto the least-prepared person — someone freshly woken up at 3am — instead of the person who wrote the alert and had full context at the time.

How to work through it (the standard for a good alert).

1. Every alerting rule should carry labels routing it to the right team/service (via Alertmanager routing rules matched on labels) so it pages the people who can actually act, not a generic catch-all rotation.
2. Every alert should include a clear annotation with what the alert means and, ideally, a link to a runbook — the response steps written calmly in advance, not improvised at 3am.
3. If an alert genuinely has no clear owner, that's itself the finding — either assign one, or seriously reconsider whether the alert should exist as a page at all versus a dashboard signal.

The root causes and trade-offs.

This usually comes from alerts written without ownership labels or a runbook link, often because they were added quickly during or right after an incident and never revisited afterward, combined with a shared or generic on-call rotation covering systems that no single person fully understands. Runbooks take real upfront effort to write and maintain, but the trade-off is against a 3am responder having to improvise blind every single time.

The trap that less-experienced engineers fall into.

Adding new alerting rules during an incident, under pressure, without also adding the ownership/routing labels and a runbook reference — solving today's visibility gap while creating tomorrow's "nobody knows what this alert means" problem.

🎯 Interviewer follow-up questions you should expect.

• "What makes a good alert, beyond the threshold itself?" Clear ownership and routing labels, plus an annotation with a runbook link, so the responder doesn't have to reconstruct context at 3am.
• "How do you route alerts to the right team?" You use Alertmanager routing rules matched on the labels the alert carries, not a single generic rotation for everything.

Users are reporting the app is down, but the team's main Grafana dashboard shows every panel green.

What is actually happening (the mental model).

A dashboard is only as honest as the metrics feeding it — it can show green because the system genuinely is healthy, or because the metric it's built on doesn't actually capture the failure mode happening right now (a health check that only tests one shallow path, an average that hides a severely affected minority of requests), which is a dashboard-design gap, not a monitoring-system malfunction.

How to work through it.

1. Confirm the underlying data is actually fresh (see the "silent scrape failure" scenario) before assuming the metric itself is the problem — check it isn't simply stale data rendering as a static "healthy" state.
2. If the data is fresh and still shows green, question what the metric actually measures — an average latency can look fine while a real minority of requests time out; a shallow health check can pass while a deeper dependency is failing.
3. Cross-check against a metric closer to the actual user experience — error rate on real traffic, a synthetic end-to-end check — rather than trusting one dashboard's framing of "healthy."
4. After resolving, add the metric or percentile (like p99 latency instead of average) that would have actually shown this outage, so the dashboard's blind spot is closed.

The root causes and trade-offs.

One cause is stale data rendering as a false-positive "healthy" state. Another is an average hiding a real problem that's only affecting a minority of traffic. A third is a shallow health check that never actually exercises the failing dependency. Averages and shallow checks are cheap and simple to build, but the trade-off is that they can miss exactly the kind of partial, tail-latency incident that matters most to users.

The trap that less-experienced engineers fall into.

Trusting the dashboard over the user reports, and spending time trying to explain why users must be wrong — instead of immediately suspecting the dashboard's own metrics might not capture this specific failure mode.

🎯 Interviewer follow-up questions you should expect.

• "Dashboard is green but users report an outage — what do you trust?" You trust the user reports, then go find out what the dashboard's metric actually measures and where its blind spot is.
• "Why can an average metric hide a real incident?" It can look fine while a real subset of requests, or one specific path, is badly affected, as long as the majority of traffic is unaffected.

A PromQL query you're confident should return data comes back completely empty in the Prometheus UI.

What is actually happening (the mental model).

An empty PromQL result almost always means the query's label selector doesn't match any existing series exactly — Prometheus label matching is exact-string by default, so a subtly wrong label value, a typo, or a metric name that doesn't exist at all produces silently empty results rather than an error.

How to work through it.

1. Strip the query down to just the bare metric name with no label filters — if that alone returns nothing, the metric doesn't exist (or isn't being scraped, see earlier scenarios); if it returns data, the problem is in your label filter.
2. Once the bare metric returns data, inspect the actual label values Prometheus has for it and compare them character-for-character against what your filter expects — case, exact spelling, and value format (a version string like "1.0" vs "1.0.0") all matter.
3. Check the query's time range — a valid query can return nothing if the time window predates when the metric started being collected, or a rate/increase function's window is shorter than the scrape interval.

The root causes and trade-offs.

Sometimes the metric genuinely doesn't exist or isn't being scraped at all. Other times a label filter has a typo or the wrong exact value. And sometimes the time range or the rate window is simply shorter than the data that's actually available. All three are query-construction issues, not Prometheus itself malfunctioning.

The trap that less-experienced engineers fall into.

Assuming a complex query is "broken" and rewriting it repeatedly, rather than methodically stripping it back to the bare metric name first to isolate whether the issue is the metric's existence or just one label filter.

🎯 Interviewer follow-up questions you should expect.

• "A query returns no data — how do you debug it?" You strip it down to the bare metric name first, then add filters back one at a time to isolate which one excludes everything.
• "Why does PromQL label matching fail silently instead of erroring?" Because it's exact-string matching by design — a non-matching filter is treated as a legitimate, if empty, result, not an error.

Two Grafana dashboards, both supposedly showing "error rate" for the same service, show visibly different numbers at the same moment.

What is actually happening (the mental model).

"Error rate" is not one unambiguous query — two dashboards can both be technically correct while measuring subtly different things: different time windows for the rate calculation, different label filters (one includes a specific route the other excludes), or one query using rate() while the other uses increase() divided by a different denominator.

How to work through it.

1. Open both dashboard panels' underlying PromQL (Grafana lets you inspect and edit the query behind any panel) side by side rather than trusting the panel titles.
2. Compare the exact time window each uses in its rate/increase function — a 1-minute window and a 5-minute window smooth differently and will legitimately show different instantaneous values.
3. Compare label filters precisely — one dashboard might be scoped to a specific route, namespace, or version that the other isn't.
4. Once you've found the actual difference, decide which definition is the "official" one for that name, and align both dashboards to it (or rename one so the difference is explicit rather than implicit).

The root causes and trade-offs.

Sometimes the two panels use different rate or increase time windows. Sometimes they use different label scoping. And sometimes they're built on an entirely different underlying metric — an application-level error counter versus an ingress-level 5xx counter, say — despite both being labelled "error rate." None of these is a bug on its own; the real problem is that two things with genuinely different meanings are sharing a name that implies they're the same.

The trap that less-experienced engineers fall into.

Assuming one dashboard must simply be "wrong" and trying to fix its data source or plumbing, instead of comparing the actual PromQL behind both panels, where the real difference almost always lives.

🎯 Interviewer follow-up questions you should expect.

• "Two dashboards show different numbers for 'the same metric' — how do you resolve it?" You compare the actual PromQL behind both panels — time windows and label scope are the usual culprits, not a data pipeline bug.
• "How do you prevent this ambiguity long-term?" You standardise a single definition, ideally through a shared, reusable query or library panel, for commonly named metrics like "error rate" across dashboards.

A community Grafana dashboard is imported but half its panels show "No data," even though Prometheus and Grafana are both clearly working.

What is actually happening (the mental model).

Community Kubernetes dashboards are built assuming kube-state-metrics is deployed and scraped — if it's missing entirely, or an old version whose metric names have since changed, every panel relying on object-state metrics (replica counts, pod phase, node conditions) comes back empty, while panels built on raw node/API metrics work fine, which is a useful clue about exactly what's missing.

How to work through it.

1. Check whether kube-state-metrics is even deployed and being scraped (Status → Targets again) — the single most common cause of "half the panels are empty" on an imported Kubernetes dashboard.
2. If it's deployed but panels are still empty, check its version against what the dashboard expects — a version mismatch can mean the exact metric names the dashboard queries no longer exist, similar to the "disappeared metric" scenario but scoped specifically to this component.
3. Reconcile by either installing/upgrading kube-state-metrics or, if pinned for a reason, importing a dashboard version compatible with the installed version instead.

The root causes and trade-offs.

Sometimes kube-state-metrics was simply never installed as part of the monitoring setup. Other times it was installed, but at a version whose metrics don't match what an imported community dashboard is actually expecting. This is essentially the earlier "metric disappeared" and "silent scrape failure" scenarios showing up again, in the specific and very common context of a fresh Kubernetes monitoring setup.

The trap that less-experienced engineers fall into.

Assuming the whole monitoring stack is broken because "half the dashboard is empty," instead of noticing that the working panels and the empty panels cleanly split along exactly which metric source (node/API vs. kube-state-metrics) they depend on.

🎯 Interviewer follow-up questions you should expect.

• "Half a Kubernetes dashboard shows No data — what's the first thing you check?" You check whether kube-state-metrics is deployed and being scraped at all.
• "The dashboard was working before an upgrade and now some panels break — why?" kube-state-metrics's own version may have changed the metric names the dashboard still queries under the old names.

A counter metric that had been steadily climbing suddenly drops back to zero after a routine pod restart, making a rate-of-change graph spike wildly.

What is actually happening (the mental model).

A Prometheus counter only ever increases while a process is alive — it lives in that process's memory and resets to zero whenever the process restarts, which is expected, documented behaviour, not data loss; the reason it doesn't ruin dashboards in practice is that PromQL's rate()/increase() functions are specifically designed to detect a counter reset and handle it correctly.

How to work through it.

1. Confirm the metric type is genuinely a counter (as opposed to a gauge, which isn't supposed to only-increase) — the reset-on-restart behaviour is correct for counters and wrong-looking for gauges.
2. Make sure any query or panel graphing this metric uses rate() or increase() rather than the raw counter value directly — graphing the raw value will show the reset as a visible cliff, while rate() correctly treats it as "the counter restarted, don't count this interval as a huge negative rate."
3. If a panel is still showing an artificial spike even through rate(), check the query's time window relative to how frequently pods restart — a very short window right around a restart can still show a brief artefact.

The root causes and trade-offs.

The actual bug is almost always a dashboard graphing the raw counter value instead of wrapping it in rate() or increase(). That leads people to confuse a counter that's genuinely just reset on restart with a real anomaly. This is a dashboard-construction issue, not something wrong with Prometheus or the application itself.

The trap that less-experienced engineers fall into.

Treating a counter reset after a restart as a sign of data loss or a monitoring bug worth investigating deeply, when it's expected behaviour that a correctly-written rate() query already accounts for — the actual bug, if any, is a panel that graphs the raw counter instead.

🎯 Interviewer follow-up questions you should expect.

• "A counter resets to zero after a pod restart — is that a bug?" No, that's expected counter behaviour — if a graph looks wrong, the fix is using rate() or increase() instead of the raw value.
• "What's the difference between a counter and a gauge in this context?" A counter only ever increases, until a restart resets it, and is meant to be queried with rate or increase; a gauge can go up or down and is graphed directly.

The company now runs five Kubernetes clusters, and leadership wants one place to see the health of all of them, not five separate Grafanas.

What is actually happening (the mental model).

A single Prometheus server is normally scoped to one cluster it can reach directly — a true multi-cluster view requires either a federated/central layer that aggregates from each cluster's own Prometheus, or a long-term storage backend that already supports multi-source ingestion (Thanos/Cortex/Mimir again), rather than trying to point one Prometheus at five clusters' worth of targets directly.

How to reason about it.

1. Keep a Prometheus per cluster for local scraping (so a network blip to one cluster doesn't blind you to the others), and use a central layer on top for the cross-cluster view — this mirrors the earlier long-term-storage answer, since both problems are solved by the same class of tool.
2. Add a cluster label to every metric at the source (Prometheus's external_labels) so once data is aggregated centrally, queries can still filter or group by cluster.
3. For genuine multi-tenancy (different teams who shouldn't see each other's data), look specifically at a system with tenant isolation (Mimir, Cortex) rather than a single shared Grafana with no access boundaries.

The root causes and trade-offs.

Trying to make one single Prometheus reach directly across every cluster is fragile, and it defeats the whole benefit of local, resilient scraping that per-cluster Prometheus gives you. The alternative, a Prometheus per cluster plus a central aggregation or long-term-storage layer on top, has more moving parts, but it's genuinely resilient and scalable. Multi-tenancy on top of that adds real access-control complexity that a single shared dashboard never had to solve.

The trap that less-experienced engineers fall into.

Trying to solve a multi-cluster view by pointing a single central Prometheus directly at every cluster's targets, which reintroduces a single point of failure and network dependency that per-cluster Prometheus was specifically avoiding.

🎯 Interviewer follow-up questions you should expect.

• "How do you monitor five Kubernetes clusters from one place?" You run a Prometheus per cluster for resilient local scraping, plus a central federation or long-term-storage layer, like Thanos, Cortex, or Mimir, for the aggregated view.
• "How do you keep cluster data distinguishable once it's aggregated?" You use external_labels to tag every metric with its cluster at the source.

Every alert and dashboard panel that exists today can be traced back to a specific past incident — nothing was set up proactively.

What is actually happening (the mental model).

Reactive-only monitoring means your coverage is a map of past failures, not a map of the system's actual risk surface — it's real evidence that monitoring has been treated as an incident-response afterthought rather than a design input alongside the system architecture itself, and it guarantees blind spots for any failure mode that hasn't happened yet.

How to work through it (shifting toward proactive).

1. For any new service, define its key signals before launch — request rate, error rate, latency (the classic "RED" trio for request-driven services, or "USE" — utilisation, saturation, errors — for resources) — rather than waiting to see what breaks first.
2. Use SLOs (service-level objectives) as a forcing function: deciding "99.9% of requests should succeed" up front makes you instrument and alert on the thing that actually matters to users, before an incident forces the question.
3. Periodically review coverage against a checklist of common failure modes for the service's type (a database dependency, an external API call, a queue) instead of only after something breaks.
4. Treat "we only just added this alert after an incident" as a signal worth tracking over time — a ratio of proactive-to-reactive additions is itself a useful health metric for the monitoring practice.

The root causes and trade-offs.

This usually comes from organisational pressure to focus only on what's visibly urgent, like fixing the last incident, rather than investing time in what's merely likely to happen next. It's made worse by genuinely not knowing what "good" monitoring looks like for a new service type, without a framework like RED, USE, or SLOs to structure the thinking. Proactive monitoring takes real upfront time that competes directly with feature work, and that's a trade-off worth making deliberately, not by default neglect.

The trap that less-experienced engineers fall into.

Treating "we have lots of alerts" as evidence of mature monitoring, without noticing that all of them are reactive patches for specific past incidents rather than proactive coverage of the system's actual risk surface.

🎯 Interviewer follow-up questions you should expect.

• "How do you tell if monitoring is proactive or just reactive patchwork?" You check whether each alert or panel traces back to a past incident, versus being derived from a framework like RED, USE, or an SLO defined up front.
• "What do you set up for a brand-new service, before anything has broken?" You set up RED metrics — rate, errors, duration — for request-driven services, or USE for resources, tied to an SLO that reflects what users actually need.

"Set up monitoring for this" — a new service is about to ship, and it currently has zero metrics, zero dashboards, and zero alerts.

What is actually happening (the mental model).

Starting from a blank slate is exactly where a framework earns its keep — rather than guessing which metrics matter, a small number of proven patterns (RED, USE, the four golden signals) tell you almost the entire starting checklist, and everything past that baseline is specific to what this particular service actually does.

The checklist (each item tied to a concrete signal).

1. RED for the service itself — Rate (requests per second), Errors (error rate/count), Duration (latency, ideally as a histogram so you can pull p50/p95/p99, not just an average that hides tail latency).
2. USE for anything it depends on as a resource — Utilisation, Saturation, Errors — for its host/pod (CPU, memory) and for any queue or connection pool it uses.
3. Dependency health — if it calls a database or another service, track that call's latency and error rate separately from the service's own, so you can tell "we're slow" from "someone we depend on is slow."
4. An SLO — a stated target (like "99.9% of requests succeed in under 300ms") that turns the RED metrics into a clear pass/fail an alert can be built on, instead of an arbitrary threshold picked with no reference point.
5. Alerts on symptoms, dashboards for causes — page on the SLO being at risk (a symptom), and keep the RED/USE/dependency breakdowns as dashboards for diagnosing why, once a symptom alert has already fired.
6. Business/custom metrics — whatever this specific service uniquely needs to prove it's doing its job (a queue depth, a job success count, a conversion counter) that no generic framework would know to include.

The trap that less-experienced engineers fall into.

Either shipping with zero monitoring "to add later" (which almost never happens until the first incident forces it), or over-instrumenting everything imaginable from day one without an SLO to anchor which of those metrics actually deserves an alert versus just existing quietly on a dashboard.

🎯 Interviewer follow-up questions you should expect.

• "A new service ships tomorrow with no monitoring — what's the minimum you'd add?" You'd add RED for the service, USE for its resources, dependency-call tracking, and an SLO to anchor the first alert.
• "Why prefer a latency histogram (p95/p99) over an average?" Because an average can look fine while a real minority of requests have much worse tail latency, which is exactly what users actually notice.