HomeSource & deliveryTerraform
Day 16 · 17 — provisioning infrastructure with code, on any cloud

Terraform & Infrastructure as Code

Every company runs its applications on servers, and those servers have to be created somewhere — on AWS, on Azure, on Google Cloud, or in a data centre of their own. Creating them by hand does not scale, and writing a different automation tool for every cloud is its own trap. This chapter rebuilds Days 16 and 17 into a workshop: first you understand the problem infrastructure as code solves and how Terraform's "API as code" idea sidesteps cloud lock-in, then you get hands-on installing Terraform and standing up real infrastructure, then you get quizzed the way an interviewer would, and finally you face the senior scenarios where state, drift, and blast radius decide whether a change is safe.

① Theory② Lab③ Interview Q&A④ Scenario drills
Day 16 · Infrastructure as Code — the concept and the problem Day 17 · Everything about Terraform — first project, remote backend, modules, interview Q&A
⚡ The whole of Terraform in one mental model

Terraform lets you describe the infrastructure you want in code, and it creates that infrastructure on any cloud for you. You write configuration files that say what you want — an EC2 instance, an S3 bucket, a database — and you tell Terraform which provider you are targeting. Terraform then translates your files into the API calls that provider understands and makes them for you. That last part is the clever bit, and it has a name: API as code. Because Terraform speaks every cloud's API for you, you learn one tool instead of a different one per cloud. And to remember what it built, Terraform keeps a state file — the single most important thing in all of Terraform.

Your config (.tf)what you want, in HCL Terraformconfig → provider API"API as code" AWS API Azure API Google Cloud API … any other provider state filewhat Terraform built
One tool, every cloud, and a memory. You write config once; Terraform turns it into each provider's API calls (API as code) and records what it created in the state file so it can change or destroy it later.

Four ideas carry the chapter: (1) the problem is per-cloud lock-in — a different automation tool for every provider, rewritten every time you move; (2) Terraform solves it with API as code, so you learn one tool and it talks to every cloud's API for you; (3) the everyday workflow is a short lifecycle — write → init → plan → apply (and sometimes destroy); and (4) the state file is the heart of Terraform, which is exactly why protecting it — remote, locked, never hand-edited — is where all the senior thinking lives.

01
Phase 1 · Theory

What infrastructure as code is, and why Terraform

First the problem — a different automation tool for every cloud — then Terraform's answer, the "API as code" idea, and finally what you gain from it and the short lifecycle you work in.

T1

The problem — a different tool for every cloud

Infrastructure as code is easiest to understand through the pain it removes, so follow one DevOps engineer's story across three migrations.

Imagine you are a DevOps engineer at a large company — call it Flipkart — with three hundred applications to run. Each application needs servers, and those servers, the compute resources, can be created in many places: AWS, Azure, Google Cloud, Oracle Cloud, DigitalOcean, or your own physical machines on-premises. You evaluate the options and decide to host everything on AWS. Being a good engineer, you do not click around the console by hand; you automate the whole thing with AWS CloudFormation Templates (CFT). You write hundreds of scripts, they stabilise, and life is good — a developer asks for ten EC2 instances and you just run the relevant CFT script.

Then management decides to move from AWS to Azure — perhaps the support or the cost was not right. Now all those hundreds of CloudFormation scripts are worthless, because CFT only works on AWS. So you rewrite everything in Azure Resource Manager, Azure's equivalent, and automate the whole estate again. Later the company moves again, this time to on-premises with OpenStack — so you rewrite everything a third time, now in OpenStack's Heat templates. Every migration means learning a new tool and rewriting all your automation from scratch.

Your automationhundreds of scripts AWS → CloudFormation Azure → Resource Manager OpenStack → Heat rewrite it all,every migration
The migration tax. Each cloud has its own infrastructure-as-code tool, and each one only works on that cloud — so moving provider means throwing away and rewriting all your automation.

It gets worse in the modern world of hybrid cloud, where a company deliberately runs part of its infrastructure on one provider and part on another — storage on AWS because it is strong there, DevOps tooling on Azure because it is strong there. Now you must know both CloudFormation and Azure Resource Manager at the same time. Note that all of these tools — CFT, ARM, Heat — are "infrastructure as code": you are creating infrastructure with code rather than by hand, which is a real improvement over clicking in a console. The problem is not the idea; it is that each tool is welded to a single provider, so the multi-cloud world forces you to learn and maintain many of them.

T2

The solution — Terraform and "API as code"

Terraform, built by HashiCorp, was created to answer exactly this problem: learn one tool, target any cloud.

Terraform's pitch to DevOps engineers is simple: do not learn a hundred tools — learn one. You write Terraform configuration files, you tell Terraform which provider you are targeting (in a file such as provider.tf you might say the provider is AWS), and Terraform takes care of automating the resources on that cloud. If the company later moves from AWS to Azure, the migration is smooth: you change the provider details and make some minimal adjustments, rather than rewriting everything in a brand-new language. One tool covers every cloud, present and future.

How does one tool talk to every cloud? Through the concept that makes Terraform special: API as code. To understand it you first need to know what an API is. When you want something from Google, you normally open a browser and type google.com — that is a user interface, a manual way to interact. But if you want to do it programmatically, from a script, that manual flow is no use. So applications expose an API (an application programming interface): a programmatic way to talk to them. Instead of logging in and clicking, your script sends an HTTP request — a curl call, say — to the application's API and gets a result back. GitHub, Google, AWS, Azure — they all expose APIs so that programs, not just people, can drive them.

You write .tf"create an EC2 instance" Terraformconverts config → API call Provider APIAWS / Azure / GCP …and the result comes back to you no API code — you write config
API as code. You never write the API calls yourself. You write plain configuration, and Terraform — which has learned every provider's API — converts your request into the right calls, runs them, and hands you the result.

Terraform uses exactly this. AWS has an API, Azure has an API, and rather than making you write those API calls in Python or shell, Terraform's maintainers (HashiCorp and open-source contributors) have already studied each provider's API and written modules for it. So to create an EC2 instance you do not make an AWS API call — you write a few lines of Terraform config, almost like English, and Terraform receives your request, converts it into the AWS API call behind the scenes, runs it, and returns the result. You are never talking to the cloud's API directly; you write config, and Terraform does the API work. That is "API as code," and it is why one tool can automate any provider — including clouds that do not exist yet, once someone writes the module.

T3

What Terraform gives you, and its lifecycle

Before touching the tool, it helps to know both what you get from it and the short cycle of commands you will live in.

Whenever someone asks why move to Terraform, these are the advantages to name. You can track your infrastructure: you do not have to log into the cloud console to see what exists — you look at the state file, and it describes your whole estate. You can automate changes: to alter something you edit code and re-run, rather than hand-editing live infrastructure. You can collaborate: because the configuration files (everything except the state file) live in a Git repository, a change can be reviewed by a peer before it is applied, just like application code. And you get standardised configuration: one consistent way of describing infrastructure instead of a different style per cloud. You do not need to recite these word for word in an interview — conveying the overview is enough.

writethe .tf files initset up the provider plandry run — preview applycreate for real destroytear down (optional) Write it, preview it, then apply it — never apply blind
The lifecycle. You write configuration, then init sets up the provider, plan shows you exactly what will change (a dry run), and apply makes it real. destroy tears it back down when you no longer need it.

The lifecycle is short. You write the configuration files — and you do not memorise syntax; you go to the HashiCorp Terraform documentation for your provider, copy the example for the resource you want, and adjust it. Then Terraform runs on four commands. terraform init initialises the project and downloads the provider you declared. terraform plan is a dry run: it shows you exactly which resources will be created, changed, or destroyed, so you can catch a mistake before anything happens — a safety feature many tools lack. terraform apply actually makes the changes. And terraform destroy, which some people count as part of the lifecycle, deletes what you created. The golden habit is to always plan before you apply, so you never apply blind.

02
Phase 2 · Lab

Hands-on — from installing Terraform to remote state and modules

Install Terraform and wire up the cloud authentication it depends on, write and run your first configuration to create a real EC2 instance, understand the all-important state file, move it to a proper remote backend with locking, and package reusable infrastructure into a module.

L1

Install Terraform and authenticate to the cloud

Install Terraform with your native package manager — not a manual curl download, which tends to cause path problems. Use Homebrew on Mac, apt on Ubuntu, yum on CentOS; on Windows follow the docs. Always run a recent version, because some cloud provider modules will not work with an old one:

bashbrew tap hashicorp/tap                # on Mac
brew install hashicorp/tap/terraform  # (brew upgrade … if an old version is already installed)
terraform --version                   # verify — you want a current version, e.g. v1.3+

There is one prerequisite people miss. Your configuration only names the provider and region — it holds no credentials — so Terraform authenticates to the cloud through that cloud's own CLI, which you must configure first. On AWS that means running aws configure and giving it your access key, secret key, and default region; on Azure you set up a service principal. A quick way to prove it works: if a plain CLI command like aws s3 ls succeeds, Terraform will authenticate too.

bashaws configure          # enter AWS access key ID, secret access key, default region, output format
aws s3 ls              # if this works, Terraform can authenticate to AWS as well

The key found under Security credentials → Access keys in the AWS console. Without this step, Terraform has no way to talk to your account, no matter how correct your configuration is.

L2

Your first configuration — the anatomy

Start with a local-state example that creates one EC2 instance, and learn each block in it. You do not memorise this — you assemble it from the documentation.

A Terraform file is built from a few blocks. The terraform block at the top declares the required providers (which provider and version — for example hashicorp/aws at ~> 4.16) and a required Terraform version (so someone with an older CLI gets a clear error instead of a mysterious failure). Then a provider block can set options such as the region (optional — it defaults to us-east-1). Finally come the resource blocks, which are the part that actually differs from file to file — everything above is nearly constant boilerplate you can copy:

terraform { } required_providers — which provider + version (e.g. hashicorp/aws ~> 4.16) required_version — the minimum Terraform CLI version provider "aws" { region = "us-west-2" }optional resource "aws_instance" "app_server" { … }the actual thing Boilerplate on top; the resource block is what changes per file
The anatomy. A terraform block (required providers + version), an optional provider block for options like region, and the resource blocks that describe what you actually want.
hclterraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.16"
    }
  }
  required_version = ">= 1.2.0"
}

provider "aws" {
  region = "us-west-2"          # optional — defaults to us-east-1
}

resource "aws_instance" "app_server" {
  ami           = "ami-0c02fb55956c7d316"
  instance_type = "t2.micro"
  tags = { Name = "terraform-demo" }
}

You will not remember the resource syntax, and you are not supposed to — AWS alone has around two hundred services. Go to the HashiCorp Terraform documentation for your provider, find the resource (say aws_instance or aws_lb), copy its example, and adjust it. If a brand-new service has no docs there, that simply means Terraform does not support it yet. Even in an interview, it is fine to say you follow the documentation rather than memorise syntax.

One good-practice habit to adopt from the start: do not hard-code values in the main file. Put the inputs — the AMI, instance type, names, CIDR blocks — into a variables.tf (often called input.tf) file, and reference them with var.name. Put the things you want Terraform to report back — a new instance's private IP, a public IP, an invoke URL — into an outputs.tf file, so a user who ran your config through a pipeline (and has no console access) still gets the details they need. Beyond cleaner code, this makes reviews safer: a change to variables.tf is obviously just a value change, easy to reason about.

L3

The four commands — init, plan, apply, destroy

With the file written and the AWS CLI configured, the whole workflow is four commands. init reads your required_providers and downloads the AWS provider. plan shows you, in detail, exactly what it will create — the AMI, the instance type, the tags — without touching anything. apply asks for confirmation and then really creates the instance. destroy removes it again:

bashterraform init      # initialise the project, download the AWS provider
terraform plan      # dry run — preview every resource that will be created/changed/destroyed
terraform apply     # create for real (prompts yes/no; nothing happens until you confirm)
terraform destroy   # tear it all down when you are finished

Run these against the local-state example and you will watch Terraform create a real EC2 instance in about forty seconds, printing its instance ID at the end. Behind the scenes, at apply, Terraform is converting your configuration into the AWS API calls that create the instance — exactly the "API as code" idea from the theory, now happening live. Two habits: read the plan output every time before you confirm, and remember that nothing is created until you answer yes to apply.

L4

The state file — the heart of Terraform

After your first apply, a new file appears that you did not write — terraform.tfstate. Understanding it is understanding Terraform.

The state file is how Terraform remembers what it built. It was not in your Git repository; Terraform created it on apply, and it records every detail of every resource — the instance ID, the region, the availability zone, the CPU, everything. This is how Terraform "tracks your infrastructure." It is also how future changes work: when you later edit the config — say, change the instance type — and re-run plan, Terraform compares your desired config against the state file to work out the difference ("1 to change") and update only what moved. If you delete the state file, Terraform forgets it ever created the instance. The state file is the single source of truth, and the heart of the whole tool.

That power comes with two rules you must never break. First: never keep state on your local machine or in version control. It holds sensitive information — secrets, VPC and KMS details — so it cannot go into Git; and if two people each run from their own clone, they each get their own state file, which cannot be sensibly merged. If someone forgets to share an updated state, the shared view of the infrastructure is lost. So state must live in one central, remote place. Second: never hand-edit the state file. By default you have write access to your own local state, but if you manually change it, Terraform's view no longer matches reality and it becomes corrupted — it will try to "fix" infrastructure based on a broken picture. State should be effectively read-only to humans; only Terraform should write it.

Why this section matters so much. Ask anyone about Terraform and the first thing they mention is the state file, because almost every serious Terraform problem — corruption, drift, two people applying at once, a near-miss destroy — traces back to how state is stored and protected. The two rules here (remote, and never hand-edited) are the foundation of every senior scenario drill later in this chapter.
L5

Remote backend — S3 and DynamoDB locking

The fix for both state rules is a remote backend. In production this is not optional — it is simply how Terraform is run.

A remote backend is a central, remote storage location for the state file — Amazon S3 on AWS, an Azure storage container on Azure. Storing state in S3 solves the sharing problem: anyone who applies, from anywhere, reads and writes the same state automatically. It solves the durability problem too, because S3 supports versioning, so you can roll back to an earlier state if something goes wrong. But there is one more danger: if two people apply at the same time, they send conflicting instructions to the cloud. To prevent that, you pair the S3 backend with a DynamoDB table for locking. When someone runs Terraform, the state is locked; anyone else is told it is in use and must wait until it is released. That is the entire job of DynamoDB here: serialise applies so only one runs at a time.

Git (VCS).tf files, reviewed CI (Jenkins /GH Actions) Cloud (AWS)real resources S3 — stateremote, versioned DynamoDB — lockone apply at a time Config in Git · applied through CI · state in S3 · locked by DynamoDB
The ideal Terraform setup. Configuration lives in version control and is applied through a CI pipeline (not from laptops); the state file lives remotely in S3 with versioning, and a DynamoDB lock guarantees only one apply runs at a time. You can swap the CI tool or VCS, but not this shape.

Switching the earlier example to a remote backend is a small change: you create the S3 bucket and DynamoDB table (with Terraform itself), then add a backend "s3" block to the terraform block naming the bucket and lock table, and re-run terraform init. One more state good-practice belongs here: isolate state per environment to reduce the blast radius. Keep separate state for dev, staging, and prod, so that if someone corrupts or mishandles state, the damage is contained to one environment rather than taking down everything. This is the shape every organisation should use — the only pieces you may swap are the CI tool (Jenkins, GitHub Actions) and the VCS (GitHub, GitLab, Bitbucket); change the rest and you are using Terraform wrongly.

L6

Modules — reusable infrastructure

The last core idea is modules, and the concept is the same across many DevOps tools (Jenkins, Terraform, and others): a module is a piece of reusable code. Suppose you have configuration that creates an EC2 instance plus a load balancer, or an S3 bucket plus a DynamoDB table, and several teams — or several environments (dev, staging, prod) — all need the same thing. Rather than copy-paste it everywhere, you put that configuration in one place and reference it as a module from other Terraform files. When Terraform runs, it executes the module first, so every consumer gets the same, already-worked-out setup — for instance, the remote-backend-and-lock configuration ready to go — without redoing the effort.

So modules are simply how you write reusable components in Terraform, exactly as functions are in a programming language. Terraform also offers existing modules you can pull in, and this reuse is what keeps a large estate consistent and DRY. In the drills you will see the senior side of this: modules must be versioned and kept focused, because an over-configurable "mega-module" with dozens of toggles becomes harder to live with than the duplication it was meant to remove.

03
Phase 3 · Interview Q&A

The Terraform questions you'll be asked

These are the questions the instructor explicitly flags across Days 16 and 17, with answers built from his explanations. Answer each out loud, then reveal.

Q1What is Infrastructure as Code (IaC)?reveal ▸
Answer

Infrastructure as code is the practice of creating and managing your infrastructure through code rather than by clicking in a console. Instead of manually launching servers, you describe them in configuration files and a tool provisions them. AWS CloudFormation, Azure Resource Manager, OpenStack Heat, and Terraform are all infrastructure-as-code tools — they let you version, review, and automate infrastructure the way you do application code.

Q2Why Terraform, when CloudFormation and Azure Resource Manager already exist?reveal ▸
Answer

Because those tools are each tied to a single cloud — CloudFormation only works on AWS, Resource Manager only on Azure. In a multi-cloud or hybrid world you would have to learn and maintain a separate tool per provider, and rewrite everything whenever you move. Terraform is one tool that works across all of them: you learn it once, and to target a different cloud you mostly just change the provider details. That is its whole reason to exist.

Q3What is "API as code," and how does Terraform work internally?reveal ▸
Answer

Every cloud exposes an API — a programmatic way to create resources. Terraform's maintainers have studied each provider's API and written modules for it, so you never write API calls yourself. You write plain configuration describing what you want; Terraform converts that into the provider's API calls behind the scenes, runs them, and returns the result. That translation from config to provider API is "API as code," and it is what lets one tool automate any cloud — even future ones, once a module is written.

Q4What is an API?reveal ▸
Answer

An API (application programming interface) is a programmatic way to talk to an application, as opposed to a user interface. Instead of opening a browser and clicking, a program sends a request — for example an HTTP or curl call — to the application's API and gets a result back. Applications like GitHub, Google, and AWS expose APIs so that scripts and tools, not just humans, can drive them.

Q5What are the advantages of Terraform?reveal ▸
Answer

You can track your infrastructure — the state file describes your whole estate, so you do not have to log into the console to see what exists. You can automate changes rather than hand-editing live infrastructure. You can collaborate, because the configuration lives in Git and can be peer-reviewed before it is applied. And you get standardised configuration — one consistent way of describing infrastructure instead of a different style per cloud.

Q6What is the Terraform lifecycle / the four main commands?reveal ▸
Answer

You write the configuration, then run four commands. terraform init initialises the project and downloads the declared provider. terraform plan is a dry run showing exactly what will be created, changed, or destroyed. terraform apply makes the changes for real. terraform destroy tears the resources down. The rule is always to plan before you apply, so you never apply blind.

Q7What is terraform plan / a dry run for?reveal ▸
Answer

plan is a dry run: it shows you every resource Terraform would create, change, or destroy, without changing anything. It lets you catch mistakes before they happen — for instance, if you copied an example and forgot to edit a value, the plan reveals it. Many tools do not offer this; in Terraform it is the safety step you always take before apply.

Q8How does Terraform authenticate to a cloud provider?reveal ▸
Answer

Through that cloud's own CLI, which you configure first — Terraform's files hold no credentials. On AWS you run aws configure with your access key, secret key, and region; on Azure you set up a service principal. A quick check: if a plain CLI command such as aws s3 ls works, Terraform will authenticate too. Without this, Terraform cannot talk to your account no matter how correct the config is.

Q9What is the state file, and why is it so important?reveal ▸
Answer

The state file (terraform.tfstate) is how Terraform remembers what it built — it records every attribute of every resource it created. It is how Terraform tracks your infrastructure and how it computes what to change on the next run, by comparing your config against it. It is the single source of truth and the heart of Terraform: delete it and Terraform forgets your infrastructure exists; corrupt it and you have effectively broken your Terraform.

Q10Why should you never store state locally or in Git?reveal ▸
Answer

Two reasons. It contains sensitive information — secrets, VPC and KMS details — so it must not go into a Git repository. And if multiple people run from their own clones, each gets a separate state file, and those cannot be merged; if someone forgets to share an update, the shared view of the infrastructure is lost, which can be catastrophic. So state must live in one central remote backend, never on a laptop or in version control.

Q11Why should you never manually edit the state file?reveal ▸
Answer

Because if you hand-edit state, Terraform's record no longer matches reality and the state becomes corrupted — Terraform will act on a broken picture and try to "fix" infrastructure incorrectly. State should be effectively read-only to humans; only Terraform should write to it. Even in a remote backend you grant read access and let Terraform be the only writer.

Q12What is a remote backend, and why S3?reveal ▸
Answer

A remote backend is a central, remote place to store the state file — S3 on AWS, an Azure storage container on Azure. It solves the sharing problem, because everyone who applies reads and writes the same state; and the durability problem, because S3 versioning lets you roll back to an earlier state. In production you always use a remote backend; local state is only for a first practice run.

Q13What is the purpose of DynamoDB in a Terraform setup?reveal ▸
Answer

DynamoDB provides state locking. If two people apply at the same time, they could send conflicting instructions to the cloud. Pairing the S3 backend with a DynamoDB lock table means the state is locked while someone is applying, and anyone else is told to wait until it is released. Its whole job is to serialise applies so only one runs at a time.

Q14Tell me about your Terraform setup.reveal ▸
Answer

The configuration lives in a Git repository and is applied through a CI pipeline — nobody applies from a laptop. The state file is stored remotely in an S3 bucket with versioning, and locked with a DynamoDB table so only one apply runs at a time. State is isolated per environment to keep the blast radius small. You can swap the CI tool or the VCS, but that shape — config in version control, state remote and locked — is the standard.

Q15What are modules, and why use them?reveal ▸
Answer

A module is reusable Terraform code. If several teams or environments need the same setup — say an EC2 instance plus a load balancer, or an S3-plus-DynamoDB backend — you put that configuration in one place and reference it as a module instead of copy-pasting. Terraform runs the module for each consumer, so everyone gets the same worked-out setup. It keeps a large estate consistent and DRY, and good modules are versioned and kept focused rather than over-configurable.

Q16What are the disadvantages / problems with Terraform?reveal ▸
Answer

Several honest ones. The state file is a single source of truth, so if it is corrupted or mishandled you have compromised your whole Terraform. It is not bi-directional: if someone changes a resource directly in the cloud console, Terraform does not detect and auto-correct it — you have to reconcile with terraform refresh. It is not naturally GitOps-friendly, since a manual cloud change makes Git no longer the true source of truth, and tools like Argo CD lack native Terraform support. It can get complex and hard to manage at large scale. And it is a poor fit as a configuration manager — that crossover with Ansible does not work well, because each tool should stay in its own lane.

Q17Do you memorise Terraform resource syntax?reveal ▸
Answer

No — and you are not expected to. AWS alone has around two hundred services, and Azure a similar number; nobody remembers every resource's syntax. You go to the HashiCorp Terraform documentation for your provider, find the resource, copy its example, and adjust it. In an interview it is perfectly good to say you follow the docs and can write it from there, rather than reciting exact syntax from memory.

04
Phase 4 · Scenario drills

The senior layer — Terraform where state and blast radius decide safety

You can now write configs, run the lifecycle, and reason about state. These 17 scenarios are the senior version: state corruption and locking, drift, blast radius, safe refactors, module design, and knowing when not to use Terraform. Answer out loud, reveal, and mark yourself.

🏗️

Terraform

17 scenarios · 1–17

The complete Terraform scenario set, worked through in depth. This is the thinking and the story behind each one, not commands to memorise. Senior Terraform is about state, blast radius, drift, module design, and safe delivery — the sharp edges that cause outages — not terraform apply. Interviewers want to hear that you treat state as the crown jewels and that you read every -/+ in a plan.

⚡ The universal Terraform principles (the "reflex")

Terraform is governed by a handful of principles that answer most questions and prevent most outages. Burn these in:

  1. State is the source of truth about reality — protect it like production data. Remote backend, locking, encryption, and versioning are non-negotiable for shared infra. Corrupt or lost state is the worst thing that can happen.
  2. The plan is the contract — read every line, especially -/+ (replace). A provider quietly recreating your RDS instance is how you cause an outage. Apply the saved plan so what runs is exactly what was reviewed.
  3. State boundaries = blast-radius boundaries. Split state by lifecycle/component and isolate environments into separate accounts, so a mistake is contained.
  4. Pin everything — provider versions, module versions, and commit the .terraform.lock.hcl. Upgrades are deliberate, never accidental.
  5. Terraform owns what's in state; drift is when reality diverges. Detect drift and either codify or reconcile it — and remove humans' ability to hand-mutate managed resources.
  6. Everything through the pipeline — plan on PR (reviewed), apply on merge (the saved plan), nobody applies to prod from a laptop.

---

📋 The full scenario inventory (distinct — no padding)

A. State management (the core of Terraform)

  1. Two engineers ran apply at once and corrupted state
  2. State structure strategy (splitting by blast radius)
  3. Refactoring / splitting a huge state safely
  4. Secrets in Terraform (state is plaintext)

B. Drift & reconciliation

  1. Someone changed infra in the console — drift
  2. plan shows a perpetual diff (never converges)

C. Failures & blast radius

  1. terraform destroy almost nuked production
  2. A provider/module upgrade broke everything
  3. apply failed halfway — partial-state recovery
  4. A resource must be replaced without downtime

D. Code design

  1. Module design — DRY without over-abstracting
  2. count vs for_each (the re-indexing trap)
  3. Importing existing (clickops) infrastructure

E. Multi-environment & delivery

  1. Multi-account / region / environment structure
  2. CI/CD for Terraform (GitOps for infra)
  3. Policy as code & cost control

F. Tool choice

  1. Terraform vs alternatives / when NOT to use it

---

1Two engineers ran apply at once and corrupted stateScenario

Two people applied against the same state at nearly the same time; now the state is inconsistent with reality — resources Terraform thinks exist don't, or vice-versa, and plans show bizarre changes.

What is actually happening (the mental model).

Terraform reads state, computes a plan, and writes state back. If two applies interleave without a lock, they race — each overwrites the other's view, and the result is a state that no longer matches reality. The root cause is no state locking, which a proper remote backend provides. Recovery leans on state versioning (you kept it, right?).

How to work through it.

  1. Stop further applies — lock the workspace / freeze.
  2. Restore from a prior state version — S3 bucket versioning (or Terraform Cloud/managed backend history) lets you roll back to the last consistent state.
  3. Reconcile carefully — run terraform plan and scrutinise it against reality before applying anything; use terraform refresh/state commands to align state with actual resources if needed.
  4. Fix the root cause: a remote backend with locking — S3 + a DynamoDB lock table, or a managed backend (Terraform Cloud/GitLab) — which serialises applies so this cannot happen.

The root causes and trade-offs.

The root cause is simply not having state locking in place at all, and a related red flag is using local state for infrastructure that's actually shared across a team. The fix is a remote backend with locking, and the trade-off is a bit of upfront setup for a safety guarantee that's genuinely mandatory for any team working together.

The trap that less-experienced engineers fall into.

Using local state for shared infrastructure (no locking, no durability, no collaboration) — and not enabling state-bucket versioning, so there's nothing to roll back to when state corrupts.

🎯 Interviewer follow-up questions you should expect.

  • "How do you prevent concurrent-apply corruption?" You use a remote backend with locking — an S3 backend with a DynamoDB lock table, or a managed backend.
  • "State got corrupted — how do you recover?" You restore a previous state version, since S3 versioning is enabled, and reconcile with a plan before applying.
  • "Why never local state for a team?" Because it has no locking, no durability, and no shared source of truth.
Say it like this"Concurrent applies mean there was no state lock — that's the real bug. A proper remote backend, S3 with a DynamoDB lock table or a managed backend, serialises applies so this can't happen. To recover I roll back to a previous state version — I always keep S3 versioning on the state bucket — and reconcile with a plan before touching anything. Local state for shared infrastructure is something I'd flag immediately; state is the crown jewels."

---

Mark:
2State structure strategy (splitting by blast radius)Scenario

"How do you structure Terraform state?" Or a real problem: one giant state file for everything, so every plan is slow and every change risks the whole estate.

What is actually happening (the mental model).

State boundaries are blast-radius boundaries. One monolithic state means slow plans, a huge blast radius (a mistake can affect everything), and constant contention between people. So you split state by lifecycle and blast radius — network, data, app — per environment, rather than one giant state.

How to reason about it.

  1. Split by lifecycle/component and blast radius — e.g. networking, data stores, and application layers in separate states, per environment. Rarely-changed foundational things (VPC) shouldn't share state with fast-moving app config.
  2. Remote backend per state, encrypted and locked.
  3. Cross-stack references — publish outputs and consume via terraform_remote_state data sources (or data sources on the resources), so states can reference each other without merging.
  4. Terragrunt to keep multi-environment/multi-component config DRY when the duplication gets painful.

The root causes and trade-offs.

A single, monolithic state file gives you slow plans, a huge blast radius, and contention between people applying changes at the same time. Splitting into many smaller states requires more wiring between them, but each one stays isolated and fast. The trade-off is a bit more structure and more cross-references between states, in exchange for real safety and speed.

The trap that less-experienced engineers fall into.

One giant state file for the whole estate — slow, high-contention, and a single mistake can ripple across everything.

🎯 Interviewer follow-up questions you should expect.

  • "How do you split state?" By lifecycle and blast radius — network, data, app — per environment, since state boundaries are blast-radius boundaries.
  • "How do states reference each other?" Through remote-state data sources or published outputs.
  • "What's the cost of one giant state?" Slow plans, a huge blast radius, and contention between people applying at the same time.
Say it like this"I never put everything in one state file — a monolith means slow plans, a huge blast radius, and everyone stepping on each other. I split by lifecycle and blast radius: networking, data, and app layers in separate states, per environment, so foundational things like the VPC aren't in the same state as fast-moving app config. They reference each other through remote-state data sources. When multi-environment duplication gets painful I reach for Terragrunt. The mental model is: state boundaries are blast-radius boundaries."

---

Mark:
3Refactoring / splitting a huge state safelyScenario

A monolithic state takes 15–20 minutes to plan, and you need to split it into smaller states — or refactor code (rename resources, move into modules) without Terraform destroying and recreating everything.

What is actually happening (the mental model).

Terraform tracks each resource by its address in state. If you rename a resource or move it into a module, its address changes, and Terraform — not knowing it's the same resource — plans to destroy the old and create the new. State surgery (moved blocks, state mv) tells Terraform "this is the same thing at a new address" so it just updates the pointer instead of recreating. This is the highest-risk operation in Terraform, so you back up state first and verify no-op plans.

How to work through it.

  1. Back up state first — copy it. State surgery is the riskiest thing you do; a mistake can orphan or destroy resources.
  2. Move resources with moved blocks (declarative, reviewable — preferred) or terraform state mv (imperative), or remove-and-re-import into the new state.
  3. One component at a time, verifying a no-op plan on both sides before proceeding (a no-op plan proves the move was clean).
  4. Use -target sparingly — routine use is a smell that state boundaries are wrong.

The root causes and trade-offs.

Renaming a resource or moving it between modules changes its address in state, and Terraform reads that as the old resource being destroyed and a brand-new one being created. State surgery — using a moved block or terraform state mv — avoids that entirely. The trade-off is spending a little care and time now against the alternative, which is an outage from recreating live resources unnecessarily.

The trap that less-experienced engineers fall into.

Renaming a resource or moving it into a module and applying without a moved block — Terraform destroys and recreates the live resource (an outage). And doing state surgery without backing up state first.

🎯 Interviewer follow-up questions you should expect.

  • "You renamed a resource and the plan shows destroy+create — why and how to avoid?" The address changed, so you use a moved block or state mv so Terraform updates the pointer instead of recreating the resource.
  • "How do you safely split a big state?" You back up the state, move one component at a time with state mv or a re-import, and verify a no-op plan on both sides.
  • "When is -target a smell?" Routine use means your state boundaries are wrong — you should refactor instead of working around every apply.
Say it like this"Terraform tracks resources by their address in state, so renaming one or moving it into a module changes the address and Terraform plans to destroy and recreate it — an outage on a live resource. I avoid that with moved blocks or state mv, which tell Terraform it's the same resource at a new address. Splitting a big state I do carefully — back up state first because it's the highest-risk operation, move one component at a time, and verify a no-op plan on both sides before moving on. And routine -target use I treat as a smell that the state boundaries are wrong."

---

Mark:
4Secrets in Terraform (state is plaintext)Scenario

A security question, or a review finding: "how do you handle secrets in Terraform?" The critical fact many miss: anything Terraform manages ends up in state in plaintext.

What is actually happening (the mental model).

Terraform state stores the full attributes of every managed resource — including generated passwords, private keys, and any secret it touches — in plaintext. So the state file itself is a secret, and protecting it is step one. Beyond that, don't hardcode secrets; source them from a secrets manager, and minimise what lands in state.

How to reason about it.

  1. Encrypt the backend and restrict access tightly — state is plaintext secrets, so the backend must be encrypted (S3 SSE-KMS) with least-privilege access, treated as a secret store.
  2. Don't hardcode — source secrets from Vault/SSM/Secrets Manager via data sources at apply time.
  3. Mark outputs sensitive so they're not printed; keep secret .tfvars out of git.
  4. Minimise what's in state — prefer letting a dedicated secrets system own the secret and having Terraform only reference it, rather than Terraform generating/holding it.

The root causes and trade-offs.

The state file being plaintext is simply inherent to how Terraform works, so the mitigation is encryption at rest, tight access control on who can read the state, and minimising how many real secrets ever end up stored in it in the first place. The trade-off is a bit of indirection, referencing a proper secret store instead of a raw value, in exchange for real security.

The trap that less-experienced engineers fall into.

Not realising state contains secrets in plaintext — leaving the state backend unencrypted or broadly readable, or committing a state file / secret .tfvars to git.

🎯 Interviewer follow-up questions you should expect.

  • "Are secrets safe in Terraform?" No — state stores them in plaintext, so you encrypt the backend and restrict access to it.
  • "How do you source secrets?" From Vault or Secrets Manager via data sources, marking outputs sensitive, and keeping secret tfvars out of git.
  • "How do you minimise exposure?" You let a secrets manager own the secret, and have Terraform reference it rather than generating or holding it.
Say it like this"The critical thing people miss is that anything Terraform manages ends up in state in plaintext — a generated password, a private key, all of it. So the state backend must be encrypted with tightly scoped access, treated as a secret itself. I pull secrets from Vault or Secrets Manager at apply time rather than hardcoding, mark sensitive outputs, and keep secret tfvars out of git. Where possible I let a dedicated secrets system own the secret and have Terraform only reference it, to minimise what lands in state."

---

Mark:
5Someone changed infra in the console — driftScenario

A manual change was made in the cloud console, and the next terraform plan shows destructive changes — Terraform wants to "fix" reality back to the code, potentially reverting something important or destroying a manually-added resource.

What is actually happening (the mental model).

Terraform's job is to make reality match the code. When someone changes a managed resource by hand, reality drifts from state/code, and the next plan wants to reconcile — which might revert a legitimate fix or, worse, be destructive. Drift is fundamentally a governance problem before it's a technical one: the real fix is removing the ability to hand-mutate managed resources.

How to work through it.

  1. Detect with terraform plan (it shows the drift).
  2. Decide: was the manual change intentional and correct? If yes, codify it — update the config (or import a manually-added resource) so Terraform is the source of truth again. If no, let Terraform reconcile it away.
  3. Prevent drift systemically: restrict console write access via IAM so humans can't mutate managed resources; run scheduled drift-detection plans in CI so you catch drift in hours, not at the next unrelated apply; a "no manual changes" policy with break-glass exceptions.

The root causes and trade-offs.

The root cause is usually humans still having console write access to resources that Terraform is supposed to manage, combined with no drift detection in place to catch it when that access gets used. The fix is tighter IAM permissions plus scheduled drift detection, and the trade-off is losing a little ad-hoc flexibility in exchange for real consistency and safety.

The trap that less-experienced engineers fall into.

Discovering drift only when an unrelated apply suddenly wants to make a surprising destructive change — because there was no scheduled drift detection, and no lock-down of console writes.

🎯 Interviewer follow-up questions you should expect.

  • "Someone changed a managed resource in the console — what do you do?" You decide whether it was intentional, and codify or import it, or unintentional, and reconcile it, then prevent it from happening again.
  • "How do you catch drift early?" With a scheduled plan in CI as drift detection, not by waiting for the next unrelated apply.
  • "How do you stop drift at the source?" By restricting console write access to managed resources through IAM.
Say it like this"Drift is a governance problem before it's a technical one. plan surfaces it; then I ask whether the manual change was legitimate. If it was, I codify it — import the resource or update config so Terraform is the source of truth again. If it wasn't, I let Terraform reconcile. But the durable fix is removing the ability to click-change managed resources: tighten IAM so humans can't mutate them, and run scheduled drift-detection plans in CI so I catch drift in hours, not at the next unrelated apply when it causes a surprise destroy."

---

Mark:
6plan shows a perpetual diff (never converges)Scenario

Every terraform plan shows the same change, even immediately after a successful apply — the plan never becomes a clean no-op. Noisy and dangerous, because it hides real diffs.

What is actually happening (the mental model).

A perpetual diff means Terraform's desired state and the provider's actual state never match for some attribute, so every plan wants to "fix" it. The usual causes: the provider normalises a value differently than your config (reordered IAM policy JSON, canonicalised strings), an attribute is computed/mutated by another system (autoscaling changing desired capacity, another tool adding tags), or a genuine bug.

How to work through it.

  1. Identify which attribute flaps — read the plan diff carefully.
  2. Diagnose the cause: provider normalisation (e.g. IAM policy JSON key ordering — use jsonencode / a canonical form), an externally-mutated attribute (ASG desired capacity, tags added by a policy/automation), or a computed value.
  3. Fix: either match the provider's canonical form in your config, or use ignore_changes (lifecycle block) on the attribute that a different system legitimately owns, so Terraform stops fighting it.

Root causes, and how to tell them apart.

Provider normalisation (config value differs from provider's canonical form — match it); external mutation (ASG capacity, auto-added tags — ignore_changes); a computed attribute. The specific flapping attribute tells you which.

The trap that less-experienced engineers fall into.

Ignoring a perpetually-noisy plan ("it always shows that change, just apply it") — which is dangerous because a real diff hides in the noise and gets applied unnoticed.

🎯 Interviewer follow-up questions you should expect.

  • "Every plan shows the same change even after apply — why?" Because the desired and actual state never match — this can come from provider normalisation, an externally-mutated attribute, or a computed value.
  • "How do you fix it?" You match the canonical form, or use ignore_changes on attributes another system owns.
  • "Why is a noisy plan dangerous?" Because it hides real diffs — people stop reading it.
Say it like this"A perpetual diff means Terraform's desired state and the provider's actual state never match for some attribute, usually because something normalises or mutates the value outside Terraform — reordered IAM policy JSON, autoscaling changing desired capacity, or another tool adding tags. I identify which attribute flaps, then either match the provider's canonical form or use ignore_changes on that attribute so Terraform stops fighting a system that legitimately owns it. A noisy plan that always shows changes is dangerous precisely because it hides real diffs."

---

Mark:
7terraform destroy almost nuked productionScenario

Someone ran terraform destroy (or an apply that planned mass destruction) believing they were in dev, but were pointed at prod — and real infrastructure started coming down before it was caught.

What is actually happening (the mental model).

The root issue is insufficient blast-radius isolation — a single human error could reach prod at all. The senior response isn't "be more careful" (not a control); it's structural guardrails: separate accounts/state per environment so a dev mistake physically cannot touch prod, prevent_destroy on critical resources, and applying the saved plan so what runs is exactly what was reviewed.

How to work through it (prevention-focused).

  1. Isolate environments into separate cloud accounts (and separate state) — a mistake in dev can't reach prod.
  2. prevent_destroy lifecycle on critical resources (databases, buckets) — Terraform refuses to destroy them.
  3. Plan-then-apply-the-saved-plan (terraform plan -out=tfplanterraform apply tfplan) so there's no gap between what was reviewed and what runs — a mass-destroy would be visible in review.
  4. Require review/approval before apply; never apply to prod from a laptop.

The root causes and trade-offs.

The root cause is usually no real environment isolation, so a single command can reach production, combined with no prevent_destroy lifecycle rule in place and a habit of applying without ever reviewing the saved plan first. The fix is proper isolation plus explicit guardrails, and the trade-off is a bit more structure in exchange for a hard safety boundary around production.

The trap that less-experienced engineers fall into.

Relying on "be careful" / naming conventions instead of hard isolation — sharing state or credentials across prod and non-prod, so a wrong-directory or wrong-workspace mistake can destroy prod.

🎯 Interviewer follow-up questions you should expect.

  • "How do you prevent a wrong-environment destroy?" You separate accounts and state per environment, so a dev mistake can't reach prod, and set prevent_destroy on critical resources.
  • "Why apply the saved plan?" So that what runs is exactly what was reviewed — a mass-destroy would be caught in review.
  • "Is 'be more careful' a fix?" No — it's not a control, you need structural guardrails.
Say it like this"This is why I isolate environments into separate accounts and state — a mistake in dev physically cannot reach prod. On critical resources I set prevent_destroy, and I enforce plan-then-apply-the-saved-plan so what gets approved is exactly what runs, which means a mass-destroy shows up in review. A near-miss like this I'd turn into a permanent guardrail, not a 'be more careful' reminder, because 'be careful' is not a control."

---

Mark:
8A provider/module upgrade broke everythingScenario

You bumped the AWS provider (or a module) version, and the next plan shows unexpected changes — or worse, wants to replace live resources like an RDS instance.

What is actually happening (the mental model).

Provider and module versions change behaviour, defaults, and sometimes resource schemas between releases. An unpinned or careless upgrade can silently plan a destroy-recreate of critical resources. The senior discipline: pin versions, commit the lock file, upgrade deliberately (read the changelog, test low first), and scrutinise the plan for every -/+ replace.

How to work through it.

  1. Pin provider and module versions (required_providers constraints), and commit .terraform.lock.hcl so upgrades are intentional, never accidental.
  2. Upgrade deliberately — read the changelog for breaking changes, bump in a lower environment first, and read the plan carefully for any -/+ (replace) — a provider recreating an RDS instance is an outage.
  3. If a refactor changes resource addresses, use moved blocks / state mv so Terraform updates the pointer instead of destroying/recreating.

The root causes and trade-offs.

The root cause is usually unpinned provider or module versions, which lets an upgrade happen accidentally, combined with a habit of not actually reading which resources the plan says will be replaced. The fix is pinning versions, upgrading deliberately, and always reading the plan, and the trade-off is a slower, more careful upgrade process in exchange for real safety.

The trap that less-experienced engineers fall into.

Not committing the lock file (so terraform init silently pulls a new provider version), and applying an upgrade without reading the plan for destructive replacements.

🎯 Interviewer follow-up questions you should expect.

  • "How do you make provider upgrades safe?" You pin versions, commit the lock file, upgrade deliberately, test in a lower environment, and read the plan for any -/+ replacements.
  • "A provider upgrade wants to replace your database — what do you do?" You stop and read why, then use a moved block or a lifecycle rule to avoid the recreation, or hold off on the upgrade entirely.
  • "Why commit the lock file?" So upgrades are intentional — without it, init can pull a new version silently.
Say it like this"I pin providers and commit the lock file so upgrades are intentional, never accidental. When I do upgrade, I read the changelog, run it in dev first, and scrutinise the plan for any -/+ replace — a provider bump silently recreating an RDS instance is how you cause an outage. If a refactor changes resource addresses, I use moved blocks so Terraform updates the state pointer instead of destroying and recreating. The plan is the contract; I read every replace carefully."

---

Mark:
9apply failed halfway — partial-state recoveryScenario

An apply errored partway through — some resources were created/modified, others weren't. Now state partially reflects reality, and you're unsure what actually got applied.

What is actually happening (the mental model).

Terraform applies resources according to the dependency graph, and if one fails (an API error, a timeout, a permissions issue), it stops — resources already applied are recorded in state, the rest aren't. Terraform is designed to be re-runnable: state tracks what succeeded, so a corrected re-apply continues from where it left off. The key is to understand the current state vs reality before re-running.

How to work through it.

  1. Read the error — which resource failed and why (API limit, timeout, permissions, a dependency that isn't ready)?
  2. Run terraform plan — it shows what's left to do based on current state, telling you what succeeded and what didn't.
  3. Fix the cause (permissions, a race, a quota) and re-apply — Terraform continues from where it stopped.
  4. If a resource was created in the cloud but not recorded in state (a partial failure mid-resource), you may need to import it or terraform state reconcile to avoid a duplicate-create on re-apply.

The root causes and trade-offs.

This can come from a transient API error, a timeout, a permissions problem, a dependency-ordering race, or a resource that was genuinely created but never recorded in state. The plan itself is what reveals the gap, and importing or reconciling the state is what handles the case where something was created but never actually tracked.

The trap that less-experienced engineers fall into.

Panicking and re-running blindly, or manually deleting half-created resources — instead of reading the plan to see what state thinks exists, fixing the cause, and letting Terraform continue.

🎯 Interviewer follow-up questions you should expect.

  • "An apply failed halfway — how do you recover?" You read the error, run plan to see what's left, fix the cause, and re-apply, since Terraform continues from where it left off.
  • "A resource was created but isn't in state — what now?" You import it, or reconcile it, so a re-apply doesn't try to create a duplicate.
Say it like this"Terraform applies along the dependency graph and stops at the first failure, recording what succeeded in state — it's designed to be re-runnable. So I read the error to see which resource failed and why, run a plan to see what's left based on current state, fix the cause — often a permissions issue, a quota, or a dependency race — and re-apply, and it continues from where it stopped. The one gotcha is a resource that got created in the cloud but wasn't recorded in state; there I import it first so the re-apply doesn't try to create a duplicate."

---

Mark:
10A resource must be replaced without downtimeScenario

A change forces a resource to be replaced (the plan shows -/+), or you deliberately need to recreate one — but the naive replace destroys the old before creating the new, causing downtime.

What is actually happening (the mental model).

Some attribute changes are immutable and force replacement — Terraform must destroy and recreate. By default it destroys then creates, which means a gap of downtime. The create_before_destroy lifecycle flips that order — create the replacement first, cut over, then destroy the old — enabling zero-downtime replacement (with care around unique constraints like names/ports). And you can force a replacement deliberately with -replace (formerly taint).

How to work through it.

  1. Understand why it's being replaced — which attribute forces replacement (the plan says "forces replacement").
  2. If downtime matters, add lifecycle { create_before_destroy = true } so the new resource is created before the old is destroyed. Handle unique constraints (a name/port collision during the overlap — often needs a name change or a moving target).
  3. To force a replace deliberately: terraform apply -replace=<address> (replaces terraform taint).
  4. Combine with the deployment strategy (blue/green at the infra level) for truly zero-downtime cutover.

The root causes and trade-offs.

Changing an immutable attribute forces Terraform to replace the resource, and its default destroy-then-create order causes real downtime while that happens. Adding create_before_destroy avoids that downtime, at the cost of briefly running two copies of the resource at once, which means you also need to handle any unique-constraint collisions that causes.

The trap that less-experienced engineers fall into.

Applying a replacement without realising it destroys-then-creates (an outage), and not knowing create_before_destroy exists to reverse the order.

🎯 Interviewer follow-up questions you should expect.

  • "A change forces a resource replacement with downtime — how do you avoid it?" You use the create_before_destroy lifecycle rule, so the new resource is created before the old one is destroyed.
  • "How do you force a replacement?" With terraform apply -replace=<addr>, which is the modern equivalent of taint.
  • "What's the catch with create_before_destroy?" Unique constraints, like a name or a port, can collide during the overlap, and you have to handle that.
Say it like this"Some attribute changes are immutable so Terraform must replace the resource, and by default it destroys then creates — a downtime gap. If that matters, I add create_before_destroy in the lifecycle block so the new resource is created and cut over before the old one is destroyed, handling any unique-name or port collision during the overlap. To force a replacement deliberately I use -replace, which is the modern taint. The plan always tells me when a change forces replacement, and I read that carefully because it's an outage risk."

---

Mark:
11Module design — DRY without over-abstractingScenario

Teams are copy-pasting Terraform, and you're asked to standardise with reusable modules — or you've inherited a mega-module with 40 variables that nobody understands.

What is actually happening (the mental model).

Reusable, versioned modules with clear interfaces are the goal — but the failure mode is over-abstraction: a single mega-module with dozens of toggles to handle every conceivable case becomes impossible to reason about and worse than duplication. The senior balance is focused, opinionated modules with tight interfaces and sane defaults, versioned like a product.

How to reason about it.

  1. Reusable modules with clear inputs/outputs, sane defaults, published to a registry; consumers pin a version (v1.2.0), not a moving target.
  2. Avoid premature/over-abstraction — a module with 40 variables to cover every case is worse than two focused modules. Prefer opinionated modules over infinitely-configurable ones.
  3. Version and changelog modules like a product — a breaking change to a shared module can break every consuming team.

The root causes and trade-offs.

Copy-pasting configuration gives you no reuse and lets each copy drift independently, while an over-parameterised mega-module becomes unmaintainable in its own way. The sweet spot is a focused module with good defaults, and the trade-off is sometimes accepting a bit of duplication rather than building one monstrous, do-everything abstraction.

The trap that less-experienced engineers fall into.

Building a mega-module with 40 toggles to handle every scenario (unmaintainable), or not versioning modules so a change breaks all consumers at once.

🎯 Interviewer follow-up questions you should expect.

  • "How do you design reusable modules?" With tight interfaces, sane defaults, versioning in a registry, and consumers pinning the versions they use.
  • "What's the over-abstraction trap?" A mega-module with dozens of toggles, which ends up worse than two focused modules.
  • "Why version modules?" Because a breaking change to a shared module can break every team consuming it.
Say it like this"I build reusable modules with tight interfaces and good defaults, versioned in a registry so teams consume v1.2.0, not a moving target. The trap is over-abstraction: a mega-module with forty toggles to handle every scenario becomes unmaintainable and impossible to reason about. I'd rather have a few focused, opinionated modules. And I version and changelog them like a product, because a breaking change to a shared module can break every team consuming it."

---

Mark:
12count vs for_each (the re-indexing trap)Scenario

You remove one item from a list of resources created with count, and the plan shows Terraform destroying and recreating several resources — not just the one you removed. A favourite interview question because it exposes real understanding.

count — positional: remove middle → re-index → destroy/recreate [0] [1] del [2]→[1] [3]→[2] ← recreated for_each — stable keys: remove one → only that one changes ["a"] ["b"] del ["c"] ✓ ["d"] ✓
count addresses by position, so removing a middle item re-indexes and recreates the rest; for_each uses stable keys.

What is actually happening (the mental model).

count addresses resources by positional index ([0], [1], [2]). Remove the middle element of a list, and everything after it shifts index — so [2] becomes [1], and Terraform sees the resource "at index 1" as changed and destroys/recreates it, cascading down the list. for_each addresses resources by a stable map key — remove one key and only that resource is affected. So for anything that changes over time, prefer for_each.

How to reason about it.

  1. for_each (keyed by a stable identifier) for collections that evolve — removing one item touches only that item.
  2. count only for truly positional/identical things that don't get items removed from the middle (e.g. "N identical instances" where you only scale the number at the end).
  3. If you have existing count-based resources and need to migrate, use moved blocks to re-key without recreation.

The root causes and trade-offs.

count indexes its resources positionally, so removing one from the middle of the list shifts every index after it and cascades into recreating them all. for_each uses stable keys instead, so removing one item never touches the others. The trade-off is that for_each needs a genuinely stable key, from a map or a set, which is a small design cost for a big safety win.

The trap that less-experienced engineers fall into.

Using count over a list that changes over time — then removing a middle element re-indexes and destroys/recreates multiple live resources, a nasty surprise (potential outage).

🎯 Interviewer follow-up questions you should expect.

  • "count vs for_each — why does it matter?" count is positional, so removing a middle item re-indexes and recreates everything after it, while for_each uses stable keys, so only the removed item is affected.
  • "When would you use count?" For truly identical, positional resources that you only ever scale at the end — otherwise you'd use for_each.
Say it like this"I default to for_each over count for collections that evolve. With count, resources are addressed by position, so deleting the second of five subnets shifts the indices and Terraform destroys and recreates three of them — a nasty surprise and a potential outage. for_each keys resources by a stable identifier, so removing one touches only that one. It's a small syntax choice with a big blast-radius consequence, and getting it wrong has caused real outages."

---

Mark:
13Importing existing (clickops) infrastructureScenario

Production was built by hand in the console, and you must bring it under Terraform management without disrupting it.

What is actually happening (the mental model).

You need Terraform's state and code to accurately represent resources that already exist. terraform import (or import blocks in 1.5+) brings an existing resource into state; then you write config to match it, and iterate until plan shows zero changes — a no-op plan is your proof the code correctly represents reality. Do it incrementally, low-risk resources first — rushing risks Terraform "fixing" prod to match imperfect code (an outage).

How to work through it.

  1. For each resource: write the matching config, then terraform import <address> <id> (or an import block).
  2. Iterate on the config until plan is a no-op — zero changes proves the code accurately captures the real resource. A diff means your code doesn't match reality yet.
  3. Incrementally — start with low-risk resources to build confidence and tooling before touching the database or load balancer.
  4. Manage forward once imported.

The root causes and trade-offs.

The root cause is simply existing infrastructure that was clicked together in the console with no code behind it at all. Importing it and then iterating on the configuration until the plan shows no changes is what brings it properly under management. The trade-off is careful, incremental work now, against the real risk of Terraform reconciling production to a config that isn't quite right yet.

The trap that less-experienced engineers fall into.

Importing a resource and applying while the config still has a diff — so Terraform "fixes" the live resource to match the imperfect code, causing an outage. Or trying to import everything big-bang.

🎯 Interviewer follow-up questions you should expect.

  • "How do you bring existing infra under Terraform?" You import it, write matching configuration, and iterate until the plan is a no-op, working incrementally and starting with the lowest-risk resources.
  • "How do you know the import is correct?" A zero-change plan proves the code matches reality.
  • "Why not import everything at once?" Because it's too risky — a config diff would make Terraform reconcile production to imperfect code.
Say it like this"I onboard existing infra incrementally, not big-bang. For each resource I write the matching config, import it into state, and iterate on the config until plan shows zero changes — that no-op plan is my proof the code accurately represents reality. I start with low-risk resources to build confidence before touching the database or load balancer. Rushing an import and getting the config wrong means Terraform 'fixing' prod to match imperfect code — which is an outage."

---

Mark:
14Multi-account / region / environment structureScenario

Design Terraform for an organisation with many accounts, regions, and environments — and the interviewer probes workspaces vs directory-per-env and how you isolate prod.

What is actually happening (the mental model).

The organising principle is isolation: separate state and separate accounts per environment so a mistake is contained. The key opinion: for prod-critical separation, prefer directory-per-environment over Terraform workspaces, because workspaces share one config and it's easy to forget which one you're in (the near-destroy scenario). Provider aliases handle multi-region.

How to reason about it.

  1. Separate state + separate accounts per environment — isolation is the security and blast-radius benefit.
  2. Directory-per-environment (with per-env .tfvars) over workspaces for prod-critical separation — workspaces share config and hide which environment you're targeting; directories make it explicit. (Workspaces are fine for ephemeral/similar envs.)
  3. Provider aliases for multi-region within a config.
  4. Landing-zone patterns (Control Tower) and CI/CD-driven applies with per-environment approvals — nobody applies to prod from a laptop.

The root causes and trade-offs.

Workspaces keep your configuration DRY, but they make it easy to accidentally target the wrong environment. A directory-per-environment layout means more duplication, but which environment you're touching is always explicit and safe. For production, that explicitness wins, and a tool like Terragrunt can reduce the duplication cost of the directory-per-environment approach.

The trap that less-experienced engineers fall into.

Using workspaces to separate prod from non-prod with shared config and credentials — making it easy to run the wrong environment (the near-destroy scenario).

🎯 Interviewer follow-up questions you should expect.

  • "Workspaces or directory-per-env?" Directory-per-environment for prod-critical separation, since it's explicit and isolated; workspaces for ephemeral or similar environments.
  • "How do you handle multi-region?" With provider aliases.
  • "How do you isolate environments?" With separate accounts and state, and CI/CD applying through per-environment approvals.
Say it like this"I lean toward directory-per-environment with separate state and separate accounts over Terraform workspaces for prod-critical separation, because workspaces share one config and it's easy to forget which one you're in — that's the near-destroy scenario. Provider aliases handle multi-region. Applies run through CI with per-environment approvals so no one applies to prod from their laptop. The organising principle is isolation: separate accounts and state so a mistake is contained to one environment."

---

Mark:
15CI/CD for Terraform (GitOps for infra)Scenario

Design the Terraform delivery pipeline — how changes get reviewed and applied safely, without people running apply from laptops.

What is actually happening (the mental model).

Infra changes go through the same rigor as application code, and the key discipline is plan-on-PR (the plan is the diff being reviewed) and apply-the-saved-plan on merge (no gap between reviewed and applied). Tools like Atlantis / Terraform Cloud / Spacelift orchestrate this with locking and a scoped role.

How to reason about it (the pipeline).

  1. On the PR: fmt / validate / tflint / security scan (tfsec/checkov) / cost estimate (Infracost), and post the plan as a PR comment so reviewers see exactly what will change.
  2. On merge: apply the saved plan (plan -out then apply that file) so what runs is exactly what was reviewed — no re-plan drift.
  3. State locking during apply; a least-privilege CI role.
  4. Per-environment approvals; nobody applies to prod from a laptop — the pipeline is the only path.
  5. Atlantis / Terraform Cloud / Spacelift to automate this.

The root causes and trade-offs.

Applying from a laptop is unreviewed, un-audited, and prone to drift, since nobody else sees what actually happened. Applying through a pipeline means the change is reviewed, audited, and run from the exact saved plan. The trade-off is the upfront work of setting up that pipeline, in exchange for real safety and auditability.

The trap that less-experienced engineers fall into.

Applying from laptops with personal credentials (no review, no audit trail), and re-planning at apply time instead of applying the saved plan (so what runs may differ from what was reviewed).

🎯 Interviewer follow-up questions you should expect.

  • "How do you deliver Terraform safely?" You run plan on the pull request as a comment, apply the saved plan on merge, use state locking, and give CI a least-privilege role.
  • "Why apply the saved plan?" So that what runs is exactly what was reviewed, with no drift between review and apply.
  • "What tools automate this?" Atlantis, Terraform Cloud, or Spacelift.
Say it like this"Infra changes go through the same rigor as code. On the PR: format, validate, lint, and security-scan with tfsec or checkov, then post the plan as a comment so reviewers see exactly what will change — the plan is the diff being reviewed. On merge, apply the saved plan so there's no gap between what was approved and what runs. Tools like Atlantis or Terraform Cloud orchestrate this with locking and a scoped CI role. Nobody applies to prod from a laptop; the pipeline is the only path."

---

Mark:
16Policy as code & cost controlScenario

You need to stop people provisioning insecure or expensive infrastructure — public S3 buckets, unencrypted volumes, oversized instances, untagged resources — before it's applied.

What is actually happening (the mental model).

Guardrails belong in the pipeline, shifted left — catch bad infra in the PR, not in a security audit or a surprise bill. Policy-as-code (OPA/Sentinel/Conftest) blocks non-compliant plans, and cost estimation (Infracost) surfaces the dollar impact on the PR so it's a conversation before merge.

How to reason about it.

  1. Policy-as-code gates in CI (OPA/Sentinel/Conftest) — block public S3, unencrypted volumes, 0.0.0.0/0 security groups, oversized instances, missing required tags.
  2. Cost estimation (Infracost) on PRs — "this change adds $3k/month" is a great pre-merge conversation.
  3. Shift left — fail in the PR, not in prod or on the invoice.
  4. Mandatory tagging for cost allocation.

The root causes and trade-offs.

Ungoverned infrastructure lets an insecure or expensive change reach production with nobody catching it. Policy gates catch exactly that kind of change during review, before it ever applies. The trade-off is the upfront work of authoring those policies, in exchange for governance that happens automatically from then on.

The trap that less-experienced engineers fall into.

Relying on manual review or a post-hoc security audit to catch insecure/expensive infra — instead of automated gates that fail the PR.

🎯 Interviewer follow-up questions you should expect.

  • "How do you stop insecure infra being provisioned?" With policy-as-code gates, using OPA or Sentinel, in CI, blocking things like public S3 buckets, unencrypted volumes, and open security groups.
  • "How do you control cost?" With Infracost on the pull request, so the dollar impact is visible before merge, plus mandatory tagging.
  • "What's the principle?" Shift governance left — catch it in review, not in an audit or on the bill.
Say it like this"I put guardrails in the pipeline so bad infra never gets applied: policy-as-code with OPA or Sentinel to block things like public S3 buckets, unencrypted volumes, or 0.0.0.0/0 security groups, plus mandatory tagging for cost allocation. I add Infracost to the PR so reviewers see the dollar impact before merge — 'this change adds $3k a month' is a great conversation to have pre-merge. The theme is shifting governance left: catch it in review, not in a security audit or a surprise bill."

---

Mark:
17Terraform vs alternatives / when NOT to use itScenario

"Would you always use Terraform?" A judgment question testing whether you know Terraform's limits and reach for the right tool per layer.

What is actually happening (the mental model).

Terraform excels at declarative, multi-cloud provisioning, but it's not the right tool for everything — it's weak at imperative sequencing, app configuration, and fast-changing app deploys. Senior means matching the tool to the layer — provisioning, configuration, and app deployment are different problems — rather than cramming everything into Terraform.

How to reason about it.

  • Terraform — declarative provisioning of infrastructure, multi-cloud. Its sweet spot.
  • Not great for: imperative sequencing, app-level configuration (use Ansible), app deployment on Kubernetes (use Helm/GitOps), fast-changing app deploys.
  • Alternatives per context: CloudFormation/CDK for deep AWS-native shops; Pulumi for real programming languages; Crossplane for Kubernetes-native provisioning.
  • Right tool per layer — provisioning vs config vs deploy.

The root causes and trade-offs.

Forcing everything into Terraform, including work it was never designed for, produces brittle, awkward code, since you end up writing imperative logic inside a declarative tool. Matching each tool to the layer it's actually good at produces cleaner code overall, at the cost of needing to know more tools.

The trap that less-experienced engineers fall into.

Using Terraform for app configuration or deployment (imperative, fast-changing work it's bad at), producing brittle, hard-to-maintain code — instead of Ansible/Helm for those layers.

🎯 Interviewer follow-up questions you should expect.

  • "Would you always use Terraform?" No — it's for declarative provisioning; application configuration goes to Ansible, and application deployment goes to Helm or GitOps.
  • "Terraform vs CDK vs Pulumi?" CDK or CloudFormation for deep AWS-native work; Pulumi if the team wants real programming languages; Terraform for declarative multi-cloud.
  • "What's Terraform bad at?" Imperative sequencing, application configuration, and fast-changing application deploys.
Say it like this"Terraform is my default for provisioning, but I don't force it everywhere. It's declarative and weak at imperative sequencing, so app-level configuration belongs in Ansible and app deployment on Kubernetes in Helm or GitOps, not Terraform. For deep AWS-native shops CDK or CloudFormation can fit better, Pulumi if the team wants real languages, Crossplane if everything's Kubernetes-native. Senior means matching the tool to the layer — provisioning, config, and deployment are different problems, and cramming all three into Terraform creates brittle, slow, awkward code."

---

Mark:
🧠 How to turn this into muscle memory
  • Drill the principles first (protect state, read the plan/-/+, state boundaries = blast-radius boundaries, pin everything, everything-through-the-pipeline) — nearly every question is an application of one.
  • Break and recover, in a real project. Corrupt state and restore from a versioned backup; rename a resource and watch it destroy-recreate, then fix it with a moved block; remove a middle count element and see the cascade, then migrate to for_each; import a clickops resource and iterate to a no-op plan. Reproducing each teaches ten times more than reading.
  • One scenario, three passes. Reason it out, then say the spoken version without looking, then explain it to a teammate — especially the state/locking, count-vs-for_each, and drift stories, which come up constantly.
  • Keep a "state decision + why" log — the state-splitting, module-design, and blast-radius decisions you make become the concrete stories you tell in the behavioural round.

Next in the course order: Docker, at this same depth.

No scenarios match that search. Clear search
Terraform & Infrastructure as Code — DevOps Zero → Hero. Theory, Lab & Interview rebuilt from Days 16 & 17 (infrastructure as code, API as code, first project, state, remote backend & modules); the 17 scenario drills are the Scenario Playbook's Terraform domain, rewritten in plain English. ← All chapters