Linux & Shell — DevOps Zero → Hero (Day 6·7·8) · Theory · Lab · Interview

Home›Cloud & OS›Linux & Shell

Day 6 · 7 · 8 — the operating system every server runs on

Linux & Shell Scripting

Almost every server you will ever touch as a DevOps engineer runs Linux, and the way you talk to those servers is the shell. This chapter rebuilds Days 6, 7 and 8 into a workshop: first you understand what Linux and the shell actually are, then you get hands-on with the commands and two real automation projects, then you get quizzed the way an interviewer would quiz you, and finally you face the senior on-call scenarios.

① Theory→② Lab→③ Interview Q&A→④ Scenario drills

Day 6 · Linux OS & the basics of shell scripting Shell · Zero-to-Hero (basics, intermediate, interview Q&A) Day 7 · Live AWS project — the resource tracker Day 8 · Real-time project — GitHub API integration

⚡ The whole of Linux & the shell in one mental model

An operating system is the bridge between the software you want to run and the hardware it runs on. Your application cannot speak directly to the CPU, the memory, or the disk; the operating system sits in the middle and carries the request down to the hardware and the response back up. Linux is the operating system that runs production because it is free, secure, and fast, and the shell is how you talk to it when there is no graphical screen to click on.

The bridge. A request travels from you, through your software, down to the operating system, and finally to the hardware — then the response comes back up the very same path.

Hold on to four ideas and the rest of the chapter follows: (1) the kernel is the heart of Linux — it manages devices, memory, processes, and system calls; (2) the shell is the command-line way you tell Linux what to do, and a shell script is just a file of those commands run in order to automate a task; (3) every file has permissions built from the numbers 4 (read), 2 (write), and 1 (execute); and (4) the real power of the shell comes from joining small commands together with a pipe, so that the output of one becomes the input of the next.

On this page

① Theory — the concepts (Day 6 + the shell series)
What an operating system is, and why Linux runs production
The anatomy of Linux — kernel, libraries, and the shell
What shell scripting is, and the shebang line
File permissions — the 4-2-1 model, users, and sudo
The anatomy of a real script — variables, flags, conditions, loops
② Lab — hands-on, command by command
Moving around and handling files
Reading a machine's health
Your first script, and giving it permission to run
The filtering toolkit — pipe, grep, awk, find, curl
Project 1 — the AWS resource tracker
Project 2 — the GitHub API access lister
③ Interview Q&A — the questions the instructor flags
④ Scenario drills — the senior on-call layer (16)

Phase 1 · Theory

What Linux and the shell actually are

Before you memorise a single command, you need the mental model: what an operating system does, what the pieces of Linux are, what a shell script is for, how permissions work, and what the building blocks of a script look like.

What an operating system is, and why Linux runs production

Before you can understand Linux, you need to understand what any operating system is for. The simplest way to see it is to follow what happens when you run a program.

When you buy a laptop or a server, what you are really buying is hardware: a CPU, some memory (RAM), and storage for input and output, such as a hard disk. On top of that hardware, you want to run software — a game, an application like Jenkins, or a program you are writing in Java or Python. The problem is that your software cannot talk directly to the CPU or the memory. There has to be something in the middle that carries the conversation back and forth, and that something is the operating system.

So the definition is simple: an operating system is the bridge that lets your software and your hardware communicate. When you install an application and it needs to use the CPU or the memory, the request travels from you, to the application, to the operating system, and finally to the hardware; the response then travels back up the very same path. This is why nothing runs without an operating system. Your hardware may be ready and your application may be installed, but until the operating system is there to connect them, nothing happens. When you buy a Dell laptop it usually comes with Windows already installed, a Mac comes with macOS, and some machines come with a version of Linux — the vendor installs an operating system on the hardware before handing it to you.

As a beginner you are probably most familiar with Windows, but once you move into software you find that Linux is used almost everywhere — in production, in staging, and in developer environments. Roughly eight or nine times out of ten, applications are both tested and deployed on Linux. There are three good reasons for this.

Why Linux, specifically:

It is free and open source. Windows is a proprietary operating system provided by Microsoft, whereas Linux is free. Anyone can build a Unix-like (that is, Linux-like) operating system, which is why there are so many free distributions to choose from, such as Ubuntu, CentOS, Debian, and Alpine.
It is very secure. On a Windows laptop you usually have to install antivirus and anti-malware software; on Linux you generally do not have to install anything of the kind, because it is secure by default. It is not perfectly secure — nothing is — but it is secure enough that this is one of the main reasons people trust it in production.
It is fast. Because the operating system is the heart of everything, a slow operating system makes every request slow, no matter how clever your application is. When a user opens Amazon or Netflix, the operating system underneath has to be able to take that request quickly. People prefer an operating system that is fast and that does not crash or slow down, and Linux fits that need.

So when someone asks you the difference between Linux and Windows, or why Linux is used, this is your answer: it is the most widely used operating system for servers because it is free, secure, and fast, and it comes in many distributions you can pick from depending on your needs.

The anatomy of Linux — kernel, libraries, and the shell

The full Linux architecture is genuinely complicated, so the goal here is not to cover every detail but to keep it crisp: the few layers you actually need to picture.

Think of the Linux operating system as a stack of layers. At the very bottom, and at the heart of everything, is the kernel. On top of the kernel sit the system libraries. And on top of those you have the things you interact with more directly: the compilers, the user processes, and the system software. When you install Linux on a machine, these are the fundamental pieces you are installing.

The layers. A user's task passes down from the software, through the system libraries, into the kernel, which is where the real work of talking to the hardware happens.

The kernel is the heart of the operating system because it is the component that actually establishes the communication between your hardware and your software. If somebody asks you in an interview what a kernel is, you can say that it is the heart of any operating system and that it has four primary responsibilities: device management, memory management, process management, and handling system calls. Those four jobs are the core of what a kernel does for any operating system, not just Linux.

Above the kernel are the system libraries. When a user wants to perform a task, that task comes in through the system libraries and is then passed down to the kernel. Different distributions may have slightly different libraries, but the concept is the same everywhere; a well-known example is libc. Finally, at the top layer, you have the compilers, user processes, and system software — for example, when you run a Java or Python program, the operating system has to compile and run that code, and this is the layer that does it. You do not need to memorise the depths of each layer; what matters is understanding the breadth, so that later, when you interview, you can go deep on whichever piece the conversation calls for.

Where the shell fits in.

Now that you can picture Linux, you can see why you need the shell. The shell is the way you talk to the operating system. On Windows you have a rich graphical interface, so to create a file you click through folders with your mouse. But most servers in a software organisation do not have a graphical interface at all — installing one would make the server heavier and slower, so on production systems it is left out. That means when there is a problem on a production server, you cannot navigate through folders with a cursor; you have to do everything through the command line. Those command-line instructions you type are your shell commands, and they are common across distributions, so the same commands work whether you are on CentOS, Debian, Ubuntu, or Fedora.

What shell scripting is, and the shebang line

Shell scripting is really just automation. The clearest way to see why you need it is to imagine a task growing until doing it by hand becomes impossible.

Automation, in any field, is the process of reducing manual, repetitive work. Imagine someone asks you to print the numbers from 1 to 10. You could just type them out. Now imagine they ask for the numbers from 1 to 1,000, and then they keep adding zeros — very quickly it becomes impossible to do by hand. The same is true if someone asks you to create a hundred files on a Linux machine, and then a thousand, and then more. Shell scripting is how you automate these day-to-day activities on a Linux machine, and because a script declares which shell should run it, the same script runs anywhere that shell is available.

A shell script is simply a file — by convention its name ends in .sh, just as a Python file ends in .py — that contains a set of shell commands to be run in order. The very first line of almost every shell script is special, and it looks like this:

bash#!/bin/bash

That first line is called the shebang. Its job is to tell Linux which executable — which shell — should run the rest of the script. Over the history of Linux, several different shells have existed, including bash, ksh, sh, and dash. They are broadly similar, but they differ slightly in syntax; for example, the way you write a for loop in bash is not quite the same as in ksh. The most widely used shell by far is bash, so rather than learning all of them, you learn bash and use it everywhere.

The /bin/sh versus /bin/bash interview question. You will often see two different shebangs in the wild: #!/bin/sh and #!/bin/bash. Historically these behaved the same, because /bin/sh was a link that pointed to /bin/bash, so even when you wrote sh, the request was quietly forwarded to bash. More recently, though, some operating systems — Ubuntu among them — have changed that link so that /bin/sh now points to /bin/dash instead of bash. The consequence is that if you write a bash script but use the #!/bin/sh shebang, and you then share it with a colleague whose machine links sh to dash, your script may fail because of the small syntax differences. The safe rule is to always write the exact shell you mean: #!/bin/bash.

File permissions — the 4-2-1 model, users, and sudo

The moment you try to run a script you have just written, Linux stops you with "permission denied." Understanding why is one of the most important things in this chapter, because Linux security is built around it.

When you create a file in Linux, you are the creator, but Linux still wants to know exactly who is allowed to do what with it before it will let anyone run it. This is because Linux is designed to be very secure. So as soon as you save a script and try to execute it, Linux essentially asks, "who are you, and do you have permission?" — and if you have not granted the right permissions, it refuses. Granting those permissions is done with the chmod command, which stands for "change mode."

Permissions are organised into three categories of people: the owner of the file (the user who created it), the group that owns the file, and everyone else who is logged into the machine. For each of those three categories, you decide what they are allowed to do, using a simple numeric formula:

The 4-2-1 formula. Read is 4, write is 2, execute is 1. You add them up for each of the three categories — owner, group, and everyone else — to get the three digits you pass to chmod.

The magic number is 7, because 4 + 2 + 1 = 7 gives read, write, and execute all at once. So chmod 777 myscript.sh grants full access to the owner, the group, and everyone else — the three digits, in order, are the owner's permissions, the group's permissions, and everyone else's. By contrast, chmod 444 grants read-only to all three (nobody can change or run the file), and chmod 700 gives the owner everything while giving the group and everyone else nothing. In your real work you should grant only the permission that is actually needed rather than reflexively using 777, which opens the file to everyone.

Root and sudo.

Linux has a special super-user called root that can do anything, including deleting files that cannot be recovered, so in an organisation you normally work as your own user and only borrow root's power when you truly need it. To become root you can run sudo su -. Breaking that down: su means "switch user," so su - switches you to root; and sudo — "substitute user do" — is what lets you run a command with root's privileges in the first place. In practice, rather than becoming root for a whole session, you usually stay as your normal user and simply put sudo in front of the one command that needs elevated rights.

The anatomy of a real script — variables, flags, conditions, loops

A one-line script is easy. What turns it into something an organisation relies on is a handful of habits and building blocks that you add on top.

Once you can create and run a file, a professional script layers a few things onto the bare commands. The first is metadata: a block of comments at the top, written with the # symbol, that records who wrote the script, when, what it does, and which version it is. Anything after a # is a comment, which the shell ignores when it runs the file; the point is that nobody can tell what a file called addition.sh actually does just from its name, so you spell it out. Comments are not optional in shared code — they let a colleague in a hurry understand your script at a glance.

The second habit is a set of flags you set at the very top of the script to control how it behaves. These come up constantly in real scripts and in interviews:

set -x puts the script into debug mode, so that before each command runs, Linux prints the command it is about to execute along with its output. This is the standard way to debug a shell script.
set -e makes the script exit as soon as any command fails. Without it, a script whose first step failed would keep running every remaining line, which is pointless and dangerous — imagine a script that is meant to create a user, then create a file, then add the user to the file; if the first step fails but the rest run, you end up with an empty file and no user.
set -o pipefail covers a gap in set -e: by itself, set -e only checks the last command in a pipe, so an earlier command in the pipe could fail silently. Turning on pipefail makes the script fail if any command in the pipe fails.

You can combine them as set -exo pipefail, though keeping them on separate lines makes it easier to comment one out later. On top of these, a script uses variables (for example a=4, with no spaces around the =, read back as $a), the echo command to print output, and two structures that behave the same in every programming language even though their syntax differs — the if/else condition and the for loop.

bash#!/bin/bash
# if / else — note the spaces inside [ ], and 'fi' closes the block
a=4
b=10
if [ $a -gt $b ]
then
  echo "a is greater than b"
else
  echo "b is greater than a"
fi

# for loop — 'do' the action for each value, 'done' closes the loop
for i in 1 2 3 4 5
do
  echo $i
done

The if block is closed by writing if backwards, as fi, and the for loop is closed with done. Bash also has while, until, and select loops, but the for loop is the one you reach for most often. You do not need to memorise every syntax detail — the important thing is to understand what each structure is for, and to practise them until the syntax comes back to you when you need it.

Phase 2 · Lab

Hands-on — from your first command to two real projects

Now you actually drive a Linux box: move around and handle files, read a machine's health, write and run your first script, learn the filtering toolkit every DevOps engineer leans on, and then build the two real automation projects from Days 7 and 8.

Moving around and handling files

You log into a server over SSH (as covered in the AWS chapter, using your key: ssh -i key.pem ubuntu@<public-ip>). Now you are on a box with no mouse and no folders to click, so every navigation and file action is a command. Start with the three you will type most:

bashpwd                 # present working directory — where am I right now?
ls                  # list the files and folders here
cd bundle           # change directory into 'bundle'
cd ..               # go back up one directory
cd ../..            # go back up two directories
ls -ltr             # long listing: permissions, owner, size, timestamp; d = directory

Whenever you log into a server, the habit is to run pwd first to see where you are, then ls to see what is around you. The longer form ls -ltr lists everything with its details — whether each entry is a file or a directory (a directory's line starts with d), who owns it, which group owns it, its size, its timestamp, and its permissions. Once you can move around, you create and read files:

bashtouch abhishek      # create an empty file called 'abhishek'
vi test             # create AND open a file to edit it
                    #   press i to enter insert mode and type
                    #   press Esc, then :wq to save and quit  (or :q to quit without saving)
cat test            # print the file's contents without opening it
mkdir myfolder      # make a directory
rm abhishek         # remove a file
rm -r myfolder      # remove a directory (and everything in it)
man ls              # the manual for any command — your go-to when you forget
history             # every command you have typed

There are two ways to create a file, and the difference matters. touch only creates the file, whereas vi (or vim, if it is installed) both creates and opens it. You might wonder why you would ever use touch when vi does more — the answer is automation. Inside a script that needs to create a thousand files, you cannot use vi, because it would try to open a thousand files at once, and just as playing a thousand movies at once would crash your machine, a program cannot keep that many files open. So touch is the tool for creating files in scripts, and vi is the tool for editing one file by hand. If you ever forget what a command does or which option to use, man is your manual — man ls shows you everything about ls.

Reading a machine's health

A huge part of a DevOps engineer's job is logging into a machine that is misbehaving and finding out which resource is under strain. Just as you check a new phone's RAM, CPU, and storage, you check a server's — but you do it with commands. There are three focused commands and one that shows everything at once:

bashfree -m             # memory: total, used, and free (in MB)
nproc               # number of CPUs on this machine
df -h               # disk usage per mount, human-readable
top                 # everything at once: live CPU, memory, and per-process usage
ps -ef              # every running process, in full detail

Use free to see the memory, nproc to count the CPUs (a small free-tier EC2 instance will report just one), and df -h to see how full the disk is. When you want the whole picture in one place, top shows live CPU and memory along with every running process and how much each one is consuming — which is why "how do you monitor the health of a node?" is such a common interview question, and why the short answer is "the top command." The last command, ps -ef, lists every process running on the machine in full format. It becomes far more powerful once you learn to filter its output, which is the next lesson.

Why this matters in real life. Imagine a DevOps engineer named John who is responsible for ten thousand Linux virtual machines. Every time a developer reports that one of them is slow or out of memory, John could log in and run these commands by hand — or he could write one shell script that logs into a machine, checks its CPU, memory, and processes, and reports back. Better still, he can schedule that script to run automatically across all ten thousand machines and email him a summary: "I checked all your machines, and ten look suspicious — five are low on memory and five are high on CPU." That is exactly the kind of automation shell scripting exists for, and it is why it is a must-learn skill for DevOps.

Your first script, and giving it permission to run

Now put it together into a real script. Create and open a file with vi first.sh, press i to insert, and write a script that starts with the shebang, records who wrote it, and does a small task:

bash#!/bin/bash
###############################################
# Author : Abhishek
# Date   : 1st December
# Purpose: create a folder and two files inside it
# Version: v1
###############################################
set -x                          # run in debug mode (optional while developing)

mkdir abhishek                  # create a folder
cd abhishek                     # go into it
touch first-file second-file    # create two files

Save with Esc then :wq, and try to run it. You have two ways to execute a script — put sh in front of it, or run it directly with ./ — but the first time you try, Linux will stop you:

bash./first.sh                       # Permission denied — Linux won't run it yet
chmod 777 first.sh               # grant permissions (see T4 on the 4-2-1 model)
./first.sh                       # now it runs
sh first.sh                      # the other way to run it

The "permission denied" message is not a bug — it is Linux insisting that you explicitly say who may run the file before it will run it, which is the permission model from T4 in action. Once you run chmod, the script executes: it creates the folder, moves into it, and creates the two files. (In real work you would grant only the permission actually required rather than 777.) While you are developing, the set -x line prints each command and its output so you can watch exactly what the script is doing, and you can comment it out once the script works.

The filtering toolkit — pipe, grep, awk, find, curl

This is where the shell becomes genuinely powerful. The idea is to join small commands together so that the output of one becomes the input of the next, and then filter that output down to exactly what you need.

The heart of it is the pipe, written |, which sends the output of the command on its left into the command on its right. On its own, ps -ef lists every process; but piped into grep, which filters lines, you get only the ones you care about; and piped again into awk, which can pull out a specific column, you get only the process IDs:

The pipe chain. Each | hands the output of one command to the next: list every process, keep only the Amazon ones, then print just the second column — the process ID.

bashps -ef | grep amazon              # only the processes belonging to Amazon
ps -ef | grep amazon | awk '{print $2}'   # just the process IDs (column 2)
cat app.log | grep error          # pull the error lines out of a log file
find / -name pam.d                # search the whole filesystem for a file by name
curl https://example.com/app.log  # fetch a file/URL from the internet (like Postman)
wget https://example.com/app.log  # download that file to disk instead

A few of these deserve a note. awk is a small pattern-processing language; the form awk '{print $2}' treats each line as columns separated by spaces and prints the second one, which is exactly how you turn a full process listing into a clean list of process IDs. find searches the filesystem — find / -name pam.d looks everywhere (the / means "start from the root"), and you usually prefix it with sudo so it can look inside protected folders. And curl retrieves information from the internet, much like Postman or Python's requests module; it is how you pull a log file that has been uploaded to storage, or make an API call, straight from a script.

curl versus wget, and one pipe gotcha. The difference between curl and wget is that curl hands you the output directly, whereas wget downloads the file and saves it to disk — so to search a remote log you can either curl <url> | grep error in one step, or wget it and then cat and grep the downloaded file in two. One more subtlety: a pipe only works when the command on the right actually reads its input. Running date | echo "today is" prints only "today is" and never the date, because echo ignores what is piped into it and simply prints its own arguments.

Project 1 — the AWS resource tracker (Day 7)

This is a project real DevOps engineers genuinely use, and it is one you can put on your resume. The goal is to report which AWS resources an account is using, so that nobody is quietly paying for things they forgot about.

Organisations move to the cloud for two reasons: to avoid the overhead of maintaining their own servers, and to save money by paying only for what they use. But that second promise only holds if someone tracks usage — if a developer spins up a hundred EC2 instances nobody uses, or leaves EBS volumes attached to nothing, AWS still charges for them. So a core DevOps responsibility is to track resource usage, and one simple way to do that is a shell script that combines what you already know about the shell with the AWS CLI. The prerequisite is that the AWS CLI is installed and authenticated with aws configure (which asks for your access key, secret key, region, and output format).

bash#!/bin/bash
###############################################
# Author : Abhishek   Date: 11th Jan   Version: v1
# Purpose: report AWS resource usage (S3, EC2, Lambda, IAM)
###############################################
set -x                                    # debug mode: show each command and its output

echo "Print list of S3 buckets"
aws s3 ls

echo "Print list of EC2 instances"
aws ec2 describe-instances | jq '.Reservations[].Instances[].InstanceId'

echo "Print list of Lambda functions"
aws lambda list-functions

echo "Print list of IAM users"
aws iam list-users

If you do not know a particular AWS CLI command, the CLI reference documentation lists them all — that is where you find that buckets are listed with aws s3 ls, instances with aws ec2 describe-instances, and so on. Two touches make the output usable. The echo lines label each section so the reader knows which output belongs to which resource, and jq — a JSON parser — trims the enormous JSON that describe-instances returns down to just the instance IDs a manager actually wants. (Its sibling yq does the same for YAML; as a DevOps engineer you deal with JSON and YAML constantly, so both are worth knowing.)

Making it run every day by itself. A report is only useful if it is produced on time, and you might not always be available at 6 pm to run it. The fix is a cron job: you redirect the script's output into a file (./aws-resource-tracker.sh > resource-tracker), and then schedule the script with crontab so that a Linux process runs it automatically at the same time every day — exactly the way you schedule a video to publish at 7 pm without being there to click the button.

Project 2 — the GitHub API access lister (Day 8)

The second project talks to another system over its API. The task is one DevOps engineers do all the time: list who has access to a GitHub repository — for example, to revoke access the day someone leaves — without clicking through the web interface every time.

There are two ways to talk to almost any application, including GitHub, Jenkins, GitLab, or Kubernetes: through its CLI (like kubectl) or through its API. An API — an application interface — lets you get information programmatically instead of through a browser. As a DevOps engineer you do not write these APIs; the developers who build the application write them and publish reference documentation. Your job is to consume them, using curl from a shell script (or the requests module from Python). You look up the endpoint you need in the API docs — for GitHub, listing a repository's collaborators is a URL of the shape https://api.github.com/repos/{owner}/{repo}/... — and then you call it.

bash#!/bin/bash
# list-users.sh — list who has access to a GitHub repo
# Usage: export the two secrets, then pass owner and repo as arguments
#   export username="your-github-username"
#   export token="your-personal-access-token"
#   ./list-users.sh <repo_owner> <repo_name>

API_URL="https://api.github.com"
REPO_OWNER=$1                     # first command-line argument
REPO_NAME=$2                      # second command-line argument

# call the GitHub API and use jq to keep only collaborators with read access
collaborators=$(curl -s -u "${username}:${token}" \
  "${API_URL}/repos/${REPO_OWNER}/${REPO_NAME}/collaborators" \
  | jq -r '.[] | select(.permissions.pull == true) | .login')

if [ -z "$collaborators" ]; then
  echo "No users have read access to ${REPO_OWNER}/${REPO_NAME}."
else
  echo "Users with read access to ${REPO_OWNER}/${REPO_NAME}:"
  echo "$collaborators"
fi

Several ideas from the whole chapter come together here. The username and token are read from the environment with export, never hardcoded, because both are sensitive — the token is a GitHub personal access token (created under Settings → Developer settings) that stands in for your password when a script talks to the API, and anyone who gets it could act as you. The owner and repository are passed as command-line arguments, read inside the script as $1 and $2, so the same script works for any repository. curl calls the API, which returns JSON, and jq filters that JSON to the users whose permissions.pull is true, printing just their login names. Finally, an if/else reports either that nobody has access or the list of people who do. (The script needs jq installed — sudo apt install jq -y.)

Two senior touches to add. First, a proper metadata block at the top explaining what the script does, what inputs it needs, and whom to contact. Second, a helper function that checks the number of arguments the user supplied ($#) against the number expected, and prints a usage message if they got it wrong — called at the very start of the script so it fails fast with a clear message instead of a confusing error. To run the check across a hundred repositories, you would wrap the call in a for loop over your list of repository names.

Phase 3 · Interview Q&A

The Linux & shell questions you'll be asked

These are the questions the instructor explicitly flags across the shell-scripting series — gathered, in his words, from real interviews. Answer each out loud, then reveal.

Q1List some of the most commonly used shell commands.reveal ▸

Answer

Be honest and name the commands you genuinely use every day: ls to list files, cp and mv to copy and move, mkdir to create directories, touch to create files, vi/vim to open them, grep to filter, find to locate files, and top or df to check a machine's health. The trap is reeling off netcat, route, or traceroute — those are debugging commands you only use when something is wrong, so leading with them signals that you do not work on Linux day to day. Start with the everyday commands, then mention the advanced ones as extras.

Q2What is the difference between /bin/sh and /bin/bash (and dash)?reveal ▸

Answer

Historically they were the same, because /bin/sh was a link that pointed to /bin/bash, so a script written with the sh shebang was quietly forwarded to bash. That is no longer guaranteed: some operating systems, such as Ubuntu, now link /bin/sh to /bin/dash instead. Because bash and dash differ slightly in syntax, a bash script run through sh on such a machine can fail unexpectedly. The safe rule is to always write the exact shell you mean — #!/bin/bash — so your script behaves the same everywhere.

Q3Write a simple shell script to list all processes (and just their process IDs).reveal ▸

Answer

To list every process, use ps -ef, which prints all processes in full detail. If the interviewer then says "I only want the process IDs," pipe that into awk to pull out the second column: ps -ef | awk '{print $2}'. You can narrow to a specific application first with grep — for example ps -ef | grep amazon | awk '{print $2}' gives just the process IDs of the Amazon processes.

Q4Write a script to print only the errors from a remote log file.reveal ▸

Answer

The log lives on a remote server — perhaps in S3 or Google storage — so you fetch it with curl, then filter it with grep, joining the two with a pipe: curl <log-url> | grep error. Three commands are doing the work: curl retrieves the whole file, grep keeps only the lines you want (change error to trace for trace-level lines), and the pipe joins them by sending curl's output into grep. Without the pipe, the second command would not receive the first command's output at all.

Q5What is the output of date | echo "today is"?reveal ▸

Answer

It prints only "today is" — the date never appears. A pipe sends the first command's output to the second command's input, but echo does not read its input at all; it simply prints whatever arguments it was given. So the output of date has nowhere to go, and only echo's own text is printed. The pipe only produces a useful result when the command on the right actually reads what is piped into it (as grep and awk do).

Q6What is the difference between curl and wget?reveal ▸

Answer

curl retrieves the content and hands it straight back to you, so you can pipe it onward in a single step — for example curl <url> | grep error. wget instead downloads the file and saves it to disk, so you then work on the saved file in a second step — wget <url>, then cat and grep it. Use curl when you just want the output; use wget when you actually want the file saved locally.

Q7Write a script to print numbers divisible by 3 and 5 but not by 15.reveal ▸

Answer

First break the question down out loud: the number must be divisible by 3 or by 5, and it must not be divisible by 15 (which is 3 × 5). Then ask the interviewer for the range. A for loop over the range, with an if using the modulo operator, does it:

bashfor i in {1..100}
do
  if [ $((i % 3)) -eq 0 -o $((i % 5)) -eq 0 ] && [ $((i % 15)) -ne 0 ]
  then
    echo $i
  fi
done

The same shape answers the whole family of these questions — even numbers, odd numbers, primes — by changing the condition.

Q8Write a script to count the number of "S"s in "Mississippi".reveal ▸

Answer

Say up front that this is a simple command, and that you can optimise it if asked — but the easy way is grep with -o (which prints each match on its own line) piped into wc -l (which counts lines):

bashx="Mississippi"
echo $x | grep -o "s" | wc -l

grep -o emits one line per matched "s", and wc -l counts those lines, giving the total. For "Mississippi" that is 4; for "Singapore" it is 1.

Q9How do you debug a shell script?reveal ▸

Answer

Add set -x near the top of the script (or run it as bash -x script.sh). That puts the script into debug mode, so before each command runs, it prints the command it is about to execute along with the resulting output, letting you see exactly where things go wrong.

Q10What is crontab? Give an example.reveal ▸

Answer

crontab is how you schedule a command or script to run automatically at a set time, like an alarm. For example, if a Linux administrator has to send a node-health report every day at 6 pm, rather than logging in and running the script by hand, they set a cron job for 6 pm and Linux runs the script for them and produces the output. So whenever a task must run on a fixed schedule, the answer is a cron job.

Q11How do you open a file in read-only mode?reveal ▸

Answer

Use vi -R <file> (or view <file>). The -R flag opens the file read-only, so you can look at its contents without any risk of accidentally editing and saving over it.

Q12What is the difference between a soft link and a hard link?reveal ▸

Answer

Both are created with ln, but they behave differently. A hard link is effectively a mirror copy of a file's data, so if the original file is deleted, the hard link still holds the content — useful for keeping a backup of a sensitive file. A soft link (symbolic link) is more like a pointer or alias to another file — the classic example is python pointing to python3, so that running python is redirected to python3. Explain each with an example, since that is what interviewers want to hear.

Q13What is the difference between break and continue in a loop?reveal ▸

Answer

The words say it. break stops the loop entirely — for example, when looping through a hundred student names to find "Abhishek", you break as soon as you find the name, so you do not process the rest. continue skips just the current iteration and moves on to the next — for example, when printing numbers divisible by 3 or 5, you continue past any number that is also divisible by 15, ignoring it but carrying on with the loop.

Q14What are the disadvantages of shell scripting?reveal ▸

Answer

Come up with your own answer from real experience rather than a canned list. A good one is that shell is not statically typed: you can leave variables undeclared or unused and — unless you turn on set -u — the shell will not complain, so mistakes slip through that a statically typed language would catch at compile time. Explaining a practical drawback like this, from your own use, lands far better than a memorised list.

Q15What are the different kinds of loops, and when do you use them?reveal ▸

Answer

Like any language, shell has several loops — the for loop, the while loop, and the until (do-while style) loop — and you pick the one that fits. You reach for a for loop most often, when you know the set of values to iterate over (a range of numbers, a list of names). You use a while loop when you want to keep going as long as a condition holds, and until when you want to keep going until a condition becomes true.

Q16Is bash dynamically or statically typed?reveal ▸

Answer

Bash is dynamically typed. You can write x=5 and then x="a string" without ever declaring a type, and the shell accepts both. A statically typed language like Go would make you declare that x is a string, and would throw an error if you then gave it an integer; scripting languages such as bash and Python do not, because their nature is dynamic.

Q17What networking troubleshooting tools do you use?reveal ▸

Answer

A good one to name is traceroute. Running traceroute google.com shows every hop a request takes on its way — from your laptop to your router, to your internet service provider, and onward — and how long each hop takes, which is how you see where a slow network is losing time. There is also tracepath, which does much the same job and does not require root privileges.

Q18How would you sort a list of names in a file?reveal ▸

Answer

Use the built-in sort command — sort <file>. Say the easy way first; you can add that you could improve the efficiency or discuss time complexity if you were writing your own sort, but Linux already provides sort natively, so there is no need to reinvent it.

Q19How will you manage the logs of a system that generates huge log files every day?reveal ▸

Answer

Use logrotate. A busy, customer-facing application emits enormous numbers of logs, and if you keep them all, the disk fills up. With logrotate you define how often to rotate — say, once every 24 hours — compress the rotated file (gzip, zip, or tar), and then delete it after a retention period such as 30 days. That keeps the logs manageable without letting them consume the disk.

Q20How do you monitor the health of a node?reveal ▸

Answer

The quickest answer is the top command, which shows CPU, memory, and per-process usage all in one place. If you want the individual pieces, use free for memory, nproc for the number of CPUs, and df -h for disk usage. And for anything beyond the basics, you can write a custom shell script that gathers exactly the parameters you care about — which is the node-health script this chapter builds toward.

Phase 4 · Scenario drills

The senior on-call layer — Linux under real pressure

You can now navigate Linux, write scripts, and automate real tasks. These 16 scenarios from the Scenario Playbook are the on-call and interview version: an unhealthy box, a full disk, a process that will not die, a server that will not come back after a reboot. Answer out loud, reveal, and mark yourself.

🐧

Linux

16 scenarios · 1–16

The complete Linux troubleshooting scenario set, worked through in depth. This is the thinking and the story behind each one, not commands to memorise. The goal is that you can walk up to any unhealthy box and methodically find the saturated resource or the root cause — and narrate it in an interview like someone who has been on-call for production Linux for years.

⚡ The universal Linux triage reflex — top-down by resource

Go top-down through CPU, memory, disk, network. Load average is not the same as CPU; %wa points to disk; always ask "what changed?".

Almost every "the box is unhealthy" problem is solved by going top-down through the four resources — CPU, memory, disk, and network — instead of guessing. This is a version of the USE method, where for each resource you look at its Utilisation, its Saturation, and its Errors. Everything below is a specialisation of it, so burn this in first:

textuptime                       # load average vs core count — is there REAL pressure?
top   (or htop)              # split the story: %us (user CPU) vs %sy (kernel) vs %wa (I/O-wait) vs memory
vmstat 1                     # run-queue (r), swap in/out (si/so), io-wait (wa) at a glance
free -m                      # memory + swap headroom
df -h                        # which mount is full
dmesg -T | tail -50          # kernel-level TRUTH: OOM kills, disk errors, segfaults, link flaps
journalctl -xe               # recent service/system errors

The mental model: a slow or broken box almost always means that one specific resource is saturated — the CPU is pinned, memory is exhausted, the disk is full or its I/O is saturated, or the network is dropping packets. Your entire job is to identify which one before you change anything. Guessing and restarting is the junior move; naming the saturated resource with evidence is the senior one.

Two habits separate senior engineers:

Read the load average relative to the core count. A load of 8 on an 8-core box is fully utilised but fine; a load of 8 on 2 cores is on fire. Always run nproc first.
Always ask "what changed?" — a deploy, a cron job firing this minute, a patch, or a traffic spike. In a running system, "it was fine an hour ago" means there is a specific recent trigger.

Going from top to dmesg and journalctl is where roughly 70% of the answers live.

---

📋 The full scenario inventory (distinct — no padding)

A. Resource saturation triage

"The server is slow" (no other information)
High load average but the CPU looks idle
A process is pinning the CPU
A process was OOM-killed

B. Disk & storage

Disk is 100% full
"Too many open files" (EMFILE)

C. Processes & scheduling

A cron job silently isn't running
Zombie / defunct processes piling up

D. Access & permissions

Can't SSH into a server
File permissions / ownership broke an app (and SELinux)

E. Change & forensics

"Something changed and broke prod" — who/what?

F. Networking from the host

Intermittent packet loss / latency from a host
Clock skew causing weird failures

G. Boot, tuning & logs

Server won't come back after a reboot
Kernel / sysctl tuning for a workload
Analysing logs at scale from the CLI

---

1"The server is slow" (no other information)Scenario▸

A vague, high-pressure report: "the server's slow", "the app is laggy", users complaining, and no detail. This is the most common real starting point, and how you begin is what marks you as senior.

What is actually happening (the mental model).

"Slow" is a symptom, not a cause. Something is saturated — one of CPU, memory, disk I/O, or network — and everything else is downstream of that. The senior instinct is to resist forming a theory and instead go top-down by resource, letting each layer's numbers tell you where to descend next. A less-experienced engineer guesses ("it must be the database") and starts restarting things; a senior engineer narrows down to the saturated resource in a couple of minutes with a methodical sweep.

How to work through it.

uptime (and nproc) — is the load actually high for this core count? If the load is close to the number of cores, the box is busy but perhaps fine; if the load is far above the cores, there is a queue.
top (press 1 for per-core, M to sort by memory, P by CPU) — read the split:
- high %us means the application is burning user CPU (a busy or looping app).
- high %sy means it is kernel-heavy or syscall-heavy (context switches, network, lots of small I/O).
- high %wa means I/O wait: the CPU is idle, waiting on disk, so you descend to iostat.
- low CPU but high load means processes are blocked in D-state (uninterruptible sleep), usually on I/O, which is scenario 2.
Follow the signal: iostat -xz 1 for disk (%util, await), free -m for memory and swap, ss -s or ss -tan for sockets and connection pileups, and vmstat 1 for the run-queue and swapping.
Correlate with change: was there a recent deploy? A cron job firing this minute (grep CRON /var/log/syslog)? A traffic spike? A patch or reboot?

Root causes, and how to tell them apart.

The split shown by top localises it: high %us points to application CPU, high %wa points to disk I/O, low free memory together with swap activity points to memory pressure, and socket pileups point to a network or connection issue. Each one tells you which tool to reach for next, and the question "what changed?" usually names the trigger.

How to fix it, verify it, and prevent it.

Address the specific resource that is saturated. Prevent a recurrence with monitoring and alerting on all four resources, so that next time "slow" arrives with data rather than as a blank report, and so that you are paged before users notice.

The trap that less-experienced engineers fall into.

Randomly restarting services or "throwing more resources at it" before naming the saturated resource, which either masks the problem or makes it worse, and teaches you nothing.

🎯 Interviewer follow-up questions you should expect.

"Walk me through diagnosing a slow server." You go top-down: uptime for load versus cores, then top for the CPU, memory, and I/O-wait split, then you follow the saturated resource and correlate it with what changed.
"High %wa — what does it mean and where next?" The CPU is idle, waiting on the disk — you go to iostat for disk saturation and await.
"What's your first question beyond the metrics?" What changed — a deploy, a cron job, traffic, or a patch.

Say it like this"I resist guessing and go top-down. uptime tells me if load is even real relative to core count. top splits the story into user CPU, kernel, I/O-wait, and memory — high %wa means the CPU's idle waiting on disk, so I go to iostat; low CPU with high load means processes blocked on I/O. I follow whichever resource is saturated, and in parallel I always ask what changed, because 'slow' is nearly always one specific recently-changed thing hitting one specific resource. Naming that resource with evidence beats restarting things and hoping."

---

Mark:

2High load average but the CPU looks idleScenario▸

uptime shows a load average of, say, 30 on an 8-core box, but top shows %us and %sy low and plenty of idle CPU. It looks contradictory — high load, bored CPU.

What is actually happening (the mental model).

This is the classic "you don't actually understand load average" test. Load average is not CPU utilisation. On Linux it counts processes that are runnable plus those in uninterruptible sleep (D-state) — processes blocked in the kernel, almost always waiting on I/O (disk, or a hung network mount). So a high load with an idle CPU means the box is not compute-bound at all; it is full of processes stuck waiting, typically on storage.

How to work through it.

Confirm the paradox: uptime shows a high load, while top shows low %us and %sy (and possibly high %wa).
Find the blocked processes: ps -eo pid,state,cmd | grep " D" — the D-state processes are what is driving the load.
Find what they are blocked on: iostat -xz 1 (is the disk at 100% util, or is await huge?), check network and NFS mounts (a hung NFS server spikes the load instantly as everything touching it wedges in D-state), and dmesg for disk errors.

Root causes, and how to tell them apart.

The disk is saturated or failing — iostat shows 100% utilisation or a huge await, so you find the I/O hog and fix or replace the storage.
A hung network mount (an NFS or EBS stall) — processes wedge in D-state forever, waiting on a mount that has gone away. You can tell it apart because the D-state processes are all touching the same mount, and df or mount may hang too; the fix is to recover the mount.
A genuinely huge run-queue of runnable threads — that one really is CPU-bound, and %us would actually be high, so it is not this scenario.

How to fix it, verify it, and prevent it.

Fix the storage or the mount. Prevent it by monitoring disk I/O (utilisation and await) and mount health, and by using mount options (such as soft and intr for NFS) so that a dead server cannot wedge every process indefinitely.

The trap that less-experienced engineers fall into.

Seeing a high load and assuming the box is CPU-bound — trying to reduce CPU work or add cores, when the CPU is idle and the real problem is blocked I/O.

🎯 Interviewer follow-up questions you should expect.

"Load is 30 on 8 cores but CPU is idle — how?" Load includes uninterruptible-sleep, D-state, processes blocked on I/O, not just CPU.
"What's a D-state process?" A process blocked in the kernel on I/O, un-killable until the I/O completes — a hung NFS mount is the classic cause.
"How do you confirm it's disk?" You use iostat for %util and await, and ps for the D-state processes.

Say it like this"Load average isn't CPU utilisation — on Linux it includes processes in uninterruptible sleep, which are almost always blocked on I/O. So high load with an idle CPU tells me the box isn't compute-bound; it's full of processes stuck waiting, usually on storage. I look for D-state processes with ps, then iostat for disk saturation and await, and I check for a hung NFS or network mount — a disappearing NFS server is a classic cause of a load spike with a completely bored CPU, because everything touching that mount wedges in D-state."

---

Mark:

3A process is pinning the CPUScenario▸

top shows one process at 100% of a core (or several cores), and the box is hot or a service is degraded.

What is actually happening (the mental model).

A process at 100% CPU is not automatically a bug — it might be legitimate heavy work. The senior distinction is to understand what it is doing (is it a tight loop, garbage-collection thrash, or a real workload?) and, crucially, to capture evidence before you kill it, because kill -9 destroys the only proof and the problem just recurs in ten minutes.

How to work through it.

Identify it: use top and note the PID (press P to sort by CPU); use top -H -p <pid> or pidstat -t 1 to see which thread if it is multi-threaded.
See what it is doing:
- strace -p <pid> -f -c — is it storming syscalls (a spin loop, repeated EAGAIN retries, or polling)? The -c summary shows which syscall dominates.
- Use language-level profilers: py-spy dump or py-spy top for Python, jstack or async-profiler for the JVM, and perf top -p <pid> for native code, to get a flamegraph of where the CPU time goes.
Judge it: a tight loop, a stuck retry, or GC pressure is a bug, whereas a process that is genuinely busy under load is a capacity or algorithm question, not a rogue process.

Root causes, and how to tell them apart.

It may be a busy-loop or an EAGAIN retry storm (strace shows one syscall hammering away), GC thrash (a JVM or runtime profiler shows GC dominating, often paired with memory pressure), an inefficient algorithm under real load (the profiler shows real work), or legitimate expected load (in which case it is a scaling decision). The profiler or strace tells you which.

How to fix it, verify it, and prevent it.

Fix the specific cause — the bug, or scaling out for legitimate load. Prevent it with CPU monitoring and alerting, and by profiling in staging under load so that hot paths are found before production.

The trap that less-experienced engineers fall into.

Blindly running kill -9 on production before capturing a stack or flamegraph — you lose the only evidence, you look like you "fixed" it, and it recurs because the root cause is untouched.

🎯 Interviewer follow-up questions you should expect.

"A process pins a core — what do you do before killing it?" You capture evidence first — strace for syscall storms, and a profiler like py-spy, jstack, or perf for where the CPU is going.
"How do you tell a bug from legitimate load?" You profile it — a tight loop or GC thrash versus real work; if it's real load, it's a scaling decision, not a rogue process.

Say it like this"One process at 100% isn't automatically a bug — it might be real work. So I identify the PID, then profile rather than kill. strace -c shows if it's storming syscalls in a spin loop, and a language profiler like py-spy, jstack, or perf shows exactly where the CPU time goes. I capture that stack or flamegraph before I touch it, because killing it destroys the evidence and it'll just recur. Then I decide whether it's a bug to fix or genuine load to scale for — but understanding first, silencing second."

---

Mark:

4A process was OOM-killedScenario▸

A critical process suddenly vanished; the app is down; dmesg shows Out of memory: Killed process 1234 (java) and the kernel picking a victim. Sometimes it presents as a service that "randomly restarts" under load.

What is actually happening (the mental model).

When the machine runs out of memory, the kernel's OOM killer steps in to save the system: it scores every process (roughly by memory footprint, adjustable with oom_score_adj) and sends SIGKILL to the highest-scoring victim (exit code 137). It is not random corruption — the box was genuinely out of memory and the kernel triaged. The senior question is whether the app was leaking, whether the box was under-provisioned, or whether a noisy neighbour ate the memory and got an innocent process killed.

How to work through it.

Confirm it: dmesg -T | grep -i -E "oom|killed process" or journalctl -k | grep -i oom — this shows the killed PID, its memory usage, and its score.
Establish the pattern: was it the app that got killed, or an innocent bystander while something else hogged the memory? Check free -m now, plus historical memory from monitoring if you have it.
Decide the fix from the shape of the memory over time:
- a genuine leak (memory climbs without bound) needs an app fix, since more RAM only delays it.
- an under-provisioned box (the legitimate working set exceeds what is available) needs a right-sized instance or more memory, or swap as a shock-absorber (which is not a real fix).
- a noisy neighbour needs cgroup or systemd memory limits so that one process cannot starve the host, and you protect critical processes with a lower oom_score_adj.

Runtime gotcha (a strong senior signal).

Runtimes such as the JVM and Node that size their heap to the host's memory rather than to their cgroup limit will allocate past a container's limit and get OOM-killed. That is a heap-configuration fix (-XX:MaxRAMPercentage, or cgroup-aware settings), not "add RAM."

Root causes, and how to tell them apart.

It is a leak (an unbounded climb), under-provisioning (a stable but too-high working set), a noisy neighbour (another process spiked and an innocent one was killed), or a cgroup-unaware runtime heap. The memory graph over time distinguishes a leak from the rest.

How to fix it, verify it, and prevent it.

Fix the leak, right-size the box, or add limits. Prevent it with memory alerting before an OOM happens, cgroup limits to contain the blast radius, and load-testing to learn a service's real working set.

The trap that less-experienced engineers fall into.

Blindly adding memory whenever something is OOM-killed, which just delays a leak's crash and does nothing for a cgroup-unaware heap.

🎯 Interviewer follow-up questions you should expect.

"How do you confirm and diagnose an OOM kill?" You use dmesg or journalctl -k for the OOM message, the PID, and the score, then work out whether it's a leak, an under-provision, or a noisy neighbour.
"Leak vs needs-more-memory — how do you tell?" From the memory graph — an unbounded climb is a leak, while stable-but-high is under-provisioned.
"A JVM OOMs well below expected — why?" Because the heap is sized to the host, not the cgroup limit — you need to make it container-aware.
"How do you stop one process taking down the host?" With cgroup or systemd memory limits, plus oom_score_adj set on critical processes.

Say it like this"dmesg confirms the OOM kill, the victim PID, and its score. Then I decide the shape of the problem: is the app genuinely leaking, is the box under-provisioned, or did a noisy neighbour eat the memory and get an innocent process killed? A leak needs an app fix — more memory only delays the crash. I contain blast radius with cgroup or systemd memory limits so one process can't take down the host, and protect critical processes with oom_score_adj. And I check the runtime heap is cgroup-aware, because a JVM or Node process sized to the host instead of the container limit will OOM no matter what."

---

Mark:

5Disk is 100% fullScenario▸

df -h shows a mount at 100%, and the app is failing on writes — can't log, can't write temp files, database errors. Everything that needs to write is breaking.

What is actually happening (the mental model).

The disk space is being held somewhere, and there are two twists that catch people out. First, inodes can be exhausted independently of space, so millions of tiny files can give you "disk full" while there are still bytes free. Second, there is the notorious deleted-but-open file trap: a process holding a file descriptor to a deleted file keeps its space allocated, so df shows the disk as full while du does not add up.

How to work through it.

df -h — which mount is full? Also run df -i — are the inodes exhausted (a directory with millions of tiny files)? That looks like "full" with space to spare.
Drill down: du -xhd1 /var | sort -h, then descend into the biggest. The -x keeps you on one filesystem so you do not wander into other mounts.
The usual culprits are /var/log (runaway logs), /var/lib/docker (use docker system df then docker system prune), old kernels filling /boot, core dumps, and forgotten backups or tarballs.
If df and du disagree (df is full but du is much smaller), a process is holding a deleted-but-open file, and the space is not freed until the descriptor closes. Find it with lsof +L1 or lsof | grep deleted, then restart that process (or truncate through its descriptor) to reclaim the space.

Root causes, and how to tell them apart.

It is either runaway logs, Docker bloat, or core dumps (in which case du finds the directory), inode exhaustion (which df -i reveals through lots of tiny files), or a deleted-but-open file (where df is full but du does not add up, and lsof finds the process holding it). Each one has a distinct signal.

How to fix it, verify it, and prevent it.

Clear, rotate, or truncate the offender, and confirm with df -h. Prevent it with logrotate and hard size caps, a docker system prune cron job, and a disk alert at around 80% so that you never reach 100% by surprise.

The trap that less-experienced engineers fall into.

Running rm on a large log that a process still has open — the space will not come back (this is the deleted-open-file trap) and you may break the app's logging. Truncate it (: > file) or restart the holder instead. This one trips almost everyone once.

🎯 Interviewer follow-up questions you should expect.

"df says full but du doesn't add up — what's going on?" A process is holding a deleted-but-open file — you run lsof | grep deleted and restart the holder to reclaim the space.
"Disk shows space free but writes fail — why?" Inode exhaustion — you check df -i.
"How do you prevent hitting 100% again?" With logrotate and caps, a Docker prune cron job, and an alert at 80%.
"Best way to reclaim space from a huge active log?" You truncate it with : > file, not rm, since a process still has it open.

Say it like this"df finds the full mount, du drills to the directory — and I check df -i too, because inode exhaustion looks like a full disk with bytes free. The gotcha that catches people is deleted-but-open files: df shows full but du doesn't add up, because a process still holds a deleted log's file descriptor and the space isn't freed until that closes. lsof | grep deleted reveals it, and restarting the process frees the space — and I truncate large active logs rather than rm-ing them for the same reason. Then I fix the root cause with logrotate, a Docker prune cron, and an alert at 80%, because reaching 100% should never be a surprise."

---

Mark:

6"Too many open files" (EMFILE)Scenario▸

An app logs EMFILE: too many open files, or accept: too many open files, and starts refusing connections or failing to open files — usually under load or after running a while.

What is actually happening (the mental model).

Every open file and every socket is a file descriptor, and there is a per-process limit (ulimit -n). Hitting it is one of two things: either the limit is genuinely too low for legitimate load, or the app is leaking descriptors (opening sockets and files and never closing them). You have to tell them apart — raising the limit "fixes" a too-low limit but only delays a leak — and there is a subtle systemd gotcha about where the limit is set.

How to work through it.

Read the effective limit: ulimit -n for your shell, and for a running process cat /proc/<pid>/limits (the authoritative value for that process).
Read the actual usage over time: ls /proc/<pid>/fd | wc -l or lsof -p <pid> | wc -l, and watch the trend.
Diagnose it:
- a count that is flat but near the limit under load means the limit is too low, so raise it.
- a count that is climbing steadily means a descriptor leak, which no limit will save you from; it is an app fix (unclosed connections or files).
Raise the limits in the right place: /etc/security/limits.conf is applied through PAM at login and does not apply to systemd services — those need LimitNOFILE= in the unit file. This gotcha wastes hours.

Root causes, and how to tell them apart.

It is either a too-low limit for real concurrency (a flat-but-high descriptor count) or a leak (a steadily climbing count that never plateaus). The trend over time is the discriminator.

How to fix it, verify it, and prevent it.

Raise the limits where appropriate (in the unit file for services) and fix the leaks. Prevent it by setting sensible limits for high-connection services in your config management, and by monitoring descriptor usage.

The trap that less-experienced engineers fall into.

Just bumping the limit for a leak (which delays the crash), or setting it in limits.conf for a systemd service and being baffled that it has no effect (systemd ignores it — you use LimitNOFILE).

🎯 Interviewer follow-up questions you should expect.

"Too-low limit or a leak — how do you tell?" You watch the descriptor count over time — flat-but-high versus steadily climbing.
"You raised the limit in limits.conf and nothing changed — why?" Because it's a systemd service, and systemd ignores limits.conf — you set LimitNOFILE in the unit instead.
"Where do you read a running process's real limit?" In /proc/<pid>/limits.

Say it like this"Two possibilities: the limit's too low for real load, or the app's leaking descriptors — and every socket counts as a file descriptor. I read the effective limit from /proc/<pid>/limits and watch the fd count over time. Flat but high under load means raise the limit; steadily climbing means a leak that no limit will fix, so it's an app fix. And I set limits in the right place — systemd services ignore limits.conf and need LimitNOFILE in the unit file, which trips a lot of people up for hours."

---

Mark:

7A cron job silently isn't runningScenario▸

A scheduled job "didn't run" — no output, no error, nothing happened. A backup didn't happen, a cleanup didn't run, and there's no obvious failure to look at.

What is actually happening (the mental model).

Cron runs jobs in a minimal environment — a bare PATH (typically just /usr/bin:/bin), no shell profile, no interactive setup, and often a different user. A script that works perfectly when you run it interactively fails under cron for environmental reasons far more often than scheduling ones. And if the job's output is not redirected, failures vanish silently, because cron mails its output nowhere useful by default.

How to work through it.

Did it even fire? Check grep CRON /var/log/syslog (Debian) or /var/log/cron (RHEL) to confirm cron triggered the job at all. This splits "did not fire" (a schedule or crontab problem) from "fired but failed" (an environment problem).
If it fired but failed, it is the environment. Cron's tiny PATH means unqualified commands are not found, so use absolute paths; environment variables you rely on interactively are not set, so set them explicitly in the script or crontab.
Capture the output: append >> /var/log/myjob.log 2>&1 so that both standard output and standard error are saved — a silent cron job is undebuggable.
Check the right user's crontab (crontab -l -u <user>), and the schedule syntax itself.

Root causes, and how to tell them apart.

If it fired but failed, the cause is environmental: a relative path or missing PATH, a missing environment variable, or the wrong working directory. If it did not fire, the cause is the wrong crontab user, a bad cron expression, or cron (or the timer) being disabled. The cron log answers "did it fire?", which points you to the right half.

How to fix it, verify it, and prevent it.

Fix the environment (absolute paths, explicit environment, output capture). Prefer systemd timers for anything important — they give you real logging through journald, dependency ordering, OnFailure= hooks, and an easy systemctl status, all of which make silent failures far less likely.

The trap that less-experienced engineers fall into.

Assuming a "did not run" job is a scheduling problem and fiddling with the cron expression, when it actually fired fine and failed on a missing PATH or environment variable — invisibly, because the output was never captured.

🎯 Interviewer follow-up questions you should expect.

"A cron job works manually but not under cron — why?" Because of cron's minimal environment — a bare PATH and no profile — so you use absolute paths and set the environment explicitly.
"How do you debug a silent cron failure?" You confirm it fired in the cron log, then redirect standard output and standard error to a log.
"Why prefer systemd timers?" For real logging through journald, dependency ordering, and failure hooks.

Say it like this"Cron failures are almost always environment, not schedule. Cron runs with a bare PATH and no shell profile, so a script that works interactively fails silently under it. I first confirm it even fired from the cron log — that splits a scheduling problem from an environment one — then I use absolute paths, set the env I need explicitly, and redirect stdout and stderr to a log so failures aren't invisible. For anything important I move it to a systemd timer, which gives real logging and failure hooks so it can't fail silently."

---

Mark:

8Zombie / defunct processes piling upScenario▸

ps shows processes marked <defunct> or state Z, sometimes many of them, and someone's worried the box is "filling up with dead processes."

What is actually happening (the mental model).

A zombie is a dead child process whose parent has not reaped it. When a child exits, it becomes a zombie holding just its exit status and a slot in the PID table, until the parent calls wait() to collect that status. Zombies consume no CPU and no memory — they are just un-reaped exit codes. The bug is always the parent (it is not reaping), and the only real danger is exhausting the PID table if they accumulate in huge numbers.

How to work through it.

Spot them: ps -eo pid,ppid,state,cmd | grep -w Z, and note the PPID, the parent that is not reaping.
The parent is the problem: it is not handling SIGCHLD or calling wait(). Fix or restart the parent — killing the zombie itself does nothing, because it is already dead.
Watch for volume: thousands of zombies risk PID exhaustion (kernel.pid_max), which would be a real problem.

Container gotcha (a classic senior signal).

In containers, the app often runs as PID 1, and PID 1 has special responsibility for reaping orphaned children — but most apps do not do it, so zombies accumulate. The fix is a minimal init as PID 1 (tini, or docker run --init) that reaps them. This is the same reaping concept, surfacing in the container world.

The root causes.

Either a parent process with buggy child-handling (no wait() or SIGCHLD handler), or, in containers, an app running as PID 1 that does not reap. You tell them apart by whether it is a normal process's children or the container's PID 1.

How to fix it, verify it, and prevent it.

Fix or restart the parent, or add an init in containers. Prevent it with proper child handling, and with --init for containers that spawn children.

The trap that less-experienced engineers fall into.

Trying to kill -9 the zombies (which is pointless, since they are already dead), or panicking that they are consuming resources (they are not, beyond a PID slot). The real fix is the parent.

🎯 Interviewer follow-up questions you should expect.

"Are zombies consuming resources?" No — just a PID slot and an exit status; the only risk is PID exhaustion at huge scale.
"How do you get rid of them?" You fix or restart the parent — killing the zombie itself does nothing.
"Why do containers accumulate zombies?" Because the app runs as PID 1 and doesn't reap orphans — you add tini or --init.

Say it like this"Zombies aren't consuming resources — they're just un-reaped exit statuses waiting for the parent to call wait(). So the bug is always the parent, and I fix or restart it; killing the zombie does nothing because it's already dead. In containers this bites people because the app runs as PID 1, which is supposed to reap orphaned children but usually doesn't, so I add a minimal init like tini or run with --init. The only real danger is exhausting the PID table if they pile up in the thousands."

---

Mark:

9Can't SSH into a serverScenario▸

SSH hangs, times out, or is refused. The box may be serving traffic fine (so it's not fully down) but you can't get a shell.

What is actually happening (the mental model).

"Can't SSH" spans four independent layers — network reachability, the port and firewall, the sshd daemon, and authentication — and you debug them in order. The single most important senior habit here is to keep an out-of-band access path (a serial console or SSM) so that a broken sshd or a full disk does not lock you out entirely.

How to work through it.

Is it reachable? Use ping <host> (if ICMP is allowed) and traceroute or mtr — is the host even up and routable?
Is the port open? Use nc -zv host 22 or curl -v telnet://host:22. Refused versus timeout is diagnostic: refused means you reached the host but sshd is not listening (the daemon is down or on the wrong port); timeout means a firewall, security group, or NACL is silently dropping the packets.
Is the daemon up? If you can reach it through the console or SSM, run systemctl status sshd and journalctl -u sshd. Note that a full / can break login entirely (it cannot write session files), so a full disk masquerades as an SSH problem.
Is it authentication? Run ssh -v (verbose) from the client — a wrong key, bad ~/.ssh permissions (which must be 600 or 700 or sshd refuses them), an authorized_keys issue, or a disabled account or shell.

Root causes, and how to tell them apart.

Refused means sshd is down or on the wrong port. Timeout means a firewall or security group is dropping the packets. Connects-then-fails-authentication means a key or permissions problem (which ssh -v shows). Cannot-log-in-despite-good-authentication means a full disk. Each layer's test isolates it.

How to fix it, verify it, and prevent it.

Fix the specific layer. Prevent it by configuring an out-of-band path — the EC2 Serial Console or SSM Session Manager — before you need it, so that a broken sshd, a misconfigured firewall, or a full disk never locks you out completely.

The trap that less-experienced engineers fall into.

Having no out-of-band access — the day sshd breaks or the disk fills is the worst possible time to discover you have no other way in.

🎯 Interviewer follow-up questions you should expect.

"Refused vs timed out on port 22 — what does each tell you?" Refused means you reached the host but the daemon is down or on the wrong port; a timeout means a firewall is dropping the traffic silently.
"How do you get in if sshd itself is broken?" Out-of-band — through SSM Session Manager or a serial console, configured beforehand.
"SSH connects then rejects your key — likely cause?" A wrong key or bad ~/.ssh permissions — ssh -v shows it.

Say it like this"I debug it as four layers in order: reachability, then the port and firewall, then the sshd daemon, then auth. nc -zv tells me refused-versus-timeout, which localises it immediately — refused means the daemon's down, timeout means something's dropping packets silently. Verbose ssh -v usually pinpoints key or permission issues, and I remember a full root disk can break login entirely. Most importantly, I always keep an out-of-band path like SSM Session Manager configured before I need it, so a broken daemon or a full disk doesn't lock me out completely."

---

Mark:

10File permissions / ownership broke an app (and SELinux)Scenario▸

After a deploy or a change, an app can't read its config or write its logs — "permission denied" — even though the file permissions look correct.

What is actually happening (the mental model).

Access is governed by more than the rwx bits. There is ownership, group membership, ACLs, and on many production boxes (especially RHEL and CentOS) SELinux or AppArmor — a mandatory-access-control layer that can deny access even when the Unix permissions are perfect. The senior instinct is to check the security context, and to fix it with least privilege rather than reaching for the chmod 777 sledgehammer.

How to work through it.

The basics: ls -l (owner, group, mode) — is the app's user the owner, or in the right group? Use id <user> for group membership.
ACLs: getfacl <path> — an ACL can grant or deny access beyond what ls -l shows.
The security context (the silent killer): ls -Z shows SELinux labels, and ausearch -m avc -ts recent or journalctl | grep -i denied shows AVC denials — SELinux blocking access despite fine permissions. For AppArmor, use aa-status and dmesg | grep apparmor.
Special bits: setuid, setgid, and the sticky bit can change the effective behaviour.

Root causes, and how to tell them apart.

It is a wrong owner, group, or mode (shown by ls -l), an ACL (shown by getfacl), or an SELinux or AppArmor denial (the permissions look fine, but ls -Z or the audit log shows a context mismatch — which is extremely common after moving files, since they do not get the right context). The audit log's AVC denial is the tell for SELinux.

How to fix it, verify it, and prevent it.

Fix the ownership or context precisely — for SELinux, use restorecon to reset the correct context, or an appropriate policy, rather than disabling SELinux. Prevent it by deploying files in ways that preserve their context, and by testing with SELinux in enforcing mode, not disabled.

The trap that less-experienced engineers fall into.

Running chmod 777 to "make it work" — a security hole masquerading as a fix, which does not even address an SELinux denial (the context is still wrong). And disabling SELinux entirely instead of fixing the context.

🎯 Interviewer follow-up questions you should expect.

"Permissions look right but access is denied — what else?" An SELinux or AppArmor context — you check ls -Z and the audit log's AVC denials.
"How do you fix an SELinux context issue?" With restorecon or a proper policy, not by disabling SELinux, and not chmod 777.
"Why is chmod 777 a red flag?" It is a security hole, and it does not fix a context or ownership problem anyway.

Say it like this"I check ownership and mode first, but the silent killer on production boxes is SELinux or AppArmor — the Unix permissions look fine yet access is denied by the security context, which is common after files are moved and don't inherit the right context. ls -Z and the audit log's AVC denials reveal it, and the fix is restorecon or a proper policy, not disabling SELinux. I fix with least privilege; chmod 777 is a red flag I'd call out in review because it's a vulnerability and it doesn't even address a context problem."

---

Mark:

11"Something changed and broke prod" — who/what?Scenario▸

Prod was fine, now it's broken, and nobody admits to changing anything. You need to reconstruct what changed — fast — under pressure.

What is actually happening (the mental model).

You need forensics now, but the deeper truth a senior names is that a hand-mutable production box is itself the bug. Reconstruct the change from the available trail, then push toward a world where "what changed?" is answered by a git diff, not by detective work.

How to work through it.

Logins: last, journalctl _COMM=sshd, and /var/log/auth.log — who was on, and when, around the breakage?
Commands: shell history with timestamps (with HISTTIMEFORMAT set), though it is easily bypassed, so it is a hint rather than proof. Better is auditd for authoritative tracking of executions and file changes.
Package changes: /var/log/dpkg.log, /var/log/apt/history.log, or yum history / dnf history — a patch is a very common "what changed", and the package history is reliable.
File changes: find /etc -mtime -1 (changed in the last day), auditd watches, or config-management drift (ansible --check shows what has diverged from the desired state).

The root causes.

A manual config edit, a patch or package update, a hand-run command, or config drift from the desired state. The package and auth logs, together with auditd, usually reconstruct it.

How to fix it, verify it, and prevent it.

Revert the offending change. The real fix is not better forensics — it is removing the ability to hand-mutate production: config management as the source of truth, immutable infrastructure, and tight IAM and sudo so that every change is code, reviewed and traceable. Then "what changed?" is a git diff.

The trap that less-experienced engineers fall into.

Treating recurring "someone changed something" incidents as a people problem ("be more careful") rather than a systems problem — the mutable production box is what allows untracked change.

🎯 Interviewer follow-up questions you should expect.

"How do you find what changed on a box?" Auth logs, auditd, shell history with timestamps, and package history through yum, dnf, or apt history.
"What's the real fix so this stops recurring?" Config management or immutable infrastructure, so changes live as code, plus tight access control.

Say it like this"Short term I reconstruct it from auth logs, auditd, shell history with timestamps, and package history — a patch or a manual edit is the usual culprit, and package history is reliable. But I'd name the real problem: a mutable, hand-edited prod server is the bug. I'd move us toward config management and immutable infrastructure so 'what changed' is answered by a git diff and a pipeline, not forensic archaeology at 2am — and tighten access so changes can't be made by hand in the first place."

---

Mark:

12Intermittent packet loss / latency from a hostScenario▸

A service has sporadic timeouts or slowness talking to a dependency — most requests fine, some fail — and it's maddeningly hard to pin down because it's not consistent.

What is actually happening (the mental model).

Intermittent problems are the hardest class, so the senior approach is to capture evidence rather than theorise. The failure is usually one of a lossy hop on the path, socket-state exhaustion, an MTU mismatch, or NIC errors — and each one leaves a distinct fingerprint that you find with the right tool.

How to work through it.

Per-hop loss and latency: mtr <dest> (let it run a while) shows which hop loses packets, distinguishing your network from the far end from a middle hop.
Socket health: ss -s and ss -tan state time-wait | wc -l — pileups of TIME_WAIT or SYN-SENT point at connection churn, port exhaustion, or half-open connections.
Capture the actual failure: tcpdump -ni <iface> host <dest> lets you see the retransmits, resets, or duplicate ACKs directly, which tells you how it is failing.
Physical and interface: ethtool -S <iface> (drops, errors, CRC — a bad cable or NIC), and check the MTU — "small requests fine, large ones hang" is the fingerprint of an MTU or PMTU blackhole (see the networking notes for the full story).

Root causes, and how to tell them apart.

It is a specific lossy hop (which mtr localises), TIME_WAIT or port exhaustion (which ss shows as a pileup), an MTU mismatch (small works, big hangs), or NIC and cable errors (shown in the ethtool counters). Matching the tool to the symptom pinpoints it.

How to fix it, verify it, and prevent it.

Fix the specific cause (a route, hardware, tuning, or MTU). Prevent it by monitoring NIC error counters and connection-state metrics, and by capturing during the failure window rather than guessing afterwards.

The trap that less-experienced engineers fall into.

Theorising and changing things blindly on an intermittent problem instead of capturing evidence (with mtr or tcpdump) during the failure — you cannot fix what you have not observed.

🎯 Interviewer follow-up questions you should expect.

"Intermittent packet loss — how do you approach it?" You capture, you don't theorise — mtr for per-hop loss, ss for socket pileups, and tcpdump for the actual retransmits and resets.
"Small requests work, large ones hang — what's that?" An MTU or PMTU-discovery problem.
"How do you rule out hardware?" With ethtool -S for the NIC drop, error, and CRC counters.

Say it like this"Intermittent is the hardest, so I capture rather than theorise. mtr shows whether loss is on a specific hop or the far end; ss reveals socket-state pileups like TIME_WAIT exhaustion; tcpdump catches the actual retransmits and resets so I see how it's failing. I also check NIC error counters with ethtool and consider MTU — 'small works, big hangs' is the signature of an MTU problem. Evidence first, hypothesis second — you can't fix an intermittent problem you haven't actually observed failing."

---

Mark:

13Clock skew causing weird failuresScenario▸

A scattering of maddening, intermittent failures across a fleet — TLS "certificate not yet valid"/"expired", token/JWT rejections, Kerberos failures, duplicate or missed cron runs, or broken distributed consensus — with no obvious common cause.

What is actually happening (the mental model).

Many systems depend on accurate, synchronised time, and when a host's clock drifts, all of them break in confusing, time-shaped ways. Certificate validity windows, token expiry, Kerberos tickets, and distributed-consensus timestamps all assume the clock is right. Time is invisible infrastructure — nobody thinks about it until it drifts and causes a 2am mystery.

How to work through it.

Suspect it when the failures are time-shaped (validity and expiry errors) and spread across hosts (auth or TLS flapping fleet-wide).
Check the sync: timedatectl (is NTP active and synchronised?), and chronyc tracking / chronyc sources (the current offset and the last sync). A large offset is your answer.
Fix it: make sure chronyd or systemd-timesyncd is running and can reach its NTP source (the firewall allows UDP 123), that the timezone is correct, and monitor the drift so it cannot silently recur.

The root causes.

The NTP daemon is not running or cannot reach its source (a firewall), a VM's clock has drifted (a virtualisation clock issue), or the timezone is misconfigured. The offset from chronyc tracking distinguishes drift from other causes.

How to fix it, verify it, and prevent it.

Restore NTP sync. Prevent it by enforcing NTP everywhere through config management and alerting on the clock offset — a boring control that prevents a whole class of baffling incidents.

The trap that less-experienced engineers fall into.

Chasing the symptoms (debugging TLS or auth in isolation on each host) without noticing that they are all time-related and fleet-wide, and so missing that the common cause is clock skew.

🎯 Interviewer follow-up questions you should expect.

"Intermittent TLS/auth failures across a fleet — what would you check?" The clock — skew breaks certificate validity windows and token expiry, so you use chronyc tracking for the offset.
"How do you prevent it?" With NTP enforced everywhere through config management, and drift monitoring.

Say it like this"Intermittent auth or TLS failures across a fleet make me check the clock. Skew silently breaks certificate validity windows, token expiry, and anything time-based, and it presents as scattered, confusing failures. chronyc tracking shows the offset. NTP is boring infrastructure people forget until it causes a 2am mystery, so I make sure chrony is running, can reach its source through the firewall, and is monitored for drift — that prevents a whole class of baffling incidents."

---

Mark:

14Server won't come back after a rebootScenario▸

A box was patched and rebooted (or rebooted for any reason) and never came back — no SSH, no service. It's just… gone, stuck somewhere in boot.

What is actually happening (the mental model).

A box that rebooted but never returned is stuck at boot, and you need console or serial access — not SSH — to see why, because it is not far enough into boot to run sshd. The usual suspects are a small, well-known set: a bad /etc/fstab, a kernel that panics after an update, or a failed critical systemd unit.

How to work through it.

Get eyes on the boot: the EC2 Serial Console or the hypervisor console. SSH is useless on a box that is not finishing boot.
Read the failure:
- a bad /etc/fstab — an entry for a volume that is not present halts boot (waiting for a device that never appears), unless it is marked nofail. This is extremely common after volume or disk changes; boot to rescue, fix fstab, and add nofail.
- a kernel panic after an update — the new kernel does not boot; boot the previous kernel from the GRUB menu, then investigate or roll back.
- a failed systemd unit blocking the boot target, or a full /boot or / (no space to boot); use rescue or emergency mode, systemctl status, and free space.
Recover through rescue or emergency mode, fix the cause, reboot, and verify a clean boot.

Root causes, and how to tell them apart.

It is an fstab entry waiting on a missing device (boot hangs at a mount), a kernel panic (visible on the console, so boot the previous kernel), a failed unit, or a full /boot. The serial console output names it.

How to fix it, verify it, and prevent it.

Fix the specific cause. Prevention is a process: never patch-and-reboot the whole fleet at once. Do a canary reboot: patch one node, reboot it, verify it returns and is healthy, and then roll the fleet. And mark non-critical mounts nofail so a missing volume cannot halt boot.

The trap that less-experienced engineers fall into.

Patching and rebooting the entire fleet simultaneously — then a kernel or fstab issue takes down everything at once, with no healthy canary to have caught it. And not knowing that you can boot the previous kernel from GRUB.

🎯 Interviewer follow-up questions you should expect.

"A server didn't come back after a reboot — how do you debug it?" Through the serial console, since SSH won't help — you look for an fstab issue, a kernel panic, or a failed unit.
"A bad fstab entry halted boot — how do you prevent that?" With nofail on non-critical mounts.
"A kernel update panics — how do you recover?" You boot the previous kernel from GRUB.
"How do you patch safely across a fleet?" With a canary reboot — you patch, reboot, and verify one node, then roll out to the rest.

Say it like this"Serial console is my lifeline — SSH is useless on a box that isn't finishing boot. The usual causes are an fstab entry for a volume that isn't there, which halts boot unless it's marked nofail, or a kernel update that panics, where I boot the previous kernel from GRUB, or a failed systemd unit. The prevention is process: patch a canary, reboot it, verify it comes back healthy, then roll the fleet — never reboot everything at once, or one bad kernel takes down the whole fleet simultaneously."

---

Mark:

15Kernel / sysctl tuning for a workloadScenario▸

A high-throughput service needs OS-level tuning — dropping connections under load, exhausting ports, hitting file limits — and someone asks how you'd tune the kernel for it.

What is actually happening (the mental model).

The OS ships with conservative defaults, and a demanding service sometimes needs specific sysctl levers raised. But tuning is evidence-driven — you find the symptom, change one lever, and measure again — and every value lives in config management, never in a manual edit that evaporates on the next reprovision. Blindly copy-pasting "performance sysctls" from a blog creates new, subtler problems.

How to work through it.

Find the symptom first. Dropping connections under a burst? Use ss -s and netstat -s (look for listen-queue overflows and SYN drops). Ephemeral ports exhausted? File limits? Then pick the matching lever — do not tune blind.
Know the common levers and what each is for:
- net.core.somaxconn and the app's listen backlog address dropped connections under connection bursts.
- net.ipv4.ip_local_port_range and net.ipv4.tcp_tw_reuse address ephemeral-port or TIME_WAIT exhaustion when you make many outbound connections.
- fs.file-max and the nofile limits address global and per-process file-descriptor ceilings.
- vm.swappiness controls how eagerly the kernel swaps.
Apply it reproducibly: put it in /etc/sysctl.d/*.conf through Ansible or cloud-init, load it with sysctl -p, and benchmark before and after to confirm the change actually helped.

The root causes, and when this comes up.

Connection drops under a burst (the backlog and somaxconn), port exhaustion (the ephemeral range and tw_reuse), or file-descriptor limits — each mapped to its lever. The symptom, from ss or netstat -s, tells you which.

How to fix it, verify it, and prevent it.

Change one lever, benchmark it, and keep it in config management. Prevent config drift by never hand-editing sysctl on a box — bake it into the image or the provisioning.

The trap that less-experienced engineers fall into.

Copy-pasting a "10 sysctls for performance" blog post onto production — untargeted tuning creates new problems, and you cannot tell which change did what. And hand-editing sysctl so that the tuning vanishes on the next reprovision.

🎯 Interviewer follow-up questions you should expect.

"How do you approach kernel tuning?" Symptom first — you change one lever at a time, benchmark before and after, and keep it in config management.
"Dropping connections under load — which lever?" somaxconn and the app's listen backlog.
"Why not just apply a known-good sysctl set?" Because untargeted tuning creates subtle new problems — you tune from evidence instead.

Say it like this"I tune from evidence, not vibes. I find the symptom first — if we're dropping connections under burst, somaxconn and the listen backlog are suspects; if we're exhausting ephemeral ports on many outbound connections, I widen the range and enable tw_reuse. I change one lever at a time and benchmark before and after, because blind sysctl copy-paste from a blog is how you create new problems. And every value goes into Ansible or cloud-init, never a manual edit that vanishes on the next reprovision."

---

Mark:

16Analysing logs at scale from the CLIScenario▸

You need to find the cause in gigabytes of logs, fast, during an incident — top errors, offending IPs, a latency pattern — straight from the command line.

What is actually happening (the mental model).

For a fast, single-box triage, a grep | awk | sort | uniq -c | sort -rn pipeline is unbeatable for ranking the top errors, sources, or slow endpoints. But there is a senior meta-point: if you are SSHing to boxes to grep logs during an incident, that is itself a signal that you are missing centralised logging — the fluency is valuable and it should tell you to invest in a queryable log store.

How to work through it (the core patterns).

Narrow the window first: journalctl --since "10 min ago", awk '$0 >= "start" && $0 <= "end"', or sed -n '/start/,/end/p'. Use zgrep for rotated .gz logs.
Rank the offenders: grep ERROR app.log | awk '{print $5}' | sort | uniq -c | sort -rn | head gives the top error types or source IPs at a glance. This sort | uniq -c | sort -rn idiom is the workhorse.
Extract a metric: awk '{sum+=$NF; n++} END{print sum/n}' for an average of a latency field; a quick sort -n plus an index for a percentile.
Correlate across services by timestamp where you can.

The root causes, and when this comes up.

Any incident where you need to find a pattern in high-volume logs quickly — a spike of one error, one bad client IP, or a slow endpoint.

How to fix it, verify it, and prevent it.

Use the pipeline to find the pattern fast. Then escalate the root cause: grepping production logs per host means the real fix is shipping the logs to a queryable store (Loki, ELK, or CloudWatch) so that this becomes one query across the whole fleet, not a per-box hunt.

The trap that less-experienced engineers fall into.

Either not being fluent with the text-processing pipeline (which is slow during an incident), or being so reliant on per-host grep that they never push for centralised logging — treating the SSH-and-grep as normal rather than as a tooling gap.

🎯 Interviewer follow-up questions you should expect.

"Find the top errors in a huge log — how?" You narrow the window, then run grep | awk | sort | uniq -c | sort -rn.
"You're SSHing to boxes to grep during incidents — what does that tell you?" That there's a gap — you need centralised logging so it becomes one query across the fleet.

Say it like this"For a fast triage I narrow the time window, then build a pipeline — grep the errors, awk the field I care about, and sort | uniq -c | sort -rn to rank the offenders. It's incredibly fast on one box. But I'd also flag that if I'm SSHing around to grep logs during an incident, that's a gap — the real fix is centralised logging so it's one Kibana or Loki query across every host, not a per-box hunt. Fluency with the pipeline is valuable, and it should also tell you when to invest in better tooling."

---

Mark:

🧠 How to turn this into muscle memory

Drill the universal reflex first (uptime → top → iostat/free → dmesg/journalctl, and "what changed?") until it is automatic — it unlocks most of the rest and keeps you methodical under pressure.
Break, then fix, in your own lab. Fill a disk with fallocate (and create a deleted-open-file with a script that holds a deleted log); spawn a CPU spinner and profile it; set a tiny memory cgroup and watch the OOM killer; break fstab on a throwaway VM and recover through rescue; misconfigure a cron's PATH and watch it fail silently. Reproducing the failure teaches ten times more than reading about it.
One scenario, three passes. Do it in the lab, then say the spoken version without looking, then explain it to a teammate. Three passes and it is yours.
Keep a "symptom → what it actually was" log as you drill — the deleted-open-file trap, load-is-not-CPU, systemd-ignores-limits.conf. Those pattern-recognitions are exactly what make you fast, and they become the war stories you tell in the behavioural round.

No scenarios match that search. Clear search

Linux & Shell — DevOps Zero → Hero. Theory, Lab & Interview rebuilt from Day 6, the shell-scripting Zero-to-Hero series, and the Day 7 & 8 projects; the 16 scenario drills are stitched in from the Scenario Playbook (Linux domain), rewritten in plain English. ← All chapters