Troubleshooting — Google SDE Interview Prep

Part 1: Questions 1–75


Section 1: Troubleshooting Methodology (Q1–Q15)

Q1: What is your systematic approach when you first encounter an unknown production issue?

Answer: Start with the five-step triage loop:

  1. Establish a timeline — when did the issue start, what changed (deploy, config push, traffic spike)?

  2. Define the blast radius — which services, users, or data are affected?

  3. Gather signals before touching anything — metrics, logs, alerts, error rates. Changing state prematurely destroys evidence.

  4. Form a short-list of hypotheses — ranked by likelihood and ease of verification.

  5. Test one hypothesis at a time — binary-search the system so each test eliminates half the remaining space.

Concretely at a command line:

Never skip step 2 — knowing scope tells you the urgency and how aggressive your mitigation can be.


Q2: How do you gather information without making the problem worse?

Answer: The principle is read-only first. Every command that doesn't mutate state is safe to run immediately. Commands that restart services, change configs, or kill processes require explicit justification.

Safe information-gathering commands:

Risky commands that look safe but aren't:

Document every command run and its output in a shared incident channel so the team has a record.


Q3: Explain divide-and-conquer as applied to distributed system debugging.

Answer: In a distributed system, a request traverses N components. Divide-and-conquer means picking the midpoint of the call graph and determining whether the fault is upstream or downstream, then repeating.

Example: User reports slow checkout. The path is: Browser → CDN → Load Balancer → App Server → Auth Service → DB.

Step 1: Check the App Server response time at the LB layer. If LB logs show 2 s, the problem is at App Server or deeper. Step 2: Add a probe call directly to the App Server, bypassing LB. If that's also 2 s, the LB is innocent. Step 3: Check App Server logs for slow DB queries. If DB query time is 1.8 s, the problem is in the DB or between App Server and DB. Step 4: Run EXPLAIN ANALYZE on the slow query. Missing index found — done.

This halves the search space at each step. With 6 components, worst case is 3 steps rather than 6 sequential checks.

Tools that help map the graph first:


Q4: What is hypothesis-driven debugging and how does it differ from random probing?

Answer: Hypothesis-driven debugging requires forming a falsifiable statement before running any command. "The database is slow because the connection pool is exhausted" is falsifiable — you can check pool metrics and prove or disprove it. "Let me restart things and see" is random probing.

Process:

  1. List all hypotheses.

  2. Rank by: (prior probability) × (ease of verification).

  3. Design a test that, if the result goes one way, confirms the hypothesis and, if it goes the other way, conclusively rules it out.

  4. Run the test.

  5. Update your ranked list and repeat.

Example:

Start with H1 because it's fast and likely. This is Bayesian reasoning applied to debugging.


Q5: When and how do you escalate an incident?

Answer: Escalate when:

How to escalate effectively: Provide a structured handoff:

  1. Symptoms — exact error messages, metrics, time of onset.

  2. What you've ruled out — don't make the next person re-investigate dead ends.

  3. Current best hypothesis — even if uncertain.

  4. Actions taken — commands run, config changes made, rollbacks done.

  5. Impact — users/QPS affected, SLO burn rate.

Use an incident management system (PagerDuty, Opsgenie) and a shared war-room channel. Never escalate silently — always provide context.


Q6: Describe the "5 Whys" technique and its limitations.

Answer: The 5 Whys is an iterative interrogation technique where you ask "why" repeatedly until you reach a root cause.

Example:

Root cause: Inadequate code review process for memory-sensitive code.

Limitations:

  1. Single-path bias — complex failures have multiple contributing causes. The 5 Whys finds one path through the causal tree.

  2. Stops at the wrong layer — "human error" is rarely the root cause; it's the system that allowed the error.

  3. Confirmation bias — the analyst's framing shapes which "why" gets asked.

  4. Not suitable for concurrent causes — if A and B together caused C, 5 Whys misses B.

Better alternative for complex failures: Fault Tree Analysis or Event Timeline reconstruction, which maps all contributing factors in parallel.


Q7: How do you verify a fix before declaring an incident resolved?

Answer: A fix is not verified until the symptom is gone AND the root cause is confirmed eliminated. Steps:

  1. Monitor the primary signal — the error rate, latency metric, or alert that triggered the incident. Wait for it to return to baseline for at least 2–3 measurement periods.

  2. Check secondary signals — related metrics that should also normalize (e.g., if you fixed a DB connection leak, pool utilization and query latency should both drop).

  3. Reproduce the original failure path (if safe) — synthetic transaction, canary request, or replay of the failing request.

  4. Confirm the root cause mechanism is closed — if the fix was "increase connection pool size," verify the pool is no longer hitting the limit.

  5. Watch for 5–10 minutes post-fix — some fixes mask rather than solve (e.g., restarting a process that will OOM again in 10 minutes).

Only then close the incident. Leave the monitoring dashboard up for 24 hours with an on-call watching it.


Q8: What belongs in a post-incident review (PIR) document?

Answer: A PIR (also called postmortem or RCA doc) should be:

Blameless — the goal is systemic improvement, not finding who to punish.

Required sections:

  1. Incident summary — one paragraph: what broke, impact, duration.

  2. Timeline — chronological log of events, detection, diagnosis steps, mitigations. Include timestamps.

  3. Root cause — the underlying technical and process failure(s).

  4. Contributing factors — what made it worse or delayed detection/mitigation.

  5. Impact — quantified: user-hours lost, SLO error budget burned, revenue affected.

  6. What went well — detection was fast, runbooks existed, on-call had context.

  7. What went poorly — alert fired too late, runbook was outdated, ownership unclear.

  8. Action items — each must have: owner, deadline, and a link to a ticket. No vague "improve monitoring."

Anti-patterns to avoid:


Q9: How do you use logs effectively during troubleshooting?

Answer: Logs are most useful when you can filter, correlate, and timestamp-anchor them.

Filtering strategy:

Correlation across services: If services emit a request_id or trace_id, grep all log files for that ID to reconstruct the full request path:

Structured logs (JSON): Use jq to filter:

Pitfalls: Log levels misconfigured (INFO swamping ERRORs), log rotation eating the evidence, timestamps in different time zones, buffered log writes causing ordering issues.


Q10: How do you handle an issue where symptoms are intermittent and hard to reproduce?

Answer: Intermittent issues require persistent capture rather than interactive investigation.

Techniques:

  1. Continuous metric collection — set up monitoring that records at high frequency (1 s intervals) so when it happens again, you have data:

  1. Audit logs / event tracing — enable debug logging temporarily in the suspect component with a time-bounded config (don't forget to turn it off).

  2. Alarm-triggered capture — use a script that dumps state when a metric crosses a threshold:

  1. Reproduction environment — try to replicate load/traffic patterns using recorded production traffic (shadow traffic, replay tools).

  2. Correlate with external events — cron jobs, garbage collection cycles, batch jobs, network reconfigurations. Check /var/log/cron and system event logs.


Q11: What is the difference between a symptom and a root cause? Give an example.

Answer:

Example:

Fixing at the symptom level (restart app servers) provides temporary relief. Fixing at the immediate cause level (increase memory limits) buys time. Only fixing the root cause prevents recurrence.

In PIRs, the goal is always root cause. In incident response, sometimes you must fix at the symptom level first (mitigate) and root-cause later (resolve).


Q12: How do you prioritize between multiple simultaneous failures?

Answer: Triage using two axes: user impact and rate of spread.

Concrete rules:

  1. If a failure is cascading (spreading to more services), stop it first even if current impact is moderate.

  2. Among equal-severity issues, fix the one that unblocks the most other work (dependency ordering).

  3. Data integrity failures take priority over availability failures — you can survive downtime; you can't easily recover corrupt data.

  4. Don't split attention: assign separate responders to each incident rather than one person context-switching.

During a multi-failure event, designate an Incident Commander who doesn't touch keyboards — their job is coordination, communication, and triage decisions only.


Q13: How do you document a troubleshooting session in real time?

Answer: Real-time documentation serves two purposes: knowledge sharing for others joining the incident, and evidence preservation for the postmortem.

Minimum viable war-room log:

Tools: Incident.io, PagerDuty Notes, a dedicated Slack channel, or a shared Google Doc. The medium matters less than the habit.

What to record: Every command run (and its output if relevant), every hypothesis tested, every action taken, timestamps on everything.


Q14: What is the role of monitoring and alerting in proactive troubleshooting?

Answer: Good monitoring lets you detect degradation before users notice and provides the data needed to diagnose issues quickly.

Four golden signals (Google SRE book):

  1. Latency — how long requests take; distinguish success latency from error latency.

  2. Traffic — request rate, QPS, messages/sec.

  3. Errors — rate of failed requests (5xx, timeouts, explicit errors).

  4. Saturation — how "full" the service is (CPU %, memory %, queue depth, connection pool utilization).

Alerting design principles:


Q15: How do you know when you've found the actual root cause vs. a contributing factor?

Answer: You've found the root cause when:

  1. Eliminating it prevents recurrence — the test is: if we undo this single change, does the failure stop?

  2. It explains all the symptoms — a contributing factor may explain some observations but not others.

  3. It's a system-level explanation, not an action — "the cache invalidation logic has a race condition" is a root cause; "the engineer didn't check for the race condition" is a contributing factor.

  4. The causal chain is complete — you can trace from root cause → intermediate effects → symptoms without gaps.

Validation test: Ask "if we fix this and only this, are we confident the issue won't recur?" If yes, that's root cause. If you're hedging ("well, it probably won't recur, but..."), keep digging.

Common traps:


Section 2: Linux Performance & High CPU (Q16–Q30)

Q16: What does load average actually mean on Linux, and what values indicate a problem?

Answer: Load average is the average number of runnable (running + waiting for CPU) or uninterruptible (waiting for I/O) processes over the last 1, 5, and 15 minutes. It is displayed by uptime and top.

The three numbers are 1-minute, 5-minute, and 15-minute averages.

What the numbers mean:

Trend analysis:


Q17: What flags do you use with top, htop, and ps for CPU troubleshooting?

Answer:

top useful keys and flags:

Inside top:

htop:

Inside htop: F6 to sort, F4 to filter, F2 for setup (show tree, customize columns).

ps flags:

stat column values:


Q18: What is CPU steal time and how do you diagnose it?

Answer: CPU steal time (%st in top) is the percentage of time a virtual CPU is waiting for the hypervisor to give it real CPU time. This happens when the physical host is overcommitted — too many VMs competing for physical cores.

Where to see it:

Example top header:

5% steal means the VM is losing 5% of its time waiting for the hypervisor.

What to do:

Important: High steal time can cause mysterious latency spikes and load average increases without apparent CPU-bound processes inside the VM.


Q19: How do you use nice and renice to manage process priority?

Answer: nice values in Linux range from -20 (highest priority) to +19 (lowest priority). Default is 0. Only root can set negative (higher) priority.

nice — start a process with a given niceness:

renice — change priority of a running process:

Verify:

Practical use case: During a production incident where a runaway analytics job is consuming CPU alongside critical services, renice the analytics job to +19 to give back CPU:

Note: nice/renice affect SCHED_OTHER (CFS) processes. For real-time scheduling, use chrt.


Q20: How do you use perf top to identify CPU hotspots?

Answer: perf top provides a real-time view of which functions (in user space and kernel) are consuming the most CPU cycles, by sampling hardware performance counters.

Output columns:

Overhead = percentage of samples this function appeared in (proxy for CPU time).

Workflow for investigating high CPU:

  1. perf top -p <pid> — find the hot function.

  2. perf record -g -p <pid> sleep 30 — record a detailed profile.

  3. perf report — interactive call-graph browser.

  4. If the hot function is in kernel space, check if it's a driver issue or system call overhead.

Prerequisites: linux-tools-$(uname -r) package, kernel symbols (/proc/kallsyms), and ideally debug symbols for userspace binaries.


Q21: How do you use strace -p to investigate a running process?

Answer: strace intercepts and records system calls made by a process. Attaching to a running process with -p is a key diagnostic tool.

Reading output:

The second line shows the process blocked on futex for 5 seconds — a lock contention issue.

Warning: strace adds significant overhead (10–100x slowdown for I/O-heavy processes) due to ptrace. Use -c mode in production to minimize impact. Prefer perf trace for lower overhead.


Q22: What is a zombie process, how does it form, and how do you clean it up?

Answer: A zombie process is a process that has terminated but whose entry remains in the process table because its parent has not yet called wait() to collect its exit status.

How it forms:

  1. Child process exits and sends SIGCHLD to parent.

  2. Parent doesn't call wait() or waitpid() to read the exit code.

  3. The kernel keeps a minimal process table entry (just PID, exit status, resource usage) until the parent retrieves it.

Identifying zombies:

Cleanup options:

  1. Send SIGCHLD to the parent — tells it to reap children:

  1. Kill the parent — when the parent dies, init (PID 1 / systemd) adopts and reaps the zombie:

  1. Fix the code — the real fix is ensuring the parent calls waitpid() or installs a SIGCHLD handler.

Impact: A few zombies are harmless. Thousands can exhaust the PID space (default max: 32768, check cat /proc/sys/kernel/pid_max), preventing new processes from starting.


Q23: How do you identify a runaway process and safely stop it?

Answer: A runaway process is one consuming excessive resources (CPU, memory, file descriptors, sockets) unexpectedly.

Identification:

Graceful stop sequence:

Caution before killing:


Q24: How do you use vmstat to investigate context switching?

Answer: vmstat reports virtual memory statistics and includes context switch information.

Output columns relevant to context switching:

Normal vs. problematic:

Corroborate with pidstat:

High involuntary switches = too many threads competing for CPU. High voluntary switches = I/O-bound processes.


Q25: How do you differentiate between user-space and kernel-space CPU consumption?

Answer: In top and vmstat, CPU time is broken into:

Diagnosing high sy:

Diagnosing high hi or si: Usually network-related. si high often means high network packet rate:

Diagnosing high us: Use perf top, profiler (gprof, async-profiler for Java), or flame graphs:


Q26: What is CPU throttling and how does it differ from CPU saturation?

Answer:

CPU saturation — the system has more work to do than CPU capacity. Characterized by:

CPU throttling — a hard limit is imposed on CPU usage by the scheduler. Common in containerized environments (cgroups).

In Kubernetes:

A container at 100% of its CPU limit is throttled even if the host has idle CPU. The fix is to raise the CPU limit (or request). This is a very common source of latency spikes in containerized services.


Q27: How do you identify CPU affinity issues and pin processes to specific cores?

Answer: CPU affinity controls which CPUs a process is allowed to run on. Incorrect affinity can cause unnecessary context switches, NUMA effects, and cache misses.

Check current affinity:

Set affinity:

NUMA topology:

Why it matters: On a multi-socket server, a process migrating between NUMA nodes incurs remote memory access latency (~2x slower than local). Pin latency-sensitive processes to a single NUMA node:


Q28: How do you investigate high system call overhead?

Answer: High sy CPU often means excessive syscall rate. Tools to investigate:

Common culprits:


Q29: How do you analyze CPU flame graphs?

Answer: Flame graphs (invented by Brendan Gregg) show the call stack hierarchy, with width proportional to time spent.

Reading a flame graph:

Generating:

Patterns to look for:


Q30: How do you handle a situation where CPU usage is high but no single process seems responsible?

Answer: When the sum of per-process CPU in ps doesn't explain total system CPU, investigate these causes:

1. Kernel threads consuming CPU:

2. Short-lived processes not captured by ps:

3. Interrupt and softirq load:

4. CPU frequency scaling (not actually high load):

If CPUs are throttled to low frequency, %CPU numbers are high but work done is low.

5. Cgroup accounting mismatch in containers — use systemd-cgtop or docker stats to get cgroup-level CPU usage.


Section 3: Memory Issues (Q31–Q45)

Q31: How do you investigate an OOM (Out of Memory) kill using dmesg and journalctl?

Answer: When the kernel OOM killer terminates a process, it logs detailed information.

Typical OOM log output:

Key fields:

Check a process's OOM score:

Protect a critical process from OOM kill:


Q32: What are the key fields in /proc/meminfo and what do they mean?

Answer:

Key fields:

FieldMeaning
MemTotalTotal physical RAM
MemFreeCompletely unused RAM
MemAvailableRAM available for new allocations (includes reclaimable cache) — more useful than MemFree
BuffersKernel buffers for raw disk block data
CachedPage cache (file contents cached in RAM)
SwapCachedData in swap that's also in RAM (being swapped back in)
ActiveRecently used memory, not reclaimed
InactiveLess recently used, can be reclaimed
Active(anon)Anonymous active pages (heap/stack)
Active(file)File-backed active pages
SwapTotal / SwapFreeSwap space total and free
DirtyMemory waiting to be written to disk
WritebackMemory actively being written to disk
SlabKernel slab allocator memory
SReclaimableSlab memory that can be reclaimed
SUnreclaimSlab memory that cannot be reclaimed
HugePages_TotalNumber of huge pages allocated
VmallocTotal / VmallocUsedVirtual memory allocated by vmalloc

Most important for "is the system low on memory?" Use MemAvailable, not MemFree. A system with 100 MB MemFree but 8 GB MemAvailable is fine — the kernel will reclaim cache as needed.


Q33: What is the slab cache and how can it cause memory pressure?

Answer: The slab allocator is the kernel's internal object cache. It pre-allocates and pools frequently used kernel objects (dentries, inodes, socket buffers, etc.) to avoid the overhead of constant allocation/deallocation.

Common high-memory slab caches:

Why it causes problems: Slab memory is counted as "used" but much of it is SReclaimable. Under memory pressure, the kernel will reclaim it, but this can cause spikes in latency (dentry/inode cache misses).

Force reclaim (use cautiously — causes latency spikes):

Investigating dentry bloat (e.g., log rotation creating/deleting millions of files):


Q34: How do you analyze swap usage and when is it a problem?

Answer: Swap allows the kernel to move inactive memory pages to disk to free RAM for active use.

When swap is a problem:

Swappiness tuning:

On servers where swap causes latency problems:


Q35: How do you detect a memory leak in a running process?

Answer: A memory leak is characterized by continuously growing RSS (resident set size) without corresponding growth in workload.

Detection:

Valgrind (for development/testing — not production):

AddressSanitizer (compile-time instrumentation):

heaptrack (lower overhead than valgrind):

For production systems — periodic pmap snapshots:

For Java: Use jmap -heap <pid>, jcmd <pid> VM.native_memory, or attach JVisualVM/async-profiler.


Q36: What are huge pages, when should you use them, and how do you configure them?

Answer: Standard Linux page size is 4 KB. Huge pages are 2 MB (or 1 GB on x86-64), which reduces Translation Lookaside Buffer (TLB) entries needed for large memory regions, decreasing TLB miss overhead.

When to use:

Types:

  1. Static huge pages — pre-allocated at boot, never swapped.

  2. Transparent Huge Pages (THP) — kernel automatically promotes 4K pages to 2MB pages. Can cause latency spikes during compaction.

Configure static huge pages:

THP control:

For databases, typically disable THP to avoid latency spikes:


Q37: How do you use vmstat -s to get a memory snapshot?

Answer: vmstat -s prints a summary table of memory and virtual memory statistics.

Key output fields:

Key metrics to check:


Q38: How do you use sar -r for memory monitoring over time?

Answer: sar (System Activity Reporter) from the sysstat package records and reports system activity. -r shows memory utilization.

Output columns (sar -r):

Set up sar collection (if not already running):


Q39: How does memory overcommitment work in Linux?

Answer: Linux allows processes to mmap/malloc more virtual memory than physically exists (RAM + swap). This works because most allocated memory is never actually used simultaneously (sparse allocation).

Overcommit modes (/proc/sys/vm/overcommit_memory):

ValueModeBehavior
0HeuristicDefault. Allow overcommit up to a heuristic limit.
1AlwaysAlways allow allocation. Never fails malloc. Used in some HPC scenarios.
2NeverFail allocation if committed memory would exceed overcommit_ratio% of RAM + swap.

Mode 2 calculation: Max commit = swap_total + (overcommit_ratio/100) × RAM_total

Check committed memory:

Relevance to OOM: With overcommit enabled (mode 0 or 1), processes can allocate more than exists. When they actually touch the pages, the kernel must provide physical memory. If it can't, the OOM killer activates. Mode 2 prevents this by failing the allocation upfront.


Q40: How do you find which process is causing high memory pressure and decide which to kill?

Answer:

Find the top memory consumers:

Identifying the right process to kill:

  1. OOM score — the kernel uses this; higher score = better kill candidate.

  1. Priority of service — killing a cache service (memcached) is better than killing the database.

  2. Recovery behavior — will it restart? Will data be lost?

  3. Memory vs. importance — don't kill a small-footprint critical service to save a large-footprint non-critical one.

Manual kill with grace:


Q41: What is the difference between VSZ, RSS, PSS, and USS?

Answer:

MetricFull NameMeaningUse Case
VSZVirtual SizeTotal virtual address space mapped by the process. Includes code, heap, stack, mmap'd files, shared libraries. Most inflated figure.Rarely useful alone
RSSResident Set SizePhysical memory currently in RAM. Includes shared pages counted multiple times (once per process).Quick gauge; overestimates total system usage
PSSProportional Set SizeRSS but shared pages divided by number of sharers. More accurate total memory footprint.Accurate per-process memory cost
USSUnique Set SizeOnly private (non-shared) pages. Memory freed if this process dies.True memory freed by killing a process

Example: 10 processes each share a 100 MB library.

In practice: Use RSS for quick triage, USS for understanding kill impact, PSS for system-wide memory accounting.


Q42: How do you diagnose memory fragmentation issues?

Answer: Memory fragmentation occurs when free memory exists but not in contiguous blocks of the required size, causing allocation failures or performance degradation.

Check fragmentation:

Compaction:

Symptoms of fragmentation:

Long-term mitigation:


Answer: For lightweight, scriptable memory monitoring without installing tools:

Polling script for trend analysis:

Import to gnuplot or Python pandas for visualization. A steadily increasing RSS without workload increase is a memory leak.


Q44: How does the Linux kernel decide which pages to evict under memory pressure?

Answer: The kernel uses a Least Recently Used (LRU) page replacement approximation via two lists: Active and Inactive.

LRU mechanism:

  1. New pages enter the Inactive list.

  2. If a page is accessed again while in the Inactive list, it moves to the Active list.

  3. Under pressure, pages are moved from Active → Inactive → evicted.

  4. Anonymous pages (heap/stack) go to swap when evicted.

  5. File-backed pages are simply dropped (they can be re-read from disk).

Key sysctl parameters:

kswapd — background page reclaim daemon:

Direct reclaim occurs when an allocation can't be satisfied immediately and the calling process performs reclaim inline — this causes high latency spikes in applications. Goal: keep memory pressure low enough that kswapd handles reclaim in the background.


Q45: How do you set and tune the OOM killer to protect critical processes?

Answer: The OOM killer selects a victim based on oom_score (0–1000). The process with the highest score is killed.

Score factors:

Protect a critical process:

Make a process more likely to be killed (sacrifice non-critical):

System-wide OOM behavior:

View current OOM scores:


Section 4: Disk & Storage (Q46–Q60)

Q46: What is the difference between df -h and df -i, and when does inode exhaustion matter?

Answer:

df -h — shows disk space usage (blocks) in human-readable format:

df -i — shows inode usage:

What is an inode? Each file and directory requires one inode (metadata entry: permissions, timestamps, size, block pointers). The number of inodes is fixed at filesystem creation time (for ext3/ext4).

When inode exhaustion matters: You can have plenty of free disk space but be unable to create new files if inodes are exhausted. This is most common when:

Symptoms:

But df -h shows plenty of free space → check df -i.

Fix:


Q47: How do you use du -sh to find disk space consumers?

Answer: du (disk usage) reports the disk space used by files and directories.

Efficient large-tree scan:

Common space hogs:


Q48: How do you use lsof to find open-but-deleted files still consuming disk space?

Answer: When a file is deleted (unlinked), the disk space is not freed until all file descriptors pointing to it are closed. If a process has a large log file open and the file is deleted, the space remains consumed until the process closes the file or is restarted.

Find open-but-deleted files:

Example output:

5 GB log file deleted but still held open by java PID 12345.

Fix without killing the process:

Fix by restarting (if safe):


Q49: How do you investigate I/O wait using iostat and iotop?

Answer: I/O wait (%iowait in top, wa in vmstat) indicates CPU time spent waiting for I/O operations to complete.

iostat — device-level I/O statistics:

Interpreting:

iotop — process-level I/O:

pidstat -d — per-process I/O from sysstat:


Q50: How do you diagnose disk errors using smartctl and dmesg?

Answer:

dmesg — kernel-level disk errors:

smartctl — SMART disk health monitoring:

NVMe drives:

If Reallocated_Sector_Ct > 0 or Current_Pending_Sector > 0: Plan drive replacement. Back up data immediately.


Q51: Explain LVM basics and how LVM helps in storage troubleshooting.

Answer: LVM (Logical Volume Manager) adds a layer of abstraction between physical disks and filesystems, enabling dynamic resizing and more flexible storage management.

Layered architecture:

Key commands:

Troubleshooting disk full without reformatting: LVM makes it possible to extend filesystems online without downtime, which is critical for production systems.


Q52: How do you find the largest files on a filesystem quickly?

Answer:

Common locations for unexpectedly large files:


Q53: What does /proc/mounts tell you and how do you use it?

Answer: /proc/mounts shows all currently mounted filesystems, including their mount options. It's the authoritative source (unlike /etc/mtab which can be stale).

Useful queries:

Why it matters in troubleshooting:

  1. Filesystem remounted read-only — a common result of detected disk errors. Application writes fail with "Read-only file system":

  1. Missing mounts — a required filesystem not mounted after reboot.

  2. Wrong mount options — performance-critical noatime or barrier=0 not applied.

Remount read-write after fixing the error:


Q54: How do you use fio or dd to benchmark disk I/O?

Answer:

dd — simple sequential I/O test:

fio — flexible, comprehensive I/O benchmark:

When to use: Baseline disk performance before production, diagnose degraded performance after a hardware event, compare storage tiers.


Q55: How do you investigate I/O scheduler behavior and tune it?

Answer: The I/O scheduler (elevator) determines the order in which block layer requests are served.

Check current scheduler:

Common schedulers:

SchedulerBest For
noneNVMe SSDs (hardware has its own queue)
mq-deadlineSSDs, general purpose (good latency guarantees)
kyberFast NVMe SSDs (low-latency target)
bfqHDDs, mixed read/write (fairness, avoids starvation)
cfq (older kernels)General purpose HDDs

Change scheduler:

Queue depth tuning:

Read-ahead:


Q56: How do you handle a filesystem corruption event?

Answer:

Detection:

If the filesystem is still mounted (with errors):

fsck — filesystem check and repair (must unmount first):

Never run fsck on a mounted filesystem (except read-only root with rescue mode).

After repair:

Root cause investigation after repair:


Q57: What is the difference between block size, sector size, and filesystem block size?

Answer:

ConceptTypical ValueMeaning
Physical sector size4096 bytes (4K)Minimum write unit of the physical disk (modern drives)
Logical sector size512 bytes (traditional), 4096 (4Kn)What the OS uses to address the disk
Filesystem block size4096 bytes (ext4 default)Minimum allocation unit for the filesystem

Check disk sector sizes:

512e drives — present 512-byte logical sectors but have 4096-byte physical sectors. Misaligned writes cause a "read-modify-write" penalty.

Check partition alignment:

Filesystem block size relevance:


Q58: How do you monitor disk health proactively with smartd?

Answer: smartd is a daemon that polls drive SMART data at configurable intervals and alerts on failures.

Configuration (/etc/smartd.conf):

Enable and check:

Key SMART thresholds to alert on:

NVMe monitoring:


Q59: How do you recover from accidentally filling a disk to 100%?

Answer: When a disk is 100% full, many operations fail. Approach systematically:

Immediate triage:

Quick wins — find and delete large files:

Freed space not showing?

Emergency space creation: Reserve blocks (ext4 reserves 5% for root by default):


Q60: How do you use blktrace or blkparse for advanced block I/O tracing?

Answer: blktrace captures block layer events (queue, issue, complete) to diagnose complex I/O problems.

Event types:

Calculate latency: Time from Q to C = total I/O latency. Time from I to C = driver/device service time. Time from Q to I = queue wait time.

btt — block trace timing analysis:

Simpler alternative — biotop (from bcc-tools):


Section 5: Services & Daemons (Q61–Q75)

Q61: How do you use systemctl status and journalctl -xe to diagnose a failed service?

Answer:

systemctl status — first stop:

Output includes:

journalctl -xe — extended context with explanations:

Systematic approach:


Q62: How do you diagnose ExecStartPre failures in a systemd unit?

Answer: ExecStartPre commands run before the main ExecStart. If any exit non-zero (unless prefixed with -), the service fails to start.

Identifying ExecStartPre failures:

Example unit file with pre-check:

If --config-check exits 1 (config invalid), the service never starts.

Debugging:

Suppress failure from optional pre-command (using - prefix):

Common ExecStartPre failures:


Q63: What causes crash loops in systemd services and how do you diagnose them with StartLimitBurst?

Answer: A crash loop is when a service repeatedly starts, fails, and restarts. Systemd's rate limiting prevents infinite restart storms.

Crash loop detection:

StartLimitBurst and StartLimitIntervalSec:

Reset the rate limit counter to try again:

Diagnosing why it's crashing:

Common crash loop causes:


Q64: How do you debug dependency failures using After= and Requires= in systemd?

Answer: systemd uses dependency directives to order and require services.

Key directives:

DirectiveEffect
After=BStart A after B, but don't require B to succeed
Requires=BA requires B; if B fails, A fails
Wants=BSoft dependency; A prefers B but continues without it
BindsTo=BStronger than Requires; if B stops, A stops too
Before=BA must start before B
Conflicts=BA and B cannot run simultaneously

Debugging dependency failures:

Common failure pattern: Service uses Requires=network.target After=network.target but actually needs network-online.target (full network, not just interface up):

Checking what network.target vs network-online.target means:

Visualize dependencies:


Q65: How does socket activation work in systemd and how do you debug it?

Answer: Socket activation allows systemd to listen on a socket and start the service only when a connection arrives. This reduces boot time and allows on-demand activation.

How it works:

  1. systemd creates and binds the socket (.socket unit).

  2. When a connection arrives, systemd starts the associated .service unit.

  3. The service inherits the already-bound socket via sd_listen_fds() mechanism.

  4. File descriptor 3 (or more) are the pre-activated sockets.

Example unit files:

Debugging socket activation:

Common problems:


Q66: How do you debug environment file issues in systemd services?

Answer: systemd services can load environment variables from files using EnvironmentFile=.

Debugging environment issues:

Common pitfalls:

Correct format:


Q67: What is the difference between Type=forking and Type=simple and when does the wrong choice break a service?

Answer: The Type= directive tells systemd how to determine when the service is "ready."

TypeMeaningsystemd waits for...
simpleDefault. The main process is ExecStart.ExecStart to start (doesn't wait for readiness)
execLike simple, but waits for exec() to succeedexec() to return
forkingProcess forks and parent exitsParent to exit (child is the daemon)
notifyService sends sd_notify("READY=1")READY=1 notification
dbusService registers on D-BusD-Bus name acquired
oneshotSingle-run taskProcess to exit
idleLike simple, delayed until active jobs doneExecStart (delayed)

Type=simple (wrong for traditional daemons): If an application double-forks (traditional daemon behavior) and you use Type=simple, systemd tracks the wrong PID (the parent that exited). Service appears "active" but the real daemon may have failed.

Type=forking (wrong for non-forking apps): If a modern app doesn't fork but you specify Type=forking, systemd waits forever for the parent to exit.

Best practice for modern apps: Use Type=notify with sd_notify() in the application, or Type=simple for apps that stay in the foreground.

Diagnose with:


Q68: How do you use SYSTEMD_LOG_LEVEL to get debug output from a service?

Answer: systemd services (and systemd itself) can be configured to emit debug-level logs.

Method 1 — Environment variable in unit file:

Method 2 — Set for a one-off restart:

Method 3 — Set on the systemd manager itself (affects all units):

Application-specific debug: SYSTEMD_LOG_LEVEL affects systemd libraries (libsystemd). For application-specific debug:

View all environment variables set for a service:


Q69: How do you check for port conflicts preventing a service from starting?

Answer: A common service startup failure is "Address already in use" — another process is already bound to the port.

Detection:

Find the conflicting process and decide:

If the service crashed and left a stale socket file:

Systemd socket reuse (avoid conflicts):


Q70: How do you use systemd-analyze to diagnose slow boot times?

Answer: systemd-analyze provides boot timing analysis.

Typical slow-boot culprits:

Fix network boot slowness:


Q71: How do you create and use systemd drop-in override files?

Answer: Drop-in files allow modifying a unit's configuration without editing the original (which may be overwritten by package updates).

Create a drop-in:

View effective unit (merged original + overrides):

Reset to default (remove overrides):

Common uses for drop-ins:


Q72: How do you debug a service that runs correctly from the command line but fails under systemd?

Answer: The most common causes of this discrepancy are environment differences, working directory, user/group, or missing stdin/tty.

Systematic comparison:

1. User and group context:

2. Environment variables:

3. Working directory:

4. No terminal / stdin:

5. ulimits:

6. SELinux/AppArmor context:


Q73: How do you manage and investigate systemd service resource limits?

Answer: systemd services run in cgroups, and resource limits can be set both via classic ulimit mechanisms and cgroup-based controls.

View current limits:

Common resource directives in unit files:

Diagnose "too many open files" error:


Q74: How do you use systemd journal for log aggregation and filtering?

Answer: The systemd journal (journald) stores structured logs that can be queried with powerful filters.

Basic queries:

Structured field filtering:

Journal disk usage:


Q75: How do you set up and debug a systemd timer (as a cron replacement)?

Answer: systemd timers replace cron jobs with better logging, dependency management, and resource control.

Timer unit files:

Management:

OnCalendar syntax:

Debug a timer that's not running:

Part 2: Questions 76–150


Section 1: Network Connectivity Troubleshooting (Q76–Q90)

Q76: Walk me through how you'd diagnose a host that can't reach an external IP address. Which tools do you run, and in what order?

Answer: Start at Layer 3 and work outward:

  1. Check local interface and route:

  2. Ping the gateway:

    If this fails, the problem is local (bad cable, misconfigured interface, VLAN issue).

  3. Ping a known external IP (bypass DNS):

    If this fails but gateway ping succeeds, the issue is upstream routing or firewall.

  4. Traceroute to find the hop where packets die:

    mtr combines ping+traceroute and shows per-hop packet loss over time, which is invaluable for intermittent issues.

  5. Check for firewall rules dropping outbound:

  6. Check routing table for unexpected routes:

  7. Confirm DNS separately:

Key distinction: packet loss at a middle hop in traceroute is often just that router de-prioritizing ICMP — look at whether loss continues at subsequent hops. If subsequent hops are reachable, the middle-hop "loss" is benign.


Q77: Explain the difference between "connection refused" and "connection timed out." What does each imply about the network path?

Answer:

Connection Refused (TCP RST):

Connection Timed Out:

Diagnostic approach:

Why it matters at Google scale: A service returning ECONNREFUSED immediately is failing fast; ETIMEDOUT means threads/goroutines pile up waiting, consuming memory and file descriptors — this is the dangerous case that cascades.


Q78: How do you use ss to find which process is listening on a specific port, and what flags do you use?

Answer:

Flag breakdown:

Find a specific port:

Show all connections to a remote host:

Show established connections with memory usage:

Count connections per state:

Compared to netstat: ss reads directly from the kernel's sock_diag netlink interface — much faster on hosts with thousands of connections. netstat reads /proc/net/tcp which scales poorly.

Privileged access: The -p flag showing PID/process requires root or the same UID as the process. Without root, you'll see ("-") for processes you don't own.


Q79: A service is intermittently unreachable. How would you use tcpdump to capture the relevant traffic without filling the disk?

Answer:

Basic targeted capture:

Limit file size and rotate:

Capture only packet headers (no payload) to save space:

Write to stdout, pipe to compress:

Trigger-based capture (stop after N packets):

Read and filter a saved file:

Key filters for troubleshooting:

Disk estimation: 100 Mbps line rate ≈ 750 MB/min full capture. Use -s 96 to reduce to ~10-20% of that.


Q80: Explain how you'd diagnose a DNS resolution failure. What files and commands are involved?

Answer:

Step 1: Determine if it's DNS or the network:

Step 2: Check resolver configuration:

Step 3: Check NSSwitch configuration:

Step 4: Query directly:

Step 5: Test local resolver connectivity:

Step 6: Check for NXDOMAIN vs SERVFAIL:

Step 7: Flush caches:

Common gotchas:


Q81: What causes TIME_WAIT accumulation and how do you mitigate it without breaking things?

Answer:

What is TIME_WAIT? After a TCP connection closes (the side that sent the final FIN enters TIME_WAIT), the kernel keeps the 4-tuple (src IP, src port, dst IP, dst port) in TIME_WAIT state for 2×MSL (Maximum Segment Lifetime = 60s on Linux, so 120 seconds total) to absorb late-arriving duplicate packets from the network.

Why it accumulates: High-throughput short-lived connections (HTTP/1.0, REST APIs without keep-alive, monitoring checks) create many connections per second. At 10,000 connections/second × 120 seconds = 1.2 million TIME_WAIT sockets possible.

Check current state:

Mitigations (safest to most aggressive):

  1. Enable TCP keep-alive and connection reuse (best fix — application level):

    • Use HTTP persistent connections (keep-alive)

    • Use connection pools in your application

  2. tcp_tw_reuse (safe for outbound connections):

  3. Increase local port range:

  4. tcp_fin_timeout (reduce TIME_WAIT duration):

  5. SO_LINGER with linger=0 (use carefully):

    • Sends RST instead of FIN on close — skips TIME_WAIT entirely

    • Can cause data loss; only appropriate for protocols that tolerate it

Do NOT use tcp_tw_recycle: Removed in kernel 4.12. Previously caused dropped connections from NATted clients (multiple clients behind same NAT with different timestamps).

At Google scale: TIME_WAIT is usually a symptom of missing connection reuse. Fix the application layer first.


Q82: How do you check whether a firewall rule is dropping packets, and how do you trace a packet through iptables?

Answer:

List rules with packet/byte counters:

Check all tables:

Trace a packet through all rules (iptables LOG target):

Use xtables-monitor (nftables era):

nftables tracing:

conntrack (check if connection is being tracked):

Common gotchas:


Q83: What is MTU and how does a misconfigured MTU cause subtle, hard-to-reproduce network issues?

Answer:

MTU (Maximum Transmission Unit): The largest IP packet that can be sent on a network path without fragmentation. Ethernet standard MTU is 1500 bytes. Jumbo frames use 9000 bytes.

How PMTU Discovery works: Linux sets the "Don't Fragment" (DF) bit on TCP packets and relies on ICMP "fragmentation needed" (Type 3, Code 4) messages from intermediate routers to discover the path MTU. TCP then reduces its MSS accordingly.

The subtle failure mode: If ICMP is blocked by a firewall (common in corporate/cloud environments), PMTU Discovery fails silently. Small packets work fine (ACKs, HTTP headers), but large packets (bulk data transfers, large responses) are silently dropped. This manifests as:

Diagnosis:

Fix options:

  1. Allow ICMP type 3 code 4 through firewalls (correct fix):

  2. Clamp MSS (TCP-specific workaround):

  3. Reduce interface MTU:

VPN-specific: VPN tunnels add overhead (GRE: 24 bytes, OpenVPN: 28+ bytes, WireGuard: 60 bytes). Always set tunnel interface MTU to outer_MTU minus tunnel overhead.


Q84: How do you use mtr for diagnosing intermittent packet loss, and how do you interpret its output?

Answer:

Run mtr in report mode:

Sample output:

Interpretation rules:

  1. Loss only at hop 3, not at hop 4+: The router at hop 3 is rate-limiting ICMP probes — not real packet loss. Traffic flows fine. This is benign.

  2. Loss starts at hop 3 and continues through all subsequent hops: Real packet loss at hop 3. Everything downstream inherits it.

  3. Loss at the final destination only: That host is rate-limiting ICMP or firewalling probes. Check with TCP-based probe: mtr -T -P 443 8.8.8.8

  4. High latency spike at one hop, normal after: That router has high ICMP processing latency (low priority queue) but forwards traffic normally. Benign for most traffic.

  5. StDev (jitter) increasing progressively: Network congestion or bufferbloat in that segment.

TCP mode for bypassing ICMP filters:

Continuous interactive mode:


Q85: How do you check which ports are open on a remote host, and what's the difference between the approaches?

Answer:

From the remote host itself (most reliable):

From a remote machine — nmap:

Lightweight alternatives:

Differences:

ToolMethodPrivilegeSpeedStealth
sskernel APIlocal onlyinstantN/A
nmap -sTfull TCPnonemediumno
nmap -sSSYN onlyrootfastpartial
nc -zTCP connectnoneslowno
bash /dev/tcpTCP connectnoneslowno

Important: Port scanners only show what's reachable from the scanner's network position. A port may be open but blocked by firewall rules between you and the host — always scan from the same network segment as the actual clients.


Q86: Describe how you would troubleshoot a situation where ping works but curl to the same host fails.

Answer:

ping uses ICMP; curl uses TCP (and DNS). The failure is somewhere in TCP or the application layer.

Step 1: Rule out DNS:

Step 2: Test TCP connectivity directly:

Step 3: If TCP connects, test HTTP manually:

Step 4: Check TLS issues (for HTTPS):

Step 5: Check for firewall blocking TCP but allowing ICMP:

Step 6: Check curl-specific issues:

Common causes:


Q87: What is ip route get and when is it more useful than looking at the routing table directly?

Answer:

Why it's more useful than ip route show:

ip route show displays the routing table, which can have many overlapping prefixes, policy routing rules, and multiple routing tables. Determining which route would actually be used requires manually evaluating prefix length, metric, and policy routing rules in order.

ip route get asks the kernel the exact same question it asks when forwarding a real packet: "For this destination, what route do I use?" The kernel evaluates all routing tables, policy rules (from ip rule show), and returns the single winner.

Examples:

Policy routing context:

When multiple routing tables exist (common in multi-homed servers, VRFs, or Google's internal routing), ip route get is the only reliable way to know which table and route wins for a specific source+destination combination.


Q88: How would you diagnose high network latency between two hosts in the same datacenter?

Answer:

Expected intra-datacenter latency: 0.1–0.5 ms. If you're seeing 1–50 ms, something is wrong.

Step 1: Confirm the latency and its variance:

Step 2: Check if it's symmetric:

Step 3: Eliminate routing hops:

Step 4: Check NIC and switch statistics:

Step 5: Check CPU and interrupt handling:

Step 6: Check for TCP bufferbloat:

Step 7: Check for CPU throttling/c-states:

Step 8: Physical layer (if nothing else explains it):


Q89: How do you use netstat or ss to find a port conflict when a service fails to start?

Answer:

When a service fails to bind to a port you'll see errors like:

Find what's using the port:

If the port appears free but service still fails:

Check for lingering socket file (Unix domain sockets):

Check all sockets including non-listening:

Force immediate reuse (application config): Most servers set SO_REUSEADDR by default, which allows binding to a port in TIME_WAIT. If not:


Q90: Explain TCP SYN flood attacks and how SYN cookies mitigate them.

Answer:

The SYN flood attack: A normal TCP 3-way handshake: Client sends SYN → Server allocates a half-open connection entry in the SYN queue → Server sends SYN-ACK → Client sends ACK → Connection promoted to accept queue.

In a SYN flood, an attacker sends millions of SYN packets with spoofed source IPs. The server allocates state for each one, fills the SYN queue (default: 128-1024 entries), and legitimate connections are rejected because the queue is full.

SYN cookies mechanism: Instead of allocating state on SYN receipt, the server encodes connection parameters into the ISN (Initial Sequence Number) of the SYN-ACK:

No state stored. When the client returns an ACK, the server decodes the ISN to reconstruct connection parameters and creates the socket only then.

Legitimate clients will complete the handshake; spoofed IPs won't receive the SYN-ACK and thus never send ACK.

Check and configure:

Tuning SYN queue:

Trade-offs of SYN cookies:

Modern complement: nftables / iptables rate limiting:


Section 2: Application Debugging (Q91–Q105)

Q91: How do you enable core dumps for a crashing process and then analyze the resulting file?

Answer:

Step 1: Enable core dumps:

Step 2: Configure core file naming and location:

Step 3: Check if systemd-coredump is active (modern systems):

Step 4: Analyze with gdb:

Step 5: Ensure debug symbols are present:

Without source/symbols — at minimum:


Q92: A process has crashed with a segfault. How do you find the root cause using dmesg and the core dump?

Answer:

Step 1: Find the segfault in dmesg:

Decode the instruction pointer:

Step 2: Check if ASLR affects the address:

Step 3: gdb analysis of core:

Common patterns:

Tools for deeper analysis:


Q93: How do you use strace to debug a hanging or misbehaving process, and how do you limit the overhead?

Answer:

Attach to a running process:

Trace from start:

Common useful flags:

Diagnosing a hanging process:

For performance overhead:

ltrace for library calls:


Q94: What information does /proc/<pid>/fd give you, and how do you use it to debug file descriptor leaks?

Answer:

/proc/<pid>/fd/ is a directory of symlinks, one per open file descriptor, pointing to the actual resource:

Count open FDs for a process:

Detect FD leak over time:

fdinfo — additional metadata per FD:

Find deleted files still held open (common cause of "disk full but no files"):

System-wide FD usage:

Increase FD limits:


Q95: How do you debug a process that is consuming unexpected amounts of CPU?

Answer:

Step 1: Confirm which process and threads:

Step 2: See what the process is doing:

Step 3: Application-level profiling:

Step 4: Check for CPU throttling (containers/cgroups):

Step 5: Check for runaway loops:


Q96: Explain how ldd and ldconfig work and how you debug missing shared library errors.

Answer:

The error:

ldd — show shared library dependencies:

ldconfig — update the linker cache:

Debugging steps:

RPATH vs RUNPATH:


Q97: How do you find and kill a zombie process, and what causes zombie processes?

Answer:

What is a zombie process? A process that has exited but whose parent has not yet called wait() to collect its exit status. The kernel keeps a minimal entry in the process table (just the exit code and resource accounting) until the parent reads it.

Zombie entries show:

How to resolve:

  1. Find the parent and send SIGCHLD:

  2. If parent won't reap — kill the parent:

  3. Cannot kill the zombie itself:

Root causes:

Impact:

Set SIGCHLD to SIG_IGN in the parent to tell the kernel to auto-reap children (no wait() needed):


Q98: How do you generate a core dump from a running process without killing it?

Answer:

Using gcore:

Using gdb:

Using /proc/<pid>/mem (manual, more complex):

For Java — generate heap dump (different from core dump):

Why non-destructive core dumps are useful:

Considerations:

Using /proc/sysrq-trigger for kernel-level core dump (NMI):


Q99: What does ulimit -n control, and what happens when a process exceeds it?

Answer:

ulimit -n controls the maximum number of open file descriptors for a process (RLIMIT_NOFILE). In Linux, everything is a file descriptor: regular files, directories, sockets, pipes, device files, eventfd, timerfd, inotify watches, etc.

Checking limits:

What happens when limit is exceeded:

Check current FD usage:

Increase limits:

Typical values:


Q100: How do you use strace to trace system calls of a multi-threaded process?

Answer:

The challenge: By default, strace -p PID attaches only to the main thread. Other threads (which are separate tasks in Linux) are not traced.

Trace all existing threads:

Trace the main process AND follow all future forks/clones:

From process start with all threads:

Focus on specific syscalls across all threads:

Interpret multi-threaded output:

Deadlock detection with strace:

Better tools for production thread analysis:


Q101: How would you debug a process that works fine alone but fails when multiple instances run concurrently?

Answer:

This is a concurrency/resource contention problem. Systematic approach:

Step 1: Identify the failure mode:

Step 2: Check for shared resource contention:

Temp files with non-unique names:

Port conflicts:

Lock files:

Database connections:

Step 3: Race conditions on shared state:

Step 4: Reproduce deterministically:

Step 5: Use helgrind for threading bugs:

Step 6: Linux futex analysis:


Q102: What are the key differences between strace and ltrace, and when would you use each?

Answer:

Featurestraceltrace
TracesKernel system callsUser-space library calls
Mechanismptrace + seccompptrace + LD_PRELOAD-like
Examplesread(), write(), open(), mmap(), fork()malloc(), free(), printf(), strcmp(), fopen()
OverheadModerate (ptrace stop per syscall)Higher (intercepts all PLT entries)
Use forI/O, networking, filesystem, signalsMemory allocation, string ops, library bugs

When to use strace:

When to use ltrace:

Combined approach:

Production alternative (lower overhead):


Q103: How do you debug a service that intermittently runs out of memory?

Answer:

Step 1: Confirm and characterize the leak:

Step 2: Differentiate leak types:

Step 3: Use heap profiler:

Step 4: Java-specific:

Step 5: Check kernel OOM killer:


Q104: How do you trace which files a process is reading at startup to debug a slow initialization?

Answer:

strace approach:

inotifywait for real-time file access:

fatrace (faster, uses fanotify):

blktrace for disk I/O timing:

perf for comprehensive I/O tracing:

Analysis pattern:


Q105: How do you use /proc/<pid>/maps and /proc/<pid>/smaps to understand a process's memory layout?

Answer:

/proc/<pid>/maps — memory map:

Permission flags:

/proc/<pid>/smaps — detailed per-region stats:

PSS (Proportional Set Size) is the most accurate measure of a process's memory cost — it divides shared pages among all processes using them.

Useful analysis scripts:

Use cases:


Section 3: Logs & Monitoring (Q106–Q120)

Q106: How do you use journalctl to filter logs for a specific service, time range, and priority level?

Answer:

Filter by service unit:

Filter by time:

Filter by priority (syslog severity levels):

Kernel messages only:

Combined filters:

Follow live (like tail -f):

Output formats:

Show last N lines:

Search for pattern:

Disk usage:


Q107: How does rsyslog work, and how would you configure it to forward logs to a central server?

Answer:

Architecture: rsyslog uses a modular pipeline: Input modules (imjournal, imudp, imtcp, imfile) → Processing (filtering, property-based routing, templates) → Output modules (omfwd for forwarding, omfile for local files, omelasticsearch, etc.)

Configuration file structure:

Basic local logging (default /etc/rsyslog.conf):

Forward to central syslog server (UDP — fire and forget):

Forward with TCP and queuing (reliable forwarding):

Forward only specific facility/severity:

Receive logs from remote (central server config):

Test and reload:


Q108: How do you use dmesg effectively for diagnosing kernel-level events, and what does -T do?

Answer:

Basic usage:

Filter by level:

Filter by facility:

Search patterns:

Common kernel messages and what they mean:

Ring buffer size and clearing:

Persist kernel messages (ring buffer is lost on reboot):


Q109: Explain how logrotate works and how you would configure it for a custom application log.

Answer:

What logrotate does: logrotate is run by cron (typically daily via /etc/cron.daily/logrotate). It reads config from /etc/logrotate.conf and /etc/logrotate.d/. For each configured log file, it:

  1. Renames the current log (e.g., app.log → app.log.1)

  2. Optionally compresses old rotations (app.log.1.gz)

  3. Signals/restarts the application to reopen its log file

  4. Removes logs older than the retention policy

Configuration for a custom app:

Common directives:

Test without executing:

Check rotation status:

Troubleshooting logrotate failures:


Q110: How do you identify which log entries indicate a real problem versus noise, and how do you build a simple alert?

Answer:

Identifying signal vs noise:

1. Categorize by frequency pattern:

2. Look for correlated spikes:

3. Cross-correlate with metrics:

Building a simple alert with shell:

Run via cron:

Systemd path unit for immediate trigger:

Production approach: Use structured logging + metrics pipeline (Fluentd/Vector → Prometheus counter on error log lines → Alertmanager rule on rate).


Q111: Describe the layout of /var/log and what the key log files are for.

Answer:

Reading binary logs:

RHEL/CentOS differences:

Key for incident response:


Q112: What is auditd and how do you use it to track file access or privilege escalation?

Answer:

auditd is the Linux kernel audit subsystem daemon. Unlike syslog (which captures what applications choose to log), auditd hooks into the kernel and can log any system call, file access, or security event regardless of whether the application wants it logged.

Check if running:

Key files:

Add rules with auditctl:

Make rules persistent:

Search audit logs:

Read a specific audit event:


Q113: What is structured logging and how does it help with log analysis at scale?

Answer:

Structured logging means emitting logs as machine-parseable key-value pairs (typically JSON) rather than free-form text strings.

Unstructured (traditional):

Structured (JSON):

Why it helps at scale:

  1. Exact field filtering without regex:

  2. Aggregation and metrics extraction:

  3. Trace ID correlation across services:

  4. Log pipeline efficiency: Fluentd/Logstash/Vector can route, filter, and aggregate without regex parsing.

Implementation examples:

journald native structured fields:

At Google scale: Structured logs feed into systems like Cloud Logging / Stackdriver where fields become queryable dimensions, enabling dashboards, SLO calculation from log data, and automated incident detection.


Q114: How do you correlate logs across multiple services to trace a single user request?

Answer:

The core technique: propagate a trace/request ID through all service calls.

Step 1: Generate a trace ID at the entry point:

Step 2: Each downstream service logs with the same trace ID:

Step 3: Correlate at query time:

Without a trace ID (retrofit approach):

OpenTelemetry (modern standard):

At Google: Every RPC carries a Trace ID via Stubby/gRPC metadata. Dapper (predecessor to Zipkin/Jaeger) provides end-to-end tracing. Log queries use the trace ID as a primary key for correlation.


Q115: How do you set up a simple monitoring check that alerts when a log file grows too fast or stops growing?

Answer:

Alert when log grows too fast (error spike):

Alert when log stops growing (service stuck/dead):

Using inotifywait for real-time monitoring:

systemd service health via log silence:

Production tools:


Answer:

Using journald (persistent journal):

Using traditional syslog (no reboot continuity by default):

Find process crash across reboots:


Q117: How do you monitor disk I/O in real time and identify which process is causing high disk usage?

Answer:

iotop — per-process I/O monitoring:

iostat — per-device I/O statistics:

pidstat — per-process I/O:

For one-shot snapshot without iotop:

blktrace for deep disk analysis:

Find which files a process is writing:

cgroup-level I/O stats:


Q118: What is log sampling and when is it appropriate?

Answer:

Log sampling means recording only a fraction of log events of a given type rather than every occurrence. Instead of logging every single HTTP request, log 1 in every 100.

When appropriate:

  1. High-frequency, low-information events at steady state:

    • Every successful health check (200 OK) at 1000 RPS → 86 million log entries/day for zero additional insight

    • Routine read queries against a database

  2. Known-good paths with predictable behavior:

    • CDN cache hits (all identical pattern)

    • Internal heartbeats between services

When NOT to sample:

Implementation strategies:

Head-based sampling (random):

Tail-based sampling (sample based on outcome):

Rate limiting (cap at N per second):

Best practice: Always log the sample rate as a field so downstream analysis can extrapolate true counts: {"sampled": true, "sample_rate": 0.01}.


Q119: How do you use watch and other tools to build a simple real-time dashboard for system health?

Answer:

watch — refresh a command every N seconds:

Composing a multi-panel dashboard with tmux:

Quick one-liners for common checks:

vmstat for trend watching:

dstat (combines vmstat, iostat, netstat):


Q120: How would you diagnose a situation where logs show no errors, but users report the service is slow?

Answer:

Absence of errors in logs while users see slowness means the problem is likely latency rather than failures. The service is responding, just slowly.

Step 1: Check if logs have latency data:

Step 2: System resource saturation (the USE method):

Step 3: Application-level latency breakdown:

Step 4: GC pauses (JVM, Go):

Step 5: Connection pool exhaustion (no errors, just queueing):


Section 4: Boot & Kernel Issues (Q121–Q135)

Q121: A server won't boot and drops into the GRUB rescue shell. What are your steps to recover?

Answer:

The GRUB rescue shell (grub rescue>) appears when GRUB cannot find its modules or the configuration file — typically because of a corrupted filesystem, changed disk layout, or accidental deletion of /boot/grub.

Step 1: Identify available disks and partitions:

Step 2: Find where GRUB modules are:

Step 3: Set root and prefix, load normal module:

Step 4: If normal module loads the menu, boot manually from GRUB prompt:

Step 5: Once booted, reinstall GRUB properly:

If unable to boot at all — use a Live USB:

Common causes:


Q122: What is the initrd/initramfs and what happens if it is corrupted or missing?

Answer:

What is initramfs? The initramfs (initial RAM filesystem) is a compressed cpio archive loaded by the bootloader into RAM before the real root filesystem is mounted. It contains:

Boot sequence:

What happens when corrupted/missing:

Recovery from initramfs emergency shell:

Rebuild initramfs from a working system or chroot:


Q123: What causes a kernel panic, and how do you analyze one after the fact?

Answer:

Common causes of kernel panics:

  1. Kernel bug / NULL pointer dereference in kernel code — the kernel equivalent of a segfault

  2. Unrecoverable hardware error — bad RAM (MCE), corrupted kernel text

  3. Corrupted kernel stack — stack overflow in kernel context

  4. Missing root filesystem — VFS: Unable to mount root fs

  5. Explicit panic() call — a driver or subsystem called panic("fatal error") on an unrecoverable condition

  6. Soft lockup / hard lockup — watchdog timer fires because a CPU was stuck in kernel code for >10s (soft) or >20s (hard) without yielding

Panic message anatomy (from serial console/kdump):

Capturing panics with kdump:

Analyze vmcore with crash:

Decode a panic from dmesg without kdump:


Q124: How do you perform filesystem checks and repairs in emergency/single-user mode?

Answer:

Entering emergency/single-user mode:

At GRUB: Edit the boot entry (press e), append to the kernel line:

Remounting root read-write (often needed in emergency mode):

Running fsck:

XFS-specific recovery:

After repair, verify:


Q125: What are kernel parameters, and how do you tune them with sysctl?

Answer:

Kernel parameters (exposed via /proc/sys/) control kernel behavior at runtime: networking, memory management, security, filesystems, and more.

View and set:

Important categories and parameters:

Network performance:

Memory:

Security:

Filesystem:


Q126: What is kexec and when would you use it?

Answer:

kexec is a Linux kernel feature that allows loading and booting a new kernel directly from the currently running kernel, bypassing the BIOS/UEFI boot sequence entirely.

Key properties:

Use cases:

  1. Fast kernel updates in production:

  2. Crash kernel for kdump (most common production use):

  3. Live patching alternative for kernel version upgrades:

    • kpatch / livepatch for in-place kernel fixes

    • kexec for full version changes without hardware re-init

Setup:

Limitations:


Q127: How does kdump work, and how do you configure it to capture kernel crash dumps?

Answer:

Architecture: kdump uses kexec to boot a second "capture kernel" when the primary kernel panics. The capture kernel runs a minimal environment (loaded into pre-reserved RAM), mounts storage, and saves the memory image of the crashed kernel to disk as /proc/vmcore.

Step 1: Reserve memory for the capture kernel:

Step 2: Install and configure kdump:

Step 3: Enable and verify:

Step 4: Test (warning: reboots the machine):

Step 5: Analyze the crash dump:

makedumpfile compression levels:


Q128: How do you manage kernel modules — loading, unloading, and debugging issues?

Answer:

List loaded modules:

Load and unload:

Module information:

Persistent loading:

Debugging module issues:


Q129: What are soft lockups and hard lockups, and what do you do when they appear in dmesg?

Answer:

Watchdog mechanism: Linux runs a high-priority watchdog thread per CPU. This thread increments a counter and checks that the CPU is making progress.

Soft lockup:

Hard lockup (NMI watchdog):

Immediate response:

Tuning watchdog thresholds:

Common causes and fixes:

SymptomLikely CauseFix
Soft lockup on all CPUsSystem overloadedReduce load, check for runaway process
Soft lockup + NFS mentionsNFS server unreachableFix NFS mount or use soft+timeo options
Soft lockup + virtioVM hypervisor overloadedReduce VM density
Hard lockupHardware faultCheck mcelog, run memtest, replace hardware
RCU stallCPU stuck with RCU read lockKernel bug, check for updates

Panic on lockup (for automated recovery):


Q130: How do you use sysrq for emergency access to a hung system?

Answer:

Magic SysRq is a kernel feature that responds to key combinations even when the system is mostly unresponsive, because it bypasses the normal input stack.

Enable SysRq:

Trigger via keyboard (physical console):

Trigger via /proc (remote/programmatic):

The REISUB sequence for safe emergency reboot:

Wait a few seconds between each step. This is the safe alternative to a hard power cycle.

Use case scenarios:


Q131: What is mcelog and how do you respond to machine check exceptions?

Answer:

Machine Check Exception (MCE): Modern x86 CPUs have a Machine Check Architecture (MCA) that detects hardware errors: ECC memory errors, cache errors, bus errors, thermal events, and more. When a hardware error occurs, the CPU records it in MSR (Model Specific Register) banks and can signal an MCE.

Two types:

Tools:

mcelog (older, legacy):

rasdaemon (modern replacement):

Direct kernel MCE messages:

Responding to MCEs:

Error TypeFrequencyAction
Single CEOccasionalMonitor; ECC doing its job
CE on same addressIncreasingBad DIMM; schedule replacement
Multiple CEsFrequentUrgent DIMM replacement; schedule maintenance
UCEAnyImmediate — risk of data corruption; take system out of service
CPU cache MCEAnyPossible CPU fault; replace CPU or motherboard

Q132: How do you debug a server that randomly reboots with no clear kernel panic in logs?

Answer:

Random reboots without panic logs are among the hardest issues to diagnose. Work through these systematically:

Step 1: Determine if it was a clean or dirty reboot:

Step 2: Check for hardware watchdog:

Step 3: Check thermal events:

Step 4: Check power issues:

Step 5: Check for OOM without MCE panic:

Step 6: Check for kernel oops (non-fatal, but indicates bug):

Step 7: Enable persistent panic logging:

Step 8: Hardware diagnostics:


Q133: How do you use /proc/sys to tune kernel behavior for a high-traffic web server?

Answer:

For a high-traffic web server, the goal is: maximize connection throughput, minimize latency, avoid resource exhaustion.

Network tuning:

File descriptor limits:

Memory:

Apply:

Validate in nginx config too:


Q134: What is the difference between dmesg ring buffer and the systemd journal for kernel messages?

Answer:

The kernel ring buffer (dmesg):

systemd-journald kernel capture:

Comparison:

Aspectdmesgjournalctl -k
PersistenceRAM only, lost on rebootPersistent (if configured)
Previous bootsNoYes (-b -1, -b -2)
TimestampsSeconds since bootReal wall-clock time
Filteringgrep/awkRich query language
AvailabilityImmediate (kernel startup)After journald starts
Size limitFixed ring bufferConfigurable (% of disk)
FormatTextBinary (queryable)

Practical difference:

When they diverge:


Q135: How do you diagnose and fix a system stuck in emergency mode at boot?

Answer:

Emergency mode is systemd's last-resort target: a minimal environment with only the root filesystem mounted (often read-only) and a root shell. It activates when a critical unit fails.

Step 1: Read the failure message:

Step 2: Identify the failing unit:

Step 3: Fix common causes:

Bad /etc/fstab entry:

Filesystem errors (fsck failed automatically):

Missing mount point directory:

NFS mount failing at boot:

Step 4: Remount and continue boot:

Prevent recurrence:


Section 5: Distributed Systems & Real-World Scenarios (Q136–Q150)

Q136: What is a cascading failure, and how do you detect and stop one in progress?

Answer:

A cascading failure occurs when the failure of one component causes increased load or degraded behavior in other components, which then fail, further increasing load on remaining components — a positive feedback loop leading to total system failure.

Classic pattern:

Detection signals:

Stopping a cascade in progress:

  1. Shed load immediately (most important):

  2. Disable retries / reduce retry aggressiveness:

  3. Add capacity fast:

  4. Isolate the root cause:

  5. Protect remaining healthy instances:

Prevention: circuit breakers, bulkheads, load shedding with priority queues, exponential backoff with jitter on clients, capacity buffers (never run at 100%).


Q137: How do you detect and handle split-brain in a distributed system?

Answer:

Split-brain occurs when a network partition causes a cluster to divide into two (or more) groups, each believing it is the authoritative primary and potentially accepting conflicting writes.

Detection:

The quorum approach (prevention): Require a majority (quorum) of nodes to agree before any write is accepted or leadership is granted. A 3-node cluster requires 2 nodes to agree — a partition into 1+1+1 means no side has quorum, so no side proceeds.

Fencing (STONITH — Shoot The Other Node In The Head):

Database-level split-brain recovery:

Key principle: prefer consistency over availability during network partition (CP in CAP theorem) for data stores. Use quorum writes and fencing to guarantee only one writer.


Q138: How does NTP clock skew affect distributed systems, and how do you diagnose and fix it?

Answer:

Why clock skew matters in distributed systems:

  1. Log correlation: Events that happened in order appear reversed; incident reconstruction is confusing

  2. TLS certificate validation: Certificates appear expired or not yet valid if clocks diverge > allowed skew (usually 5 minutes)

  3. Kerberos authentication: Fails if clock skew > 5 minutes

  4. Distributed databases: CockroachDB, Spanner, Cassandra, and others use wall-clock time for ordering — skew causes stale reads or write conflicts

  5. Caches: TTL expiry calculated incorrectly; stale data served longer than intended

  6. Distributed tracing: Span timestamps wrong; waterfall diagrams inaccurate

  7. Lease/lock expiry: A node holds a lease that has expired on the master's clock — split-brain risk

Diagnosing clock skew:

Acceptable thresholds:

Fixing clock skew:


Q139: What is the thundering herd problem and how do you mitigate it?

Answer:

The thundering herd occurs when many processes/threads are simultaneously awakened or triggered by an event, but only one can handle it, causing all others to go back to sleep — wasting CPU and causing latency spikes.

Classic forms:

  1. Accept thundering herd (pre-kernel 3.9): Multiple worker processes all accept() on the same listening socket. When a connection arrives, all workers wake up, one gets the connection, others go back to sleep. At 10k workers, this is massively wasteful.

  2. Cache stampede (cache thundering herd): A hot cache entry expires. All concurrent requests miss the cache simultaneously, all query the database, all compute the same value, all try to update the cache. The database gets hit with N identical queries at once.

  3. Reconnect storms: All clients lose connection to a server simultaneously (server restart). All reconnect immediately, overwhelming the server.

Mitigations:

Accept thundering herd:

Cache stampede / cache thundering herd:

Reconnect storms:

epoll thundering herd (Linux kernel):


Q140: How do you debug an issue that only occurs on one of many hosts in a fleet?

Answer:

When a problem is isolated to one host out of many, the most productive approach is differential diagnosis — compare the affected host to healthy hosts.

Step 1: Confirm the problem is truly host-specific:

Step 2: Diff the host against healthy peers:

Step 3: Check host-specific metadata:

Step 4: Check resource exhaustion specific to that host:

Step 5: Check if it's a DNS/network routing issue specific to that host:

Step 6: Parallel-ssh for fleet-wide checks:


Q141: Walk through the steps of a structured incident response for a production outage.

Answer:

Phase 1: Detect and Acknowledge (0–5 minutes)

Phase 2: Triage (5–15 minutes)

Phase 3: Communicate (ongoing throughout)

Phase 4: Mitigate (primary goal — restore service, worry about root cause later)

Phase 5: Resolve and Monitor

Phase 6: Postmortem (within 48 hours)


Q142: How do you write a good blameless postmortem?

Answer:

A blameless postmortem (post-incident review) focuses on improving systems and processes, not on identifying and punishing individuals. The premise: engineers make the best decisions with the information available at the time; system design allowed the error to have impact.

Structure:

Key principles:


Q143: A canary deployment is causing elevated errors. Walk through your response.

Answer:

Context: A canary deployment sends a small percentage (1-10%) of traffic to the new version while the rest goes to stable. If canary shows elevated errors, you must decide: roll back, fix-forward, or reduce canary percentage.

Step 1: Confirm the canary is the source:

Step 2: Characterize the error:

Step 3: Decision matrix:

Error RateTrendAction
< 0.1% above baselineStableContinue canary, investigate
0.1-1% above baselineStableReduce canary %, investigate
> 1% above baselineAnyRoll back immediately
AnyIncreasingRoll back immediately

Step 4a: Roll back (most likely action):

Step 4b: Reduce canary percentage:

Step 5: Investigate root cause before re-deploying:

Step 6: Communicate:


Q144: How do you design health checks that accurately reflect service readiness?

Answer:

Health checks have two distinct purposes — liveness and readiness — and conflating them causes problems.

Liveness probe: "Is the process alive or should it be restarted?"

Readiness probe: "Is this instance ready to receive traffic?"

Kubernetes example:

Health check endpoint implementation:

Startup probe (Kubernetes): For slow-starting services that would fail liveness during initialization:

Anti-patterns:


Q145: How do you use parallel-ssh (pssh) or similar tools to diagnose issues across a fleet of hosts?

Answer:

pssh (parallel-ssh):

Common diagnostic patterns:

pdsh (alternative, often faster):

Ansible for more complex fleet operations:

For Google-scale fleets: Tools like mosh + tmux synchronize-panes, or internal fleet management systems (Borg, Kubernetes) provide native fleet-wide querying via control plane APIs rather than SSH.


Q146: What is exponential backoff with jitter, and why is it important in distributed systems?

Answer:

Exponential backoff: After a failed request, wait before retrying. Double the wait time with each subsequent failure.

The problem with pure exponential backoff (no jitter): If 1,000 clients all started retrying at the same time (e.g., server restart), they all hit the same backoff schedule. After the wait period, they ALL retry simultaneously → server gets hit by 1,000 requests at once → server struggles → all fail again → all wait same 2s → all retry again simultaneously. This synchronized retry is itself a load problem.

Jitter: Add randomness to desynchronize retries:

Full jitter distribution: Over many clients, requests are spread uniformly across the [0, cap] window → server sees steady trickle rather than synchronized bursts.

When NOT to retry:

Circuit breaker pattern (complement to backoff):


Q147: How do you perform a safe rolling restart of a critical service across a fleet?

Answer:

A rolling restart updates/restarts instances one at a time (or in small batches) to avoid taking the entire service offline.

Kubernetes rolling restart:

Manual rolling restart with health checks:

Safety checks to add:


Q148: What on-call best practices do you follow, and how do you avoid alert fatigue?

Answer:

Alert quality principles:

1. Every alert must be actionable:

2. Alert on symptoms, not causes:

3. Tune thresholds to minimize false positives:

4. Include runbook links in every alert:

Reducing alert fatigue:

On-call health practices:

Measuring on-call health:


Q149: How do you diagnose and respond to a memory leak in a production service without restarting it?

Answer:

Step 1: Confirm and quantify the leak:

Step 2: Identify which allocations are growing:

Step 3: Mitigate without restart:

Memory limit + graceful restart (if using systemd):

Blue-green swap (zero-downtime memory drain):

cgroup memory limit as circuit breaker:

Step 4: Permanent fix:


Q150: How do you balance speed of recovery versus preserving evidence during a production incident?

Answer:

This is one of the most nuanced on-call decisions. The tension: restoring service quickly may destroy the forensic evidence needed to understand the root cause.

The guiding principle: In most cases, restore service first, then investigate. Users being down is usually more costly than a delayed root cause analysis. However, there are exceptions.

Decision framework:

Evidence preservation techniques that don't delay mitigation:

Parallel tracks approach:

When to delay recovery for evidence:

Communication during the decision:

Post-mitigation evidence gathering: Even after restoring service, evidence exists:

The act of restoring service rarely destroys all evidence — it mainly loses the live process state (memory, open files, etc.).

At Google: The standard is clear: restore service first (SLO/error budget drives urgency), then conduct thorough postmortem. kdump and pre-configured diagnostics collection scripts run automatically, so engineers don't have to choose between speed and evidence.


End of Part 2: Questions 76–150