Answer: Start with the five-step triage loop:
Establish a timeline — when did the issue start, what changed (deploy, config push, traffic spike)?
Define the blast radius — which services, users, or data are affected?
Gather signals before touching anything — metrics, logs, alerts, error rates. Changing state prematurely destroys evidence.
Form a short-list of hypotheses — ranked by likelihood and ease of verification.
Test one hypothesis at a time — binary-search the system so each test eliminates half the remaining space.
Concretely at a command line:
xxxxxxxxxx# Check recent system eventsdmesg -T | tail -50journalctl -xe --since "30 min ago"
# Network, disk, memory at a glanceuptime; free -h; df -h; netstat -s | grep -E 'failed|error'Never skip step 2 — knowing scope tells you the urgency and how aggressive your mitigation can be.
Answer: The principle is read-only first. Every command that doesn't mutate state is safe to run immediately. Commands that restart services, change configs, or kill processes require explicit justification.
Safe information-gathering commands:
xxxxxxxxxxps aux --sort=-%cpu | head -20 # CPU hogs, read-onlyss -tulnp # open socketslsof -p <pid> # file descriptors of a processstrace -p <pid> -c # syscall summary (adds overhead — note this)cat /proc/<pid>/status # memory, threads, stateRisky commands that look safe but aren't:
kill -9 — obvious
service restart — if the issue is a data-corruption bug, restart may replay the corruption
tcpdump on a heavily loaded NIC — can cause packet drops in extreme cases
Document every command run and its output in a shared incident channel so the team has a record.
Answer: In a distributed system, a request traverses N components. Divide-and-conquer means picking the midpoint of the call graph and determining whether the fault is upstream or downstream, then repeating.
Example: User reports slow checkout. The path is: Browser → CDN → Load Balancer → App Server → Auth Service → DB.
Step 1: Check the App Server response time at the LB layer. If LB logs show 2 s, the problem is at App Server or deeper.
Step 2: Add a probe call directly to the App Server, bypassing LB. If that's also 2 s, the LB is innocent.
Step 3: Check App Server logs for slow DB queries. If DB query time is 1.8 s, the problem is in the DB or between App Server and DB.
Step 4: Run EXPLAIN ANALYZE on the slow query. Missing index found — done.
This halves the search space at each step. With 6 components, worst case is 3 steps rather than 6 sequential checks.
Tools that help map the graph first:
Distributed traces (Dapper/Zipkin/Jaeger)
Service mesh telemetry (Istio, Linkerd)
Request-id correlation across logs
Answer: Hypothesis-driven debugging requires forming a falsifiable statement before running any command. "The database is slow because the connection pool is exhausted" is falsifiable — you can check pool metrics and prove or disprove it. "Let me restart things and see" is random probing.
Process:
List all hypotheses.
Rank by: (prior probability) × (ease of verification).
Design a test that, if the result goes one way, confirms the hypothesis and, if it goes the other way, conclusively rules it out.
Run the test.
Update your ranked list and repeat.
Example:
H1: DB connection pool exhausted → check SHOW STATUS LIKE 'Threads_connected' and pool metrics. Takes 5 seconds.
H2: Disk I/O saturated on DB host → iostat -x 1 5. Takes 30 seconds.
H3: Network partition between App and DB → ping, traceroute. Takes 2 minutes.
Start with H1 because it's fast and likely. This is Bayesian reasoning applied to debugging.
Answer: Escalate when:
You've exhausted your immediate diagnosis options (> 15–30 min with no progress).
The blast radius is expanding.
The fix requires permissions or resources you don't have.
The issue crosses team boundaries (your service is fine, but a dependency is misbehaving).
How to escalate effectively: Provide a structured handoff:
Symptoms — exact error messages, metrics, time of onset.
What you've ruled out — don't make the next person re-investigate dead ends.
Current best hypothesis — even if uncertain.
Actions taken — commands run, config changes made, rollbacks done.
Impact — users/QPS affected, SLO burn rate.
Use an incident management system (PagerDuty, Opsgenie) and a shared war-room channel. Never escalate silently — always provide context.
Answer: The 5 Whys is an iterative interrogation technique where you ask "why" repeatedly until you reach a root cause.
Example:
Problem: Website is down.
Why? → The application server is crashing.
Why? → It's running out of memory.
Why? → A memory leak in the new image-processing code.
Why? → The code doesn't release image buffers after processing.
Why? → No code review caught it; the diff was approved without testing memory-intensive paths.
Root cause: Inadequate code review process for memory-sensitive code.
Limitations:
Single-path bias — complex failures have multiple contributing causes. The 5 Whys finds one path through the causal tree.
Stops at the wrong layer — "human error" is rarely the root cause; it's the system that allowed the error.
Confirmation bias — the analyst's framing shapes which "why" gets asked.
Not suitable for concurrent causes — if A and B together caused C, 5 Whys misses B.
Better alternative for complex failures: Fault Tree Analysis or Event Timeline reconstruction, which maps all contributing factors in parallel.
Answer: A fix is not verified until the symptom is gone AND the root cause is confirmed eliminated. Steps:
Monitor the primary signal — the error rate, latency metric, or alert that triggered the incident. Wait for it to return to baseline for at least 2–3 measurement periods.
Check secondary signals — related metrics that should also normalize (e.g., if you fixed a DB connection leak, pool utilization and query latency should both drop).
Reproduce the original failure path (if safe) — synthetic transaction, canary request, or replay of the failing request.
Confirm the root cause mechanism is closed — if the fix was "increase connection pool size," verify the pool is no longer hitting the limit.
Watch for 5–10 minutes post-fix — some fixes mask rather than solve (e.g., restarting a process that will OOM again in 10 minutes).
xxxxxxxxxx# Example: verifying a fix for high DB connection countwatch -n 5 'mysql -e "SHOW STATUS LIKE \"Threads_connected\""'# Confirm it stays below the threshold for several minutesOnly then close the incident. Leave the monitoring dashboard up for 24 hours with an on-call watching it.
Answer: A PIR (also called postmortem or RCA doc) should be:
Blameless — the goal is systemic improvement, not finding who to punish.
Required sections:
Incident summary — one paragraph: what broke, impact, duration.
Timeline — chronological log of events, detection, diagnosis steps, mitigations. Include timestamps.
Root cause — the underlying technical and process failure(s).
Contributing factors — what made it worse or delayed detection/mitigation.
Impact — quantified: user-hours lost, SLO error budget burned, revenue affected.
What went well — detection was fast, runbooks existed, on-call had context.
What went poorly — alert fired too late, runbook was outdated, ownership unclear.
Action items — each must have: owner, deadline, and a link to a ticket. No vague "improve monitoring."
Anti-patterns to avoid:
Assigning personal blame ("X deployed bad code").
Action items with no owner.
Root cause listed as "human error" — that's a symptom, not a root cause.
Answer: Logs are most useful when you can filter, correlate, and timestamp-anchor them.
Filtering strategy:
xxxxxxxxxx# Find errors in a time windowjournalctl -u myservice --since "2026-05-17 14:00" --until "2026-05-17 14:30" -p err
# Follow logs in real-time with grepjournalctl -fu myservice | grep -E 'ERROR|WARN|exception'
# Find the first occurrence of an errorgrep -n "connection refused" /var/log/app.log | head -1
# Count error types to find the most frequentgrep ERROR /var/log/app.log | awk '{print $5}' | sort | uniq -c | sort -rnCorrelation across services:
If services emit a request_id or trace_id, grep all log files for that ID to reconstruct the full request path:
xxxxxxxxxxgrep "req-abc123" /var/log/frontend.log /var/log/backend.log /var/log/db-proxy.logStructured logs (JSON):
Use jq to filter:
xxxxxxxxxxcat app.log | jq 'select(.level=="ERROR" and .duration > 1000)'Pitfalls: Log levels misconfigured (INFO swamping ERRORs), log rotation eating the evidence, timestamps in different time zones, buffered log writes causing ordering issues.
Answer: Intermittent issues require persistent capture rather than interactive investigation.
Techniques:
Continuous metric collection — set up monitoring that records at high frequency (1 s intervals) so when it happens again, you have data:
xxxxxxxxxxsar -u -r -d 1 3600 > /tmp/sar_capture.txt & # record for 1 hourAudit logs / event tracing — enable debug logging temporarily in the suspect component with a time-bounded config (don't forget to turn it off).
Alarm-triggered capture — use a script that dumps state when a metric crosses a threshold:
xxxxxxxxxx# Triggered capture when load > 20while true; do load=$(uptime | awk '{print $NF}') if (( $(echo "$load > 20" | bc -l) )); then ps aux --sort=-%cpu > /tmp/ps_$(date +%s).txt ss -s >> /tmp/ss_$(date +%s).txt fi sleep 5doneReproduction environment — try to replicate load/traffic patterns using recorded production traffic (shadow traffic, replay tools).
Correlate with external events — cron jobs, garbage collection cycles, batch jobs, network reconfigurations. Check /var/log/cron and system event logs.
Answer:
Symptom — the observable, user-visible effect of a failure. It's what alerts fire on.
Root cause — the underlying condition that, if eliminated, prevents the symptom from recurring.
Example:
Symptom: HTTP 503 errors from the web tier.
Immediate cause: All backend app servers are refusing connections.
Deeper cause: App servers are OOM-killed by the kernel.
Deeper still: A memory leak in a library introduced in the last deploy.
Root cause: The library update bypassed integration testing which would have caught the leak.
Fixing at the symptom level (restart app servers) provides temporary relief. Fixing at the immediate cause level (increase memory limits) buys time. Only fixing the root cause prevents recurrence.
In PIRs, the goal is always root cause. In incident response, sometimes you must fix at the symptom level first (mitigate) and root-cause later (resolve).
Answer: Triage using two axes: user impact and rate of spread.
xxxxxxxxxxHigh Spread|P1: Stop | P2: Containbleeding | and fix|Low ---------|--------- HighImpact | Impact|P4: Monitor| P3: Fixand defer | next|Low Spread
Concrete rules:
If a failure is cascading (spreading to more services), stop it first even if current impact is moderate.
Among equal-severity issues, fix the one that unblocks the most other work (dependency ordering).
Data integrity failures take priority over availability failures — you can survive downtime; you can't easily recover corrupt data.
Don't split attention: assign separate responders to each incident rather than one person context-switching.
During a multi-failure event, designate an Incident Commander who doesn't touch keyboards — their job is coordination, communication, and triage decisions only.
Answer: Real-time documentation serves two purposes: knowledge sharing for others joining the incident, and evidence preservation for the postmortem.
Minimum viable war-room log:
xxxxxxxxxx14:03 [alice] Alert fired: checkout_error_rate > 1%14:05 [alice] Checked app server logs: seeing "dial tcp: connection refused" to db-primary14:07 [alice] db-primary responding to ping, but port 5432 not listening14:09 [alice] Checked db-primary: PostgreSQL service crashed. dmesg shows OOM kill at 14:0114:11 [alice] Restarted PostgreSQL. Service up. Error rate dropping.14:14 [alice] Error rate back to baseline. Mitigation complete.14:15 [alice] Root cause TBD: OOM on db-primary. Creating RCA ticket: DB-4421
Tools: Incident.io, PagerDuty Notes, a dedicated Slack channel, or a shared Google Doc. The medium matters less than the habit.
What to record: Every command run (and its output if relevant), every hypothesis tested, every action taken, timestamps on everything.
Answer: Good monitoring lets you detect degradation before users notice and provides the data needed to diagnose issues quickly.
Four golden signals (Google SRE book):
Latency — how long requests take; distinguish success latency from error latency.
Traffic — request rate, QPS, messages/sec.
Errors — rate of failed requests (5xx, timeouts, explicit errors).
Saturation — how "full" the service is (CPU %, memory %, queue depth, connection pool utilization).
Alerting design principles:
Alert on symptoms (user-visible: error rate, latency SLO), not just causes (CPU high).
Every alert should have a runbook linked.
Alert fatigue kills on-call culture — tune alert thresholds so they fire only when human action is required.
Use error budget burn rate alerts (multi-window, multi-burn-rate) rather than static thresholds for SLO-based alerting.
xxxxxxxxxx# Quick check of your monitoring coverage gaps# Are all critical services represented in Prometheus/Grafana?# Does each alert have a runbook?curl -s http://alertmanager:9093/api/v2/alerts | jq '.[].annotations.runbook_url'Answer: You've found the root cause when:
Eliminating it prevents recurrence — the test is: if we undo this single change, does the failure stop?
It explains all the symptoms — a contributing factor may explain some observations but not others.
It's a system-level explanation, not an action — "the cache invalidation logic has a race condition" is a root cause; "the engineer didn't check for the race condition" is a contributing factor.
The causal chain is complete — you can trace from root cause → intermediate effects → symptoms without gaps.
Validation test: Ask "if we fix this and only this, are we confident the issue won't recur?" If yes, that's root cause. If you're hedging ("well, it probably won't recur, but..."), keep digging.
Common traps:
Stopping at the first technical finding ("the query was slow") instead of asking why it became slow now.
Confusing the trigger (traffic spike that exposed the bug) with the root cause (the bug itself).
Multiple root causes — some incidents genuinely have two independent failures coinciding; the 5 Whys may not surface both.
Answer: Load average is the average number of runnable (running + waiting for CPU) or uninterruptible (waiting for I/O) processes over the last 1, 5, and 15 minutes. It is displayed by uptime and top.
xxxxxxxxxx$ uptime 14:23:01 up 42 days, 3:17, 2 users, load average: 2.45, 1.87, 1.32The three numbers are 1-minute, 5-minute, and 15-minute averages.
What the numbers mean:
On a single-core system, load 1.0 = 100% CPU busy. Load 2.0 = 2x overloaded.
On a multi-core system, load N = 100% across N cores. On a 16-core system, load 16.0 is fully saturated.
Rule of thumb: load average ÷ number of CPUs > 1.0 means the system is overloaded.
xxxxxxxxxx# Get CPU countnproc# orgrep -c ^processor /proc/cpuinfoTrend analysis:
Load 2.45, 1.87, 1.32 → load is increasing (recent value is highest). Investigate now.
Load 1.32, 1.87, 2.45 → load is decreasing. Peak may have passed.
Load much higher than CPU count but CPU% is low → I/O wait is a major contributor. The load includes uninterruptible sleeps.
top, htop, and ps for CPU troubleshooting?Answer:
top useful keys and flags:
xxxxxxxxxxtop -b -n 1 # batch mode, single snapshot (good for scripting)top -H -p <pid> # show threads of a specific processtop -d 0.5 # refresh every 0.5 secondsInside top:
P — sort by CPU usage
M — sort by memory usage
1 — toggle per-CPU breakdown
H — toggle thread view
c — show full command line
htop:
xxxxxxxxxxhtop -d 5 # refresh delay in tenths of seconds (5 = 0.5s)htop -p 1234,5678 # watch specific PIDshtop -u www-data # filter by userInside htop: F6 to sort, F4 to filter, F2 for setup (show tree, customize columns).
ps flags:
xxxxxxxxxxps aux --sort=-%cpu | head -20 # top CPU consumersps aux --sort=-%mem | head -20 # top memory consumersps -eLf | grep <pid> # threads of a process (-L = LWP)ps -o pid,ppid,cmd,%cpu,%mem,stat -p <pid> # specific fields for a PIDps axo pid,stat,pcpu,comm | grep -v ' S\| I' # show running/zombie processesstat column values:
R = running, S = sleeping, D = uninterruptible sleep (I/O wait), Z = zombie, T = stopped.
Answer: CPU steal time (%st in top) is the percentage of time a virtual CPU is waiting for the hypervisor to give it real CPU time. This happens when the physical host is overcommitted — too many VMs competing for physical cores.
Where to see it:
xxxxxxxxxxtop # look at %st column in the CPU summary linevmstat 1 # st column in the cpu sectionsar -u 1 5 # %steal columnExample top header:
xxxxxxxxxx%Cpu(s): 45.2 us, 2.1 sy, 0.0 ni, 40.0 id, 5.3 wa, 0.0 hi, 2.4 si, 5.0 st
5% steal means the VM is losing 5% of its time waiting for the hypervisor.
What to do:
Steal > 5–10% consistently → the hypervisor host is overloaded. Options:
Move the VM to a less-loaded host (live migration).
Request a dedicated host or host tenancy (AWS dedicated instances).
Scale out — add more VMs and distribute the load.
Cannot be fixed by tuning inside the VM — it's a hypervisor-level resource issue.
Important: High steal time can cause mysterious latency spikes and load average increases without apparent CPU-bound processes inside the VM.
nice and renice to manage process priority?Answer: nice values in Linux range from -20 (highest priority) to +19 (lowest priority). Default is 0. Only root can set negative (higher) priority.
nice — start a process with a given niceness:
xxxxxxxxxxnice -n 10 ./backup_script.sh # run backup at lower prioritynice -n -5 ./critical_service # run at higher priority (root only)renice — change priority of a running process:
xxxxxxxxxxrenice +15 -p 1234 # lower priority of PID 1234renice -5 -p 1234 # raise priority (root only)renice +10 -u www-data # lower priority for all processes of user www-datarenice +10 -g 500 # lower priority for all processes of group 500Verify:
xxxxxxxxxxps -o pid,ni,comm -p 1234# or in top: look at the NI columnPractical use case: During a production incident where a runaway analytics job is consuming CPU alongside critical services, renice the analytics job to +19 to give back CPU:
xxxxxxxxxxrenice +19 -p $(pgrep analytics_job)Note: nice/renice affect SCHED_OTHER (CFS) processes. For real-time scheduling, use chrt.
perf top to identify CPU hotspots?Answer: perf top provides a real-time view of which functions (in user space and kernel) are consuming the most CPU cycles, by sampling hardware performance counters.
xxxxxxxxxx# Basic usage — requires root or perf_event_paranoid <= 1perf top
# Focus on a specific processperf top -p <pid>
# Show call graphs (requires frame pointers or DWARF)perf top -g -p <pid>
# Count specific hardware eventsperf top -e cache-missesperf top -e instructions,cyclesOutput columns:
xxxxxxxxxxOverhead Shared Object Symbol42.33% [kernel] [k] __softirqentry_text_start18.21% libc-2.31.so [.] malloc9.87% myapp [.] process_packet
Overhead = percentage of samples this function appeared in (proxy for CPU time).
Workflow for investigating high CPU:
perf top -p <pid> — find the hot function.
perf record -g -p <pid> sleep 30 — record a detailed profile.
perf report — interactive call-graph browser.
If the hot function is in kernel space, check if it's a driver issue or system call overhead.
Prerequisites: linux-tools-$(uname -r) package, kernel symbols (/proc/kallsyms), and ideally debug symbols for userspace binaries.
strace -p to investigate a running process?Answer: strace intercepts and records system calls made by a process. Attaching to a running process with -p is a key diagnostic tool.
xxxxxxxxxx# Attach to running processstrace -p <pid>
# Attach to all threads of a processstrace -p <pid> -f
# Summary of syscall counts and time (less intrusive, good for production)strace -p <pid> -c -f
# Show timestamps and duration for each syscallstrace -p <pid> -T -tt
# Filter to specific syscallsstrace -p <pid> -e trace=read,write,open,connect
# Trace file operations onlystrace -p <pid> -e trace=file
# Write output to a file (don't let it scroll past)strace -p <pid> -o /tmp/strace_output.txtReading output:
xxxxxxxxxx14:23:01.123456 read(5, "", 4096) = 0 <0.000021>14:23:01.123500 futex(0x7f..., FUTEX_WAIT, ...) = -1 ETIMEDOUT <5.000123>
The second line shows the process blocked on futex for 5 seconds — a lock contention issue.
Warning: strace adds significant overhead (10–100x slowdown for I/O-heavy processes) due to ptrace. Use -c mode in production to minimize impact. Prefer perf trace for lower overhead.
Answer: A zombie process is a process that has terminated but whose entry remains in the process table because its parent has not yet called wait() to collect its exit status.
How it forms:
Child process exits and sends SIGCHLD to parent.
Parent doesn't call wait() or waitpid() to read the exit code.
The kernel keeps a minimal process table entry (just PID, exit status, resource usage) until the parent retrieves it.
Identifying zombies:
xxxxxxxxxxps aux | grep Z # Z in the STAT columnps axo pid,ppid,stat,comm | grep Z # also shows parent PIDCleanup options:
Send SIGCHLD to the parent — tells it to reap children:
xxxxxxxxxxkill -s SIGCHLD <parent_pid>Kill the parent — when the parent dies, init (PID 1 / systemd) adopts and reaps the zombie:
xxxxxxxxxxkill -9 <parent_pid>Fix the code — the real fix is ensuring the parent calls waitpid() or installs a SIGCHLD handler.
Impact: A few zombies are harmless. Thousands can exhaust the PID space (default max: 32768, check cat /proc/sys/kernel/pid_max), preventing new processes from starting.
Answer: A runaway process is one consuming excessive resources (CPU, memory, file descriptors, sockets) unexpectedly.
Identification:
xxxxxxxxxx# Top CPU consumersps aux --sort=-%cpu | head -10
# Top memory consumers ps aux --sort=-%mem | head -10
# Process tree to understand parent/child relationshipspstree -p
# How long has a process been running?ps -o pid,etime,cmd -p <pid>
# What files/sockets does it have open?lsof -p <pid> | wc -lGraceful stop sequence:
xxxxxxxxxx# Step 1: SIGTERM — polite request to shut downkill -15 <pid># Wait 5-10 seconds
# Step 2: Check if still runningps -p <pid>
# Step 3: If still running, SIGKILL — cannot be caught or ignoredkill -9 <pid>
# For a process group (kills parent and all children):kill -9 -<pgid> # negative PID = process groupCaution before killing:
Is this process holding a lock (database, file lock)? Killing it may leave the lock held.
Is it in the middle of a write? kill -9 doesn't allow cleanup — possible data corruption.
Does it have children that need to die too? Check with pstree -p <pid>.
Prefer SIGTERM first to allow graceful shutdown.
vmstat to investigate context switching?Answer: vmstat reports virtual memory statistics and includes context switch information.
xxxxxxxxxxvmstat 1 10 # report every 1 second, 10 timesOutput columns relevant to context switching:
xxxxxxxxxxprocs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----r b swpd free buff cache si so bi bo in cs us sy id wa st2 0 0 1234567 12345 234567 0 0 0 12 450 1200 25 5 70 0 0
cs — context switches per second. High values indicate many processes competing for CPU.
in — interrupts per second (includes timer interrupts).
r — run queue length: processes waiting for CPU. If consistently > number of CPUs, system is CPU-saturated.
b — blocked processes (uninterruptible sleep, usually I/O).
Normal vs. problematic:
Context switches of 1,000–10,000/s are normal for a busy server.
Context switches of 100,000+/s with high sy (system time) may indicate a problem (too many threads, lock contention, excessive signals).
Corroborate with pidstat:
xxxxxxxxxxpidstat -w 1 # context switches per process per second# cswch/s = voluntary context switches (process yielded, e.g., waiting for I/O)# nvcswch/s = involuntary context switches (preempted by scheduler)High involuntary switches = too many threads competing for CPU. High voluntary switches = I/O-bound processes.
Answer: In top and vmstat, CPU time is broken into:
us — user space: application code.
sy — system (kernel) space: syscalls, kernel threads.
ni — niced user processes.
wa — I/O wait.
hi — hardware interrupt handling.
si — software interrupt (softirq) handling.
Diagnosing high sy:
xxxxxxxxxx# Find which syscalls are being called mostperf top -e raw_syscalls:sys_enterstrace -c -p <pid> # syscall summary for one process
# Kernel thread CPU usageps aux | grep '\[.*\]' # kernel threads shown in bracketsDiagnosing high hi or si:
Usually network-related. si high often means high network packet rate:
xxxxxxxxxxcat /proc/interrupts # per-CPU interrupt countscat /proc/softirqs # softirq counts (NET_RX, NET_TX, etc.)ethtool -S eth0 # NIC statisticsDiagnosing high us:
Use perf top, profiler (gprof, async-profiler for Java), or flame graphs:
xxxxxxxxxxperf record -g -p <pid> sleep 30perf script | stackcollapse-perf.pl | flamegraph.pl > flame.svgAnswer:
CPU saturation — the system has more work to do than CPU capacity. Characterized by:
Load average > number of CPUs.
Run queue r in vmstat consistently > CPU count.
Processes waiting for CPU time.
CPU throttling — a hard limit is imposed on CPU usage by the scheduler. Common in containerized environments (cgroups).
xxxxxxxxxx# Check cgroup CPU quota (container)cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us # microseconds of CPU per periodcat /sys/fs/cgroup/cpu/cpu.cfs_period_us # period in microseconds# Quota 50000 / Period 100000 = 50% of one CPU allowed
# Check throttling statisticscat /sys/fs/cgroup/cpu/cpu.stat# nr_periods — total CFS periods elapsed# nr_throttled — number of periods this cgroup was throttled# throttled_time — total nanoseconds throttledIn Kubernetes:
xxxxxxxxxxkubectl top pod <pod-name>kubectl describe pod <pod-name> # look at Limits and RequestsA container at 100% of its CPU limit is throttled even if the host has idle CPU. The fix is to raise the CPU limit (or request). This is a very common source of latency spikes in containerized services.
Answer: CPU affinity controls which CPUs a process is allowed to run on. Incorrect affinity can cause unnecessary context switches, NUMA effects, and cache misses.
Check current affinity:
xxxxxxxxxxtaskset -p <pid> # shows hex bitmask of allowed CPUstaskset -cp <pid> # shows CPU list formatSet affinity:
xxxxxxxxxx# Run new process on CPUs 0 and 1 onlytaskset -c 0,1 ./myprocess
# Change affinity of running processtaskset -cp 0,1 <pid>NUMA topology:
xxxxxxxxxxnumactl --hardware # show NUMA nodes and their CPUs/memorynumactl --show # show current process NUMA policynumastat # show NUMA memory allocation statisticsWhy it matters: On a multi-socket server, a process migrating between NUMA nodes incurs remote memory access latency (~2x slower than local). Pin latency-sensitive processes to a single NUMA node:
xxxxxxxxxxnumactl --cpunodebind=0 --membind=0 ./latency_sensitive_serviceAnswer: High sy CPU often means excessive syscall rate. Tools to investigate:
xxxxxxxxxx# Per-process syscall frequency summary (safe for production)strace -c -f -p <pid>
# System-wide syscall rate with perfperf stat -e 'raw_syscalls:sys_enter' -a sleep 5
# Find the top syscall-making processesperf top -e raw_syscalls:sys_enter
# ftrace — lower overhead than straceecho function > /sys/kernel/debug/tracing/current_tracerecho 1 > /sys/kernel/debug/tracing/tracing_oncat /sys/kernel/debug/tracing/trace_pipeCommon culprits:
Excessive gettimeofday/clock_gettime — use vDSO; if not, consider caching timestamps.
Excessive write syscalls — application flushing logs unbuffered; add buffering.
futex contention — lock contention in multi-threaded app; reduce lock granularity.
mmap/mprotect — JIT compilers, dynamic linker; profile the application.
epoll_wait with many spurious wakeups — edge-triggered epoll misconfiguration.
Answer: Flame graphs (invented by Brendan Gregg) show the call stack hierarchy, with width proportional to time spent.
Reading a flame graph:
X-axis — alphabetical ordering of stack frames (not time).
Y-axis — call stack depth (bottom = on-CPU, top = what was running).
Width — proportion of total samples that included this frame.
A wide plateau at the top is a hot function worth optimizing.
Generating:
xxxxxxxxxx# CPU flame graph with perfperf record -F 99 -a -g sleep 30perf script > perf.outgit clone https://github.com/brendangregg/FlameGraph./FlameGraph/stackcollapse-perf.pl perf.out > perf.folded./FlameGraph/flamegraph.pl perf.folded > cpu_flame.svg
# For Java (async-profiler)./profiler.sh -d 30 -f flame.html <pid>Patterns to look for:
Tall stacks with wide bottoms → deep call chains consuming time.
Unexpected library (libc malloc wide) → memory allocation overhead.
Kernel frames wide → syscall-heavy or interrupt-heavy.
"Broken" stacks (lost frames) → missing frame pointers; recompile with -fno-omit-frame-pointer.
Answer: When the sum of per-process CPU in ps doesn't explain total system CPU, investigate these causes:
1. Kernel threads consuming CPU:
xxxxxxxxxxps aux | grep '\[' | sort -k3 -rn | head -10# Common offenders: kswapd (paging), ksoftirqd (softirqs), kworker, jbd2 (journal)2. Short-lived processes not captured by ps:
xxxxxxxxxx# Use execsnoop (from bcc-tools) to trace all exec() callsexecsnoop
# Or audit with process accountingaccton /var/log/pacctlastcomm --file /var/log/pacct | head -303. Interrupt and softirq load:
xxxxxxxxxxwatch -n 1 'cat /proc/softirqs'watch -n 1 'cat /proc/interrupts | head -20'4. CPU frequency scaling (not actually high load):
xxxxxxxxxxcpupower frequency-info # current frequencycat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freqIf CPUs are throttled to low frequency, %CPU numbers are high but work done is low.
5. Cgroup accounting mismatch in containers — use systemd-cgtop or docker stats to get cgroup-level CPU usage.
Answer: When the kernel OOM killer terminates a process, it logs detailed information.
xxxxxxxxxx# Recent OOM eventsdmesg -T | grep -i 'oom\|killed process\|out of memory'
# Full OOM event with contextdmesg -T | grep -A 20 'Out of memory'
# Via journalctl (systemd systems)journalctl -k | grep -i oomjournalctl -k --since "1 hour ago" | grep -i 'killed process'Typical OOM log output:
xxxxxxxxxx[Mon May 17 14:23:01 2026] Out of memory: Kill process 12345 (java) score 847 or sacrifice child[Mon May 17 14:23:01 2026] Killed process 12345 (java) total-vm:8192000kB, anon-rss:7168000kB, file-rss:45000kB, shmem-rss:0kB
Key fields:
score — OOM score (0–1000); higher = more likely to be killed. Based on memory usage relative to system total.
total-vm — virtual memory size of the killed process.
anon-rss — actual physical anonymous memory (heap, stack).
file-rss — physical memory backed by files (mmap'd files, shared libraries).
Check a process's OOM score:
xxxxxxxxxxcat /proc/<pid>/oom_score # current scorecat /proc/<pid>/oom_score_adj # adjustment (-1000 to 1000)Protect a critical process from OOM kill:
xxxxxxxxxxecho -1000 > /proc/<pid>/oom_score_adj # very unlikely to be killed (root only)/proc/meminfo and what do they mean?Answer:
xxxxxxxxxxcat /proc/meminfoKey fields:
| Field | Meaning |
|---|---|
MemTotal | Total physical RAM |
MemFree | Completely unused RAM |
MemAvailable | RAM available for new allocations (includes reclaimable cache) — more useful than MemFree |
Buffers | Kernel buffers for raw disk block data |
Cached | Page cache (file contents cached in RAM) |
SwapCached | Data in swap that's also in RAM (being swapped back in) |
Active | Recently used memory, not reclaimed |
Inactive | Less recently used, can be reclaimed |
Active(anon) | Anonymous active pages (heap/stack) |
Active(file) | File-backed active pages |
SwapTotal / SwapFree | Swap space total and free |
Dirty | Memory waiting to be written to disk |
Writeback | Memory actively being written to disk |
Slab | Kernel slab allocator memory |
SReclaimable | Slab memory that can be reclaimed |
SUnreclaim | Slab memory that cannot be reclaimed |
HugePages_Total | Number of huge pages allocated |
VmallocTotal / VmallocUsed | Virtual memory allocated by vmalloc |
Most important for "is the system low on memory?"
Use MemAvailable, not MemFree. A system with 100 MB MemFree but 8 GB MemAvailable is fine — the kernel will reclaim cache as needed.
Answer: The slab allocator is the kernel's internal object cache. It pre-allocates and pools frequently used kernel objects (dentries, inodes, socket buffers, etc.) to avoid the overhead of constant allocation/deallocation.
xxxxxxxxxx# View slab usageslabtop -s c # sort by cache size
# Via /proccat /proc/slabinfo
# Summarycat /proc/meminfo | grep SlabCommon high-memory slab caches:
dentry — directory entry cache. High if working with millions of files.
inode_cache — cached inodes.
ext4_inode_cache — filesystem-specific inode cache.
sock_inode_cache — socket inodes (high on servers with many connections).
kmalloc-* — general-purpose kernel allocations.
Why it causes problems:
Slab memory is counted as "used" but much of it is SReclaimable. Under memory pressure, the kernel will reclaim it, but this can cause spikes in latency (dentry/inode cache misses).
Force reclaim (use cautiously — causes latency spikes):
xxxxxxxxxxecho 2 > /proc/sys/vm/drop_caches # drop dentries and inodesecho 3 > /proc/sys/vm/drop_caches # drop page cache + dentries + inodesInvestigating dentry bloat (e.g., log rotation creating/deleting millions of files):
xxxxxxxxxxslabtop -s c | grep dentry# If dentry cache is in GB, investigate file creation patternsAnswer: Swap allows the kernel to move inactive memory pages to disk to free RAM for active use.
xxxxxxxxxx# Swap summaryfree -hswapon -s # active swap devices and their usage
# Real-time swap activity (si = swap in, so = swap out)vmstat 1# si/so in KB/s — non-zero means active swapping
# Which processes are using swapfor pid in /proc/[0-9]*/status; do awk '/VmSwap/{print FILENAME, $2, $3}' $pid 2>/dev/nulldone | sort -k2 -rn | head -10
# Or with smemsmem -s swap -r | head -20When swap is a problem:
so (swap out) in vmstat is non-zero consistently → system is actively paging out, meaning RAM is insufficient.
High swap usage + high si (swap in) → thrashing: system is swapping pages in and out repeatedly, causing massive I/O.
Swap usage alone (without active si/so) is less concerning — it may just be cold memory that was paged out long ago.
Swappiness tuning:
xxxxxxxxxxcat /proc/sys/vm/swappiness # default 60 (0-100)# Lower value = kernel less eager to swap. For database servers: 1-10sysctl vm.swappiness=10On servers where swap causes latency problems:
xxxxxxxxxxswapoff -a # disable swap (only if enough RAM)Answer: A memory leak is characterized by continuously growing RSS (resident set size) without corresponding growth in workload.
Detection:
xxxxxxxxxx# Watch RSS growth over timewatch -n 5 'ps -o pid,rss,vsz,cmd -p <pid>'
# More detail with /proccat /proc/<pid>/status | grep -E 'VmRSS|VmSize|VmPeak|RssAnon|RssFile'
# pmap — show memory mapping summarypmap -x <pid> | tail -3 # summary at the bottom (RSS, Dirty)
# Track heap growthpmap -x <pid> | grep -E 'heap|anon'Valgrind (for development/testing — not production):
xxxxxxxxxxvalgrind --leak-check=full --track-origins=yes ./myapp 2>&1 | tee valgrind.outAddressSanitizer (compile-time instrumentation):
xxxxxxxxxxgcc -fsanitize=address -g -o myapp myapp.c./myapp # leak report on exitheaptrack (lower overhead than valgrind):
xxxxxxxxxxheaptrack ./myappheaptrack_gui heaptrack.myapp.*.gzFor production systems — periodic pmap snapshots:
xxxxxxxxxx# Capture pmap at intervalswhile true; do pmap -x <pid> > /tmp/pmap_$(date +%s).txt sleep 60done# Then diff successive snapshots to find growing regionsFor Java: Use jmap -heap <pid>, jcmd <pid> VM.native_memory, or attach JVisualVM/async-profiler.
Answer: Standard Linux page size is 4 KB. Huge pages are 2 MB (or 1 GB on x86-64), which reduces Translation Lookaside Buffer (TLB) entries needed for large memory regions, decreasing TLB miss overhead.
When to use:
Databases (PostgreSQL, Oracle, MySQL) with large buffer pools.
In-memory caches with large datasets.
HPC applications processing large arrays.
Any application mapping > 1 GB of memory that benefits from reduced TLB pressure.
Types:
Static huge pages — pre-allocated at boot, never swapped.
Transparent Huge Pages (THP) — kernel automatically promotes 4K pages to 2MB pages. Can cause latency spikes during compaction.
Configure static huge pages:
xxxxxxxxxx# Check current huge page statecat /proc/meminfo | grep -i hugegrep -i hugepage /proc/meminfo
# Allocate 512 huge pages (512 × 2MB = 1 GB)echo 512 > /proc/sys/vm/nr_hugepages
# Persistent (sysctl.conf)echo "vm.nr_hugepages = 512" >> /etc/sysctl.confsysctl -p
# Mount hugetlbfsmount -t hugetlbfs none /mnt/hugepagesTHP control:
xxxxxxxxxxcat /sys/kernel/mm/transparent_hugepage/enabled# Options: always, madvise, neverecho madvise > /sys/kernel/mm/transparent_hugepage/enabled# madvise = only for regions that explicitly request huge pages with madvise(MADV_HUGEPAGE)For databases, typically disable THP to avoid latency spikes:
xxxxxxxxxxecho never > /sys/kernel/mm/transparent_hugepage/enabledecho never > /sys/kernel/mm/transparent_hugepage/defragvmstat -s to get a memory snapshot?Answer: vmstat -s prints a summary table of memory and virtual memory statistics.
xxxxxxxxxxvmstat -sKey output fields:
xxxxxxxxxx16384000 K total memory1234567 K used memory8901234 K active memory4567890 K inactive memory1234567 K free memory123456 K buffer memory5678901 K swap cache8388608 K total swap0 K used swap8388608 K free swap1234567 non-nice user cpu ticks5678 nice user cpu ticks234567 system cpu ticks12345678 idle cpu ticks123456 IO-wait cpu ticks1234 IRQ cpu ticks12345 softirq cpu ticks0 stolen cpu ticks1234567 pages paged in234567 pages paged out0 pages swapped in0 pages swapped out1234567 interrupts2345678 CPU context switches1716000000 boot time123456 forks
Key metrics to check:
pages swapped in/out — non-zero means the system is swapping.
pages paged in/out — normal I/O paging.
CPU context switches — cumulative since boot.
used memory — note this includes buffers/cache; use MemAvailable from /proc/meminfo for free memory assessment.
sar -r for memory monitoring over time?Answer: sar (System Activity Reporter) from the sysstat package records and reports system activity. -r shows memory utilization.
xxxxxxxxxx# Real-time memory stats every 1 second, 10 samplessar -r 1 10
# Review today's historical datasar -r
# Review a specific date (data stored in /var/log/sa/)sar -r -f /var/log/sa/sa17 # May 17th data
# Memory + swap combinedsar -r -S 1 10Output columns (sar -r):
xxxxxxxxxxTime kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty14:00:01 123456 8901234 8765432 53.56 123456 5678901 12345678 75.46 8901234 456789 12345
kbavail — available memory (analogous to MemAvailable).
%memused — percentage of total RAM used.
%commit — percentage of total memory (RAM + swap) committed by processes. > 100% means overcommit.
kbdirty — dirty pages waiting to be flushed to disk. High values indicate write-back bottleneck.
Set up sar collection (if not already running):
xxxxxxxxxx# Enable sysstat data collectionsystemctl enable --now sysstat# Data is collected by cron: /etc/cron.d/sysstatAnswer: Linux allows processes to mmap/malloc more virtual memory than physically exists (RAM + swap). This works because most allocated memory is never actually used simultaneously (sparse allocation).
Overcommit modes (/proc/sys/vm/overcommit_memory):
| Value | Mode | Behavior |
|---|---|---|
| 0 | Heuristic | Default. Allow overcommit up to a heuristic limit. |
| 1 | Always | Always allow allocation. Never fails malloc. Used in some HPC scenarios. |
| 2 | Never | Fail allocation if committed memory would exceed overcommit_ratio% of RAM + swap. |
xxxxxxxxxxcat /proc/sys/vm/overcommit_memorycat /proc/sys/vm/overcommit_ratio # default 50 (50% of RAM)Mode 2 calculation: Max commit = swap_total + (overcommit_ratio/100) × RAM_total
Check committed memory:
xxxxxxxxxxgrep CommitLimit /proc/meminfo # max allowed committedgrep Committed_AS /proc/meminfo # currently committedRelevance to OOM: With overcommit enabled (mode 0 or 1), processes can allocate more than exists. When they actually touch the pages, the kernel must provide physical memory. If it can't, the OOM killer activates. Mode 2 prevents this by failing the allocation upfront.
Answer:
Find the top memory consumers:
xxxxxxxxxx# Sort by RSS (resident set size — actual physical memory)ps aux --sort=-%mem | head -15ps -eo pid,ppid,cmd,%mem,rss --sort=-%mem | head -15
# Include swap usagesmem -s rss -r | head -15
# For per-process memory breakdown (private vs shared)smem -r -t | head -15 # -t shows totals# USS = Unique Set Size (private only, most accurate for "how much I'm using")# PSS = Proportional Set Size (USS + shared/number_sharing_it)# RSS = USS + all sharedIdentifying the right process to kill:
OOM score — the kernel uses this; higher score = better kill candidate.
xxxxxxxxxxfor p in /proc/[0-9]*/oom_score; do printf '%s %s\n' $(cat $p) $p; done | sort -rn | head -10Priority of service — killing a cache service (memcached) is better than killing the database.
Recovery behavior — will it restart? Will data be lost?
Memory vs. importance — don't kill a small-footprint critical service to save a large-footprint non-critical one.
Manual kill with grace:
xxxxxxxxxxkill -15 <pid> # SIGTERM firstsleep 10kill -9 <pid> # SIGKILL if necessaryAnswer:
| Metric | Full Name | Meaning | Use Case |
|---|---|---|---|
| VSZ | Virtual Size | Total virtual address space mapped by the process. Includes code, heap, stack, mmap'd files, shared libraries. Most inflated figure. | Rarely useful alone |
| RSS | Resident Set Size | Physical memory currently in RAM. Includes shared pages counted multiple times (once per process). | Quick gauge; overestimates total system usage |
| PSS | Proportional Set Size | RSS but shared pages divided by number of sharers. More accurate total memory footprint. | Accurate per-process memory cost |
| USS | Unique Set Size | Only private (non-shared) pages. Memory freed if this process dies. | True memory freed by killing a process |
Example: 10 processes each share a 100 MB library.
RSS per process: 100 MB (shared) + private pages → inflated
PSS per process: 10 MB (100/10) + private pages → accurate
USS per process: only private pages → accurate for kill impact
xxxxxxxxxx# Get PSS and USS for a processcat /proc/<pid>/smaps_rollup# orsmem -P myapp -c "pid name uss pss rss"In practice: Use RSS for quick triage, USS for understanding kill impact, PSS for system-wide memory accounting.
Answer: Memory fragmentation occurs when free memory exists but not in contiguous blocks of the required size, causing allocation failures or performance degradation.
Check fragmentation:
xxxxxxxxxx# Buddy allocator fragmentation infocat /proc/buddyinfo# Shows free blocks by order (powers of 2) for each NUMA zone# Example: Node 0, zone Normal 23 15 10 8 5 3 2 1 0 0 0# Each column is 2^n pages free. Last column (order 10 = 4MB) being 0 is bad for huge pages.
# External fragmentation index (higher = more fragmented)cat /proc/pagetypeinfoCompaction:
xxxxxxxxxx# Manually trigger memory compaction (reclaim and compact free pages)echo 1 > /proc/sys/vm/compact_memory # kernel 2.6.35+Symptoms of fragmentation:
Huge page allocation fails despite enough free memory.
dmesg shows page allocation failure for high-order allocations.
vmalloc failures.
Long-term mitigation:
Pre-allocate huge pages at boot (before fragmentation occurs).
Enable khugepaged for THP compaction.
vm.min_free_kbytes — increase to keep more contiguous free pages:
xxxxxxxxxxsysctl vm.min_free_kbytes=524288 # 512 MB minimum free/proc filesystem polling?Answer: For lightweight, scriptable memory monitoring without installing tools:
xxxxxxxxxx# Snapshot current process memorycat /proc/<pid>/status | grep -E 'VmRSS|VmSize|VmPeak|VmSwap|RssAnon'
# Detailed memory maps (each mapping line)cat /proc/<pid>/smaps | grep -E 'Size|Rss|Pss|Private'
# Rollup summary (kernel 4.14+)cat /proc/<pid>/smaps_rollup
# System-widecat /proc/meminfo
# NUMA-specific memory statscat /proc/<pid>/numa_maps | head -20Polling script for trend analysis:
xxxxxxxxxxPID=$1LOG=/tmp/mem_trend_${PID}.csvecho "timestamp,rss_kb,vsz_kb,swap_kb" > $LOGwhile true; do TS=$(date +%s) RSS=$(awk '/VmRSS/{print $2}' /proc/$PID/status 2>/dev/null) VSZ=$(awk '/VmSize/{print $2}' /proc/$PID/status 2>/dev/null) SWAP=$(awk '/VmSwap/{print $2}' /proc/$PID/status 2>/dev/null) echo "$TS,$RSS,$VSZ,$SWAP" >> $LOG sleep 30doneImport to gnuplot or Python pandas for visualization. A steadily increasing RSS without workload increase is a memory leak.
Answer: The kernel uses a Least Recently Used (LRU) page replacement approximation via two lists: Active and Inactive.
LRU mechanism:
New pages enter the Inactive list.
If a page is accessed again while in the Inactive list, it moves to the Active list.
Under pressure, pages are moved from Active → Inactive → evicted.
Anonymous pages (heap/stack) go to swap when evicted.
File-backed pages are simply dropped (they can be re-read from disk).
Key sysctl parameters:
xxxxxxxxxx# Tendency to reclaim anonymous vs. file pagescat /proc/sys/vm/swappiness # 0=prefer file reclaim, 100=equal
# Minimum free memory the kernel tries to maintaincat /proc/sys/vm/min_free_kbytes
# Ratio of active to inactive anonymous pagescat /proc/sys/vm/inactive_ratiokswapd — background page reclaim daemon:
xxxxxxxxxx# Watch kswapd activityvmstat 1 | awk '{print $7, $8}' # si (swap in), so (swap out)# Orsar -B 1 5 # pgscank (pages scanned by kswapd), pgsteal (pages reclaimed)Direct reclaim occurs when an allocation can't be satisfied immediately and the calling process performs reclaim inline — this causes high latency spikes in applications. Goal: keep memory pressure low enough that kswapd handles reclaim in the background.
Answer: The OOM killer selects a victim based on oom_score (0–1000). The process with the highest score is killed.
Score factors:
Proportional to memory usage (RSS + swap) relative to total memory.
Adjusted by oom_score_adj (-1000 to +1000).
Protect a critical process:
xxxxxxxxxx# Make it almost immune (score adjusted to 0 at minimum)echo -1000 > /proc/<pid>/oom_score_adj# Persist for a service in its unit file:# [Service]# OOMScoreAdjust=-1000Make a process more likely to be killed (sacrifice non-critical):
xxxxxxxxxxecho 500 > /proc/<pid>/oom_score_adj # increases score by 500 points# In systemd unit:# OOMScoreAdjust=500System-wide OOM behavior:
xxxxxxxxxx# Panic instead of killing (for systems where corruption is worse than crash)sysctl vm.panic_on_oom=1
# OOM killer policycat /proc/sys/vm/oom_kill_allocating_task # 0=kill highest scorer, 1=kill the task that triggered OOMView current OOM scores:
xxxxxxxxxxfor p in $(ls /proc | grep '^[0-9]'); do score=$(cat /proc/$p/oom_score 2>/dev/null) cmd=$(cat /proc/$p/comm 2>/dev/null) echo "$score $p $cmd"done | sort -rn | head -20df -h and df -i, and when does inode exhaustion matter?Answer:
df -h — shows disk space usage (blocks) in human-readable format:
xxxxxxxxxxdf -h# Filesystem Size Used Avail Use% Mounted on# /dev/sda1 50G 45G 5.0G 90% /df -i — shows inode usage:
xxxxxxxxxxdf -i# Filesystem Inodes IUsed IFree IUse% Mounted on# /dev/sda1 3276800 3276799 1 100% /What is an inode? Each file and directory requires one inode (metadata entry: permissions, timestamps, size, block pointers). The number of inodes is fixed at filesystem creation time (for ext3/ext4).
When inode exhaustion matters: You can have plenty of free disk space but be unable to create new files if inodes are exhausted. This is most common when:
A process creates millions of small files (log files, mail queues, temp files, session files).
A PHP/Python application creates a session file per user.
Misconfigured log rotation leaving many small log files.
Symptoms:
xxxxxxxxxxtouch: cannot touch 'newfile': No space left on device
But df -h shows plenty of free space → check df -i.
Fix:
xxxxxxxxxx# Find directories with the most filesfind / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -rn | head -20
# Delete unnecessary small files# On ext4, you can't easily increase inode count without reformatting# Consider using XFS (dynamic inode allocation) or migrating to a new filesystemdu -sh to find disk space consumers?Answer: du (disk usage) reports the disk space used by files and directories.
xxxxxxxxxx# Size of current directorydu -sh .
# Size of all subdirectories in /vardu -sh /var/*
# Find top 10 largest directories under /du -sh /* 2>/dev/null | sort -h | tail -10
# Recursive breakdown, sorteddu -h --max-depth=2 /var | sort -h | tail -20
# Find files larger than 1 GBfind / -xdev -size +1G -type f -ls 2>/dev/null | sort -k7 -rnEfficient large-tree scan:
xxxxxxxxxx# Use -x to stay on one filesystem (don't cross mounts)du -xsh /var/* | sort -h
# ncdu — interactive disk usage viewerncdu /varCommon space hogs:
/var/log — log files; check for unrotated or misconfigured logs.
/var/lib/docker — container images and volumes.
/tmp — temporary files; should be cleaned on reboot.
/home — user files.
Core dumps: find / -name 'core' -o -name 'core.[0-9]*' 2>/dev/null
lsof to find open-but-deleted files still consuming disk space?Answer: When a file is deleted (unlinked), the disk space is not freed until all file descriptors pointing to it are closed. If a process has a large log file open and the file is deleted, the space remains consumed until the process closes the file or is restarted.
Find open-but-deleted files:
xxxxxxxxxx# Show all deleted files still openlsof | grep '(deleted)'
# Focus on large deleted fileslsof | grep '(deleted)' | awk '{print $7, $1, $2}' | sort -rn | head -10
# Show size explicitlylsof -nP | awk '/deleted/{print $7"\t"$1"\t"$2"\t"$NF}' | sort -rn | head -10Example output:
xxxxxxxxxxjava 12345 tomcat 123u REG 8,1 5368709120 ... /var/log/app.log (deleted)
5 GB log file deleted but still held open by java PID 12345.
Fix without killing the process:
xxxxxxxxxx# Truncate the file via its file descriptor (frees the space)> /proc/12345/fd/123 # truncate to 0 bytes
# Or find the fd number firstls -la /proc/12345/fd/ | grep deletedFix by restarting (if safe):
xxxxxxxxxxsystemctl restart tomcat # process closes the file descriptor on exitiostat and iotop?Answer: I/O wait (%iowait in top, wa in vmstat) indicates CPU time spent waiting for I/O operations to complete.
iostat — device-level I/O statistics:
xxxxxxxxxx# Install: sysstat packageiostat -x 1 10 # extended stats, 1-second intervals, 10 samples
# Key columns in extended output:# r/s, w/s — reads and writes per second# rkB/s, wkB/s — KB/s read and written# await — average time (ms) for I/O requests to complete# r_await, w_await — separate read/write await times# %util — percentage of time device was busy (100% = saturated)# svctm — service time (deprecated, unreliable on SSDs)Interpreting:
%util near 100% → disk is saturated.
await > 10 ms for SSD or > 20 ms for HDD is elevated; > 100 ms is a problem.
High r/s with high await → too many concurrent reads; consider queue depth tuning.
iotop — process-level I/O:
xxxxxxxxxx# Install: iotop package (or iotop-c on newer systems)iotop -o # show only processes doing I/O (-o = only)iotop -boa # batch mode, only active, accumulated
# Key columns: TID, PRIO, USER, DISK READ, DISK WRITE, SWAPIN, IO, COMMANDpidstat -d — per-process I/O from sysstat:
xxxxxxxxxxpidstat -d 1 5 # disk I/O per process, 1-second intervalssmartctl and dmesg?Answer:
dmesg — kernel-level disk errors:
xxxxxxxxxxdmesg -T | grep -iE 'error|fail|reset|ata|sd[a-z]|blk'# Look for:# ata1.00: error: { UNC } — uncorrectable read error# sd a: FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE — SCSI sense error# Buffer I/O error on device sda1 — filesystem-level read error# EXT4-fs error — filesystem corruptionsmartctl — SMART disk health monitoring:
xxxxxxxxxx# Install: smartmontools package
# Quick health checksmartctl -H /dev/sda# Output: SMART overall-health self-assessment test result: PASSED (or FAILED)
# Full SMART datasmartctl -a /dev/sda
# Key SMART attributes to check:# ID 5 Reallocated_Sector_Ct — non-zero means bad sectors remapped (wear)# ID 187 Reported_Uncorrect — uncorrectable errors# ID 197 Current_Pending_Sector — sectors pending reallocation (unstable)# ID 198 Offline_Uncorrectable — sectors found bad during offline scan# ID 9 Power_On_Hours — drive age
# Run a short self-testsmartctl -t short /dev/sdasmartctl -l selftest /dev/sda # view test resultsNVMe drives:
xxxxxxxxxxnvme smart-log /dev/nvme0nvme error-log /dev/nvme0If Reallocated_Sector_Ct > 0 or Current_Pending_Sector > 0: Plan drive replacement. Back up data immediately.
Answer: LVM (Logical Volume Manager) adds a layer of abstraction between physical disks and filesystems, enabling dynamic resizing and more flexible storage management.
Layered architecture:
xxxxxxxxxxPhysical Drives (PV: Physical Volumes)↓Volume Groups (VG: pool of storage)↓Logical Volumes (LV: virtual partitions)↓Filesystems (ext4, xfs, etc.)
Key commands:
xxxxxxxxxx# Display all LVM infopvdisplay # physical volumesvgdisplay # volume groupslvdisplay # logical volumes
# Compact summarypvs; vgs; lvs
# Extend a logical volume (if VG has free space)lvextend -L +10G /dev/vg0/dataresize2fs /dev/vg0/data # resize ext4 filesystem (online if mounted)xfs_growfs /mount/point # resize XFS (must be mounted)
# Add a new disk to expand VGpvcreate /dev/sdbvgextend vg0 /dev/sdblvextend -l +100%FREE /dev/vg0/dataresize2fs /dev/vg0/data
# Create a snapshot (for backups/testing)lvcreate -s -n snap0 -L 5G /dev/vg0/datamount /dev/vg0/snap0 /mnt/snapshotTroubleshooting disk full without reformatting: LVM makes it possible to extend filesystems online without downtime, which is critical for production systems.
Answer:
xxxxxxxxxx# Find files > 100 MB, sorted by size (stay on one filesystem with -xdev)find / -xdev -type f -size +100M -printf '%s\t%p\n' 2>/dev/null | sort -rn | head -20
# Human-readable sizesfind / -xdev -type f -size +100M -exec ls -lh {} \; 2>/dev/null | sort -k5 -rh | head -20
# Fastest method: ncdu (interactive)ncdu -x /
# If looking in /var specificallyfind /var -type f -printf '%s %p\n' | sort -rn | head -20
# Find files modified in the last 24 hours that are large (recent growth)find / -xdev -type f -mtime -1 -size +50M -ls 2>/dev/null | sort -k7 -rn
# Find core dumpsfind / -xdev -name 'core' -o -name 'core.[0-9]*' 2>/dev/null -ls | sort -k7 -rnCommon locations for unexpectedly large files:
/var/log/ — unsegmented logs
/var/lib/docker/ — Docker layer cache
/tmp/ — temp files not cleaned up
Application-specific data directories
Home directories (/home/)
/var/spool/mail/ — mail queue
/proc/mounts tell you and how do you use it?Answer: /proc/mounts shows all currently mounted filesystems, including their mount options. It's the authoritative source (unlike /etc/mtab which can be stale).
xxxxxxxxxxcat /proc/mounts# <device> <mount_point> <fstype> <options> <dump> <pass># /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0# tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0Useful queries:
xxxxxxxxxx# Find all NFS mountsgrep nfs /proc/mounts
# Find all tmpfs mounts (memory-backed)grep tmpfs /proc/mounts
# Find read-only mounts (filesystem mounted ro after error)grep ' ro,' /proc/mounts
# Check mount options for a specific pathgrep ' /var ' /proc/mountsWhy it matters in troubleshooting:
Filesystem remounted read-only — a common result of detected disk errors. Application writes fail with "Read-only file system":
xxxxxxxxxxgrep ' ro,' /proc/mounts # find filesystems that went read-onlydmesg | grep 'remounting read-only' # find the error that caused itMissing mounts — a required filesystem not mounted after reboot.
Wrong mount options — performance-critical noatime or barrier=0 not applied.
Remount read-write after fixing the error:
xxxxxxxxxxmount -o remount,rw /dev/sda1# But first fix the underlying error (fsck, disk replacement)fio or dd to benchmark disk I/O?Answer:
dd — simple sequential I/O test:
xxxxxxxxxx# Sequential write speed (direct I/O, bypasses cache)dd if=/dev/zero of=/tmp/testfile bs=1M count=1024 oflag=direct conv=fdatasync# Result: ... 1073741824 bytes (1.1 GB) copied, 5.23 s, 205 MB/s
# Sequential read speeddd if=/tmp/testfile of=/dev/null bs=1M iflag=direct
# Cleanuprm /tmp/testfilefio — flexible, comprehensive I/O benchmark:
xxxxxxxxxx# Install: apt install fio / yum install fio
# Random read IOPS test (simulates database workload)fio --name=randread --ioengine=libaio --iodepth=32 --rw=randread \ --bs=4k --direct=1 --size=1G --numjobs=4 --runtime=60 \ --group_reporting --filename=/tmp/fio_test
# Sequential write throughputfio --name=seqwrite --ioengine=libaio --iodepth=8 --rw=write \ --bs=128k --direct=1 --size=4G --numjobs=1 --runtime=60 \ --group_reporting --filename=/tmp/fio_test
# Key output metrics:# IOPS — I/O operations per second# BW — bandwidth in MB/s# lat (avg) — average latency# lat (99.00th) — 99th percentile latencyWhen to use: Baseline disk performance before production, diagnose degraded performance after a hardware event, compare storage tiers.
Answer: The I/O scheduler (elevator) determines the order in which block layer requests are served.
Check current scheduler:
xxxxxxxxxx# For each block devicecat /sys/block/sda/queue/scheduler# Example: [mq-deadline] kyber bfq none# The active scheduler is in brackets
# All devices at oncefor dev in /sys/block/*/queue/scheduler; do echo "$dev: $(cat $dev)"doneCommon schedulers:
| Scheduler | Best For |
|---|---|
none | NVMe SSDs (hardware has its own queue) |
mq-deadline | SSDs, general purpose (good latency guarantees) |
kyber | Fast NVMe SSDs (low-latency target) |
bfq | HDDs, mixed read/write (fairness, avoids starvation) |
cfq (older kernels) | General purpose HDDs |
Change scheduler:
xxxxxxxxxxecho mq-deadline > /sys/block/sda/queue/scheduler # temporary# Persistent: add udev rule or set in /etc/default/grub (elevator= kernel param)Queue depth tuning:
xxxxxxxxxxcat /sys/block/nvme0n1/queue/nr_requests # current queue depthecho 256 > /sys/block/nvme0n1/queue/nr_requests # increase for NVMeRead-ahead:
xxxxxxxxxxblockdev --getra /dev/sda # get read-ahead (in 512-byte sectors)blockdev --setra 4096 /dev/sda # set to 2 MB (for sequential workloads)Answer:
Detection:
xxxxxxxxxxdmesg | grep -iE 'EXT4-fs error|XFS.*error|filesystem.*error|corruption'journalctl -k | grep -iE 'fs error|corruption'If the filesystem is still mounted (with errors):
xxxxxxxxxx# Most ext4 filesystems remount read-only on errorgrep ' ro,' /proc/mounts# This protects data from further corruption
# Check the journal for the errortune2fs -l /dev/sda1 | grep 'Last mounted\|Errors\|Mount count'fsck — filesystem check and repair (must unmount first):
xxxxxxxxxx# Unmount the filesystemumount /dev/sda1# If it's root or can't be unmounted, boot from recovery/rescue media
# Run fsckfsck -y /dev/sda1 # -y = answer yes to all repair questions# For XFS:xfs_repair /dev/sda1# For ext4 with journal replay:e2fsck -f /dev/sda1Never run fsck on a mounted filesystem (except read-only root with rescue mode).
After repair:
xxxxxxxxxxmount /dev/sda1 /mnt/datadmesg | tail -20 # verify no new errorsRoot cause investigation after repair:
Check SMART data for bad sectors.
Review power loss events (unclean shutdown can cause journal inconsistency).
Check for hardware errors (bad RAM causing data corruption during write).
Answer:
| Concept | Typical Value | Meaning |
|---|---|---|
| Physical sector size | 4096 bytes (4K) | Minimum write unit of the physical disk (modern drives) |
| Logical sector size | 512 bytes (traditional), 4096 (4Kn) | What the OS uses to address the disk |
| Filesystem block size | 4096 bytes (ext4 default) | Minimum allocation unit for the filesystem |
Check disk sector sizes:
xxxxxxxxxxblockdev --getss /dev/sda # logical sector sizeblockdev --getpbsz /dev/sda # physical sector sizecat /sys/block/sda/queue/hw_sector_sizecat /sys/block/sda/queue/physical_block_size512e drives — present 512-byte logical sectors but have 4096-byte physical sectors. Misaligned writes cause a "read-modify-write" penalty.
Check partition alignment:
xxxxxxxxxxparted /dev/sda print # check if Start is aligned to physical block size# Good: partition starts at 1049kB (2048 sectors × 512 bytes = 1 MB = aligned to 4K)Filesystem block size relevance:
Small files on large block size → internal fragmentation (wasted space).
Large files on small block size → more metadata overhead.
Default 4096 byte ext4 block size is good for most workloads.
smartd?Answer: smartd is a daemon that polls drive SMART data at configurable intervals and alerts on failures.
Configuration (/etc/smartd.conf):
xxxxxxxxxx# Monitor all drives, email on failure, run short test weekly, long test monthlyDEVICESCAN -d removable -n standby -m admin@example.com -M exec /usr/share/smartmontools/smartd-runner \ -R 5 -R 187 -R 197 -R 198 \ -s (S/../.././02|L/../../6/03)# -R 5 = report if Reallocated_Sector_Ct changes# -R 187 = report if Reported_Uncorrect changes# -s = schedule short (S) and long (L) self-testsEnable and check:
xxxxxxxxxxsystemctl enable --now smartdsmartctl -a /dev/sda # manual checksmartd -q onecheck # run one cycle immediatelyKey SMART thresholds to alert on:
Reallocated_Sector_Ct > 0 (any value, but trends matter)
Current_Pending_Sector > 0
Offline_Uncorrectable > 0
Reported_Uncorrect any increase
Temperature > 55°C for HDD, > 70°C for SSD
NVMe monitoring:
xxxxxxxxxxnvme smart-log /dev/nvme0# Key fields: critical_warning (non-zero = bad), available_spare (low = wear)# media_errors, num_err_log_entriesAnswer: When a disk is 100% full, many operations fail. Approach systematically:
Immediate triage:
xxxxxxxxxxdf -h # confirm which partition is fulldf -i # check inode usage tooQuick wins — find and delete large files:
xxxxxxxxxx# Compressed logs (safe to delete old ones)ls -lhS /var/log/*.gz | head -10# Truncate active log (never delete an open log file; truncate instead)> /var/log/app/large.log
# Find large filesfind /var -type f -size +100M -ls 2>/dev/null | sort -k7 -rn | head -10find /tmp -type f -ls | sort -k7 -rn | head -10
# Find and clear core dumpsfind / -xdev -name 'core' -o -name 'core.[0-9]*' 2>/dev/null -delete
# Clear package manager cacheapt-get clean # Debian/Ubuntuyum clean all # RHEL/CentOSdnf clean all # Fedora/RHEL 8+Freed space not showing?
xxxxxxxxxx# Check for open-but-deleted fileslsof | grep '(deleted)' | awk '{print $7}' | sort -rn | head -10# Truncate via fd or restart the holding process> /proc/<pid>/fd/<fd_number>Emergency space creation: Reserve blocks (ext4 reserves 5% for root by default):
xxxxxxxxxxtune2fs -m 1 /dev/sda1 # reduce reserved blocks to 1% (for non-root partitions)blktrace or blkparse for advanced block I/O tracing?Answer: blktrace captures block layer events (queue, issue, complete) to diagnose complex I/O problems.
xxxxxxxxxx# Install: blktrace package
# Capture 30 seconds of I/O on /dev/sdablktrace -d /dev/sda -o /tmp/sda_trace -w 30
# Parse the traceblkparse /tmp/sda_trace.blktrace.0 -o /tmp/sda_trace.txt
# Or combine capture and parseblktrace -d /dev/sda -o - | blkparse -i -Event types:
Q — queued to generic block layer
G — get request (merged into request queue)
I — issued to driver
C — completed
M — merged with existing request
D — dispatched to driver
Calculate latency:
Time from Q to C = total I/O latency.
Time from I to C = driver/device service time.
Time from Q to I = queue wait time.
btt — block trace timing analysis:
xxxxxxxxxxbtt -i /tmp/sda_trace.blktrace.0# Outputs: Q2C (queue-to-complete), D2C (dispatch-to-complete), Q2D (queue-to-dispatch)# and system call latency breakdownsSimpler alternative — biotop (from bcc-tools):
xxxxxxxxxxbiotop 1 # top-like view of I/O by process, refreshes every secondsystemctl status and journalctl -xe to diagnose a failed service?Answer:
systemctl status — first stop:
xxxxxxxxxxsystemctl status myserviceOutput includes:
Active state: failed, activating, deactivating, active (running).
Exit code and signal (e.g., code=exited, status=1/FAILURE or code=killed, signal=SEGV).
Last 10 journal lines from the service.
PID, memory usage, cgroup.
journalctl -xe — extended context with explanations:
xxxxxxxxxxjournalctl -xe -u myservice# -x = add explanatory text (catalog entries)# -e = jump to end of journal# -u = filter to specific unit
# Last 100 linesjournalctl -u myservice -n 100
# Follow in real-time during a restartjournalctl -fu myservice &systemctl restart myserviceSystematic approach:
xxxxxxxxxx# Step 1: Status overviewsystemctl status myservice
# Step 2: Full logs for this servicejournalctl -u myservice --since "10 min ago"
# Step 3: Check the unit file for cluessystemctl cat myservice
# Step 4: Verify ExecStart binary exists and is executablewhich <binary_from_ExecStart>ls -la <full_path_to_binary>
# Step 5: Dependency checksystemctl list-dependencies myserviceExecStartPre failures in a systemd unit?Answer: ExecStartPre commands run before the main ExecStart. If any exit non-zero (unless prefixed with -), the service fails to start.
Identifying ExecStartPre failures:
xxxxxxxxxxjournalctl -u myservice -n 50# Look for: "ExecStartPre= failed with result 'exit-code'"# or specific error from the pre commandExample unit file with pre-check:
xxxxxxxxxx[Service]ExecStartPre=/usr/bin/myapp --config-check /etc/myapp.confExecStart=/usr/bin/myappIf --config-check exits 1 (config invalid), the service never starts.
Debugging:
xxxxxxxxxx# Run the ExecStartPre command manually as the service usersudo -u myapp /usr/bin/myapp --config-check /etc/myapp.confecho $? # exit code
# Check if required files existls -la /etc/myapp.conf
# Verify permissions (service may run as non-root)stat /etc/myapp.confSuppress failure from optional pre-command (using - prefix):
xxxxxxxxxxExecStartPre=-/usr/bin/cleanup_pidfile.sh # OK to failExecStartPre=/usr/bin/required_check.sh # must succeedCommon ExecStartPre failures:
Config validation script failing (bad config syntax).
Pre-flight check for required resources (ports, databases) not ready.
Filesystem checks for required directories that don't exist.
StartLimitBurst?Answer: A crash loop is when a service repeatedly starts, fails, and restarts. Systemd's rate limiting prevents infinite restart storms.
Crash loop detection:
xxxxxxxxxxsystemctl status myservice# Active: failed (Result: start-limit-hit)# or: activating (auto-restart) repeated rapidly in logs
journalctl -u myservice | grep -E 'Started|Failed|Stopped' | tail -20StartLimitBurst and StartLimitIntervalSec:
xxxxxxxxxx[Service]Restart=on-failureRestartSec=2s
[Unit]StartLimitBurst=5 # allow 5 starts...StartLimitIntervalSec=30s # ...within 30 seconds# If exceeded: systemd stops trying and sets status to "start-limit-hit"Reset the rate limit counter to try again:
xxxxxxxxxxsystemctl reset-failed myservicesystemctl start myserviceDiagnosing why it's crashing:
xxxxxxxxxx# Get the actual error before systemd stopped retryingjournalctl -u myservice --since "5 min ago" | grep -E 'error|Error|fail|Fail|fatal|FATAL'
# Check exit codejournalctl -u myservice | grep 'Main process exited'# "Main process exited, code=exited, status=1/FAILURE"# "Main process exited, code=killed, signal=SEGV" → segfault
# Run service binary manually to see its outputsudo -u <service_user> /usr/bin/myapp # see raw outputCommon crash loop causes:
Missing config file or invalid configuration.
Port already in use (Address already in use).
Required dependency service not available.
Permissions error on data directory.
Segfault or unhandled exception in code.
Missing shared library (ldd /usr/bin/myapp).
After= and Requires= in systemd?Answer: systemd uses dependency directives to order and require services.
Key directives:
| Directive | Effect |
|---|---|
After=B | Start A after B, but don't require B to succeed |
Requires=B | A requires B; if B fails, A fails |
Wants=B | Soft dependency; A prefers B but continues without it |
BindsTo=B | Stronger than Requires; if B stops, A stops too |
Before=B | A must start before B |
Conflicts=B | A and B cannot run simultaneously |
Debugging dependency failures:
xxxxxxxxxx# See full dependency treesystemctl list-dependencies myservicesystemctl list-dependencies --reverse myservice # what depends on myservice
# Check if required units are activesystemctl is-active postgresql.service
# See why a dependency isn't metjournalctl -u postgresql.service | tail -20
# Show all failed unitssystemctl --failedsystemctl --state=failedCommon failure pattern: Service uses Requires=network.target After=network.target but actually needs network-online.target (full network, not just interface up):
xxxxxxxxxx[Unit]After=network-online.targetWants=network-online.targetChecking what network.target vs network-online.target means:
xxxxxxxxxxsystemctl cat network-online.targetVisualize dependencies:
xxxxxxxxxxsystemd-analyze dot myservice | dot -Tsvg > deps.svgAnswer: Socket activation allows systemd to listen on a socket and start the service only when a connection arrives. This reduces boot time and allows on-demand activation.
How it works:
systemd creates and binds the socket (.socket unit).
When a connection arrives, systemd starts the associated .service unit.
The service inherits the already-bound socket via sd_listen_fds() mechanism.
File descriptor 3 (or more) are the pre-activated sockets.
Example unit files:
xxxxxxxxxx# myservice.socket[Socket]ListenStream=8080Accept=false
[Install]WantedBy=sockets.targetxxxxxxxxxx# myservice.service[Service]ExecStart=/usr/bin/myappDebugging socket activation:
xxxxxxxxxx# Check socket unit statussystemctl status myservice.socket
# Verify the socket is listeningss -tlnp | grep 8080
# Test activation (connect to the port)nc -zv localhost 8080 # should trigger myservice.service start
# Check if service was activatedsystemctl status myservice.servicejournalctl -u myservice.service
# Verify socket is passed correctlysystemd-socket-activate -l 8080 /usr/bin/myapp # test socket activation manually
# Check socket file descriptorsls -la /run/systemd/units/Common problems:
Service doesn't call sd_listen_fds() and tries to bind its own socket (bind fails — port in use).
Accept=yes vs Accept=no mismatch with application expectations.
Missing StandardInput=socket for stdin-based protocols.
Answer: systemd services can load environment variables from files using EnvironmentFile=.
xxxxxxxxxx[Service]EnvironmentFile=/etc/myapp/environmentEnvironmentFile=-/etc/myapp/environment.local # - = optional fileExecStart=/usr/bin/myappDebugging environment issues:
xxxxxxxxxx# Show the environment a service runs withsystemctl show myservice -p Environment# orsystemctl show myservice | grep Environment
# Check the unit file for EnvironmentFile directivessystemctl cat myservice | grep EnvironmentFile
# Verify the file exists and has correct permissionsls -la /etc/myapp/environmentcat /etc/myapp/environment # review contents
# See the actual environment in the running processcat /proc/<pid>/environ | tr '\0' '\n'# orstrings /proc/<pid>/environCommon pitfalls:
File doesn't exist and no - prefix → service fails to start.
Wrong file permissions (should be readable by the service user).
Variable syntax errors (no export keyword; format is KEY=VALUE).
Quoting issues: PATH="/usr/local/bin:$PATH" — $PATH is not expanded in EnvironmentFile.
Spaces around = are not allowed in EnvironmentFile.
Correct format:
xxxxxxxxxx# /etc/myapp/environmentDB_HOST=localhostDB_PORT=5432APP_ENV=productionLOG_LEVEL=info# NOT: export DB_HOST=localhost (export keyword ignored)# NOT: DB_HOST = localhost (spaces not allowed)Type=forking and Type=simple and when does the wrong choice break a service?Answer: The Type= directive tells systemd how to determine when the service is "ready."
| Type | Meaning | systemd waits for... |
|---|---|---|
simple | Default. The main process is ExecStart. | ExecStart to start (doesn't wait for readiness) |
exec | Like simple, but waits for exec() to succeed | exec() to return |
forking | Process forks and parent exits | Parent to exit (child is the daemon) |
notify | Service sends sd_notify("READY=1") | READY=1 notification |
dbus | Service registers on D-Bus | D-Bus name acquired |
oneshot | Single-run task | Process to exit |
idle | Like simple, delayed until active jobs done | ExecStart (delayed) |
Type=simple (wrong for traditional daemons):
If an application double-forks (traditional daemon behavior) and you use Type=simple, systemd tracks the wrong PID (the parent that exited). Service appears "active" but the real daemon may have failed.
xxxxxxxxxx# Symptom: service shows active but process isn't runningsystemctl status myservice # shows active (running)pgrep myapp # no process foundType=forking (wrong for non-forking apps):
If a modern app doesn't fork but you specify Type=forking, systemd waits forever for the parent to exit.
xxxxxxxxxx# Symptom: service gets stuck in "activating" statesystemctl status myservice # Active: activating (start) for 30+ secondsBest practice for modern apps: Use Type=notify with sd_notify() in the application, or Type=simple for apps that stay in the foreground.
Diagnose with:
xxxxxxxxxxsystemctl show myservice -p Typesystemctl cat myservice | grep ^TypeSYSTEMD_LOG_LEVEL to get debug output from a service?Answer: systemd services (and systemd itself) can be configured to emit debug-level logs.
Method 1 — Environment variable in unit file:
xxxxxxxxxx[Service]Environment=SYSTEMD_LOG_LEVEL=debugExecStart=/usr/bin/myappMethod 2 — Set for a one-off restart:
xxxxxxxxxx# Override temporarily without editing the unit filesystemctl edit --runtime myservice# Add in the override file:# [Service]# Environment=SYSTEMD_LOG_LEVEL=debug
systemctl restart myservicejournalctl -fu myservice # watch debug outputMethod 3 — Set on the systemd manager itself (affects all units):
xxxxxxxxxxsystemd-analyze log-level debug # set systemd log level to debugjournalctl -fu systemd # watch systemd's own debug outputsystemd-analyze log-level info # reset to normalApplication-specific debug:
SYSTEMD_LOG_LEVEL affects systemd libraries (libsystemd). For application-specific debug:
xxxxxxxxxx# Common environment variables for debug loggingEnvironment=DEBUG=1Environment=LOG_LEVEL=DEBUGEnvironment=VERBOSE=trueView all environment variables set for a service:
xxxxxxxxxxsystemctl show myservice | grep -E '^Environment'cat /proc/$(systemctl show -p MainPID --value myservice)/environ | tr '\0' '\n'Answer: A common service startup failure is "Address already in use" — another process is already bound to the port.
Detection:
xxxxxxxxxx# Check what's listening on a specific portss -tlnp | grep :8080lsof -i :8080# Output: COMMAND PID USER FD TYPE ... NAME# nginx 1234 root ... TCP *:8080 (LISTEN)
# Check all listening portsss -tlnpnetstat -tlnp # older systemsFind the conflicting process and decide:
xxxxxxxxxx# Is it a stale/zombie process?ps -p <pid>systemctl status --pid=<pid> # which unit owns this PID?
# Kill the conflicting process (if appropriate)kill -15 <pid>
# Or find which service owns it and disable that servicesystemctl stop nginxsystemctl disable nginxIf the service crashed and left a stale socket file:
xxxxxxxxxxls -la /run/myapp.sockrm /run/myapp.sock # remove stale socketsystemctl start myappSystemd socket reuse (avoid conflicts):
xxxxxxxxxx[Socket]ListenStream=8080ReusePort=yes # allows multiple sockets on same port (load balancing)systemd-analyze to diagnose slow boot times?Answer: systemd-analyze provides boot timing analysis.
xxxxxxxxxx# Total boot time breakdownsystemd-analyze# Startup finished in 1.234s (kernel) + 3.456s (initrd) + 12.345s (userspace) = 17.035s
# Per-unit timing (sorted by time)systemd-analyze blame | head -20
# Critical path (longest chain that determined total boot time)systemd-analyze critical-chain
# Critical chain for a specific servicesystemd-analyze critical-chain myservice.service
# Visualize as a timeline SVGsystemd-analyze plot > boot-timeline.svg
# Check for security issues in unit filessystemd-analyze security myservice.service
# Verify unit file syntaxsystemd-analyze verify myservice.serviceTypical slow-boot culprits:
NetworkManager-wait-online.service — waiting for network (often slow on cloud VMs).
Services with long TimeoutStartSec that are hanging.
dev-sdb1.device — waiting for a disk device that's absent.
Fix network boot slowness:
xxxxxxxxxxsystemctl disable NetworkManager-wait-online.service# or tune its timeoutsystemctl edit NetworkManager-wait-online.service# Add: [Service]# TimeoutStartSec=10sAnswer: Drop-in files allow modifying a unit's configuration without editing the original (which may be overwritten by package updates).
Create a drop-in:
xxxxxxxxxx# Interactive editor (creates /etc/systemd/system/myservice.service.d/override.conf)systemctl edit myservice
# Non-interactive creationmkdir -p /etc/systemd/system/myservice.service.d/cat > /etc/systemd/system/myservice.service.d/override.conf << 'EOF'[Service]Environment=DEBUG=1Restart=alwaysRestartSec=5sLimitNOFILE=65536EOF
# Reload systemd to pick up changessystemctl daemon-reloadsystemctl restart myserviceView effective unit (merged original + overrides):
xxxxxxxxxxsystemctl cat myservice # shows all layers with file pathsReset to default (remove overrides):
xxxxxxxxxxsystemctl revert myserviceCommon uses for drop-ins:
Increase LimitNOFILE (file descriptor limits) without changing the package's unit.
Set environment variables.
Change Restart= policy.
Override After=/Requires= dependencies.
Set TimeoutStartSec= or TimeoutStopSec=.
Adjust CPUQuota= or MemoryLimit= for resource control.
Answer: The most common causes of this discrepancy are environment differences, working directory, user/group, or missing stdin/tty.
Systematic comparison:
1. User and group context:
xxxxxxxxxx# What user does systemd run it as?systemctl show myservice -p User -p Group# Run manually as that usersudo -u myapp /usr/bin/myapp2. Environment variables:
xxxxxxxxxx# Shell has many variables systemd doesn't have# Check what systemd passessystemctl show myservice -p Environment
# Common missing vars: HOME, PATH, TERM, DISPLAY, USER# Add to unit file if needed:# [Service]# Environment=HOME=/home/myapp# Environment=PATH=/usr/local/bin:/usr/bin:/bin3. Working directory:
xxxxxxxxxxsystemctl show myservice -p WorkingDirectory# Application may expect to be run from a specific directory# Fix: WorkingDirectory=/opt/myapp4. No terminal / stdin:
xxxxxxxxxx# Application may try to read stdin or use tty functions that fail with no TTY# Fix: StandardInput=null or StandardInput=file:/dev/null# Or: redirect in ExecStart: ExecStart=/usr/bin/myapp < /dev/null5. ulimits:
xxxxxxxxxx# Shell has different ulimits than systemdulimit -a # your shell limitssystemctl show myservice | grep Limit # systemd limits# Fix: LimitNOFILE=65536 in [Service]6. SELinux/AppArmor context:
xxxxxxxxxxausearch -m avc -ts recent # SELinux denialsjournalctl | grep apparmor # AppArmor denialsAnswer: systemd services run in cgroups, and resource limits can be set both via classic ulimit mechanisms and cgroup-based controls.
View current limits:
xxxxxxxxxx# For a running servicesystemctl show myservice | grep -E 'Limit|Memory|CPU|Tasks'cat /proc/<pid>/limits # kernel-enforced limits for a specific PID
# Cgroup resource usagesystemd-cgtop # live cgroup resource usagecat /sys/fs/cgroup/system.slice/myservice.service/memory.currentCommon resource directives in unit files:
xxxxxxxxxx[Service]# File descriptor limitLimitNOFILE=65536
# Max processes/threadsLimitNPROC=4096
# Core dump size (0 = disable core dumps)LimitCORE=0
# Max locked memoryLimitMEMLOCK=infinity # needed for some databases, realtime apps
# Cgroup-based memory limit (hard limit, triggers OOM kill)MemoryMax=4GMemoryHigh=3G # soft limit, triggers throttling
# CPU limitCPUQuota=200% # 2 cores worth of CPU
# I/O weightIOWeight=100 # 1-10000, default 100
# Task count (threads + processes)TasksMax=1024Diagnose "too many open files" error:
xxxxxxxxxx# Check current fd usage vs limitcat /proc/<pid>/limits | grep 'open files'ls /proc/<pid>/fd | wc -l # current open file count
# If at limit, increase:# systemctl edit myservice → add LimitNOFILE=131072Answer: The systemd journal (journald) stores structured logs that can be queried with powerful filters.
Basic queries:
xxxxxxxxxx# Logs for a specific unitjournalctl -u myservice
# Follow in real-timejournalctl -fu myservice
# Time-bounded queriesjournalctl --since "2026-05-17 14:00" --until "2026-05-17 15:00"journalctl --since "1 hour ago"journalctl --since yesterday
# Priority filtering (-p: emerg, alert, crit, err, warning, notice, info, debug)journalctl -p err -u myservicejournalctl -p 0..3 # emerg through err (most critical)
# Kernel messages onlyjournalctl -k
# Boot-specific logsjournalctl -b # current bootjournalctl -b -1 # previous bootjournalctl --list-boots # all recorded bootsStructured field filtering:
xxxxxxxxxx# Filter by executable pathjournalctl _EXE=/usr/bin/myapp
# Filter by PIDjournalctl _PID=12345
# Filter by UIDjournalctl _UID=1000
# Combine filters (AND)journalctl _SYSTEMD_UNIT=myservice.service _PRIORITY=3
# JSON output for machine parsingjournalctl -u myservice -o json | jq '.MESSAGE' | tail -20
# Export for analysisjournalctl -u myservice --since "1 hour ago" -o json-pretty > /tmp/service_logs.jsonJournal disk usage:
xxxxxxxxxxjournalctl --disk-usagejournalctl --vacuum-size=500M # keep only 500 MBjournalctl --vacuum-time=30d # keep only 30 daysAnswer: systemd timers replace cron jobs with better logging, dependency management, and resource control.
Timer unit files:
xxxxxxxxxx# backup.timer[Unit]Description=Daily backup timer
[Timer]OnCalendar=daily # run at midnight dailyOnCalendar=Mon-Fri 09:00 # weekdays at 9 AMRandomizedDelaySec=1h # randomize within 1 hour (prevents stampede)Persistent=true # run if missed (e.g., system was off)
[Install]WantedBy=timers.targetxxxxxxxxxx# backup.service[Unit]Description=Daily backup job
[Service]Type=oneshotExecStart=/usr/local/bin/backup.shUser=backupManagement:
xxxxxxxxxx# Enable and start the timersystemctl enable --now backup.timer
# List all timers and next run timesystemctl list-timerssystemctl list-timers --all # includes inactive timers
# Check timer statussystemctl status backup.timer
# Run the job immediately (for testing)systemctl start backup.service # runs the service directly
# View logs from timer-triggered runsjournalctl -u backup.serviceOnCalendar syntax:
xxxxxxxxxx# Verify calendar expressionsystemd-analyze calendar "Mon-Fri 09:00"# Next elapse: Mon 2026-05-18 09:00:00 UTC
systemd-analyze calendar "*:0/15" # every 15 minutessystemd-analyze calendar "hourly"systemd-analyze calendar "weekly"Debug a timer that's not running:
xxxxxxxxxxsystemctl status backup.timer # is it active?journalctl -u backup.timer # any error activating it?journalctl -u backup.service # did the service fail when triggered?systemctl list-timers | grep backup # when is next run?timedatectl # is system time correct?Answer: Start at Layer 3 and work outward:
Check local interface and route:
xxxxxxxxxxip addr show # confirm interface has an IPip route show # confirm default gateway existsPing the gateway:
xxxxxxxxxxping -c 4 $(ip route | awk '/default/{print $3}')If this fails, the problem is local (bad cable, misconfigured interface, VLAN issue).
Ping a known external IP (bypass DNS):
xxxxxxxxxxping -c 4 8.8.8.8If this fails but gateway ping succeeds, the issue is upstream routing or firewall.
Traceroute to find the hop where packets die:
xxxxxxxxxxtraceroute -n 8.8.8.8 # ICMP/UDPtraceroute -T -p 443 8.8.8.8 # TCP SYN — useful when ICMP is blockedmtr --report --report-cycles 20 8.8.8.8mtr combines ping+traceroute and shows per-hop packet loss over time, which is invaluable for intermittent issues.
Check for firewall rules dropping outbound:
xxxxxxxxxxiptables -L -n -v --line-numbersiptables -t nat -L -n -vnft list rulesetCheck routing table for unexpected routes:
xxxxxxxxxxip route get 8.8.8.8 # shows exact route the kernel would useConfirm DNS separately:
xxxxxxxxxxdig +short google.com @8.8.8.8 # bypass local resolverKey distinction: packet loss at a middle hop in traceroute is often just that router de-prioritizing ICMP — look at whether loss continues at subsequent hops. If subsequent hops are reachable, the middle-hop "loss" is benign.
Answer:
Connection Refused (TCP RST):
The remote host is reachable at the IP level
The kernel on the remote host received the SYN but no process is listening on that port
The kernel immediately sends back a TCP RST packet
errno: ECONNREFUSED
Diagnosis: service is down, wrong port, or firewall sending RST
Connection Timed Out:
The SYN packet never received a SYN-ACK or RST response
The client retried (default 6 times with exponential backoff, ~127 seconds total on Linux) and gave up
errno: ETIMEDOUT
Implies: packet is being silently dropped somewhere — firewall rule with DROP (not REJECT), routing black hole, host is unreachable, overloaded NIC/queue
Diagnostic approach:
xxxxxxxxxx# Refused: service check on the remotess -tulnp | grep :PORTsystemctl status <service>
# Timeout: path checktraceroute -T -p PORT remote_hosttcpdump -i eth0 host remote_host and port PORT# Look: do you see SYN going out? Do you see SYN-ACK coming back?Why it matters at Google scale: A service returning ECONNREFUSED immediately is failing fast; ETIMEDOUT means threads/goroutines pile up waiting, consuming memory and file descriptors — this is the dangerous case that cascades.
ss to find which process is listening on a specific port, and what flags do you use?Answer:
xxxxxxxxxxss -tulnpFlag breakdown:
-t — TCP sockets
-u — UDP sockets
-l — listening sockets only
-n — numeric (don't resolve hostnames/service names, faster)
-p — show process (PID and program name)
Find a specific port:
xxxxxxxxxxss -tulnp | grep ':8080'# orss -tlnp sport = :8080Show all connections to a remote host:
xxxxxxxxxxss -tn dst 10.0.0.5Show established connections with memory usage:
xxxxxxxxxxss -tnm state establishedCount connections per state:
xxxxxxxxxxss -tan | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rnCompared to netstat: ss reads directly from the kernel's sock_diag netlink interface — much faster on hosts with thousands of connections. netstat reads /proc/net/tcp which scales poorly.
Privileged access: The -p flag showing PID/process requires root or the same UID as the process. Without root, you'll see ("-") for processes you don't own.
tcpdump to capture the relevant traffic without filling the disk?Answer:
Basic targeted capture:
xxxxxxxxxxtcpdump -i eth0 -n host 10.0.1.5 and port 443 -w /tmp/capture.pcapLimit file size and rotate:
xxxxxxxxxxtcpdump -i eth0 -n host 10.0.1.5 and port 443 \ -C 100 -W 10 -w /tmp/capture_%Y%m%d_%H%M%S.pcap# -C 100: rotate at 100 MB# -W 10: keep max 10 files (ring buffer)Capture only packet headers (no payload) to save space:
xxxxxxxxxxtcpdump -i eth0 -s 96 -n port 443 -w /tmp/headers.pcap# -s 96: snap length, captures first 96 bytes (headers + some payload)# -s 0: full packetWrite to stdout, pipe to compress:
xxxxxxxxxxtcpdump -i eth0 -n port 443 -w - | gzip > /tmp/capture.pcap.gzTrigger-based capture (stop after N packets):
xxxxxxxxxxtcpdump -i eth0 -n port 443 -c 10000 -w /tmp/capture.pcapRead and filter a saved file:
xxxxxxxxxxtcpdump -r /tmp/capture.pcap -n 'tcp[tcpflags] & tcp-rst != 0'# Show only RST packetsKey filters for troubleshooting:
xxxxxxxxxx# SYN without SYN-ACK (half-open connections)tcpdump 'tcp[tcpflags] == tcp-syn'
# HTTP errorstcpdump -A port 80 | grep -E 'HTTP/1\.[01] [45]'
# ICMP unreachabletcpdump 'icmp[icmptype] == icmp-unreach'Disk estimation: 100 Mbps line rate ≈ 750 MB/min full capture. Use -s 96 to reduce to ~10-20% of that.
Answer:
Step 1: Determine if it's DNS or the network:
xxxxxxxxxxping -c 2 8.8.8.8 # works? IP connectivity is fineping -c 2 google.com # fails? DNS issue confirmedStep 2: Check resolver configuration:
xxxxxxxxxxcat /etc/resolv.conf# Look for: nameserver, search, options# nameserver 127.0.0.53 means systemd-resolved is in useStep 3: Check NSSwitch configuration:
xxxxxxxxxxcat /etc/nsswitch.conf | grep hosts# e.g.: hosts: files dns myhostname# Order matters: 'files' = /etc/hosts, 'dns' = nameserversStep 4: Query directly:
xxxxxxxxxxdig google.com # uses /etc/resolv.confdig google.com @8.8.8.8 # bypass local resolverdig google.com @127.0.0.53 # query systemd-resolved directlyresolvectl query google.com # systemd-resolved statusresolvectl status # shows per-interface DNS settingsStep 5: Test local resolver connectivity:
xxxxxxxxxxdig google.com @$(awk '/^nameserver/{print $2; exit}' /etc/resolv.conf)nc -zuv <nameserver_ip> 53 # UDP port 53 reachable?Step 6: Check for NXDOMAIN vs SERVFAIL:
xxxxxxxxxxdig +short google.com# NXDOMAIN = name doesn't exist (misconfigured zone, typo)# SERVFAIL = resolver couldn't get an authoritative answer (broken upstream, DNSSEC failure)# no response = resolver unreachableStep 7: Flush caches:
xxxxxxxxxxresolvectl flush-caches # systemd-resolvedsystemctl restart nscd # if nscd is runningCommon gotchas:
/etc/hosts overrides DNS — check it first
search domain in resolv.conf can cause unexpected lookups
ndots:5 option causes short names to be tried with all search domains first
systemd-resolved stub listener on 127.0.0.53 may not forward to the right upstream
Answer:
What is TIME_WAIT? After a TCP connection closes (the side that sent the final FIN enters TIME_WAIT), the kernel keeps the 4-tuple (src IP, src port, dst IP, dst port) in TIME_WAIT state for 2×MSL (Maximum Segment Lifetime = 60s on Linux, so 120 seconds total) to absorb late-arriving duplicate packets from the network.
Why it accumulates: High-throughput short-lived connections (HTTP/1.0, REST APIs without keep-alive, monitoring checks) create many connections per second. At 10,000 connections/second × 120 seconds = 1.2 million TIME_WAIT sockets possible.
Check current state:
xxxxxxxxxxss -tan state time-wait | wc -lss -tan | awk 'NR>1{print $1}' | sort | uniq -ccat /proc/net/sockstat | grep TCPMitigations (safest to most aggressive):
Enable TCP keep-alive and connection reuse (best fix — application level):
Use HTTP persistent connections (keep-alive)
Use connection pools in your application
tcp_tw_reuse (safe for outbound connections):
xxxxxxxxxxsysctl -w net.ipv4.tcp_tw_reuse=1# Allows reusing TIME_WAIT sockets for new outbound connections# Only applies when source port would be reused; safe with timestampsIncrease local port range:
xxxxxxxxxxsysctl -w net.ipv4.ip_local_port_range="10000 65000"# More ephemeral ports = less chance of port exhaustiontcp_fin_timeout (reduce TIME_WAIT duration):
xxxxxxxxxxsysctl -w net.ipv4.tcp_fin_timeout=30# Reduces FIN_WAIT_2 (not TIME_WAIT directly, but related)SO_LINGER with linger=0 (use carefully):
Sends RST instead of FIN on close — skips TIME_WAIT entirely
Can cause data loss; only appropriate for protocols that tolerate it
Do NOT use tcp_tw_recycle: Removed in kernel 4.12. Previously caused dropped connections from NATted clients (multiple clients behind same NAT with different timestamps).
At Google scale: TIME_WAIT is usually a symptom of missing connection reuse. Fix the application layer first.
Answer:
List rules with packet/byte counters:
xxxxxxxxxxiptables -L -n -v --line-numbers# -v shows packet and byte counts per rule# Zero counts = rule never matchedCheck all tables:
xxxxxxxxxxiptables -t filter -L -n -v # default tableiptables -t nat -L -n -v # NAT rulesiptables -t mangle -L -n -v # packet manglingiptables -t raw -L -n -v # raw/connection tracking bypassTrace a packet through all rules (iptables LOG target):
xxxxxxxxxx# Add temporary logging before DROP rulesiptables -I INPUT 1 -p tcp --dport 8080 -j LOG --log-prefix "IPTABLES-IN: " --log-level 4iptables -I OUTPUT 1 -p tcp --sport 8080 -j LOG --log-prefix "IPTABLES-OUT: "
# Watch the logsjournalctl -f -k | grep IPTABLES# ordmesg -w | grep IPTABLES
# Remove when doneiptables -D INPUT 1iptables -D OUTPUT 1Use xtables-monitor (nftables era):
xxxxxxxxxxxtables-monitor --trace# Shows real-time rule matchingnftables tracing:
xxxxxxxxxxnft add table ip debugnft add chain ip debug prerouting '{ type filter hook prerouting priority -300; }'nft add rule ip debug prerouting tcp dport 8080 meta nftrace set 1nft monitor trace# When done:nft delete table ip debugconntrack (check if connection is being tracked):
xxxxxxxxxxconntrack -L | grep "dport=8080"# Shows NAT translations and connection stateCommon gotchas:
Default policy DROP is silent — add explicit ACCEPT rules before it
Rules are evaluated top-to-bottom; first match wins
ESTABLISHED,RELATED rule missing = return traffic dropped
Docker/Kubernetes add their own chains; always check iptables-save for full picture
Answer:
MTU (Maximum Transmission Unit): The largest IP packet that can be sent on a network path without fragmentation. Ethernet standard MTU is 1500 bytes. Jumbo frames use 9000 bytes.
How PMTU Discovery works: Linux sets the "Don't Fragment" (DF) bit on TCP packets and relies on ICMP "fragmentation needed" (Type 3, Code 4) messages from intermediate routers to discover the path MTU. TCP then reduces its MSS accordingly.
The subtle failure mode: If ICMP is blocked by a firewall (common in corporate/cloud environments), PMTU Discovery fails silently. Small packets work fine (ACKs, HTTP headers), but large packets (bulk data transfers, large responses) are silently dropped. This manifests as:
SSH connects but hangs after authentication
HTTPS pages load headers but not body
VPN connects but large transfers fail
Works for small files, fails for large files
Diagnosis:
xxxxxxxxxx# Check interface MTUip link show eth0# orcat /sys/class/net/eth0/mtu
# Test specific packet sizesping -M do -s 1472 remote_host # 1472 + 28 byte IP/ICMP header = 1500ping -M do -s 1400 remote_host # if 1472 fails, try smaller# -M do = set DF bit, -s = payload size
# Check for ICMP blockingiptables -L -n | grep icmp# or trace with tcpdumptcpdump -i eth0 icmpFix options:
Allow ICMP type 3 code 4 through firewalls (correct fix):
xxxxxxxxxxiptables -I INPUT -p icmp --icmp-type fragmentation-needed -j ACCEPTiptables -I OUTPUT -p icmp --icmp-type fragmentation-needed -j ACCEPTiptables -I FORWARD -p icmp --icmp-type fragmentation-needed -j ACCEPTClamp MSS (TCP-specific workaround):
xxxxxxxxxxiptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \ -j TCPMSS --clamp-mss-to-pmtu# Rewrites MSS in SYN packets to match path MTUReduce interface MTU:
xxxxxxxxxxip link set eth0 mtu 1400VPN-specific: VPN tunnels add overhead (GRE: 24 bytes, OpenVPN: 28+ bytes, WireGuard: 60 bytes). Always set tunnel interface MTU to outer_MTU minus tunnel overhead.
mtr for diagnosing intermittent packet loss, and how do you interpret its output?Answer:
Run mtr in report mode:
xxxxxxxxxxmtr --report --report-cycles 100 --no-dns 8.8.8.8# --report-cycles 100: send 100 probes per hop (more reliable statistics)# --no-dns: skip reverse DNS lookups (faster, less noise)Sample output:
xxxxxxxxxxStart: 2024-01-15T10:00:00+0000HOST: myserver Loss% Snt Last Avg Best Wrst StDev1.|-- 192.168.1.1 0.0% 100 0.4 0.5 0.3 2.1 0.22.|-- 10.0.0.1 0.0% 100 1.2 1.3 1.0 3.4 0.33.|-- 203.0.113.1 20.0% 100 15.1 14.8 13.9 22.1 1.14.|-- 198.51.100.1 0.0% 100 15.3 15.1 14.8 18.2 0.45.|-- 8.8.8.8 0.0% 100 16.2 16.0 15.7 19.1 0.3
Interpretation rules:
Loss only at hop 3, not at hop 4+: The router at hop 3 is rate-limiting ICMP probes — not real packet loss. Traffic flows fine. This is benign.
Loss starts at hop 3 and continues through all subsequent hops: Real packet loss at hop 3. Everything downstream inherits it.
Loss at the final destination only: That host is rate-limiting ICMP or firewalling probes. Check with TCP-based probe: mtr -T -P 443 8.8.8.8
High latency spike at one hop, normal after: That router has high ICMP processing latency (low priority queue) but forwards traffic normally. Benign for most traffic.
StDev (jitter) increasing progressively: Network congestion or bufferbloat in that segment.
TCP mode for bypassing ICMP filters:
xxxxxxxxxxmtr -T -P 443 --report --report-cycles 50 google.comContinuous interactive mode:
xxxxxxxxxxmtr --curses 8.8.8.8 # press 'p' to pause, 'r' to resetAnswer:
From the remote host itself (most reliable):
xxxxxxxxxxss -tulnp # all listening socketsss -tlnp sport = :80 # specific portFrom a remote machine — nmap:
xxxxxxxxxx# Basic TCP connect scan (completes 3-way handshake)nmap -sT -p 1-1024 target_host
# SYN scan (half-open, faster, requires root)nmap -sS -p 1-65535 target_host
# Service version detectionnmap -sV -p 80,443,8080 target_host
# UDP scan (slow, often unreliable)nmap -sU -p 53,123,161 target_host
# Fast scan of common portsnmap -F target_host
# OS detection + service + scriptsnmap -A target_hostLightweight alternatives:
xxxxxxxxxx# netcat port scannc -zv target_host 80 443 8080 2>&1
# nc range scanfor port in $(seq 20 25); do nc -zv -w1 target_host $port 2>&1 | grep -v refuseddone
# bash /dev/tcptimeout 1 bash -c ">/dev/tcp/target_host/80" && echo open || echo closedDifferences:
| Tool | Method | Privilege | Speed | Stealth |
|---|---|---|---|---|
ss | kernel API | local only | instant | N/A |
| nmap -sT | full TCP | none | medium | no |
| nmap -sS | SYN only | root | fast | partial |
| nc -z | TCP connect | none | slow | no |
| bash /dev/tcp | TCP connect | none | slow | no |
Important: Port scanners only show what's reachable from the scanner's network position. A port may be open but blocked by firewall rules between you and the host — always scan from the same network segment as the actual clients.
ping works but curl to the same host fails.Answer:
ping uses ICMP; curl uses TCP (and DNS). The failure is somewhere in TCP or the application layer.
Step 1: Rule out DNS:
xxxxxxxxxxcurl -v http://hostname/ # may fail on DNScurl -v http://IP_ADDRESS/ # bypass DNS# If IP works: DNS problemStep 2: Test TCP connectivity directly:
xxxxxxxxxxnc -zv hostname 80 # or 443 for HTTPStelnet hostname 80 # manual HTTP testStep 3: If TCP connects, test HTTP manually:
xxxxxxxxxx# Raw HTTP requestnc hostname 80 << 'EOF'GET / HTTP/1.0Host: hostname
EOFStep 4: Check TLS issues (for HTTPS):
xxxxxxxxxxopenssl s_client -connect hostname:443 -servername hostname# Look for certificate errors, TLS version mismatchescurl -v --insecure https://hostname/ # skip cert validationcurl -v --tlsv1.2 https://hostname/ # force specific TLS versionStep 5: Check for firewall blocking TCP but allowing ICMP:
xxxxxxxxxxiptables -L -n | grep -E '(DROP|REJECT).*tcp'traceroute -T -p 80 hostname # TCP tracerouteStep 6: Check curl-specific issues:
xxxxxxxxxxcurl -v --trace - http://hostname/ # full protocol tracecurl --interface eth0 http://hostname/ # force specific interfacecurl --resolve hostname:80:IP http://hostname/ # manual DNS overrideCommon causes:
Firewall allows ICMP but drops TCP port 80/443
HTTP proxy required (check $http_proxy, $https_proxy env vars)
TLS certificate mismatch (ping doesn't care about certs)
Service listening on IPv4 only but curl connecting over IPv6 (or vice versa): curl -4 or curl -6
HTTP redirect to HTTPS that fails: use curl -L to follow redirects
ip route get and when is it more useful than looking at the routing table directly?Answer:
xxxxxxxxxxip route get 8.8.8.8# Output: 8.8.8.8 via 192.168.1.1 dev eth0 src 192.168.1.100 uid 0Why it's more useful than ip route show:
ip route show displays the routing table, which can have many overlapping prefixes, policy routing rules, and multiple routing tables. Determining which route would actually be used requires manually evaluating prefix length, metric, and policy routing rules in order.
ip route get asks the kernel the exact same question it asks when forwarding a real packet: "For this destination, what route do I use?" The kernel evaluates all routing tables, policy rules (from ip rule show), and returns the single winner.
Examples:
xxxxxxxxxx# What interface and source IP would be used to reach Google?ip route get 8.8.8.8
# What about traffic from a specific source address? (policy routing)ip route get 8.8.8.8 from 10.0.0.5
# Via a specific interface?ip route get 8.8.8.8 oif eth1
# Check policy routing (multiple routing tables)ip rule show # shows rules evaluated before main tableip route get 8.8.8.8 from 10.1.0.5 iif eth1Policy routing context:
xxxxxxxxxxip rule show# 0: from all lookup local# 100: from 10.1.0.0/24 lookup table1# 32766: from all lookup main# 32767: from all lookup defaultWhen multiple routing tables exist (common in multi-homed servers, VRFs, or Google's internal routing), ip route get is the only reliable way to know which table and route wins for a specific source+destination combination.
Answer:
Expected intra-datacenter latency: 0.1–0.5 ms. If you're seeing 1–50 ms, something is wrong.
Step 1: Confirm the latency and its variance:
xxxxxxxxxxping -c 100 -i 0.01 peer_host # fast pings, 100 samples# Look at: avg, max, and mdev (standard deviation = jitter)Step 2: Check if it's symmetric:
xxxxxxxxxx# From both sides simultaneously — asymmetric latency points to one directionssh peer_host "ping -c 50 $(hostname)"Step 3: Eliminate routing hops:
xxxxxxxxxxtraceroute -n peer_host # should be 1-2 hops in same DC# If 5+ hops: traffic is taking an unexpected path (BGP misconfiguration, wrong VLAN)Step 4: Check NIC and switch statistics:
xxxxxxxxxx# Interface errors and dropsip -s link show eth0ethtool -S eth0 | grep -E '(error|drop|miss|collision)'# Errors on the NIC = bad cable, bad SFP, duplex mismatch
# Check duplex (half duplex = severe latency/retransmit)ethtool eth0 | grep -E '(Speed|Duplex)'Step 5: Check CPU and interrupt handling:
xxxxxxxxxx# Soft IRQ: is the NIC IRQ handler getting CPU time?cat /proc/interrupts | grep eth0# IRQ affinitycat /proc/irq/*/smp_affinity_list
# Check for RX queue dropsethtool -S eth0 | grep rx_no_buffercat /proc/net/dev | grep eth0Step 6: Check for TCP bufferbloat:
xxxxxxxxxx# Large buffer fills = latency spike under loadsysctl net.core.rmem_max net.core.wmem_maxsysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmemStep 7: Check for CPU throttling/c-states:
xxxxxxxxxxcpupower frequency-info # check if CPU is in power-saving mode# High c-state latency can add 100-200μs to interrupt handlingStep 8: Physical layer (if nothing else explains it):
Check switch port error counters (via network team)
Check if hosts are on the same ToR switch (cross-rack adds ~100μs)
Jumbo frames misconfiguration
netstat or ss to find a port conflict when a service fails to start?Answer:
When a service fails to bind to a port you'll see errors like:
xxxxxxxxxxbind: Address already in uselisten tcp :8080: bind: address already in use
Find what's using the port:
xxxxxxxxxx# With ss (preferred)ss -tlnp | grep :8080# Output: LISTEN 0 128 0.0.0.0:8080 0.0.0.0:* users:(("nginx",pid=1234,fd=6))
# With lsoflsof -i :8080lsof -i TCP:8080 -sTCP:LISTEN
# With fuserfuser 8080/tcp # prints PIDfuser -v 8080/tcp # verbose with process name
# With netstat (older systems)netstat -tlnp | grep :8080If the port appears free but service still fails:
xxxxxxxxxx# Check if service is running as non-root trying to bind port < 1024# Ports 1-1023 require CAP_NET_BIND_SERVICEgetcap /usr/sbin/nginx # check capabilitiessystemctl show nginx | grep User # check which user it runs as
# Check SELinux/AppArmor port labelssemanage port -l | grep http_port_t # SELinuxaa-status # AppArmorCheck for lingering socket file (Unix domain sockets):
xxxxxxxxxxls -la /tmp/*.sock /run/*.sock /var/run/*.sock# Remove stale socket files from crashed processesCheck all sockets including non-listening:
xxxxxxxxxxss -anp | grep 8080 # -a = all states (not just LISTEN)# A socket in CLOSE_WAIT or TIME_WAIT on the same port can block re-bind# unless SO_REUSEADDR is setForce immediate reuse (application config):
Most servers set SO_REUSEADDR by default, which allows binding to a port in TIME_WAIT. If not:
xxxxxxxxxxsysctl -w net.ipv4.tcp_tw_reuse=1 # helps for outbound connectionsAnswer:
The SYN flood attack: A normal TCP 3-way handshake: Client sends SYN → Server allocates a half-open connection entry in the SYN queue → Server sends SYN-ACK → Client sends ACK → Connection promoted to accept queue.
In a SYN flood, an attacker sends millions of SYN packets with spoofed source IPs. The server allocates state for each one, fills the SYN queue (default: 128-1024 entries), and legitimate connections are rejected because the queue is full.
SYN cookies mechanism: Instead of allocating state on SYN receipt, the server encodes connection parameters into the ISN (Initial Sequence Number) of the SYN-ACK:
xxxxxxxxxxISN = hash(src_ip, src_port, dst_ip, dst_port, timestamp, secret) | encoded_MSS
No state stored. When the client returns an ACK, the server decodes the ISN to reconstruct connection parameters and creates the socket only then.
Legitimate clients will complete the handshake; spoofed IPs won't receive the SYN-ACK and thus never send ACK.
Check and configure:
xxxxxxxxxx# Check if syn cookies are enabledsysctl net.ipv4.tcp_syncookies# 0 = disabled, 1 = enable when queue full, 2 = always enable
# Enablesysctl -w net.ipv4.tcp_syncookies=1
# See if cookies are being sentnetstat -s | grep cookies# orcat /proc/net/netstat | awk '/TcpExt/{getline; print}' | tr ' ' '\n' | grep -A1 CookieTuning SYN queue:
xxxxxxxxxxsysctl net.ipv4.tcp_max_syn_backlog # per-socket SYN queuesysctl net.core.somaxconn # listen() backlog (accept queue)Trade-offs of SYN cookies:
TCP options (window scaling, SACK, timestamps) cannot be negotiated (no space in ISN) — some implementations encode basic options
Slight CPU overhead for hash computation per packet
Not a full solution: attacker can still exhaust bandwidth or accept queue with completed connections
Modern complement: nftables / iptables rate limiting:
xxxxxxxxxxiptables -A INPUT -p tcp --syn -m limit --limit 100/s --limit-burst 200 -j ACCEPTiptables -A INPUT -p tcp --syn -j DROPAnswer:
Step 1: Enable core dumps:
xxxxxxxxxx# Check current limit (0 = disabled)ulimit -culimit -Sa | grep core
# Enable for current shell sessionulimit -c unlimited
# Enable permanently system-wideecho '* soft core unlimited' >> /etc/security/limits.confecho '* hard core unlimited' >> /etc/security/limits.confStep 2: Configure core file naming and location:
xxxxxxxxxx# Default: "core" in CWD (may be non-writable)cat /proc/sys/kernel/core_pattern
# Set to write to /var/crash with PID and program namesysctl -w kernel.core_pattern=/var/crash/core.%e.%p.%t# %e = executable name, %p = PID, %t = timestamp
# Verifycat /proc/sys/kernel/core_uses_pidStep 3: Check if systemd-coredump is active (modern systems):
xxxxxxxxxxcat /proc/sys/kernel/core_pattern# If it starts with |/usr/lib/systemd/systemd-coredump: systemd manages cores
# List stored corescoredumpctl listcoredumpctl list myapp
# Show info about last crashcoredumpctl info
# Extract core filecoredumpctl dump -o /tmp/myapp.core myappStep 4: Analyze with gdb:
xxxxxxxxxxgdb /usr/bin/myapp /var/crash/core.myapp.1234.1705123456
# In gdb:(gdb) bt # backtrace — shows call stack at crash(gdb) bt full # backtrace with local variables(gdb) info registers # register state(gdb) info threads # all threads(gdb) thread apply all bt # backtrace for every thread(gdb) frame 2 # switch to frame 2(gdb) list # show source around current frame(gdb) print variable_name # inspect variable value(gdb) x/20x $rsp # examine 20 hex words at stack pointerStep 5: Ensure debug symbols are present:
xxxxxxxxxx# Install debug symbols (Debian/Ubuntu)apt-get install myapp-dbgsym# ordebuginfod-find debuginfo $(file /usr/bin/myapp | grep -oP 'BuildID\[sha1\]=\K\w+')
# Check if binary has symbolsfile /usr/bin/myapp # "not stripped" = has symbolsnm /usr/bin/myapp | head # symbol tableWithout source/symbols — at minimum:
xxxxxxxxxx(gdb) bt # even without symbols, shows crash address(gdb) info sharedlibrary # shows loaded .so files(gdb) disassemble # assembly of current framedmesg and the core dump?Answer:
Step 1: Find the segfault in dmesg:
xxxxxxxxxxdmesg -T | grep segfault# Example output:# [Mon Jan 15 10:23:45 2024] myapp[12345]: segfault at 7f00000000 ip 00007f1234567890 sp 00007fff12345678 error 4
# error codes (octal, bit field):# bit 0: 0=no page, 1=protection fault# bit 1: 0=read, 1=write# bit 2: 0=kernel, 1=user# bit 3: 1=use of reserved bit in page table# error 4 = user-space read of unmapped memory# error 6 = user-space write of unmapped memory (common: NULL ptr deref write)# error 7 = user-space write to read-only page (const data write, buffer overflow into .text)Decode the instruction pointer:
xxxxxxxxxx# ip 00007f1234567890 — translate to function nameaddr2line -e /usr/bin/myapp 0x00007f1234567890 -f# or with gdb:gdb /usr/bin/myapp -batch -ex "info symbol 0x00007f1234567890"Step 2: Check if ASLR affects the address:
xxxxxxxxxxcat /proc/sys/kernel/randomize_va_space# 2 = full ASLR (default), addresses differ per run# 0 = disabled (useful for debugging)sysctl -w kernel.randomize_va_space=0 # disable temporarilyStep 3: gdb analysis of core:
xxxxxxxxxxgdb /usr/bin/myapp core.myapp.12345
(gdb) bt# Look for the frame where crash occurred(gdb) frame N # switch to crash frame(gdb) print ptr # inspect the NULL/invalid pointer(gdb) info locals(gdb) x/1x address_from_dmesg # examine that memory addressCommon patterns:
NULL pointer dereference: at 0000000000000000 or small offset (like at 8 = 8 bytes into a NULL struct)
Use-after-free: address in freed heap region, often with error 4 (read)
Stack overflow: at address is in stack range, error 6
Buffer overflow into return address: crash in a strange function after a strcpy/memcpy
Tools for deeper analysis:
xxxxxxxxxx# Valgrind (run the program under it)valgrind --leak-check=full --track-origins=yes ./myapp
# AddressSanitizer (recompile with -fsanitize=address)gcc -fsanitize=address -g -o myapp_asan myapp.c./myapp_asan # detailed error with stack trace on crashstrace to debug a hanging or misbehaving process, and how do you limit the overhead?Answer:
Attach to a running process:
xxxxxxxxxxstrace -p PID# Ctrl+C to detach (process continues running)Trace from start:
xxxxxxxxxxstrace ./myapp arg1 arg2Common useful flags:
xxxxxxxxxx# Follow forked childrenstrace -f ./myapp
# Timestamp each call (absolute and relative)strace -t -r ./myapp # -t: absolute time, -r: relative to prev call
# Filter to specific syscallsstrace -e trace=open,read,write,close ./myappstrace -e trace=network ./myapp # all network syscallsstrace -e trace=file ./myapp # all file-relatedstrace -e trace=signal ./myapp
# Show string contents (default truncates at 32 bytes)strace -s 256 ./myapp
# Write output to file (doesn't mix with program output)strace -o /tmp/strace.out ./myapp
# Count syscalls (summary, low overhead)strace -c ./myapp# Output: sorted table of syscall counts, time spent, errorsDiagnosing a hanging process:
xxxxxxxxxxstrace -p PID -e trace=all 2>&1 | head -20# A process hung on:# read(fd, ...) — waiting for input on a pipe/socket/file# futex(addr, FUTEX_WAIT, ...) — blocked on a mutex/lock (deadlock?)# epoll_wait(fd, ...) — event loop waiting for events (often normal)# nanosleep(...) — sleeping (check if loop with backoff or just stuck)# wait4(-1, ...) — waiting for child processFor performance overhead:
xxxxxxxxxx# strace adds 5-100x overhead due to ptrace stops on every syscall# Use -c for summary (lower overhead, no per-call recording)strace -c -p PID -e trace=all
# Use perf to see syscall distribution with lower overheadperf stat -e syscalls:sys_enter_read,syscalls:sys_enter_write -p PID sleep 5
# For production: use eBPF (zero overhead by comparison)bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s\n", str(args->filename)); }'ltrace for library calls:
xxxxxxxxxxltrace -p PID # trace library calls (malloc, free, printf, etc.)ltrace -e malloc+free # only malloc/free — memory allocation trackingltrace -c ./myapp # summary count/proc/<pid>/fd give you, and how do you use it to debug file descriptor leaks?Answer:
/proc/<pid>/fd/ is a directory of symlinks, one per open file descriptor, pointing to the actual resource:
xxxxxxxxxxls -la /proc/1234/fd/# lrwx------ 1 user user 64 ... 0 -> /dev/pts/0 (terminal)# lrwx------ 1 user user 64 ... 1 -> /dev/pts/0 (stdout)# lrwx------ 1 user user 64 ... 2 -> /dev/pts/0 (stderr)# lr-x------ 1 user user 64 ... 3 -> /lib/x86_64.so.6 (shared lib)# lrwx------ 1 user user 64 ... 5 -> socket:[123456] (TCP socket)# lr-x------ 1 user user 64 ... 7 -> /var/log/app.log (log file)# lrwx------ 1 user user 64 ... 9 -> anon_inode:[eventfd] (eventfd)Count open FDs for a process:
xxxxxxxxxxls /proc/1234/fd | wc -l# Compare to limit:cat /proc/1234/limits | grep 'open files'Detect FD leak over time:
xxxxxxxxxx# Watch FD count growingwatch -n 2 'ls /proc/1234/fd | wc -l'
# More detail: what types are accumulating?ls -la /proc/1234/fd | awk '{print $NF}' | sed 's/\(.*\)\/.*/\1/' | sort | uniq -c | sort -rnfdinfo — additional metadata per FD:
xxxxxxxxxxcat /proc/1234/fdinfo/5# pos: 12345 (current file offset)# flags: 0100002 (O_RDWR | O_NONBLOCK in octal)# mnt_id: 23# For sockets: additional socket stateFind deleted files still held open (common cause of "disk full but no files"):
xxxxxxxxxxlsof -p 1234 | grep deleted# orls -la /proc/1234/fd | grep '(deleted)'# These are files unlinked from the filesystem but still consuming inodes/blocks# Fix: rotate logs (logrotate), which sends SIGHUP to reopen filesSystem-wide FD usage:
xxxxxxxxxxcat /proc/sys/fs/file-nr# <open> <available-for-reuse> <max># e.g.: 12345 0 9223372036854775807
cat /proc/sys/fs/file-max # system-wide maxulimit -n # per-process limit (current shell)Increase FD limits:
xxxxxxxxxx# Temporary (current session)ulimit -n 65535
# Permanent (systemd services)# In /etc/systemd/system/myapp.service:# [Service]# LimitNOFILE=65535
# Or in /etc/security/limits.conf:# myapp-user soft nofile 65535# myapp-user hard nofile 65535Answer:
Step 1: Confirm which process and threads:
xxxxxxxxxxtop -H -p PID # -H shows individual threads# orps -eLo pid,tid,pcpu,comm | sort -rk3 | head -20
# Identify specific threadscat /proc/PID/task/TID/status | grep -E '(Name|State|VmRSS)'Step 2: See what the process is doing:
xxxxxxxxxx# strace syscall summarystrace -c -p PID sleep 5# High sys_futex: lock contention (mutex thrashing)# High nanosleep: spin-wait with sleep# High read/write: I/O bound (not CPU)
# perf — kernel-level profilingperf top -p PID # live, per-function CPU usageperf record -F 99 -p PID -g -- sleep 10 # record 10 secondsperf report # interactive flame-graph-like viewperf script | flamegraph.pl > flames.svg # generate flame graphStep 3: Application-level profiling:
xxxxxxxxxx# For Java: thread dump shows what's runningkill -3 PID # SIGQUIT: dumps thread info to stderr/logjstack PID # Java thread dump with stack traces
# For Pythonpy-spy top --pid PID # live viewpy-spy record -o profile.svg --pid PID # flame graph
# For Gokill -SIGABRT PID # dump goroutines# or use pprof endpoint: curl http://localhost:6060/debug/pprof/profile?seconds=30
# For Node.jskill -SIGUSR1 PID # enable inspector, then connect with node --inspectStep 4: Check for CPU throttling (containers/cgroups):
xxxxxxxxxxcat /sys/fs/cgroup/cpu/myapp/cpu.stat# nr_throttled: times the container was throttled# throttled_time: total nanoseconds throttled
cat /sys/fs/cgroup/cpu/myapp/cpu.cfs_quota_us # CPU limit in microsecondscat /sys/fs/cgroup/cpu/myapp/cpu.cfs_period_us # period# quota/period = number of CPUs allocatedStep 5: Check for runaway loops:
xxxxxxxxxx# Is the process in kernel or user space?cat /proc/PID/status | grep State# R (running) — confirmed CPU bound# S (sleeping) — waiting on something
# Check if it's a tight spin loopstrace -p PID 2>&1 | head # if output flies by with no waits: spin loopldd and ldconfig work and how you debug missing shared library errors.Answer:
The error:
xxxxxxxxxxerror while loading shared libraries: libfoo.so.2: cannot open shared object file: No such file or directory
ldd — show shared library dependencies:
xxxxxxxxxxldd /usr/bin/myapp# Output:# linux-vdso.so.1 (0x00007fff...)# libfoo.so.2 => /usr/lib/libfoo.so.2 (0x00007f...)# libbar.so.1 => not found <-- problem# libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f...)
# Security note: never run ldd on untrusted binaries# ldd executes the binary (via ld.so); use readelf instead for untrusted:readelf -d /usr/bin/myapp | grep NEEDEDldconfig — update the linker cache:
xxxxxxxxxx# The dynamic linker reads /etc/ld.so.cache (binary cache of library paths)# It searches paths from:# 1. RPATH/RUNPATH embedded in the binary# 2. LD_LIBRARY_PATH environment variable# 3. /etc/ld.so.cache (built from /etc/ld.so.conf and /etc/ld.so.conf.d/)# 4. /lib and /usr/lib (default)
# Rebuild cache after installing a libraryldconfigldconfig -v | grep libfoo # verify it found the library
# Check where ldconfig lookscat /etc/ld.so.confls /etc/ld.so.conf.d/Debugging steps:
xxxxxxxxxx# 1. Find where the library actually isfind /usr /lib /opt -name "libfoo.so*" 2>/dev/null
# 2. If found in non-standard path, add to ldconfigecho '/opt/myapp/lib' > /etc/ld.so.conf.d/myapp.confldconfig
# 3. Or set LD_LIBRARY_PATH (less preferred, but useful for testing)LD_LIBRARY_PATH=/opt/myapp/lib ./myapp
# 4. Check version mismatch (libfoo.so.2 vs libfoo.so.1)ls -la /usr/lib/libfoo*# libfoo.so -> libfoo.so.2.1.0 (symlink set by ldconfig)# libfoo.so.2 -> libfoo.so.2.1.0 (soname symlink)# libfoo.so.2.1.0 (actual file)
# 5. Check what soname the binary requiresobjdump -p /usr/bin/myapp | grep NEEDED# NEEDED libfoo.so.2 (this exact soname must be present)
# 6. Use LD_DEBUG for detailed linker traceLD_DEBUG=libs ./myapp 2>&1 | grep libfooLD_DEBUG=all ./myapp 2>&1 | head -50RPATH vs RUNPATH:
xxxxxxxxxxreadelf -d /usr/bin/myapp | grep -E '(RPATH|RUNPATH)'# RPATH: searched before LD_LIBRARY_PATH (cannot be overridden by env var)# RUNPATH: searched after LD_LIBRARY_PATH (can be overridden)Answer:
What is a zombie process?
A process that has exited but whose parent has not yet called wait() to collect its exit status. The kernel keeps a minimal entry in the process table (just the exit code and resource accounting) until the parent reads it.
xxxxxxxxxxps aux | grep Z # State Z = zombieps -eo pid,ppid,stat,comm | awk '$3 ~ /^Z/'Zombie entries show:
xxxxxxxxxxPID PPID STAT COMMAND12345 1000 Z myapp <defunct>
How to resolve:
Find the parent and send SIGCHLD:
xxxxxxxxxx# Find parent PIDps -o ppid= -p 12345 # or from ps -eo output
# Send SIGCHLD to parent (signal that a child has changed state)kill -SIGCHLD PPID# If parent handles SIGCHLD correctly, it will call wait() and reap the zombieIf parent won't reap — kill the parent:
xxxxxxxxxxkill PPID# When parent dies, init (PID 1) adopts all children and reaps them# Systemd/init calls wait() continuously for adopted childrenCannot kill the zombie itself:
xxxxxxxxxxkill -9 12345 # does NOT work — zombie is already dead# Only the parent's wait() call or parent's death removes itRoot causes:
Parent process has a bug: forked a child but doesn't call wait() or waitpid()
Parent is caught in a long-running operation and hasn't gotten to wait()
Improper SIGCHLD handling (SIG_IGN is actually fine — kernel auto-reaps; custom handler that doesn't call wait is the bug)
Parent itself is hung/deadlocked
Impact:
Zombies don't consume CPU or memory (except one process table slot)
If PID space fills up (typically 32768 PIDs), new processes cannot be created
Large numbers of zombies indicate a bug in the parent process
Set SIGCHLD to SIG_IGN in the parent to tell the kernel to auto-reap children (no wait() needed):
xxxxxxxxxxsignal(SIGCHLD, SIG_IGN); // children auto-reapedAnswer:
Using gcore:
xxxxxxxxxxgcore -o /tmp/myapp_snapshot PID# Generates /tmp/myapp_snapshot.PID# Process is briefly suspended (SIGSTOP) while the dump is written, then resumedUsing gdb:
xxxxxxxxxxgdb -p PID(gdb) generate-core-file /tmp/myapp_snapshot.core(gdb) detach(gdb) quitUsing /proc/<pid>/mem (manual, more complex):
xxxxxxxxxx# cat /proc/PID/maps shows memory regions# Requires reading and writing specific regions# gcore is the right tool for thisFor Java — generate heap dump (different from core dump):
xxxxxxxxxxjmap -dump:format=b,file=/tmp/heap.hprof PID# For live objects only:jmap -dump:live,format=b,file=/tmp/heap_live.hprof PIDWhy non-destructive core dumps are useful:
Inspect state of a production process at a point in time without restarting it
Compare snapshots before and after a memory leak grows
Capture state when intermittent bug first appears (watchdog detects anomaly, triggers gcore)
Considerations:
Writing the dump suspends the process — for large heap sizes this can be seconds to minutes
Core file can be very large (full virtual memory size, though sparse)
Compress immediately: gcore -o /tmp/snap PID && gzip /tmp/snap.PID
Check available disk space first: df -h /tmp
Using /proc/sysrq-trigger for kernel-level core dump (NMI):
xxxxxxxxxxecho c > /proc/sysrq-trigger # kernel panic + crash dump (destructive!)# Use only for kernel debugging, not application debuggingulimit -n control, and what happens when a process exceeds it?Answer:
ulimit -n controls the maximum number of open file descriptors for a process (RLIMIT_NOFILE). In Linux, everything is a file descriptor: regular files, directories, sockets, pipes, device files, eventfd, timerfd, inotify watches, etc.
Checking limits:
xxxxxxxxxxulimit -n # soft limit for current shellulimit -Hn # hard limit for current shellcat /proc/PID/limits # limits for a running process# Max open files 1024 4096 files# (soft) (hard)What happens when limit is exceeded:
open() returns -1 with errno = EMFILE ("Too many open files")
socket() returns -1 with errno = EMFILE
accept() returns -1 with errno = EMFILE
Application logs typically show: "Error accepting connection: too many open files"
Services like nginx, Java apps, databases fail to accept new connections
Check current FD usage:
xxxxxxxxxxls /proc/PID/fd | wc -l# Compare to limit in /proc/PID/limitsIncrease limits:
xxxxxxxxxx# Temporary for current shellulimit -n 65535
# Permanent via /etc/security/limits.conf (PAM-based sessions)echo 'myuser soft nofile 65535' >> /etc/security/limits.confecho 'myuser hard nofile 65535' >> /etc/security/limits.conf# Requires logout/login to take effect
# For systemd services (overrides limits.conf for system services)# In /etc/systemd/system/myservice.service:# [Service]# LimitNOFILE=65535systemctl daemon-reload && systemctl restart myservice
# System-wide maximum (affects all processes)sysctl -w fs.file-max=2097152echo 'fs.file-max = 2097152' >> /etc/sysctl.confTypical values:
Default Linux: 1024 soft, 4096 hard (very low for production)
Database servers (PostgreSQL, MySQL): 65536+
High-traffic web servers (nginx): 65536–1048576
Absolute hard ceiling: /proc/sys/fs/nr_open (default 1048576)
strace to trace system calls of a multi-threaded process?Answer:
The challenge: By default, strace -p PID attaches only to the main thread. Other threads (which are separate tasks in Linux) are not traced.
Trace all existing threads:
xxxxxxxxxxstrace -p PID $(ls /proc/PID/task | awk '{printf " -p %s", $1}')# Gets all thread IDs from /proc/PID/task/ and attaches to eachTrace the main process AND follow all future forks/clones:
xxxxxxxxxxstrace -f -p PID# -f: follow forks (also works for threads created via clone(CLONE_THREAD))From process start with all threads:
xxxxxxxxxxstrace -f -o /tmp/strace.out ./myapp# Each line prefixed with PID/TID: [12345] read(3, ...Focus on specific syscalls across all threads:
xxxxxxxxxxstrace -f -e trace=futex -p PID 2>&1 | head -50# futex: mutex operations — shows lock contention patterns# FUTEX_WAIT = blocked waiting for lock# FUTEX_WAKE = releasing lock and waking waitersInterpret multi-threaded output:
xxxxxxxxxx# With -f, output is interleaved by TID# Use grep to focus on one threadstrace -f -o /tmp/out.txt ./myappgrep '^1234 ' /tmp/out.txt # filter by specific TID
# Unfinished syscalls (context switched away mid-call)# [pid 1234] read(5, <unfinished ...># [pid 1235] futex(0x7f1234, FUTEX_WAIT, ...# [pid 1234] <... read resumed> "data", 100) = 100Deadlock detection with strace:
xxxxxxxxxxstrace -f -e trace=futex -p PID 2>&1 | grep FUTEX_WAIT# Multiple threads blocked on FUTEX_WAIT at same address = deadlock candidate# Note the addresses: two threads waiting on each other's locks = deadlockBetter tools for production thread analysis:
xxxxxxxxxx# gdb for all thread backtracesgdb -p PID -batch -ex 'thread apply all bt' 2>/dev/null
# perf for low-overhead thread profilingperf record -F 99 -p PID -g --call-graph dwarf sleep 10Answer:
This is a concurrency/resource contention problem. Systematic approach:
Step 1: Identify the failure mode:
xxxxxxxxxx# What error does it produce?strace -f ./myapp 2>&1 | grep -E 'E(EXIST|AGAIN|BUSY|LOCK|DEADLK)'# EEXIST: trying to create something that already exists (no uniqueness)# EAGAIN/EWOULDBLOCK: non-blocking operation would block (shared resource)# EDEADLK: deadlock detected# EBUSY: device/resource busyStep 2: Check for shared resource contention:
Temp files with non-unique names:
xxxxxxxxxxls -la /tmp/myapp* # multiple instances creating same temp file# Fix: use mkstemp() or include PID in filenamePort conflicts:
xxxxxxxxxxss -tlnp | grep myapp # multiple instances binding same port# Fix: use SO_REUSEPORT (Linux 3.9+) — kernel distributes connectionsLock files:
xxxxxxxxxxflock -n /var/lock/myapp.lock -c './myapp' || echo "already running"# Proper lock file usage with flock prevents thisDatabase connections:
xxxxxxxxxx# Connection pool exhaustion — use connection pooling middleware# PgBouncer for PostgreSQL, ProxySQL for MySQLStep 3: Race conditions on shared state:
xxxxxxxxxx# Use ltrace to see shared memory / semaphore callsltrace -e shm_open+sem_open+sem_wait ./myapp
# Check for /dev/shm usagels /dev/shm/
# strace to see file lockingstrace -e trace=flock,fcntl ./myappStep 4: Reproduce deterministically:
xxxxxxxxxx# Run N instances simultaneouslyfor i in $(seq 1 10); do ./myapp & donewait
# Increase likelihood of race with tight timingtaskset -c 0 ./myapp & # pin to same CPU to maximize scheduling interleavetaskset -c 0 ./myapp &Step 5: Use helgrind for threading bugs:
xxxxxxxxxxvalgrind --tool=helgrind ./myapp# Detects lock ordering violations, missing locks, data racesStep 6: Linux futex analysis:
xxxxxxxxxxperf record -e 'syscalls:sys_enter_futex' -p PID sleep 5perf script | head -50strace and ltrace, and when would you use each?Answer:
| Feature | strace | ltrace |
|---|---|---|
| Traces | Kernel system calls | User-space library calls |
| Mechanism | ptrace + seccomp | ptrace + LD_PRELOAD-like |
| Examples | read(), write(), open(), mmap(), fork() | malloc(), free(), printf(), strcmp(), fopen() |
| Overhead | Moderate (ptrace stop per syscall) | Higher (intercepts all PLT entries) |
| Use for | I/O, networking, filesystem, signals | Memory allocation, string ops, library bugs |
When to use strace:
xxxxxxxxxx# Process failing to open a file? strace shows exact pathstrace -e trace=openat,open ./myapp 2>&1 | grep ENOENT
# Network issues? strace shows connect() callsstrace -e trace=connect,bind,socket ./myapp
# Process stuck? strace shows what syscall it's blocked onstrace -p PID # immediately shows: futex, read, poll, etc.
# Permission denied? strace shows which file/syscallstrace -e trace=all -p PID 2>&1 | grep EPERMWhen to use ltrace:
xxxxxxxxxx# Memory allocation patternsltrace -e malloc+free+realloc ./myapp 2>&1 | grep malloc | wc -l
# String operations (buffer overflow investigation)ltrace -e strcpy+strcat+sprintf ./myapp
# Configuration parsing (what config files are being read via fopen)ltrace -e fopen+fread ./myapp
# Crypto operationsltrace -e EVP_+SSL_ ./myappCombined approach:
xxxxxxxxxx# Get both at oncestrace -f -o /tmp/strace.txt ./myapp <race -f -o /tmp/ltrace.txt ./myapp# Compare timelines to correlate library call → syscallProduction alternative (lower overhead):
xxxxxxxxxx# bpftrace for syscall tracingbpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s\n", str(args->filename)); }'
# uprobes for library function tracingbpftrace -e 'uprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc { printf("%d bytes\n", arg0); }'Answer:
Step 1: Confirm and characterize the leak:
xxxxxxxxxx# Monitor RSS over timewatch -n 5 'ps -o pid,rss,vsz,comm -p PID'
# Or log itwhile true; do echo "$(date): $(cat /proc/PID/status | grep VmRSS)" sleep 60done >> /tmp/memory_trend.log
# smaps for detailed breakdowncat /proc/PID/smaps | awk '/^Size/{size+=$2} END{print size/1024 "MB"}'cat /proc/PID/smaps_rollup # summary of all memory areasStep 2: Differentiate leak types:
xxxxxxxxxx# Virtual vs RSS: VSZ growing fast = mapping lots of memory but not using (mmap leak)# RSS growing steadily = actual heap/stack leak# Heap inspectioncat /proc/PID/maps | grep heap# [heap] shows current heap boundaries
# malloc stats (glibc)kill -SIGUSR1 PID # if app has malloc_info() dump on SIGUSR1Step 3: Use heap profiler:
xxxxxxxxxx# Valgrind memcheckvalgrind --leak-check=full --show-leak-kinds=all --track-origins=yes ./myapp
# gperftools (Google's TCMalloc heap profiler)LD_PRELOAD=/usr/lib/libprofiler.so HEAPPROFILE=/tmp/heap_profile ./myapppprof --pdf /usr/bin/myapp /tmp/heap_profile.0001.heap > leak.pdf
# jemalloc profilingLD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2 \ MALLOC_CONF=prof:true,prof_leak:true,lg_prof_interval:30 ./myappjeprof --show_bytes --pdf /usr/bin/myapp jeprof.out.*.heap > leak.pdfStep 4: Java-specific:
xxxxxxxxxx# Enable GC loggingjava -Xmx512m -Xlog:gc*:file=/tmp/gc.log ./myapp.jar
# Heap histogram without full dumpjmap -histo:live PID | head -30
# Generate heap dump when OOMjava -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/heapdump.hprof ./myapp.jar
# Analyze with MAT (Memory Analyzer Tool) or jhatjhat /tmp/heapdump.hprof # web UI on port 7000Step 5: Check kernel OOM killer:
xxxxxxxxxxdmesg -T | grep -E '(oom-killer|Killed process|Out of memory)'# Shows which processes were killed, their memory usage at time of kill
# OOM score — higher = more likely to be killedcat /proc/PID/oom_scorecat /proc/PID/oom_score_adj # adjust: -1000 = never kill, 1000 = always killAnswer:
strace approach:
xxxxxxxxxxstrace -e trace=openat,open,read,stat,lstat -T -o /tmp/startup.txt ./myapp# -T: show time spent in each syscall# Sort by time to find slow file readsgrep 'openat\|open(' /tmp/startup.txt | grep -v ENOENT | head -30
# Find files that took longestgrep '<[0-9]' /tmp/startup.txt | sort -t'<' -k2 -rn | head -20inotifywait for real-time file access:
xxxxxxxxxx# Monitor a directory for access eventsinotifywait -m -r -e access,open /etc /usr/lib /var/lib &WATCH_PID=$!./myappkill $WATCH_PIDfatrace (faster, uses fanotify):
xxxxxxxxxxfatrace -p PID -f O -o /tmp/fatrace.txt & # -f O = open events only# or system-widefatrace | grep myappblktrace for disk I/O timing:
xxxxxxxxxxblktrace -d /dev/sda -o /tmp/blktrace &./myappkill %1 # stop blktraceblkparse /tmp/blktrace.blktrace.0 | grep myappperf for comprehensive I/O tracing:
xxxxxxxxxxperf trace -p PID -e openat,read 2>&1 | head -50# Shows timing per syscall with microsecond resolutionAnalysis pattern:
xxxxxxxxxx# After collecting strace output# 1. Find all unique files openedgrep 'openat.*O_RDONLY' /tmp/startup.txt | grep -oP '"[^"]*"' | sort -u
# 2. Find files accessed many times (reading config in loop)grep 'openat' /tmp/startup.txt | grep -oP '"[^"]*"' | sort | uniq -c | sort -rn | head
# 3. Find large readsgrep 'read(' /tmp/startup.txt | grep -v '= -1' | awk -F'= ' '{print $NF}' | sort -rn | head/proc/<pid>/maps and /proc/<pid>/smaps to understand a process's memory layout?Answer:
/proc/<pid>/maps — memory map:
xxxxxxxxxxcat /proc/1234/maps# address perm offset dev inode pathname# 55a1234000-55a1456000 r-xp 00000000 08:01 123456 /usr/bin/myapp (text segment)# 55a1658000-55a1678000 r--p 00224000 08:01 123456 /usr/bin/myapp (read-only data)# 55a1678000-55a1690000 rw-p 00244000 08:01 123456 /usr/bin/myapp (writable data/BSS)# 55a1a00000-55a2000000 rw-p 00000000 00:00 0 [heap]# 7f1234000000-7f1234021000 r-xp 00000000 08:01 789 /lib/libc.so.6# 7fff12340000-7fff12360000 rw-p 00000000 00:00 0 [stack]# 7fff12361000-7fff12363000 r--p 00000000 00:00 0 [vvar]# 7fff12363000-7fff12365000 r-xp 00000000 00:00 0 [vdso]Permission flags:
r = readable, w = writable, x = executable, p = private (CoW), s = shared
/proc/<pid>/smaps — detailed per-region stats:
xxxxxxxxxxcat /proc/1234/smaps | head -30# For each mapping:# Size: 1024 kB (total virtual size)# KernelPageSize: 4 kB# MMUPageSize: 4 kB# Rss: 512 kB (resident in physical RAM)# Pss: 256 kB (proportional: RSS / number of processes sharing)# Shared_Clean: 256 kB (shared, not modified)# Shared_Dirty: 0 kB (shared, modified)# Private_Clean: 128 kB (private, not modified — zero pages)# Private_Dirty: 128 kB (private, modified — actual memory cost)# Referenced: 512 kB# Anonymous: 128 kB (no file backing)# Swap: 0 kB# SwapPss: 0 kBPSS (Proportional Set Size) is the most accurate measure of a process's memory cost — it divides shared pages among all processes using them.
Useful analysis scripts:
xxxxxxxxxx# Total RSS and PSSawk '/^Rss/{rss+=$2} /^Pss/{pss+=$2} END{print "RSS:", rss/1024, "MB\nPSS:", pss/1024, "MB"}' \ /proc/1234/smaps
# Find largest anonymous regions (heap allocations, thread stacks)awk '/^[0-9a-f]/{addr=$1} /Anonymous:/{print $2, addr}' /proc/1234/smaps | \ sort -rn | head -10
# smaps_rollup (kernel 4.14+) — precomputed summarycat /proc/1234/smaps_rollup
# Check huge pages usagegrep -i huge /proc/1234/smaps | head
# Find memory-mapped filesgrep -v '^\(00000\|7fff\|7f\)' /proc/1234/maps | grep -v '^\[' | awk '{print $6}' | sort -uUse cases:
Diagnosing why VSZ is huge but RSS is small (lots of mmap'd but not accessed memory)
Finding which shared library contributes most to memory usage
Confirming copy-on-write is working (Shared_Clean should dominate for shared libs)
Debugging memory fragmentation (many small Private_Dirty anonymous regions)
journalctl to filter logs for a specific service, time range, and priority level?Answer:
Filter by service unit:
xxxxxxxxxxjournalctl -u nginx.servicejournalctl -u nginx.service -u php-fpm.service # multiple unitsFilter by time:
xxxxxxxxxxjournalctl --since "2024-01-15 10:00:00" --until "2024-01-15 11:00:00"journalctl --since "1 hour ago"journalctl --since todayjournalctl --since yesterday --until todayFilter by priority (syslog severity levels):
xxxxxxxxxxjournalctl -p err # err and above (err, crit, alert, emerg)journalctl -p warning # warning and abovejournalctl -p 3 # numeric: 0=emerg, 1=alert, 2=crit, 3=err, 4=warning, 5=notice, 6=info, 7=debugjournalctl -p 0..3 # range: emerg through errKernel messages only:
xxxxxxxxxxjournalctl -k # kernel ring buffer (equivalent to dmesg)journalctl -k --since "10 min ago"Combined filters:
xxxxxxxxxxjournalctl -u nginx -p err --since "1 hour ago" --no-pagerFollow live (like tail -f):
xxxxxxxxxxjournalctl -fu nginx.service # -f = follow, -u = unitjournalctl -f -p err # follow all errors system-wideOutput formats:
xxxxxxxxxxjournalctl -u nginx -o json-pretty # full JSON with all metadata fieldsjournalctl -u nginx -o short-precise # default with microsecond timestampsjournalctl -u nginx -o cat # message only, no headersShow last N lines:
xxxxxxxxxxjournalctl -u nginx -n 100Search for pattern:
xxxxxxxxxxjournalctl -u nginx | grep -E "error|CRIT"journalctl -u nginx --grep "connection refused" # journalctl built-in grep (systemd 243+)Disk usage:
xxxxxxxxxxjournalctl --disk-usagejournalctl --vacuum-size=500M # trim old entries to stay under 500MBjournalctl --vacuum-time=30d # remove entries older than 30 daysrsyslog work, and how would you configure it to forward logs to a central server?Answer:
Architecture: rsyslog uses a modular pipeline: Input modules (imjournal, imudp, imtcp, imfile) → Processing (filtering, property-based routing, templates) → Output modules (omfwd for forwarding, omfile for local files, omelasticsearch, etc.)
Configuration file structure:
xxxxxxxxxx# /etc/rsyslog.conf and /etc/rsyslog.d/*.conf# Three config statement types:# 1. $legacy directives (old style)# 2. RainerScript (modern, preferred)# 3. Basic syslog lines: facility.priority destinationBasic local logging (default /etc/rsyslog.conf):
xxxxxxxxxx# Log all auth messages to auth.logauth,authpriv.* /var/log/auth.log# Log all except debug to syslog*.*;auth,authpriv.none -/var/log/syslog # '-' prefix = async write# Kernel messageskern.* /var/log/kern.logForward to central syslog server (UDP — fire and forget):
xxxxxxxxxx# /etc/rsyslog.d/99-forward.conf*.* @central-syslog.example.com:514# @ = UDP, @@ = TCPForward with TCP and queuing (reliable forwarding):
xxxxxxxxxx# Load TCP output modulemodule(load="omfwd")
action( type="omfwd" target="central-syslog.example.com" port="514" protocol="tcp" # Queue to handle central server unavailability queue.type="LinkedList" queue.size="10000" queue.filename="fwdRule" queue.saveOnShutdown="on" action.resumeRetryCount="-1")Forward only specific facility/severity:
xxxxxxxxxxif $syslogfacility-text == 'kern' and $syslogseverity <= 4 then { action(type="omfwd" target="central-syslog.example.com" port="514" protocol="tcp")}Receive logs from remote (central server config):
xxxxxxxxxx# Enable UDP receptionmodule(load="imudp")input(type="imudp" port="514")
# Enable TCP receptionmodule(load="imtcp")input(type="imtcp" port="514")
# Write remote logs to per-host files$template RemoteLogs,"/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log"*.* ?RemoteLogsTest and reload:
xxxxxxxxxxrsyslogd -N1 # syntax checkrsyslogd -N2 # more detailed validationsystemctl reload rsysloglogger -p auth.err "test error message" # send test messagedmesg effectively for diagnosing kernel-level events, and what does -T do?Answer:
Basic usage:
xxxxxxxxxxdmesg # print kernel ring buffer (raw timestamps in seconds since boot)dmesg -T # -T: convert timestamps to human-readable date/time # Example: [Mon Jan 15 10:23:45 2024] instead of [12345.678901]dmesg -w # follow: equivalent of tail -f for kernel messagesdmesg -H # human-readable with paging (implies -T on modern systems)Filter by level:
xxxxxxxxxxdmesg -l err,crit,alert,emerg # only errors and abovedmesg -l warn # warnings and abovedmesg --level=err # sameFilter by facility:
xxxxxxxxxxdmesg -f kern # kernel messages onlydmesg -f daemon # daemon messagesdmesg -f syslog # syslog facilitySearch patterns:
xxxxxxxxxxdmesg -T | grep -E '(error|fail|warn|OOM|panic|BUG|segfault|oom-killer)'dmesg -T | grep -i 'ata\|disk\|nvme\|scsi' # storage errorsdmesg -T | grep 'eth\|ens\|bond\|br-' # network driver messagesdmesg -T | grep 'usb' # USB events (device attach/detach)dmesg -T | grep 'ACPI\|thermal\|fan' # thermal eventsCommon kernel messages and what they mean:
xxxxxxxxxx# CPU/Memory errorsdmesg | grep 'mce\|MCE\|Machine Check' # hardware errorsdmesg | grep 'EDAC\|ECC' # memory error detection/correctiondmesg | grep 'Out of memory' # OOM killer activationsdmesg | grep 'Killed process' # processes killed by OOM
# Filesystemdmesg | grep 'EXT4-fs error\|XFS.*error\|BTRFS.*error'dmesg | grep 'I/O error' # disk read/write failures
# Networkdmesg | grep 'NOHZ\|RCU\|hung_task' # kernel stall warningsdmesg | grep 'link is not ready\|Link is Up\|Link is Down' # NIC link stateRing buffer size and clearing:
xxxxxxxxxxdmesg -s 65536 # set buffer size to read (default: 16392)dmesg -c # print and clear ring bufferdmesg --clear # just clear (no print)
# Ring buffer size is set at boot or viasysctl kernel.printk_devkmsgPersist kernel messages (ring buffer is lost on reboot):
xxxxxxxxxx# systemd-journald captures kernel messages automaticallyjournalctl -k # all kernel messagesjournalctl -k -b-1 # kernel messages from previous bootjournalctl -k --since "1 hour ago" -p errlogrotate works and how you would configure it for a custom application log.Answer:
What logrotate does:
logrotate is run by cron (typically daily via /etc/cron.daily/logrotate). It reads config from /etc/logrotate.conf and /etc/logrotate.d/. For each configured log file, it:
Renames the current log (e.g., app.log → app.log.1)
Optionally compresses old rotations (app.log.1.gz)
Signals/restarts the application to reopen its log file
Removes logs older than the retention policy
Configuration for a custom app:
xxxxxxxxxx# /etc/logrotate.d/myapp/var/log/myapp/*.log { daily # rotate daily rotate 14 # keep 14 rotated copies compress # gzip old logs delaycompress # don't compress most recent (app may still write to .1) missingok # don't error if log file doesn't exist notifempty # don't rotate if empty create 0640 myapp adm # create new log file with these permissions/owner sharedscripts # run postrotate script once even if multiple files match postrotate # Signal app to reopen its log file /bin/kill -HUP $(cat /var/run/myapp.pid 2>/dev/null) 2>/dev/null || true # Or for systemd services that don't handle SIGHUP: # systemctl reload myapp || true endscript}Common directives:
xxxxxxxxxxweekly # rotate weekly instead of dailysize 100M # rotate when file reaches 100MB (overrides daily)maxsize 500M # rotate even if time hasn't come but size exceededminsize 10M # don't rotate unless file is at least 10MBdateext # add date to rotated filename (app.log-20240115 instead of app.log.1)dateformat -%Y%m%d # custom date formatnocreate # don't create new empty log file (app creates it)copytruncate # copy log, then truncate original (for apps that can't reopen) # WARNING: small window where writes are lostsu root adm # run as specific user/groupTest without executing:
xxxxxxxxxxlogrotate -d /etc/logrotate.d/myapp # dry run, shows what would happenlogrotate -v /etc/logrotate.d/myapp # verbose executionlogrotate -f /etc/logrotate.d/myapp # force rotation even if not dueCheck rotation status:
xxxxxxxxxxcat /var/lib/logrotate/status # last rotation time per file# orcat /var/lib/logrotate/logrotate.statusTroubleshooting logrotate failures:
xxxxxxxxxx# Run manually with debug outputlogrotate -dv /etc/logrotate.conf
# Common issues:# - Log file owned by different user than configured# - Postrotate script fails (check exit codes)# - App doesn't handle SIGHUP (use copytruncate as fallback)# - Log file path has glob that matches nothingAnswer:
Identifying signal vs noise:
1. Categorize by frequency pattern:
xxxxxxxxxx# High-frequency errors that are always present = likely noise or known issuejournalctl -p err --since "7 days ago" -o json | \ jq -r '.MESSAGE' | sort | uniq -c | sort -rn | head -20
# New errors (appear in last hour but not in last week baseline)# Build a baseline:journalctl -p err --since "7 days ago" --until "1 day ago" -o json | \ jq -r '._SYSTEMD_UNIT + " " + .MESSAGE' | sort -u > /tmp/baseline_errors.txt# Compare to recent:journalctl -p err --since "1 hour ago" -o json | \ jq -r '._SYSTEMD_UNIT + " " + .MESSAGE' | sort -u > /tmp/recent_errors.txtcomm -23 /tmp/recent_errors.txt /tmp/baseline_errors.txt# Lines only in recent = new errors2. Look for correlated spikes:
xxxxxxxxxx# Error rate per minutejournalctl -p err --since "1 hour ago" -o json | \ jq -r '.__REALTIME_TIMESTAMP' | \ awk '{print int($1/1000000/60)}' | sort | uniq -c# Spike at a specific minute = event-driven, not constant noise3. Cross-correlate with metrics:
xxxxxxxxxx# Did error rate change when a deployment happened?journalctl -u myapp --since "2 hours ago" | grep -E 'started|stopped|error' | \ awk '{print $1, $2, $3, $5}' | head -30Building a simple alert with shell:
xxxxxxxxxx# /usr/local/bin/check_errors.shTHRESHOLD=10WINDOW="5 min ago"SERVICE="myapp.service"
ERROR_COUNT=$(journalctl -u "$SERVICE" -p err --since "$WINDOW" --no-pager | wc -l)
if [ "$ERROR_COUNT" -gt "$THRESHOLD" ]; then echo "ALERT: $SERVICE has $ERROR_COUNT errors in last 5 minutes" | \ mail -s "Error spike on $(hostname)" oncall@example.com # Or send to Slack: curl -s -X POST "$SLACK_WEBHOOK_URL" \ -H 'Content-type: application/json' \ -d "{\"text\": \"ERROR: $SERVICE spike: $ERROR_COUNT errors in 5min on $(hostname)\"}"fiRun via cron:
xxxxxxxxxx*/5 * * * * /usr/local/bin/check_errors.shSystemd path unit for immediate trigger:
xxxxxxxxxx# /etc/systemd/system/myapp-error-watch.path[Path]PathChanged=/var/log/myapp/error.log[Install]WantedBy=multi-user.target
# /etc/systemd/system/myapp-error-watch.service[Service]ExecStart=/usr/local/bin/check_errors.shProduction approach: Use structured logging + metrics pipeline (Fluentd/Vector → Prometheus counter on error log lines → Alertmanager rule on rate).
/var/log and what the key log files are for.Answer:
xxxxxxxxxx/var/log/├── syslog (or messages on RHEL) — general system messages from rsyslog├── auth.log (or secure on RHEL) — authentication: SSH logins, sudo, PAM├── kern.log — kernel messages (subset of syslog)├── daemon.log — background daemon messages├── user.log — user-level messages├── dpkg.log (Debian) / dnf.log — package manager activity├── apt/│ ├── history.log — apt install/remove/upgrade history│ └── term.log — terminal output of apt operations├── nginx/│ ├── access.log — HTTP requests (IP, URL, status, bytes)│ └── error.log — nginx errors (configuration, upstream)├── mysql/ or postgresql/ — database server logs├── journal/ — systemd-journald binary logs (if persistent)├── wtmp — binary: login history (read with last)├── btmp — binary: failed login attempts (read with lastb)├── utmp — binary: currently logged in users (read with who/w)├── lastlog — binary: last login per user (read with lastlog)├── faillog — binary: failed login counts (read with faillog)├── dmesg (if saved) — kernel ring buffer snapshot from boot├── boot.log — boot sequence messages├── cron (or cron.log) — cron job execution log├── mail.log (or maillog) — mail transfer agent logs├── audit/audit.log — auditd security audit log└── tallylog — binary: PAM pam_tally2 data
Reading binary logs:
xxxxxxxxxxlast -F # /var/log/wtmp: login history with full timestampslastb # /var/log/btmp: failed logins (requires root)who /var/log/wtmp # alternative wtmp readerlastlog # /var/log/lastlog: per-user last loginfaillog -a # /var/log/faillog: failed login counts per userRHEL/CentOS differences:
/var/log/messages instead of syslog (general)
/var/log/secure instead of auth.log
/var/log/dnf.log instead of dpkg.log
Key for incident response:
xxxxxxxxxx# Timeline reconstructiongrep "Jan 15 10:" /var/log/auth.log /var/log/syslog /var/log/nginx/access.log# Shows all events in a 1-minute window across servicesauditd and how do you use it to track file access or privilege escalation?Answer:
auditd is the Linux kernel audit subsystem daemon. Unlike syslog (which captures what applications choose to log), auditd hooks into the kernel and can log any system call, file access, or security event regardless of whether the application wants it logged.
Check if running:
xxxxxxxxxxsystemctl status auditdauditctl -s # show audit status (enabled, pid, rules count)Key files:
xxxxxxxxxx/etc/audit/auditd.conf # daemon configuration/etc/audit/rules.d/*.rules # persistent audit rules/var/log/audit/audit.log # audit log (binary-ish, read with ausearch/aureport)Add rules with auditctl:
xxxxxxxxxx# Watch a file for read and writeauditctl -w /etc/passwd -p rwa -k passwd_changes# -w: watch path# -p: permissions to audit: r=read, w=write, x=execute, a=attribute change# -k: key tag for searching
# Watch a directory recursivelyauditctl -w /etc/sudoers.d/ -p wa -k sudo_changes
# Audit all executions of a specific programauditctl -w /usr/bin/ssh -p x -k ssh_executions
# Audit syscall-level (privilege escalation via setuid)auditctl -a always,exit -F arch=b64 -S execve -F euid=0 -F auid!=0 -k privesc# Logs any execve() where effective UID is root but audit UID is not rootMake rules persistent:
xxxxxxxxxx# /etc/audit/rules.d/hardening.rules-w /etc/passwd -p wa -k identity-w /etc/group -p wa -k identity-w /etc/shadow -p wa -k identity-w /etc/sudoers -p wa -k sudo_changes-a always,exit -F arch=b64 -S execve -F euid=0 -F auid!=unset -k privesc
# Reloadaugenrules --load# orservice auditd reloadSearch audit logs:
xxxxxxxxxxausearch -k passwd_changes # search by keyausearch -k passwd_changes -i # interpret (show usernames, not UIDs)ausearch -ua 1000 --start today # all events for user UID 1000 todayausearch -sc execve --start "01/15/2024 10:00:00" # by syscall
# Generate summary reportsaureport --summary # overall summaryaureport --auth # authentication attemptsaureport --login # login eventsaureport -x --summary # executable summaryaureport -f --summary # file access summaryRead a specific audit event:
xxxxxxxxxxausearch -a 12345 -i # look up by event serial number# Example output:# type=SYSCALL msg=audit(1705123456.789:12345): arch=c000003e syscall=2 success=yes# exit=3 a0=7f.. a1=0 a2=1b6 a3=0 items=1 ppid=1234 pid=5678 auid=1000 uid=0# gid=0 euid=0 ses=15 comm="cat" exe="/usr/bin/cat" key="passwd_changes"# type=PATH msg=audit(...): name="/etc/passwd" inode=... dev=08:01 mode=0100644Answer:
Structured logging means emitting logs as machine-parseable key-value pairs (typically JSON) rather than free-form text strings.
Unstructured (traditional):
xxxxxxxxxx[2024-01-15 10:23:45] ERROR: Failed to connect to database at 10.0.0.5:5432 after 3 retries, latency=2341ms
Structured (JSON):
xxxxxxxxxx{"timestamp":"2024-01-15T10:23:45Z","level":"ERROR","event":"db_connect_failed", "host":"10.0.0.5","port":5432,"retries":3,"latency_ms":2341, "service":"order-api","trace_id":"abc123","user_id":9876}Why it helps at scale:
Exact field filtering without regex:
xxxxxxxxxx# Find all DB failures with latency > 1000msjournalctl -o json | jq 'select(.event=="db_connect_failed" and .latency_ms > 1000)'# vs. grep-and-parse on unstructured log — fragile, slowAggregation and metrics extraction:
xxxxxxxxxx# P99 latency from logsgrep '"event":"db_connect_failed"' app.log | \ jq -r '.latency_ms' | sort -n | awk 'END{print int(NR*0.99) " " NR}' | \ xargs -I{} awk 'NR=={}'Trace ID correlation across services:
xxxxxxxxxx# Find all log entries for a specific requestjq 'select(.trace_id == "abc123")' /var/log/app/*.logLog pipeline efficiency: Fluentd/Logstash/Vector can route, filter, and aggregate without regex parsing.
Implementation examples:
xxxxxxxxxx# Python with structlogimport structloglog = structlog.get_logger()log.error("db_connect_failed", host="10.0.0.5", port=5432, retries=3, latency_ms=2341)xxxxxxxxxx// Go with zaplogger.Error("db_connect_failed", zap.String("host", "10.0.0.5"), zap.Int("port", 5432), zap.Int("retries", 3), zap.Duration("latency", 2341*time.Millisecond),)journald native structured fields:
xxxxxxxxxxsystemd-cat -t myapp -p err <<< '{"MESSAGE":"db failed","host":"10.0.0.5"}'# or use sd_journal_send() in C for proper structured journal entriesAt Google scale: Structured logs feed into systems like Cloud Logging / Stackdriver where fields become queryable dimensions, enabling dashboards, SLO calculation from log data, and automated incident detection.
Answer:
The core technique: propagate a trace/request ID through all service calls.
Step 1: Generate a trace ID at the entry point:
xxxxxxxxxx# API gateway or first serviceimport uuidtrace_id = str(uuid.uuid4())# Add to log context:log.info("request_received", trace_id=trace_id, user_id=user_id, path=request.path)# Pass downstream via HTTP header:headers["X-Trace-ID"] = trace_idStep 2: Each downstream service logs with the same trace ID:
xxxxxxxxxx# All services emit: {"trace_id": "abc123-...", "service": "order-api", ...}Step 3: Correlate at query time:
xxxxxxxxxx# If logs are centralized (Elasticsearch, Loki, BigQuery)# ES query: GET /logs/_search { "query": { "term": { "trace_id": "abc123" }}}
# If local files:grep '"trace_id":"abc123"' /var/log/service-*.log | sort -t'"' -k... # Better: use jqfor logfile in /var/log/services/*.log; do jq -r 'select(.trace_id == "abc123") | [.timestamp, .service, .level, .message] | @tsv' \ "$logfile" 2>/dev/nulldone | sort
# journald (structured fields):journalctl TRACE_ID=abc123 # if services use sd_journal_send() with TRACE_ID fieldWithout a trace ID (retrofit approach):
xxxxxxxxxx# Correlate by user ID and time windowTIMESTAMP="2024-01-15 10:23"USER_ID="9876"
grep "$TIMESTAMP" /var/log/nginx/access.log | grep "user_id=$USER_ID"grep "$TIMESTAMP" /var/log/order-service.log | grep "user_id=$USER_ID"grep "$TIMESTAMP" /var/log/payment-service.log | grep "user_id=$USER_ID"
# Build a timelinefor svc in nginx order-service payment-service; do grep "$TIMESTAMP" "/var/log/$svc.log" | grep "$USER_ID" | \ awk -v svc="$svc" '{print $1, $2, svc, $0}'done | sortOpenTelemetry (modern standard):
W3C Trace Context header: traceparent: 00-traceId-spanId-flags
Propagated automatically by OTel SDKs
Exported to Jaeger, Zipkin, Cloud Trace for visual waterfall diagrams
At Google: Every RPC carries a Trace ID via Stubby/gRPC metadata. Dapper (predecessor to Zipkin/Jaeger) provides end-to-end tracing. Log queries use the trace ID as a primary key for correlation.
Answer:
Alert when log grows too fast (error spike):
xxxxxxxxxx# /usr/local/bin/log_growth_check.shLOG_FILE="/var/log/myapp/error.log"STATE_FILE="/tmp/log_growth_state"GROWTH_THRESHOLD_KB=1024 # alert if grew more than 1MB in check interval
current_size=$(stat -c %s "$LOG_FILE" 2>/dev/null || echo 0)
if [ -f "$STATE_FILE" ]; then last_size=$(cat "$STATE_FILE") growth=$(( (current_size - last_size) / 1024 )) if [ "$growth" -gt "$GROWTH_THRESHOLD_KB" ]; then echo "ALERT: $LOG_FILE grew ${growth}KB since last check" | \ mail -s "Log spike on $(hostname)" oncall@example.com fifi
echo "$current_size" > "$STATE_FILE"Alert when log stops growing (service stuck/dead):
xxxxxxxxxxLOG_FILE="/var/log/myapp/access.log"STATE_FILE="/tmp/log_activity_state"STALE_SECONDS=300 # alert if no new writes in 5 minutes
last_modified=$(stat -c %Y "$LOG_FILE" 2>/dev/null || echo 0)now=$(date +%s)age=$(( now - last_modified ))
if [ "$age" -gt "$STALE_SECONDS" ]; then echo "ALERT: $LOG_FILE has not been written to in ${age}s — service may be down" | \ mail -s "Service stalled on $(hostname)" oncall@example.comfiUsing inotifywait for real-time monitoring:
xxxxxxxxxx# Alert on rapid writes (more than N in 60 seconds)inotifywait -m -e modify /var/log/myapp/error.log 2>/dev/null | \ awk 'BEGIN{count=0; t=systime()} {count++; if (systime() - t > 60) { if (count > 100) print "SPIKE: " count " writes in 60s" | "mail -s alert oncall@" count=0; t=systime() }}'systemd service health via log silence:
xxxxxxxxxx# /etc/systemd/system/log-watchdog.service[Unit]Description=Log activity watchdog
[Service]Type=oneshotExecStart=/usr/local/bin/log_growth_check.sh
# /etc/systemd/system/log-watchdog.timer[Timer]OnBootSec=5minOnUnitActiveSec=5min
[Install]WantedBy=timers.targetProduction tools:
mtail (Google): parses log files, emits Prometheus metrics — alert on metric thresholds
Promtail/Vector: stream logs to Loki/InfluxDB, alert with Alertmanager
Filebeat: ships logs to ELK stack with metadata
Answer:
Using journald (persistent journal):
xxxxxxxxxx# Enable persistent journal firstmkdir -p /var/log/journalsystemd-tmpfiles --create --prefix /var/log/journalsystemctl restart systemd-journald
# Now journal persists across reboots
# List available bootsjournalctl --list-boots# -2 abc123 Mon 2024-01-13 ... Mon 2024-01-13 ...# -1 def456 Mon 2024-01-14 ... Mon 2024-01-14 ...# 0 ghi789 Mon 2024-01-15 ... now
# View logs from previous bootjournalctl -b -1 -u myapp.service
# View logs from all boots for a servicejournalctl -u myapp.service --no-pager
# Filter by process name across all bootsjournalctl _COMM=myapp --since "7 days ago"
# Filter by executable pathjournalctl _EXE=/usr/bin/myappUsing traditional syslog (no reboot continuity by default):
xxxxxxxxxx# Rotated logs are kept in /var/log/zgrep "myapp" /var/log/syslog* # includes .1, .2.gz, etc.zgrep "myapp" /var/log/syslog.1zcat /var/log/syslog.2.gz | grep myapp
# Search all rotated logs with a single commandfor f in /var/log/syslog /var/log/syslog.[0-9]* /var/log/syslog.*.gz; do if [[ "$f" == *.gz ]]; then zgrep "myapp" "$f" | sed "s|^|$f:|" else grep "myapp" "$f" | sed "s|^|$f:|" fidone | sort -t: -k2Find process crash across reboots:
xxxxxxxxxx# journald fields for crash detectionjournalctl _SYSTEMD_UNIT=myapp.service --no-pager | grep -E '(segfault|oom-killer|Killed|core dumped)'
# coredumpctl shows crashes across rebootscoredumpctl list myappcoredumpctl info myapp # shows all crashes with timestamps, signals, and traceAnswer:
iotop — per-process I/O monitoring:
xxxxxxxxxxiotop -o # -o: only show processes with I/O activityiotop -a # accumulate totals instead of current rateiotop -p PID # specific processiotop -u user # specific user# Columns: TID, PRIO, USER, DISK READ, DISK WRITE, SWAPIN, IO>, COMMANDiostat — per-device I/O statistics:
xxxxxxxxxxiostat -x 1 # extended stats, refresh every 1 second# Key columns:# %util: % of time device was busy (near 100% = bottleneck)# await: average wait time (ms) for I/O requests (service + queue time)# r_await/w_await: read/write wait separately# rrqm/s, wrqm/s: requests merged by elevator (high = sequential I/O)pidstat — per-process I/O:
xxxxxxxxxxpidstat -d 1 # disk I/O per process, refresh every 1 secondpidstat -d -p PID 1For one-shot snapshot without iotop:
xxxxxxxxxx# /proc/PID/io — per-process I/O accountingcat /proc/1234/io# rchar: bytes read (including cache)# wchar: bytes written (including cache)# read_bytes: bytes read from storage (bypassed cache)# write_bytes: bytes written to storage
# Compare all processes:for pid in /proc/[0-9]*/io; do p=$(dirname $pid | xargs basename) awk -v p=$p '/write_bytes/{print $2, p}' $pid 2>/dev/nulldone | sort -rn | head -10blktrace for deep disk analysis:
xxxxxxxxxxblktrace -d /dev/sda -o - | blkparse -i - | head -50# Shows per-request: process, timestamp, request type (R/W), sector, sizeFind which files a process is writing:
xxxxxxxxxxlsof -p PID | grep -E 'REG|CHR' # open regular/character filesinotifywait -m -r -e modify /var/log & # watch directory for writescgroup-level I/O stats:
xxxxxxxxxxcat /sys/fs/cgroup/blkio/myservice/blkio.throttle.io_service_bytes# Shows bytes read/written per block device per cgroupAnswer:
Log sampling means recording only a fraction of log events of a given type rather than every occurrence. Instead of logging every single HTTP request, log 1 in every 100.
When appropriate:
High-frequency, low-information events at steady state:
Every successful health check (200 OK) at 1000 RPS → 86 million log entries/day for zero additional insight
Routine read queries against a database
Known-good paths with predictable behavior:
CDN cache hits (all identical pattern)
Internal heartbeats between services
When NOT to sample:
Error events (always log 100% of errors)
Authentication/authorization events (security requirement)
Financial transactions
Events that trigger alerts
First occurrence of a new error type
Implementation strategies:
Head-based sampling (random):
xxxxxxxxxximport randomif random.random() < 0.01: # log 1% of requests logger.info("request", path=path, status=200, latency_ms=latency)Tail-based sampling (sample based on outcome):
xxxxxxxxxx# Buffer events for a request, then decide to keep or discard# Keep all if: any error occurred, latency > thresholdif any_error or latency_ms > 500: flush_buffered_events() # keep 100%elif random.random() < 0.001: flush_buffered_events() # keep 0.1% of clean requestselse: discard_buffered_events()Rate limiting (cap at N per second):
xxxxxxxxxxfrom collections import defaultdictimport time
_log_counts = defaultdict(int)_log_window = defaultdict(float)
def rate_limited_log(key, message, limit=10): now = time.time() if now - _log_window[key] > 1.0: # new window _log_counts[key] = 0 _log_window[key] = now if _log_counts[key] < limit: logger.info(message) _log_counts[key] += 1 # else: silently dropBest practice: Always log the sample rate as a field so downstream analysis can extrapolate true counts: {"sampled": true, "sample_rate": 0.01}.
watch and other tools to build a simple real-time dashboard for system health?Answer:
watch — refresh a command every N seconds:
xxxxxxxxxxwatch -n 2 'ps aux --sort=-%cpu | head -15' # top CPU processeswatch -n 1 'ss -s' # socket summarywatch -n 5 'df -h | grep -v tmpfs' # disk usagewatch -n 2 'cat /proc/loadavg; free -h' # load + memorywatch -d -n 2 'netstat -s | grep -E "(failed|error|retrans)"' # -d highlights changesComposing a multi-panel dashboard with tmux:
xxxxxxxxxx# /usr/local/bin/dashboard.shtmux new-session -d -s dashboardtmux split-window -htmux split-window -vtmux select-pane -t 0tmux split-window -v
# Pane 0: CPU/Loadtmux send-keys -t dashboard:0.0 "watch -n 1 'uptime; top -bn1 | head -15'" Enter# Pane 1: Memorytmux send-keys -t dashboard:0.1 "watch -n 2 'free -h; cat /proc/meminfo | grep -E \"(Mem|Swap|Dirty|Writeback)\"'" Enter# Pane 2: Disk I/Otmux send-keys -t dashboard:0.2 "iostat -x 2" Enter# Pane 3: Networktmux send-keys -t dashboard:0.3 "watch -n 1 'ss -s; ip -s link show eth0'" Enter
tmux attach -t dashboardQuick one-liners for common checks:
xxxxxxxxxx# All-in-one system snapshotecho "=== Load ===" && uptime && \echo "=== Memory ===" && free -h && \echo "=== Top CPU ===" && ps aux --sort=-%cpu | head -5 && \echo "=== Disk ===" && df -h && \echo "=== Network ===" && ss -s && \echo "=== Recent Errors ===" && journalctl -p err -n 5 --no-pagervmstat for trend watching:
xxxxxxxxxxvmstat 2 30 # every 2 seconds, 30 samples# r: run queue (>2×CPU count = CPU saturation)# b: blocked on I/O# si/so: swap in/out (any = memory pressure)# wa: I/O wait % (>20% sustained = I/O bottleneck)dstat (combines vmstat, iostat, netstat):
xxxxxxxxxxdstat -cdngy 2 # CPU, disk, net, page, system every 2 secondsdstat --top-cpu --top-io --top-mem # show top process per resourceAnswer:
Absence of errors in logs while users see slowness means the problem is likely latency rather than failures. The service is responding, just slowly.
Step 1: Check if logs have latency data:
xxxxxxxxxx# Web server access logs usually include response time# nginx: $request_time (total) and $upstream_response_timetail -1000 /var/log/nginx/access.log | awk '{print $NF}' | \ sort -n | awk 'NR==int(NR*0.95)' | tail -1 # P95 latency
# Or compute percentilesawk '{print $NF}' /var/log/nginx/access.log | sort -n | \ awk 'BEGIN{n=0}{a[n++]=$1}END{ print "P50:", a[int(n*0.5)]; print "P95:", a[int(n*0.95)]; print "P99:", a[int(n*0.99)] }'Step 2: System resource saturation (the USE method):
xxxxxxxxxx# CPU: high load even without errorsuptime # load average vs CPU countmpstat -P ALL 1 5 # per-CPU utilizationsar -u 1 10 # historical CPU
# Memory: swapping causes latencyvmstat 1 5 | awk '{print $7, $8}' # si/so columns (swap in/out)free -h # swap in use?
# Disk I/O: high awaitiostat -x 1 5 | grep -v '^$'# await > 10ms for SSD, > 20ms for HDD = saturation
# Network: packet drops or retransmitsnetstat -s | grep -E '(retransmit|failed|error)'ip -s link show eth0 | grep -E '(errors|dropped)'Step 3: Application-level latency breakdown:
xxxxxxxxxx# Check upstream dependencies# Is the slow part: DB query? External API? Internal computation?
# For web apps: enable access log with upstream timing# nginx: log_format with $upstream_response_time# If upstream_response_time ≈ request_time: backend is slow# If upstream_response_time << request_time: nginx/network overhead
# Database slow query log# MySQL: slow_query_log = ON, long_query_time = 0.5tail -f /var/log/mysql/slow.log
# PostgreSQLtail -f /var/log/postgresql/postgresql-*.log | grep "duration:"Step 4: GC pauses (JVM, Go):
xxxxxxxxxx# Java GC pauses show as latency spikesgrep "GC pause" /var/log/app/gc.log | tail -20# "GC pause (G1 Humongous Allocation) 1.234ms" — minor# "Full GC 12.345s" — major, explains 12-second latency spikes
# Go runtime GC (via GODEBUG)GODEBUG=gctrace=1 ./myapp 2>&1 | grep '^gc'Step 5: Connection pool exhaustion (no errors, just queueing):
xxxxxxxxxx# Check DB connection pool stats — app may be waiting for connectionsss -tn dst DB_HOST | wc -l # number of connections to DB# If at max pool size, requests queue silently
# Check thread pool / goroutine countcat /proc/PID/status | grep ThreadsAnswer:
The GRUB rescue shell (grub rescue>) appears when GRUB cannot find its modules or the configuration file — typically because of a corrupted filesystem, changed disk layout, or accidental deletion of /boot/grub.
Step 1: Identify available disks and partitions:
xxxxxxxxxxgrub rescue> ls# (hd0) (hd0,msdos1) (hd0,msdos2) (hd1) ...# or with GPT: (hd0,gpt1) (hd0,gpt2)
grub rescue> ls (hd0,gpt2)/# If this shows boot/ grub/ etc/ — this is the root partitionStep 2: Find where GRUB modules are:
xxxxxxxxxxgrub rescue> ls (hd0,gpt1)/grub/# Should show: i386-pc/ or x86_64-efi/ directory with *.mod filesStep 3: Set root and prefix, load normal module:
xxxxxxxxxxgrub rescue> set root=(hd0,gpt2)grub rescue> set prefix=(hd0,gpt1)/boot/grubgrub rescue> insmod normalgrub rescue> normal# This loads the full GRUB menuStep 4: If normal module loads the menu, boot manually from GRUB prompt:
xxxxxxxxxxgrub> linux (hd0,gpt2)/boot/vmlinuz-5.15.0 root=/dev/sda2 rogrub> initrd (hd0,gpt2)/boot/initrd.img-5.15.0grub> bootStep 5: Once booted, reinstall GRUB properly:
xxxxxxxxxx# From the running system:grub-install /dev/sda # MBR-basedgrub-install --target=x86_64-efi --efi-directory=/boot/efi # UEFIupdate-grub # regenerate grub.cfgIf unable to boot at all — use a Live USB:
xxxxxxxxxx# Boot from Live USB, then:mount /dev/sda2 /mntmount /dev/sda1 /mnt/bootmount --bind /dev /mnt/devmount --bind /proc /mnt/procmount --bind /sys /mnt/syschroot /mntgrub-install /dev/sdaupdate-grubexitumount -R /mntrebootCommon causes:
Disk UUID changed (LVM resize, disk replacement) — update /etc/fstab and regenerate grub.cfg
/boot partition accidentally formatted or deleted
Dual-boot Windows update overwrote MBR
Answer:
What is initramfs? The initramfs (initial RAM filesystem) is a compressed cpio archive loaded by the bootloader into RAM before the real root filesystem is mounted. It contains:
A minimal Linux environment (busybox or systemd-based)
Kernel modules needed to mount the real root filesystem (disk drivers, filesystem drivers, LVM, LUKS crypto, etc.)
Scripts/udev rules to discover and assemble the root device
The init script that mounts root and execs the real init
Boot sequence:
xxxxxxxxxxBIOS/UEFI → Bootloader (GRUB) → Kernel + initramfs loaded into RAM→ Kernel extracts initramfs → Runs /init in initramfs→ /init discovers + mounts real root (may need to unlock LUKS, assemble RAID/LVM)→ Switches root to real filesystem → Exec /sbin/init (systemd)
What happens when corrupted/missing:
Missing: GRUB loads kernel but fails at "Unpacking initramfs" — kernel panics: VFS: Cannot open root device or drops to kernel panic with not syncing: No working init found
Corrupted: Kernel extracts garbage, /init crashes, kernel panics with Attempted to kill init!
Incomplete (missing modules): initramfs loads but can't find the root device — drops to initramfs emergency shell (busybox shell prompt)
Recovery from initramfs emergency shell:
xxxxxxxxxx# You see: (initramfs) prompt# Try to manually find and mount root(initramfs) ls /dev/sd*(initramfs) mount /dev/sda2 /root(initramfs) exit # or: exec switch_root /root /sbin/init
# If LVM:(initramfs) vgscan(initramfs) vgchange -ay(initramfs) mount /dev/mapper/vg0-root /rootRebuild initramfs from a working system or chroot:
xxxxxxxxxx# Debian/Ubuntu:update-initramfs -u -k all # update for all installed kernelsupdate-initramfs -u -k $(uname -r) # current kernel onlyupdate-initramfs -c -k $(uname -r) # create fresh
# RHEL/CentOS/Fedora (dracut):dracut --force /boot/initramfs-$(uname -r).img $(uname -r)dracut --regenerate-all --force # all kernels
# Verify the initramfs contents:lsinitrd /boot/initrd.img-$(uname -r) | head -30lsinitrd /boot/initrd.img-$(uname -r) | grep -E '(sd_mod|ext4|dm-crypt)'Answer:
Common causes of kernel panics:
Kernel bug / NULL pointer dereference in kernel code — the kernel equivalent of a segfault
Unrecoverable hardware error — bad RAM (MCE), corrupted kernel text
Corrupted kernel stack — stack overflow in kernel context
Missing root filesystem — VFS: Unable to mount root fs
Explicit panic() call — a driver or subsystem called panic("fatal error") on an unrecoverable condition
Soft lockup / hard lockup — watchdog timer fires because a CPU was stuck in kernel code for >10s (soft) or >20s (hard) without yielding
Panic message anatomy (from serial console/kdump):
xxxxxxxxxxBUG: kernel NULL pointer dereference at 0000000000000028IP: [<ffffffff81234567>] some_function+0x123/0x456 [some_module]PGD 0Oops: 0002 [#1] SMP...Call Trace:[<ffffffff81234890>] caller_function+0x45/0x89[<ffffffff81235000>] parent_function+0x12/0x34...RIP: 0010:[<ffffffff81234567>] some_function+0x123/0x456RSP: ...
Capturing panics with kdump:
xxxxxxxxxx# Installapt-get install kdump-tools crash # Debianyum install kexec-tools crash # RHEL
# Configure crashkernel in GRUB (reserve RAM for crash kernel)# /etc/default/grub: GRUB_CMDLINE_LINUX="... crashkernel=256M"update-grub && reboot
# Verify kdump is readysystemctl status kdumpcat /sys/kernel/kexec_crash_loaded # should be 1
# After a crash, vmcore is at:ls /var/crash/*/vmcoreAnalyze vmcore with crash:
xxxxxxxxxxcrash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux /var/crash/*/vmcore
# Inside crash:crash> bt # backtrace at time of paniccrash> bt -a # all CPUs' backtracescrash> log # kernel message buffer at time of crashcrash> ps # running processes at crash timecrash> files PID # open files of a processcrash> vm PID # virtual memory areascrash> rd address # read memory at addressDecode a panic from dmesg without kdump:
xxxxxxxxxx# If system recovered (restarted), check previous boot:journalctl -b -1 -k | grep -A 30 'Oops\|panic\|BUG:'
# Decode address to function+offsetaddr2line -e /usr/lib/debug/vmlinux 0xffffffff81234567# orgdb /usr/lib/debug/vmlinux -batch -ex "list *0xffffffff81234567"Answer:
Entering emergency/single-user mode:
At GRUB: Edit the boot entry (press e), append to the kernel line:
xxxxxxxxxxsystemd.unit=emergency.target # emergency mode (root-only, minimal, no mounts)systemd.unit=rescue.target # rescue mode (more services, single user)# or legacy:single # single-user modeinit=/bin/bash # bypass init entirely (remount read-write needed)
Remounting root read-write (often needed in emergency mode):
xxxxxxxxxxmount -o remount,rw /Running fsck:
xxxxxxxxxx# NEVER run fsck on a mounted filesystem (data corruption risk)# In emergency mode, root is often mounted read-only — safe to check
# Check which filesystem needs repaircat /etc/fstab # see device for /blkid # identify device by UUID
# Run fsckfsck -y /dev/sda1 # -y: automatically answer yes to all repairsfsck.ext4 -v /dev/sda1 # verbose ext4 checkxfs_repair /dev/sda2 # XFS repair (xfs has no fsck -equivalent)btrfs check /dev/sda3 # BTRFS check (use --repair carefully)
# Force check on next boot (ext4)tune2fs -C 1000 /dev/sda1 # fake high mount count to force checktouch /forcefsck # on some older systems# Modern: systemd-fsck runs automatically if dirty bit is set
# Check result (ext4)tune2fs -l /dev/sda1 | grep -E '(state|mount count|last checked)'# Filesystem state: clean = OK, with errors = needs fsckXFS-specific recovery:
xxxxxxxxxxxfs_check /dev/sda2 # check (read-only, deprecated on newer systems)xfs_repair -n /dev/sda2 # dry-run check, no changesxfs_repair /dev/sda2 # repair
# XFS journal replay (if dirty):xfs_repair -L /dev/sda2 # zero the log (last resort, may lose data)After repair, verify:
xxxxxxxxxxfsck.ext4 -n /dev/sda1 # re-check without changes; should report "clean"sysctl?Answer:
Kernel parameters (exposed via /proc/sys/) control kernel behavior at runtime: networking, memory management, security, filesystems, and more.
View and set:
xxxxxxxxxx# View a single parametersysctl net.ipv4.ip_forwardcat /proc/sys/net/ipv4/ip_forward
# Set temporarily (lost on reboot)sysctl -w net.ipv4.ip_forward=1echo 1 > /proc/sys/net/ipv4/ip_forward
# Set permanentlyecho 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf# or create a drop-in:echo 'net.ipv4.ip_forward = 1' > /etc/sysctl.d/99-routing.confsysctl --system # reload all sysctl.d filessysctl -p /etc/sysctl.d/99-routing.conf # reload specific file
# View all parameterssysctl -asysctl -a | grep net.coreImportant categories and parameters:
Network performance:
xxxxxxxxxxnet.core.rmem_max = 134217728 # max socket receive buffer (128MB)net.core.wmem_max = 134217728 # max socket send buffernet.ipv4.tcp_rmem = 4096 87380 134217728 # min default max for TCP recvnet.ipv4.tcp_wmem = 4096 65536 134217728 # min default max for TCP sendnet.core.netdev_max_backlog = 250000 # NIC receive queue depthnet.ipv4.tcp_congestion_control = bbr # congestion control algorithmnet.core.somaxconn = 65535 # listen backlog maxnet.ipv4.tcp_max_syn_backlog = 65535 # SYN queue sizeMemory:
xxxxxxxxxxvm.swappiness = 10 # prefer to keep things in RAM (0-100, lower = less swap)vm.dirty_ratio = 10 # max % of memory that can be dirty before syncvm.dirty_background_ratio = 5 # background flush starts at this %vm.overcommit_memory = 1 # allow memory overcommit (0=heuristic, 1=always, 2=never)vm.drop_caches = 3 # drop page/inode/dentry cache (use cautiously)Security:
xxxxxxxxxxkernel.randomize_va_space = 2 # full ASLRkernel.dmesg_restrict = 1 # non-root can't read dmesgnet.ipv4.conf.all.rp_filter = 1 # reverse path filtering (anti-spoofing)net.ipv4.conf.all.accept_redirects = 0 # ignore ICMP redirectskernel.core_pattern = /var/crash/core.%e.%p.%tFilesystem:
xxxxxxxxxxfs.file-max = 2097152 # system-wide max open filesfs.inotify.max_user_watches = 524288 # for IDEs, file watcherskernel.pid_max = 4194304 # max PIDskexec and when would you use it?Answer:
kexec is a Linux kernel feature that allows loading and booting a new kernel directly from the currently running kernel, bypassing the BIOS/UEFI boot sequence entirely.
Key properties:
BIOS POST and hardware initialization are skipped
Boot time reduction: typically 30-60 seconds faster than full reboot
Memory contents are preserved during the transition (used by kdump)
The running kernel loads the new kernel into memory, then hands off control
Use cases:
Fast kernel updates in production:
xxxxxxxxxx# Load new kernel into memory (doesn't reboot yet)kexec -l /boot/vmlinuz-5.15.0 \ --initrd=/boot/initrd.img-5.15.0 \ --command-line="$(cat /proc/cmdline)"
# Execute the loaded kernel (reboots instantly into new kernel)kexec -eCrash kernel for kdump (most common production use):
xxxxxxxxxx# The crashkernel= boot parameter reserves RAM# When kernel panics, kexec automatically boots the crash kernel# Crash kernel runs kdump.service, which saves /proc/vmcore to disk
# Check if crash kernel is loadedcat /sys/kernel/kexec_crash_loaded # 1 = readycat /proc/iomem | grep Crash # shows reserved memory regionLive patching alternative for kernel version upgrades:
kpatch / livepatch for in-place kernel fixes
kexec for full version changes without hardware re-init
Setup:
xxxxxxxxxx# Installapt-get install kexec-tools# oryum install kexec-tools
# Systemd integration (automatic kexec on 'systemctl reboot')# /etc/default/kexec: LOAD_KEXEC=true
systemctl enable kexec-load.service
# Manual reboot with kexec:systemctl kexec # systemd 209+: does kexec rebootLimitations:
Hardware is not re-initialized — if hardware is in a bad state, kexec won't fix it
UEFI Secure Boot may require signed kernels
Some drivers don't cleanly survive kexec transition (reinit issues)
Not suitable when BIOS/hardware update is the reason for reboot
kdump work, and how do you configure it to capture kernel crash dumps?Answer:
Architecture:
kdump uses kexec to boot a second "capture kernel" when the primary kernel panics. The capture kernel runs a minimal environment (loaded into pre-reserved RAM), mounts storage, and saves the memory image of the crashed kernel to disk as /proc/vmcore.
Step 1: Reserve memory for the capture kernel:
xxxxxxxxxx# Add to GRUB kernel command line:# /etc/default/grub:GRUB_CMDLINE_LINUX="... crashkernel=256M"# For systems with more RAM: crashkernel=512M# Auto-size: crashkernel=auto (RHEL)
update-grubrebootStep 2: Install and configure kdump:
xxxxxxxxxx# Debian/Ubuntu:apt-get install kdump-tools crash linux-image-$(uname -r)-dbg
# RHEL/CentOS:yum install kexec-tools crash kernel-debuginfo
# Configure: /etc/kdump.conf (RHEL) or /etc/default/kdump-tools (Ubuntu)# Set dump destination:path /var/crash # local directory#core_collector makedumpfile -l --message-level 1 -d 31 # compress, exclude zero pages
# RHEL /etc/kdump.conf:path /var/crashcore_collector makedumpfile -l --message-level 1 -d 31default reboot # action after dump (reboot|halt|shell)Step 3: Enable and verify:
xxxxxxxxxxsystemctl enable kdump && systemctl start kdumpsystemctl status kdump
# Verify crash kernel is loaded:cat /sys/kernel/kexec_crash_loaded # must be 1cat /proc/iomem | grep 'Crash kernel'Step 4: Test (warning: reboots the machine):
xxxxxxxxxx# Force a kernel panic (use on test systems only)echo c > /proc/sysrq-trigger# After reboot, check /var/crash/Step 5: Analyze the crash dump:
xxxxxxxxxxls /var/crash/# 2024-01-15-10:23:00/vmcore
# With crash tool:crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux \ /var/crash/2024-01-15-10:23:00/vmcore
crash> bt # backtrace at paniccrash> log # kernel message buffercrash> sys # system infocrash> mod # loaded modules at time of crashmakedumpfile compression levels:
xxxxxxxxxx# -d 31: exclude zero pages, cache pages, user pages (smallest dump)# -d 1: exclude only zero pages (larger but more complete)# -l: LZO compressionmakedumpfile -l --message-level 1 -d 31 /proc/vmcore /var/crash/vmcore.compressedAnswer:
List loaded modules:
xxxxxxxxxxlsmod# Output: Module Size Used by# e1000e 147456 0 (NIC driver, not in use by anything)# dm_mod 122880 7 dm_log,dm_mirror,... (used by 7 other modules)Load and unload:
xxxxxxxxxxmodprobe module_name # load module and its dependenciesmodprobe -v module_name # verbose: shows what's being loadedmodprobe -r module_name # remove module (if not in use)modprobe -r -v module_name # verbose removal
insmod /path/to/module.ko # load specific .ko file (no dependency resolution)rmmod module_name # remove (no dependency handling)Module information:
xxxxxxxxxxmodinfo module_name # description, author, parameters, license, filenamemodinfo e1000e | grep -E '(filename|version|parm)'
# Module parameters (configurable at load time)modprobe e1000e InterruptThrottleRate=3000,3000 # pass parameters# or set permanently:echo 'options e1000e InterruptThrottleRate=3000,3000' > /etc/modprobe.d/e1000e.confPersistent loading:
xxxxxxxxxx# Load at bootecho 'module_name' >> /etc/modules # Debianecho 'module_name' > /etc/modules-load.d/module_name.conf # systemd
# Blacklist a module (prevent loading)echo 'blacklist nouveau' > /etc/modprobe.d/blacklist-nouveau.confupdate-initramfs -u # if module might be loaded from initramfs
# Alias (substitute one module for another)echo 'alias eth0 e1000e' > /etc/modprobe.d/net.confDebugging module issues:
xxxxxxxxxx# Module fails to load — check kernel logdmesg | tail -20 # immediately after failed modprobe
# Dependency issuesmodprobe --show-depends module_name # show dependency tree without loadingdepmod -a # rebuild module dependency database (after kernel update)
# Module verificationmodinfo --field=vermagic module_nameuname -r# vermagic must match uname -r, or module was built for different kernel
# Check if module is taintedcat /proc/sys/kernel/tainted# 0 = clean, non-zero = tainted (out-of-tree, unsigned, error, etc.)# Bit flags: see Documentation/admin-guide/tainted-kernels.rst
# Dynamic debug for moduleecho 'module e1000e +p' > /sys/kernel/debug/dynamic_debug/control# Enables pr_debug() messages from e1000edmesg -w | grep e1000eAnswer:
Watchdog mechanism: Linux runs a high-priority watchdog thread per CPU. This thread increments a counter and checks that the CPU is making progress.
Soft lockup:
A CPU was in kernel context for >10 seconds without calling the scheduler
The watchdog reports it but the system continues running
Message: BUG: soft lockup - CPU#3 stuck for 23s! [myprocess:12345]
Causes: busy loop in kernel code, holding a spinlock too long, interrupt storm
Hard lockup (NMI watchdog):
A CPU was stuck for >20 seconds (no interrupts processed)
Detected via NMI (Non-Maskable Interrupt) which fires even when interrupts are disabled
Often fatal (the CPU is completely stuck)
Message: Watchdog detected hard LOCKUP on cpu 2
Causes: hardware hang, deadlock in interrupt handler, defective hardware
Immediate response:
xxxxxxxxxx# Check for lockup messagesdmesg -T | grep -E '(lockup|hung_task|RCU stall|NMI)'journalctl -k | grep -E '(lockup|hung_task)'
# Lockup message includes a stack trace — decode it# The trace shows which kernel function the CPU was stuck inTuning watchdog thresholds:
xxxxxxxxxx# Soft lockup threshold (seconds, default 20)sysctl -w kernel.watchdog_thresh=30
# Disable soft lockup detection (on overloaded systems where false positives occur)sysctl -w kernel.soft_watchdog=0
# Enable/disable NMI watchdogsysctl -w kernel.nmi_watchdog=1 # 1=enable, 0=disable
# hung_task detection (D-state processes)sysctl -w kernel.hung_task_timeout_secs=120 # default: 120sCommon causes and fixes:
| Symptom | Likely Cause | Fix |
|---|---|---|
| Soft lockup on all CPUs | System overloaded | Reduce load, check for runaway process |
| Soft lockup + NFS mentions | NFS server unreachable | Fix NFS mount or use soft+timeo options |
| Soft lockup + virtio | VM hypervisor overloaded | Reduce VM density |
| Hard lockup | Hardware fault | Check mcelog, run memtest, replace hardware |
| RCU stall | CPU stuck with RCU read lock | Kernel bug, check for updates |
Panic on lockup (for automated recovery):
xxxxxxxxxxsysctl -w kernel.softlockup_panic=1 # panic on soft lockup → triggers kdump → auto-rebootsysctl -w kernel.hardlockup_panic=1sysrq for emergency access to a hung system?Answer:
Magic SysRq is a kernel feature that responds to key combinations even when the system is mostly unresponsive, because it bypasses the normal input stack.
Enable SysRq:
xxxxxxxxxxsysctl -w kernel.sysrq=1 # enable all (or use bitmask for specific features)# Persist: echo 'kernel.sysrq = 1' > /etc/sysctl.d/99-sysrq.conf
# Bitmask values (sum them):# 1 = enable sysrq at all# 2 = allow reboot (b)# 4 = allow sync (s)# 8 = allow remount read-only (u)# 16 = allow kill signals (e, i)# 32 = allow memory dump (m)# 64 = allow nice all RT tasks (n)# 128 = allow sync + kill + reboot (for OOM situations)# 128 = FULL (all = 1+2+4+8+16+32+64+128 = 255)Trigger via keyboard (physical console):
xxxxxxxxxxAlt+SysRq+<key> # PC keyboardAlt+PrintScreen+<key> # same
Trigger via /proc (remote/programmatic):
xxxxxxxxxxecho b > /proc/sysrq-trigger # reboot immediately (no sync)echo s > /proc/sysrq-trigger # sync all filesystemsecho u > /proc/sysrq-trigger # remount all filesystems read-onlyecho e > /proc/sysrq-trigger # send SIGTERM to all processes except initecho i > /proc/sysrq-trigger # send SIGKILL to all processes except initecho o > /proc/sysrq-trigger # power offecho t > /proc/sysrq-trigger # dump task states to console/dmesgecho m > /proc/sysrq-trigger # dump memory infoecho p > /proc/sysrq-trigger # dump current register state per CPUecho c > /proc/sysrq-trigger # crash kernel (trigger kdump)echo l > /proc/sysrq-trigger # backtrace all CPUsecho 0-9 > /proc/sysrq-trigger # set console log levelThe REISUB sequence for safe emergency reboot:
xxxxxxxxxxR - unRaw (take keyboard from X)E - tErminate (SIGTERM to all processes)I - kIll (SIGKILL to remaining processes)S - Sync (flush disk writes)U - Unmount (remount read-only)B - reBoot
Wait a few seconds between each step. This is the safe alternative to a hard power cycle.
Use case scenarios:
System appears frozen but is still responsive to NMI: use REISUB
OOM death spiral: s u e i s u b sequence
Kernel panic capture: c to trigger kdump capture kernel
Diagnosing hung system: t to dump task states to dmesg (then read remotely via serial or iDRAC/iLO console)
mcelog and how do you respond to machine check exceptions?Answer:
Machine Check Exception (MCE): Modern x86 CPUs have a Machine Check Architecture (MCA) that detects hardware errors: ECC memory errors, cache errors, bus errors, thermal events, and more. When a hardware error occurs, the CPU records it in MSR (Model Specific Register) banks and can signal an MCE.
Two types:
Correctable errors (CE): hardware detected and corrected the error (single-bit ECC), system continues
Uncorrectable errors (UCE): hardware cannot correct — if not handled, causes kernel panic
Tools:
mcelog (older, legacy):
xxxxxxxxxx# Install and runapt-get install mcelogsystemctl start mcelogcat /var/log/mcelog# Decodes MCE records into human-readable descriptionsrasdaemon (modern replacement):
xxxxxxxxxxapt-get install rasdaemonsystemctl start rasdaemonras-mc-ctl --status # show error countsras-mc-ctl --errors # show all recorded errorsDirect kernel MCE messages:
xxxxxxxxxxdmesg -T | grep -E '(mce|MCE|EDAC|ECC|Machine Check)'# Example:# [Mon Jan 15 10:00:00 2024] mce: [Hardware Error]: Machine check events logged# [Mon Jan 15 10:00:00 2024] EDAC MC0: 1 CE page:0x12345 offset:0x678 grain:8 syndrome:0xabcd# CE = correctable error, MC0 = memory controller 0
# EDAC (Error Detection and Correction) subsystemcat /sys/bus/platform/drivers/EDAC/*mc/mc0/ce_count # correctable errorscat /sys/bus/platform/drivers/EDAC/*mc/mc0/ue_count # uncorrectable errorsResponding to MCEs:
| Error Type | Frequency | Action |
|---|---|---|
| Single CE | Occasional | Monitor; ECC doing its job |
| CE on same address | Increasing | Bad DIMM; schedule replacement |
| Multiple CEs | Frequent | Urgent DIMM replacement; schedule maintenance |
| UCE | Any | Immediate — risk of data corruption; take system out of service |
| CPU cache MCE | Any | Possible CPU fault; replace CPU or motherboard |
xxxxxxxxxx# Identify which DIMM has errorsras-mc-ctl --errors | grep -i 'corrected'# Output includes memory bank and DIMM slot information
# Verify with IPMI/BMCipmitool sel list | grep -i 'memory\|ECC'ipmitool sdr type Memory
# Map error address to DIMM slot (on DELL/HP servers)# Vendor-specific: use iDRAC/iLO web console or:edac-util -s 0 # show EDAC statusAnswer:
Random reboots without panic logs are among the hardest issues to diagnose. Work through these systematically:
Step 1: Determine if it was a clean or dirty reboot:
xxxxxxxxxx# Check last reboot reasonlast reboot | head -5who -b # last boot timeuptime
# If clean shutdown: systemd shutdown logjournalctl -b -1 | grep -E '(shutdown|reboot|poweroff)' | tail -10
# If dirty: previous boot journal will be cut off abruptlyjournalctl -b -1 | tail -20 # ends mid-stream = hard resetStep 2: Check for hardware watchdog:
xxxxxxxxxx# Some systems have a hardware watchdog that reboots if not refresheddmesg | grep -i watchdogsystemctl status watchdogls /dev/watchdog*# If watchdog service fails, hardware watchdog fires after timeoutStep 3: Check thermal events:
xxxxxxxxxxdmesg -b -1 -T | grep -i 'thermal\|temperature\|overheat\|throttl'journalctl -b -1 -k | grep -i thermal
# Current temperaturessensors # lm-sensors packageipmitool sdr type Temperature# BIOS may reboot if CPU/ambient temperature exceeds thresholdStep 4: Check power issues:
xxxxxxxxxx# IPMI/BMC system event log (BMC survives OS crashes)ipmitool sel list # all system eventsipmitool sel list | grep -i 'power\|voltage\|AC loss'# "AC Loss" = power cut# "Voltage" = PSU instability# "Power Unit" events
# ACPI power eventsjournalctl -b -1 | grep -i 'acpi\|power'Step 5: Check for OOM without MCE panic:
xxxxxxxxxxjournalctl -b -1 -k | grep -E '(oom-killer|Out of memory|Killed process)'dmesg -b -1 | grep 'oom\|OOM'# OOM killer kills processes but usually doesn't reboot unless panic_on_oom is setsysctl vm.panic_on_oom # if 1: OOM causes panicStep 6: Check for kernel oops (non-fatal, but indicates bug):
xxxxxxxxxxjournalctl -b -1 -k | grep -E '(Oops|BUG:|WARNING:|kernel BUG)'# Kernel oops doesn't reboot by default, but panic_on_oops=1 makes itsysctl kernel.panic_on_oopsStep 7: Enable persistent panic logging:
xxxxxxxxxx# /etc/sysctl.d/99-panic.conf:kernel.panic = 30 # reboot 30 seconds after panic (0=don't reboot)kernel.panic_on_oops = 1 # oops → panic → captured by kdump
# Ensure kdump is configured to capture the panic causeStep 8: Hardware diagnostics:
xxxxxxxxxx# Memory test (run offline)memtest86+ # from bootloader menu
# CPU stress teststress-ng --cpu $(nproc) --timeout 1h # will panic if CPU is faulty
# Disk healthsmartctl -a /dev/sda | grep -E '(Reallocated|Uncorrectable|Pending)'/proc/sys to tune kernel behavior for a high-traffic web server?Answer:
For a high-traffic web server, the goal is: maximize connection throughput, minimize latency, avoid resource exhaustion.
Network tuning:
xxxxxxxxxx# /etc/sysctl.d/99-webserver.conf
# Increase connection backlognet.core.somaxconn = 65535net.ipv4.tcp_max_syn_backlog = 65535
# Allow port reuse (faster connection teardown)net.ipv4.tcp_tw_reuse = 1
# Increase ephemeral port range (more outbound connections to upstreams)net.ipv4.ip_local_port_range = 10000 65000
# TCP receive/send buffers (for high-bandwidth connections)net.core.rmem_max = 134217728net.core.wmem_max = 134217728net.ipv4.tcp_rmem = 4096 87380 134217728net.ipv4.tcp_wmem = 4096 65536 134217728
# Faster ACK for short connectionsnet.ipv4.tcp_sack = 1net.ipv4.tcp_timestamps = 1
# NIC receive queuenet.core.netdev_max_backlog = 250000
# SYN cookies (protect against SYN flood)net.ipv4.tcp_syncookies = 1
# BBR congestion control (if kernel >= 4.9)net.core.default_qdisc = fqnet.ipv4.tcp_congestion_control = bbrFile descriptor limits:
xxxxxxxxxx# /etc/sysctl.d/99-webserver.conffs.file-max = 2097152fs.nr_open = 2097152
# In systemd service unit:# LimitNOFILE=1048576Memory:
xxxxxxxxxx# Reduce swappiness (keep web cache in RAM)vm.swappiness = 5
# Increase inode/dentry cache for serving many small filesvm.vfs_cache_pressure = 50 # default 100; lower = keep more in cacheApply:
xxxxxxxxxxsysctl --system # reload all /etc/sysctl.d/ files# orsysctl -p /etc/sysctl.d/99-webserver.conf
# Verifysysctl net.core.somaxconnsysctl net.ipv4.tcp_congestion_controlValidate in nginx config too:
xxxxxxxxxxworker_processes auto;worker_rlimit_nofile 1048576;events { worker_connections 65535; use epoll; multi_accept on;}dmesg ring buffer and the systemd journal for kernel messages?Answer:
The kernel ring buffer (dmesg):
A circular buffer in kernel memory, fixed size (default ~512KB, tunable via log_buf_len)
Contains messages written via printk() from kernel/drivers/modules
Volatile: lost on reboot, overwritten when full (oldest messages dropped)
Timestamps are in seconds-since-boot (hardware clock not always available at early boot)
Access: dmesg reads /dev/kmsg or via syslog() syscall
Always available even before journald starts
systemd-journald kernel capture:
journald reads /dev/kmsg continuously and indexes kernel messages into its binary journal
Persistent if /var/log/journal/ exists (survives reboots)
Proper wall-clock timestamps (journald converts monotonic to real time)
Indexed by boot ID, time, priority, allowing journalctl -b -1 -k to see previous boot's kernel messages
Rich metadata: cursor position, sequence numbers, priority
Comparison:
| Aspect | dmesg | journalctl -k |
|---|---|---|
| Persistence | RAM only, lost on reboot | Persistent (if configured) |
| Previous boots | No | Yes (-b -1, -b -2) |
| Timestamps | Seconds since boot | Real wall-clock time |
| Filtering | grep/awk | Rich query language |
| Availability | Immediate (kernel startup) | After journald starts |
| Size limit | Fixed ring buffer | Configurable (% of disk) |
| Format | Text | Binary (queryable) |
Practical difference:
xxxxxxxxxx# dmesg: see what kernel is saying RIGHT NOW, in real timedmesg -w # follow kernel messages live
# journalctl -k: analyze what kernel said over time, across bootsjournalctl -k -b -1 -p err # previous boot's kernel errors onlyjournalctl -k --since "30 min ago" # recent kernel messages with real timestampsWhen they diverge:
Very early boot messages (before journald starts) appear in dmesg but may not be in journal
If journald crashes/restarts, there's a gap in journal but dmesg is unaffected
If system was swapped/overloaded, dmesg may overwrite old messages but journal retains them
Answer:
Emergency mode is systemd's last-resort target: a minimal environment with only the root filesystem mounted (often read-only) and a root shell. It activates when a critical unit fails.
Step 1: Read the failure message:
xxxxxxxxxx# The prompt usually shows:# "Welcome to emergency mode! ... Press Enter for maintenance"# or shows a failed unit error
# After entering the root password:journalctl -xb # full journal for current boot with contextsystemctl --failed # list all failed unitsStep 2: Identify the failing unit:
xxxxxxxxxxjournalctl -xb | grep -E '(Failed|failed|error)' | head -20# Common culprits:# - systemd-fsck@dev-sda1.service — filesystem check failed# - local-fs.target — filesystem mount failed# - network.target — network configuration failedStep 3: Fix common causes:
Bad /etc/fstab entry:
xxxxxxxxxx# Mount root read-write firstmount -o remount,rw /
cat /etc/fstab# Look for: wrong UUID, wrong filesystem type, typo in mount point# Verify UUID:blkid
# Test mounting manually:mount UUID=xxxx-xxxx /mnt# If this fails: the device/filesystem is the problem
# Fix fstab:nano /etc/fstab # remove or fix bad entryFilesystem errors (fsck failed automatically):
xxxxxxxxxx# The dirty/errored filesystem is usually mentioned in the journal# Find which device:dmesg | grep -E '(EXT4-fs error|XFS.*error|fsck)'
# Run fsck manually (filesystem should NOT be mounted, or mounted read-only)fsck -y /dev/sda2 # repair ext4/ext3/ext2xfs_repair /dev/sda2 # repair XFS (run only when unmounted)
# After repair:exit # or rebootMissing mount point directory:
xxxxxxxxxxmkdir -p /mnt/data # create missing directorymount -o remount,rw /mkdir -p /data # create mount point that fstab referencesNFS mount failing at boot:
xxxxxxxxxx# NFS mounts should use _netdev option in fstab# Change: /etc/fstab: nfsserver:/share /mnt/nfs nfs defaults,_netdev 0 0# _netdev tells systemd to only mount after network is upStep 4: Remount and continue boot:
xxxxxxxxxx# After fixing the issuemount -o remount,rw / # ensure root is writable# Then either:exit # try to continue boot# or:systemctl default # start default.target# or:rebootPrevent recurrence:
xxxxxxxxxx# Add 'nofail' to optional mounts in fstab (system boots even if mount fails)# /dev/sdb1 /backup ext4 defaults,nofail 0 0
# For network filesystems:# server:/share /mnt/nfs nfs defaults,_netdev,nofail,x-systemd.automount 0 0Answer:
A cascading failure occurs when the failure of one component causes increased load or degraded behavior in other components, which then fail, further increasing load on remaining components — a positive feedback loop leading to total system failure.
Classic pattern:
xxxxxxxxxxOne backend fails→ Load redistributed to remaining backends→ Remaining backends become overloaded→ They start timing out / dropping requests→ Clients retry, multiplying load→ More backends fail→ Entire tier collapses
Detection signals:
xxxxxxxxxx# Latency climbing across all instances (not just one)# Error rate rising system-wide# Retry rate spiking (clients retrying makes it worse)# Thread/connection pool exhaustion spreading
# Monitor in real time:watch -n 2 'for host in web1 web2 web3; do echo -n "$host: " ssh $host "ss -s | grep estab; uptime | awk \"{print \\\$NF}\""done'
# Check load balancer stats# nginx upstream healthtail -f /var/log/nginx/error.log | grep "upstream"
# Check error ratesjournalctl -u myapp -f -p err | grep -c "error"Stopping a cascade in progress:
Shed load immediately (most important):
xxxxxxxxxx# Enable load shedding: return 503 to some % of requests# nginx: limit_req zone# Application: check queue depth, reject if above threshold
# Reduce haproxy max connections temporarilyecho "set maxconn frontend myapp 1000" | socat stdio /var/run/haproxy.sockDisable retries / reduce retry aggressiveness:
xxxxxxxxxx# Retries amplify load during cascade# Circuit breakers (if implemented) open automatically# Manual: set retry timeout to 0 or disable at load balancerAdd capacity fast:
xxxxxxxxxx# Scale out: add instances, increase autoscaling limits# Drain and restart overloaded instancesIsolate the root cause:
xxxxxxxxxx# Was there a bad deployment? Roll back:kubectl rollout undo deployment/myapp
# Is one slow backend dragging the others? Remove it:# nginx: mark upstream server down# haproxy: disable serverProtect remaining healthy instances:
xxxxxxxxxx# Bulkheads: limit connections from each client/tier# Drop traffic at the edge before it reaches overloaded backendsPrevention: circuit breakers, bulkheads, load shedding with priority queues, exponential backoff with jitter on clients, capacity buffers (never run at 100%).
Answer:
Split-brain occurs when a network partition causes a cluster to divide into two (or more) groups, each believing it is the authoritative primary and potentially accepting conflicting writes.
Detection:
xxxxxxxxxx# For etcd cluster: check member healthetcdctl endpoint health --clusteretcdctl endpoint status --cluster# Healthy quorum requires > N/2 members reachable
# For Pacemaker/Corosync clustercorosync-cmapctl | grep quorumcrm_mon -1 # cluster resource status# Look for "UNCLEAN" members or "split brain suspected"
# For PostgreSQL streaming replication# On primary:psql -c "SELECT * FROM pg_stat_replication;"# No entries = replicas think primary is gone
# For Redis Sentinelredis-cli -h sentinel_host -p 26379 sentinel mastersredis-cli -h sentinel_host -p 26379 sentinel sentinels mymasterThe quorum approach (prevention): Require a majority (quorum) of nodes to agree before any write is accepted or leadership is granted. A 3-node cluster requires 2 nodes to agree — a partition into 1+1+1 means no side has quorum, so no side proceeds.
xxxxxxxxxx# etcd: quorum requires ceil((N+1)/2) members# 3 nodes: 2 required — a 1/2 split means the 2-node side has quorum, 1-node side does not# Even split (2/2 with 4 nodes): neither side has quorum — both go read-only
# Check etcd quorum statusetcdctl endpoint status --cluster --write-out=table# If leader blank for any member: that member is partitionedFencing (STONITH — Shoot The Other Node In The Head):
xxxxxxxxxx# When split is detected, fence the node that shouldn't be primary# Power fence via IPMI:ipmitool -H 192.168.1.10 -U admin -P password chassis power off
# Pacemaker fence action:stonith_admin --fence node2# Forces the potentially-split node offline before the winner proceeds with writesDatabase-level split-brain recovery:
xxxxxxxxxx# PostgreSQL: if old primary was promoted as replica, it has diverged timeline# On new primary: check timelinepsql -c "SELECT timeline_id FROM pg_control_checkpoint();"
# Old primary must be demoted and resynced:pg_ctl stop -D /var/lib/postgresql/data# Reinitialize from new primary:pg_basebackup -h new_primary -D /var/lib/postgresql/data -P --wal-method=streamKey principle: prefer consistency over availability during network partition (CP in CAP theorem) for data stores. Use quorum writes and fencing to guarantee only one writer.
Answer:
Why clock skew matters in distributed systems:
Log correlation: Events that happened in order appear reversed; incident reconstruction is confusing
TLS certificate validation: Certificates appear expired or not yet valid if clocks diverge > allowed skew (usually 5 minutes)
Kerberos authentication: Fails if clock skew > 5 minutes
Distributed databases: CockroachDB, Spanner, Cassandra, and others use wall-clock time for ordering — skew causes stale reads or write conflicts
Caches: TTL expiry calculated incorrectly; stale data served longer than intended
Distributed tracing: Span timestamps wrong; waterfall diagrams inaccurate
Lease/lock expiry: A node holds a lease that has expired on the master's clock — split-brain risk
Diagnosing clock skew:
xxxxxxxxxx# Check current NTP synchronization statustimedatectl status# Look for: "System clock synchronized: yes", NTP service active
# Check NTP peer offsetsntpq -p # NTP (ntpd)chronyc tracking # Chronychronyc sources -v # Chrony: all NTP sources with offsets
# chronyc tracking output:# System time: 0.000123 seconds fast of NTP time# RMS offset: 0.000456 seconds# Frequency: 5.234 ppm fast# Stratum: 3
# Check offset across multiple hostsfor host in web1 web2 web3 db1 db2; do offset=$(ssh $host "chronyc tracking | grep 'System time' | awk '{print \$4, \$5}'") echo "$host: $offset"doneAcceptable thresholds:
General distributed systems: < 100ms
Kerberos: < 300 seconds (configurable, default 5 minutes)
Spanner / TrueTime: < 7ms (enforced by atomic clocks + GPS)
Cassandra: < 1 second recommended
Fixing clock skew:
xxxxxxxxxx# Check if NTP daemon is runningsystemctl status chronyd # or ntpd
# If not synced, force immediate syncchronyc makestep # Chrony: step the clock immediatelyntpdate -u pool.ntp.org # ntpdate: one-time sync (deprecated but works)
# Check for firewall blocking UDP 123nc -zuv pool.ntp.org 123iptables -L -n | grep 123
# Chrony configuration: /etc/chrony.conf# server 169.254.169.123 prefer iburst # AWS NTP server# server time.google.com iburst
# For VMs: ensure VM clock source is reliablecat /sys/devices/system/clocksource/clocksource0/current_clocksource# kvm-clock or tsc preferred for VMs
# Force chrony to accept large step (if clock is far off):# /etc/chrony.conf: makestep 1.0 3 (step if offset > 1s, for first 3 updates)systemctl restart chronydAnswer:
The thundering herd occurs when many processes/threads are simultaneously awakened or triggered by an event, but only one can handle it, causing all others to go back to sleep — wasting CPU and causing latency spikes.
Classic forms:
Accept thundering herd (pre-kernel 3.9): Multiple worker processes all accept() on the same listening socket. When a connection arrives, all workers wake up, one gets the connection, others go back to sleep. At 10k workers, this is massively wasteful.
Cache stampede (cache thundering herd): A hot cache entry expires. All concurrent requests miss the cache simultaneously, all query the database, all compute the same value, all try to update the cache. The database gets hit with N identical queries at once.
Reconnect storms: All clients lose connection to a server simultaneously (server restart). All reconnect immediately, overwhelming the server.
Mitigations:
Accept thundering herd:
xxxxxxxxxx# SO_REUSEPORT (kernel 3.9+): each worker has its own socket, kernel distributes# nginx:listen 80 reuseport;
# Verify:ss -tlnp | grep :80# Multiple sockets on same port, each owned by different worker processCache stampede / cache thundering herd:
xxxxxxxxxx# Option 1: Mutex/lock on cache miss (one request recomputes, others wait)import threading_cache_locks = {}
def get_value(key): value = cache.get(key) if value is not None: return value lock = _cache_locks.setdefault(key, threading.Lock()) with lock: # Double-check after acquiring lock value = cache.get(key) if value is None: value = expensive_computation(key) cache.set(key, value, ttl=300) return value
# Option 2: Probabilistic early expiry (XFetch / PER algorithm)# Start recomputing slightly before expiry based on computation time# Spreads recomputation across time window
# Option 3: Staggered TTLs# Add random jitter to TTL: ttl = base_ttl + random(0, base_ttl * 0.1)cache.set(key, value, ttl=300 + random.randint(0, 30))Reconnect storms:
xxxxxxxxxx# Exponential backoff with jitterimport random, time
def connect_with_backoff(base_delay=1, max_delay=60, jitter=True): delay = base_delay while True: try: return connect() except ConnectionError: if jitter: sleep_time = delay + random.uniform(0, delay * 0.5) else: sleep_time = delay time.sleep(sleep_time) delay = min(delay * 2, max_delay)epoll thundering herd (Linux kernel):
xxxxxxxxxx# EPOLLEXCLUSIVE flag (kernel 4.5+): only ONE epoll waiter is woken per event# Prevents all threads in a thread pool from waking on each socket event# Set via epoll_ctl() with EPOLLEXCLUSIVE flag in application codeAnswer:
When a problem is isolated to one host out of many, the most productive approach is differential diagnosis — compare the affected host to healthy hosts.
Step 1: Confirm the problem is truly host-specific:
xxxxxxxxxx# Is it consistent on that host? Or intermittent?for i in $(seq 1 20); do ssh affected_host "curl -s -o /dev/null -w '%{http_code} %{time_total}\n' http://localhost/health" sleep 1done# vs. same from a healthy hostStep 2: Diff the host against healthy peers:
xxxxxxxxxx# Compare key system parametersfor param in kernel.version os.release cpu.model memory.total disk.free; do echo "=== $param ===" ssh affected_host "uname -r; cat /etc/os-release | head -3; lscpu | grep 'Model name'; free -h | head -2; df -h /"done
# Package versionsssh affected_host "dpkg -l myapp" vs ssh healthy_host "dpkg -l myapp"
# Configuration filesssh affected_host "md5sum /etc/myapp/*.conf" > /tmp/affected_conf.md5ssh healthy_host "md5sum /etc/myapp/*.conf" > /tmp/healthy_conf.md5diff /tmp/affected_conf.md5 /tmp/healthy_conf.md5Step 3: Check host-specific metadata:
xxxxxxxxxx# Hardware differencesssh affected_host "lshw -short 2>/dev/null | head -20"ssh affected_host "dmidecode -t processor | grep 'Version'"ssh affected_host "smartctl -H /dev/sda"
# Uptime (could be stale state from long uptime)ssh affected_host "uptime; last reboot | head -3"
# Recent changesssh affected_host "rpm -qa --last | head -10" # RHELssh affected_host "dpkg -l --no-pager | diff - <(ssh healthy_host dpkg -l --no-pager)"
# Kernel parametersfor key in net.ipv4.tcp_tw_reuse vm.swappiness fs.file-max; do affected=$(ssh affected_host "sysctl $key") healthy=$(ssh healthy_host "sysctl $key") if [ "$affected" != "$healthy" ]; then echo "DIFF $key: affected=$affected healthy=$healthy" fidoneStep 4: Check resource exhaustion specific to that host:
xxxxxxxxxxssh affected_host " echo '=== FD usage ===' && cat /proc/sys/fs/file-nr echo '=== Memory ===' && free -h echo '=== Disk ===' && df -h echo '=== Inode usage ===' && df -i echo '=== Network ===' && ss -s echo '=== Top processes ===' && ps aux --sort=-%cpu | head -10"Step 5: Check if it's a DNS/network routing issue specific to that host:
xxxxxxxxxx# Can the affected host reach its dependencies?ssh affected_host "for dep in db1 cache1 api1; do ping -c1 -W1 \$dep && echo OK || echo FAIL; done"
# Routing differencesdiff <(ssh affected_host "ip route") <(ssh healthy_host "ip route")Step 6: Parallel-ssh for fleet-wide checks:
xxxxxxxxxx# pssh / parallel-sshpssh -H "host1 host2 host3 affected_host" -i "uptime; ss -s | grep estab"
# Or with pdshpdsh -w host[1-10],affected_host "df -h /" | sortAnswer:
Phase 1: Detect and Acknowledge (0–5 minutes)
xxxxxxxxxx- Alerting fires (PagerDuty, OpsGenie)- On-call engineer acknowledges alert- Declare incident: create incident channel (#incident-YYYY-MM-DD-brief)- Set incident commander (IC) — single person coordinates- Announce: "I am taking IC. Brief description of alert. Assessing."
Phase 2: Triage (5–15 minutes)
xxxxxxxxxx# Quantify impact immediately:# - What % of users affected? (check error rate, not just alerts)# - Which regions/services?# - Is it getting better or worse?
# Quick health checks:curl -s https://api.example.com/health | jq .kubectl get pods --all-namespaces | grep -v Running# Check monitoring dashboard: error rate, latency, throughput
# Declare severity:# SEV1: total outage, all users affected# SEV2: major degradation, >20% users affected# SEV3: minor degradationPhase 3: Communicate (ongoing throughout)
xxxxxxxxxx- Status page update within 10 minutes of declaring incident- "We are investigating increased error rates. More updates in 15 minutes."- Regular updates every 15-30 minutes even if nothing changes- Never go silent — users/leadership need acknowledgment
Phase 4: Mitigate (primary goal — restore service, worry about root cause later)
xxxxxxxxxx# Look for recent changes first (most outages are caused by changes)git log --since="2 hours ago" --oneline # recent commitskubectl rollout history deployment/myapp # recent k8s deploys# "When did this start?" → correlate with change timeline
# Rollback if change-correlated:kubectl rollout undo deployment/myappgit revert HEAD --no-edit && git push
# If not change-correlated:# - Restart unhealthy instances# - Scale out to absorb load# - Enable maintenance mode / serve cached content# - Fail over to backup regionPhase 5: Resolve and Monitor
xxxxxxxxxx# Confirm mitigation worked:# Error rate back to baseline# Latency back to normal# No further alerts firing
# Keep watching for 15-30 minutes before declaring resolution# Declare: "Incident resolved. Error rate returned to baseline at HH:MM UTC."# Update status pagePhase 6: Postmortem (within 48 hours)
xxxxxxxxxx- Timeline (not blame)- Root cause- Contributing factors- What went well- Action items with owners and deadlines- Blameless culture: "The system failed, not the person"
Answer:
A blameless postmortem (post-incident review) focuses on improving systems and processes, not on identifying and punishing individuals. The premise: engineers make the best decisions with the information available at the time; system design allowed the error to have impact.
Structure:
xxxxxxxxxx# Incident Postmortem: [Brief Title]**Date:** 2024-01-15**Duration:** 14:23 UTC – 15:47 UTC (1h 24m)**Severity:** SEV2**Impact:** 23% of API requests returned 503; ~15,000 users affected**On-call:** [names]
## SummaryOne-paragraph description of what happened, impact, and how it was resolved.
## Timeline (UTC)| Time | Event ||-------|-------|| 13:55 | Deploy of v2.3.1 to production (25% canary) || 14:18 | Error rate starts rising (0.1% → 2%) — not yet alerting || 14:23 | Alert fires: error rate > 5% || 14:25 | On-call acknowledges, begins investigation || 14:31 | Identified correlation with v2.3.1 deploy || 14:35 | Rollback initiated || 15:12 | Rollback completes; error rate declining || 15:47 | Error rate returns to baseline; incident resolved |
## Root Cause[Specific technical cause, not "human error"]Example: "The new connection pool implementation in v2.3.1 had a race conditionwhen pool size was exhausted. Under the load of the canary traffic, the poolexhausted within 8 minutes, causing all new requests to fail."
## Contributing Factors- Load testing did not replicate production connection patterns- Canary rollout alert threshold (5%) was too high to catch gradual degradation- Connection pool exhaustion was not a tracked metric
## What Went Well- On-call was paged promptly- Rollback procedure was practiced and completed in 37 minutes- Status page was updated within 8 minutes
## What Could Be Improved- Load test environment did not match production (action item)- Alert fired after 5 minutes of impact; lower threshold needed
## Action Items| Action | Owner | Due Date ||--------|-------|----------|| Add connection pool saturation metric + alert | @engineer1 | 2024-01-22 || Lower canary error rate alert to 1% | @engineer2 | 2024-01-19 || Update load test to use production connection patterns | @engineer3 | 2024-01-29 || Document rollback procedure in runbook | @engineer4 | 2024-01-22 |Key principles:
No names in the root cause section — describe system behavior, not "John forgot to..."
Specific, measurable action items — not "improve monitoring" but "add alert for connection pool > 80% saturation"
Timeline is factual — what happened, when, no editorializing
Share widely — postmortems improve the whole organization
Answer:
Context: A canary deployment sends a small percentage (1-10%) of traffic to the new version while the rest goes to stable. If canary shows elevated errors, you must decide: roll back, fix-forward, or reduce canary percentage.
Step 1: Confirm the canary is the source:
xxxxxxxxxx# Compare error rate by version tag# In Prometheus:rate(http_requests_total{status=~"5..", version="v2.3.1"}[5m]) /rate(http_requests_total{version="v2.3.1"}[5m])# vs:rate(http_requests_total{status=~"5..", version="v2.3.0"}[5m]) /rate(http_requests_total{version="v2.3.0"}[5m])
# In kubectl:kubectl logs -l version=v2.3.1 --tail=50 | grep -c ERRORkubectl logs -l version=v2.3.0 --tail=50 | grep -c ERROR
# nginx split_clients: check access log by upstreamgrep "canary" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -cStep 2: Characterize the error:
xxxxxxxxxx# What type of errors? (5xx, timeouts, crashes?)kubectl logs -l version=v2.3.1 --tail=200 | grep -E 'ERROR|FATAL|panic' | head -20
# Is it all endpoints or specific?# Is it correlated with load? (errors started at time X = certain traffic level)# Is it getting worse (leak) or stable (configuration bug)?Step 3: Decision matrix:
| Error Rate | Trend | Action |
|---|---|---|
| < 0.1% above baseline | Stable | Continue canary, investigate |
| 0.1-1% above baseline | Stable | Reduce canary %, investigate |
| > 1% above baseline | Any | Roll back immediately |
| Any | Increasing | Roll back immediately |
Step 4a: Roll back (most likely action):
xxxxxxxxxx# Kubernetes:kubectl rollout undo deployment/myappkubectl rollout status deployment/myapp # watch progress
# Argo Rollouts:kubectl argo rollouts abort myappkubectl argo rollouts undo myapp
# Verify rollback worked:kubectl get pods -l app=myapp -o wide | grep version# Confirm error rate dropsStep 4b: Reduce canary percentage:
xxxxxxxxxx# Argo Rollouts: reduce weightkubectl argo rollouts set weight myapp 5 # reduce from 25% to 5%
# nginx split_clients: change percentage in config# Reload nginx:nginx -s reloadStep 5: Investigate root cause before re-deploying:
xxxxxxxxxx# Get the specific error from canary podskubectl exec -it canary-pod -- journalctl -u myapp -n 100 | grep ERROR
# Compare config between versionskubectl get deployment myapp-canary -o yaml > canary.yamlkubectl get deployment myapp-stable -o yaml > stable.yamldiff canary.yaml stable.yaml
# Check if new version has different dependencies/migrationsStep 6: Communicate:
xxxxxxxxxx"Canary v2.3.1 showing 3% error rate vs 0.05% baseline.Rolling back to v2.3.0. Investigating connection pool issue in new version."
Answer:
Health checks have two distinct purposes — liveness and readiness — and conflating them causes problems.
Liveness probe: "Is the process alive or should it be restarted?"
Should be lightweight and almost always pass
Failure → container restart
Check: is the process running? Can it respond to a simple request?
Should NOT check dependency health (if DB is down, restarting the app won't help)
Readiness probe: "Is this instance ready to receive traffic?"
Can check dependency health (DB, cache, upstreams)
Failure → remove from load balancer pool (no restart)
Recovers when dependencies recover
Check: can the service actually handle requests right now?
Kubernetes example:
xxxxxxxxxxlivenessProbe httpGet path/healthz # simple: returns 200 if process is alive port8080 initialDelaySeconds10 periodSeconds10 failureThreshold3 # 3 failures = restart
readinessProbe httpGet path/ready # checks: DB connection, cache ping, disk space port8080 initialDelaySeconds5 periodSeconds5 failureThreshold1 # 1 failure = remove from LB immediatelyHealth check endpoint implementation:
xxxxxxxxxx# /healthz - liveness.route('/healthz')def liveness(): # Just check the process can handle requests return jsonify({"status": "ok"}), 200
# /ready - readiness.route('/ready')def readiness(): checks = {} status_code = 200 # Database try: db.execute("SELECT 1") checks["database"] = "ok" except Exception as e: checks["database"] = f"error: {e}" status_code = 503 # Cache try: cache.ping() checks["cache"] = "ok" except Exception as e: checks["cache"] = f"error: {e}" status_code = 503 # Disk space usage = shutil.disk_usage("/") if usage.free / usage.total < 0.05: # less than 5% free checks["disk"] = "low" status_code = 503 else: checks["disk"] = "ok" return jsonify({"status": "ok" if status_code == 200 else "degraded", "checks": checks}), status_codeStartup probe (Kubernetes): For slow-starting services that would fail liveness during initialization:
xxxxxxxxxxstartupProbe httpGet path/healthz port8080 failureThreshold30 # up to 5 minutes to start (30 × 10s) periodSeconds10Anti-patterns:
Liveness check that queries the database (DB down → all pods restart → thundering herd on DB)
Readiness check that's too strict (checks non-critical dependencies → unnecessary traffic shedding)
Health check that takes > 1 second (check timeout must be longer than check duration)
No health check at all (load balancer never removes dead backends)
parallel-ssh (pssh) or similar tools to diagnose issues across a fleet of hosts?Answer:
pssh (parallel-ssh):
xxxxxxxxxx# Installapt-get install pssh# or: pip install parallel-ssh
# Basic usage: run command on multiple hostspssh -H "web1 web2 web3 web4" -i "uptime"# -i: print output inline (with hostname prefix)# -o /tmp/pssh_out: save output per host to files
# From a hosts file:pssh -h /etc/pssh/webservers -i "df -h /"# /etc/pssh/webservers contains one hostname per line
# Parallel execution with timeoutpssh -H "$(cat /etc/pssh/all_hosts | tr '\n' ' ')" -t 30 -i "systemctl status myapp"# -t 30: timeout 30 seconds per hostCommon diagnostic patterns:
xxxxxxxxxx# Find which hosts have a service downpssh -h hosts.txt -i "systemctl is-active myapp || echo FAILED" 2>/dev/null | grep FAILED
# Check disk space across fleetpssh -h hosts.txt -i "df -h / | tail -1" | sort -k5 -rn | head -10
# Find hosts with high loadpssh -h hosts.txt -i "uptime" | awk '{print $NF, $1}' | sort -rn | head -10
# Check specific log entries across all hostspssh -h hosts.txt -i "journalctl -u myapp -p err --since '30 min ago' --no-pager | wc -l"
# Compare config file MD5 across fleetpssh -h hosts.txt -i "md5sum /etc/myapp/config.yaml" | sort -k1
# Find which host has the most connectionspssh -h hosts.txt -i "ss -s | grep estab" | sort -t: -k2 -rn
# Check kernel version across fleetpssh -h hosts.txt -i "uname -r" | sort | uniq -c | sort -rnpdsh (alternative, often faster):
xxxxxxxxxx# Installapt-get install pdsh
# Run on rangepdsh -w web[1-20] "df -h /" | sort
# Run on comma-separated listpdsh -w web1,web2,db1,db2 "uptime"
# With WCOLL (host list file)export WCOLL=/etc/pdsh/hostspdsh "free -h | head -2"Ansible for more complex fleet operations:
xxxxxxxxxx# Ad-hoc commandansible webservers -m command -a "df -h /"ansible webservers -m shell -a "journalctl -u myapp -p err -n 5 --no-pager"
# Gather factsansible webservers -m setup -a "filter=ansible_memfree_mb" | grep memfreeFor Google-scale fleets: Tools like mosh + tmux synchronize-panes, or internal fleet management systems (Borg, Kubernetes) provide native fleet-wide querying via control plane APIs rather than SSH.
Answer:
Exponential backoff: After a failed request, wait before retrying. Double the wait time with each subsequent failure.
xxxxxxxxxxAttempt 1 fails → wait 1sAttempt 2 fails → wait 2sAttempt 3 fails → wait 4sAttempt 4 fails → wait 8s...up to max (e.g., 60s)
The problem with pure exponential backoff (no jitter): If 1,000 clients all started retrying at the same time (e.g., server restart), they all hit the same backoff schedule. After the wait period, they ALL retry simultaneously → server gets hit by 1,000 requests at once → server struggles → all fail again → all wait same 2s → all retry again simultaneously. This synchronized retry is itself a load problem.
Jitter: Add randomness to desynchronize retries:
xxxxxxxxxximport randomimport time
def retry_with_backoff(func, max_attempts=5, base_delay=1.0, max_delay=60.0): for attempt in range(max_attempts): try: return func() except TransientError as e: if attempt == max_attempts - 1: raise # Full jitter (AWS recommendation): random between 0 and cap cap = min(base_delay * (2 ** attempt), max_delay) sleep_time = random.uniform(0, cap) # Alternative: Equal jitter: half fixed, half random # sleep_time = cap/2 + random.uniform(0, cap/2) # Alternative: Decorrelated jitter (better spread): # sleep_time = random.uniform(base_delay, prev_sleep * 3) time.sleep(sleep_time)Full jitter distribution: Over many clients, requests are spread uniformly across the [0, cap] window → server sees steady trickle rather than synchronized bursts.
When NOT to retry:
xxxxxxxxxx# Don't retry on non-transient errors# 400 Bad Request → bug in client; retry won't fix it# 401 Unauthorized → credentials invalid; retry won't fix it# 404 Not Found → resource missing; retry won't fix it# 429 Too Many Requests → retry ONLY after Retry-After header# 503 Service Unavailable → retry with backoff# Connection timeout → retry with backoffCircuit breaker pattern (complement to backoff):
xxxxxxxxxx# If 50% of requests fail in last 60 seconds → open circuit# While circuit is open: fail fast (don't even try the request)# After 30 seconds: allow one request through (half-open)# If it succeeds: close circuit; if it fails: stay open# Prevents cascading failures by detecting when a service is downAnswer:
A rolling restart updates/restarts instances one at a time (or in small batches) to avoid taking the entire service offline.
Kubernetes rolling restart:
xxxxxxxxxx# Trigger rolling restart (all pods restarted one by one)kubectl rollout restart deployment/myapp
# Watch progresskubectl rollout status deployment/myapp
# Configure rolling update parameters in deployment spec:spec: strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 # max extra pods during update maxUnavailable: 0 # never reduce below desired count
# Pause and resume if issues detected:kubectl rollout pause deployment/myappkubectl rollout resume deployment/myapp
# Rollback if needed:kubectl rollout undo deployment/myappManual rolling restart with health checks:
xxxxxxxxxxHOSTS=(web1 web2 web3 web4 web5 web6)LB_POOL="myapp_pool"
for host in "${HOSTS[@]}"; do echo "=== Restarting $host ===" # Remove from load balancer echo "Draining $host from LB..." haproxy_cmd "disable server $LB_POOL/$host" # Wait for in-flight requests to complete echo "Waiting for connections to drain..." sleep 10 # Verify no active connections remaining=$(ssh $host "ss -tn state established dport :80 | wc -l") echo "Remaining connections: $remaining" # Restart service ssh $host "systemctl restart myapp" # Wait for service to be ready echo "Waiting for health check..." for i in $(seq 1 30); do if ssh $host "curl -sf http://localhost:8080/health" > /dev/null 2>&1; then echo "$host is healthy" break fi if [ $i -eq 30 ]; then echo "ERROR: $host failed health check after restart!" # Don't continue the rolling restart — something is wrong exit 1 fi sleep 2 done # Re-enable in load balancer haproxy_cmd "enable server $LB_POOL/$host" echo "$host back in service" echo "Sleeping 30s before next host..." sleep 30done
echo "Rolling restart complete"Safety checks to add:
xxxxxxxxxx# Before starting: confirm current state is healthycurrent_error_rate=$(prometheus_query 'rate(http_errors_total[5m])')if (( $(echo "$current_error_rate > 0.01" | bc -l) )); then echo "ERROR: Error rate $current_error_rate is already elevated. Aborting." exit 1fi
# After each restart: confirm error rate didn't increasesleep 60 # wait for metrics to stabilizenew_error_rate=$(prometheus_query 'rate(http_errors_total[5m])')if (( $(echo "$new_error_rate > $current_error_rate * 2" | bc -l) )); then echo "ERROR: Error rate doubled after restarting $host. Stopping." exit 1fiAnswer:
Alert quality principles:
1. Every alert must be actionable:
xxxxxxxxxxBad alert: "CPU > 80%" (so what? what do I do?)Good alert: "API latency P99 > 2s for >5min on payment-service — check DB query performance"
2. Alert on symptoms, not causes:
xxxxxxxxxxSymptom: User-visible error rate > 1% (alert on this)Cause: CPU high, DB slow, memory low (use these for diagnosis, not paging)
3. Tune thresholds to minimize false positives:
xxxxxxxxxx# Too sensitive: pages constantly for transient spikes# Too loose: misses real problems
# Use for=5m to require sustained condition before alerting# Prometheus example:- alert: HighErrorRate expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.01 for: 5m # must persist for 5 minutes labels: severity: critical annotations: summary: "Error rate {{ $value | humanizePercentage }} on {{ $labels.service }}" runbook_url: "https://wiki/runbook/high-error-rate"4. Include runbook links in every alert:
xxxxxxxxxxannotations runbook_url"https://internal-wiki/runbooks/service-X-alert-Y" # Runbook content: What is this alert? What to check first? How to mitigate?Reducing alert fatigue:
xxxxxxxxxx# Audit your alerts regularly# For each alert that fired in the past month:# - Was it actionable? (Did it require human intervention?)# - Was it a true positive? (Was there actually a problem?)# - Did it wake someone up unnecessarily?
# Categorize alerts by urgency:# PAGE (wake someone up): customer impact occurring right now# TICKET (next business day): degradation that doesn't require immediate action# LOG (just log it): informational, no action needed
# Implement alert inhibition: don't page for downstream effects# if the root cause is already being alerted oninhibit_rules: - source_match: alertname: DatabaseDown target_match: alertname: APIHighErrorRate equal: ['datacenter']On-call health practices:
xxxxxxxxxx- Rotation: spread on-call across team (no single person always on-call)- Handoff: clear handoff notes (what's currently broken, known issues)- Sleep protection: secondary on-call for after-hours escalation; don't page for non-critical at 3am- Post-incident: every page that woke someone up gets an action item to prevent recurrence or remove the alert- On-call review meetings: weekly review of all alerts fired, tune thresholds- Escalation paths: clear escalation matrix (what warrants waking up a senior engineer vs. handling alone)
Measuring on-call health:
xxxxxxxxxx# Metrics to track:# - Pages per on-call shift (target: <2 actionable pages per day)# - Pages outside business hours# - Mean time to acknowledge (MTTA)# - Mean time to resolve (MTTR)# - Alert signal-to-noise ratio (actionable / total alerts)# - Number of repeat alerts (same alert firing repeatedly = not fixed properly)Answer:
Step 1: Confirm and quantify the leak:
xxxxxxxxxx# Monitor RSS over timewatch -n 10 'ps -o pid,rss,vsz,pmem,comm -p PID'
# Or collect historywhile true; do echo "$(date +%s) $(cat /proc/PID/status | awk '/VmRSS/{print $2}')" sleep 60done | tee /tmp/mem_trend.log
# Plot rate of growth: if consistently linear → leak confirmedawk '{print $2 - prev_mem, $1} BEGIN{prev_mem=0}' /tmp/mem_trend.log | head -20Step 2: Identify which allocations are growing:
xxxxxxxxxx# For C/C++ processes: attach gdb and call malloc_stats() / malloc_info()gdb -p PID -batch -ex 'call malloc_stats()' 2>/dev/null
# For Java: heap histogram without stoppingjmap -histo:live PID | head -30# Run twice, 60 seconds apart, compare growing classes
# For Python: tracemalloc (if enabled in app)# Or: use py-spy or memraymemray attach PID # if memray is installed
# For Go: pprof heap profilecurl http://localhost:6060/debug/pprof/heap > heap1.out# Wait 60 secondscurl http://localhost:6060/debug/pprof/heap > heap2.outgo tool pprof -http=:8081 -diff_base heap1.out heap2.outStep 3: Mitigate without restart:
Memory limit + graceful restart (if using systemd):
xxxxxxxxxx# /etc/systemd/system/myapp.service# MemoryMax=512M — systemd kills process if it exceeds limit# Restart=on-failure — systemd restarts it automatically
# Trigger manual graceful restart (if app supports it):kill -SIGUSR2 PID # some apps do graceful restart on SIGUSR2Blue-green swap (zero-downtime memory drain):
xxxxxxxxxx# Start new instance (fresh, no leak)systemctl start myapp-green
# Wait for health checkcurl http://localhost:8081/health
# Shift load to green# (nginx: update upstream, haproxy: enable green/disable blue)
# Let blue drain connectionssleep 30
# Stop blue (old leaky instance)systemctl stop myapp-blue
# Now blue is fresh, restart and add back:systemctl start myapp-blue# Shift some load back, let green become the next blue...cgroup memory limit as circuit breaker:
xxxxxxxxxx# Prevent the process from killing the hostsystemctl set-property myapp MemoryMax=2G# Process will OOM within its cgroup before affecting other processesStep 4: Permanent fix:
xxxxxxxxxx# Instrument the code to find the leak:# 1. AddressSanitizer (recompile with -fsanitize=address)# 2. Valgrind in staging environment# 3. heaptrack for C++ allocations# 4. Memory profiler specific to runtime (JProfiler, pyspy, pprof)Answer:
This is one of the most nuanced on-call decisions. The tension: restoring service quickly may destroy the forensic evidence needed to understand the root cause.
The guiding principle: In most cases, restore service first, then investigate. Users being down is usually more costly than a delayed root cause analysis. However, there are exceptions.
Decision framework:
xxxxxxxxxxIs the incident currently causing user impact?├─ YES: Restore service immediately (rollback, restart, failover)│ Then: preserve any remaining evidence for postmortem│└─ NO (degraded but not down, or intermittent):Take time to collect evidence firstThen: mitigate carefully
Evidence preservation techniques that don't delay mitigation:
xxxxxxxxxx# 1. Capture state BEFORE rollback (takes 30-60 seconds)# Core dump of crashing processgcore -o /tmp/incident_snapshot PID
# Memory map and open filescat /proc/PID/maps > /tmp/incident_maps.txtls -la /proc/PID/fd > /tmp/incident_fds.txtcat /proc/PID/status > /tmp/incident_status.txt
# Network connections at time of incidentss -antp > /tmp/incident_connections.txtnetstat -s > /tmp/incident_netstat.txt
# Current log tailjournalctl -u myapp -n 500 --no-pager > /tmp/incident_logs.txt
# 2. Take a snapshot/checkpoint BEFORE rolling back# k8s: describe pods with eventskubectl describe pods -l app=myapp > /tmp/incident_pods.txtkubectl get events --sort-by='.lastTimestamp' > /tmp/incident_events.txt
# 3. Preserve metrics screenshots (dashboards show state at incident time)
# 4. Add extra logging before restarting (if time permits)# Increase log verbosity temporarilykill -SIGUSR1 PID # many apps increase log level on SIGUSR1Parallel tracks approach:
xxxxxxxxxxPerson A: Execute mitigation (rollback, restart)Person B: Simultaneously collect evidence→ Both happening at same time, not sequentially
When to delay recovery for evidence:
Security incident: preserving forensic evidence may be legally required
Data corruption bug: restarting may overwrite audit trail
Very intermittent bug that's hard to reproduce: the current state may be the only chance to see it
Bug that's not causing user impact yet: time to safely investigate
Communication during the decision:
xxxxxxxxxx"We are investigating a production issue.We have two options:1. Rollback immediately (service restored in ~5 minutes, root cause TBD)2. Collect diagnostics first (~10 minutes, then rollback)Current user impact: X% error rate.Recommendation: [Option 1 / Option 2] because [reason].Awaiting confirmation."
Post-mitigation evidence gathering: Even after restoring service, evidence exists:
Logs in journald/Elasticsearch (timestamped, don't disappear)
Metrics in Prometheus/Grafana (retained per retention policy)
Traces in Jaeger/Zipkin
Old pods in Kubernetes events
Previous deployment artifacts
The act of restoring service rarely destroys all evidence — it mainly loses the live process state (memory, open files, etc.).
At Google: The standard is clear: restore service first (SLO/error budget drives urgency), then conduct thorough postmortem. kdump and pre-configured diagnostics collection scripts run automatically, so engineers don't have to choose between speed and evidence.
End of Part 2: Questions 76–150