System Internals — Top 150 Interview Q&A


System Calls

Q1: What is a system call?

Answer: A system call is the interface between user-space programs and the kernel. Since user processes run in a restricted CPU mode (ring 3 on x86), they cannot directly access hardware or kernel data structures. A syscall is a controlled entry point into the kernel: the process sets up arguments (in registers) and executes a special instruction (syscall on x86-64, svc on ARM) that transitions the CPU to privileged mode (ring 0), where the kernel performs the requested operation and returns the result.

Q2: How does a system call work at the hardware level on x86-64?

Answer:

  1. User code puts syscall number in rax, arguments in rdi, rsi, rdx, r10, r8, r9.

  2. Executes syscall instruction — CPU saves rip/rflags/rsp to kernel-defined locations, switches to kernel stack, loads kernel entry point from MSR (LSTAR).

  3. Kernel handler (entry_SYSCALL_64) runs, dispatches to syscall table entry.

  4. Result returned in rax (negative = errno).

  5. sysretq returns to user mode, restores registers.

Q3: What is the system call table?

Answer: A table (array of function pointers) in the kernel mapping syscall numbers to handler functions. On Linux x86-64: sys_call_table[] in arch/x86/entry/syscall_64.c. Syscall 0 = read, 1 = write, 2 = open, etc. (per <asm/unistd_64.h>).

Q4: What is the difference between syscall() and a libc wrapper?

Answer: libc wrappers (e.g., read(), write()) call the actual kernel via syscall. They handle: errno setting (from negative return value), argument type conversions, and sometimes add POSIX compatibility behavior. The raw syscall(SYS_read, ...) bypasses libc and calls the kernel directly.

Q5: What is vDSO (virtual dynamic shared object)?

Answer: vDSO maps a small kernel-provided shared library into every process's address space. It allows certain syscalls (gettimeofday, clock_gettime, getcpu) to execute in user space without mode switch by reading kernel-maintained data (e.g., time in shared memory). Eliminates syscall overhead for high-frequency time reads.

Q6: What is VVAR and how does it relate to vDSO?

Answer: vvar is a read-only memory region mapped alongside vDSO containing data that the kernel updates (e.g., current time, timezone). The vDSO code reads from vvar to implement clock_gettime without entering the kernel. Visible in /proc/<pid>/maps as [vvar] and [vdso].

Q7: What does errno represent?

Answer: errno is a thread-local integer variable (defined in <errno.h>) that holds the error code from the last failed syscall. The kernel returns negative errno values; the libc wrapper negates and stores them in errno, then returns -1. EAGAIN, EINTR, ENOENT, etc. are defined error codes.

Q8: How do you find all available syscalls on a Linux system?

Q9: What is seccomp and how does it restrict syscalls?

Answer: Secure Computing mode (seccomp) is a kernel facility to restrict which syscalls a process may make. Mode 1 (strict): only read, write, _exit, sigreturn allowed — anything else → SIGKILL. Mode 2 (BPF filter): a BPF program filters syscalls by number/arguments. Used in containers (Docker seccomp profiles), browsers (Chrome sandbox), and security-critical applications.

Q10: What is ptrace() and how does it work?

Answer: ptrace is a syscall that allows one process (tracer) to inspect and control another (tracee). The tracer can: stop the tracee at each syscall entry/exit, read/write its memory and registers, inject signals, and single-step execution. Basis for strace, gdb, and sandboxes. PTRACE_SEIZE allows non-stopping attach; PTRACE_O_EXITKILL kills the tracee if the tracer dies.

Q11: What is the overhead of making a system call?

Answer: A syscall costs ~100ns on modern hardware (vs. ~1ns for a function call). Costs include: mode switch (ring3→ring0→ring3), kernel stack setup, register save/restore, TLB/cache effects. Spectre/Meltdown mitigations (KPTI page table isolation) add ~5–10% overhead. Batch I/O (readv/writev, io_uring) amortizes this cost.

Q12: What is io_uring and why is it significant?

Answer: io_uring (Linux 5.1+) is an async I/O interface using shared ring buffers (submission queue + completion queue) between user space and kernel. Eliminates syscall overhead per operation by batching I/O through shared memory. Supports pre-registered buffers, fixed files, linked requests. Used in high-performance databases (PostgreSQL, RocksDB) and networking. Can operate entirely without syscalls for hot paths.

Q13: What is the difference between read() and pread()?

Answer: read(fd, buf, n) reads from the current file position (maintained per open file description) and advances it. pread(fd, buf, n, offset) reads from the specified offset without changing the file position — atomic offset+read. Essential for multi-threaded file access where threads share an FD.

Q14: What is the difference between select(), poll(), and epoll()?

Answer:

Q15: What is the splice() syscall?

Answer: splice() moves data between a file descriptor and a pipe (or between two pipes) without copying to user space — zero-copy within the kernel. Used for efficient data forwarding (e.g., web server sending file to network socket: splice(file, NULL, pipe[1], ...) then splice(pipe[0], NULL, socket, ...)).

Q16: What is sendfile()?

Answer: sendfile(out_fd, in_fd, offset, count) transfers data from a regular file to a socket directly in the kernel, bypassing user-space copy. One syscall for data transfer. Used by web servers (nginx) for efficient static file serving. Similar to splice() but specifically for file-to-socket.

Q17: What is mmap()'s MAP_SHARED vs MAP_PRIVATE?

Answer: MAP_SHARED: writes are visible to other processes mapping the same file/object and are written through to the file. MAP_PRIVATE: copy-on-write — writes create private copies invisible to others and not written to file. MAP_ANONYMOUS | MAP_PRIVATE: anonymous memory (not backed by file) — basis for malloc's large allocations.

Q18: What does munmap() do?

Answer: munmap(addr, length) removes a memory mapping. Pages are returned to the OS. Any further access to the region causes SIGSEGV. Does not guarantee writes to MAP_SHARED mappings are flushed to disk — use msync() for that.

Q19: What is msync() and when is it needed?

Answer: msync(addr, length, flags) flushes mmap-ed file data back to disk. MS_SYNC — synchronous (waits for completion). MS_ASYNC — asynchronous (schedules flush). MS_INVALIDATE — invalidates other mappings. Required for durability of file-backed shared mappings.

Q20: What is fcntl() used for?

Answer: File control operations on open FDs:


Kernel vs User Space

Q21: What is the kernel/user space boundary?

Answer: User space: where processes run, restricted CPU mode (ring 3), cannot access hardware directly, isolated virtual address space. Kernel space: privileged CPU mode (ring 0), full hardware access, manages all processes. The boundary is crossed via: system calls, interrupts, exceptions, and signals. The CPU hardware enforces this via protection rings.

Q22: What is the kernel's role in memory management?

Answer: The kernel manages: page tables (virtual-to-physical mapping), physical page allocation (buddy allocator), page replacement (when RAM is full), copy-on-write for fork, demand paging (load pages on fault), memory-mapped files, swap. User space calls mmap/brk and the kernel manages the underlying physical resources.

Q23: What is the difference between kernel threads and user threads?

Answer: Kernel threads (e.g., kworker, ksoftirqd) run entirely in kernel space, have no user address space, and are scheduled by the kernel. User threads (pthreads on Linux) are implemented via clone(CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|...) — they are kernel-scheduled tasks sharing the parent's address space. Linux uses a 1:1 model (one kernel scheduling entity per user thread).

Q24: What is the Linux kernel preemption?

Answer: Kernel preemption (enabled in most kernels via CONFIG_PREEMPT) allows the kernel to preempt a task running in kernel mode (e.g., in a syscall) when a higher-priority task becomes runnable. Without preemption, a task can hold the CPU until it explicitly yields or exits the kernel. Low-latency kernels use PREEMPT_RT for near-hard-real-time behavior.

Q25: What is a kernel oops vs. a kernel panic?

Answer: A kernel oops is a non-fatal kernel error — the kernel logs the error (register dump, call trace) and continues (usually killing the offending process). A kernel panic is unrecoverable — the kernel halts, printing the panic message and optionally triggering kdump. panic_on_oops=1 in sysctl converts oops to panics.

Q26: What are kernel modules and how do they extend the kernel?

Answer: Kernel modules are loadable object files (.ko) that extend kernel functionality without rebooting — add device drivers, filesystems, networking protocols. Loaded with insmod/modprobe, unloaded with rmmod. They run in ring 0 with full kernel access. A buggy module can cause oops/panic.

Q27: What is kprobes and how is it used?

Answer: kprobes is a kernel instrumentation mechanism that allows setting breakpoints at arbitrary kernel addresses (functions or instructions). When hit, a handler runs. Used by strace alternatives, SystemTap, and bpftrace. No kernel recompilation needed. kretprobes intercept function returns.

Q28: What is the difference between user space and kernel address space on x86-64?

Answer: The 64-bit virtual address space is split: user space occupies the lower half (0 to ~128TB, 0x0 to 0x7FFFFFFFFFFFFFFF) and kernel space the upper half (0xFFFF800000000000 onwards). KPTI (Kernel Page Table Isolation, Meltdown mitigation) maps minimal kernel pages in user-space page tables to prevent speculative leakage.

Q29: What is CONFIG_KALLSYMS?

Answer: A kernel build option that includes all kernel symbol names in the kernel image, enabling human-readable call traces in oops/panics (instead of raw addresses). Exposed via /proc/kallsyms. Required for many profiling tools (perf, ftrace, kprobes).

Q30: What is KASLR (Kernel Address Space Layout Randomization)?

Answer: KASLR randomizes the load address of the kernel image at boot to prevent attackers from knowing kernel symbol addresses (needed for exploit techniques like ROP). Combined with ASLR for user space. Can be disabled at boot with nokaslr (needed for some debugging tools that use hardcoded addresses).


Context Switching

Q31: What is a context switch?

Answer: Saving the CPU state (registers, program counter, stack pointer, FPU state) of the currently running process and restoring the state of the next scheduled process. Triggered by: scheduler time slice expiry, blocking syscall, higher-priority wakeup, voluntary sched_yield(). The task_struct stores the context.

Q32: What does task_struct contain?

Answer: The kernel's per-process control block. Key fields: PID/TGID, state, priority/niceness, scheduler class, CPU affinity, signal handlers, credentials (uid/gid), file descriptor table, virtual memory (mm_struct), namespace pointers, cgroup membership, accounting statistics, and saved CPU registers (in embedded thread_struct).

Q33: What is the cost of a context switch?

Answer: Direct costs: save/restore registers (~100ns), switch page tables (if different process). Indirect costs: TLB flush (most expensive — all virtual-to-physical translations invalidated unless ASID/PCID support), CPU cache cold (next process's data not in L1/L2). PCID (Process Context ID) allows TLB tagging, reducing TLB flush cost on modern Intel CPUs.

Q34: What is a voluntary vs. involuntary context switch?

Answer: Voluntary (vcsw): process gives up CPU by making a blocking syscall or calling sched_yield(). Involuntary (ivcsw): kernel preempts the process (time slice expired or higher-priority task woke up). High involuntary context switches suggest the process needs more CPU than available. Check with pidstat -w or /proc/<pid>/status (VoluntaryCtxtSwitches, NonvoluntaryCtxtSwitches).

Q35: What is a TLB shootdown?

Answer: When a process modifies its page tables (unmap, remap), all CPUs that have cached that process's TLB entries must be notified to flush them (via IPI — Inter-Processor Interrupt). This is a TLB shootdown — expensive on multi-core systems with many CPUs. High rate: perf stat -e tlb:tlb_flush.

Q36: What is the run queue and how does the scheduler use it?

Answer: The run queue is the per-CPU data structure holding all runnable tasks (state = TASK_RUNNING). CFS uses a red-black tree ordered by vruntime. The scheduler picks the leftmost node (smallest vruntime = least CPU time received). Real-time queues are separate priority-ordered lists.

Q37: What is schedule() in the kernel?

Answer: schedule() is the kernel function that performs a context switch. It: selects the next task (calls pick_next_task() on active scheduling class), calls context_switch() to save/restore state, and returns in the context of the new task. Called from: schedule() explicit calls in blocking paths, and from the tick interrupt when preemption is needed.

Q38: What is a spinlock vs. a mutex in the kernel?

Answer: Spinlock: busy-waits (spins) in a tight loop until the lock is available. Cannot sleep while holding. For very short critical sections in interrupt context where sleeping is forbidden. Mutex: blocks (sleeps) the thread if the lock is unavailable, allowing the CPU to run other tasks. For longer critical sections in process context. Spinlocks avoid context switch overhead but waste CPU; mutexes are CPU-efficient for longer waits.

Q39: What is RCU (Read-Copy-Update)?

Answer: RCU is a synchronization mechanism optimized for read-heavy, rarely-written shared data. Readers proceed without locks (zero overhead). Writers: make a new version of the data, update the pointer atomically, then wait for all current readers to finish a grace period before freeing the old version. Used extensively in the Linux kernel for routing tables, network configuration, process lists.

Q40: What is the softirq and tasklet in the Linux kernel?

Answer: softirq (software interrupt): deferred processing units that run in interrupt context after hardware IRQs, but can run in parallel across CPUs. Fixed set (NET_RX, NET_TX, TIMER, etc.). tasklet: built on softirq, runs serialized (same tasklet can't run on two CPUs simultaneously), used for simpler deferred work. Both cannot sleep. For sleepable deferred work: work queues.


Virtual Memory & Paging

Q41: How does the kernel implement virtual memory?

Answer: Each process has an mm_struct containing a list of VMAs (Virtual Memory Areas, vm_area_struct), and a page table root (PGD). The CPU MMU uses multi-level page tables (4-level on x86-64: PGD→PUD→PMD→PTE) to translate virtual addresses to physical frames. The kernel manages page allocation, swapping, and COW.

Q42: What is a VMA (Virtual Memory Area)?

Answer: vm_area_struct describes a contiguous region of a process's virtual address space with uniform properties (permissions, backing store). Each mmap() call, stack growth, or heap expansion creates/extends a VMA. cat /proc/<pid>/maps shows all VMAs. The kernel searches VMAs on page faults to determine if an access is valid.

Q43: What is demand paging?

Answer: Pages are not loaded into physical memory until accessed — memory is allocated lazily. When a process accesses a not-yet-present page, a page fault occurs, and the kernel loads the page from its backing store (file or swap) or allocates a zero page. This makes program startup fast and allows address spaces larger than RAM.

Q44: What is the page cache?

Answer: The kernel's cache of disk file data in RAM. When a file is read, data is placed in the page cache (mapped to specific file+offset). Subsequent reads return cached data. Writes to regular files go to cache first (dirty pages), written back by pdflush/writeback kernel threads or on sync/fsync. cat /proc/meminfo | grep -E "Cached|Buffers".

Q45: What is the difference between anonymous and file-backed pages?

Answer: File-backed: backed by a file (executable text, mmap'd files, shared libraries). Can be evicted and re-read from disk. Anonymous: no file backing (heap, stack, BSS, MAP_ANONYMOUS). Cannot be re-read from disk — must go to swap if evicted. Anonymous dirty pages are far more expensive to evict.

Q46: What is swap and when does the kernel use it?

Answer: Swap provides disk-backed storage for evicted anonymous pages. The kernel swaps out pages when physical RAM is low (via kswapd or direct reclaim). Pages are chosen by LRU lists (active/inactive anon and file lists). Swap-heavy workloads suffer performance degradation (disk I/O vs. RAM access).

Q47: What is kswapd?

Answer: kswapd is a kernel thread (one per NUMA node) that asynchronously frees pages when memory gets low (below the pages_high watermark). It scans LRU lists, writes dirty pages, and frees clean ones. If it can't keep up, direct reclaim happens synchronously in the allocation path, causing latency spikes.

Q48: What is the vm.swappiness parameter?

Answer: Sysctl tunable (0–200, default 60) that controls the kernel's tendency to swap anonymous pages vs. evicting file cache. 0 = prefer evicting page cache, avoid swap. 100 = balance. 200 = aggressively swap anonymous pages. For latency-sensitive workloads: set to 10 or 0. Databases often set it low.

Q49: What is madvise() and its key hints?

Q50: What is posix_fadvise()?

Q51: What is huge pages and how do they affect page faults?

Answer: 2MB (or 1GB) huge pages vs. default 4KB pages. Fewer TLB entries needed for same amount of memory → fewer TLB misses. With huge pages, a 2MB allocation requires 1 TLB entry vs 512. Page fault cost is similar per fault but faults are far less frequent. mmap(MAP_HUGETLB) or Transparent Huge Pages (THP) provide these automatically.

Q52: What is MAP_POPULATE in mmap()?

Answer: When MAP_POPULATE is set, mmap() pre-faults all pages in the mapping (populates page tables) before returning. Eliminates subsequent page fault overhead at the cost of upfront latency and memory use. Useful for mappings that will be immediately accessed fully (e.g., reading a file into memory).

Q53: What is MADV_HUGEPAGE vs THP enabled globally?

Answer: THP (Transparent Huge Pages) can be set system-wide: always (promote all eligible), madvise (only for regions with MADV_HUGEPAGE), or never. Per-region control via madvise is preferred for databases/latency-sensitive apps as THP promotions cause occasional latency spikes during compaction. Set via /sys/kernel/mm/transparent_hugepage/enabled.

Q54: What is a page frame number (PFN) and how do you find it?

Answer: A PFN identifies a physical memory page (physical_addr / PAGE_SIZE). /proc/<pid>/pagemap maps each virtual page to its PFN (or indicates swap). Useful for: DMA buffer management, NUMA analysis, and understanding physical memory layout.

Q55: What happens to memory after free() in C?

Answer: free() returns the block to malloc's internal freelist — the pages remain in the process's address space (still in RSS). The kernel only reclaims physical pages if malloc calls munmap() (for large blocks) or brk() to shrink the heap. For small blocks, pages stay until the process exits or the OS reclaims them under memory pressure.


Inter-Process Communication (IPC)

Q56: What are the main IPC mechanisms in Linux?

Answer:

  1. Pipes — anonymous, unidirectional, kernel FIFO, between related processes.

  2. Named pipes (FIFOs) — filesystem entry, between unrelated processes.

  3. Signals — asynchronous notification.

  4. POSIX message queues — typed messages with priorities.

  5. SysV message queues — older message API.

  6. Shared memory — fastest IPC, requires synchronization (semaphores/mutexes).

  7. POSIX semaphores — process synchronization.

  8. SysV semaphores — older semaphore API.

  9. Unix domain sockets — bidirectional, stream or datagram.

  10. Netlink — kernel↔userspace communication.

  11. D-Bus — message bus for desktop/system services.

Q57: How does a pipe work?

Answer: A pipe is a kernel-managed circular buffer (default 64KB on Linux). pipe(fd[2]) creates two FDs: fd[0] (read) and fd[1] (write). Data written to fd[1] is buffered in the kernel until read from fd[0]. Writes block when full; reads block when empty. Atomicity: writes ≤ PIPE_BUF (4096) bytes are atomic.

Q58: What is POSIX shared memory?

Fastest IPC — processes share physical pages directly; no kernel copy in data path.

Q59: What is the difference between SysV shared memory and POSIX shared memory?

Answer: SysV (shmget/shmat/shmdt/shmctl): older API, IPC key-based, persistent until explicitly deleted or reboot, visible in ipcs -m. POSIX (shm_open/mmap): file-descriptor-based, appears in /dev/shm/, more POSIX portable, easier permissions model. Prefer POSIX for new code.

Q60: How do you use POSIX semaphores for IPC?

Q61: What is the difference between a semaphore and a mutex?

Answer: A mutex (mutual exclusion lock) is owned by the thread that locked it — only that thread can unlock it. Binary (0/1). Used for protecting critical sections. A semaphore is a counter — any thread can increment it; used for signaling and resource counting. Semaphores don't have ownership; mutexes do. Priority inversion protocols (PI) apply to mutexes.

Q62: What is a POSIX message queue?

Messages have priorities; higher-priority messages are delivered first. Bounded queue size. Non-blocking via O_NONBLOCK. Notifications via mq_notify().

Q63: What is a Unix domain socket and when would you use it over a pipe?

Answer: Unix domain sockets support: bidirectional communication (pipes are one-way), datagram mode (message boundaries preserved), passing open FDs (SCM_RIGHTS), credentials passing (SCM_CREDENTIALS), and connection to unrelated processes (by path). Use over pipes when you need full duplex, message-oriented protocol, or FD passing.

Q64: How do you pass file descriptors between processes?

Answer: Via Unix domain sockets using SCM_RIGHTS ancillary data:

Used by privilege separation daemons, containers, and sandboxes.

Q65: What is a futex (fast userspace mutex)?

Answer: A futex is a kernel primitive for efficient synchronization. When uncontested, lock/unlock happens entirely in user space (atomic operations on a shared integer) — no syscall. Only when contention occurs (futex() syscall with FUTEX_WAIT) does the thread block in the kernel. pthreads mutexes, condition variables, and semaphores are built on futexes.

Q66: What is FUTEX_WAIT and FUTEX_WAKE?

Answer: futex(addr, FUTEX_WAIT, expected, timeout) — if *addr == expected, put the calling thread to sleep. futex(addr, FUTEX_WAKE, n) — wake up to n threads waiting on addr. The atomic check-and-sleep prevents lost wakeups. Core building block for all pthreads synchronization primitives.

Q67: What is a condition variable and how does it work?

Answer: A condition variable allows threads to wait for a condition to become true. pthread_cond_wait(cond, mutex): atomically releases the mutex and blocks. When pthread_cond_signal(cond) or pthread_cond_broadcast(cond) is called, waiting threads are woken and reacquire the mutex. Always check the condition in a while loop (spurious wakeups possible).

Q68: What is eventfd() and how is it used for IPC?

Lightweight in-kernel counter for signaling between processes/threads. Integrates with epoll/select — used in async I/O notification (libuv, Java NIO).

Q69: What is signalfd()?

Delivers signals as file descriptor reads — allows signals to be handled via epoll in event loops, avoiding signal handler complexity.

Q70: What is timerfd_create()?

Creates a timer that delivers expirations as file descriptor reads. Integrates with epoll for event-driven timer handling.


Scheduling Internals

Q71: How does the Linux CFS scheduler calculate vruntime?

Answer: vruntime += actual_run_time * (NICE_0_WEIGHT / task_weight). Higher-priority (lower nice) tasks have larger task_weight, so they accumulate vruntime more slowly — they appear to have less runtime proportionally and are picked more often. The task with the smallest vruntime in the red-black tree is always picked next.

Q72: What is the scheduler's tick and how often does it fire?

Answer: The scheduler tick (via CONFIG_HZ, typically 250Hz or 1000Hz) runs scheduler_tick() which updates vruntime, checks if preemption is needed, and updates load statistics. CONFIG_HZ=250 means 4ms tick. CONFIG_NO_HZ (tickless mode) disables idle ticks, improving power efficiency and reducing latency jitter.

Q73: What is sched_setaffinity() and why use it?

Pinning a process to specific CPUs improves cache locality (hot L1/L2 cache), reduces TLB shootdowns, and enables NUMA optimization. Used for real-time tasks and high-performance services.

Q74: What is the SCHED_DEADLINE policy and how does it work?

Answer: SCHED_DEADLINE implements the Earliest Deadline First (EDF) algorithm. Tasks specify: runtime (compute time per period), deadline (relative), and period. The scheduler guarantees each task receives at least runtime nanoseconds before its deadline. Uses CBS (Constant Bandwidth Server) to prevent overrun. Most stringent RT guarantee in Linux.

Q75: What is sched_yield() and when should you use it?

Answer: sched_yield() voluntarily gives up the CPU, allowing other runnable tasks to run. The calling task is moved to the end of its priority queue. Rarely needed — usually indicates a design problem (busy-waiting). Better alternatives: proper blocking primitives (futex, condition variable, epoll).

Q76: What is the difference between O(1) scheduler and CFS?

Answer: The older O(1) scheduler (pre-2.6.23) maintained run queues indexed by priority, giving O(1) pick-next but had interactive performance issues (heuristics for distinguishing interactive vs. batch tasks). CFS uses the red-black tree with vruntime — O(log N) pick-next but perfectly fair and no heuristics needed. CFS handles interactive performance naturally via vruntime lag.

Q77: What is load balancing in the Linux scheduler?

Answer: CFS periodically (and on CPU idle) balances the run queues across CPUs to ensure fairness. load_balance() migrates tasks from overloaded to underloaded CPUs. Considers: CPU affinity, cache topology (SMT siblings, LLC domains, NUMA nodes). Migration cost increases with NUMA distance.

Q78: What is NUMA balancing in Linux?

Answer: Automatic NUMA Balancing (controlled by kernel.numa_balancing sysctl) periodically unmaps process pages and re-faults them to detect which NUMA node accesses them. It then migrates pages to the node where the accessing thread runs, improving memory locality. Can cause latency spikes from the fault-and-migrate process.

Q79: What are scheduling domains and groups?

Answer: The scheduler organizes CPUs into a hierarchy of scheduling domains reflecting hardware topology: SMT siblings (cores sharing an L1 cache) → CPU cores → NUMA nodes. Load balancing within a domain considers the topology's migration cost. This lets the scheduler prefer keeping tasks close to their data.

Q80: What is isolcpus and how is it used?

Answer: isolcpus=2,3 kernel boot parameter isolates CPUs 2 and 3 from the scheduler — no tasks are automatically scheduled there. To run on isolated CPUs: taskset -c 2 ./prog. Used for real-time tasks, DPDK, or latency-sensitive applications that need exclusive CPU access.


Interrupts & Signals

Q81: What is a hardware interrupt (IRQ)?

Answer: A hardware interrupt is an asynchronous signal from a hardware device (network card, disk controller, timer) to the CPU, requesting attention. The CPU saves its current state and executes the registered interrupt handler (ISR). Linux splits IRQ handling into: top half (ISR — fast, runs with interrupts disabled) and bottom half (softirq/tasklet — deferred, runs with interrupts enabled).

Q82: What is an interrupt service routine (ISR) and what constraints does it have?

Answer: An ISR runs in interrupt context: no process context, cannot sleep, cannot call blocking functions, must be fast (< few microseconds). It should acknowledge the hardware and schedule deferred work (bottom half) for slower processing. Accesses shared data with spinlocks (not mutexes).

Q83: What is interrupt coalescing (NAPI in networking)?

Answer: NAPI (New API) for network cards: instead of an interrupt per packet (high-rate bombardment), receive one interrupt, then poll the NIC in a softirq to drain the receive queue. After draining, re-enable interrupts. Reduces interrupt overhead at high traffic rates. ethtool -C eth0 rx-usecs 50 configures coalescing in the NIC hardware.

Q84: What is IRQ affinity and how do you set it?

Pinning IRQs to specific CPUs reduces cache bouncing and enables RSS/RPS network scaling.

Q85: What is the difference between hardware IRQ and software IRQ (softirq)?

Answer: Hardware IRQ is triggered by physical devices, running in hardware interrupt context. Softirq is a deferred mechanism that runs after hardware IRQs, in interrupt context but with interrupts re-enabled. There are 10 fixed softirq types (HI, TIMER, NET_TX, NET_RX, BLOCK, etc.). Each CPU has its own ksoftirqd thread for soft IRQ overload.

Q86: What is ksoftirqd?

Answer: A per-CPU kernel thread that handles softirq processing when softirqs are generated faster than they can be processed inline (after hard IRQ handlers or local_bh_enable()). If too many softirqs are pending, ksoftirqd runs to prevent monopolizing interrupt context. High ksoftirqd CPU use = high softirq load (often network receive).

Q87: What is the difference between kill() and raise() system calls?

Answer: kill(pid, sig) sends a signal to another process (or process group if pid=0, or all processes if pid=-1). raise(sig) sends a signal to the calling process (equivalent to kill(getpid(), sig)). Also: tkill(tid, sig) sends to a specific thread; tgkill(tgid, tid, sig) does so with safety check.

Q88: What is sigaction() vs signal()?

Answer: signal() is simple but has implementation-defined behavior (SA_RESTART, signal mask during handler). sigaction() provides full control: SA_RESTART (auto-restart interrupted syscalls), SA_SIGINFO (receive siginfo_t with sender PID, etc.), SA_NODEFER (don't block signal in handler), custom signal mask during handler. Always use sigaction() for robust signal handling.

Q89: What is async-signal safety?

Answer: Only async-signal-safe functions can be called from signal handlers. These are functions that are reentrant or don't use non-reentrant resources. Safe: write(), _exit(), sigprocmask(), simple arithmetic. Unsafe: printf(), malloc(), free(), mutex_lock(). Using unsafe functions in handlers causes undefined behavior (deadlock, corruption).

Q90: What is the real-time signal queue size limit?

RT signals queue (multiple signals of same number are NOT merged). Queue depth bounded by RLIMIT_SIGPENDING. If full, sigqueue() returns EAGAIN.


Namespaces & Cgroups

Q91: What Linux namespaces exist and what do they isolate?

NamespaceFlagIsolates
Mount (mnt)CLONE_NEWNSFilesystem mount tree
UTSCLONE_NEWUTSHostname, NIS domain
IPCCLONE_NEWIPCSysV IPC, POSIX MQs
PIDCLONE_NEWPIDProcess IDs
NetworkCLONE_NEWNETNetwork interfaces, routing, firewall
UserCLONE_NEWUSERUIDs/GIDs (UID 0 in namespace = unprivileged outside)
CgroupCLONE_NEWCGROUPcgroup root view
TimeCLONE_NEWTIMEClock offsets

Q92: How are namespaces created?

Or from shell: unshare --pid --mount --net bash.

Q93: What is a PID namespace?

Answer: A PID namespace gives processes an independent PID numbering. Inside the namespace, processes see PIDs starting from 1. PID 1 in the namespace is the namespace's init — if it dies, all processes in the namespace receive SIGKILL. Nested PID namespaces: processes have different PIDs in each ancestor namespace. docker run creates a new PID namespace.

Q94: What is a network namespace?

Answer: A network namespace gives processes their own network stack: independent interfaces, routing tables, iptables rules, sockets, and ports. Used by containers for network isolation. ip netns add myns, ip link set eth0 netns myns. Containers connect namespaces via veth (virtual ethernet) pairs.

Q95: What are cgroups and what can they limit?

Answer: cgroups organize processes into hierarchical groups with resource limits and accounting:

Q96: What is the difference between cgroup v1 and v2?

Answer: v1: each subsystem has its own hierarchy; a process can be in different groups for different subsystems — complex and inconsistent. v2 (unified hierarchy): single hierarchy tree, all controllers applied together, better delegation to non-root. systemd uses v2 on modern distros. /sys/fs/cgroup/ shows v2; previously separate directories.

Q97: How does Docker use namespaces and cgroups?

Answer: Docker creates a container by:

  1. Namespaces: clone() with CLONE_NEWPID | CLONE_NEWNET | CLONE_NEWNS | CLONE_NEWUTS | CLONE_NEWIPC | CLONE_NEWUSER flags.

  2. cgroups: creates cgroup entries under /sys/fs/cgroup/ for CPU/memory/blkio limits.

  3. Filesystem: overlayfs for layered filesystem; pivot_root() to switch root.

  4. Capabilities: drops most Linux capabilities (CAP_NET_RAW, etc.).

  5. seccomp: applies syscall filter profile.

Q98: What are Linux capabilities?

Answer: Capabilities split the all-or-nothing root privilege into granular permissions: CAP_NET_BIND_SERVICE (bind ports < 1024), CAP_SYS_PTRACE (ptrace), CAP_NET_ADMIN (network config), CAP_SYS_ADMIN (many admin ops), CAP_KILL (send signals to any process), etc. Allows services to run as non-root with only needed privileges.

Q99: What is unshare and nsenter?

unshare creates new namespaces for the calling process. nsenter joins existing namespaces.

Q100: What is an overlay filesystem and how does Docker use it?

Answer: OverlayFS merges multiple directory layers into a single view. A container has: immutable lower layers (image layers) and a upper (writable container layer). Reads check upper first, then lower. Writes go to upper (copy-on-write). Deletes create whiteout files. Efficient: image layers are shared across containers.


Advanced System Internals

Q101: What is the difference between fork() and vfork()?

Answer: fork() creates a full copy-on-write address space clone. vfork() creates a child that shares the parent's address space — parent is suspended until the child calls exec() or _exit(). vfork() is faster (no page table clone) but dangerous (child must not return from the function that called vfork or modify any data). Largely obsolete — posix_spawn() is preferred.

Q102: What is clone() and how does it differ from fork()?

Answer: clone() is the underlying Linux syscall for both fork() and pthread_create(). It takes flags controlling which resources are shared: CLONE_VM (share address space → thread), CLONE_FILES (share FD table), CLONE_SIGHAND (share signal handlers). fork() = clone(SIGCHLD). pthread_create() = clone(CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|...).

Q103: What is execve() and what happens during it?

Answer: execve(path, argv, envp) replaces the current process image. Kernel actions: open the ELF binary, verify headers, map text/data/BSS segments, set up stack with argv/envp/auxv, map vDSO and ld.so (dynamic linker), jump to dynamic linker entry point (which resolves shared libraries), then jump to _start in the program. PID, open FDs (without FD_CLOEXEC), signal handlers reset to default.

Q104: What is ELF and what are its components?

Answer: ELF (Executable and Linkable Format) is the standard binary format. Components:

Q105: What is the PLT/GOT (Procedure Linkage Table/Global Offset Table)?

Answer: Lazy dynamic linking mechanism. First call to printf: jumps to PLT stub → GOT entry (initially points back to PLT resolver) → dynamic linker resolves printf's address, stores in GOT. Subsequent calls: PLT stub → GOT (now holds real address) → printf directly. RELRO (-Wl,-z,relro,-z,now) makes GOT read-only after startup for security.

Q106: What is ASLR and how does it work?

Answer: Address Space Layout Randomization randomizes the base addresses of the stack, heap, and memory-mapped regions (libraries) at program start. Prevents attackers from hardcoding addresses for exploits. Controlled by /proc/sys/kernel/randomize_va_space: 0=off, 1=stack/libs/mmap, 2=also heap. Compile-time PIE (-fPIE -pie) required for executable ASLR.

Q107: What is the difference between stack frame and stack pointer?

Answer: The stack pointer (RSP/SP) points to the current top of the stack. A stack frame (or activation record) is the region of the stack allocated for a function call — contains: local variables, saved registers, return address, function arguments (some in registers on x86-64). The frame pointer (RBP/BP) tracks the base of the current frame (may be omitted with -fomit-frame-pointer).

Q108: What is a return-oriented programming (ROP) attack?

Answer: ROP is an exploit technique that chains existing code snippets ("gadgets" ending in ret) in memory to execute arbitrary logic without injecting new code, bypassing NX (no-execute) protections. Mitigations: ASLR (randomizes gadget addresses), stack canaries (detect overflows), shadow stack (CET), CFI (control flow integrity).

Q109: What are stack canaries?

Answer: A random value placed on the stack between local variables and the return address at function entry, checked at function exit. If a buffer overflow overwrites the canary, the check fails and the program is terminated (SIGABRT). Enabled with -fstack-protector or -fstack-protector-all. Does not protect against precise overwrites or non-stack vulnerabilities.

Q110: What is the kernel's slab allocator?

Answer: The SLUB allocator (default since 2.6.22) manages kernel memory for fixed-size objects (inodes, dentries, sk_buff, etc.) using per-CPU caches to minimize locking. Objects of the same size share slabs (groups of pages). Reduces fragmentation and allocation overhead. Exposed via /proc/slabinfo and slabtop.

Q111: What is kmalloc() vs vmalloc() in the kernel?

Answer: kmalloc(size, GFP_KERNEL) allocates physically and virtually contiguous memory from the slab allocator — fast, suitable for DMA. vmalloc(size) allocates virtually contiguous but physically non-contiguous pages — used for large allocations where physical contiguity is not required. vmalloc has overhead: extra page table entries, TLB pressure.

Q112: What is GFP flags in kernel memory allocation?

Answer: GFP_KERNEL — can sleep, normal allocation. GFP_ATOMIC — cannot sleep (used in interrupt context). GFP_DMA — allocate in DMA-accessible zone. GFP_HIGHUSER — user-space page allocation. GFP_NOWAIT — return NULL if can't immediately satisfy. Using wrong GFP flags (e.g., GFP_KERNEL in interrupt context) is a kernel bug.

Q113: What is a memory barrier and why is it needed?

Answer: CPUs and compilers reorder memory operations for performance. Memory barriers (smp_mb(), smp_rmb(), smp_wmb(), C11 atomics) prevent reordering across the barrier. Needed when: sharing data between CPUs without locks, implementing lock-free data structures, communicating with hardware (DMA). Without barriers, the other CPU may see stores in a different order.

Q114: What is the difference between __builtin_expect and branch prediction?

Answer: Modern CPUs predict branch outcomes; misprediction costs ~15 cycles. __builtin_expect(cond, expected) hints to the compiler about likely vs. unlikely paths, enabling better code layout:

Q115: What is a CPU cache line and why does it matter for programming?

Answer: Cache lines are the unit of cache memory (typically 64 bytes on x86). Any access to any byte loads the entire 64-byte line. False sharing: two threads modifying different variables in the same cache line causes cache line ping-pong between CPUs → severe performance degradation. Fix: align shared variables to cache line boundaries (__attribute__((aligned(64))), __cacheline_aligned).

Q116: What is __attribute__((packed)) and what are its implications?

Answer: Forces the compiler to pack a struct with no padding between members — reduces size but may cause misaligned access on strict-alignment architectures (ARM, SPARC), leading to performance penalties or SIGBUS. Useful for network protocol headers or hardware register maps.

Q117: What is DMA (Direct Memory Access) and how does it work?

Answer: DMA allows devices (NIC, disk controller) to transfer data directly to/from main memory without CPU involvement. The driver: allocates a DMA buffer (physically contiguous, sometimes with IOMMU mapping), gives the device the physical address, and the device writes directly to RAM. On completion, the device raises an interrupt. Requires cache coherency handling (dma_map_* functions in the kernel manage this).

Q118: What is IOMMU and why does it matter?

Answer: IOMMU (I/O Memory Management Unit) provides virtual-to-physical address translation for DMA-capable devices, similar to how the CPU's MMU works for processes. Benefits: prevents DMA from accessing unauthorized memory (security/isolation), allows non-contiguous physical memory for DMA, enables virtualization (devices see guest physical addresses). Intel VT-d, AMD-Vi.

Q119: What is mlock() and why use it?

Used by: real-time applications (prevent latency from page faults), security-sensitive code (prevent keys from reaching swap/disk), databases. Requires CAP_IPC_LOCK or setting RLIMIT_MEMLOCK.

Q120: What is perf_event_open() used for?

Answer: The syscall for the perf subsystem. Allows user space to access hardware performance counters (cycles, instructions, cache misses, branch misses), software events (context switches, faults), tracepoints, and kprobes. Used by perf, bpftrace (for hardware events), and custom profiling tools. Returns a file descriptor for reading counts or setting up overflow notifications.

Q121: What is the O_DIRECT flag in file I/O?

Answer: open(file, O_DIRECT) bypasses the kernel page cache — I/O goes directly between user buffer and disk. Requires: aligned buffer (posix_memalign), aligned offset, aligned size (multiple of block size). Used by databases (PostgreSQL, MySQL) that manage their own cache and don't want kernel cache double-buffering.

Q122: What is O_SYNC and O_DSYNC?

Answer: O_SYNC: each write() blocks until data and metadata are written to the physical device (equivalent to fsync() after each write). O_DSYNC: blocks until data is written but not necessarily metadata (equivalent to fdatasync()). Provides durability at the cost of latency. Used for journals and WALs.

Q123: What is epoll's edge-triggered vs. level-triggered mode?

Answer: Level-triggered (LT, default): epoll_wait returns the FD as ready as long as data is available — safe, easy, but may return the same FD multiple times. Edge-triggered (ET): returns once per new data arrival — must drain completely each time; miss it and you won't be notified again. ET requires non-blocking FDs and careful implementation but is more efficient for high-performance servers.

Q124: What is SO_KEEPALIVE and how does TCP keepalive work?

Answer: SO_KEEPALIVE enables TCP keepalive probes — after idle for tcp_keepalive_time seconds (default 2h), the kernel sends probe packets at tcp_keepalive_intvl intervals (default 75s), up to tcp_keepalive_probes times (default 9). If no response, the connection is closed. Used to detect dead peers and prevent firewall/NAT from expiring idle connections.

Q125: What is TCP_NODELAY and when to use it?

Answer: By default, TCP uses Nagle's algorithm — buffers small writes and combines them into larger segments to reduce packet count. TCP_NODELAY disables Nagle's algorithm — each write is sent immediately. Needed for low-latency request-response protocols (interactive SSH, game servers, financial trading). Default: enabled (Nagle on), which improves throughput for bulk transfers.

Q126: What is SO_REUSEPORT and why is it useful?

Answer: Multiple sockets can bind to the same IP:port. The kernel load-balances incoming connections/packets across them using consistent hashing. Enables: multiple processes/threads to accept without a common accept loop, eliminates the "thundering herd" on accept(), better multi-core utilization. Used by Nginx worker processes, HAProxy.

Q127: What is the difference between SOCK_STREAM and SOCK_SEQPACKET?

Answer: SOCK_STREAM (TCP, Unix): reliable, ordered, byte-stream — no message boundaries. Reads may return partial messages. SOCK_SEQPACKET (Unix domain, SCTP): reliable, ordered, with message boundaries preserved — each write is a discrete message, read returns exactly one message. Avoids manual framing protocol for Unix domain IPC.

Q128: What is SCTP (Stream Control Transmission Protocol)?

Answer: SCTP provides: reliable, ordered delivery like TCP, but with message framing (no need for length-prefixing), multi-homing (multiple network paths), multi-streaming (independent streams within one connection, avoiding head-of-line blocking), and four-way handshake. Used in telecom signaling (SS7/Diameter), some clustering protocols.

Q129: What is recv() vs read() on a socket?

Answer: Both work on sockets but recv(fd, buf, len, flags) has additional flags: MSG_PEEK (read without consuming), MSG_WAITALL (block until buffer full), MSG_DONTWAIT (non-blocking), MSG_TRUNC (return actual size even if truncated). read(fd, buf, len) is equivalent to recv(fd, buf, len, 0).

Q130: What is TCP_CORK and how does it work?

Answer: TCP_CORK (Linux) accumulates data in the send buffer until either the MSS is reached or TCP_CORK is cleared. Used to build complete response headers+body before sending, avoiding Nagle algorithm limitations for chunked writes. Example: write HTTP headers, cork, write body, uncork → single segment.

Q131: What is SO_LINGER and what is it used for?

Answer: By default, close() on a socket returns immediately (kernel handles remaining data). SO_LINGER with l_onoff=1, l_linger=N makes close() block up to N seconds waiting for data to be sent and acknowledged, or resets the connection if N=0. Used when you need to ensure data delivery before the socket is fully closed.

Q132: What is inotify and how does it work?

inotify lets processes watch filesystem events efficiently without polling. Up to fs.inotify.max_user_watches watches per user.

Q133: What is fanotify?

Answer: fanotify is a more powerful filesystem notification API (requires CAP_SYS_ADMIN). Unlike inotify: can watch entire mounts, receive pre-event access decisions (approve/deny file opens — used by antivirus scanners, access control). Used by security tools (ClamAV, auditd, container security).

Q134: What is landlock (Linux 5.13+)?

Answer: Landlock is a security module that allows unprivileged processes to restrict their own access to the filesystem. Unlike seccomp (syscall level), landlock operates at the VFS layer. Useful for sandboxing: a process can restrict itself to only specific directories/file types, even without root. Used in browsers and sandboxed applications.

Q135: What are cgroups freezer and cgroups devices controllers?

Answer: Freezer: suspends and resumes all tasks in a cgroup simultaneously (SIGSTOP/SIGCONT equivalent for groups). Used by container checkpoint/restore (CRIU). Devices: controls which device files (by major:minor, type c/b) tasks in a cgroup can open. Used to restrict containers to only allowed devices.

Q136: What is CRIU (Checkpoint/Restore In Userspace)?

Answer: CRIU dumps and restores the complete state of running processes (memory, FDs, network connections, process tree). The checkpoint creates a set of image files; restore re-creates the exact process state on any compatible system. Used for: container migration, live migration, fast startup (restore vs. cold start).

Q137: What is memfd_create() and what is it used for?

Creates an anonymous file backed by memory. Unlike shm_open: no filesystem visibility by default, supports sealing (make read-only/immovable after writing). Used for secure IPC (pass fd via SCM_RIGHTS), JIT code buffers, and container image loading.

Q138: What is pidfd_open() (Linux 5.3+)?

A file descriptor representing a process — race-free PID handling (PID reuse races are eliminated). Can be passed between processes. Integrates with epoll to watch for process exit.

Q139: What is eBPF from a kernel perspective?

Answer: eBPF is a safe in-kernel virtual machine: user submits a BPF bytecode program → kernel verifier checks for safety (no loops, no invalid accesses, bounded execution) → JIT-compiled to native code → attached to kernel hooks (kprobes, tracepoints, XDP, etc.). Programs communicate via BPF maps (hash, array, ring buffer). No kernel module required, no stability risk (verifier enforces safety).

Q140: What is XDP (eXpress Data Path)?

Answer: XDP allows eBPF programs to run at the network driver receive path — before the kernel network stack processes the packet. Enables: ultra-high-performance packet processing (line-rate), DDoS mitigation, load balancing (Facebook Katran), software firewalls. Actions: XDP_PASS (normal stack), XDP_DROP (drop), XDP_TX (bounce back), XDP_REDIRECT (redirect to another NIC/CPU queue).

Q141: What is a network socket buffer (sk_buff)?

Answer: sk_buff (socket buffer) is the kernel's central network packet structure. Contains: packet data (via DMA to NIC), protocol headers, metadata (timestamps, checksums, routing info). Passed between network stack layers by adjusting head/tail pointers (no data copy). Memory layout allows prepending/appending headers efficiently.

Q142: What are netfilter hooks?

Answer: Netfilter hooks are points in the kernel's network stack where packet processing modules can register. Hooks: PREROUTING → INPUT → FORWARD → OUTPUT → POSTROUTING. iptables/nftables use these hooks. Each hook can: accept, drop, modify, or queue packets to user space (NFQUEUE).

Q143: What is the difference between process address space and kernel address space?

Answer: Each process has its own virtual address space (isolated by page tables). The kernel address space is shared across all processes (same physical mapping for all page tables). When a syscall executes, the same kernel code/data is mapped in the current process's page tables (or in a separate PGD with KPTI). Kernel sees all physical memory; a process sees only its own virtual pages.

Q144: What is prctl() used for?

Q145: What is the vsyscall region?

Answer: vsyscall (older mechanism, pre-vDSO) mapped a fixed virtual address (0xFFFFFFFFFF600000) with commonly used syscall implementations. Fixed address was a security issue (bypasses ASLR). Modern Linux uses vDSO (randomized mapping) instead. vsyscall=emulate kernel parameter makes vsyscall page fault through the kernel for security.

Q146: What is copy_to_user() / copy_from_user() in the kernel?

Answer: Kernel functions that safely copy data between kernel and user address spaces. They check that user pointers are valid (not kernel addresses, within user address range) and handle faults gracefully (instead of crashing the kernel). Using plain dereference on user pointers is a kernel bug — use these functions. Returns bytes not copied on error.

Q147: What is the difference between wait4() and waitid()?

Answer: wait4(pid, status, options, rusage) waits for a child, optionally filling resource usage. waitid(idtype, id, infop, options) is more flexible: can wait on PID, PGID, or pidfd; uses siginfo_t for richer exit info (exit code, signal, CPU times). WNOWAIT option lets you inspect exit status without reaping. waitid with P_PIDFD is the modern race-free interface.

Q148: What is getrandom() syscall?

Direct cryptographic random bytes from the kernel. Preferred over /dev/urandom (avoids open+read+close overhead) and safer (available before filesystem, works in chroot/container).

Q149: What is renameat2() with RENAME_ATOMIC?

Answer: Standard rename() atomically replaces the destination. renameat2() with RENAME_EXCHANGE atomically swaps two files (both paths remain valid with exchanged content). RENAME_NOREPLACE fails if destination exists (instead of overwriting). Used for atomic configuration updates (swap new config into place without a window where neither exists).

Q150: What is the clone3() syscall and what improvements does it bring?

Answer: clone3() (Linux 5.3+) is an extensible replacement for clone(). It takes a struct instead of individual arguments, allowing future fields without a new syscall. Added features: set_tid (specify desired PID in namespace), pidfd (return a pidfd for the child), cgroup (place child in a cgroup atomically). Enables race-free process creation with full resource control.