Inside the OOM Killer: How Linux Decides What to Kill When Memory Runs Out

Linux out-of-memory handling is not one global alarm that always kills the largest process. An allocation can fail in a constrained cgroup, NUMA memory-policy domain, cpuset, or the whole system. The kernel considers reclaim, allocation context, eligibility, and configured policy before selecting a victim; userspace managers may act earlier under sustained pressure.

The killed task is evidence about one decision, not proof of the original memory leak or capacity mistake.

Define the allocation domain first

A global OOM means the allocator could not satisfy an eligible allocation after reclaim within the relevant system constraints. A cgroup OOM occurs when a memory cgroup cannot reduce usage below memory.max. A cpuset or memory-policy restriction can produce an OOM despite free memory elsewhere that the allocation is not permitted to use.

Capture the complete kernel message. Phrases identifying constraint=, mems_allowed, task/cgroup, allocation order/mask, and memory state distinguish domains. Do not grep only the final “Killed process” line.

Reclaim happens before killing

The allocator can reclaim clean page cache, write dirty data, shrink kernel caches, compact memory, and invoke filesystem/device paths subject to context. Reclaim can stall for a long time and still fail because pages are dirty, pinned, unevictable, fragmented for a high-order allocation, or outside the allowed nodes/cgroup.

This means a host can be operationally unavailable from pressure before an OOM kill occurs. Monitor Pressure Stall Information and latency, not only the final event.

Virtual memory reservations and physical page allocation happen at different times. vm.overcommit_memory controls address-space commitment accounting:

0: kernel heuristic;
1: always overcommit from the accounting perspective;
2: refuse commitments beyond a limit derived from swap and overcommit_ratio or overcommit_kbytes.

Strict mode does not reserve physical RAM for every accepted allocation and cannot prevent all OOM scenarios involving page cache, kernel memory, constrained nodes, cgroups, or locked/pinned pages. Programs also frequently fail to handle malloc() failure safely. Do not change overcommit globally as a generic OOM fix.

Victim selection uses badness and eligibility

For eligible tasks, the kernel’s OOM badness heuristic heavily reflects memory footprint relative to the allowed memory domain, then applies oom_score_adj. The implementation evolves; folklore about fixed root bonuses or “the newest allocation always dies” should not replace current kernel evidence.

Inspect values before an incident:

cat /proc/"$pid"/oom_score
cat /proc/"$pid"/oom_score_adj
grep -E '^(Name|Pid|VmRSS|VmSwap|RssAnon|RssFile|RssShmem):' /proc/"$pid"/status

oom_score is a current estimate, not the historical score at kill time, and processes can exit/change memory between samples.

`oom_score_adj` is policy, not diagnosis

The adjustment range is -1000 through 1000. Negative values reduce likelihood; -1000 disables OOM killing for that task. A process may raise its own score, while decreasing it requires appropriate privilege.

Protecting many services can leave no useful victim or force the kernel to kill an even more damaging task. Do not set SSH, databases, or PID 1 to -1000 from a generic checklist. Define recovery dependencies, memory limits, restart behavior, and what should be sacrificed first.

Service managers expose policy declaratively. With systemd, review OOMScoreAdjust=, OOMPolicy=, cgroup memory settings, and any systemd-oomd integration together.

The triggering task may differ from the victim

An innocent task can request a small allocation when the system is already exhausted. A large process with the highest eligible score may be selected even if a separate leak or unbounded cache caused the trajectory. Conversely, the allocating task can be killed under some constraints.

Investigate memory growth before the event: cgroup series, anonymous/file/shmem/slab metrics, swap, page faults, PSI, application workload, and deployment changes. The kernel victim table is a pressure snapshot, not causal history.

Threads and process groups complicate the record

Memory is often shared across threads and processes. OOM selection identifies a task, then killing/reaping behavior and logged identity can be confusing around thread groups, shared address spaces, fork-heavy workloads, and containers. Match PID, TGID, cgroup, unit/container ID, executable start time, and deployment instance.

PIDs are reusable. A later /proc/<pid> does not prove it is the killed process.

cgroup v2 provides containment and evidence

Use memory.high as a pressure/throttling boundary and memory.max as the final hard limit. Monitor:

cat /sys/fs/cgroup/your.slice/memory.current
cat /sys/fs/cgroup/your.slice/memory.events
cat /sys/fs/cgroup/your.slice/memory.pressure

Paths vary and managed hierarchies should be configured through the service/orchestrator. memory.events distinguishes high, max, oom, and oom_kill counters. An event at an ancestor can affect descendants even when a leaf’s local view looks quiet.

`memory.oom.group` changes workload semantics

Setting memory.oom.group=1 asks the kernel to treat a cgroup’s workload as an indivisible unit for OOM killing, except tasks protected with oom_score_adj=-1000. This can avoid a partially killed service continuing in a corrupt/degraded state.

It can also terminate healthy workers and increase restart cost. Use it only when the service’s consistency model favors all-or-nothing termination, and test supervisor behavior. It does not make the victim selection extend outside the constrained ancestor arbitrarily.

systemd-oomd is not the kernel OOM killer

systemd-oomd is a userspace daemon that monitors cgroup v2 and PSI to act before severe thrashing, based on configured candidates and pressure/swap policy. A process killed by systemd-oomd may have no kernel “Out of memory” victim record.

Inspect its journal and unit properties:

oomctl
journalctl -u systemd-oomd -b
systemctl show your.service -p ManagedOOMMemoryPressure -p ManagedOOMSwap

Availability/property names depend on systemd version. Distinguish a userspace SIGKILL from kernel OOM evidence before tuning.

Collect the right forensic bundle

Preserve untruncated kernel journal, previous boot if available, unit/container events, cgroup files and ancestors, /proc/meminfo, /proc/zoneinfo, slab summary, PSI, swap status, kernel command line, overcommit sysctls, and time-series monitoring. Sanitize command lines/environment that can contain secrets.

Useful read-only checks include:

journalctl -k -b | grep -i -E 'out of memory|oom-kill|killed process'
grep -E '^(MemAvailable|MemFree|SwapFree|Dirty|Writeback|Slab|SReclaimable|SUnreclaim):' /proc/meminfo
cat /proc/pressure/memory

Do not run memory-heavy diagnostic collectors during active thrashing without considering their footprint.

Avoid dangerous global reactions

vm.drop_caches is not an OOM repair and can worsen performance; it only targets clean caches after synchronization conditions and does not fix anonymous growth. Adding swap can provide breathing room but may turn failure into prolonged thrashing and can expose sensitive pages unless storage is protected.

vm.panic_on_oom can reboot/panic systems and lose availability/data. Never toggle it interactively on production to make OOM “cleaner.” Kernel/sysctl changes require architecture-specific recovery design and testing.

Design limits from measured peaks

Classify working set: anonymous, page cache, shmem/tmpfs, kernel/slab, swap, huge pages, and pinned/locked memory. Set requests/protections and limits with startup/compaction/headroom. Introduce memory.high, observe throttling/latency, then choose memory.max and restart policy.

Load-test below and above boundaries. Confirm alerts fire before kill, essential management remains responsive, the intended cgroup—not a sibling—records the event, native data survives abrupt termination, and restart does not loop into pressure.

Close with cause and decision separately

An incident report should state: allocation domain, pressure trajectory, trigger context, eligible set, selected victim/adjustment, kill mechanism (kernel cgroup/global or userspace), reclaimed outcome, root cause, and preventive control.

OOM handling is successful only when limits match service semantics, monitoring detects sustained pressure early, an intentional victim policy preserves recovery, and a repeat test cannot destabilize unrelated workloads.

Related:

Sources: