現象:服務明明還有一些可用記憶體,Linux 卻突然結束程序,甚至讓容器內的 AI 推論中斷。這不一定是整台機器的 RAM 用光;配置範圍、swap、memcg 限制與保護分數都可能讓核心進入 OOM 路徑。
Phenomenon: Linux may terminate a process even when the machine appears to have free memory. The trigger can be a cgroup, NUMA policy, swap, or allocation scope—not only total RAM.
圖 1|Linux OOM killer 的簡化決策路徑:先確認耗盡範圍,再以記憶體用量與 oom_score_adj 評分。
Figure 1 | Simplified Linux OOM-killer path: identify the exhausted scope, then score candidates using memory usage and oom_score_adj.
機制:OOM 先看耗盡範圍,再挑候選者 / Mechanism: scope first, victim second
核心在無法透過 reclaim 或 compaction 滿足配置時,才會進入 OOM killer。mm/oom_kill.c 的 oom_badness() 依候選程序記憶體用量與 oom_score_adj 計算分數;-1000 代表強力保護。允許使用的記憶體集合取決於當次 OOM 上下文,因此 memcg OOM 不等同於整機 OOM。
The kernel enters the OOM path only after reclaim and compaction cannot satisfy the allocation. In mm/oom_kill.c, oom_badness() scores candidates using memory footprint and oom_score_adj; -1000 strongly protects a task. The allowed memory set depends on context, so a memcg OOM is not necessarily machine-wide.
影響:被殺的不一定是最吃 RAM 的程序 / Impact: the largest process is not always the victim
只看 free 會漏掉容器的 memory.max、匿名頁與 swap。端側 LLM 或影像推論的一次大配置,可能在 reclaim 來不及完成時觸發 OOM;同一 memcg 中分數較高的輔助程序也可能先被殺。
Checking only free misses a container's memory.max, anonymous pages, and swap. A large edge-LLM or image-inference allocation can hit OOM before reclaim catches up, and a higher-scored helper in the same memcg may be killed first.
可操作步驟:先留下證據,再調整邊界 / Actionable steps: capture evidence first
# Kernel OOM messages and selected victim
journalctl -k -b | grep -E 'oom|Out of memory|Killed process'
# Per-process score and adjustment
cat /proc/$PID/oom_score
cat /proc/$PID/oom_score_adj
# cgroup v2 boundary and usage
cat /sys/fs/cgroup/$CGROUP/memory.current
cat /sys/fs/cgroup/$CGROUP/memory.max
# pressure and swap
cat /proc/pressure/memory
free -h先把 OOM 日誌、oom_score_adj、cgroup 用量與 PSI 放在同一時間線,再決定降低 batch、提高上限、增加 swap,或重新設計模型載入。不要把所有關鍵程序設成 -1000,否則可能把程序級故障換成整機失穩。
Correlate the OOM log, process adjustment, cgroup usage, and PSI before changing batch size, limits, swap, or model loading. Protecting every critical task with -1000 can trade a process failure for system-wide instability.
查核與限制:本文依 Linux 6.18.55 本機 mm/oom_kill.c(2026-11-13 查核)及官方文件整理。指令只觀測狀態,請在測試環境驗證變更。
Verification and limits: Checked against local Linux 6.18.55 mm/oom_kill.c on 2026-11-13 and official kernel documentation. Commands observe state; validate changes in a test environment.
Linux kernel VM sysctl documentation · Linux memory-management concepts · /proc documentation
留言
張貼留言