Linux io_uring 為什麼仍可能卡住？從固定 Buffer 到 Completion Queue 的 I/O 路徑 / Why Linux io_uring Can Still Stall: Fixed Buffers and Completion Queues



現象：io_uring 非同步，為什麼服務仍會卡住？ / Symptom: Why can an io_uring service still stall?

你把大量檔案讀取改成 Linux io_uring，CPU 使用率下降，卻仍看到請求延遲突然拉高。提交請求變得非同步，不代表儲存裝置立刻完成；queue depth、檔案系統與 block I/O 仍會把等待時間留在 completion 之前。

You moved file reads to Linux io_uring, CPU usage fell, yet request latency still spikes. Submission becomes asynchronous; storage, the filesystem, and block I/O can still delay completion.

io_uring 的基本路徑是：應用程式準備 submission queue entry（SQE），核心取出並執行，最後把結果寫成 completion queue entry（CQE）。共享 ring 減少系統呼叫，但 CQE 的 res 才是實際結果；不能只看到 SQE 已提交，就把工作算成完成。

The path is: the application prepares an SQE, the kernel executes it, and a CQE reports the result. Shared rings reduce syscall traffic, but CQE res is the real outcome. Submission is not completion.

圖一｜io_uring 的 SQE→核心執行→CQE 路徑；固定 buffer 的生命週期與完成延遲必須分開觀察。 / Figure 1 | The SQE-to-kernel-to-CQE path; buffer lifetime and completion latency are separate signals. Original conceptual diagram.

機制：固定 Buffer 解決什麼，又帶來什麼責任？ / Mechanism: what registered buffers solve—and require

對固定大小、重複出現的 I/O，registered buffer 可以先把 buffer 註冊給 io_uring，之後請求只引用已知 buffer，可能減少每次操作的檢查與映射成本。代價是應用程式必須維持 buffer 有效，不能在核心仍可能使用時重用或釋放。這不是零延遲開關，而是把成本與責任搬到初始化與所有權管理。

For repeated I/O with stable buffers, registered buffers can reduce per-operation checking and mapping work. The trade-off is ownership: the application must keep each buffer valid until the kernel is done. This is not a zero-latency switch; it moves cost and responsibility into setup and lifetime management.

可重算示例：每個請求讀 64 KiB，同時維持 32 個未完成請求，理論上的 in-flight 資料量是 64 KiB × 32 = 2 MiB。這只是記憶體配置估算，不是吞吐量保證；若 CQE 回收速度低於提交速度，queue depth 會增加，延遲也可能惡化。

Example: 64 KiB per request with 32 in-flight requests gives 64 KiB × 32 = 2 MiB of theoretical in-flight data. That is a memory estimate, not a throughput guarantee. If CQEs are consumed more slowly than SQEs are submitted, queue depth grows and latency can worsen.

影響與可操作步驟 / Impact and practical steps

先量測，不要先調大 queue：記錄 SQE 提交時間、CQE 回來時間、cqe->res、錯誤碼與未完成數量，再對照 block 裝置延遲。若只記錄 submit 成功，你看不到真正 I/O 等待。對 AI 推論服務，模型載入與資料預取特別容易把高併發誤認成高效能。

Measure before increasing the queue: record SQE submission time, CQE arrival time, cqe->res, errors, and in-flight operations, then compare them with block-device latency. Logging only successful submission hides the wait. Model loading and prefetching make this mistake common in AI inference services.

cat /proc/diskstats; cat /sys/block/nvme0n1/stat; # read-only first checks

安全的第一輪排查是：固定 buffer 大小與數量，為每個 request 建立唯一 ID；只有收到 CQE 才回收 buffer；將提交延遲與完成延遲分開統計；最後才依服務的 p95/p99 延遲基準調整併發。若裝置延遲沒有下降，繼續加大 queue 只會增加排隊，不會創造 IOPS。

A safe first pass is to fix buffer size and count, assign each request a unique ID, recycle a buffer only after its CQE arrives, and report submission and completion latency separately. Tune concurrency from the service p95/p99 baseline. If device latency does not fall, a larger queue adds waiting rather than creating IOPS.

實務上還要分辨核心忙碌、裝置忙碌與應用程式沒有及時收 CQE 三種情況；同一個延遲數字可能來自不同層。把 request ID、buffer ID 與 CQE 結果放在同一筆 trace，才有機會把症狀對回機制。

In practice, distinguish a busy kernel, a busy device, and an application that is late to consume CQEs. The same latency number can come from different layers. Put request ID, buffer ID, and CQE result in one trace to connect symptoms to mechanisms.

資料與限制 / Sources and limits

查核日期：2026-11-12（Asia/Taipei）。本文依 Linux Kernel io_uring 官方文件與 Linux v6.18 原始碼整理；2 MiB 是示範算式，不是實測。本文未宣稱在讀者機器上執行指令，也不把非同步提交當作效能保證。 / Checked on 2026-11-12 (Asia/Taipei). Based on official Linux io_uring documentation and Linux v6.18 source. The 2 MiB value is explanatory, not a benchmark.

1. Linux Kernel Documentation: io_uring
2. Linux Kernel Documentation: io_uring setup
3. Linux v6.18 io_uring source