io_uring Explained: Linux's Modern Asynchronous I/O Interface

io_uring is a Linux-specific asynchronous I/O API built around submission and completion queues shared between userspace and the kernel. It can batch work, reduce copies and syscall transitions, register reusable resources, and express operations beyond file reads and writes.

It is not automatically zero-syscall, nonblocking, faster, or safer than a mature event loop. Correctness depends on queue protocol, memory lifetime, completion handling, kernel/liburing versions, operation semantics, and the workload being measured.

The interface begins with three syscalls

io_uring_setup() creates a ring context and returns a file descriptor. Userspace maps queue memory exposed by the kernel. io_uring_enter() submits queued work and/or waits for completions. io_uring_register() registers resources, personalities, restrictions, or other ring features.

Applications should normally use the maintained liburing helpers rather than reproducing ring layout and memory barriers from examples. Kernel UAPI headers define availability, while the running kernel decides which flags and opcodes are accepted.

Submission and completion are separate contracts

Userspace obtains a submission queue entry (SQE), fills its opcode-specific fields and user_data, publishes it to the submission queue, and submits. The kernel later posts a completion queue entry (CQE) containing the same user_data, a signed result, and flags.

A successful submit count does not mean the I/O succeeded. Per-operation errors normally arrive as negative error codes in cqe->res; applications must handle every CQE and advance the completion head correctly. Lost identity or ignored negative results create silent data corruption.

Shared rings reduce coordination overhead

The SQ and CQ are shared mappings, so queue metadata and entries do not need to be copied through a distinct syscall argument for every operation. Normal rings still generally call io_uring_enter() (often via io_uring_submit()) to tell the kernel about new SQEs. The advantage is that one transition can submit a batch and optionally wait for completions.

Therefore “placing an SQE never needs a syscall” is incomplete. Avoiding submission syscalls requires SQ polling and still has wakeup/wait/registration cases.

Asynchronous does not mean one execution mechanism

Some operations can complete inline during submission. Others execute through native async support. Potentially blocking work may be punted to io-wq worker threads. Filesystem, file type, flags, cache state, opcode, and kernel version determine the path.

Latency-sensitive applications must account for inline completions, worker creation, queueing, and resource limits. Observe actual behavior rather than assuming every buffered file operation is offloaded identically.

Queue depth needs backpressure

Ring size bounds outstanding queue bookkeeping, not unlimited work. Applications need a policy for a full SQ, a full CQ, partial submissions, CQ overflow, allocation failures, and downstream saturation. A deeper ring can improve batching but increase memory, queueing latency, and the amount of work in flight during cancellation or shutdown.

Use bounded application queues and monitor completions. Do not retry every -EAGAIN in a tight loop or increase ring size until overload disappears.

Buffer and file lifetimes are explicit

Many SQEs contain pointers. The referenced memory must remain valid for the operation’s documented lifetime; SQPOLL can extend submission-side lifetime assumptions because a kernel thread consumes entries asynchronously. Stack buffers that leave scope early are a classic use-after-free bug.

Registered buffers and files reduce repeated lookup/mapping costs and hold kernel references. They also pin memory or file objects and interact with RLIMIT_MEMLOCK, RLIMIT_NOFILE, cgroups, and cleanup. Huge-page registration may pin more memory than the slice actually used.

Linked requests express dependencies

IOSQE_IO_LINK and hard links can chain operations, but cancellation-on-failure semantics differ. Timeouts can be linked to an operation; multishot opcodes can produce multiple CQEs; provided buffer rings can select buffers dynamically. CQE flags must be interpreted, not discarded.

An “open, read, close” chain is not automatically transaction-safe. Partial results, short reads/writes, link failure, timeout races, and close ordering still need application logic.

Cancellation is inherently racy

Async cancel requests identify target work, but the target may already be executing or completed. Completion of the cancel operation and completion of the target are separate events. Closing a ring initiates teardown, yet applications still need a protocol that stops producers, accounts for in-flight resources, and consumes required completions.

Design every operation so normal completion, timeout, cancellation, and shutdown release ownership exactly once.

SQPOLL is a specialized mode

IORING_SETUP_SQPOLL creates a kernel thread that watches the SQ. While active, applications can publish work without an io_uring_enter() per batch. After its configured idle interval, the thread sleeps and userspace must detect IORING_SQ_NEED_WAKEUP and call into the kernel.

SQPOLL is not “a dedicated core is always consumed.” It creates a polling thread whose affinity, idle behavior, sharing, permissions, and resource use depend on setup and kernel version. Many rings can consume significant kernel resources. Benchmark it only for sustained workloads with a clear latency/CPU tradeoff.

IOPOLL is different from SQPOLL

IORING_SETUP_IOPOLL busy-polls storage completions and has strict device, driver, opcode, and often O_DIRECT requirements. It can reduce latency for appropriately configured polling-capable storage while consuming CPU. Combining it with SQPOLL does not make arbitrary buffered/network I/O faster.

Use current io_uring_setup documentation and device evidence. A flag accepted on one kernel/device is not a portability contract.

Feature probing beats kernel-version guessing

io_uring arrived in Linux 5.1, but opcodes, setup flags, restrictions, multishot behavior, and fixes have evolved continuously. Distribution kernels also backport selectively. Probe supported operations/features through liburing rather than branching only on uname -r.

Define fallback paths for -EINVAL, -EOPNOTSUPP, policy denial, and operation-specific limitations. Test the fallback as seriously as the ring path.

Security policy is part of deployment

io_uring exposes a broad kernel attack surface and has had security fixes. Current kernels provide kernel.io_uring_disabled policy values that can allow, restrict to an authorized group, or disable new ring creation depending on configuration/version. Containers or sandbox profiles may block io_uring_setup.

Do not weaken a host-wide sysctl or seccomp/LSM profile to gain an unmeasured optimization. Patch kernels, minimize exposed opcodes, and retain a synchronous/epoll/thread-pool path when the threat model forbids rings.

Ring restrictions can create an allowlist

A ring created with IORING_SETUP_R_DISABLED can register restrictions that allow specific registration operations, SQE opcodes, and flags before enabling it. Once enabled, those restrictions govern submissions. This can reduce what delegated code can request, but it does not remove kernel bugs or replace process sandboxing.

Combine restrictions with least privilege, file-descriptor discipline, seccomp/LSM policy, namespaces, cgroups, and an immutable loader. Verify the restriction path on the minimum supported kernel.

Benchmark end-to-end rather than counting syscalls

Compare against a well-tuned baseline using the same files, cache state, direct/buffered mode, queue depth, request size, durability semantics, CPU affinity, storage, and concurrency. Measure throughput, p50/p99/p999 latency, CPU time, context switches, worker count, memory/pinned pages, errors, and tail behavior under overload.

Warm-cache tiny I/O can exaggerate syscall overhead; large storage-bound I/O may show little difference. fsync and data-integrity requirements must remain equivalent. A faster benchmark that omits durability is not the same workload.

Build a correctness acceptance test

Exercise short reads/writes, EOF, partial submission, invalid descriptor, CQ pressure, timeout/target races, cancellation, signals, worker saturation, fork/exec policy, shutdown with work in flight, and kernel-policy denial. Use checksummed data and fault injection on disposable storage.

Record kernel and liburing versions, probed flags/opcodes, setup parameters, registered resources, limits, security policy, benchmark procedure, error counts, and fallback result. Adoption is justified when correctness is equivalent, measured workload benefit exceeds complexity, resource use is bounded, and disabling io_uring remains operational.

Related:

Sources: