Kernel Bypass over PCIe: Doorbells and DMA

2026-10 · John Rose · john@lineargp.com · code: github.com/jnros/sacc

When a CUDA kernel for inference gets submitted, how does it get to the GPU? I recently found GPUs routing around the Linux kernel using buffers and doorbells. I built a Synthetic PCIe Accelerator (spec, QEMU model, and kernel driver) to build and break these mechanisms.

A "kernel bypass" job takes an elegant path through host memory, across the PCIe wire, and back up through completion. Not only does the kernel get bypassed¹, but the CPU itself gets bypassed in part as DMA works its magic.

Now the rings are wrapping and doorbells are ringing. How does this work, and why this design? Let's embed and inspect.

Picture a host OS sitting on top of PCIe, with a GPU or SACC device below. Kernel bypass relies on two ring buffers, or queues: submissions and completions. Two whiteboards with numbered job lists.

Host perspective

The host OS writes a number of jobs onto the list and rings the device doorbell. The SACC driver populates the submission queue (one job for now, submitted in kernel code). When "ringing the doorbell", the driver writes sq_tail to the device over iomapped memory, saying "here's the last job # I wrote down". In PCIe lingo, the doorbell is a posted write: the host doesn't wait around to confirm.

Now the host watches the completion queue until the job's slot is marked complete. The device reads and writes the submission and completion queues (both in host memory) with DMA. On production systems, the IOMMU serves as gatekeeper for these accesses. No IOMMU here, for now.

Device perspective

Because SACC includes the QEMU model, we get a nice symmetrical view of both sides. When the device hears the doorbell, it uses DMA to read the submission queue. The doorbell carried the tail, so the device loops to read all new jobs.

For each job, the device does some work. Matrix multiplication, CUDA kernel execution, or in my case a NOOP. When complete, the device uses DMA to mark the job done on the host's completion list.

Why this design

Doorbell page. The spec gives the doorbell its own 4K page after device register space. When userspace later maps the doorbell page, it won't get access to critical PCIe registers. Note that ringing the doorbell sends a 32-bit tail value. So why dedicate an entire page? Because a page is the smallest blob of memory one can map into userspace.

Phase bit completion. The host never asks the device "are you done?" That question would be an MMIO read: a CPU stall for a round trip across PCIe. Instead, each completion entry speaks for itself. The device writes the completion payload first and the phase bit last. The host polls its own memory, which costs almost nothing until the line changes.

The phase bit flips every time the ring wraps. On the second pass, slot 0 still holds the old entry from pass one. Because the host knows this lap's phase, stale entries don't match.

sacc_cq_post cq_tail=63 req_id=0x40 ... phase 0x1
sacc_cq_post cq_tail=0 req_id=0x41 ... phase 0x0

Neighbors

This can all seem dry and arbitrary. Think about garages. You (Linux) collaborate with your neighbor (GPU) on a multi-step woodworking project. Crucial points:

1. Your neighbor is free to use your garage (DMA) as well as their own (HBM).
2. Work is coordinated entirely with two whiteboards that never leave your garage (host memory).
3. A security keypad (IOMMU) keeps your garage safe.
4. No face to face. (Shout out, engineers.)
5. Your house (kernel) plays a vital role attached to the garage, but it's easier to bypass the house and go garage to garage.

Debugging

When I submitted my first job from the SACC driver, the device detected the doorbell, read the job entry, and saw all zeros. Driver problem or device problem? On real silicon, this would take a root cause analysis across OS and hardware.

It turns out QEMU provides a monitor mode: freeze the guest VM and directly inspect guest memory. It even looks like GDB.

Above: register reads and writes during PCI probe, then the exact host memory contents at the descriptor address. The green box shows the descriptor in memory is not zero. Request ID 1, right there in the hex.

This rules out a kernel-side descriptor or submission problem. My bug was in the emulated device, and it surfaced in 30 seconds. QEMU is the gift that keeps on giving.

Note to self: VM freeze and memory inspection has broad potential for host-side optimization of inference, mech interp, and more.

1. The current SACC implementation uses kernel code to submit and process requests. In production systems, and a later version of SACC, userland handles submissions and completions for complete kernel bypass.

Spec, QEMU device model, and kernel driver: github.com/jnros/sacc