Your serving loop is the other half of the model

2026-07-07 · Caldrix Engineering

Ask an engineer to make inference faster and they reach for the kernel. It has the interesting math and the published papers. It is also the part everyone else has already optimized, which is a good sign you should look elsewhere.

Between a request arriving and a token leaving there is a lot of software that is not the model: parsing, admission control, the batching policy, the paging scheme for the KV cache, the sampler, the detokenizer, the streaming encoder, and the HTTP layer carrying all of it. In a conventional stack every one of those is a fixed runtime component, configured rather than compiled, written without knowledge of the model it would serve or the machine it would land on.

That has a specific consequence: no optimization can cross the boundary. The batching policy cannot be specialized against the kernel's occupancy curve, because it does not have one. The paging scheme cannot be laid out against a cache hierarchy it never measured. The sampler cannot fuse with the last matmul — one is a library, the other is a loop in Python.

Today Inferno compiles the model's kernels against the machine it probed; the server around them is still ordinary hand-written code. Caldrix IR will put them in the same program. Scheduler and serving layer will be described alongside attention and the feed-forward blocks, and cache policy and sampler will join them, all compiled together against the same probed hardware model. Admission control will know the throughput curve because the compiler derived both. The cache will be paged against a measured cache line. The token leaving the sampler will already be in the wire format the endpoint wants.

None of this is exotic. It is mostly the observation that the boundary between "the model" and "the server around it" is an artifact of how the ecosystem grew — and a compiler that cannot see past it has to leave everything on the other side alone.