Why we JIT-compile the whole inference stack

2026-07-14 · Caldrix Engineering

Every inference engine optimizes kernels. Almost none optimize the stack — the scheduler deciding what batches together, the KV-cache manager deciding what lives where, the HTTP layer feeding tokens back out. Those pieces are written once, in general-purpose code, for a hypothetical average machine.

Your machine is not average. An MI300X has 192 GB at one bandwidth profile; an M3 Max shares memory with the OS; a Snapdragon phone would rather run four small kernels than one big one. When the stack is fixed, every one of those machines runs the same scheduling decisions — tuned, at best, for an H100.

Inferno probes the machine it is on — CPU instruction set, core count, cache-line geometry, which GPU runtime is present — and compiles the model's kernels against what it found. Inferno will treat the entire serving path as one program in Caldrix IR and compile the whole program against the probe. The probe will grow to measure FLOPS, memory bandwidth and whether CPU offload is worth modeling at all; continuous batching thresholds will become compile-time constants, KV-cache layout will be chosen for the measured bandwidth, and kernel boundaries will dissolve into megakernels when fusion pays and stay separate when it doesn't.

The result will be a binary that could only have been built for your machine — because it was.