caldrix serve

caldrix serve <model> compiles a model for this machine and serves it over an OpenAI-compatible HTTP API on http://127.0.0.1:8425. It takes the same model argument as caldrix run, and every serving flag of Inferno's server; the endpoints are in api.md.

Usage

caldrix serve <model> [--port 8425] [--host 127.0.0.1] [--ctx N] [--backend auto|hip|metal|cpu] [serving flags]

The model comes first: a directory, an alias or a Hugging Face repo id owner/name[@revision] (run.md, models.md). --backend picks the lane as in caldrix run (doctor.md), and --ctx N is --max-context N.

An exchange

caldrix serve qwen3-0.6b --backend cpu

Once the server is up, stderr ends with the line caldrix: serving Qwen/Qwen3-0.6B on http://127.0.0.1:8425. From another terminal:

curl http://localhost:8425/v1/models

{"data":[{"id":"Qwen/Qwen3-0.6B","object":"model","owned_by":"inferno"}],"object":"list"}

curl http://localhost:8425/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-0.6B", "messages": [{"role": "user", "content": "What is the capital of France?"}], "max_tokens": 16}'

{"choices":[{"finish_reason":"length","index":0,"message":{"content":"<think>\nOkay, the user is asking for the capital of France. I know","role":"assistant"}}],"created":1792079067,"id":"chatcmpl-1","model":"Qwen/Qwen3-0.6B","object":"chat.completion","usage":{"completion_tokens":16,"prompt_tokens":15,"total_tokens":31}}

Both were captured on a Linux x86-64 machine on the CPU lane. The model name in a request must be the name the server answers to: the repo id for an alias or a repo id, the directory's name for a directory, or --served-name. Ctrl-C stops the server.

Defaults and your flags

caldrix serve starts Inferno's server with its own defaults first and your flags after them:

--model-dir <the model's directory> --backend <the lane> --served-name <the repo id or directory name> --max-context 2048 --port 8425

Where a flag is given twice, the last one wins, so any of your flags overrides the default. --model-dir is the one you cannot give: the model is the first argument.

Flags

Every flag Inferno's server parses, with the default caldrix serve runs it at. Each is checked before any download starts.

FlagMeaningDefault
--port Nthe HTTP port; 0 takes any free port8425
--host ADDRthe address to listen on127.0.0.1
--served-name NAMEthe model name the API answers tothe repo id, or the directory's name
--max-context N (or --ctx N)the compiled context length, in tokens2048
--dtype bf16|f32the type the model is compiled inbf16; f32 for a quantized model, which refuses bf16
--queue-depth Nrequests that may wait for the engine; past it a request gets 4298
--max-batch 1|8requests decoded together; 8 only on the AMD lane8 on the AMD lane; 1 on metal and cpu, with --weight-budget, with --admin and for a quantized model, which refuses 8
--admission fill|onehow prompts are admitted between decode steps: fill admits into every free slot, one at most one per loop iterationfill
--prefill-grid off|onAMD lane only: run each prompt's prefill as one persistent train beside the decode step; on needs --prefill-attention tiles, --prefill-gemm inferno, --max-batch 8, bf16, resident weights and a dense Llama model, and is refused otherwise, naming the other flagoff
--threads Ncpu only: the size of the thread poolthe physical cores, capped by a cgroup CPU quota
--cpu-isa ROWcpu only: the instruction-set row to compile for (x86-64-v2, x86-64-v3, x86-64-v4), at or below the probed onethe probed row
--weight-budget BYTESstream the weights through this much memory instead of holding them all (5g, 4500m or a byte count); refused on the AMD laneevery weight resident
--weight-ring Nwith --weight-budget: slots in the streaming ring, at least 22
--weight-pin Nwith --weight-budget: at most this many layers held residentno cap
--weight-pins spread|prefixwith --weight-budget: which layers are held resident, spread over the stack or the first onesspread
--weight-digest every-fill|at-openwith --weight-budget: when a streamed segment's checksum is checkedevery-fill
--weight-codec raw|planeswith --weight-budget: how the streamed layout file stores the weightsplanes
--layout-dir DIRwith --weight-budget: where the streamed layout file is keptlayout under Inferno's cache: $INFERNO_CACHE_DIR, else $XDG_CACHE_HOME/inferno, else ~/.cache/inferno
--telemetry off|host|sampledhow much of each request's timing is recorded for /metrics and /debug/requestssampled
--device-sample-period Nwith sampled: device timing on one step in N8
--device-sample-offset Nwith sampled: which step of each period, below the period4
--telemetry-log FILEappend each request's telemetry record to FILEnone
--batch-gemv rows|x2|x4AMD lane only: the batched GEMV kernel's spelling; x4 is refused when the plan compilesrows
--tail generatedAMD lane only: the output-layer kernel's spellinggenerated
--prefill-attention rows|tilesAMD lane only: the prefill attention kernel's spelling; tiles needs --max-batch 8, a bf16 Llama model and, with --prefill-chunk N, N a multiple of 32, and is refused otherwiserows
--batch-attention rows|groupedAMD lane only: the batched attention kernels' spellingrows
--sampler device|hostAMD lane only, with --max-batch 8: where a sampled token is picked, on the GPU or on the CPU (api.md)device
--draft-model DIR|ALIAS|REPOspeculate with this draft model on a lane that can check a draft (speculation=yes in doctor.md); a draft that fails its checks falls back to n-gram lookup, with a stderr line (run.md)none
--adminopen the variant-swap admin endpoint, on loopback only; needs --admin-portoff
--admin-port Nthe admin endpoint's port, different from --portnone
--shadow-ring Nwith --admin: recent requests kept to check a candidate variant against8
--shadow-requests Nwith --admin: requests a candidate must match before it serves4
--shadow-timeout SECONDSwith --admin: how long a candidate has to match600
--temperature T, --top-k N, --top-p P, --min-p M, --repetition-penalty R, --seed Nthe server's default sampling settings, the same knobs as run.md; a request's own fields override them (api.md)greedy (0, 0, 1, 0, 1, fresh entropy)

The kernel flags (--batch-gemv, --tail, --prefill-attention, --batch-attention, --sampler) are refused on the metal and cpu lanes. The --admin flags are refused on the cpu lane and with --weight-budget. The sampling flags set the server's defaults for every request; a request's own temperature, top_p, top_k, min_p, repetition_penalty and seed override them per call (api.md).

A vision model

caldrix serve smolvlm-256m --backend cpu serves SmolVLM-256M, which takes image parts in a chat message (api.md); at startup stderr carries the line vision: siglip 12×768, 64 tokens/tile. It runs on the cpu lane only, one request at a time, from its safetensors weights held resident.

  • --dtype bf16 (the default) runs the text model in bf16; --dtype f32 runs it in f32. The vision tower runs in f32 at either, and its output enters the text model in the text model's type.
  • Refused by name at startup, each naming the vision tower: a --backend other than cpu, --max-batch above 1, --weight-budget, a kernel flag (--batch-gemv, --tail, --prefill-attention, --batch-attention, --sampler) at other than its default, --admin, --speculate draft and --speculate ngram, and a GGUF file or quantized weights.
  • It decodes without speculation. If --speculate auto is passed (or caldrix's own speculation would turn it on), a startup line says speculation is off and why; --speculate draft and --speculate ngram are refused.
  • Its request body limit is 32 MiB, on every route, and at most 2 requests decode images at once (api.md).

Binding and proxies

The server listens on 127.0.0.1, so only this machine can reach it. --host 0.0.0.0 (or any address that is not loopback) exposes it to that network, and the server prints a warning: there is no authentication and no TLS.

To serve other machines, keep the default and put a reverse proxy in front that terminates TLS and authenticates clients, forwarding to http://127.0.0.1:8425. Answers stream as server-sent events, so turn off response buffering for /v1/ in the proxy. A request body over 4 MiB is refused with 413 by the server itself (32 MiB for a vision model).