caldrix quickstart

caldrix is one binary that downloads a model from Hugging Face, compiles it with the Inferno compiler for the hardware it finds (an AMD GPU, an Apple silicon GPU or the CPU), and either answers a prompt or serves an OpenAI-compatible HTTP API. Four commands take you from nothing to a running server: install, caldrix doctor, caldrix run, caldrix serve.

These docs track the main branch, which can be ahead of the latest release; what each release changed is in the changelog.

Supported hardware and models

A lane is supported only on hardware it has been verified on. Hardware the binary accepts but that has not been verified is listed as not verified on hardware; everything else is not supported.

Release archives are built for x86_64-linux (glibc 2.38 or newer) and, from v0.1.1, aarch64-darwin (macOS 14.0 or newer on Apple silicon). The installer refuses any other platform.

Models: the import accepts checkpoints of six architectures (LlamaForCausalLM, MistralForCausalLM, Qwen2ForCausalLM, Qwen3ForCausalLM, the mixture-of-experts Qwen3MoeForCausalLM and Qwen3.5's Qwen3_5ForCausalLM, with Qwen3_5ForConditionalGeneration as text only) in safetensors, bf16, f32 or f16. Mistral is accepted by the import but has not been verified on hardware. Aliases name the models Inferno has run on hardware, each pinned to a revision: smollm2, smollm2-instruct, llama-8b, qwen3-0.6b, qwen3-30b-a3b and decider-0.8b (models.md); smollm2-instruct (from v0.1.2) is verified on the CPU lane (x86-64-v3), not on Metal or AMD GPUs: its tokens on a plain-text prompt; its chat template is not exercised by any gate. qwen3-30b-a3b is verified by its token gates on one MI300X (resident), one M1 Max streamed under a 56 GiB --weight-budget, and the CPU lane (x86-64-v3) streamed under a 20 GiB budget (models.md). decider-0.8b (Qwen3.5) is a decision model, not a chat model; its token gates and the Qwen3.5 base model's ran at the desk on the CPU lane (x86-64-v3), on one MI300X and on one Apple M1 Max (through the engine), and caldrix runs Qwen3.5 on the cpu lane only: the hip and metal lanes refuse it (models.md). Quantized: Ollama Q4_K_M and Q8_0 tags on the metal lane and, at the desk, on the cpu lane (x86-64-v3); on the hip lane, five tags were token-exact in f32 on one MI300X (models.md); quantized safetensors checkpoints are not supported.

Generation is greedy unless asked: --temperature, --top-k, --top-p, --min-p, --repetition-penalty and --seed turn sampling on, and the chat changes them with /temperature and its siblings (run.md).

Start

Install the binary into ~/.local/bin (install.md):

curl -fsSL https://caldrix.ai/install.sh | sh

See which lanes this machine can use (doctor.md):

caldrix doctor

Ask a model something (run.md). The first run downloads the model into the Hugging Face cache:

caldrix run qwen3-0.6b -p "What is the capital of France?"

Serve it over HTTP on port 8425 (serve.md):

caldrix serve qwen3-0.6b

Then, from another terminal (api.md):

curl http://localhost:8425/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-0.6B", "messages": [{"role": "user", "content": "What is the capital of France?"}]}'

Any OpenAI client works with its base URL set to http://localhost:8425/v1.

All docs

• Quickstart: this page.
• Install: the installer, its options, checksums and uninstalling.
• Kubernetes: the Helm chart, what it installs, its values and what it does not do.
• caldrix doctor: what each line of the report means.
• caldrix run: one prompt, or a chat in the terminal.
• caldrix serve: the HTTP server and its flags.
• HTTP API: the endpoints, the request fields and the errors.
• caldrix pool: the daemon that holds models for local clients.
• Models: the aliases, the Hugging Face cache and what the import refuses.
• Troubleshooting: each refusal, its cause and its fix.

To run a model in the browser instead, see caldrix.js, the npm package @caldrix-ai/caldrix.