caldrix run

caldrix run <model> compiles a model for this machine and answers one prompt with -p, or, without -p, chats in the terminal line by line. Generation is greedy unless a sampling flag asks otherwise (Sampling); the answer goes to stdout and status lines go to stderr.

Usage

caldrix run <model> [-p PROMPT] [-i IMAGE]... [--system TEXT|@FILE] [--max-tokens N] [--ctx N] [--backend auto|hip|metal|cpu] [--plain]
    [--draft-model DIR|ALIAS|REPO] [--temperature T] [--top-k N] [--top-p P] [--min-p M] [--repetition-penalty R] [--seed N]
    [serving flags]

The model comes first; flags follow it.

The model argument

<model> is one of, checked in this order:

1. A directory holding config.json, a supported tokenizer (the models page lists them) and *.safetensors. An existing directory always wins. An argument starting with ./, / or ~ that is not a directory is refused as no such directory.
2. An alias: smollm2, smollm2-instruct (from v0.1.2), llama-8b, qwen3-0.6b, qwen3-8b, qwen3-30b-a3b, decider-0.8b or smolvlm-256m, each a Hugging Face repo pinned to a revision (models.md).
3. A Hugging Face repo id, owner/name or owner/name@revision. Without @revision it is main.

An alias or a repo id is downloaded into the Hugging Face cache the first time and read from there after. Anything else is refused before any download.

One prompt

caldrix run smollm2 --backend cpu -p "The capital of France is" --max-tokens 8

On a Linux x86-64 machine this printed, on stdout:

 the capital of the country.

SmolLM2-135M is a small base model that continues text. Before the answer, caldrix printed status lines on stderr: that the model was already in the Hugging Face cache (a first run lists the files it downloads instead), the compile, the lane and its settings, and, for this model, that it has no chat template, so the prompt is sent as plain text.

With --backend auto (the default) one more stderr line names the lane chosen, for example caldrix: backend cpu (auto; `caldrix doctor` shows why).

Chat

Without -p, caldrix run reads one message per line after a > prompt:

caldrix run qwen3-0.6b

• Each answer streams as it is generated, and the conversation is kept: each message is sent with the earlier turns through the model's chat template.
• /reset clears the conversation.
• When the conversation outgrows the compiled context (--ctx), the oldest turns are dropped, and stderr says how many. Room is kept for the answer: --max-tokens when given, else a quarter of the context, at most half of it.
• A single message too long for the context is refused, and the chat continues.
• A model without a chat template (a base model such as SmolLM2-135M) gets each line as plain text, without the conversation; stderr says so once. A base model continues text and rarely ends its own reply, so from v0.1.2, without --max-tokens, each reply stops at 256 tokens. To chat with a SmolLM2, use (from v0.1.2) caldrix run smollm2-instruct (SmolLM2-135M-Instruct), which has a chat template. It is verified on the CPU lane (x86-64-v3), not on Metal or AMD GPUs, and what was verified is its tokens on a plain-text prompt: the chat template is not exercised by any gate.
• When the model's chat template refuses a system message (--system, /system), the system text is placed at the top of the first user message instead, and stderr says so once.
• Ctrl-C while an answer is streaming stops that answer. Ctrl-C at the prompt exits, as does end of input (Ctrl-D).

Images

A model that takes images (smolvlm-256m, on the cpu lane only; caldrix doctor prints images on <lane>: for each lane, doctor.md) answers about a picture. -i attaches an image file to the first message, and it may be given up to 8 times; in the chat, /image PATH attaches one to the next message.

caldrix run smolvlm-256m -i photo.png -p "What is in this picture?"

On a Linux x86-64 machine on the CPU lane, caldrix run smolvlm-256m -i wide.png -p "What is in this picture?" --max-tokens 16 --plain on a test image (geometric shapes on a colour gradient) printed, on stdout:

 A colorful gradient background with five geometric shapes in various colors: a square, a

and, last on stderr, the image line:

image: 512×512 ×13 tiles, 832 rows

• A file must be PNG, JPEG or WebP, told from its bytes, not its name, and at most 8 MiB. It is read once, before the model is downloaded or loaded, so a refused file costs nothing; a pipe such as -i <(curl …) works.
• Each image is cut into tiles the way the HTTP API's "detail": "auto" does (api.md), and stderr prints one line for it, image: 512×512 ×13 tiles, 832 rows: the tile size, the tiles and the prompt rows they take. A side over 4096 pixels is refused when the image is attached.
• In the chat the images stay in the conversation: each later message is sent with them again, and they are decoded again for it. Past 8 images, or past --ctx, the oldest turns are dropped, and stderr says which limit dropped them. /reset clears the conversation and any image not yet sent.
• An image on a model that takes text only is refused (… takes text only; -i and /image need a vision model), as is -i with --raw (an image needs the chat template).

Flags

Any other flag goes to Inferno's serving library, which checks it: --dtype, --threads, --cpu-isa, --weight-budget and the rest are the serving flags in serve.md. A flag is checked, and the lane chosen, before any download starts.

Speculative decoding

--draft-model names a small model that guesses tokens ahead of the main one; the main model checks the guesses, and a greedy reply is the same tokens plain decoding gives. It works on a lane that can check a draft: caldrix doctor prints speculation=yes or speculation=no on each lane line (doctor.md), and today cpu and hip say yes. On another lane the flag is not used, and stderr says why.

• Before any download, a draft the model index describes must pass the architecture gate and, when the index also holds the main model, fit beside it: in memory with both models resident, and on disk, where the draft's own download is compared with the free space (the main model's download still to come is not counted). A draft named as a directory, or a Hub draft the index does not describe at that revision, gets neither check from caldrix; Inferno's serving library checks it when it loads it. Once the draft's files are fetched, its tokenizer must match the main model's.
• A suggested draft that fails the gate or the fit is passed over for the next candidate. A draft that cannot be fetched, whose tokenizer differs, or that fails to load is not replaced. None of this stops the run: stderr names the draft and why, and the run speculates with n-gram lookup (guesses taken from the text already in the conversation) instead.
• Without --draft-model, caldrix chooses a draft by itself from the model index only on the AMD lane, which can check a draft; on the cpu lane a draft is used only when you name it. caldrix models drafts <repo> lists the candidates the index knows (models.md).
• A suggested draft that was turned off on this machine in an earlier run is not tried again for the same model. A draft you name with --draft-model is still tried, and stderr says it lost last time.
• Speculation runs one request at a time, so --max-batch other than 1 leaves it off; --weight-budget does too. Pool members do not speculate, and an Ollama model is not a draft.

Nothing here claims a speed-up: no speed has been measured.

Sampling

Without a sampling flag every reply is greedy: each token is the most likely one, and the same prompt gives the same reply. The six flags above turn sampling on. The penalty is applied first, then --top-k, --top-p and --min-p cut the candidates, then --temperature reshapes what is left and one token is drawn. --temperature 0 with a penalty keeps the most likely token after the penalty, with no random draw. On the AMD lane the GPU picks the token, in the same order but with its own draw, so a seed's reply there can differ from the CPU's pick; --sampler host picks on the CPU (api.md). A value out of range is refused by name before any download starts.

In the chat, the same knobs are commands, applied from the next reply on (a reply already streaming is not touched, and /reset leaves them alone):

/temperature T   /top-k N   /top-p P   /min-p M   /repetition-penalty R   /seed N

Each takes a value, off (back to its default), or nothing (print its current value); /sampling prints all six on one line.

Refused flags

Exit status