Inferno

Inferno.

Most AI servers run every model on the same generic code. Inferno compiles the model's kernels for the hardware it finds, then serves them.

install
curl -fsSL https://caldrix.ai/install.sh | sh
caldrix serve llama-8b

How it works

From a model name to a running server in four steps.

  1. Load the model

    From Hugging Face or a folder; its config is checked before any weights are downloaded.

  2. Find the hardware

    Checks which lanes this machine can use: an AMD GPU, an Apple GPU, or the CPU.

  3. Compile

    Compiles the model's kernels for that lane.

  4. Run

    Answers a prompt, or serves an OpenAI-compatible API.

Where the speed comes from

Speed work covers the whole server, not just the model.

Fewer trips to memory

Steps that usually run separately will be combined, so data stays close to the chip.

Memory sized to your machine

Inferno will size its memory to your GPU, so more requests fit at once.

Many users at once

On the AMD lane, --max-batch 8 lets up to eight requests share each step. It is the default there; --max-batch 1 serves one at a time.

Smaller models when needed

If a model won't fit, Inferno will pick a compressed version that does.

Hardware

One compiler for the chips you run.

  • NVIDIA GPUs
  • AMD GPUs (MI300X, gfx942)
  • Apple Silicon Macs
  • Intel GPUs
  • Qualcomm phone chips
  • x86_64 CPUs (Linux)
  • ARM CPUs
  • iOS and Android apps
  • Web browsers (WebGPU)

FAQ

Straight answers to common questions.

The first run builds the server. How long does that take?
Compiling takes a while the first time on each machine. Saving the build so later starts are instant, and a way to do it ahead of time.
Will it work with my OpenAI client?
Yes. Point its base URL at http://localhost:8425/v1, set model to the name the server answers to, and send sampling fields (temperature, top_p and the rest) when you want them: without them generation is greedy. An Anthropic-compatible endpoint.
What if my hardware isn't on the list?
caldrix doctor says which lanes this machine can use and why; --backend picks one yourself.
Is Inferno open source?
No. It ships as the caldrix command-line tool and runs entirely on your machine.
Is there a hosted version?
Caldrix Cloud will run the same engine on our machines.

Get started

Build it for your machine.

Install the CLI and run a model. The first run compiles; later runs will start instantly.

Read the docsFaster inference on your fleet
$ caldrix serve llama-8b
→ ready at 127.0.0.1:8425