Colibri: Self-Host a 744B LLM on Hardware You Already Own

Colibri is an inference engine written in pure C — one file, no BLAS, no Python at runtime, no GPU required — that runs Mixture-of-Experts models from 744 billion to 2.8 trillion parameters on hardware you already own.

Not a quantized 7B model pretending to be big. The actual frontier-scale weights, running on a desktop.

The trick is not compression. It is placement: Colibri treats VRAM, RAM and disk as one continuous memory hierarchy and streams the parts of the model it needs, exactly when the router proves it needs them.

Nine model families run today, each as its own C file behind the same coli chat / coli serve / coli web front end: GLM-5.2/5.3 (744B), GLM-5.3-Flash (321B, vision), Inkling (975B), Kimi K3 (2.8T), DeepSeek V4 Flash (284B), DeepSeek V4.1 Flash (552B, vision), Qwen3.8-Flash-Next, Qwen3.6 and OLMoE (7B).

The repository is Apache-2.0, and it has taken 30,956 stars and 3,319 forks since 1 July 2026 — roughly ten weeks. That trajectory is extraordinary for a systems project written in C, and worth keeping in mind when weighing how settled it is.

This is the insight the whole project rests on, and it is genuinely elegant.

A 744B Mixture-of-Experts model does not use 744B parameters per token. It activates roughly 40B — about 5.4% — and of those, only around 11 GB actually change from one token to the next.

So the model never needed to fit in fast memory. It needed to be placed:

• The dense part — attention, shared experts, embeddings, about 17B parameters — stays resident in RAM at int4, roughly 9.9 GB.
• The 19,456 routed experts — 75 MoE layers × 256, plus the MTP head, about 19 MB each at int4 — live on disk (~372 GB) and are streamed on demand, with a per-layer LRU cache, a learned hot-store of pinned experts, and an optional VRAM tier.

The project's own analogy is the best one: a JIT, but for weights.

A compiler JIT never compiles the whole program. It watches what actually runs and compiles the hot paths, just in time. Colibri makes the same bet about a 744B parameter space — parameters are not resident state to be held, they are data to be staged across a storage hierarchy, exactly when needed.

The router runs a layer ahead so prefetching hides the staging latency, and measured routing heat decides which experts earn which tier. The more you run it, the hotter the right experts get. It works because expert routing has measurable structure — and structure is cacheable.

Here is where most coverage of projects like this becomes vague. Colibri publishes measured numbers, per hardware class, with experiment logs. Same engine, same int4 container — the hardware only changes where the experts live.

Read those numbers carefully, because they are the honest answer to "can I run a 744B model at home?"

Yes — and at roughly one to two tokens per second on a normal machine. That is about a word per second. A 200-word answer takes two to three minutes.

That is not interactive chat. It is a batch tool: ask a question, come back later. For an overnight analysis job, a long document pass, or anything where you would otherwise pay per token to an API, that may be an excellent trade. For replacing ChatGPT in a browser tab, it is not.

To the maintainer's credit, none of this is hidden. The project states plainly that it offers "no SLA on speed, and a hard guarantee on semantics" — and that the default policy never silently changes model precision or router semantics. Insufficient fast memory may make it slow; it must not quietly redefine the model. That is the right priority, and rarer than it should be.

The headline says consumer hardware. The disk requirement deserves equal billing.

Storage: about 372 GB, for the pre-converted GLM-5.2 int4 container. Ideally on a fast NVMe, because expert streaming is read-bound — the project supports a dual-SSD mode specifically to double read bandwidth. This is the real barrier for most people, not the GPU.

RAM: around 10 GB for the resident dense part at int4, plus headroom for the expert cache. More RAM means a larger hot-store and fewer disk reads.

GPU: optional. It becomes another tier for hot experts rather than a requirement. A CPU-only 128 GB desktop works.

Getting started is three steps — clone and build, fetch the model, run:

./coli chat

One caveat worth carrying over verbatim, because it will bite you otherwise: use the group-scaled (gs64) container with the int8 MTP head, not the older per-row int4 mirrors. Those measure about 9 percentage points worse on quality and were the root cause of think-mode loops and never-terminating generations. The MTP head specifically must be int8 — an int4 head collapses draft acceptance to 0%, silently removing the speculative-decoding speedup you were counting on.

You can also convert from the FP8 source yourself with ./coli convert, which is resumable and never needs the full 756 GB on disk at once.

Ollama and LM Studio solve a different problem. They make it easy to run a model that already fits your machine — 7B, 14B, maybe 70B quantized. They are the right tool for most people most of the time, and Colibri does not replace them. Colibri exists for the case where the model emphatically does not fit.

llama.cpp is the closest technical relative and shares the ethos: C/C++, minimal dependencies, run it anywhere. llama.cpp does support MoE models and offloading. Colibri pushes the streaming idea much further — expert-level granularity, learned hot-stores, router-ahead prefetch — and narrows its scope to frontier MoE specifically.

AirLLM (which we covered in an earlier video) attacks the same wall from the other side: it streams layers in and out to fit big dense models on small GPUs, in Python on top of PyTorch. Colibri streams experts, exploits MoE sparsity rather than fighting dense weights, and drops the Python runtime entirely. Same headline claim, genuinely different engineering.

vLLM is the datacenter answer — built for throughput across many concurrent requests on hardware that holds the whole model. Different universe, different budget.

The honest position: Colibri occupies a narrow and real niche. If you want a frontier-scale MoE model on hardware that cannot hold it, and you can tolerate batch-speed inference, very little else does this.

Three things to weigh before clearing 372 GB of disk.

Speed is the headline trade. Covered above, but worth restating: one to two tokens per second on typical hardware. Everything about whether this project is useful to you follows from whether that is acceptable for your workload.

It is a research platform as much as a product. The README says so directly — it exists to test aggressive systems ideas across the software/hardware boundary. That is a strength for anyone interested in inference engineering and a caution for anyone wanting a stable tool. 127 open issues on a ten-week-old project is not alarming for something moving this fast, but it tells you where it is on the maturity curve.

Configuration has sharp edges. The gs64-versus-per-row container issue and the int8 MTP head requirement are both cases where a wrong-but-plausible choice degrades output quality or silently disables a feature, rather than failing loudly. The documentation flags both, but you have to read it.

None of this is a reason to avoid the project. It is a reason to go in knowing that you are running a fast-moving research engine, not installing an app.

Try it if you want to run frontier-scale models on your own hardware and speed is negotiable — batch analysis, overnight jobs, privacy-critical work where the alternative is sending data to an API. The economics can be compelling when you are otherwise paying per token.

Try it if you find inference engineering interesting. The expert atlas, the routing-heat measurements and the tier-placement work are genuinely novel, well documented, and the issue tracker reads like a research log. This is one of the more instructive open source codebases going right now.

Skip it if you want interactive chat. Ollama with a model that fits your machine will give you a far better experience, instantly.

Skip it if you do not have ~372 GB of fast storage to spare. That constraint does not bend.

Be realistic about hardware. The 6× RTX 5090 figure is the ceiling, not the expectation. Look at the 128 GB CPU row — ~1.8 tok/s — because that is far closer to what most people will see.

Everything here was checked against the repository on 14 September 2026, at 30,956 stars with the last commit that morning. This project moves quickly; verify the numbers before relying on them.