FreeToken: Run Frontier MoE Models on Your Own GPU, and the RAM It Takes

FreeToken is an open-source serving engine for running frontier-scale Mixture-of-Experts (MoE) language models on personal and consumer hardware. Instead of assuming a datacenter GPU with room for every weight, it treats the GPU, the CPU, system memory and the links between them as one elastic inference platform, and it exposes OpenAI and Anthropic compatible APIs so coding agents can use a local model the way they use a cloud one.

It comes from FlashML-org, and the accompanying paper, "FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution," was posted to arXiv on August 17, 2026, with authors including Ion Stoica, Matei Zaharia, Song Han and Kurt Keutzer. The abstract reports more than 20 MoE models served on hardware ranging from an 8 GB laptop GPU to a single workstation GPU: a 35B model on a laptop, a 284B model on a gaming desktop, and the 753B GLM-5.2 on one workstation card.

The repository was created on July 20, 2026, is licensed under Apache 2.0, and had 15,782 stars and 1,571 forks when we checked on November 2, 2026. The project credits mini-sglang as its main inspiration and says it reuses designs and code from SGLang, vLLM, FlashInfer, flash-linear-attention, LightLLM and llama.cpp.

The trick is in how MoE models work. Each layer holds many small expert networks, but a router sends each token to only a few of them, so most weights sit idle on any given step. Qwen3.6-35B-A3B, the model FreeToken's quick start uses, has 35 billion parameters in total but only about 3 billion active per token, which is what the A3B in its name means.

FreeToken builds on that. In its default offload strategy the experts live in host RAM and the GPU keeps a least-recently-used cache of expert slots: a token whose expert is already cached runs at GPU speed, and a miss is streamed over PCIe. The cpu strategy computes misses on the CPU instead of fetching them, and hybrid splits each step's misses between the two, overlapped, using bandwidth figures you measure once per machine with ft bench bw. If the experts fit in VRAM, fused keeps them all resident. Left on auto, dense models resolve to fused and MoE models to offload, except on unified-memory machines such as DGX Spark, where copying between CPU and GPU memory would only move data between two names for the same RAM.

Around that core sit the features the README highlights: double-buffered prefill streaming, one global LRU pool of experts shared across layers, FreeToken's own FTW fast-load weight format, VRAM that can move between the expert cache and the KV cache at runtime without a restart, and semantic anchor checkpoints that let agent edits such as tool calls and thinking blocks reuse context instead of recomputing it.

Offloading moves the problem rather than removing it: every expert still has to live somewhere, and that somewhere is your system memory. A back-of-envelope estimate shows what the paper's headline implies. A 284-billion-parameter model stored at 4 bits per weight is about 142 GB. In NVFP4, which adds an 8-bit scale for every 16 weights, it is closer to 160 GB, before the operating system, the KV cache and anything else you run. By the same arithmetic, the 753B GLM-5.2 result implies well over 400 GB of RAM.

So the "gaming desktop" in that claim is a PC with a GPU you can buy off the shelf and around 192 GB of RAM, which is the top end of consumer builds rather than a typical one. The laptop result scales the same way: a 35B model at 4.5 bits per weight is roughly 20 GB, far more than an 8 GB laptop GPU holds, so most of it sits in system RAM and a 16 GB laptop would not fit it. These are our estimates from the parameter counts, not figures FreeToken publishes, and the real footprint depends on each checkpoint's formats.

Storage matters too. DeepSeek-V4.1-Flash's two 98 GiB Engram tables stream from the checkpoint shards on demand, and the docs recommend keeping them on a fast NVMe drive. Qwen3.8-Flash-Next keeps a 47.7 GiB n-gram table pinned in host RAM. On Windows and WSL, CUDA caps pinned memory at around half of RAM, so FreeToken pins what it can from the middle layers outward and OS-locks the rest, which works but copies more slowly.

Generating a token touches only a few experts per layer, but reading a long prompt does not. FreeToken's prefill guide explains that with offloaded experts, every MoE layer copies all of its experts from host to GPU for every prefill chunk, because a large chunk routes to nearly every expert anyway. Copy time per chunk is therefore roughly the total expert bytes divided by PCIe bandwidth, and the guide measures about 55.6 GB/s on PCIe 5.0. The default chunk is 8,192 tokens.

Put the earlier estimate through that formula. If a 284B model carries about 150 GB of expert weights, each chunk needs about 2.7 seconds of copying, which caps prefill at roughly 3,000 tokens per second while the copy is the bottleneck. A 100,000-token coding-agent context is 13 chunks, about 35 seconds before the first output token, and roughly twice that on a PCIe 4.0 slot with about half the bandwidth. This is our arithmetic using the project's formula, not a published benchmark.

That number explains several design choices. Because the copy cost is per chunk, not per token, raising --max-prefill-length gives the GPU more work to hide each copy behind, until compute catches up. --moe-prefill-hit-d2d (CUDA 13 or newer) skips experts already in the GPU cache so only misses cross PCIe. Overlap, on by default, copies layer N+1 while layer N computes, and the server logs a warning at startup if the expert cache is too small to hold the two staging layers it needs. And the semantic anchor checkpoints exist so an agent's tool call does not send the whole context back through this pipeline.

The command-line engine needs Linux on x86_64, an NVIDIA GPU with driver r580 or newer (CUDA 13), and Python 3.10 or later, and the README lists native support for RTX 30, 40 and 50 series cards. If you would rather avoid the terminal, the desktop app for Windows and Linux at flashml.ai sets up the engine and gives you a GUI for running models, chatting and tuning. For the CLI, install with uv:

uv venv && source .venv/bin/activate
uv pip install "freetoken[accel]"

The PyPI package JIT-compiles its CUDA kernels on first use, so it needs a CUDA 13 toolkit with nvcc on your PATH. Nightly wheels with the kernels prebuilt are published on the repository's rolling nightly release; pin a wheel URL rather than the tag, because the tag moves every night.

Then start a server. --model takes a local checkpoint directory or a Hugging Face repo id, and the dtype, attention and MoE backends, cache sizes and tool-call and reasoning parsers all resolve from the checkpoint and the GPU:

ft serve --model ~/models/Qwen3.6-35B-A3B

The server is ready when the log says it is serving on 127.0.0.1:1919. It speaks the OpenAI API (/v1/chat/completions, /v1/responses, /v1/models) and the Anthropic API (/v1/messages, /v1/messages/count_tokens), so a client library for either works once you point its base URL at the server:

curl http://127.0.0.1:1919/v1/chat/completions -H 'Content-Type: application/json' -d '{"model": "Qwen3.6-35B-A3B", "messages": [{"role": "user", "content": "What is a Mixture-of-Experts model?"}], "max_tokens": 256}'

The reply comes back as one long line of JSON, and pasting it into a JSON formatter makes the choices and usage blocks readable. For a quick conversation without writing requests, ft shell opens a terminal chat attached to the running server, and ft shell --model starts an engine and chats in one process.

Local serving is most useful when the tools you already use can call it, and FreeToken has a shortcut for that. ft launch claude writes Claude Code's provider config, installs the CLI if it is missing and starts it against your server. The same command works for codex, opencode, openclaw, hermes and dsh, and --dry-run previews the changes first. The README names Codex, Claude Code, OpenCode, OpenClaw and DeepSeek Harness among the agents it works with.

Tool calling runs through parsers inferred from the model family, and reasoning models get their chain of thought split into a separate reasoning_content field. When you debug an agent, remember that in OpenAI-style responses a tool call's arguments arrive as a JSON-encoded string inside the JSON reply, full of escaped quotes; a JSON unescape tool turns them back into readable JSON.

By default the server binds to 127.0.0.1 and accepts any request, which is fine on a single machine. If you expose it to your network with --host, set --api-key (or the FREETOKEN_API_KEY environment variable) as well, so every route except /health requires the key as a Bearer token or an x-api-key header. A long value from a random string generator does the job. The server runs four requests at once by default, adjustable with --max-running-requests.

FreeToken is very young. The repository is about three and a half months old, it had 473 open issues and pull requests on November 2, 2026, and its docs show a weight format still in motion: several notes say FTW checkpoints converted by older builds no longer load, and one warns that an old Qwen3.5, Qwen3.6 or Qwen3.8-27B conversion loads without an error but gives wrong output. If you convert checkpoints with ft checkpoint, plan to reconvert after upgrading, and run the bundled checkpoint diagnosis script when a model refuses to load.

Hardware support is narrower than "your gaming PC" suggests. The CLI is Linux x86_64 and NVIDIA only, there is no macOS build, Windows users get the desktop app or WSL with its pinned-memory cap, and AMD support for RDNA3 and RDNA4 cards on ROCm 7.14 is labelled experimental, without the CUDA-only kernel backends. Not every feature covers every model either: DeepSeek-V4 experts have no resident path, and DSpark speculative decoding is not served for DeepSeek-V4.1-Flash.

Finally, the abstract makes capability claims, not speed claims. It says which models run on which class of machine, not how many tokens per second you will get, so measure on your own hardware. ft bench bw reports your real PCIe and CPU bandwidth, and --moe-collect-stats logs the expert-cache hit rate. Together they tell you whether a model will feel interactive on your machine before you buy more RAM for it.