nanochat: Train Your Own GPT-2 Grade ChatGPT in About Two Hours

nanochat is Andrej Karpathy's experimental harness for training a large language model from scratch, end to end, on a single GPU node. One codebase covers every stage: training the tokenizer, pretraining, supervised finetuning, evaluation and inference, ending in a chat model you can talk to from the command line. The repository's own tagline is "The best ChatGPT that $100 can buy."

The name comes from Karpathy's earlier nanoGPT, which only covered pretraining. nanochat goes the rest of the way to a chatbot and deliberately stays small. Its contributing notes promise no giant configuration objects, no model factories and no if-then-else monsters, just a single, cohesive, minimal, readable, hackable and maximally forkable strong baseline. Most of the logic sits in a handful of files, such as nanochat/gpt.py for the Transformer, nanochat/engine.py for inference with a KV cache and nanochat/optim.py for the AdamW and Muon optimizers.

It is written in Python on PyTorch, licensed under MIT, and was first published on October 13, 2025. When we checked on November 2, 2026 it had 59,646 stars and 8,388 forks. For questions, the README points to DeepWiki, the repository's Discussions tab and a #nanochat channel on Discord.

The design idea that makes nanochat easy to use is a single complexity dial: --depth, the number of Transformer layers. Everything else is derived from it automatically, including the width of the model, the number of attention heads, learning rate adjustments, the training horizon and weight decay, all chosen so the result comes out compute optimal. You ask for a smaller or bigger model and the rest follows.

Sweeping that one dial gives what Karpathy calls a miniseries of compute-optimal models at different sizes. GPT-2 capability currently sits between depth 24 and depth 26. For quick experiments he likes depth 12, roughly GPT-1 sized, where a pretraining run takes about five minutes: change something, rerun d12 and watch validation bits per byte, the CORE score and training throughput to see whether it helped. Odd depths such as 25 are not recommended, because the head-dimension math stops coming out neatly.

The dial is also the bar for contributions. A change has to be principled enough to work at every depth rather than being tuned to one model, which is what keeps the codebase small while it improves.

Training length has its own knob, --target-param-data-ratio, the number of training tokens per non-embedding parameter. Compute optimal is currently 10.5. The GPT-2 runs deliberately go lower and slightly undertrain a d24, because the goal there is to cross the GPT-2 line as fast as possible rather than to get the best model for the compute spent.

The main focus of development is a "time to GPT-2" speedrun, inspired by the modded-nanogpt project: the wall-clock time an 8XH100 node needs to train a model that beats GPT-2 (1.6B) on the DCLM CORE score, an ensemble of 22 evaluations such as ARC and MMLU. GPT-2's CORE score is 0.256525, and a run has to land above it.

The record has fallen quickly. The first entry, on January 29, 2026, took 3.04 hours with a slightly overtrained d24. FP8 training brought it to 2.91 hours on February 2, doubling the batch to about 1M tokens got it to 2.76 hours on February 5, and switching the pretraining data to NVIDIA ClimbMix cut it to 2.02 hours on March 4, after FineWeb, DCLM and Olmo data had all failed to help. The last two records, 1.80 hours on March 9 and 1.65 hours on March 14, 2026, came from autonomous research: improvements found by a private version of Karpathy's autoresearch on a d12 model, which carried over to d24.

The current record, 1.65 hours or 99 minutes, is the average of five runs with a mean CORE of 0.262634. Whatever holds the record is always the recipe in runs/speedrun.sh, so cloning the repository gives you the current best rather than last year's tutorial.

The leaderboard notes give enough numbers to check the headline claim. OpenAI trained GPT-2 in 2019 on 32 TPU v3 chips for 168 hours at about $8 per chip-hour: 32 x 168 x $8 = $43,008, the "approx. $43K" the README quotes. At the README's assumed $3 per GPU-hour, an 8XH100 node costs about $24 an hour, so the 1.65-hour record is about $39.60 of training time.

That is roughly 1,086 times cheaper and about 100 times faster in wall-clock time, and the hardware tells the same story: 5,376 TPU-hours then, 13.2 H100-hours now. Run 1's log puts its total compute at 4.330784131228946e+19 floating point operations; a scientific notation converter turns that into a plain number, about 43.3 quintillion.

The catch is that the leaderboard measures only the training iterations. Its notes say the reported time excludes evaluations and logging, while the full speedrun script also downloads data, trains a tokenizer, evaluates the base model and runs supervised finetuning. So the repository quotes three numbers for one job: about 1.5 hours in the script's header, 1.65 hours on the leaderboard, and about 2 hours, or $48, in the README's introduction. The 1.5-hour figure is already shorter than the training phase alone, so plan on the last one. The README adds that a spot instance can bring the total closer to $15.

nanochat uses uv for dependencies. On a CUDA machine run uv sync --extra gpu, or uv sync --extra cpu for CPU-only and Apple Silicon, then activate the environment with source .venv/bin/activate. For the real thing, rent an 8XH100 node (the README mentions Lambda, which provided compute for the project), clone the repository and start the whole pipeline in one command, ideally inside screen so it survives a dropped connection:

screen -L -Logfile runs/speedrun.log -S speedrun bash runs/speedrun.sh

The script is short enough to read before you pay for the node. It downloads 8 data shards, about 2 billion characters, and trains a BPE tokenizer with a vocabulary of 32,768 on them while the download continues in the background up to 170 shards. About 150 are needed for GPT-2 capability, and at roughly 100 MB each that is around 17 GB of text to pull before pretraining starts; a unit converter helps when you are sizing the disk. Then it pretrains a d24 with FP8 and a data ratio of 8, evaluates the CORE score, and runs supervised finetuning to teach conversation tokens, tool use and multiple choice, with evaluations on tasks such as ARC, GSM8K, MMLU and a simple Python coding set.

When it finishes, chat with your model:

python -m scripts.chat_cli

Add -p "Why is the sky blue?" for a single prompt instead of an interactive session. Logging to Weights & Biases is optional: run wandb login and set WANDB_RUN before starting the script, otherwise it skips wandb entirely.

You do not need eight H100s to use nanochat, only to match the leaderboard. The README says the code runs fine on an 8XA100 node, a bit slower. On a single GPU, drop torchrun and the code switches to gradient accumulation automatically and produces nearly identical results, but it takes eight times longer, so a run measured in hours on a node becomes most of a day.

If your GPU has less than 80 GB of memory, lower --device-batch-size from its default of 32 to 16, 8, 4, 2 or even 1 until it fits, keeping it a power of two. The training script compensates with more gradient accumulation steps, so the total batch the optimizer sees stays the same and only the number of passes changes.

Precision is set explicitly rather than through autocast. A single COMPUTE_DTYPE defaults to bfloat16 on A100, H100 and newer cards, and to float32 on older ones such as the V100 and T4, where float16 is available with NANOCHAT_DTYPE=float16 and a gradient scaler (reinforcement learning does not support it yet). CPU and Apple MPS default to float32, and recent macOS also runs bfloat16 with about 25% less memory at similar speed. The --fp8 flag is optional: without it training runs in bf16 and produces a slightly stronger model.

For a laptop, runs/runcpu.sh shrinks the model until training fits in a few tens of minutes on CPU or Apple Silicon. The README is blunt that you will not get strong results that way. It is for learning the pipeline, not for building a useful chatbot.

Set expectations for the model itself. The speedrun trains with about 4e19 FLOPs, which the README compares to talking to a kindergartener: it writes stories and poems, explains why the sky is blue with confident nonsense, and hallucinates if you ask it who you are. That is the nature of a model this size, not a flaw in your run.

A single run is a sample, not a guarantee. When Karpathy ran the Run 4 recipe seven times, CORE scores ranged from 0.25119 to 0.26765, and three of the seven landed below GPT-2's 0.256525. Later records were set on averages with a larger safety margin, and the leaderboard notes recommend tracking validation bits per byte as a smoother signal, with the warning that it is not comparable across runs on different datasets.

Some documentation has drifted behind the code. The README's file tree still describes speedrun.sh as training "the ~$100 nanochat d20" while the script now trains a d24, and the original October 2025 announcement is marked as containing deprecated information. Non-CUDA backends such as XPU and MPS should work because the code is mostly plain PyTorch, but the README says not all of those paths have been exercised.

If you plan to contribute, read the AI policy first: pull requests must disclose any parts with substantial LLM contribution that you did not write or do not fully understand. The latest leaderboard record dates from March 14, 2026 and the last push to the repository was on September 7, 2026, so check the Discussions tab for current activity before building on it.