What Whisper Is: One Model That Listens, Translates, and Identifies Languages
Whisper is a general-purpose speech recognition model from OpenAI, and it is free and open source under the MIT license, with 113,221 stars on GitHub. It is trained on a large dataset of diverse audio, and it is a multitasking model: the same model does multilingual speech recognition, speech translation, spoken language identification, and voice activity detection. Under the hood it is a Transformer sequence-to-sequence model. The training format represents all of those tasks as a sequence of tokens for the decoder to predict, with special tokens that tell the model which task it is doing, so a single model replaces the many stages of a traditional speech pipeline.
Six Model Sizes: From Tiny on a Laptop to Large on a GPU
There are six model sizes, and picking one is a trade between speed, accuracy, and memory. Tiny has 39 million parameters, needs about 1 GB of graphics memory, and runs about 10 times faster than large. Base has 74 million parameters, also needs about 1 GB, and runs about 7 times faster. Small has 244 million parameters, needs about 2 GB, and runs about 4 times faster. Medium has 769 million parameters, needs about 5 GB, and runs about 2 times faster. Large has 1.55 billion parameters, needs about 10 GB, and is the reference speed. Turbo has 809 million parameters, needs about 6 GB, and runs about 8 times faster than large. Four of the sizes also come in English-only versions, which tend to do better for English-only work, especially tiny and base.
The Turbo Catch: The Default Model Does Not Translate
This is the one thing that trips people up. The default model is turbo, which is an optimized version of large-v3 that transcribes much faster with only a minimal drop in accuracy, and it works well for transcribing English. But turbo is not trained for translation tasks. If you ask it to translate non-English speech into English, it will return the original language instead, even if you pass the translate flag. So if you need translation, do not use turbo. Use one of the multilingual models instead, medium or large for the best results, with the translate task and the language set, like whisper japanese.wav --model medium --language Japanese --task translate.
Languages: Dozens, With Honest Accuracy Numbers
Whisper works in dozens of languages, and it detects which language is spoken on its own, so you do not have to tell it in advance. Its accuracy varies widely by language, and the project is honest about that: the README shows a word error rate breakdown by language for the large models, measured on the Common Voice 15 and Fleurs datasets, with character error rates for languages where that is the fairer measure. More detailed scores live in the paper, including translation quality scores. The practical takeaway is that English and other high-resource languages transcribe very well, while rarer languages are rougher, and the larger models close more of that gap.
Installing Whisper and Transcribing Your First File
Installing Whisper takes two steps. First install ffmpeg, which Whisper needs to read audio, from your package manager. Then run pip install -U openai-whisper, which pulls the latest release. The codebase works with Python 3.8 to 3.11 and recent PyTorch versions. After that, transcription is one command: whisper audio.mp3 --model turbo transcribes a file with the default model. You can pass several files at once, set the language with --language, or pick a different model with --model. If you would rather use it from Python, load a model with whisper.load_model("turbo") and call model.transcribe("audio.mp3"). Internally the transcribe method reads the audio with a sliding 30-second window and predicts the text a window at a time.
Running It Locally: Your Audio Stays on Your Machine
The part that matters most for many people is that Whisper runs locally. Your audio never has to leave your computer, which is a big deal for interviews, meetings, and voice notes you would rather not upload to a cloud service. The smallest models run on about 1 GB of graphics memory, so even a modest machine can transcribe. Once you have a transcript, it is just text, so the usual text tools apply. Whisper can also write its output as JSON with word-level timestamps, which is what caption tools want. If your timestamps live in a spreadsheet instead, a quick CSV to JSON pass turns them into JSON without retyping anything. And when a transcript is ready to share, a Text to HTML pass turns it into a clean web page you can send to anyone.



