Whisper is a general-purpose automatic speech recognition (ASR) model made by OpenAI: it turns spoken audio into text. It was trained on 680,000 hours of multilingual and multitask supervised data collected from the web. Besides transcription in many languages, it can translate speech into English and identify the spoken language.
Whisper is a Transformer sequence-to-sequence model, an encoder-decoder neural network. The audio is cut into 30-second chunks and converted into a log-Mel spectrogram, a picture of sound frequencies over time. The encoder processes the spectrogram, and the decoder predicts the text, with special tokens that select the task, such as transcription, translation or timestamps. There are six model sizes: from tiny (39 million parameters, about 1 GB of graphics memory) to large (1,550 million parameters, about 10 GB), plus turbo, an optimized version of large-v3. Smaller models are faster and less accurate. Whisper runs locally from the command line or from Python and needs the ffmpeg tool.
Whisper's code and model weights are free and open source under the MIT license.