2024¶

NVIDIA NeMo Canary Model Pushes the Frontier of Speech Recognition and Translation

NVIDIA NeMo team is thrilled to announce Canary, a multilingual model that sets a new standard in speech-to-text recognition and translation. Canary transcribes speech in English, Spanish, German, and French and also generates text with punctuation and capitalization. Canary supports bi-directional translation, between English and three other supported languages. Canary achieves the first place on HuggingFace Open ASR leaderboard with an average word error rate of 6.67%, outperforming all other open source models by a wide margin.

Unveiling NVIDIA NeMo's Parakeet-TDT -- Turbocharged ASR with Unrivaled Accuracy

Earlier this month, we announced Parakeet, a cutting-edge collection of state-of-the-art ASR models built by NVIDIA's NeMo toolkit, developed jointly with Suno.ai. Today, we're thrilled to announce the latest addition to the Parakeet family -- Parakeet TDT. Parakeet TDT achieves unrivaled accuracy while running 64% faster over our previous best model, making it a great choice for powering speech recognition engines in diverse environments.

The "TDT" in Parakeet-TDT is short for "Token-and-Duration Transducer", a novel sequence modeling architecture developed by NVIDIA and is open-sourced through NVIDIA's NeMo toolkit. Our research on TDT models, presented in a paper at the ICML 2023 conference, showcases the superior speed and recognition accuracy of TDT models compared to conventional Transducers of similar sizes.

To put things in perspective, our Parakeet-TDT model with 1.1 billion parameters outperforms similar-sized Parakeet-RNNT-1.1b in accuracy, as measured as the average performance among 9 benchmarks on the HuggingFace Leaderboard. Notably, Parakeet-TDT is the first model to achieve an average WER below 7.0 on the leaderboard. Additionally, it achieves an impressive real-time factor (RTF) of 8.8e-3, 64% faster than Parakeet-RNNT-1.1b's RTF of 14.4e-3. Remarkably, Parakeet-TDT's RTF is even 40% faster than Parakeet-RNNT-0.6b (RTF 12.3), despite the latter having about half the model size.

Use Parakeet-TDT model in your code

To run speech recognition with Parakeet-TDT, NVIDIA NeMo needs to be installed as a pip package as shown below. Cython and PyTorch (2.0 and above) should be installed before attempting to install NeMo Toolkit.

pip install nemo_toolkit['asr']

Once NeMo is installed, you can use Parakeet-TDT to recognize your audio files as follows:

import nemo.collections.asr as nemo_asr
asr_model = nemo_asr.models.ASRModel.from_pretrained(model_name="nvidia/parakeet-tdt-1.1b")
transcript = asr_model.transcribe(["some_audio_file.wav"])

Understanding Token-and-Duration Transducers

Token-and-Duration Transducers (TDT) represent a significant advancement over traditional Transducer models by drastically reducing wasteful computations during the recognition process. To grasp this improvement, let's delve into the workings of a typical Transducer model.

Transducer models, as illustrated in Figure 2, consist of an encoder, a decoder, and a joiner. During speech recognition, the encoder processes audio signals, extracting crucial information from each frame. The decoder extracts information from already predicted text. The joiner then combines the outputs from the encoder and decoder, and predict a text token for each audio frame. From the joiner's perspective, a frame typically covers 40 to 80 milliseconds of audio signal, while on average people speak a word per 400 milliseconds. This discrepancy makes it so that certain frames don't associate with any text output. For those frames, the Transducer would predict a "blank" symbol. A typical sequence of predictions of a Transducer looks something like,

_ _ _ _ NVIDIA _ _ _ _ is _ _ _ a _ _ great _ _ _ _ _ place _ _ _ _ to work _ _ _

where _ represents the blank symbol. To generate the final recognizion output, the model would delete all the blanks, and generate the output

NVIDIA is a great place to work

As we can see, there are many blanks symbols in the original output and this means the Transducer model wasted a lot of time on "blank frames" -- frames for which the model predicts blanks which don't contribute to the final output.

TDT is designed to mitigate wasted computation by intelligently detecting and skipping blank frames during recognition. As Figure 3 shows, when a TDT model processes a frame, it simultaneously predicts two things:

1. probability of token P_T(v|t, u): the token that should be predicted at the current frame;
2. probability of duration P_D(d|t, u): the number of frames the current token lasts before the model can make the next token prediction.

The TDT model is trained to maximize the number of frames skipped by using the duration prediction while maintaining the same recognition accuracy. For example in the example above, unlike a conventional Transducer that predict a token for every speech frame, the TDT model can simply the process as follows,

frame 1:  predict token=_,      duration=4
frame 5:  predict token=NVIDIA, duration=5
frame 10: predict token=is,     duration=4
frame 14: predict token=a,      duration=3
frame 17: predict token=great,  duration=6
frame 23: predict token=place,  duration=5
frame 28: predict token=to,     duration=1
frame 29: predict token=work,   duration=4
frame 33: reached the end of audio, recognition completed.

In this toy example, TDT can reduce the number of predictions the model have to make from 33 to 8. In our extensive experiments with TDT models, we see this optimization indeed leads to a substantial acceleration in recognition speed. Our research has also demonstrated that TDT models exhibit enhanced robustness to noisy speech and token repetitions in the text compared to traditional Transducer models. Note, this blog post simplies certain aspects of Transducer models in order to better illustrate the design differences between Transducers and TDT, and we would refer interested readers to our paper for technical details.

Additional Resources

• HuggingFace ASR Leaderboard
• HuggingFace ASR Leaderboard Evaluation
• NeMo Parakeet Models on HuggingFace
• NVIDIA NeMo Webpage
• NVIDIA NeMo ASR Documentation
• Papers:
  • Efficient Sequence Transduction by Jointly Predicting Tokens and Durations
  • Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition
  • Investigating End-to-End ASR Architectures for Long Form Audio Transcription

Announcing NVIDIA NeMo Parakeet ASR Models for Pushing the Boundaries of Speech Recognition

NVIDIA NeMo, a leading open-source toolkit for conversational AI, announces the release of Parakeet, a family of state-of-the-art automatic speech recognition (ASR) models (Figure 1.), capable of transcribing spoken English with exceptional accuracy. Developed in collaboration with Suno.ai, Parakeet ASR models mark a significant leap forward in speech recognition, paving the way for more natural and efficient human-computer interactions.

NVIDIA announces four Parakeet models based on RNN Transducer / Connectionist Temporal Classification decoders and the size of the models. They boast 0.6-1.1 billion parameters and capable of tackling diverse audio environments. Trained on only a 64,000-hour dataset encompassing various accents, domains, and noise conditions, the models deliver exceptional word error rate (WER) performance across benchmark datasets, outperforming previous models.

• Parakeet RNNT 1.1B - Best recognition accuracy, modest inference speed. Best used when the most accurate transcriptions are necessary.
• Parakeet CTC 1.1B - Fast inference, strong recognition accuracy. A great middle ground between accuracy and speed of inference.
• Parakeet RNNT 0.6B - Strong recognition accuracy and fast inference. Useful for large-scale inference on limited resources.
• Parakeet CTC 0.6B - Fastest speed, modest recognition accuracy. Useful when transcription speed is the most important.