Nvidia’s new speech model cuts errors on Indian doctors’ dictation by 62%

Nvidia released Nemotron 3.5 Transcribe on October 28, an open weight model that handles transcription, translation and speaker labeling in one pass. It is the open weight English leaderboard with a 5.33% word error rate.

Then Nvidia showed it to Indian doctors, and it scored 30.8%.

Same model, same settings. The gap between the two numbers is the part of this release that matters to you. A score of 5.33% means that about one word in nineteen comes out wrong. On the Eka Care benchmark, 3,619 clips of Indian doctors and medical students dictating, the out-of-the-box rate was close to one word in three.

A better prompt, one that simply told the model to expect English in the Latin alphabet, brought that down to 23.8%. Fine-tuning has it 11.8%about 62% fewer errors than where it started, and that fine-tuning costs about 14 GPU-hours on a single Nvidia GB300.

So the model is good, the published benchmark is real, and neither tells you what it will do to your audio.

Inside India Result

These numbers need unpacking because they are the clearest published example of adapting a general speech model to Indian-accented professional speech.

The benchmark didn’t have a training split, so Nvidia split it by speaker: 855 clips to choose from prompts and checkpoints, and 2,764 held. On that held set, the error rate went from 22.44% to 12.17%, a relative reduction of 45.8%. Keyword error rate improved from 9.2% to 6.5%, and semantic error rate from 10.0% to 7.6%.

On the same held data, an open medical specialist model scored 13.79% and what Nvidia describes only as a widely used general speech model 14.92%. The finely tuned Nemotron beat both. Nvidia doesn’t mention this second model anywhere in the post, which makes comparison difficult against anything you’re running today.

Whether 12.17% is good enough is a separate question from whether it is better. At that rate, about one word in eight is wrong, which works for a draft of a clinician’s assessment and signs, and doesn’t work for anything written on a record without supervision.

The training material was modest: around 153 hours of synthetic speech, 29 hours of accented English from Common Voice, and 63 hours of medical records from MultiMed. The method trained the speech encoder and projection layer while applying LoRA to the language model.

For anyone in Indian healthcare technology, this combination is the point. Not a bespoke model trained from scratch, but an off-the-shelf open-weight model plus several hundred hours of mixed data in less than a day of single-GPU time.

One model instead of four

The terrain for Transcribe is consolidation. Meeting audio usually required separate tools for recognition, speaker identification and translation, each with its own error modes and its own output format to reconcile.

This does it all from natural language instructions. The same recording can produce a verbatim transcript with the ums left, a cleaned version with them removed, or a transcript with the speaker marked. You can also ask questions about what was discussed instead of reading the transcript yourself.

You can also feed it vocabulary in the prompt. Drug names, product names, internal jargon, and a specified formatting convention, all provided as instructions rather than requiring a retrained model.

The operating limits: one hour of audio per pass, up to eight speakers tagged, word-level timestamps so you can jump to any point in the recording.

Architecture is a speech encoder that feeds a hybrid mamba-transformer mix-of-expert language model, with sortformer speaker cues coming in alongside the acoustic features, so that speaker tags come out during generation rather than from a separate pass.

Size is inconsistently reported. The blog post says about 30 billion total parameters with about 4 billion active per token, and the model is named Nemotron-3.5-Transcribe-30B-A4B. The Hugging Face listing shows 34B. Treat 30B total and 4B active as working numbers.

Watch the model tackle the fast-paced, overlapping dialogue of the 1940 screwball classic His girl Friday.

The accuracy numbers

On the Hugging Face Open ASR leaderboard for English, it ranks first among open-weighted models with a 5.33% average word error rate, with an RTFx of 462.09. API-only models are excluded from that ranking, so this is not a claim to beat any commercial service.

The artificial analysis tie is the result worth sitting. Nemotoron matches the top open-weight model at 2.8%, but that model runs about 24 billion active parameters per token versus Transcribe’s four. Equal accuracy at about one-sixth of the active computation is a greater claim than direct gain would have been.

On throughput, Nvidia reports processing 100 hours of audio in under three minutes on an RTX PRO 6000 Blackwell Server Edition with NVFP4, measured on 30-second clips at high concurrency.

Where it falls short

Nvidia is unusually direct about the gaps.

• No live transcription. It is built for recorded audio. Streaming is not supported, which excludes live captioning and real-time agent assistance.
• Speaker labels are not identities. It separates voices and labels them, but it can’t tell you who someone is.
• The answer to the audio question is English trained. The transcription covers 40 languages; Do not ask questions about a recording.
• Fine-tuning has a cost elsewhere. After medical adaptation, overall English accuracy on LibriSpeech Test-Clean slightly worsened, from 1.54% to 1.69%. Specialized pulls the model away from everything else.
• It still loses to brand names. The physician-specialist model remains better at exact matches on drug brand names and diagnoses. Nvidia mentions true clinical records of brand names as the next thing to fix.

Nvidia also recommends keeping at least 100 properly marked clips purely for the checkpoint selection before adapting it to your own domain.

Granary v2

Alongside the model, Nvidia published Granary v2: 463,062 hours of annotated speech across 48 languages, drawn from 16 source corpora. The previous version covered 25 languages.

Each recording comes with three transcript forms, raw spoken, standard written and cleaned of fillers, plus labels, translations, captions, question-answer pairs and contextual vocabulary terms. The NeMo Curator pipeline used to build it is open, so you can run it over your own Corpa.

One condition: these are annotations. The source audio must be downloaded separately under the license of each original corpus.

Run it

Self-hosting is through vLLM, or NeMo-Speech.cpp for local use. It runs on a single GPU, with Nvidia everything from Jetson AGX Thor and DGX Spark to Blackwell data center cards. Hosted endpoints are available from Baseten, DeepInfra and Simplismart.

Checkpoints and datasets are available at Hugging Face and ModelScope, with demos at Hugging Face Space and ModelScope Studios. The blog post does not state the license of the model, so check the model card before building anything commercial on it.

Who should bother

If you transcribe clean English meetings, this is a powerful free option that runs on a GPU and consolidates three tools into one. Take the leaderboard numbers at something close to face value.

If your audio is Indian accent, technical or both, the lesson from Nvidia’s own case study is that the leaderboard number does not apply to you. Budget for adaptation from the beginning: a few hundred hours of mixed data, a day of GPU time, and at least 100 properly labeled clips held purely to select a control point. This is a project, not a download.

And if you need live signature, this is not the model. Streaming is not supported, and no version of the fine-tuning history will change that.