NVIDIA Releases NemotronLabs VoiceChat, an Open Full-Duplex Voice Model

NVIDIA released NemotronLabs VoiceChat on August 3, 2026 — an 11-billion-parameter, end-to-end speech model that listens and speaks at the same time, published with open weights on Hugging Face. It is the first open full-duplex model that can call tools mid-conversation, and independent testing from Artificial Analysis places it as the only open-weights speech model ranking top-three on both conversational dynamics and speech reasoning.

Intermediate

One Model Instead of Three

Most voice assistants are built as a relay race: automatic speech recognition transcribes what you said, a language model decides what to reply, and a text-to-speech system reads the answer aloud. Every handoff adds latency, and the pipeline can only work in strict turns — the system cannot listen while it is talking.

VoiceChat collapses that stack into a single streaming network. A Fast Conformer speech encoder (borrowed from NVIDIA’s 0.6B streaming speech model) ingests 16 kHz audio, an NVIDIA Nemotron Nano v2 9B hybrid Mamba/Transformer backbone does the reasoning, and an NVIDIA TTS decoder plus streaming codec emits 22.05 kHz audio. An RNN-T decoder runs alongside to produce a live transcript of the user, and a dedicated head emits tool-calling scripts on a separate output channel so function calls never contaminate the spoken response.

Because both streams stay open, the model handles barge-in: interrupt it mid-sentence and it yields in roughly 480 ms. On Full-Duplex-Bench 1.0, smooth turn-taking latency measures 448 ms with a 0.82 turn-over rate, and GPT-4o rates the quality of its interruption handling at 4.33 out of 5. NVIDIA trained the system on roughly 550,000 hours of audio — a blend of synthetic TTS output, public corpora including Fisher, LibriVox, LibriTTS, HiFi-TTS and VCTK, and internal studio recordings.

Tool Calling While Talking

The headline capability is function calling that does not break the conversation. VoiceChat can fire a tool call and keep the dialogue alive with “on-hold” filler speech while the call resolves — the voice-agent equivalent of “let me look that up for you.” On the AU Harness BFCL-v3 suite it averages 56.1% (58.5% simple, 62.5% multiple, 42.5% parallel, and 89.6% on irrelevance detection). On Full-Duplex-Bench v3, it picks the right tool 82.5% of the time but gets the arguments right only 44.2% of the time, for a 33% pass@1.

That gap between selecting a tool and populating it correctly is the honest limit of the release. NVIDIA recommends no more than five tools per session, notes the model cannot reliably issue parallel calls, and warns that users cannot interrupt during tool execution.

Where It Lands Against the Field

Artificial Analysis scored VoiceChat at 38.8% on Big Bench Audio — the best of any open full-duplex model, ahead of Freeze-Omni (31.7%), NVIDIA’s own PersonaPlex (19.1%), FLM-Audio (16.0%) and Moshi (4.3%). On conversational dynamics it takes second at 77.8%, behind PersonaPlex’s 91.0%. Being strong on both axes at once is the point: as Artificial Analysis put it, VoiceChat “is the only open weights model that performs amongst the top 3 on both — making it the clear leader on the pareto frontier.”

Widen the chart to include closed models and the gap is stark. GPT-Realtime scores in the low 80s on speech reasoning with conversational dynamics near 96%, and Grok Voice Agent reaches 93% on reasoning. VoiceChat is not competitive with proprietary realtime APIs on raw audio reasoning — but it is weights you can download, inspect, and fine-tune.

What This Means

Open-weight voice has lagged open-weight text by a wide margin, in part because a speech-to-speech model is harder to assemble than a chat model. VoiceChat is the first open release that treats duplex conversation and agentic tool use as one problem rather than two, which matters for anyone building voice interfaces where audio cannot leave the building — clinical intake, classroom tutoring, regulated support desks.

The practical caveats are real. VoiceChat is English-only, capped at a two-minute audio context, weak on multi-step arithmetic, unable to handle backchanneling, and explicitly unsuited to noisy or reverberant rooms. It needs an 80 GB GPU (A100, H100, H200, B100, B200, or RTX 6000) running vLLM on Linux, and the weights ship under the OpenMDW-1.1 license marked for research purposes, with the surrounding NeMo Speech code under Apache 2.0.

The open drop also sits alongside a larger sibling. NVIDIA is running an early-access program for Nemotron 3 VoiceChat, described as a 12B model targeting sub-300 ms end-to-end latency by processing 80 ms audio chunks faster than real time. Parameter counts in NVIDIA’s own materials shift between 11B and 12B depending on the page, so treat the exact figure as approximate until the early-access documentation settles.

Related Coverage

• OpenAI Launches GPT-Live: Full-Duplex Voice for ChatGPT — the proprietary full-duplex release from July 2026 that VoiceChat is chasing
• OpenAI Launches GPT-Realtime-2 with GPT-5-Class Voice Reasoning — the closed realtime API family that still leads on speech reasoning
• Voxtral TTS: Mistral’s Open-Weight Text-to-Speech Model Rivals ElevenLabs — open-weight progress on the synthesis half of the stack
• NVIDIA Nemotron 3 Super: 120B Hybrid Model Activates Only 12B Parameters for Agentic AI — the hybrid Mamba/Transformer line VoiceChat’s backbone comes from
• NVIDIA Star Elastic: One Checkpoint, Three Reasoning Models, Zero-Shot Slicing — earlier open Nemotron research release

Sources

• nvidia/NVIDIA-NemotronLabs-VoiceChat-11B — Hugging Face model card
• NVIDIA Nemotron 3 VoiceChat: Leading the Open Weights Frontier of Conversational Dynamics vs. Speech Reasoning — Artificial Analysis
• Building NVIDIA Nemotron 3 Agents for Reasoning, Multimodal RAG, Voice, and Safety — NVIDIA Technical Blog
• NVIDIA-NeMo/Speech — nemotron-labs-voicechat branch
• NVIDIA Nemotron 3 VoiceChat Early Access Program