AdvancedAI

Next-Concept Modeling Beats Diffusion LMs at 10-31x Speed

Stanford’s Hazy Research group has published Next-Concept Modeling (NCM), a language model that writes text as a hierarchy of codes, coarse structure first and fine detail last, in eight to 32 steps rather than one token at a time. In a preprint posted to arXiv on October 9, 2026, and a blog post three days later, the team reports that NCM matches or beats strong diffusion language models while generating 10 to 31 times faster, and trains on four to five times fewer tokens. The work is a deliberate attempt to fix the three weaknesses the authors attribute to diffusion language models: too many sampling steps, a full-sequence pass at every step, and expensive training.

Diagram of a four-level coarse-to-fine generation trajectory, from a short level of global ideas to a full paragraph of text
NCM generates four levels of increasing detail, from global ideas to a final paragraph. Image credit: Hazy Research, Stanford

Two stages, one hierarchy

The first stage is a tokenizer, not a model of text. NCM trains a multiscale residual VQ-VAE that compresses a passage into a stack of discrete code levels: the earliest level holds a handful of codes that capture the broad shape of the sentence, and each later level is longer and adds finer detail. The design follows VAR, a 2024 image-generation method that predicts images scale by scale, with three changes for text. Each level gets its own codebook; nested dropout pushes earlier levels to carry meaning on their own rather than leaning on later ones; and a token-level reconstruction objective keeps the decoded text accurate. Depending on the dataset, NCM uses 8 to 32 levels with codebooks of 4,096 codes, against a text vocabulary of 8,192 to 151,680 tokens.

The second stage freezes that tokenizer and trains a transformer to predict one level at a time, conditioned on the cumulative reconstruction of everything before it. All codes inside a level are predicted in a single parallel pass. Training covers every level in one forward pass, with a fraction of the conditioning codes deliberately corrupted so the model does not fall apart when its own earlier predictions are wrong. At inference the levels are generated in order, and the tokenizer’s decoder converts the finished hierarchy into text.

Where the speed comes from

Diffusion language models win their parallelism by denoising the whole sequence repeatedly, but the number of passes is what limits them, and cutting passes degrades quality quickly. Distillation can reduce that count, at the cost of an extra training stage and lost sample diversity. NCM takes a different route: the sequence of steps is short because there are only as many steps as levels, and the early steps are cheap because early levels are short. In the 12-level configurations used for LM1B, TinyStories and Sudoku, generation takes 12 function evaluations, with most of the compute spent in the last few levels.

Line chart of generative perplexity against seconds per sequence on OpenWebText, with the NCM curve reaching comparable quality in about one tenth of the time
Generative perplexity against wall-clock generation time on OpenWebText, measured on a single NVIDIA H100. Image credit: Hazy Research, Stanford

The headline numbers, all from the paper: on LM1B, NCM reaches better generative perplexity than the strongest diffusion baseline while running 21 times faster; on OpenWebText it is competitive while running 10 times faster; on XSum summarization it matches the strongest baseline’s ROUGE-L while running 31 times faster. TinyStories and LM1B reach comparable quality with about 11 times fewer function evaluations. Counting both tokenizer and generator training, the team says NCM needs about 4.9 times fewer tokens than strong diffusion baselines on LM1B and about 4 times fewer on OpenWebText.

The results, and their limits

The reasoning result is the most interesting, because it tests the paper’s argument that generating an overall structure first helps tasks with global constraints. On Sudoku the baselines come from a prior comparison paper rather than being retrained here, and the parameter counts are close enough that size does not explain the gap:

Method Params Function evaluations Easy (40 clues) Medium (35) Hard (30)
Autoregressive (greedy) 25M 128 14.6% 5.1% 1.0%
MDLM (diffusion) 29M 128 92.0% 77.1% 30.2%
FLM 29M 128 94.2% 82.7% 44.5%
FMLM+ 29M 16 97.7% 92.8% 81.3%
NCM 25M 12 99.5% 96.5% 86.1%

Read the table with the paper’s own scope in mind. These are tens-of-millions-parameter models on benchmark datasets built for diffusion research, not frontier-scale language models; the largest NCM model in the scaling study is 725 million parameters, and the authors call those curves preliminary. The 21- and 31-fold speedups are measured against diffusion baselines on one H100, not against a production autoregressive serving stack with paged attention and batching. The VQ-VAE tokenizer is a second model with its own 15 to 144 million parameters in the paper’s configurations, and the authors note that tokenizers of this kind have failure cases in reconstruction. Nor has the work been peer-reviewed.

The authors are explicit about what they have not done: they have not tested how NCM scales in parameters, FLOPs and tokens; they trained the tokenizer and generator separately rather than jointly; and they use a fixed hierarchy for every input instead of learning when to stop or spend more computation. Each of those is a plausible source of further gains, and each is also a reason the current numbers are an early result rather than a settled one.

What This Means

Most work on faster generation tries to squeeze the number of denoising passes or to distill a big sampler into a small one. NCM changes the axis: it makes the representation itself hierarchical, so a short sequence of steps covers the whole output. If that holds up at larger scale, the practical argument is not that diffusion language models are beaten on quality, but that parallel text generation may not need a diffusion process at all. The test-time scaling result points the same way: because each sample is cheap, the model can draw more candidates within the same time budget, and minimum Bayes risk decoding over those candidates raised BLEU on WMT14 and ROUGE on XSum in the reported experiments.

For now, the honest summary is that a small research team has shown a genuinely different generation paradigm working at small scale, with released code and an unusually clear list of open questions. Whether it survives the jump to billions of parameters is the next paper’s problem.

This post was drafted with AI assistance and reviewed by RITS staff.

Sources

Related stories

AI

DFlash: Block Diffusion Delivers 6x Faster LLM Inference

DFlash is a new speculative decoding framework that uses block diffusion models to generate draft tokens in parallel rather than sequentially, achieving over 6× lossless acceleration on large language models — up to 2.5× faster than the previous state-of-the-art method EAGLE-3. The paper was published in February 2026 by Jian Chen, Yesheng Liang, and Zhijian […]

diffusion

Inception Launches Mercury 2: Diffusion-Powered Reasoning at 1,000 Tokens per Second

Key points On February 24, 2026, Inception launched Mercury 2, which the company calls the first reasoning-capable diffusion large language model and the fastest reasoning LLM available today. Inception reports that Mercury 2 reaches 1,009 tokens per second on NVIDIA Blackwell GPUs with 1.7 seconds of end-to-end latency, over five times faster than leading speed-optimized […]

benchmark

DLoop: Looped Speculative Decoding Cuts Target-Model Passes

NAVER AI Lab has proposed DLoop, a change to speculative decoding that lets a single verification pass cover several rounds of drafting instead of just one. In a preprint posted to arXiv on October 6, 2026, the team reports wall-clock speedups 5 to 41 percent higher than the draft models it builds on — EAGLE-3, […]

Never miss a RITS update

Subscribe to the RSS feed for AI, emerging technology, and digital scholarship news curated for the NYU Shanghai community, or get in touch to suggest a story.