NInfer: 700 tok/s on One RTX 5090, With a Rare Honest Audit

July 28, 2026Provided by Utku Ege Tuluk

NInfer is a from-scratch C++/CUDA inference engine that runs exactly two model checkpoints on exactly one GPU. Published under Apache 2.0 and first committed on June 26, 2026, the project by developer Neroued supports only Qwen3.6-27B and Qwen3.6-35B-A3B, only on an NVIDIA GeForce RTX 5090, and only one request at a time. In return for that narrowness it reports 15,544 prefill tokens/sec and up to 714 decode tokens/sec on a single consumer card — and, unusually, it ships a 225-response audit showing where those throughput numbers do not correspond to usable output.

Advanced

A single consumer graphics card on a dark surface with two streams of glowing token blocks rising from it — one dense and teal, one sparse and amber — representing accepted and rejected speculative tokens — Illustration generated by AI

Specialization as a design constraint

Most open-source inference stacks compete on breadth: vLLM, SGLang, and llama.cpp all try to load arbitrary checkpoints across arbitrary hardware. NInfer inverts that. Its README states the position plainly — “Selected checkpoints. Maximum single-GPU inference performance” — and the build system enforces it, rejecting any CUDA architecture other than sm_120a, the RTX 5090’s compute capability. The requirements are correspondingly narrow: 64-bit Linux, a single RTX 5090, CUDA Toolkit 13.1 or newer, CMake 3.28+, a C++20 compiler, FFmpeg development libraries for video decoding, and libcurl. There is no install target and no packaged binary, so NInfer runs from its source build tree or the provided Docker image. On any other GPU it does not build at all.

Models are not loaded from Transformers, Safetensors, or GGUF. Each target is distributed as a single native .ninfer container with a published SHA-256 — 16.29 GiB for Qwen3.6-27B, 21.22 GiB for Qwen3.6-35B-A3B — bundling weights, vision tower, MTP proposal head, tokenizer, chat template, and media processor as registered objects. GPU residency is fixed at startup: speculative decoding and vision are off by default, and a capability not requested at launch cannot be enabled later.

The list of things NInfer does not implement is equally explicit: no continuous batching, no multi-GPU execution, no CPU/GPU offload, no distributed serving. One engine owns one resident sequence, with context configurable up to the models’ native 262,144-token limit as the 32 GiB card and KV-cache settings allow. What is present is the fast path — chunked prefill, CUDA Graph decode, BF16 and INT8 group-64 KV cache, compatible-prefix reuse, and roughly forty hand-written CUDA kernel benchmarks covering GQA attention, gated delta rule, sparse MoE, RoPE, and INT8 linear layers. Serving speaks both OpenAI Chat Completions and Anthropic Messages, with streaming, multimodal input, and tool-call parsing.

The measured numbers

All figures were collected on one RTX 5090 with CUDA 13.1, INT8 group-64 KV cache, CUDA Graphs enabled, and five fixed seeds per fixture after a warm-up; the two targets are reported independently, not as a comparison. With speculative decoding off, Qwen3.6-35B-A3B — a mixture-of-experts model with roughly 3B active parameters — sustains 15,544.3 prefill tok/s and 271.1 decode tok/s at a 7,680-token prompt, degrading gracefully to 5,157.1 prefill tok/s and 188.2 decode tok/s at 260,096 tokens, where time to first token is about 50.6 seconds. The dense Qwen3.6-27B, which must move every parameter per token, lands at 3,218.1 prefill and 77.6 decode tok/s at the short prompt.

Turning on multi-token prediction with a three-token draft window changes the decode picture substantially. On the 35B-A3B target, MTP3 reaches 695.1 tok/s at 83.3% acceptance on an AIME 2026 reasoning fixture and 714.3 tok/s at 87.7% acceptance on structured output — roughly 2.6× the non-speculative baseline. Prose is the weak case: story generation accepts only 38.2% of drafts and manages 434.9 tok/s.

./build/apps/ninfer models/qwen3_6_35b_a3b.ninfer \
  --prompt "Explain prefill and decode in three sentences." \
  --max-context 16384 --max-new 256 \
  --spec mtp --draft-tokens 3 --lm-head-draft

The 35B-A3B artifact also supports a second speculative backend, DFlash, using companion weights from z-lab. Its advantage is workload-dependent rather than uniform: at block=8 it gains +10.1% on structured output and +9.9% on one reasoning fixture, but loses 11.4% on code and 39.8% on story generation against MTP3. Capability scores, measured through NInfer’s own serving route with EvalScope 1.9.0 at 0-shot with thinking enabled, are single samples rather than pass@k: the 27B scores 86.67% on AIME 2025, 93.33% on AIME 2026, and 86.87% on GPQA-Diamond; the 35B-A3B scores 90.00%, 90.00%, and 85.35%.

One of NInfer’s committed multimodal CLI fixtures — synthetic images designed to probe counting and spatial reasoning through the vision path. Image credit: Neroued/ninfer

A synthetic test image with three red circles labeled for counting and a blue square with an arrow pointing to a green triangle, used as a vision input fixture — One of NInfer’s committed multimodal CLI fixtures — synthetic images designed to probe counting and spatial reasoning through the vision path. Image credit: Neroued/ninfer

The audit most benchmarks omit

The most interesting section of NInfer’s performance document is not a throughput table. It is an audit of all 225 stored responses from the speculative-decoding campaigns, checking termination, exact repetition, and per-fixture structural requirements — and it repeatedly contradicts the headline numbers.

The clearest case: under greedy DFlash, one AIME 2026 fixture recorded 994.9 tok/s at 98.0% acceptance, the fastest decode rate in the entire corpus. The audit discloses that this generation is a deterministic repetition loop — the line Wait, $x_7 x_1 x_3$ is $x_7 x_1 x_3$. appears 2,406 times among 2,538 non-empty reasoning lines, the answer field is empty, and the number is explicitly excluded from comparisons. High acceptance can be a symptom of pathological predictability rather than evidence of speed.

The code category tells a similar story from the other direction. It posts a respectable 635.0 tok/s under MTP3, but only 1 of 15 samples stops naturally and zero satisfy the prompt’s requirement for complete runnable multi-file deliverables. Structured output — the highest-acceptance, highest-throughput category at 714.3 tok/s — produces 49 to 60 valid JSONL records against a requested 160, and 0 of 15 satisfy the full contract. Translation is the one clean case: 15 of 15 natural stops, 15 of 15 passing every structural check. The document draws the boundary itself: “Decode throughput is a transport/execution measurement, not a correctness score.”

Why this matters

NInfer’s engineering result is real — a single hobbyist-accessible card driving a 35B-class MoE at roughly 700 tok/s with a quarter-million-token context window is a meaningful data point for anyone running local inference. But the project’s more transferable contribution is methodological. Speculative decoding is now standard across the stack, and acceptance rate has become a headline metric; NInfer demonstrates that acceptance rate and tokens/sec can both peak precisely where the model has stopped doing useful work. A benchmark table without a completion audit cannot distinguish the two.

The caveats are obvious: one developer, 140 stars, roughly a month of history, locked to one GPU SKU and two checkpoints, with no batching. It is a demonstration of a ceiling, not a serving platform, and the numbers are self-reported on hardware most readers cannot easily match. Still, the reproduction commands, git revisions, seeds, and standard deviations are all committed to the repository — more than most performance claims in this space offer.

Related Coverage

Luce DFlash Brings 2x Speculative Decoding to Qwen3.6-27B on a Single RTX 3090 — the DFlash method NInfer uses as its second speculative backend
Qwen3.6-35B-A3B: Alibaba Open-Sources a Frontier-Class Agentic Coder — the MoE checkpoint behind NInfer’s fastest target
Qwen3.6-27B: A Dense 27B Model That Beats a 397B MoE on Coding — NInfer’s second registered checkpoint
JetSpec: Causal Parallel Tree Drafting Hits 9.64x Faster LLM Inference — recent work pushing speculative decoding acceptance rates higher
Gemma 4 Gets Multi-Token Prediction Drafters: 3x Faster Inference, Same Outputs — background on the MTP approach NInfer implements

This post was drafted with AI assistance and reviewed by RITS staff.

Sources

Neroued/ninfer on GitHub — repository README, capabilities, and current limits
NInfer: Single-GPU serving performance — full methodology, per-fixture results, and output audit
neroued/Qwen3.6-35B-A3B-NInfer — artifact and model card
neroued/Qwen3.6-27B-NInfer — artifact and model card
NInfer evaluation harness — EvalScope configuration used for AIME and GPQA scores