AdvancedAI
Eagle Ships EAGLE-4: One Drafter for Every Batch Size
- eagle
- inference
- llm
- nvidia
- speculative decoding
Eagle released EAGLE-4 on October 20, 2026, the fourth generation of its speculative decoding family, built around a single claim: one trained drafter can serve a lone request and a fully loaded server without being retrained or hand-tuned. The release ships code, a paper, and plugins for three inference engines, and reports gains that hold up at 512 concurrent requests — a regime where earlier drafters lose ground.
One Drafter, Two Regimes
Speculative decoding works by having a small model guess several tokens ahead and letting the large model verify them in one pass. The guesses are only worth making if enough of them survive verification, and the cost of drafting rises with the size of the speculation tree. Previous EAGLE generations tuned that tree for a benchmark configuration — usually a single request — and left server operators to re-calibrate when their batch size changed.
EAGLE-4 makes the tree a runtime decision. eagle-harness measures what drafting and verification actually cost on the hardware in front of it, then sizes the speculation from those measurements for the batch size and workload it is serving. The drafter itself is unchanged between the two regimes; on Qwen3.8-27B the same trained weights produce both the single-request and the 512-request numbers below.
The comparison table is the argument. Against vanilla decoding, DSpark and DFlash 2 all look respectable at concurrency 1 — 2.57× and 2.80× — and then collapse at concurrency 512, where they deliver 0.67× and 0.78× of vanilla throughput. A drafter tuned for an empty server actively costs throughput on a busy one. EAGLE-4 is the only method in the table that improves on vanilla in both columns.
What Changed in EAGLE-4
Three architectural changes support the adaptivity. The draft layer uses a sliding window, so the drafter attends over a bounded recent context instead of the whole sequence. The candidate head is low-rank, which reduces the cost of scoring a large tree of candidate tokens — the EAGLE team calls this the Efficient LM Head. And the draft cache is append-only, which avoids rewriting state as the tree changes shape between rounds.
Dense and latent mixture-of-experts draft structures are both implemented, and the paper covers targets with GDN and Mamba2 layers, not just standard attention. Verification keeps the target model’s output distribution, which is the property that makes speculative decoding safe to ship: a greedy mismatch fails the release’s correctness gate even when the speculative server is faster.
The Vendor Numbers
On Qwen3.8-27B, four NVIDIA B300 GPUs and tensor parallelism 4, EAGLE-4 reaches 863.8 tokens per second for a single request against 240.5 for vanilla — 3.59×, and 1.28× ahead of DFlash 2. At 512 concurrent requests it sustains 28,340 aggregate tokens per second against 21,203 for vanilla, 1.34×, while DFlash 2 falls to 16,641.
The pattern repeats on a mixture-of-experts target. Nemotron 3.5 Super VL, a 120B model with 12B active parameters, goes from 236 to 647 tokens per second at one request (2.74×) and from 19,823 to 33,480 aggregate tokens per second at 512 (1.69×). All of these are paper measurements on a specific SPEED-Bench split at temperature 0, and none has been independently reproduced.
AMD ran its own evaluation as day-0 support, on a single Instinct MI350X with no tensor parallelism: 75.27 to 262.89 tokens per second at one request, a 3.49× speedup, decaying to 1.70× at 32 concurrent requests. The decay is the honest part of the chart. Speedups that shrink as the batch fills are what speculative decoding looks like once the target model is no longer the only bottleneck.
What This Means
Speculative decoding has spent two years being benchmarked at batch size one, because that is where the numbers are largest. Serving economics run the other way: a production endpoint is judged on aggregate throughput at high concurrency, and a method that helps the demo while hurting the fleet is worse than no method at all. EAGLE-4’s contribution is to treat the speculation budget as a scheduling variable rather than a model property.
The licence deserves equal attention. Eagle describes the release as source available and ships a commercial licence agreement next to the code; the weights for the Qwen drafter are on Hugging Face under a separate custom licence. That is a different posture from DeepSeek’s Apache 2.0 or Kyutai’s MIT, and it means the components are readable and testable but not unambiguously redistributable. Teams evaluating it for production should read the agreement before benchmarking it, not after.
This post was drafted with AI assistance and reviewed by RITS staff.
沪公网安备31011502017015号