DeepSeek Open-Sources DeepInfer, Its Production Inference Engine

DeepSeek published DeepInfer on GitHub on October 20, 2026, releasing under Apache 2.0 the inference framework it says powers its own large-scale production services. The release covers offline inference for a single model, DeepSeek-V4.1-Flash, and arrives with the kernel stack, parallelism strategy and scheduler DeepSeek uses internally rather than a wrapper around someone else’s server.

What Is in the Box

DeepInfer is not a general-purpose server. Its README lists exactly one supported model — deepseek-ai/DeepSeek-V4.1-Flash-1017, the checkpoint DeepSeek shipped on October 17 — and describes the scope as “offline inference with the core features needed for serving at scale”. The bundled example runs a batch job; there is no HTTP front end.

What it does include is the layer that decides throughput on expensive hardware. The kernel set is DeepSeek’s own published work: FlashMLA for attention, DeepGEMM for matrix multiplication, DeepEP for expert-parallel communication, plus TileKernels, DeepSelect and custom TileLang kernels. Parallelism is attention data-parallel or balanced context-parallel on the prefill side, and expert-parallel with expert-parallel load balancing (EPLB) on the decode side.

Three scheduling features are worth naming. DSpark speculative decoding uses confidence estimates to decide how many draft tokens to verify, with a maximum draft length that defaults to five. SWA bounded replay cuts decoder prefill work by replaying only a bounded suffix of the context — 140 tokens by default for V4.1. And an asynchronous batch scheduler overlaps CPU-side batching with device execution, enabled by default, which is the standard fix for a host process that cannot keep a fast GPU fed.

The framework also offers batch-invariant inference and deterministic sampling, both NVIDIA-only, and structured outputs constrained to a regex, JSON Schema or EBNF grammar. Batch invariance means the same prompt returns the same logits regardless of what else is in the batch — a property worth stating explicitly, because it is often not true.

The Numbers

DeepSeek’s benchmarks are its own, measured on 4 SM100 GPUs for prefill and 64 for decode, each within a single scale-up domain. The decode run uses expert parallelism across 64 ranks with a 131,072-token context, three fixed draft tokens and a measured acceptance rate of 75%, for an average of 3.25 accepted tokens per verification round.

At 256 requests per data-parallel rank, throughput reaches about 26,000 output tokens per second per GPU, with each individual user seeing roughly 105 tokens per second. Dropping to 64 requests per rank cuts throughput to about 15,700 tokens per second per GPU while raising per-user speed to roughly 250 tokens per second. The curve is the usual trade: aggregate throughput and interactivity pull in opposite directions, and the useful operating point depends on whether the workload is a chat product or a batch job.

Prefill scales more gently. On four GPUs, a 4,096-token batch runs at about 60,000 tokens per second per GPU with a 63 ms step; at 65,536 tokens the step takes about 730 ms and throughput rises to roughly 89,000 tokens per second per GPU. That flattening is the signature of a memory-bandwidth-bound kernel — the larger batch amortises weight reads, and beyond a point there is little left to amortise.

What Is Missing

The README is unusually direct about the gap between this release and a service. Three components are listed as required but not included: a KV cache system, VE/prefill/decode disaggregation, and load balancing across data-parallel replicas for both prefill and decode. Disaggregation is described as mostly implemented but not shipped; the other two are absent.

That matters for how the release should be read. A KV cache system is what lets a server hold conversation state across requests, and disaggregation is what lets prefill and decode run on separate pools sized for their different bottlenecks. Without them, DeepInfer is a very fast single-shot engine and a reference implementation of DeepSeek’s kernels — not something an operator can drop in front of users.

What This Means

DeepSeek has now open-sourced both halves of its stack in the space of a week: the V4.1-Flash weights on October 17 and the serving framework on October 20. The framework credits vLLM as its original foundation, with SGLang and TensorRT-LLM as influences, which makes the release a return contribution to projects it built on rather than a fork war.

The Ascend 950 target is the detail with the longest reach. DeepInfer ships separate requirement files and a build path for Huawei’s NPUs alongside the NVIDIA one, at CUDA 13 and CANN 9.2 respectively. Whatever the benchmark numbers say on SM100 hardware, a production framework that treats a domestic accelerator as a first-class build target is a statement about where DeepSeek expects its inference to run. Reproducing the charts on either platform is left to whoever has 64 of those GPUs.

This post was drafted with AI assistance and reviewed by RITS staff.

Sources

• GitHub: deepseek-ai/DeepInfer repository and README (Apache 2.0)
• DeepInfer: decode throughput benchmark chart
• DeepInfer: prefill throughput benchmark chart
• SiliconANGLE: “DeepSeek open-sources DeepInfer, the inference framework behind its production services” (October 20, 2026)
• Hugging Face: DeepSeek-V4.1-Flash-1017 model card