DeepSeek open-sources DeepInfer inference framework

DeepSeek open-sourced DeepInfer on October 20, 2026 under the Apache 2.0 license, releasing the inference framework that runs its production services and powers DeepSeek-V4.1-Flash. A single DeepInfer cluster serves 155 trillion tokens per day, achieving above 25,000 output tokens per second per GPU at scale. The framework supports offline inference only and currently supports one model, unlike vLLM's 200+ architectures.

DeepInfer release details

• ▪DeepInfer supports one model, DeepSeek-V4.1-Flash
• ▪DeepInfer ships under the Apache 2.0 license
• ▪DeepSeek open-sourced DeepInfer on October 20, 2026
• ▪DeepSeek released deepseek-recipe, an MIT-licensed token encoding and decoding layer
• ▪DeepInfer is the inference framework that runs DeepSeek's production services

Technical architecture

• ▪DeepInfer's SWA bounded replay recomputes only the last 128 tokens of sliding-window attention state on a cache hit
• ▪DeepInfer was originally built primarily on vLLM, with ideas from SGLang and TensorRT-LLM
• ▪DeepInfer uses data parallelism for attention and expert parallelism with an expert-parallel load balancer for MoE layers
• ▪DeepSeek-V4.1-Flash supports a 1M-token context window per the model card
• ▪DeepInfer supports structured outputs including regex, JSON schema and EBNF grammars
• ▪DeepInfer stacks DeepSeek's kernels including FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels
• ▪DeepInfer's DSpark feature uses a draft model that proposes 5 tokens per step with a Confidence Head predicting each draft token's acceptance rate

Performance benchmarks

• ▪DeepInfer's internal FlashMLA decoding kernel is 5 to 10 percent faster than the public FlashMLA kernel
• ▪DeepInfer achieves above 25,000 output tokens per second per GPU at 256 requests per rank with EP64 and 131,072 context
• ▪DeepInfer's DSpark cuts verified draft tokens by about 30 percent, from 5.00 to 3.2 to 4.4
• ▪A single DeepInfer cluster serves 155 trillion tokens per day
• ▪DeepSeek reports overall revenue of US$38 per GPU-hour at valley pricing for DeepInfer
• ▪DeepInfer achieves about 60,000 non-cached tokens per second per GPU at 4,096 tokens per rank for prefill with EP4DP4 and 140-token bounded replay
• ▪DeepInfer's SWA bounded replay nearly halves total prefill compute in decoder layers

Framework comparison

• ▪DeepInfer is offline inference only while vLLM and SGLang provide OpenAI-compatible servers
• ▪DeepInfer supports 1 model, DeepSeek-V4.1-Flash, while vLLM supports 200+ architectures
• ▪DeepInfer uses DSpark confidence-scheduled speculative decoding while vLLM uses n-gram, suffix, EAGLE, and DFlash methods

Hardware requirements

• ▪DeepInfer's batch-invariant inference and deterministic sampling are available on NVIDIA only
• ▪DeepInfer builds require Linux, GCC 13 or later, Python 3.12, and CUDA 13 or CANN 9.2
• ▪DeepInfer runs on NVIDIA SM100 Blackwell GPUs and Huawei Ascend 950 NPUs
• ▪DeepInfer's smallest efficient production unit is 64 prefill GPUs plus 64 decode GPUs
• ▪DeepInfer's benchmarks for DeepSeek-V4.1-Flash used 4 GPUs for prefill and 64 for decode

Production deployment limitations

• ▪DeepInfer does not include load balancing across data-parallel replicas in this October 20, 2026 open-source release
• ▪DeepInfer supports offline inference only, not online serving