deepseek deepinfer
deepseek deepinfer is the high-performance inference framework that powers DeepSeek's large-scale production services. Released under the Apache 2.0 license, this open-source framework provides offline inference with core features needed for serving at scale, supporting DeepSeek-V4.1-Flash on both NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.
Official documentationdeepseek deepinfer: Developer Overview
DeepInfer is the production inference stack that DeepSeek uses to serve its V4.1-Flash model at scale, now available for developers to self-host and study.
Production Provenance and Scale
DeepSeek has open-sourced DeepInfer, the framework it describes as "the high-performance inference framework that powers DeepSeek's large-scale production services." The framework runs 155 trillion tokens per day per cluster in production with a 96% input cache hit rate, according to AI Weekly. This release provides offline inference with the core features needed for serving at scale, targeting DeepSeek-V4.1-Flash. The README credits its lineage directly: DeepInfer was "originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM." The repo currently shows 5.5k stars and 415 forks, and it lands into a busy week for the lab. The Apache 2.0 licensing permits commercial self-hosting, directly threatening managed inference pricing at providers like Fireworks, Together, and Groq.
Core Feature Set and Custom Kernels
The feature list reads as a bundle of DeepSeek's recent systems papers made operational. Custom kernels include FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels. Parallelism covers Attention DP with balanced CP plus MoE expert-parallel with EPLB. Decoding uses "DSpark and confidence-scheduled verification: accelerate decoding with draft tokens and use confidence estimates to schedule verification." An asynchronous zero-overhead batch scheduler sits alongside a batch-invariant inference mode, and the sampler is advertised to "Produce identical outputs for identical input tokens and sampling parameters." Generation can be constrained to regex, JSON schema, or EBNF grammar. DeepSeek says that future model inference code and new optimizations will land on this stack too.
deepseek deepinfer: Setup and Integration
Setting up DeepInfer requires specific hardware and software prerequisites, with build instructions for both NVIDIA and Ascend platforms.
Prerequisites and Build Process
Before building DeepInfer, you need to create and activate an empty Python 3.12 environment. Prerequisites include Linux with GCC >= 13.0, Python 3.12, and either CUDA 13 (for NVIDIA only) or CANN 9.2 (for Ascend only). The build process involves exporting the accelerator environment variable and running the build script. For NVIDIA, use `export DEEPINFER_ACCELERATOR=nvidia` and for Ascend use `export DEEPINFER_ACCELERATOR=ascend`. Then run `./build.sh`. The default checkpoint, `deepseek-ai/DeepSeek-V4.1-Flash-1017`, is downloaded into the Hugging Face cache on the first run. You can set `DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017` to run from a local copy instead. Offline inference can be tested with `python examples/offline_inference.py`.
Hardware Support and Limitations
DeepInfer supports two hardware platforms: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The Huawei Ascend 950 integration is the first time a frontier lab has shipped a production open-source stack that treats a Chinese chip as first-class hardware. However, the framework currently does not support consumer NVIDIA GPUs, meaning it is not suitable for local AI on typical desktop hardware. Required for a complete inference service but not included in this release are a KV cache system, VE/Prefill/Decode disaggregation (most required engine functionality is already implemented), and load balancing across data-parallel replicas for both prefill and decode. The framework is offline-optimized and not suited for real-time latency workloads.
Availability and Licensing
DeepInfer is publicly available under an open-source license, with clear terms for commercial use and community access.
Open-Source Release and License
DeepInfer is released under the Apache License 2.0, which permits commercial self-hosting. This licensing directly threatens managed inference pricing at providers like Fireworks, Together, and Groq. The repository is hosted on GitHub at `github.com/deepseek-ai/DeepInfer` and currently shows 5.5k stars and 415 forks. The release targets DeepSeek-V4.1-Flash and ships support for both Nvidia SM100 GPUs and Huawei Ascend 950 NPUs. The Apache 2.0 license allows developers to freely use, modify, and distribute the framework, making it accessible for a wide range of applications.
Production Metrics and Performance
DeepInfer runs 155 trillion tokens per day per cluster in production with a 96% input cache hit rate for serving context from memory. Performance benchmarks are available for prefill and decode, measured on 4 SM100 GPUs for prefill and 64 for decode — each within one scale-up domain. To reproduce the benchmarks, see `run_sweep.sh`. Evaluation benchmarks are also available in `eval/README.md`. These metrics demonstrate the framework's capability to handle massive-scale inference workloads efficiently.
Configuration Options
DeepInfer offers a range of configuration options to optimize performance, determinism, and output structure for different use cases.
DSpark and Confidence-Scheduled Verification
DSpark and confidence-scheduled verification accelerate decoding with draft tokens and use confidence estimates to schedule verification. To enable this feature, set `DEEPINFER_ENABLE_MTP_VARLEN=1`. The `DEEPINFER_USE_NEXTN_DECODE` variable controls the maximum draft length, with a default value of 5. This configuration allows the framework to generate tokens more efficiently by leveraging speculative decoding techniques.
Batch-Invariant and Deterministic Sampling
Batch-invariant inference keeps model results identical across different batch compositions. Enable this with `DEEPINFER_BATCH_INVARIANT=1` (NVIDIA only). Deterministic sampling produces identical outputs for identical input tokens and sampling parameters. To enable deterministic sampling, set `DEEPINFER_BATCH_INVARIANT=1` and `DEEPINFER_ENABLE_MTP_VARLEN=0`, and pass a fixed `seq_seed` to `SamplingParams` (NVIDIA only). These features are crucial for reproducibility in testing and production environments.
Practical Notes
Key considerations for developers working with DeepInfer, including structured output support and asynchronous scheduling.
Structured Outputs and Asynchronous Scheduling
DeepInfer supports structured outputs that constrain generation to a regex, JSON schema, or EBNF grammar. Pass `so_args={"schema_type": ..., "schema": ...}` to `SamplingParams`; see examples for details. Asynchronous scheduling overlaps CPU batch scheduling with device execution and is enabled by default. This feature reduces idle time and improves overall throughput by allowing the CPU to prepare the next batch while the GPU is processing the current one.
Decoder SWA Bounded Replay
Decoder SWA bounded replay reduces decoder prefill work by replaying only a bounded suffix. Set `DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact` and `DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140` for the default V4.1 settings. The retained suffix must cover the SWA window plus the maximum backtracking length. This optimization is particularly useful for long-context scenarios where full prefill would be computationally expensive.
FAQ
Frequently asked questions about deepseek deepinfer.
What should I know about deepseek deepinfer?
deepseek deepinfer is the high-performance inference framework that powers DeepSeek's large-scale production services. It is released under the Apache 2.0 license and supports DeepSeek-V4.1-Flash on both NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The framework includes custom kernels like FlashMLA, DeepGEMM, and DeepEP, and features such as DSpark speculative decoding, confidence-scheduled verification, and asynchronous scheduling. It runs 155 trillion tokens per day per cluster in production with a 96% input cache hit rate.
What hardware does DeepInfer support?
DeepInfer supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It does not support consumer NVIDIA GPUs, so it is not suitable for local AI on typical desktop hardware. The Huawei Ascend 950 integration is the first time a frontier lab has shipped a production open-source stack that treats a Chinese chip as first-class hardware.
How do I get started with DeepInfer?
To get started, create an empty Python 3.12 environment, ensure you have Linux with GCC >= 13.0, and either CUDA 13 (NVIDIA) or CANN 9.2 (Ascend). Export the accelerator environment variable (e.g., `export DEEPINFER_ACCELERATOR=nvidia`) and run `./build.sh`. The default checkpoint is downloaded automatically on first run. Test offline inference with `python examples/offline_inference.py`.
Original sources
- GitHub - deepseek-ai/DeepInfer: DeepInfer: High-performance, large-scale inference framework for DeepSeek models · GitHub
- DeepSeek open-sources DeepInfer, its V4.1-Flash serving stack | AI Weekly
- David Hendrickson on X: "DeepSeek open-sourced the inference engine behind its own production AI service. DeepInfer is the framework DeepSeek uses to serve V4.1 Flash and they're giving away production tricks such as ⚡ DSpark speculative decoding 🧠 confidence-scheduled verification 📚 SWA bounded… / X