Nake
Settings
Language
ThemeAtelier
Privacy policyTerms of serviceRefund policy
Premium-50%50% off
Lite

Premium membership from $19 per month

Premium benefitsMore benefits
Fast generationHD outputAll image toolsAll video tools
Nake
  • Home
  • For You
  • AI Video
  • AI Image
  • Anime / Manga
  • Video Edit
  • Video Extend
  • AI Tools
  • Generate
  • Agent
  • Explore
  • Home
  • For You
  • AI Tools

Create & chat

  • AI Video
  • AI Image
  • Anime / Manga
  • AgentBeta
  • Explore
  • Generate
  • Video Edit
  • Video Extend+5s
Upgrade for more
    • Home
    • For You
    • Create
    • Pricing
    • More

    deepseek deepinfer

    deepseek deepinfer is the high-performance inference framework that powers DeepSeek's large-scale production services. Released under the Apache 2.0 license, this open-source framework provides offline inference with core features needed for serving at scale, supporting DeepSeek-V4.1-Flash on both NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.

    Official documentation
    deepseek logo
    Publisher identity: deepseek logo.

    On this page

    deepseek deepinfer: Developer Overview

    DeepInfer is the production inference stack that DeepSeek uses to serve its V4.1-Flash model at scale, now available for developers to self-host and study.

    Production Provenance and Scale

    DeepSeek has open-sourced DeepInfer, the framework it describes as "the high-performance inference framework that powers DeepSeek's large-scale production services." The framework runs 155 trillion tokens per day per cluster in production with a 96% input cache hit rate, according to AI Weekly. This release provides offline inference with the core features needed for serving at scale, targeting DeepSeek-V4.1-Flash. The README credits its lineage directly: DeepInfer was "originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM." The repo currently shows 5.5k stars and 415 forks, and it lands into a busy week for the lab. The Apache 2.0 licensing permits commercial self-hosting, directly threatening managed inference pricing at providers like Fireworks, Together, and Groq.

    Core Feature Set and Custom Kernels

    The feature list reads as a bundle of DeepSeek's recent systems papers made operational. Custom kernels include FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels. Parallelism covers Attention DP with balanced CP plus MoE expert-parallel with EPLB. Decoding uses "DSpark and confidence-scheduled verification: accelerate decoding with draft tokens and use confidence estimates to schedule verification." An asynchronous zero-overhead batch scheduler sits alongside a batch-invariant inference mode, and the sampler is advertised to "Produce identical outputs for identical input tokens and sampling parameters." Generation can be constrained to regex, JSON schema, or EBNF grammar. DeepSeek says that future model inference code and new optimizations will land on this stack too.

    deepseek deepinfer: Setup and Integration

    Setting up DeepInfer requires specific hardware and software prerequisites, with build instructions for both NVIDIA and Ascend platforms.

    Prerequisites and Build Process

    Before building DeepInfer, you need to create and activate an empty Python 3.12 environment. Prerequisites include Linux with GCC >= 13.0, Python 3.12, and either CUDA 13 (for NVIDIA only) or CANN 9.2 (for Ascend only). The build process involves exporting the accelerator environment variable and running the build script. For NVIDIA, use `export DEEPINFER_ACCELERATOR=nvidia` and for Ascend use `export DEEPINFER_ACCELERATOR=ascend`. Then run `./build.sh`. The default checkpoint, `deepseek-ai/DeepSeek-V4.1-Flash-1017`, is downloaded into the Hugging Face cache on the first run. You can set `DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017` to run from a local copy instead. Offline inference can be tested with `python examples/offline_inference.py`.

    Hardware Support and Limitations

    DeepInfer supports two hardware platforms: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The Huawei Ascend 950 integration is the first time a frontier lab has shipped a production open-source stack that treats a Chinese chip as first-class hardware. However, the framework currently does not support consumer NVIDIA GPUs, meaning it is not suitable for local AI on typical desktop hardware. Required for a complete inference service but not included in this release are a KV cache system, VE/Prefill/Decode disaggregation (most required engine functionality is already implemented), and load balancing across data-parallel replicas for both prefill and decode. The framework is offline-optimized and not suited for real-time latency workloads.

    Availability and Licensing

    DeepInfer is publicly available under an open-source license, with clear terms for commercial use and community access.

    Open-Source Release and License

    DeepInfer is released under the Apache License 2.0, which permits commercial self-hosting. This licensing directly threatens managed inference pricing at providers like Fireworks, Together, and Groq. The repository is hosted on GitHub at `github.com/deepseek-ai/DeepInfer` and currently shows 5.5k stars and 415 forks. The release targets DeepSeek-V4.1-Flash and ships support for both Nvidia SM100 GPUs and Huawei Ascend 950 NPUs. The Apache 2.0 license allows developers to freely use, modify, and distribute the framework, making it accessible for a wide range of applications.

    Production Metrics and Performance

    DeepInfer runs 155 trillion tokens per day per cluster in production with a 96% input cache hit rate for serving context from memory. Performance benchmarks are available for prefill and decode, measured on 4 SM100 GPUs for prefill and 64 for decode — each within one scale-up domain. To reproduce the benchmarks, see `run_sweep.sh`. Evaluation benchmarks are also available in `eval/README.md`. These metrics demonstrate the framework's capability to handle massive-scale inference workloads efficiently.

    Configuration Options

    DeepInfer offers a range of configuration options to optimize performance, determinism, and output structure for different use cases.

    DSpark and Confidence-Scheduled Verification

    DSpark and confidence-scheduled verification accelerate decoding with draft tokens and use confidence estimates to schedule verification. To enable this feature, set `DEEPINFER_ENABLE_MTP_VARLEN=1`. The `DEEPINFER_USE_NEXTN_DECODE` variable controls the maximum draft length, with a default value of 5. This configuration allows the framework to generate tokens more efficiently by leveraging speculative decoding techniques.

    Batch-Invariant and Deterministic Sampling

    Batch-invariant inference keeps model results identical across different batch compositions. Enable this with `DEEPINFER_BATCH_INVARIANT=1` (NVIDIA only). Deterministic sampling produces identical outputs for identical input tokens and sampling parameters. To enable deterministic sampling, set `DEEPINFER_BATCH_INVARIANT=1` and `DEEPINFER_ENABLE_MTP_VARLEN=0`, and pass a fixed `seq_seed` to `SamplingParams` (NVIDIA only). These features are crucial for reproducibility in testing and production environments.

    Practical Notes

    Key considerations for developers working with DeepInfer, including structured output support and asynchronous scheduling.

    Structured Outputs and Asynchronous Scheduling

    DeepInfer supports structured outputs that constrain generation to a regex, JSON schema, or EBNF grammar. Pass `so_args={"schema_type": ..., "schema": ...}` to `SamplingParams`; see examples for details. Asynchronous scheduling overlaps CPU batch scheduling with device execution and is enabled by default. This feature reduces idle time and improves overall throughput by allowing the CPU to prepare the next batch while the GPU is processing the current one.

    Decoder SWA Bounded Replay

    Decoder SWA bounded replay reduces decoder prefill work by replaying only a bounded suffix. Set `DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact` and `DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140` for the default V4.1 settings. The retained suffix must cover the SWA window plus the maximum backtracking length. This optimization is particularly useful for long-context scenarios where full prefill would be computationally expensive.

    FAQ

    Frequently asked questions about deepseek deepinfer.

    What should I know about deepseek deepinfer?

    deepseek deepinfer is the high-performance inference framework that powers DeepSeek's large-scale production services. It is released under the Apache 2.0 license and supports DeepSeek-V4.1-Flash on both NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The framework includes custom kernels like FlashMLA, DeepGEMM, and DeepEP, and features such as DSpark speculative decoding, confidence-scheduled verification, and asynchronous scheduling. It runs 155 trillion tokens per day per cluster in production with a 96% input cache hit rate.

    What hardware does DeepInfer support?

    DeepInfer supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It does not support consumer NVIDIA GPUs, so it is not suitable for local AI on typical desktop hardware. The Huawei Ascend 950 integration is the first time a frontier lab has shipped a production open-source stack that treats a Chinese chip as first-class hardware.

    How do I get started with DeepInfer?

    To get started, create an empty Python 3.12 environment, ensure you have Linux with GCC >= 13.0, and either CUDA 13 (NVIDIA) or CANN 9.2 (Ascend). Export the accelerator environment variable (e.g., `export DEEPINFER_ACCELERATOR=nvidia`) and run `./build.sh`. The default checkpoint is downloaded automatically on first run. Test offline inference with `python examples/offline_inference.py`.

    Original sources

    • GitHub - deepseek-ai/DeepInfer: DeepInfer: High-performance, large-scale inference framework for DeepSeek models · GitHub
    • DeepSeek open-sources DeepInfer, its V4.1-Flash serving stack | AI Weekly
    • David Hendrickson on X: "DeepSeek open-sourced the inference engine behind its own production AI service. DeepInfer is the framework DeepSeek uses to serve V4.1 Flash and they're giving away production tricks such as ⚡ DSpark speculative decoding 🧠 confidence-scheduled verification 📚 SWA bounded… / X

    Related reading

    AI Tools DirectoryText to Image GenerationImage to Video Generation

    Read the sources for deepseek deepinfer

    Creative tools & scenes
    • dall e nsfw
    • cogvideox nsfw
    • claude nsfw
    • deepseek nsfw
    • dreamshaper nsfw
    • epicrealism nsfw