# DeepInfer DeepInfer is the inference framework behind [[Deepseek]]'s own large-scale API and chat service. DeepSeek open-sourced it on 20 October 2026 under the [[Apache 2.0 License]]. Not to be confused with DeepInfra, the model hosting company: one letter apart, no relation. For now it supports exactly one model, [[DeepSeek V4.1 Flash]], on two hardware families: NVIDIA SM100 GPUs (Blackwell) and Huawei Ascend 950 NPUs. In April 2025, DeepSeek said it would NOT publish its engine. In "The Path to Open-Sourcing the DeepSeek Inference Engine", they gave three reasons: the engine was an early fork of [[vLLM]] that had drifted far from upstream, it was tightly coupled to their cluster tooling, and a small research team couldn't maintain a big open source project. Their plan was to contribute features upstream instead. Eighteen months later, the engine is public anyway, and DeepSeek says the inference code for future DeepSeek models will land in DeepInfer too. ## What's in the release The README is clear about the scope: this is **offline inference** (a Python batch API), with "the core features needed for serving at scale". There is no HTTP server. - Kernels from DeepSeek's own libraries, pulled in as git submodules: FlashMLA, DeepGEMM and DeepEP (plus Ascend ports of the last two), TileKernels, DeepSelect, and custom TileLang kernels - Data parallelism and balanced context parallelism for attention; expert parallelism with an expert-parallel load balancer (EPLB) for the [[AI Mixture of Experts (MoE)|MoE]] layers - DSpark [[AI Speculative Decoding|speculative decoding]] with confidence-scheduled verification (up to 5 draft tokens by default) - SWA bounded replay, which rebuilds the sliding-window part of the cache by recomputing a short suffix of tokens instead of storing it - An asynchronous "zero-overhead" batch scheduler that prepares step N+1 on the CPU while the GPU runs step N (DeepSeek credits SGLang and NanoFlow for the approach) - Batch-invariant inference and deterministic sampling: same input and seed, same output, whatever else is in the batch (NVIDIA only) - [[LLM Structured Outputs|Structured outputs]] constrained by a regex, a JSON schema or an EBNF grammar What's missing is everything that turns an engine into a service: the [[AI KV Cache|KV cache]] storage system, the orchestration of the disaggregated vision encoder / prefill / decode deployment ("most required engine functionality is already implemented"), and load balancing across data-parallel replicas. DeepSeek says those parts depend on its internal infrastructure; it describes their design in a Zhihu post and in a deployment Q&A shipped with the repo. To build it, you need Linux, GCC 13+, Python 3.12, and CUDA 13 or CANN 9.2. The default checkpoint (`DeepSeek-V4.1-Flash-1017`) is downloaded from [[HuggingFace]] on first run. ## How DeepSeek runs it in production The deployment Q&A (`doc/inference_deployment_qa.md`) is the part I'd read first. Frontier labs rarely explain how they actually serve a model, and this one goes deep. All numbers below are DeepSeek's: - Prefill runs on ordinary 8-GPU machines with small expert parallelism (EP2 to EP8). Decode runs on 64 to 128-GPU supernodes with "Wide-EP" (EP64 to EP128), because spreading the experts wide cuts the memory traffic per step. The smallest efficient deployment is 64 GPUs of prefill plus 64 of decode - One of the three design goals is keeping the KV cache for more than 72 hours, so repeated context doesn't have to be prefilled again. It lives on a dedicated 3FS cluster with roughly 100 PiB of SSD, 10+2 erasure coding, and a TTL of 3 days for agent traffic and 7 days for chatbot traffic - FP4 is used for the main KV cache, the indexer keys, and the routed-expert weights (MXFP4) - DSpark's variable-length verification checks about 30% fewer draft tokens than a fixed 5, for 4 to 7% more throughput at the production operating point (about 60 concurrent sequences per GPU) - Their load balancing reaches about 80% of a perfectly balanced offline benchmark - The internal FlashMLA has one hand-tuned SASS optimization the public version lacks, which makes the internal decoding kernel 5 to 10% faster. End to end, internal and open versions are within 5% - One cluster serves 155 trillion tokens per day, with revenue of US$38 per GPU-hour at off-peak ("valley") pricing The README's own benchmarks are two throughput-versus-latency Pareto charts (4 SM100 GPUs for prefill, 64 for decode), with a sweep script to reproduce them. There's no table of raw numbers, and no head-to-head comparison with any other engine. ## How it compares to vLLM, SGLang, TensorRT-LLM and llama.cpp DeepSeek publishes no comparison. The acknowledgments say DeepInfer "was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM". The rest of this section is my own read: - [[vLLM]] and [[SGLang]] are general-purpose servers: dozens of model families, lots of hardware, an OpenAI-compatible HTTP API, and the surrounding pieces (prefix caching, routing) included. DeepInfer is one model, two chip families, offline only. If you want to serve V4.1 Flash to users today, vLLM or SGLang is still the practical choice - TensorRT-LLM is NVIDIA's engine and runs on NVIDIA only. DeepInfer treats Huawei Ascend as a first-class target; "support accelerators from multiple vendors" is the first of DeepSeek's three stated goals for the system - [[llama.cpp]] is at the opposite end. DeepInfer's smallest efficient deployment is 128 datacenter GPUs, so it does nothing for [[Running AI Models Locally]] I think the real value is as a reference. This is the code DeepSeek runs, so it's the ground truth for how V4.1 Flash should behave: bounded replay, DSpark verification, determinism. Other engines can read it and match it, which fits the 2025 promise to sync inference work with the community before each model launch. ## From Open Source Week to a full engine During its February 2025 Open Source Week, DeepSeek released one infrastructure repo per day: FlashMLA (day 1), DeepEP (day 2), DeepGEMM (day 3), DualPipe and EPLB (day 4), and the 3FS file system (day 5), followed by an overview of the DeepSeek-V3/R1 inference system. DeepInfer is the engine those pieces came out of: the kernel libraries are vendored into it, EPLB is part of its parallelism, and 3FS holds the KV cache in the production setup the Q&A describes. Alongside DeepInfer, DeepSeek released deepseek-recipe (MIT license), the Rust layer (with Python bindings) that converts Chat Completions, Responses and Messages API requests into token sequences and parses model output back. DeepSeek says it is fully aligned with the official API, so third-party hosts get the same behavior, including protection against special-token injection. ## My take - It's a reference implementation more than a product. The code landed as a single initial commit on 20 October 2026, there are no tagged releases yet, and GitHub issues are turned off, so expect breaking changes - Batch invariance and deterministic sampling in a production engine is a big deal for evaluations and RL. Most servers can't promise you the same output twice once batching kicks in - The skipped parts (KV cache storage, load balancers) are where most of the operational difficulty lives. The Q&A tells you how DeepSeek built them; you still have to build them yourself ## References - DeepInfer repository: https://github.com/deepseek-ai/DeepInfer - README: https://github.com/deepseek-ai/DeepInfer/blob/main/README.md - Deployment Q&A: https://github.com/deepseek-ai/DeepInfer/blob/main/doc/inference_deployment_qa.md - License (Apache 2.0): https://github.com/deepseek-ai/DeepInfer/blob/main/LICENSE - Evaluation scripts (AIME 2026, HLE, GPQA Diamond): https://github.com/deepseek-ai/DeepInfer/blob/main/eval/README.md - Inference system blog post (Zhihu, Chinese): https://zhuanlan.zhihu.com/p/2095560295930508953 - TileLang and inference kernel libraries announcement (WeChat, Chinese): https://mp.weixin.qq.com/s/X41mKH4Ds-VXUAnK6M8Eww - deepseek-recipe: https://github.com/deepseek-ai/deepseek-recipe - Default checkpoint: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017 - The Path to Open-Sourcing the DeepSeek Inference Engine (April 2025): https://github.com/deepseek-ai/open-infra-index/blob/main/OpenSourcing_DeepSeek_Inference_Engine/README.md - DeepSeek Open Infra index (Open Source Week, February 2025): https://github.com/deepseek-ai/open-infra-index - FlashMLA: https://github.com/deepseek-ai/FlashMLA - DeepEP: https://github.com/deepseek-ai/DeepEP - DeepGEMM: https://github.com/deepseek-ai/DeepGEMM - EPLB: https://github.com/deepseek-ai/EPLB - 3FS: https://github.com/deepseek-ai/3FS - DeepSelect: https://github.com/deepseek-ai/DeepSelect - TileKernels: https://github.com/deepseek-ai/TileKernels ## Related - [[Deepseek]] - [[DeepSeek V4.1 Flash]] - [[DeepSeek v4]] - [[DeepSeek V3]] - [[AI Inference]] - [[AI KV Cache]] - [[AI Mixture of Experts (MoE)]] - [[AI Speculative Decoding]] - [[AI Sparse Attention]] - [[LLM Structured Outputs]] - [[vLLM]] - [[SGLang]] - [[llama.cpp]] - [[Running AI Models Locally]] - [[AI Open Weight Models]] - [[HuggingFace]] - [[Apache 2.0 License]] - [[Open Source]]