Everything About DeepSeek DeepInfer

DeepInfer is the inference framework DeepSeek uses to run its own AI models in production. DeepSeek released it as open source on GitHub on October 20, 2026, under the Apache 2.0 license.

An inference engine is the software that runs a trained model and serves its answers quickly and cheaply. It decides how work is split across chips, how memory gets used, and how many requests run at the same time.

Training builds the model. Inference is where most of the running cost sits.

Models

Only DeepSeek-V4.1-Flash, at launch

Hardware

Nvidia SM100 GPUs and Huawei Ascend 950 chips

Mode

Offline inference

License

Apache 2.0, free to use commercially

• Kernels optimized for DeepSeek’s model architecture
• Parallelism for attention and mixture-of-experts layers
• Asynchronous scheduling
• Batch-invariant inference, so results stay the same at any batch size
• Deterministic sampling, for reproducible outputs
• Structured outputs
• Speculative decoding, which drafts several tokens ahead and verifies them in a single pass

DeepInfer is not a complete serving system. DeepSeek says you provide:

• Your own KV cache system
• Prefill and decode disaggregation
• Load balancing across your replicas

That makes it a strong engine, not a plug-and-play API server.

DeepSeek says DeepInfer started out built mainly on vLLM, borrowed ideas from SGLang and TensorRT-LLM, and was later heavily rewritten. Its speed claims come from DeepSeek alone and have not yet been independently tested.

vLLM

Widely used, with support for many models and hardware and full serving features

SGLang

Fast serving with strong scheduling and support for structured output

TensorRT-LLM

Nvidia’s optimized engine for its own GPUs

DeepInfer

Tuned specifically for DeepSeek models on particular chips

• Teams running DeepSeek models at large scale
• Infrastructure engineers with supported hardware
• Researchers who study efficient inference
• Not beginners who simply want to chat with a model

• Read the README at github.com/deepseek-ai/DeepInfer
• Confirm that your hardware is supported
• Begin with the offline inference examples
• Benchmark it against your current engine on your own workload

• Open-sourcing production infrastructure helps others lower serving costs
• Huawei Ascend support matters for teams outside the Nvidia ecosystem
• Its ideas may find their way into other open-source engines

• Check the license and model terms for your use case
• Follow your organization’s rules on hosting models and data
• Test reliability before you run it in production