DeepSeek open-sources DeepInfer for large-scale model inference

The inference engine becomes public

DeepSeek announced the open-source release of DeepInfer on October 20, making the inference core used behind its large-scale services available to developers. The Apache 2.0 project currently supports DeepSeek-V4.1-Flash, and its example loads the Flash-1017 checkpoint. This is an infrastructure release rather than a new model announcement.

What developers can run

The public engine includes optimized kernels, distributed execution, DSpark speculative decoding, asynchronous scheduling and structured outputs. Speculative decoding proposes several tokens before verification; scheduling overlaps CPU work with device execution. These features address the work of serving a model efficiently after its weights have been trained.

The supported hardware list names NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It does not promise compatibility with every NVIDIA or Ascend device. Batch-invariant inference and deterministic sampling are currently NVIDIA-only features with specific configuration requirements.

A system built around scale and reuse

DeepSeek’s deployment Q&A describes a design that separates processing incoming context from generating subsequent tokens, alongside a separate vision stage and cache storage. The official diagram reproduced here shows that full production system, including components outside this open-source release. The company reports that one cluster serves 155 trillion tokens daily, without a breakdown into input, cached and output tokens. This is not an independently reproduced benchmark.

One design goal is retaining KV cache for more than 72 hours to avoid repeating computation for reusable context. This describes DeepSeek’s production architecture, not an included long-lived cache service in the public package.

What remains outside the release

The README identifies three additions needed for a complete online service: a KV-cache system, disaggregation of vision encoding, prefill and decode, and load balancing across data-parallel replicas. Much of the engine functionality needed for disaggregation is already implemented. DeepSeek says cache management and load balancing are tightly coupled to its internal infrastructure, so that code has not yet been fully opened.

Recipe connects API requests to model tokens

Released alongside DeepInfer, deepseek-recipe provides Rust libraries and Python bindings for translating Messages, Chat Completions and Responses requests into a common conversation format. It renders DeepSeek prompts and converts model output back into API responses. The project’s code and public documentation use the MIT license.

Recipe supplies an interface layer rather than a complete hosted API. Developers still provide model inference, tool execution and HTTP transport. Matching a request format therefore does not automatically reproduce every part of DeepSeek’s online service.

Editorial context: weights are one part of deployment

For teams hosting an open model, the useful question is how well the full service performs under their own traffic. An inference engine can make a deployment more practical, but memory capacity, request lengths, concurrency and storage still shape the result.

A sensible evaluation would measure time to the first token, generation speed and total hardware requirements separately. Testing cold requests alongside repeated context would also reveal what the available engine delivers and what depends on a team’s additional cache and routing systems.