DeepSeek Open source inference framework DeepInfer: Inference cost has become a key layer for AI inclusiveness. DeepSeek announced on October 20 that it will fully open source its internal inference framework DeepInfer. This is another development after the previous open source series of large models, TileLang compilation tools and inference operator libraries.

DeepSeek Open source inference framework DeepInfer: Inference cost has become a key layer for AI inclusiveness. DeepSeek announced on October 20 that it will fully open source its internal inference framework DeepInfer. This is another open source after the previous open source series of large models, TileLang compilation tools and inference operator libraries. Officials pointed out that the capability boundary of large models is delineated by the model itself, but how to provide ultra-large model services at a reasonable cost also depends on the middle layer of the inference framework - it coordinates and dispatches huge computing power clusters, and its efficiency directly determines the computing power cost and power consumption of AI services. DeepInfer integrates high-performance operators and a deeply optimized inference engine, supports distributed reasoning, speculative decoding, SWA Bounded Replay, structured output and other capabilities, and is compatible with NVIDIA and Huawei Ascend platforms. Based on this framework, DeepSeek has implemented the high-throughput, low-latency large-scale inference service of DeepSeek V4.1Flash. In the future, the inference code and subsequent optimization of new models will continue to be updated here. The supporting synchronous open source DeepSeek-recipe is a production-level Token encoding and decoding component, which is completely aligned with the processing logic of the DeepSeek official API, ensuring that the community gets an API experience consistent with the official deployment. Both parts of the code have been opened in GitHub, and DeepSeek has also released a complete technical report on Zhihu. Officials stated that the first purpose of the open source inference framework is to help developers and enterprise organizations optimize the throughput and latency of large-scale inference, and efficiently build privatized and public cloud deployments, thereby reducing computing power costs and energy consumption, allowing intelligence to reach a wider range of people at lower prices, and will continue to support the community in large-scale deployment optimization. The layer-by-layer opening of models, operator libraries, and reasoning frameworks continues the industry path of domestic large models using an open source ecosystem to lower the threshold of reasoning and promote inclusive intelligence. Open source warehouse DeepInfer: DeepSeek-recipe: DeepSeek-ai/DeepSeek-recipe via AI News (author: AI Base)