DeepSeek Open-Sourced the Engine Behind Its AI Service. The Interesting Part Is What It Kept
Synergised Consulting Ltd
Last Updated: 27 October 2026
DeepSeek has open-sourced DeepInfer, the inference engine that serves its own AI API in production, and the disclosure that came with it is more interesting than the code. According to DeepSeek's own deployment notes [1], a single cluster running this engine handles 155 trillion tokens per day and earns $38 per GPU-hour at off-peak prices. The engine itself is genuine, runnable Apache 2.0 code. But the parts that make those economics work, the context cache and the load balancers, are described in prose and deliberately kept back. For a business owner buying AI capability, that gap is the actual news: the cost of running AI is decided less by the model and more by how well a provider avoids recomputing work you have already paid for.
What DeepSeek Actually Released
DeepInfer is a large-scale inference framework: the software that turns a trained model into a service that can answer millions of requests a day. According to DeepSeek's announcement and deployment Q&A [1], it is released under the Apache 2.0 licence, runs on both Nvidia and Huawei Ascend hardware, and is the same engine behind the company's own DeepSeek V4.1 Flash service. It ships with distributed inference, speculative decoding, structured outputs and the performance kernels the company has been releasing in pieces for over a year.
The design is organised around ordinary hardware rather than exotic clusters. Prefill, the stage that reads your prompt, runs on commodity 8-GPU machines; decode, the stage that writes the answer, runs on supernodes of 64 to 128 GPUs [1]. That matters because it shows the economics were engineered, not inherited: the goal was to serve more requests with less compute and energy, which is what lets a provider sustain low prices.
The Economics DeepSeek Put on the Record
Alongside the code, DeepSeek disclosed serving figures that few AI companies publish about themselves. A single cluster serves 155 trillion tokens per day, which is roughly 1.8 billion tokens every second, and overall revenue runs at $38 per GPU-hour at off-peak pricing [1]. Reporting by 36kr, based on DeepSeek's own technical write-up [2], adds the detail that gives the number its shape: the system keeps its context cache on a dedicated storage cluster of roughly 100 PiB of SSDs, retains cached context for around three days for agent traffic and about seven days for chatbot traffic, and records hit rates above 98% on its short-term in-memory cache.
The cache is the business model
Those retention windows are not a technical footnote. When a conversation with an AI agent continues across many turns, the entire history travels with every request. A provider that keeps the earlier computation for days can serve turn fifty without recomputing turns one to forty-nine. A provider that cannot must redo that work and charge for it. DeepSeek's disclosure says, in effect, that the margin in AI serving is won in the cache: the same model, served without a serious context-reuse layer, is a materially worse business. The company's deployment notes put a number to the ambition too, stating the KV cache is retained for more than 72 hours to cut repeated computation on a cache hit [1].
What Is Missing From the Release
The repository is not a drop-in replacement for DeepSeek's API, and the company is candid about it. According to the README and deployment Q&A [1], three components a complete public service needs are not included: the KV cache store itself, the load balancers that spread requests across machines, and the full system that splits the work across stages. The stated reason is that this code is tightly coupled to internal infrastructure. There is also no HTTP server; today the released entrypoint runs offline batch jobs only. As the Jarvis Labs analysis of the release puts it, if you want to run this as a service, the hardest parts are still yours to build [3].
This is worth reading closely, because it is the opposite of the usual open-source announcement. The engine, the part everyone can now inspect and fork, is the commodity. The cache and the routing, the part that decides the unit economics, stay inside the company. DeepSeek has open-sourced its factory floor while keeping the inventory system.
What This Changes for a Business Owner
The bill follows context, not just tokens
The practical takeaway is not about DeepSeek specifically. It is that the price you pay per token is downstream of an engineering decision about context reuse, and providers differ in how well they have made it. In our own work running long agent sessions, the pattern is consistent: a session that has grown over days of work resends most of its history with every turn, and the cost of that session is decided by how much of that history the provider recognises as material it has already processed. Two providers can quote similar headline prices and behave very differently on a long-running job.
So when you are comparing AI providers or designing an agent workflow, ask the question this disclosure makes visible: how long is context retained, what does a cache hit cost versus a cache miss, and how does the pricing structure treat a conversation that has been running for a week. According to DeepSeek's own deployment notes [1], its answer is a cache measured in days and billed far below fresh computation, and that is precisely why it can sustain the per-token prices it advertises. Falling serving economics are good news for any owner buying agent capability: the work that used to be priced like scarce compute is increasingly priced like storage, and that trend favours exactly the long-running, context-heavy agent workflows a small business is most likely to want.
Sources
- [1] DeepSeek, DeepInfer repository and deployment Q&A:
- [2] 36kr, coverage of the DeepInfer release and DeepSeek's technical disclosure:
Background reading (qualitative guidance, no figures):
