DeepSeek has published two components of its inference stack in deepseek-ai/DeepInfer and deepseek-ai/deepseek-recipe. One executes the model on specialized accelerators. The other translates API conversations into model inputs and turns generated output back into responses. Neither repository contains the model weights.
As inspected October 22, the practical answer depends on your machine. A serving team can evaluate DeepInfer on supported hardware, then supply the surrounding service. A laptop or RTX owner should investigate the separate community runtime for the exact V4.1-Flash-1017 checkpoint. That route is experimental and memory-heavy.
One contract. Two deployment paths.
Recipe translates API conversations into model tokens and parses generated tokens back into API events. It does not make weights smaller or provide inference hardware.
DeepInfer on supported hardware
A different local runtime
The local low-bit guide estimates 128–145 GB memory plus SSD. Conceptual integration; backend adapter required. Separate GPU/RAM placement represents a discrete-GPU system; Macs share unified memory. Capacity is not speed. Moving slips show placement, not quantities or measured transfers.
What do the two drops actually supply?
DeepInfer combines optimized kernels, expert parallelism, speculative decoding, asynchronous scheduling and constrained generation. Its published requirements are Linux, Python 3.12, GCC 13+, and the CUDA 13 family for NVIDIA SM100 or CANN 9.2 for Ascend 950. The engine explicitly describes this release as lacking an online-serving wrapper.
Recipe supplies Rust libraries and Python bindings for Messages, Chat Completions and Responses conversion, DeepSeek prompt encoding, and streaming output parsing. Its scope includes reasoning and tool-call formats; actual inference and tool execution belong to the application. The FastAPI example defaults to a mock yielding “Hello world!”. Starting that server demonstrates protocol plumbing, not a working model.
Production practicality: useful engine, substantial integration
DeepSeek describes production pools for vision, prompt processing and token generation, coordinated through cache storage and load balancing. The release omits the complete infrastructure for those responsibilities. It does contain device-side KV-cache allocation and layouts; “no KV cache” would misstate the boundary. The absent layer makes cached state useful across the larger service.
The deployment Q&A distinguishes a deployable prefill/decode pair from its recommended efficient unit of 64 prefill plus 64 decode GPUs. That is an efficiency recommendation under its architecture, not a universal 128-GPU minimum. DeepInfer merits an evaluation by teams already operating distributed inference. Smaller teams should compare integration effort with established serving runtimes before committing infrastructure.
There are alternatives for this exact checkpoint. A vLLM recipe specifies 1017, vLLM 0.31.0 and several accelerator configurations. SGLang lists 1017 configurations but marks their GPU validation pending. Those different evidence levels matter more than a generic “supports DeepSeek” badge.
The performance charts answer a narrower question
DeepInfer’s benchmark harness uses random non-special text token IDs and forced load balancing, then measures prefill and decode separately. Its decode throughput calculation assumes full draft acceptance; the plotter can apply a supplied acceptance rate. These are useful engine experiments. They do not directly measure request arrival, queueing, cache misses or complete API latency.
A production trial should replay representative requests and report first-token latency, gaps between tokens, accepted output throughput, errors and memory at realistic concurrency. Test cold caches, cancellation and worker loss. Calculate cost per successfully completed workload from the whole deployment. We did not reproduce the GPU benchmarks or establish service economics.
An RTX 5090 is Blackwell, but it is not SM100
NVIDIA distinguishes B200’s compute capability 10.0 from the RTX 5090’s 12.0 and RTX 4090’s 8.9. DeepInfer’s pinned FlashMLA dependency requires SM100/SM103. Its expert computation also relies on specialized FP8/FP4 kernels and communication choices. The pinned attention library specifically requires CUDA 13.1 or newer. An RTX port would need compatible kernels, layouts and memory management. Installing the same CUDA version cannot supply them.
The model’s size is another independent obstacle. DeepSeek describes 552 billion backbone parameters plus 196 billion Engram conditional-memory parameters. Activating only a subset per token reduces computation; it does not eliminate storage for the remaining weights. The official 1017 repository is roughly 510 GB, already using mixed precision.
The local route moves the problem into RAM and SSD
Unsloth documents exact-1017 quantizations and CPU, Metal and CUDA instructions. Its guide uses llama.cpp PR #30817, open and unmerged at inspection. This is a concrete port to evaluate, not evidence that an arbitrary stock llama.cpp or Ollama binary works. Engram tables can remain on SSD while other weights occupy system memory and GPU memory.
| Quantization | Download | Engram | Other weights |
|---|---|---|---|
| UD-IQ1_S | 233.0 GB | 110.6 GB | 122.4 GB |
| UD-IQ3_XXS | 394.7 GB | 161.3 GB | 233.5 GB |
| UD-Q4_K_XL | 500.7 GB | 202.8 GB | 297.9 GB |
The guide estimates 128–145 GB combined RAM/VRAM or unified memory for its one-bit variants with Engram on SSD. Context and runtime overhead still matter. The smallest file is not a promise of unchanged quality: reported next-token agreement tests are not coding-task accuracy. Quantization names also conceal mixed formats; the Q4 variants preserve original MXFP4 experts.
| Machine | What to expect |
|---|---|
| 16–64 GB laptop | Recipe conversion is accessible; the documented full-model configuration exceeds this memory class. |
| 128 GB Mac | An experimental low-bit, SSD-backed route; allow for OS and runtime headroom. |
| RTX workstation | Use the separate CUDA port with substantial host RAM and SSD. A single consumer RTX’s VRAM is insufficient. |
Stock MLX is a separate question. Its V4.1 module rejects configurations with Engram layers. A model file bearing the right name does not establish complete checkpoint support.
Cold prompts expose the real consumer bottleneck
A September RTX 5090 experiment paired roughly 32 GiB VRAM with 126 GiB system RAM and NVMe. Its initial results separated 5.12 tokens/s on new content from 21.27 on repeated resident content; a later direct-I/O experiment improved first-pass decoding. These concern the earlier checkpoint, not 1017. Their lesson is valuable: repeating a prompt can warm the experts its answer needs. Measure fresh tasks as well as repeats before judging an offloaded system.
How to adapt recipe without changing the conversation
The useful integration seam is the example’s inference callback. Replace the mock with your chosen backend, map generation options and stream actual output back. Use one owner for prompt formatting: applying a second chat template to recipe’s encoded conversation can change the model input.
The tokenizer contract distinguishes real control tokens from identical strings written by a user. Encode the conversation with its matching tokenizer; blindly re-tokenizing rendered text can lose that distinction. Map output constraints into backend-enforced decoding, resolve token limits against configured context, and propagate cancellation. The example already closes disconnected streams, but backend cleanup remains your responsibility.
Before exposure, add authentication, request limits and an image-destination policy; the examples do not provide authentication. Tool-call conversion does not execute a tool or reproduce hosted web search. Validate one complete tool round trip, malformed output and a cancelled request. Recipe makes protocol behavior reusable; production behavior still belongs to the assembled application.
