DeepSeek V4.1 Flash Inference Explained
Answer me with artifact · 2026-10-25 07:06

New version 0.5.7

You have 0.5.5. Found when this page was made.

What's new:

Run in a terminal:

npx skills update answer-me-with-html -y

Or tell your agent:

Update answer-me-with-html
See what's new

DeepSeek V4.1 Flash Inference Explained

How DeepSeek serves 155T tokens/day — a plain read of the DeepInfer deployment Q&A

sourcegithub.com/deepseek-ai/DeepInfer/doc/inference_deployment_qa.md

Lead: The doc is DeepSeek's playbook for serving V4.1 Flash cheaply at huge scale. Three core ideas: run each stage on its own GPU pool, spread MoE experts very wide, and keep KV cache on disk for days.

A

The big picture

8-GPU boxes64–128-GPU supernodeAPI → tokensprefix-cache lookupimagesnew tokens onlyembeddingscached KVKV handofftokens → streamRequestRecipeEntrypointVEPrefill3FS KV storeDecode
B

Glossary in one line each

Term Plain meaning
Prefill Read the whole prompt once and build its KV cache. Compute-heavy.
Decode Make output tokens one by one. Memory-bandwidth-heavy.
KV cache Saved attention state, so the model never re-reads past tokens.
MoE / EP The model has many "experts"; Expert Parallelism puts different experts on different GPUs.
DP Each GPU runs its own copy of attention on its own requests.
E/P/D split Vision Encoder, Prefill and Decode run on separate GPU pools, each sized for its job.
TTFT / TBT Time to first token / time between tokens.
C

Why split prefill and decode

  • Prefill uses small EP (2–8) on ordinary 8-GPU machines. Use the smallest size the memory allows.
  • Decode uses "Wide-EP" (64–128 GPUs). Each GPU holds few experts, so each step moves less weight data. Throughput goes up.
  • Smallest efficient deployment: 64 prefill GPUs + 64 decode GPUs, roughly 1:1.
  • Images go to a separate vision encoder on one cheap GPU.
D

Keeping work balanced

  • Experts (EPLB): hot experts get spare copies on extra GPU slots. The system reshuffles them every 1,000 (prefill) or 10,000 (decode) steps. Slowest GPU is only 4–9% behind average.
  • Prefill routing: requests go into 4 length buckets (<16K, 16–64K, 64–256K, ≥256K). A cost model picks the chunk size that fits a time budget.
  • Decode routing: balance both request count and total KV length per GPU.
  • Result: about 80% of a perfectly balanced benchmark.
E

The KV cache bet: keep it for days

Layer Where it lives
Active requests GPU memory
Sliding-window (SWA) state Host RAM of the machine that made it; fetched over 400G network on a hit
Paged block KV Dedicated 3FS SSD cluster, ~100 PiB, 10+2 erasure coding
  • TTL: 3 days for agent traffic, 7 days for chat. Chat has a long tail of reused prompts.
  • Any prefill machine can hit any prefix, so routing needs no "sticky" prefix affinity.
  • Loads are overlapped: disk → host RAM in a separate process, host → GPU layer by layer during compute.
  • SWA bounded replay: if the sliding-window state is missing, recompute only the last 128 tokens. Cost stays constant, whatever the context length.
F

Other speed tricks

Trick What it does
FP4 Main KV, indexer KV and expert weights stored in 4-bit formats
Mega MoE kernel Fused MoE kernel; faster than unfused at every size
DSpark speculative decoding Draft 5 tokens, verify only the ones likely accepted. +4–7% throughput at ~60 seqs/GPU
Async scheduling CPU plans step N+1 while GPU runs step N, so the GPU never idles
Structured output JSON/regex grammars compiled on a separate machine, applied as a token mask
Fault tolerance Spare capacity + reroute requests. A dying decode engine hands its live requests to others within 1 min
G

DeepSeek Recipe (the API front door)

  • Converts Chat Completions, Responses and Messages API requests into one token format, in Rust.
  • Special tokens typed by users (like <｜User｜>) become plain text tokens. This blocks prompt injection of control tokens.
  • Splits long prompts at known special tokens and tokenizes the pieces in parallel.
  • Streams tool-call JSON as it is generated, so clients see arguments sooner.
H

What it means for you

Takeaway

This is a hyperscale recipe, not a homelab one. The minimum efficient setup is ~128 GPUs plus a petabyte-class SSD cluster. Lessons that work at any scale: separate prefill from decode, cache prompt prefixes hard, and overlap CPU scheduling with GPU work. The KV store and load balancers are not open-sourced yet; only the engine, kernels and Recipe are.

I

How much money they make

Only per-GPU revenue is public

The doc gives one money figure: US$38 revenue per GPU-hour, at valley (off-peak) pricing. It does not give the GPU count, total revenue, costs or profit.

Figure Value Source
Revenue per GPU-hour $38 stated, valley pricing
Revenue per GPU-day $912 $38 × 24
Revenue per GPU-year ~$333K $912 × 365
Tokens per cluster per day 155 trillion stated
Cluster GPU count not given —
Total revenue, profit not given —
  • Example only: if a cluster had 10,000 GPUs, revenue would be ~$9M/day, ~$3.3B/year. The 10,000 is an assumption, not a figure from the doc.
  • Revenue is not profit. A rented H100-class GPU costs roughly $2–3/hour, so $38/hour suggests a very large gross margin per GPU. Power, storage, staff and model training are extra.