Problem
What had to be solved
Self-hosted DeepSeek-class inference for real chat traffic: long prompts, heavy prefix reuse, streaming API, TTFT matters more than end-to-end latency.
What I tried
Eight setups, in the order I actually ran them
Chat traffic reuses the same system prompt about 80% of the time. The question was how to stop recomputing that prefix on every request. I started on the production-class model, burned money, then dropped to a smaller model and compared three ways of moving the KV cache between GPUs.
Shorthand: 1P2D means 1 GPU does prefill (read the prompt) and 2 GPUs do decode (stream tokens). LMCache MP is a shared KV store in host RAM that every GPU talks to, instead of handing cache privately from one GPU to another.
| What I tried | Model and GPUs | How KV moved | What happened |
|---|---|---|---|
| BF16 baseline | DeepSeek V3.2 on 8× H200 | All 8 GPUs used for tensor parallelism. No split between prefill and decode on one node. | The model ran, but the cluster was too expensive (~$60/hr class) to iterate on daily. |
| Classic LMCache P/D | DeepSeek V2-Lite on A100s | Prefiller pushed KV to the decoder per request, using LMCache’s built-in P/D connector. | Worked in smoke tests. Decoder ran out of memory under real concurrency. |
| NIXL P/D | V2-Lite on 2× A100 1 prefiller + 1 decoder |
GPU-to-GPU KV handoff over NVLink/UCX. Each request copies cache privately to its decoder. | First stable split. Fast for a single request, but later chats could not reuse that prefix. |
| NIXL scale-out | V2-Lite on A100s 2 prefillers + 2 decoders |
Same GPU-to-GPU handoff, more workers. | Throughput went up. Still no shared prefix cache across requests. |
| Shared L1 | V2-Lite on A100s 1 prefiller + 1 decoder |
One central LMCache multiprocess server. Prefiller writes prefix KV once; decoder reads it back. | Prefix reuse started working. This is the architecture that matched the 80% hit rate. |
| Best V2-Lite run | V2-Lite on 3× A100 1 prefiller + 2 decoders |
Same shared LMCache store, extra decode GPUs so token streaming can scale independently. | Asked 10 req/s, delivered 8.5. TTFT P50 0.52s vs ~2s in production. The headline result. |
| Production-class model | V3.2 NVFP4 on 8× B200 1 prefiller + 1 decoder |
Same shared LMCache store, now on the real model and Blackwell GPUs. | The path works end to end. Sub-second TTFT only held to ~2 req/s, not the V2-Lite curve. |
| Cross-node scale-out | V3.2 NVFP4 on two 8× B200 rentals 1 prefiller + 3 decoders |
Same design, split across two machines so decode can grow past one node. | Runbook is done. The provider had no private network between rentals, so the sweep never ran. |
What worked
Shared L1 via LMCache MP + ZMQ
One central LMCache Multiprocess server (ZMQ port 6000) as a shared L1 KV layer in host RAM.
Every prefiller and decoder uses LMCacheMPConnector against the same store.
Prefiller writes prefix KV once; later requests with the same prefix skip full prefill.
Results
Asked vs delivered
Best V2-Lite stack: 1 prefiller + 2 decoders on 3× A100, shared LMCache store. Production wanted ~2s time-to-first-token and ~60 req/s. Here is what that stack actually did.
Measured on DeepSeek V2-Lite (3× A100), not a production-class model. On V3.2 NVFP4 (8× B200) the same architecture held sub-second TTFT only to ~2 req/s. This proves a shared L1 KV layer is the right shape for prefix-heavy chat, not production capacity.
Could it keep up?
Gold: load we requested. Blue: load the stack actually served. After 10 they split: served stalls near 11.* That is this box’s ceiling, not 60.
How long until the first token?
Wait time in seconds, not speed. Lower is better. Green: typical request (median). Red: slow tail. Dashed lines: production (~2s / ~3.5s). At 10 requested, median is 0.52s.
* One 1P2D box (3 GPUs). ~11 req/s is that hardware ceiling, not a failed design. More QPS means more replicas at about 1:2 to 1:3 prefill:decode, e.g. 3 prefiller + 6 decoder GPUs.
Cost
Total GPU spend
Insights
What I would do again
Technical depth
Everything else is in the repo
Full write-ups, sweep tables, per-phase runbooks, and the resource bibliography.