A 1,048,576-token context on two RTX 5090s: forking a C++ inference engine for tensor-parallel and YaRN

Qwen3.8-27B ships with a 262,144-token native context and a model card that says 1,048,576 with YaRN scaling. On a single RTX 5090 you get neither in practice: at 253k tokens the KV cache and weights already take 27.9 GB of the card’s 32, and the NVFP4 checkpoints ship without a YaRN block, so a serving stack has to be told about it explicitly (vLLM takes it through --hf-overrides). I wanted the full million on hardware I own.

So I forked NInfer — a from-scratch C++20/CUDA engine that runs a closed set of Qwen checkpoints on one 5090 — and added two things it did not have: tensor-parallel execution across two GPUs and YaRN rope scaling. The result is wamansou/ninfer-tp2-1m. It holds a 1,048,576-token window at 27.4 GB per card, retrieves needles at 1,046k tokens, and decodes faster than vLLM once you leave the native window. It also prefills slower than vLLM, and I’ll show that too.

Why tensor-parallel and not pipeline-parallel#

Two 5090s give you 64 GB, but how you split a model across them decides what you get for it. Pipeline-parallel puts layers 0–31 on one card and 32–63 on the other; each token walks through both cards in sequence, so per-token latency does not improve and one card idles while the other works. Tensor-parallel splits every weight matrix in half — each card holds half the attention heads and half the MLP columns — and both cards work on every token simultaneously. The cost is a cross-GPU reduction after every layer’s projections: about 128 of them per token for this model.

The engine’s structure made tensor-parallel the natural fit. NInfer is not a generic runtime; it has one execution plan per model family, compiled in, and every op is a hand-written kernel with a numerical oracle test. Splitting a projection means producing a shard-shaped weight tensor and running the existing kernel on it, which keeps every kernel’s tolerance proof valid. The new code is the shard plan, the collectives, the head-local attention geometry, and the single cross-device CUDA graph that stitches it together.

Two facts about consumer hardware shaped the design:

• No NVLink, and no peer-to-peer. GeForce cards on this board report canAccessPeer = 0; the two 5090s sit at PCIe 5.0 x8/x8 through the CPU root complex. Every cross-GPU copy is host-staged. I measured about 16 µs for a 10 KB reduction, which is 128 × 16 µs ≈ 2 ms per token if done naively.
• cudaMemcpyPeerAsync cannot be captured into a CUDA graph. The whole engine runs decode as a replayed graph; the collectives had to use plain cudaMemcpyAsync over unified addressing to be captured at all.

Under the graph, the 128 reductions cost ≈0.2 ms per token net, not 2 ms — the copies overlap with compute the graph already had queued. That number is what makes the rest of this post possible.

YaRN, and matching what vLLM actually does#

YaRN stretches the rotary position encoding so that a model trained on 262k positions reads a 1M-token prompt without its attention pattern falling apart. The textbook formula has two knobs, a correction range that decides which frequency pairs get interpolated and an attention temperature mscale. I wanted bit-for-bit agreement with vLLM’s YaRN so that any quality difference later would be the engine’s, not a rope drift.

That turned out to be a trap. Qwen3.8’s rope parameters carry an mrope_section (it is a multimodal model), which routes vLLM through MRotaryEmbedding — whose constructor multiplies max_position_embeddings by four before computing the correction range. The deployed correction range is (16, 24), not the (14, 22) the paper’s formula gives for these parameters. I dumped vLLM’s actual cos/sin tables from the serving environment, checked them into the repo as a reference, and wrote a drift test that regenerates them and fails if the vLLM version changes the numbers. NInfer applies mscale inside the rope kernel on the rotary dimensions, exactly where vLLM does, and has no attention-level scale factor.

One more thing I would not have guessed: rope never enters the gated-delta layers. Qwen3.8-27B is a hybrid — 48 of its 64 layers are linear-attention (Gated DeltaNet) and only 16 are full attention. YaRN touches those 16. The other 48 do not see position at all.

Does it actually retrieve at a million tokens?#

The floor test for long context is needle-in-a-haystack: hide a sentence at some depth in a huge prompt, ask for it, score the answer by strict exact match. I ran it with EvalScope at 262k, 653k and 1,046k tokens, depths 10 through 90 percent.

The first thing I found was that the stock EvalScope haystack corpus is 644 KB of Paul Graham essays — about 150k tokens. Any context tier above that tiles the corpus, and I confirmed the 262k tier repeated every window exactly twice. Retrieval over repeated text is easier than over novel text, so for the real tiers I built two disjoint haystacks from ~30 public-domain books each and verified, through EvalScope’s own prompt-construction code, that no 120-character window repeats.

Two controls matter more than the table. First, a novel needle — an invented sentence about an invented town, which cannot be in any training set — came back verbatim from a 1,045,956-token prompt, so this is retrieval, not recall. Second, the 1M tier is only two samples per depth (each request is ~19 minutes of prefill; ten per depth would have been 17 hours), so statistically 10/10 means “consistent with a per-sample retrieval rate of at least 74% at 95% confidence,” not 100%. I say that in the README because the alternative is quietly implying more than I measured.

What needle retrieval does not tell you: whether the model still reasons well over 800k tokens of material. It tells you the positions are addressable and the rope is not broken. A curious side result: with YaRN deliberately disabled, the model still retrieved a needle at 270k — plain extrapolation tolerates about 10% past the native window. So the >262k needle legs prove addressability; the 653k and 1M results are what show YaRN earning its keep.

The soak: 98,692 generated tokens, twice, byte-identical#

Retrieval is one request. A serving engine also has to not fall over at a million tokens. I started from a 949,885-token prompt and let greedy decode run until the window was full — 98,692 generated tokens, about 52 minutes — then did it again in a fresh process. Memory sat at exactly 28,070 MB on both cards for the entire run, and the two passes produced byte-identical token streams and reply text (SHA-256 match on both).

An accidental gift inside that run: with end-of-sequence suppressed, greedy decode settles into a 1,903-token loop, and the same 1,903-token block was decoded 46 times identically across positions 959k to 1,047k. That is a controlled experiment nobody designed: the same input at 46 different context lengths in the region no one had run before, and not one argmax flipped. It says a lot about numerical stability at the end of the window — and, honestly, very little about decode diversity, since windows 2 through 46 are one probe repeated. A seeded temperature > 0 soak would be the coverage complement; it is on the list.

Performance, and the power-limit story#

Every number in this section names the GPU power limit it was measured under, because the limit changed twice during the campaign. All development and the first measurement pass ran with both cards capped at 400 W — the minimum the driver allows. When I lifted both to 575 W for the publishable numbers, the machine hard-crashed under sustained dual-GPU load; the PSU could not carry it. The cross-engine comparison was then done at 500 W on both cards. Lifting the cap helped the single-GPU configuration more than the dual one (one card saturates 575 W; two cards sharing the model peak around 400 W each), so the TP2-over-TP1 ratio narrowed from 1.44× to 1.40× — a correction against my own headline that I would rather report than hide.

Two cards are faster than one, not just bigger#

At 250k context and 575 W, splitting the model gives 75 tok/s versus 54 without speculative decoding and 159 versus 114 with it. Halving the weights and KV traffic each card reads per token outweighs the collectives. And a single card cannot run the 1M column at all.

Against vLLM: prefill loses, decode wins, and one surprise#

I ran vLLM 0.25.1 on the same two cards, same prompts to the token, temperature 0, 512 generated tokens, at 500 W. vLLM served unsloth’s NVFP4 quantization with YaRN injected through --hf-overrides, fp8 KV cache and its MTP speculative decoder, at its maximum feasible window of 750k tokens.

The surprise is in vLLM’s speculative decoding. At 250k it accepts 51% of drafted tokens, and decode runs at 111 tok/s — a bit below NInfer’s 156 with MTP but a perfectly good number. Past 262k, vLLM’s acceptance rate is exactly zero: at 653k and 700k it drafted 1,533 tokens and accepted none, paying for the drafter every step and gaining nothing. Its decode falls to 42 tok/s. NInfer’s acceptance stays flat at 52–60% all the way out, so at 653k it decodes at 119 tok/s with MTP — and even without speculation, at 57 tok/s, it is ahead of vLLM there.

Prefill goes the other way. vLLM’s attention kernels are heavily tuned and it batches 16,768 prompt tokens per step; NInfer chunks at 1,024 and its long-context prefill has had no tuning at all. vLLM is 1.17× faster at 250k and 1.32× faster at 653k and 700k. Time to first token on a 653k prompt is 354 s against NInfer’s 466 s, and a full 1M prompt takes NInfer about 18 minutes. At 512-token replies vLLM wins end-to-end everywhere; NInfer with MTP pulls ahead once a reply runs past roughly 5k–9k tokens.

And the memory picture: vLLM’s fp8 KV pool tops out at 759,297 tokens on these cards at 28.9 GB per GPU; NInfer’s INT8 cache (16.5 KB per token per GPU, quantization scales included) holds 1,048,576 tokens at 27.4 GB. Thirty-eight percent more context in less memory. Adding the MTP draft head at 1M costs another 1.4 GB per card — one full extra attention layer’s worth of KV over the whole window, which a prediction earlier in the project had called “negligible.” It was not.

The caveats, which I would rather state than be told: vLLM and NInfer ran different NVFP4 quantizations of different fine-tunes (unsloth’s base quant versus the abliterated one my artifact was converted from), so this is engine plus quantizer, not identical weights. fp8 versus INT8 KV. One measurement per cell. No quality claim beyond needle retrieval. And the prefill-chunk size was never swept, so some of the prefill gap may be a config choice rather than a kernel gap.

Things that bit me#

A hard-coded attention limit outside the plan. The decode attention kernel had a compile-time ceiling of 262,144 visible keys, with a page-ID staging array sized from it. Nothing in the design document mentioned it, and it would have made 1M unplannable — and past ~348k keys, silently overrun shared memory. Widening it to 1,048,576 grew the staging array from 64 to 194 entries and put the worst case at 99,232 bytes of shared memory per two co-resident blocks against a 102,400-byte carveout. Three kilobytes of margin, now guarded by a static_assert. The negative control — an illegal memory access at 400k keys with the old bound — is what convinced me the fix was real.

Relative error stops meaning anything at a million keys. The attention oracle test compares kernel output to an FP64 reference with a relative-L2 criterion. At 1M keys the softmax over nearly zero-mean keys flattens so far that the reference output’s RMS (~5.5e-4) drops below the BF16 storage granularity of the output itself. Relative L2 has a floor of ~1.7e-3 from output rounding alone, and the kernel sits at 1.8× that — about 0.43 of one output ULP. The test now calibrates its bound against a BF16-rounded oracle per case instead of a fixed tolerance. Conditioning the queries to sharpen the softmax was tried and measured to fail.

The parity gate the spec asked for was unattainable. The plan wanted ≥99.9% greedy token agreement between the one-GPU and two-GPU paths. Any reordering of floating-point reductions in this model breaks that: the same engine on one GPU agrees with itself only 96.6% of the time between prefill-then-decode and decode-only on identical positions. The gate became comparative — the TP2 path must match TP1 as well as TP1 matches itself — and it does.

Speculative decoding is lossless per position, and streams still diverge. The MTP verify step evaluates K+1 columns in one GEMM where ordinary decode evaluates one; the two reduce in different orders, so a near-tie token can flip argmax. A teacher-forced oracle confirmed every token the speculative path commits equals the target model’s argmax for that prefix (64/64), and yet a greedy MTP stream diverged from a greedy non-MTP stream at token 45. Both are individually lossless; they are two different decodes. Needle answers were identical across the two paths, which is what matters, but “MTP is bit-identical” would be false and I removed it from the docs.

A cross-device pointer that greedy decoding had masked. Rank 1 was reading a token-count array through a pointer that pointed at rank 0’s device memory. Every greedy test passed because the penalty path that dereferences it was never taken. It surfaced in review, not in a crash. Per-rank counters with rank 0 as the source of truth fixed it, and a debug-only cross-rank consistency check now measures the lockstep argument instead of assuming it.

--tp 1 had to stay byte-identical to upstream. That was a contract from day one and the gate for it was, for most of the project, embarrassingly weak: exit code, a trivial token check and two load-summary rows. The final review called it out. I built upstream at its base commit in a separate worktree and compared 128 greedy tokens on three prompts (23, 2,191 and 28,677 tokens): identical token IDs, identical text, identical load summaries. The golden files are in the repo and the regression script compares against them now.

What vLLM does better, plainly#

Prefill, by 20–30%. Batching, continuous scheduling, prefix caching across requests, chunked-prefill flexibility, structured output, multi-model serving, hundreds of concurrent requests, an ecosystem. NInfer is one model, one to eight fixed request slots — and at 1M, exactly one — with no prefix reuse under MTP on two GPUs. It is a specialist. The specialty is that on two consumer cards it runs this model at a window nothing else I tested reaches, and keeps speculative decoding alive out there.

Takeaways#

1. Tensor-parallel on consumer GPUs works without NVLink if the collectives live inside the CUDA graph: ~0.2 ms per token for 128 host-staged reductions, against a naive estimate ten times that.
2. Two 5090s beat one on latency, not only capacity — 1.40× decode at 250k — because each card reads half the weights and half the KV per token.
3. A million tokens fits in 27.4 GB per card with INT8 KV, and needle retrieval holds at 1,046k on non-repeating text. Retrieval is the floor, not the ceiling; I did not measure reasoning quality out there.
4. YaRN “as deployed” is not YaRN “as published.” The correction range vLLM actually uses for this model is (16, 24), not (14, 22), because of a multimodal code path. Match the deployment, and check in a drift test.
5. Speculative decoding past the native window is where engines separate. vLLM’s acceptance went to zero beyond 262k in my runs; NInfer’s stayed above 50% to 1M. That single fact is worth a 2.5–2.8× decode gap at 650k+.
6. Every performance number needs its power condition next to it. The cap changed three times; a table without it would have been fiction. And do not run two 5090s at 575 W on a 1,000 W-class supply.
7. The test that matters most is the one that tries to break the claim. The tiled corpus, the YaRN-nulled control, the vLLM MTP zero-acceptance and the weak --tp 1 gate were all found by asking “what would make this number hollow?” — not by running the benchmark again.

References#

1. wamansou/ninfer-tp2-1m — the fork; docs/performance.md carries every table with its power condition
2. Neroued/ninfer — the upstream engine, Apache 2.0
3. YaRN: Efficient Context Window Extension of Large Language Models — Peng et al., 2023
4. Qwen3.8-27B — model card, including the 1M-with-YaRN claim
5. vLLM — v0.25.1, used as the control
6. EvalScope — needle-in-a-haystack harness, v1.10