A 1,048,576-token context on two RTX 5090s: forking a C++ inference engine for tensor-parallel and YaRN
I forked NInfer, a from-scratch C++/CUDA inference engine, to split Qwen3.8-27B across two consumer GPUs and stretch its context to one million tokens with YaRN. Needle retrieval holds at 1M, speculative decoding keeps working where vLLM's stops, and the whole thing fits in 27.4 GB per card. The measurements, the numerical surprises, and what vLLM still does better.