A 35B model at 237 tokens/sec in 13.6GB — here's the stack

For a while the honest answer to "can you run a 30B-class model well on a single consumer card" was "sort of, if you squint." That's changed, and it changed because of stacking, not one magic trick.

Here's the stack that got a 35B model to 237 tokens/sec in 13.6GB on a single RTX 5090.

Start with pruning. REAP — the expert-pruning work out of Cerebras — clips roughly the low-value 20% of experts from a mixture-of-experts model. A Qwen3 35B comes down to about 13.6GB without the quality collapse people expect, because you're removing experts that were barely contributing, not amputating capability at random.

Then generation speed. Multi-token prediction buys around a 65% gain in generation throughput — the model commits to more than one token per step where it can, instead of crawling one at a time.

Then the KV cache — the part everyone forgets until they run out of memory mid-context. Quantise it (iq4_nl / q4_0) and the memory you were spending to remember the conversation drops sharply, which is what leaves you room to actually use a long context.

Put those together on a 5090: the 13.6GB model runs at roughly 237 tok/s generation, 208 tok/s prefill, with about 17GB free for context. That last number is the one that matters — headroom is what turns a benchmark into something you can actually work with.

None of these is exotic on its own. The result comes from stacking them and measuring, instead of chasing a single hero optimisation. That's the whole discipline: compound the boring gains.

What's the largest model you're getting real work out of on a single card — and what got you there?

Getting large models to run — fast — on hardware you own is core to what VigyanBytes does, including pruning, quantisation, and serving locally so your data never leaves the building. Tell us what you're trying to run.