"Supported" is the most misleading word in AI infrastructure

"Supported" is one of the most misleading words in AI infrastructure.

I spent roughly 24 hours trying to run Gemma 4 31B NVFP4 on an RTX 5090 (32GB) through vLLM. On paper it looked close enough to be worth the attempt. Reality had better manners than the benchmark charts — it failed honestly.

The usual fixes came up: reduce context, lower GPU utilisation, use FP8 KV cache, force eager mode. Reasonable, polished suggestions. I did all of it. It still failed. vLLM recognised the model. Quantisation was correct. KV cache settings were correct. Engine initialisation advanced far enough to create hope — always the expensive part.

Then CUDA OOM. Roughly 112MB short. Close enough for social-media optimism, far enough for a dead runtime.

This wasn't a tuning issue anymore. It was an architecture mismatch: a dense 31B path trying to behave like it belonged comfortably on a consumer 32GB card. Consumer-GPU marketing vs enterprise-grade runtime reality.

Sometimes the right optimisation is to stop optimising.

Where have you seen "supported" diverge sharply from "actually runs"?

Getting real models to run on real hardware — cloud or on-prem — is a lot of what we do at VigyanBytes, including fine-tuning and serving models locally so your data never leaves the building. Tell us what you're trying to run.