On eight gigabytes of VRAM, model size is not the number that matters
A dense 27B runs at 2.5 tokens per second. A 35-billion MoE, on the same card, does thirty. The difference is not size.
4 min read
The plan was to run Qwen3.8-27B on an RTX 3070 Ti with 8 GB. It ran: 2.5 tokens per second. At the speed of reading one sentence in fifteen seconds, an agent that is supposed to write code is not a tool, it is a penance.
The instinctive response is to start tuning. I did, thoroughly, measuring everything. Tighter VRAM margin, thread count, batch sizes, ngram speculation, an MTP head, a smaller quant.
From 2.29 to 2.48 tokens per second. Eight per cent.
Then I changed model — not size, shape — and went to thirty.
The number that counts is not the one on the box
| model | parameters | active per token | generation |
|---|---|---|---|
| Qwen3.8-27B UD-Q3_K_XL | 27 B | 27 B | 2.5 t/s |
| Qwen3.6-35B-A3B Q4_K_M | 35 B | 3 B | ~30 t/s |
The bigger model runs twelve times faster. That is not a paradox. It is the only thing that could have happened.
Neither of them fits in 8 GB of VRAM. Half the weights live on DDR4, and the DDR4 in this machine delivers about 14.5 GB/s usable against the card’s 608. Forty-two times less. That bus is the bottleneck, and the only question that matters becomes: how many bytes do I have to push across it per token?
The dense model has to read all twenty-seven billion of its weights to produce one word. Every time. The MoE activates three out of thirty-five: the expert weights sit in RAM, but only a fraction is read per token, and attention stays on the GPU where the bandwidth is.
Parameter count is a property of the file. Active parameter count is a property of the architecture, and that is the one your bus pays for.
Three things I took for granted that were wrong
Thread count runs in opposite directions between the two architectures.
| model | 6 threads | 12 threads |
|---|---|---|
| 27B dense | 2.16 t/s | 2.48 t/s |
| 35B-A3B MoE | 30.6 t/s | 22.9 t/s |
The received wisdom on llama.cpp is to stop at physical cores. On the dense model that is false: the work is dominated by dequantisation latency, and SMT fills the bubbles. On the MoE it is true and then some, because little is activated per token, there is not much work per thread, and synchronising twelve threads costs more than it returns.
I had tuned twelve threads on the dense model and carried them over to the MoE. They were costing me twenty-five per cent. The launcher now picks for itself: six on the MoE, twelve on the dense.
The smaller quant is slower. Q2_K_XL is three gigabytes lighter than the Q3, and you can see it: it reads the prompt at 354 tokens per second instead of 250, because more of it fits on the GPU. Then it generates at 1.52 instead of 2.48. The two-bit dequantisation path costs the CPU far more than it saves in bytes moved. Fewer bytes, more time.
The MTP head is a net loss. 1.80 against 2.48. It takes 1.37 GB of VRAM, and on eight gigabytes you were already using that VRAM for weights. On a larger card the arithmetic flips — but the arithmetic has to be done on the card you own.
And a fourth, which is a non-finding and counts just as much: how many experts you keep in VRAM changes nothing. From 28 to 64, everything landed between 29.66 and 30.70 t/s, which is inside the noise. So you may as well pick the value that leaves the most VRAM free for context.
“Fast” is not an answer if the model is wrong
There is a quicker model still: qwen3.5 at 9 billion, through Ollama, fits entirely in VRAM, sixty-five tokens per second. Twice the MoE.
I put them in front of the same task: find a non-obvious bug in a
merge_intervals. The bug is there, and it is that the function mutates the
caller’s list, because the outgoing list holds references to the same objects
as the incoming one.
The 35B-A3B finds it, and explains that assigning to last[1] writes into the
original object. Correct.
The 9b claims the initial if not intervals: check returns an empty list when
the input has a single element. That is false — not [x] is False — and it
is not a detail: it invented a bug that is not there and missed the one that
is.
The 9b is still excellent for repetitive code and for answers that have to arrive immediately. But “fast” and “useful” are different axes, and for reasoning about code the first does not buy you the second.
The rule
If the GPU cannot hold the model, stop asking how big it is and start asking how much of it is active. Then measure on the machine you have, not on somebody else’s benchmark rig: here, three monitors were costing six hundred and fifty megabytes of framebuffer, which on eight gigabytes is an eighth of the card.
The 27B is still installed. It is the only one of the two with an image encoder, and for that it is perfectly good — slow, but it is the one that can see.