boltbeam.fyi

Which GPU is fastest for local LLMs? Measured tok/s, one user

Decode speed of Qwen3-14B and Qwen3.8-27B on RTX 3090, 4090, 5090, L40S, H100 and H200, each against the GPU's measured roofline. An RTX 5090 decodes the 27B as fast as an H100.

For one person chatting with a local model, decode speed is set by one number more than any other: how fast the GPU moves memory. Every token reads every weight once. So the fastest a GPU can possibly go is its memory speed divided by the model's size. That ceiling is the roofline.

The tables below are measured, one user at a time, batch 1. "Share of roofline" is how close each GPU got to its own ceiling.

Qwen3.8-27B, 4-bit

GPUClassMemory, measuredBest enginetok/sShare of roofline
H100 80GB HBM3data center3237 GB/svLLM (W4A16-AWQ)87.841%
RTX 5090consumer1710 GB/svLLM (W4A16-AWQ)87.580%
H200 NVLdata center4513 GB/sllama.cpp (UD-Q4_K_M)78.627%
RTX 4090consumer962 GB/svLLM (W4A16-AWQ)54.488%
RTX 3090consumer898 GB/svLLM (W4A16-AWQ)46.280%
L40Sdata center762 GB/svLLM (W4A16-AWQ)43.689%

Qwen3-14B, 4-bit

GPUClassMemory, measuredBest enginetok/sShare of roofline
RTX 5090consumer1710 GB/svLLM (AWQ)153.076%
H100 80GB HBM3data center3237 GB/svLLM (AWQ)147.639%
H200 NVLdata center4513 GB/sllama.cpp (Q4_K_M)135.026%
RTX 4090consumer962 GB/svLLM (AWQ)97.786%
RTX 3090consumer898 GB/svLLM (AWQ)82.478%
L40Sdata center762 GB/svLLM (AWQ)77.787%

What the numbers say

An RTX 5090 decodes the 27B as fast as an H100. 87.5 tokens per second against 87.8. The H100 moves memory almost twice as fast (3,237 GB/s against 1,710), but at one user it reaches only about 40% of its roofline. The H200 does worse, about 27%.

Consumer cards run close to their ceiling. The 3090, 4090 and 5090 reach 73 to 88% of their roofline. At one user, the small fixed costs of each step (launching kernels, waiting between them) are a small share of a consumer card's token. On a data-center card the token is so short that those same costs dominate.

So for one user, buy memory speed you can use. Data-center GPUs are built to serve many users at once. That is where their bandwidth pays off. For a single chat, a 5090 gives the same tokens per second as an H100 for this model.

What this does not cover

  • One user only. With many users at once, the data-center cards pull ahead.
  • Models that fit in VRAM. A model larger than the card is a different problem: see the Qwen3.8-Flash-Next result, which runs a 58.4 GB model on a 32 GB card.
  • One run per point. Read a few percent of slack into every number.