Qwen3-14B AWQ on H100 80GB HBM3
awq INT4_G128_ZP+BF16
Engines
vLLM
0.31.0 ↗measured 2026-11-09 18:20 UTCnewest release then: 0.33.0 (checked 2026-11-09)
147.6 tok/s
39% of the roofline380.3 tok/s possible
- now, a token
- 6.78 ms
- floor
- 2.63 ms
- read, a token
- 8.45 GB
More
- memory, measured
- 3237 GB/s
What was run
python vllm_decode_drive.py --model Qwen3-14B-AWQ --contexts 512 --batches 1 --tokens 64 --reps 1 --once 64 --max-num-batched-tokens 584 --model-context-limit 584 --capture-pass pairedSpeed at context
128 tokens
512 tokens
Where the time goes
- Now, a token
- 6.78 ms
- Roofline floor, a token
- 2.63 ms
- Gap to the floor
- 4.15 ms
- Kernels a token
- –
Per role ms a token. floor (bytes ÷ measured memory speed) above the floor
This run timed the whole step only, so there is no per-role split.
No kernel capture for this run.
Run h100-qwen3-14b-awq-vllm
Tested on
- GPU
- H100 80GB HBM3 · 80 GB
- System RAM
- 443 GB
- CPU threads
- 52
- Memory, measured
- 3237 GB/s
- Temp, max
- 39 °C
- Power
- 143 W mean of 700 W
- SM clock
- 1979 MHz
- Driver
- 580.126.20
- Measured
- 2026-11-09 18:20 UTC · context 512 tokens
Qwen3-14B on other GPUs
H200data center166.9 tok/s · 30%RTX 5090consumer151.5 tok/s · 76%H100 80GB HBM3data center147.6 tok/s · 39%H200 NVLdata center135.0 tok/s · 26%RTX 4090consumer97.7 tok/s · 86%RTX 3090consumer82.4 tok/s · 78%L40Sdata center77.7 tok/s · 87%
Best engine and quant on each GPU, one user. The percentage is that GPU's own roofline.