Qwen3-8B Q4_K_M on H100 80GB
GGUF Q4_K+Q6_K
Engines
llama.cpp
b11433 · 50569eb87 ↗measured 2026-11-08 20:30 UTC
227.0 tok/s
33% of the roofline684.4 tok/s possible
- now, a token
- 4.41 ms
- floor
- 1.46 ms
- read, a token
- 4.75 GB
Speed at context
128 tokens
512 tokens
Where the time goes
- Now, a token
- 4.41 ms
- Roofline floor, a token
- 1.46 ms
- Gap to the floor
- 2.94 ms
- Kernels a token
- 690
Launch heavy: 690 kernel launches per token
Per role ms a token. floor (bytes ÷ measured memory speed) above the floor
Per kernel (10): ms now / ms at the roofline, launches a token
No floor = the kernel reads no weights (norms, conversions, copies): all of its time is gap.
Run h100-in-model
Tested on
- GPU
- H100 80GB · 80 GB
- System RAM
- 2015 GB
- CPU threads
- 208
- Memory, measured
- 3252 GB/s
- Temp, max
- see notes
- Power
- see notes
- SM clock
- –
- Driver
- 595.91.07
- Measured
- 2026-11-08 20:30 UTC · context 512 tokens · measured, in-model (nsys): 7 of 7 weight roles timed
Qwen3-8B on other GPUs
H100 80GBllama.cpp · Q4_K_M227.0 tok/s · 33%RTX 3090llama.cpp · Q4_K_M124.5 tok/s · 66%RTX 5090TensorRT-LLM · BF1695.9 tok/s · 85%
The fastest setup measured on each GPU, one user. The percentage is that GPU's own roofline.