boltbeam.fyi

Qwen3-8B Q4_K_M on H100 80GB

GGUF Q4_K+Q6_K

Engines

llama.cpp

b11433 · 50569eb87 ↗measured 2026-11-08 20:30 UTC

227.0 tok/s

33% of the roofline684.4 tok/s possible

now, a token
4.41 ms
floor
1.46 ms
read, a token
4.75 GB

Speed at context

128 tokens
llama.cpp226.9 tok/s of 692.6
512 tokens
llama.cpp227.0 tok/s of 684.4
The pink tick is the roofline at that context.

Where the time goes

Now, a token
4.41 ms
Roofline floor, a token
1.46 ms
Gap to the floor
2.94 ms
Kernels a token
690

Launch heavy: 690 kernel launches per token

Per role ms a token. floor (bytes ÷ measured memory speed) above the floor

ffn_gate_up Q4_K1.41 / 0.63 ms 45%
attn_kv Q4_K0.36 / 0.04 ms 11%
attn_qo Q4_K0.51 / 0.21 ms 41%
ffn_down Q6_K0.53 / 0.23 ms 43%
ffn_down Q4_K0.34 / 0.16 ms 46%
attention0.13 / 0.02 ms 18%
lm_head Q6_K0.26 / 0.16 ms 60%
attn_kv Q6_K0.08 / 0.02 ms 24%
kernel gaps1.42 ms · no floor
quantize0.30 ms · no floor
norm0.23 ms · no floor
KV cache write0.06 ms · no floor
copy0.00 ms · no floor
elementwise0.00 ms · no floor
Per kernel (10): ms now / ms at the roofline, launches a token
mul_mat_vec_q · Q4_K2.62 / 1.03 ms ×180
mul_mat_vec_q · Q6_K0.87 / 0.40 ms ×37
quantize_q8_10.30 ms · no floor ×217
rms_norm_f320.13 ms · no floor ×73
rms_norm_mul_rope_f320.10 ms · no floor ×72
flash_attn_ext_vec0.07 ms · no floor ×36
flash_attn_combine_results0.07 ms · no floor ×36
k_set_rows0.06 ms · no floor ×36
k_get_rows_float0.00 ms · no floor ×2
k_bin_bcast0.00 ms · no floor ×1

No floor = the kernel reads no weights (norms, conversions, copies): all of its time is gap.

Run h100-in-model

Tested on

GPU
H100 80GB · 80 GB
System RAM
2015 GB
CPU threads
208
Memory, measured
3252 GB/s
Temp, max
see notes
Power
see notes
SM clock
–
Driver
595.91.07
Measured
2026-11-08 20:30 UTC · context 512 tokens · measured, in-model (nsys): 7 of 7 weight roles timed

Qwen3-8B on other GPUs

H100 80GBllama.cpp · Q4_K_M227.0 tok/s · 33%RTX 3090llama.cpp · Q4_K_M124.5 tok/s · 66%RTX 5090TensorRT-LLM · BF1695.9 tok/s · 85%

The fastest setup measured on each GPU, one user. The percentage is that GPU's own roofline.