boltbeam.fyi

Qwen3-8B Q4_K_M on RTX 3090

GGUF Q4_K+Q6_K

Engines

llama.cpp

b11433 · 50569eb87 ↗measured 2026-11-09 17:01 UTCnewest release then: 0.9.0 (checked 2026-11-09)

124.5 tok/s

66% of the roofline188.9 tok/s possible

now, a token
8.03 ms
floor
5.29 ms
read, a token
4.75 GB
More
memory, measured
898 GB/s

What was run

llama-bench -m Qwen3-8B-Q4_K_M.gguf -p 0 -n 32 -d 512 -ngl 99 -r 1 --no-warmup -o json -ub 512 -b 2048 -ctk f16 -ctv f16 -fa -1

Speed at context

128 tokens
llama.cpp124.5 tok/s of 191.2
512 tokens
llama.cpp124.5 tok/s of 188.9
Measured at short contexts only. The pink tick is the roofline at that context.

Where the time goes

Now, a token
8.03 ms
Roofline floor, a token
5.29 ms
Gap to the floor
2.74 ms
Kernels a token
253

Launch heavy: 690 kernel launches per token

Per role ms a token. floor (bytes ÷ measured memory speed) above the floor

attention0.42 / 0.09 ms 21%
attn_qo Q4_K1.06 / 0.76 ms 72%
ffn_gate_up Q4_K2.46 / 2.27 ms 92%
attn_kv Q4_K0.28 / 0.14 ms 50%
ffn_down Q6_K0.96 / 0.83 ms 86%
ffn_down Q4_K0.65 / 0.57 ms 87%
attn_kv Q6_K0.13 / 0.07 ms 55%
lm_head Q6_K0.58 / 0.57 ms 98%
kernel gaps2.21 ms · no floor
norm0.50 ms · no floor
quantize0.35 ms · no floor
KV cache write0.06 ms · no floor
copy0.00 ms · no floor
elementwise0.00 ms · no floor
Per kernel (2): ms now / ms at the roofline, launches a token
mul_mat_vec_q · Q4_K6.43 / 3.74 ms ×216
mul_mat_vec_q · Q6_K1.91 / 1.47 ms ×37

No floor = the kernel reads no weights (norms, conversions, copies): all of its time is gap.

Run qwen3-8b-q4-k-m-nvidia_geforce_rtx_3090_24g-001

Tested on

GPU
RTX 3090 · 24 GB
System RAM
252 GB
CPU threads
80
Memory, measured
898 GB/s
Temp, max
60 °C
Power
146 W mean of 280 W
SM clock
1869 MHz, throttled
Driver
595.71.05
Measured
2026-11-09 17:01 UTC · context 512 tokens · measured, in-model (nsys): 7 of 7 weight roles timed

Qwen3-8B on other GPUs

H100 80GB HBM3data center227.0 tok/s · 33%RTX 3090consumer124.5 tok/s · 66%RTX 5090consumer95.9 tok/s · 85%

Best engine and quant on each GPU, one user. The percentage is that GPU's own roofline.