Qwen3-8B Q4_K_M on RTX 3090
GGUF Q4_K+Q6_K
Engines
llama.cpp
b11433 · 50569eb87 ↗measured 2026-11-09 17:01 UTCnewest release then: 0.9.0 (checked 2026-11-09)
124.5 tok/s
66% of the roofline188.9 tok/s possible
- now, a token
- 8.03 ms
- floor
- 5.29 ms
- read, a token
- 4.75 GB
More
- memory, measured
- 898 GB/s
What was run
llama-bench -m Qwen3-8B-Q4_K_M.gguf -p 0 -n 32 -d 512 -ngl 99 -r 1 --no-warmup -o json -ub 512 -b 2048 -ctk f16 -ctv f16 -fa -1Speed at context
128 tokens
512 tokens
Where the time goes
- Now, a token
- 8.03 ms
- Roofline floor, a token
- 5.29 ms
- Gap to the floor
- 2.74 ms
- Kernels a token
- 253
Launch heavy: 690 kernel launches per token
Per role ms a token. floor (bytes ÷ measured memory speed) above the floor
Per kernel (2): ms now / ms at the roofline, launches a token
No floor = the kernel reads no weights (norms, conversions, copies): all of its time is gap.
Run qwen3-8b-q4-k-m-nvidia_geforce_rtx_3090_24g-001
Tested on
- GPU
- RTX 3090 · 24 GB
- System RAM
- 252 GB
- CPU threads
- 80
- Memory, measured
- 898 GB/s
- Temp, max
- 60 °C
- Power
- 146 W mean of 280 W
- SM clock
- 1869 MHz, throttled
- Driver
- 595.71.05
- Measured
- 2026-11-09 17:01 UTC · context 512 tokens · measured, in-model (nsys): 7 of 7 weight roles timed
Qwen3-8B on other GPUs
H100 80GB HBM3data center227.0 tok/s · 33%RTX 3090consumer124.5 tok/s · 66%RTX 5090consumer95.9 tok/s · 85%
Best engine and quant on each GPU, one user. The percentage is that GPU's own roofline.