Llama-3.3-70B-Instruct AWQ on 2× RTX 3090
awq INT4_G128_ZP+F16
Engines
vLLM
0.31.0 ↗measured 2026-11-09 02:12 UTC
35.2 tok/s
84% of the roofline42.1 tok/s possible
- now, a token
- 28.42 ms
- floor
- 23.75 ms
- read, a token
- 21.15 GB
Speed at context
128 tokens
512 tokens
Where the time goes
- Now, a token
- 28.42 ms
- Roofline floor, a token
- 23.75 ms
- Gap to the floor
- 4.68 ms
- Kernels a token
- –
Per role ms a token. floor (bytes ÷ measured memory speed) above the floor
This run timed the whole step only, so there is no per-role split.
No kernel capture for this run.
Run 3090x2-vllm-70b-awq-row
Tested on
- GPU
- 2× RTX 3090 · 24 GB each
- System RAM
- 63 GB
- CPU threads
- 8
- Memory, measured
- 896 GB/s
- Temp, max
- see notes
- Power
- see notes
- SM clock
- –
- Driver
- 595.112.07
- Measured
- 2026-11-09 02:12 UTC · context 512 tokens · measured, in-model (nsys): the whole step only, no per-role time
Llama-3.3-70B-Instruct on other GPUs
Llama-3.3-70B-Instruct has only been measured on the 2× RTX 3090 so far.