boltbeam.fyi

Qwen3.8-27B W4A16-AWQ on RTX 3090

safetensors INT4_G128_ZP+BF16

Engines

vLLM

0.31.0 ↗measured 2026-11-09 18:14 UTCnewest release then: 0.33.0 (checked 2026-11-09)

46.2 tok/s

80% of the roofline57.6 tok/s possible

now, a token
21.64 ms
floor
17.36 ms
read, a token
15.59 GB
More
memory, measured
898 GB/s

What was run

python vllm_decode_drive.py --model Qwen3.8-27B-W4A16-AWQ --contexts 512 --batches 1 --tokens 64 --reps 1 --once 64 --max-num-batched-tokens 584 --model-context-limit 584 --capture-pass paired

Speed at context

128 tokens
vLLM47.4 tok/s
512 tokens
vLLM46.2 tok/s of 57.6
Measured at short contexts only. The pink tick is the roofline at that context.

Where the time goes

Now, a token
21.64 ms
Roofline floor, a token
17.36 ms
Gap to the floor
4.28 ms
Kernels a token
838

Per role ms a token. floor (bytes ÷ measured memory speed) above the floor

ffn_gate_up INT4_G128_ZP7.59 / 6.61 ms 87%
ffn_down INT4_G128_ZP3.78 / 3.30 ms 87%
ssm_out INT4_G128_ZP1.35 / 0.88 ms 65%
attn_gate + attn_qkv INT4_G128_ZP2.77 / 2.33 ms 84%
recurrent0.64 / 0.34 ms 54%
attention0.21 / 0.04 ms 20%
ssm_alpha_beta BF160.20 / 0.05 ms 26%
attn_kv + attn_qo INT4_G128_ZP0.83 / 0.68 ms 82%
attn_qo INT4_G128_ZP0.40 / 0.29 ms 73%
lm_head BF162.86 / 2.83 ms 99%
triton_red_fused__to_copy_add_fused_add…0.49 ms · no floor
flash::flash_fwd_splitkv_combine_kernel0.11 ms · no floor
cublasLt::splitKreduce_kernel0.10 ms · no floor
triton_poi_fused_mul_silu_slice_10.10 ms · no floor
layer_norm_fwd_kernel0.10 ms · no floor
26 other small kernels0.38 ms · no floor
Per kernel (34): ms now / ms at the roofline, launches a token
Marlin16.71 / 14.09 ms ×256
gemv2T_kernel_val3.06 / 2.89 ms ×49
_causal_conv1d_update_kernel + fused_recurrent_gated_delta_rule_packed_decode_kernel0.64 / 0.34 ms ×96
triton_red_fused__to_copy_add_fused_add…0.49 ms · no floor ×128
flash_fwd_splitkv_kernel0.21 / 0.04 ms ×16
flash::flash_fwd_splitkv_combine_kernel0.11 ms · no floor ×16
cublasLt::splitKreduce_kernel0.10 ms · no floor ×48
triton_poi_fused_mul_silu_slice_10.10 ms · no floor ×48
layer_norm_fwd_kernel0.10 ms · no floor ×48
_fused_qk_rmsnorm_rope_gate_kernel0.05 ms · no floor ×16
vllm::reshape_and_cache_flash_kernel0.04 ms · no floor ×16
triton_poi_fused_zeros_30.04 ms · no floor ×32
triton_poi_fused_mul_silu_slice_20.03 ms · no floor ×16
triton_poi_fused_mul_sigmoid_view_00.03 ms · no floor ×16
_gumbel_sample_kernel0.02 ms · no floor ×1
triton_poi_fused_zeros_40.02 ms · no floor ×15
at::native::index_elementwise_kernel0.01 ms · no floor ×3
_bias_kernel0.01 ms · no floor ×1
_gather_block_tables_kernel0.01 ms · no floor ×1
at::native::reduce_kernel0.01 ms · no floor ×1
_prepare_rope_positions_kernel0.01 ms · no floor ×1
_combine_sampled_and_draft_tokens_kernel0.01 ms · no floor ×1
at::native::unrolled_elementwise_kernel0.00 ms · no floor ×1
triton_red_fused__to_copy_add_embedding…0.00 ms · no floor ×1
_post_update_kernel0.00 ms · no floor ×1
_get_num_sampled_and_rejected_kernel0.00 ms · no floor ×1
at::native::vectorized_elementwise_kern…0.00 ms · no floor ×3
_compute_slot_mappings_kernel0.00 ms · no floor ×1
at::native::_scatter_gather_elementwise…0.00 ms · no floor ×1
_prepare_pos_seq_lens_kernel0.00 ms · no floor ×1

No floor = the kernel reads no weights (norms, conversions, copies): all of its time is gap.

Run rtx3090-qwen3.8-27b-awq-vllm

Tested on

GPU
RTX 3090 · 24 GB
System RAM
252 GB
CPU threads
80
Memory, measured
898 GB/s
Temp, max
59 °C
Power
144 W mean of 280 W
SM clock
1762 MHz, throttled
Driver
595.71.05
Measured
2026-11-09 18:14 UTC · context 512 tokens · measured, in-model (nsys): split by role from the engine's own labels, 9 weight roles

Qwen3.8-27B on other GPUs