- Now, a token
- 11.43 ms
- Roofline floor, a token
- 9.11 ms
- Gap to the floor
- 2.31 ms
- Kernels a token
- 790
Per role ms a token. floor (bytes ÷ measured memory speed) above the floor
ffn_gate_up INT4_G128_ZP3.85 / 3.47 ms 90%
ffn_down INT4_G128_ZP1.94 / 1.73 ms 89%
attn_gate + attn_qkv INT4_G128_ZP1.39 / 1.22 ms 88%
recurrent0.34 / 0.18 ms 53%
ssm_out INT4_G128_ZP0.62 / 0.46 ms 74%
ssm_alpha_beta BF160.19 / 0.03 ms 15%
lm_head BF161.63 / 1.49 ms 92%
attention0.11 / 0.02 ms 20%
attn_kv + attn_qo INT4_G128_ZP0.43 / 0.36 ms 83%
attn_qo INT4_G128_ZP0.21 / 0.15 ms 75%
triton_red_fused__to_copy_add_fused_add…0.36 ms · no floor
kernel gaps0.19 ms · no floor
flash::flash_fwd_splitkv_combine_kernel0.06 ms · no floor
layer_norm_fwd_kernel0.05 ms · no floor
_fused_qk_rmsnorm_rope_gate_kernel0.03 ms · no floor
24 other small kernels0.15 ms · no floor
Per kernel (33): ms now / ms at the roofline, launches a token
Marlin8.44 / 7.40 ms ×256
gemm · lm_head BF161.63 / 1.49 ms ×1
triton_red_fused__to_copy_add_fused_add…0.36 ms · no floor ×128
_causal_conv1d_update_kernel + fused_recurrent_gated_delta_rule_packed_decode_kernel0.34 / 0.18 ms ×96
gemm · ssm_alpha_beta BF160.19 / 0.03 ms ×48
flash_fwd_splitkv_kernel0.11 / 0.02 ms ×16
flash::flash_fwd_splitkv_combine_kernel0.06 ms · no floor ×16
layer_norm_fwd_kernel0.05 ms · no floor ×48
_fused_qk_rmsnorm_rope_gate_kernel0.03 ms · no floor ×16
triton_poi_fused_mul_silu_slice_10.03 ms · no floor ×48
vllm::reshape_and_cache_flash_kernel0.02 ms · no floor ×16
_gumbel_sample_kernel0.02 ms · no floor ×1
triton_poi_fused_zeros_30.01 ms · no floor ×32
triton_poi_fused_mul_sigmoid_view_00.01 ms · no floor ×16
triton_poi_fused_mul_silu_slice_20.01 ms · no floor ×16
at::native::index_elementwise_kernel0.01 ms · no floor ×3
triton_poi_fused_zeros_40.01 ms · no floor ×15
triton_red_fused__to_copy_add_embedding…0.01 ms · no floor ×1
_bias_kernel0.00 ms · no floor ×1
at::native::reduce_kernel0.00 ms · no floor ×1
_gather_block_tables_kernel0.00 ms · no floor ×1
_post_update_kernel0.00 ms · no floor ×1
_compute_slot_mappings_kernel0.00 ms · no floor ×1
at::native::unrolled_elementwise_kernel0.00 ms · no floor ×1
_combine_sampled_and_draft_tokens_kernel0.00 ms · no floor ×1
_prepare_rope_positions_kernel0.00 ms · no floor ×1
_get_num_sampled_and_rejected_kernel0.00 ms · no floor ×1
at::native::vectorized_elementwise_kern…0.00 ms · no floor ×3
at::native::_scatter_gather_elementwise…0.00 ms · no floor ×1
_prepare_pos_seq_lens_kernel0.00 ms · no floor ×1
No floor = the kernel reads no weights (norms, conversions, copies): all of its time is gap.
Run rtx5090-27b-vllm-batch-b