- Now, a token
- 2.64 ms
- Roofline floor, a token
- 1.46 ms
- Gap to the floor
- 1.18 ms
- Kernels a token
- 534
Per role ms a token. floor (bytes ÷ measured memory speed) above the floor
moe_shared_expert_down NVFP4_G160.39 / 0.08 ms 19%
ssm_out FP8_E4M30.29 / 0.15 ms 51%
moe_expert_down NVFP4_G160.34 / 0.23 ms 66%
moe_expert_gate_up NVFP4_G160.33 / 0.23 ms 69%
ssm_in FP8_E4M30.47 / 0.37 ms 80%
attention0.09 / 0.00 ms 1%
moe_shared_expert_gate_up NVFP4_G160.16 / 0.08 ms 46%
mamba2 state_update0.08 / 0.03 ms 38%
mamba2 convolution0.04 / 0.00 ms 3%
moe_router F320.05 / 0.02 ms 38%
attn_kv + attn_qo BF160.11 / 0.09 ms 76%
attn_qo BF160.09 / 0.08 ms 83%
lm_head NVFP4_G160.13 / 0.12 ms 92%
kernel gaps0.08 ms · no floor
vllm::moe::single_group_topk::detail::s…0.08 ms · no floor
vllm::moe::moe_align_block_size_kernel0.07 ms · no floor
at::native::vectorized_elementwise_kern…0.06 ms · no floor
sm89_xmma_gemm_e4m3bf16_e4m3f32_f32_tn_…0.05 ms · no floor
26 other small kernels-0.26 ms · no floor
Per kernel (40): ms now / ms at the roofline, launches a token
Marlin0.68 / 0.27 ms ×47
marlin_moe_wna16::Marlin0.67 / 0.45 ms ×46
sm89_xmma_gemm_e4m3bf16_e4m3f32_f32_tn_n_tilesize64x128x64_stage4_warpsize2x2x1_tensor16x8x32_execute_kernel__5x_cublas0.47 / 0.37 ms ×23
nvjet_sm120_qqtst_mma_64x32x64_16_32x16x64_tmaAB_bz_TNNN0.29 / 0.15 ms ×23
gemm · attn_kv + attn_qo BF160.11 / 0.09 ms ×6
gemm · attn_qo BF160.09 / 0.08 ms ×6
kernel_mha0.09 / 0.00 ms ×6
vllm::moe::single_group_topk::detail::s…0.08 ms · no floor ×23
_selective_scan_update_kernel0.08 / 0.03 ms ×23
vllm::moe::moe_align_block_size_kernel0.07 ms · no floor ×23
at::native::vectorized_elementwise_kern…0.06 ms · no floor ×49
sm89_xmma_gemm_e4m3bf16_e4m3f32_f32_tn_…0.05 ms · no floor ×23
triton_per_fused__to_copy_add_clamp_mea…0.05 ms · no floor ×23
dot_kernel0.05 / 0.02 ms ×23
triton_red_fused__to_copy_add_clamp_fus…0.05 ms · no floor ×22
_causal_conv1d_update_kernel0.04 / 0.00 ms ×23
triton_red_fused__to_copy_fused_add_rms…0.03 ms · no floor ×23
triton_poi_fused_pow_relu_00.02 ms · no floor ×23
vllm::moe::moe_sum_vec_kernel0.02 ms · no floor ×23
reduce_1Block_kernel0.02 ms · no floor ×23
vllm::moe::count_and_sort_expert_tokens…0.02 ms · no floor ×23
vllm::reshape_and_cache_flash_kernel0.02 ms · no floor ×6
_gumbel_sample_kernel0.01 ms · no floor ×1
triton_red_fused_fused_add_rms_norm_vie…0.01 ms · no floor ×6
at::native::index_elementwise_kernel0.01 ms · no floor ×3
_bias_kernel0.00 ms · no floor ×1
triton_red_fused__to_copy_clamp_embeddi…0.00 ms · no floor ×1
_gather_block_tables_kernel0.00 ms · no floor ×1
at::native::unrolled_elementwise_kernel0.00 ms · no floor ×1
at::native::reduce_kernel0.00 ms · no floor ×1
No floor = the kernel reads no weights (norms, conversions, copies): all of its time is gap.
Run rtx5090-nemotron-3.5-lightning-30b-nvfp4-vllm-i