boltbeam.fyi

Qwen3-8B BF16 on RTX 5090

safetensors BF16

Qwen's BF16 safetensors; llama.cpp ran unsloth's BF16 GGUF of the same weights (unsloth/Qwen3-8B-GGUF).

Engines

TensorRT-LLM

1.2.1 ↗measured 2026-11-09 22:46 UTC

95.9 tok/s

85% of the roofline112.4 tok/s possible

now, a token
10.43 ms
floor
8.90 ms
read, a token
15.22 GB

Speed at context

128 tokens
TensorRT-LLM96.1 tok/s of 112.8
512 tokens
TensorRT-LLM95.9 tok/s of 112.4
The pink tick is the roofline at that context.

Where the time goes

Now, a token
10.43 ms
Roofline floor, a token
8.90 ms
Gap to the floor
1.53 ms
Kernels a token
405

Cannot split: attn_kv BF16. No kernel group matched their calls and bytes

Per role ms a token. floor (bytes ÷ measured memory speed) above the floor

attn_qo BF161.95 / 1.41 ms 72%
attention0.35 / 0.05 ms 13%
ffn_down BF162.36 / 2.12 ms 90%
ffn_gate_up BF164.36 / 4.24 ms 97%
lm_head BF160.73 / 0.73 ms 99%
other0.47 ms · no floor
kernel gaps0.20 ms · no floor
elementwise0.07 ms · no floor
copy0.00 ms · no floor
limit of roles not split out0.00 ms · no floor
Per kernel (20): ms now / ms at the roofline, launches a token
sm80_xmma_gemm_bf16bf16_bf16f32_f32_tn_n_tilesize32x32x64_stage6_warpsize2x2x1_tensor16x8x16_execute_kernel_trt grid=1x768x1 block=128x1x1 · BF164.36 / 4.24 ms ×36
sm80_xmma_gemm_bf16bf16_bf16f32_f32_tn_n_tilesize32x32x64_stage6_warpsize2x2x1_tensor16x8x16_execute_kernel_trt grid=1x128x1 block=128x1x1 role=ffn_down · BF162.36 / 2.12 ms ×36
sm80_xmma_gemm_bf16bf16_bf16f32_f32_tn_n_tilesize32x32x64_stage6_warpsize2x2x1_tensor16x8x16_execute_kernel_trt grid=1x192x1 block=128x1x1 · BF161.15 / 1.15 ms ×36
sm80_xmma_gemm_bf16bf16_bf16f32_f32_tn_n_tilesize32x32x64_stage6_warpsize2x2x1_tensor16x8x16_execute_kernel_trt grid=1x128x1 block=128x1x1 role=attn_qo · BF160.80 / 0.80 ms ×36
sm80_xmma_gemm_bf16bf16_bf16f32_f32_tn_n_tilesize32x32x64_stage6_warpsize2x2x1_tensor16x8x16_fused grid=1x4748x1 block=128x1x1 · BF160.73 / 0.73 ms ×1
tensorrt_llm::_v1::kernels::mmha::masked_multihead_attention_kernel0.35 ms · no floor ×36
__myl_AddCastMulMeanAddSqrtDivMulCastMul_0x97b088fb90bfac882f7e93343b239b21_com_vec grid=1x1x1 block=256x1x10.17 ms · no floor ×71
tensorrt_llm::_v1::kernels::batchApplyPenalty0.12 ms · no floor ×1
tensorrt_llm::_v1::kernels::addBiasSoftMax0.07 ms · no floor ×1
__myl_SlicAddSqrtDivMulCastMulAddSqrtDivMulCastMulConc_0x04970cc326eb89f245740ba790b01f42_com_vec grid=6x1x1 block=256x1x10.05 ms · no floor ×36
__myl_SiluMul_0x25eea546d980c4542edb933ba6d5e74c_com_vec grid=6x1x1 block=256x1x10.04 ms · no floor ×36
__myl_SlicCastMulMean_0x35cd38c00c83f1e6176c03463b10a79c_com grid=32x1x1 block=128x1x10.03 ms · no floor ×36
__myl_SlicCastMulMean_0xb573e987908735ca1cd479bc711e6a26_com grid=8x1x1 block=128x1x10.03 ms · no floor ×36
tensorrt_llm::_v1::kernels::topKStage10.02 ms · no floor ×1
__myl_GathCastMulMeanAddSqrtDivMulCastMul_0x16ddbece911f1b78bf06483280f8dc08_com grid=1x1x1 block=256x1x10.00 ms · no floor ×1
tensorrt_llm::_v1::kernels::topKStage2Sampling0.00 ms · no floor ×1
__myl_AddCastMulMean_0x9509b312e137e786b2c5bcc258932a8e_com_vec grid=1x1x1 block=256x1x10.00 ms · no floor ×1
__myl_AddAddSqrtDivMulCastMulGath_0xde205e21caa0752819b22f8b2502c596_com grid=16x1x1 block=256x1x10.00 ms · no floor ×1
tensorrt_llm::_v1::kernels::copyNextStepIds0.00 ms · no floor ×1
tensorrt_llm::_v1::kernels::lengthCriterion0.00 ms · no floor ×1

No floor = the kernel reads no weights (norms, conversions, copies): all of its time is gap.

Run rtx5090-qwen3-8b-bf16-trtllm

Tested on

GPU
RTX 5090 · 32 GB
System RAM
31 GB
CPU threads
16
Memory, measured
1710 GB/s
Temp, max
see notes
Power
see notes
SM clock
–
Driver
595.99.02
Measured
2026-11-09 22:46 UTC · context 512 tokens · measured, in-model (nsys): 4 of 5 weight roles timed

Qwen3-8B on other GPUs

H100 80GBllama.cpp · Q4_K_M227.0 tok/s · 33%RTX 3090llama.cpp · Q4_K_M124.5 tok/s · 66%RTX 5090TensorRT-LLM · BF1695.9 tok/s · 85%

The fastest setup measured on each GPU, one user. The percentage is that GPU's own roofline.