3.5
FLOPs per token (6N)6 × 109
Tokens per step per GPU16,384
FLOPs per step per GPU100.7 TFLOP
MI250X peak (BF16/die)95.74 TFLOP/s
Model FLOP utilization30.0%
30.0%
0%25%50%75%100%
28.7
useful TFLOP/s per die
35-45%
Llama (attn, NVIDIA)
46-58%
PaLM (attn, TPUv4)
30% on AMD with Mamba SSM scans is reasonable — attention parallelizes better