Flash-MSA: Accelerating Million-Token Training With Sparse Attention Kernels
- [Github] [MiniMax Paper] [Trainer] Flash-MSA vs Flash-Attention isolated train step.1 Several frontier models [1, 2, 3, 4, 5] use sparse attention to greatly speedup their inference, though no one has posted code to train it efficiently.
- Today I introduce the world's first performant open-source training kernels for Minimax Sparse Attention in CuTeDSL for Hopper and Blackwell GPUs.
- I did all of the dev work on Spheron H100 and B200 rentals and with the help of referencing FA4, MSA inference, and Codex.
Unverified
- [Github] [MiniMax Paper] [Trainer] Flash-MSA vs Flash-Attention isolated train step.1 Several frontier models [1, 2, 3, 4, 5] use sparse attention to greatly speedup their inference, though no one has posted code to train it efficiently.
- Today I introduce the world's first performant open-source training kernels for Minimax Sparse Attention in CuTeDSL for Hopper and Blackwell GPUs.
- I did all of the dev work on Spheron H100 and B200 rentals and with the help of referencing FA4, MSA inference, and Codex.
Sources: Github