XMA (Accelerated Model Architectures)

XMA is a repository comprising of fast kernels for model training. We are planning on adding lots of experimental and fun model architectures with support for multiple accelerators like NVIDIA, AMD GPUs, Google TPUs and Amazon Trainiums.

Installation

git clone https://github.com/open-lm-engine/accelerated-model-architectures
cd accelerated-model-architectures
pip install .
cd ..

Layers

Layer

CUDA

MPS

Pallas

NKI

ROCm

Triton

GRU

❌

❌

❌

❌

❌

✅

LinearAttention

❌

❌

❌

❌

❌

✅

M2RNN

❌

❌

❌

❌

❌

✅

MoE

✅

❌

❌

❌

❌

✅

RNN

❌

❌

❌

❌

❌

✅

Functional

Function

CUDA

MPS

Pallas

NKI

ROCm

Triton

continuous_count

✅

❌

❌

❌

❌

❌

cross_entropy

❌

❌

❌

❌

❌

✅

fused_linear_cross_entropy

❌

❌

❌

❌

❌

✅

fused_residual_add_rmsnorm

❌

❌

❌

❌

❌

✅

gru

❌

❌

❌

❌

❌

✅

linear_attention

❌

❌

❌

❌

❌

✅

m2rnn

❌

❌

❌

❌

❌

✅

p_norm

❌

❌

❌

❌

❌

✅

pack_sequence

✅

❌

❌

❌

❌

✅

rmsnorm

❌

❌

❌

❌

❌

✅

rnn

❌

❌

❌

❌

❌

✅

softmax

❌

❌

❌

❌

❌

✅

swiglu

✅

✅

❌

✅

❌

✅

swiglu_packed

✅

❌

❌

✅

❌

✅

unpack_sequence

✅

❌

❌

❌

❌

✅

Community

Join the Discord server if you are interested in LLM architecture or distributed training/inference research.