sdm.nn

RotaryEmbedding	Rotary Positional Embedding (RoPE) from the "RoFormer: Enhanced Transformer with Rotary Position Embedding" paper.
SwiGLU	\(\mathrm{FFN}_{\mathrm{SwiGLU}}\) block from the "GLU Variants Improve Transformer" paper.
SoftplusScale	Apply a learned positive scale to the final input dimension.
QueryScaling	Base class for query scaling modules in SDPA.
QASSMax	Query-Aware Scalable SoftMax (QASSMax).
LogScale	Logarithmic query scaling.
GatedLogScale	Gated logarithmic query scaling.
SDPA	Scaled Dot-Product Attention (SDPA).
Attention	Multi-head attention layer with grouped-query attention support.
TransformerBlock	Transformer block with normalization and feedforward residual modules.
InducedTransformerBlock	Transformer block using learned inducing points.