sdm.nn

RotaryEmbedding	Rotary Positional Embedding (RoPE) from the "RoFormer: Enhanced Transformer with Rotary Position Embedding" paper.
SinusoidalEmbedding	Sinusoidal positional embedding from the "Attention Is All You Need" paper.
ResidualMLP	Two-layer MLP with a learned linear residual projection.
SwiGLU	\(\mathrm{FFN}_{\mathrm{SwiGLU}}\) block from the "GLU Variants Improve Transformer" paper.
SoftplusScale	Apply a learned positive scale to the final input dimension.
RMSNorm	A torch.nn.RMSNorm that dispatches to an efficient kernel.
RevIN	Reversible instance normalization from the "Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift" paper.
QueryScaling	Base class for query scaling modules in SDPA.
QASSMax	Query-Aware Scalable SoftMax (QASSMax).
LogScale	Logarithmic query scaling.
GatedLogScale	Gated logarithmic query scaling.
SDPA	Scaled Dot-Product Attention (SDPA).
Attention	Multi-head attention layer with grouped-query attention support.
TransformerBlock	Transformer block with normalization and feedforward residual modules.
InducedTransformerBlock	Transformer block using learned inducing points.