Skip to main content

Ctrl+K

Transformer Engine 2.20.0.dev0

GitHub

Transformer Engine 2.20.0.dev0

GitHub

Table of Contents

Home

Getting Started

Installation
Getting Started
Frequently Asked Questions (FAQ)
Transformer Engine vX.YZ Release Notes

Python API documentation

Common API
Framework-specific API
- PyTorch
- Jax

Features

Low precision training
Other optimizations
- CPU Offloading

Examples and Tutorials

Using FP8 and FP4 with Transformer Engine
Performance Optimizations
Accelerating Hugging Face Llama 2 and 3 Fine-Tuning with Transformer Engine
Accelerating Hugging Face Gemma Inference with Transformer Engine
Accelerating Hugging Face Mixtral MoE Fine-Tuning with Transformer Engine
Export to ONNX and inference using TensorRT
JAX: Integrating TransformerEngine into an existing framework
Operation fuser API
GEMM Profiling Tutorial

Advanced

C/C++ API
Precision debug tools
Environment Variables
Attention Is All You Need!
Deep Dive into CP + THD + AG + Striped>1 + SWA support for Transformer Engine JAX

JAX: Integrating TransformerEngine into an existing framework
JAX: Expert Parallelism with TransformerEngine

Is this page helpful?

JAX: Expert Parallelism with TransformerEngine#

TODO — Coming soon.

← Back to the JAX integration overview

previous

JAX: Context-Parallel Attention with TransformerEngine

next

Operation fuser API

Privacy Policy | Your Privacy Choices | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2022-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved..