Transformer Engine

2.17.0.dev0-54e5bfc
Version select:

Home

Getting Started

Installation
Getting Started
Frequently Asked Questions (FAQ)
- FP8 checkpoint compatibility

Python API documentation

Common API
Framework-specific API
- PyTorch
- Jax

Features

Low precision training
Other optimizations
- CPU Offloading

Examples and Tutorials

Using FP8 and FP4 with Transformer Engine
Performance Optimizations
Accelerating Hugging Face Llama 2 and 3 Fine-Tuning with Transformer Engine
Accelerating Hugging Face Gemma Inference with Transformer Engine
Accelerating Hugging Face Mixtral MoE Fine-Tuning with Transformer Engine
Export to ONNX and inference using TensorRT
JAX: Integrating TransformerEngine into an existing framework
Operation fuser API
GEMM Profiling Tutorial

Advanced

C/C++ API
Precision debug tools
Environment Variables
Attention Is All You Need!
Deep Dive into CP + THD + AG + Striped>1 + SWA support for Transformer Engine JAX

Transformer Engine

JAX: Integrating TransformerEngine into an existing framework
JAX: Expert Parallelism with TransformerEngine
View page source

JAX: Expert Parallelism with TransformerEngine

TODO — Coming soon.

← Back to the JAX integration overview

Previous Next

Privacy Policy | Manage My Privacy | Do Not Sell or Share My Data | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

© Copyright 2022-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved..