Megatron-Energon Documentation

This is the documentation of Megatron’s multi-modal data loader “Energon”.

We recommend getting started in the Introduction section, which explains what Energon is and how to install it.

Once installed, check out the Basic Usage section starting with Quickstart for some basic examples and tutorials. Some underlying concepts, will be explained in the rest of that section.

For specific use cases and advanced usage, please read Advanced Usage.

In the end you will also find some documentation on how to interface with energon programmatically and how to contribute to the code base.

Introduction

• General
• Installation
  • Normal Installation
  • Installation from Source Code

Basic Usage

• Quickstart
  • Tutorial 1: Preparing the Dataset
  • Tutorial 2: Loading a Dataset
  • Tutorial 3: Batch Size
  • Tutorial 3: Blending using Metadataset
  • Tutorial 4: Distributed Loading
  • Tutorial 5: Saving and Restoring
  • More Features
• Data Preparation
  • Step 1: Creating a WebDataset
  • Step 2: Preparing the Dataset
  • Dataset Format on Disk
• Data Flow
  • Dataset Flavors
• Task Encoder
• Metadataset
  • Subflavors
  • Classes
  • Functions
• Save and Restore
  • Scenarios
  • Summary

Advanced Usage

• Remote Dataset
  • Prerequisites
  • The URL syntax
• Custom Sample Loader
  • Example: Interleaved Data and Arbitrary Image Count
• Crude Data and How to Cook It 👨‍🍳
  • Other Filters for Cookers
• Reproducible Scaling
• Packing
  • How to pack samples on the fly
• Grouping
  • Example use cases
  • How to group
  • Fixed global batch size
• Joining Datasets
  • Example File Structure
  • Minimal Example
  • Extensive Example
  • Custom Join Type
• Epochized Blending
• Customized Blending
• Parallelism
  • Example

API

• Packages and Modules
  • megatron.energon
• Command-Line Interface
  • energon prepare
  • energon info
  • energon lint
  • energon preview

Internals

• Contribution Guidelines
• Code Structure

Indices and tables

• Index

• Module Index