Megatron-Energon Documentation

This is the documentation of Megatron’s multi-modal data loader “Energon”.

We recommend getting started in the Introduction section, which introduces this package and contains a few code snippets to get you started quickly.

To better understand the underlying concepts, please read the Basic Concepts section.

In the end you will also find a code documentation with the most important classes and methods.

Introduction

• General
• Quickstart
• Installation
  • Normal Installation
  • Installation from Source Code

Usage

• Simple Library Usage
  • Tutorial 1
  • Tutorial 2
  • Tutorial 3
  • Tutorial 4
  • Tutorial 5
• FAQ
  • What’s the best way to combine multiple datasets together?
  • On autoresume, the dataset restarts from the beginning
  • How to customize the blending of datasets through metadatasets
  • Process batches of mixed datasets separately (before mixing samples)
• Command-Line Interface
  • energon prepare
  • energon lint
• Data Preparation
  • Webdataset format
  • Convert to webdataset
• Remote Dataset
  • Prerequisites
  • The URL syntax

Basic Concepts

• Dataset Format on Disk
  • Files in .nv-meta
• Data Flow
  • Dataset Flavors
  • Task Encoders
• Dataset Concepts
  • Dataset Design Guidelines
  • Statistical Analysis of Dataset Loading
  • Types
• Task Encoders
• Metadataset
  • Customized Blending
  • Classes
  • Functions
• Blending datasets
  • Mixing datasets of homogeneous type
  • Mixing datasets of heterogeneous types
  • Sample weights

Advanced Usage

• Custom Sample Loader
• Crude Data and How to Cook It 👨‍🍳
  • Other Filters for Cookers
• Reproducible Scaling
• Packing
  • How to pack samples on the fly
• Joining Datasets / Adding Columns to Metadataset
  • Minimal Example
  • Extensive Example
  • Custom Join Type
  • Example Structure

Developer's Manual

• Contribution Guidelines
• Packages and Modules
  • megatron.energon

Indices and tables

• Index

• Module Index