packed_memmap

Read-only fixed-sequence token storage shared by every distributed stage.

Classes

PackedTokenMemmapDataset

Map N x (S+1) uint32 tokens to shifted input/target tensors.

class PackedTokenMemmapDataset

Bases: Dataset

Map N x (S+1) uint32 tokens to shifted input/target tensors.

__init__(path, *, limit=None, sequence_length=None)

Parameters:

• path (str | Path)

• limit (int | None)

• sequence_length (int | None)