dataloaders

DataLoader utilities for language model training and validation.

Functions

create_train_dataloader

Create an infinite training DataLoader over ConstantLengthDataset.

create_validation_dataloader

create_padded_tensor

prepare_validation_dataloader

Build the shared deterministic validation loader without a model backend dependency.

prepare_automodel_text_validation_dataloader

Build AutoModel-native padded or neat-packed text validation data.

prepare_multimodal_validation_dataloader

Build the native AutoModel processor/collator validation path for VLM batches.
create_padded_tensor(tensor, desired_shape, padding_value=0)
Parameters:

  • tensor (TensorT)

  • desired_shape (Sequence[int])

  • padding_value (float)

Return type:

TensorT

create_train_dataloader(seed, tokenizer, block_size, dataset_path, content_field, fim_rate, fim_spm_rate, micro_batch_size, load_dataset_fn=<function load_from_disk_fn>, dataset_name='train', keep_in_memory=False, shuffle_seed=None, source_datasets_to_discard=(), bos_rate=1.0, num_workers=0, packed_token_cache_path=None)

Create an infinite training DataLoader over ConstantLengthDataset.

Parameters:

  • seed (int)

  • tokenizer (PreTrainedTokenizerBase)

  • block_size (int)

  • dataset_path (str | Mapping[str, Dataset])

  • content_field (str)

  • fim_rate (float)

  • fim_spm_rate (float)

  • micro_batch_size (int)

  • load_dataset_fn (LoadDatasetFn)

  • dataset_name (str)

  • keep_in_memory (bool)

  • shuffle_seed (int | None)

  • source_datasets_to_discard (Sequence[str])

  • bos_rate (float)

  • num_workers (int)

  • packed_token_cache_path (str | Path | None)

Return type:

DataLoader

create_validation_dataloader(accelerator, seed, tokenizer, block_size, dataset, content_field, fim_rate, fim_spm_rate, micro_batch_size, eval_samples=None, load_dataset_fn=<function load_from_disk_fn>, dataset_name='__auto__', keep_in_memory=False, source_datasets_to_discard=(), bos_rate=1.0, varlen=True, shuffle_seed=None, realized_cache_dir=None, packed_token_cache_path=None)
Parameters:

  • accelerator (Accelerator | None)

  • seed (int)

  • tokenizer (PreTrainedTokenizerBase)

  • block_size (int)

  • dataset (str | Mapping[str, Dataset])

  • content_field (str)

  • fim_rate (float)

  • fim_spm_rate (float)

  • micro_batch_size (int)

  • eval_samples (int | None)

  • load_dataset_fn (LoadDatasetFn)

  • dataset_name (str)

  • keep_in_memory (bool)

  • source_datasets_to_discard (Sequence[str])

  • bos_rate (float)

  • varlen (bool)

  • shuffle_seed (int | None)

  • realized_cache_dir (str | Path | None)

  • packed_token_cache_path (str | Path | None)

prepare_automodel_text_validation_dataloader(args, *, tokenizer, data_layout)

Build AutoModel-native padded or neat-packed text validation data.

Parameters:

  • tokenizer (PreTrainedTokenizerBase)

  • data_layout (str)

prepare_multimodal_validation_dataloader(args, *, checkpoint_dir, data_layout)

Build the native AutoModel processor/collator validation path for VLM batches.

Parameters:

  • args (Mapping[str, Any])

  • checkpoint_dir (str | Path)

  • data_layout (str)

prepare_validation_dataloader(args, tokenizer=None, *, data_layout=None)

Build the shared deterministic validation loader without a model backend dependency.

Parameters:

  • tokenizer (PreTrainedTokenizerBase | None)

  • data_layout (str | None)