lmms

Run lmms-eval for a local checkpoint with durable attempt artifacts.

Functions

run_lmms_eval_checkpoint

Evaluate one local checkpoint and preserve an isolated lmms-eval attempt.
exception LmmsEvalTimeoutError

Bases: TimeoutError

Report a timed-out lmms-eval process with its captured output.

__init__(argv, timeout, *, output, stderr)
Parameters:

  • argv (Sequence[str])

  • timeout (float)

  • output (str)

  • stderr (str)

run_lmms_eval_checkpoint(checkpoint, *, output_root, settings)

Evaluate one local checkpoint and preserve an isolated lmms-eval attempt.

Parameters:

  • checkpoint (str | Path) – Local Hugging Face checkpoint directory.

  • output_root (str | Path) – Root under which a unique attempt directory is created.

  • settings (Mapping[str, Any]) – lmms-eval tasks, vLLM model arguments, topology, and runtime controls.

Returns:

Flattened metrics and paths to the normalized summary, raw result, command, stdout, and stderr artifacts.

Return type:

dict[str, Any]