loky

Loky (joblib) execution backend.

Uses joblib.Parallel with the loky backend for robust parallel execution. Loky provides better process management than the standard multiprocessing module.

Classes

LokyBackend

Execute pipeline items using joblib with the loky backend.

Module Contents

class physicsnemo_curator.run.loky.LokyBackend

Bases: physicsnemo_curator.run.base.RunBackend

Execute pipeline items using joblib with the loky backend.

Loky is a robust process executor that handles worker crashes gracefully and provides better memory management than standard multiprocessing.

Warning

Stateful filters accumulate per-process state that is not merged back into the parent process.

Backend Options

preferstr

Soft hint for parallelization (“processes” or “threads”).

requirestr

Hard constraint for parallelization.

verboseint

Verbosity level (0-50). If not set, uses 10 when progress=True.

batch_sizeint | str

Number of tasks per batch (“auto” or int).

pre_dispatchstr | int

Number of batches to pre-dispatch.

temp_folderstr | None

Folder for memmapping large arrays.

timeoutfloat | None

Timeout in seconds for retrieving results.

run(

pipeline: physicsnemo_curator.core.base.Pipeline[Any],

config: physicsnemo_curator.run.base.RunConfig,

) → list[list[str]]

Execute pipeline indices using joblib/loky.

Parameters:

• pipeline (Pipeline) – The pipeline to execute.

• config (RunConfig) – Execution configuration.

Returns:

Sink outputs, one list per index.

Return type:

list[list[str]]

Raises:

ImportError – If joblib is not installed.

description: ClassVar[str] = 'Joblib with loky backend (robust process management)'

name: ClassVar[str] = 'loky'

requires: ClassVar[tuple[str, Ellipsis]] = ('joblib',)