Pipelines and Runners¶
Execution tools in earth2studio.run make it easy to run workflows composing
multiple components (models, data sources, etc.) over many dates and ensemble
members. This is done using three core layers, from smallest to largest:
- A
WorkItemidentifies one forecast to produce: an initialization time, a final lead time, and optional ensemble members. An inference campaign usually spans many different work items. - A
Runnerdefines what is run for each work item. It holds the models and data sources and specifies how an item's inputs are fetched, how models are stepped, and what streams of output data are produced. - A
Pipelineexecutes theRunnerover many work items. It acts as an orchestrator: distributing items across ranks, selecting, reducing and regridding outputs, writing each output stream to storage, and potentially resuming interrupted runs or scoring the outputs as they're produced.
Changing what a forecast produces means choosing or writing a Runner; changing
how and where forecasts run means configuring the Pipeline. Models can also be
called directly (see Prognostic Models); Pipeline execution is
best suited for more complex workflows and larger inference campaigns run at scale.
Interface¶
Work Items¶
A WorkItem describes one forecast: a reference time, a final_lead_time, and
optional ensemble member_ids. Build items with work_items:
import numpy as np
from earth2studio.run import work_items
times = [np.datetime64("2024-01-01T00"), np.datetime64("2024-01-02T00")]
items = work_items(times, final_lead_time=np.timedelta64(240, "h"))
ensemble = work_items(
times, final_lead_time=np.timedelta64(240, "h"), members=8, member_batch=4
)
member_batch sets how many members each item runs together. Each item produces
lead times from zero through its final lead time, which must be a multiple of the
model's step.
Runners¶
A Runner defines the core execution loop for a WorkItem and declares its inputs
(data fetches) and outputs (prediction fields). Runners group outputs into
distinct streams: each stream is a named sequence of outputs with one set of
variables, one grid, and one lead-time pattern.
| Member | Purpose |
|---|---|
supports_member_batching |
Whether one item may carry several members. |
to(device) |
Place models on a device, and fetch inputs onto it. |
output_coords(final_lead_time) |
Declare each stream's output signature without running. |
data_requests(item) |
Describe every fetch an item will make, or None if not known upfront. |
run_item(item) |
Yield one mapping per step, from stream name to output field. |
A runner is built once with its models and sources, then reused for every item;
data is fetched, from live or predownloaded sources, inside run_item.
The built-in runners are:
PrognosticRunnerrolls out one prognostic model, yielding theforecaststream. Each diagnostic indiagnosticsadds a stream under its key. It accepts an optionalperturbationfor initial-condition ensembles.DiagnosticRunnerapplies diagnostics directly to source data. Stepkreads the source at the item's time plusk * step.CoupledRunner(earth2studio.coupling) runs several models together; see Coupled Models.
Sources default to each model's default_sources(). The protocol is defined in
earth2studio.run.runner.Runner; inheritance is not required.
Pipeline¶
A Pipeline provides the outer orchestration layer around a Runner, managing
distributed execution of work items, tracking progress, and writing each output
stream to its store.
| Argument | Purpose |
|---|---|
runner |
Runner executing each item. |
output |
IO backend per stream; streams left out are not written. |
output_variables |
Variables kept per stream. |
reduction |
Lead-time reduction per stream, such as LeadWindow("7d") for weekly means. |
regridder |
Regridder per stream, applied after variable selection and any reduction. |
scorer |
Online scoring hook, called with each output before it is written. |
progress |
ProgressStore recording finished items so a rerun resumes. |
device |
Runner device; "auto" selects cuda:{LOCAL_RANK} or CPU. |
At least one of output and scorer is required.
Construct a Pipeline explicitly from a Runner and per-stream settings, or use
Pipeline.from_model(model, source, ...) to build one for a single prognostic
model, as shown below.
See E2S-11: Execution Contracts for the full work item, runner, pipeline, seeding, and progress requirements.
Usage¶
Run a Forecast¶
Given a loaded prognostic model, from_model runs it from a data source and writes
the forecast stream:
from earth2studio.data import GFS
from earth2studio.io import ZarrBackend
from earth2studio.run import Pipeline
Pipeline.from_model(
model,
GFS(),
output={"forecast": ZarrBackend("forecast.zarr")},
output_variables={"forecast": ["t2m", "u10m", "v10m"]},
).run(items)
The store is planned from all items before any runs, so its time axis spans every
initialization. The initial condition is written at lead time zero when the model's
outputs include its inputs. Pass ensemble items to write a store with an ensemble
dimension.
Add Diagnostics or Use a Runner Directly¶
Each diagnostic yields its own stream, written to its own backend:
from earth2studio.models.dx import PrecipitationAFNO
from earth2studio.perturbation import Gaussian
from earth2studio.run import PrognosticRunner
precip = PrecipitationAFNO.load_model(PrecipitationAFNO.load_default_package())
runner = PrognosticRunner(
model,
GFS(),
diagnostics={"precip": precip},
perturbation=Gaussian(noise_amplitude=0.05),
)
Pipeline(
runner,
output={"forecast": forecast_io, "precip": precip_io},
).run(ensemble)
Each member's perturbation is seeded from the runner's seed, the item time, and
the member ID, so results are reproducible regardless of how members are batched.
Stochastic models are seeded once per item; use member_batch=1 to reproduce each
member independently.
Runners can also be used on their own, without Pipeline:
Resume and Distribute¶
A ProgressStore marks an item done once its outputs are flushed. Running the
same items again skips finished ones:
from earth2studio.run import ProgressStore
progress = ProgressStore("out/.progress", tags={"model": "SFNO"})
Pipeline.from_model(model, GFS(), output=output, progress=progress).run(items)
The store refuses to resume a run whose stream layout, seed, member grouping, or
tags differ. It cannot tell models apart, so record the model and checkpoint in
tags. An interrupted item reruns in full.
For multi-GPU runs, launch the script with torchrun and initialize a process
group before calling run. Each rank runs a contiguous slice of the remaining
items, and rank 0 creates the stores. Build backends on every rank, letting rank 0
create storage first:
from earth2studio.run.work import rank0_first
output = rank0_first(lambda: {"forecast": ZarrBackend("out/forecast.zarr")})
Distributed stores must use chunks of 1 along time and ensemble, which is the
ZarrBackend default; Pipeline checks this before running.
Coupled Models¶
CoupledRunner runs several models as named components. Connector declarations
describe which component feeds another's inputs, including variable mapping, time
selection, and optional regridding. Each component runs at its own cadence and
writes its own stream; inputs not fed by a connector are fetched from sources.
Because CoupledRunner is a runner, the same Pipeline drives it. For example,
DLESyM's atmosphere and ocean each advance in 96-hour cycles, publishing outputs
every 6 and 48 hours respectively. The final lead time must be a multiple of the
96-hour cycle:
from earth2studio.coupling import Connector, CoupledRunner
runner = CoupledRunner(
{"atmos": atmos, "ocean": ocean},
[
Connector("ocean", "atmos", time="hold"), # Latest SST before each cycle
Connector("atmos", "ocean"), # Atmospheric means per ocean step
],
)
final_lead_time = np.timedelta64(192, "h") # Two coupled cycles
print(runner.describe(final_lead_time)) # Inspect the derived schedule
coupled_items = work_items(times, final_lead_time=final_lead_time)
Pipeline(runner, output={"atmos": atmos_io, "ocean": ocean_io}).run(coupled_items)
See E2S-12: Coupling for connector and scheduling requirements, and the DLESyM seasonal example for a complete coupled run.
Write Your Own Runner¶
Any object with the runner members above works with Pipeline. Wrap a built-in
runner to change its behavior, such as stopping a rollout early, or write one for
a workflow the built-ins do not cover. Keep stream names stable across items and
runs, since they key output stores and resume. Declare every stream in
output_coords, and give outputs with different grids or cadences separate
streams. Return None from data_requests when fetches depend on results.