E2S-12: Earth2Studio Coupling¶
Purpose¶
Run several models together, each at its own cadence and on its own grid, with
fields passed between them, and run the result under Pipeline like any other
Runner. Components are ordinary models: the coupler never wraps, subclasses or
mutates them, so every component also runs alone and under its conformance
checker.
Status: earth2studio.coupling exports Connector, CoupledRunner and
CouplingError. Tests are in test/coupling/; the tutorials are
dev/examples/10_coupling.py and dev/examples/11_stormscope_coupling.py.
DLESyMAtmos and DLESyMOcean are the atmosphere and ocean of the fused
DLESyM as standalone models; examples/06_seasonal/02_dlesym_example.py
couples them.
Lineage: this replaces the nvcoupler prototype of PR #1229, which grew out of
PR #1114 (the hand-computable toy models, lagged versus sequential coupling and
the DLESyM split originate there), and keeps their intent (connectors between
components, a derived schedule, eager validation and a readable plan) while building on the model, regrid, time-statistics and
execution contracts. The impact-modeling requirements it serves are in the
earth2-impact-modeling docs (CR1–CR7).
Contract¶
Interface¶
from earth2studio.coupling import Connector, CoupledRunner
runner = CoupledRunner(
{"weather": weather_model, "soil": soil_model},
[Connector("weather", "soil", regridder=to_soil)],
sources={"weather": weather_ic, "soil": (soil_ic, None, retrievals)},
)
print(runner.describe(np.timedelta64(72, "h")))
Pipeline(runner, output={"weather": w, "soil": s}).run(items)
@dataclass(frozen=True)
class Connector:
src: str
dst: str
slot: int = 0
variables: Mapping[str, str] | Sequence[str] | None = None # fed base -> src variable; None feeds all
regridder: Regridder | None = None
time: Literal["exact", "hold"] = "exact"
state: bool = False
class CoupledRunner: # satisfies earth2studio.run.Runner
supports_member_batching = True
def __init__(
self, components, connectors=(), sources=None, seed=0, devices=None,
source_member=None, perturbation=None,
): ...
def describe(self, final_lead_time) -> str: ...
def to(self, device) -> CoupledRunner: ...
def device_of(self, name) -> torch.device: ...
def output_coords(self, final_lead_time) -> Mapping[str, CoordinateSystem]: ...
def data_requests(self, item) -> tuple[DataRequest, ...]: ...
def run_item(self, item) -> Iterator[Mapping[str, xr.DataArray]]: ...
sources maps a component to one source per input-then-forcing slot, in the
order of default_sources(); a single source covers the first input slot and
None entries fall back to the model's recommendation, which is built only
when a fetch needs it. CouplingError is a
ValueError. A sequence of variables keeps each name.
devices pins components that to leaves in place, such as CPU-only impact
models; the rest move with to. Each component's fetches and connected fields
are staged onto its device, and a connector's regridder runs there.
Components¶
| Protocol | Runs | Calls |
|---|---|---|
PrognosticModel |
Every cadence | initialize(*x, *forcing), then step(*y, *forcing, state=state) |
StatefulAssimilationModel |
Every cadence | As above; DataFrame slots may be None |
DiagnosticModel |
At each valid time its inputs arrive | model(*inputs) |
A component may have several gridded output slots. A stateful component's
output slots share their lead times; models whose outputs advance at different
cadences, such as fused DLESyM, are split into components. The cadence is the
model's step interval: the outputs' last lead time minus the later of the first
input slot's last lead time and zero. Output chunk k covers the output lead times
plus k cadences and is due at its last lead time; chunk 0 comes from
initialize. A component whose forcing reduces a day of another's output
therefore initializes once that day exists, with no special case.
Imports are a stateful component's time-varying gridded forcing slots, or a
diagnostic's input slots. Connectors feed an import: each feeds the labels its
variables names, or every label without variables, so several connectors can
feed one slot from different producers. Labels no connector feeds, and every
other slot, are fetched from sources. Input slots, static forcing and DataFrame slots are
always fetched. A diagnostic input slot without lead_time, as most
diagnostics declare, takes the call's valid time as a leading dimension, as
under DiagnosticRunner.
A state slot is a stateful component's input slot whose output slot in the same
position declares its variables on its grid. A state connector (state=True) overwrites
the variables it names there before each call, for models that carry a coupled
field such as SST in their own state: in the inputs initialize receives, at
the input slot's lead times, and in the output each step advances, at its
lead times. Several state connectors may share a slot if they overwrite
different variables. Required times resolve as for imports, except that values
the producer cannot supply at or before the item time are kept rather than
fetched. The overwrite is not published; streams carry each model's own output.
Connectors and resolution¶
For each call, every import needs a set of valid times, following the model
contract: initialize takes the whole forcing window; step takes the window's
final lead time shifted by each lead time of the output it advances; a
diagnostic takes the valid time it is called for. For each required time T
and slot label:
- Split the label:
tp:sum:24his basetpwith modifiersum:24h. Map the base throughvariablesto the producer's variable, which exactly one of the producer's output slots publishes. - Plain label,
exact: the producer's frame atT. Plain label,hold: the latest frame at or beforeTthat the producer published on an earlier tick. Qualified label: the producer's frames in(T - w, T]at its frame spacing, reduced withapply_time_statistic. - If the producer cannot supply it and
Tis at or before the item time, a stateful consumer fetches it from the slot's source. A window ending after the item time whose only missing frames are at or before it, as when the producer publishes no initial condition, takes those frames of the base variable from the slot's source and reduces them with the producer's. Otherwise the consumer waits: frames after the item time always come from the producer. - Stage producer frames onto the consumer's device, regrid them with the connector's regridder, then map onto the slot signature with the slot's temporal-statistics metadata.
Connectors select, reduce and regrid; they do not convert. Derived variables,
unit changes and vertical levels are a DiagnosticModel component placed
between the models, such as DerivedWS for wind speed or
DerivedPressureLevels, which interpolates pressure or model levels onto
pressure levels. The component publishes its own stream, so conversions are
visible in describe and in the outputs.
Scheduling¶
Ticks are the lead times at which some stateful component's chunk is due. Before the first tick, every stateful component fetches its inputs, holds them as frames at their lead times, and publishes an initial condition for each output slot that declares its same-position input slot's variables on the same grid. State slots come first under the model contract, so output-only slots start at the first chunk. At each tick, components run in declaration order whenever their imports resolve, repeating until nothing more can run. A component whose import waits on a frame published later runs later; a diagnostic runs once per valid time at which all its exact inputs exist.
Lagged and sequential coupling are not declared. An exact slot needing a
frame at the same valid time as its producer's output runs after it; a hold
slot, or one declaring an earlier lead time, lags. A loop of exact edges
cannot run and is reported as a cycle.
Construction dry-runs the schedule over two periods of the cadences' least
common multiple beyond the latest first chunk. The same scheduler drives
describe, output_coords, data_requests and run_item, so a plan cannot
drift from a run. Frames older than the oldest one any consumer read in that
dry run, plus the longest cadence, are dropped as the run advances.
Known limitation: early reduction windows are partly backfilled from sources,
so the dry run can see a long window below its full width. A window much
longer than its consumer's step (about 30 h or more for a 6 h consumer, 96 h
or more for a 24 h consumer) can then need older frames than are kept, and the
run fails partway with a CouplingError that names the history kept for the
producer. No shipped model or example reads such a window.
A work item's final lead time must be a lead time at which some stateful
component's output is due; any other value raises in describe,
output_coords, data_requests and run_item, as PrognosticRunner raises
for a final lead time that is not a multiple of its step. A slower component
still stops at its last whole chunk.
Components' iterator hooks (front_hook, rear_hook) are not run: the runner
calls initialize and step, which apply no hooks (MODEL_CONTRACT P10). The
runner logs a warning at construction when a stateful component sets one. Use
perturbation= for the initial condition, or a diagnostic component or
connector for per-step changes.
Streams¶
Each output slot yields one stream: named after the component when it has one
output slot, else name.0, name.1 and so on. Stream names must be unique. A
stateful slot publishes its initial condition, when it has one, and every chunk;
a diagnostic slot publishes at every call.
Streams carry the producing model's grid and the item's time, as X1–X4
require. Seeding follows X5, from a base seed offset per component
by the CRC-32 of its name, so stochastic components draw independent noise.
perturbation maps stateful component names to initial-condition
perturbations, applied to the first input slot per member as X6 requires,
from the same offset seed.
With source_member, every fetched slot whose source is an ensemble source reads
the item's mapped member, as X22 requires; connected slots read what their
producer published, and DataFrame slots are shared. Batched members travel on a
leading ensemble dimension through connectors, buffers, time statistics,
regridders and state connectors; one observation frame serves every member, so a
DA component must accept a batched background with unbatched observations.
Independent assimilation models¶
An AssimilationModel without initialize and step, such as HealDA, computes
one analysis per call from observation DataFrames, with no lead times. It is not
a component today: as a diagnostic it would have no connected input to tell it
when to run, so the runner rejects its DataFrame slots. It fits elsewhere:
- As a source. Initializing a prognostic model from HealDA means its input
slot's source is an analysis. A
DataSourceadapter fetches each observation slot at the requested times, calls the model and returns the analysis, so it servesPrognosticRunnerandCoupledRunneralike, behindRegriddedSourcewhen grids differ. Models still never fetch (P22); the adapter does. - As a component, later. Publishing analyses on a schedule during a run needs components that run on a fixed cadence from sources alone.
- Cycling is different. An analysis that replaces a prognostic model's state
mid-run, with a forecast as background, is a
StatefulAssimilationModel, or warm-start work items (execution contract open decision 1).
Rules¶
| Rule | Requirement |
|---|---|
C1 |
Components are prognostic, stateful assimilation or diagnostic models with gridded output slots; a stateful component's slots carry the same lead_time |
C2 |
Stateful components step at their declared cadence; diagnostics run when their inputs arrive and need at least one exact connector |
C3 |
Connectors target a time-varying gridded forcing slot or a diagnostic input slot; several may feed one slot if their labels differ; state connectors target state slots and may share one if their variables differ |
C4 |
Every label a connector feeds has a base that, after variables, exactly one producer output slot publishes; variables names declared bases |
C5 |
Without a regridder each producer slot read and the consumer slot have the same geometry; with one, its source grid matches each slot read and its target grid the consumer slot |
C6 |
Required valid times follow the model contract; DataFrame slots are requested at the chunk's end, static slots only initially |
C7 |
exact waits for frames at each required time; hold takes the latest earlier-published frame and applies to plain labels only |
C8 |
Qualified labels reduce producer frames over (T - w, T]; the window must divide by the producer's frame spacing |
C9 |
Frames at or before the item time that the producer does not hold come from the slot's source, for stateful consumers only, including the early frames of a window ending later; state connectors keep the model's own values instead |
C10 |
An empty DataFrame from a source is passed as None |
C11 |
A stateful chunk due by the final lead time that cannot run raises, naming the cycle or the missing frame; construction checks this first |
C12 |
One uniquely named stream per output slot; describe, output_coords and data_requests come from the scheduler that runs the item; data_requests lists fetch_data calls only, not DataFrame fetches |
Out of scope or future work¶
- Time interpolation and conservative regridding. Unit conversion and
vertical interpolation are diagnostic components, such as
DerivedPressureLevels. - A
DataSourceadapter over independent assimilation models, with per-member seeding when they are stochastic. - Components that run on a fixed cadence from sources alone, such as observation-only analyses.
- DataFrame payloads through connectors, and DataFrame-valued streams.
- Output slots that advance at different lead times; split them, as for DLESyM.
- Mid-item checkpoints and iterator hooks on coupled models.
- Fields with a
leveldimension. Levels are encoded in variable names; a diagnostic component can map other layouts onto them. - Training rollouts: nothing differentiates through the exchange, which stages and regrids frames outside autograd.
Open decisions¶
- Units. Connected slots exchange lexicon units. A lexicon units table would let connectors reject renames across units and sums over rates.
- DataFrame streams.
Runner.run_itemyields DataArrays only, which blocks tabular products such as TC tracks as coupled outputs.