E2S-10: Earth2Studio IO Backend Protocol¶
Purpose¶
Write model and data-source DataArrays to storage without converting them to
tensors and coordinate dictionaries, preserving the coordinates and metadata that
make stored fields self-describing. Models, sources and regridders already exchange
xr.DataArray (see E2S-06_MODEL_CONTRACT_SPEC.md); this contract
moves IO onto the same payload so drivers write what models yield:
This revision also adds lifecycle methods, so the protocol changes once for DataArray writes and finalization together.
Contract¶
Interface¶
@runtime_checkable
class IOBackend(Protocol):
def add_array(self, template: xr.DataArray) -> None: ...
def write(self, x: xr.DataArray) -> None: ...
def flush(self) -> None: ...
def close(self) -> None: ...
add_arraycreates storage for the arrays described bytemplate: dimension and auxiliary coordinates, then one array per variable.writestoresxat the positions its coordinate labels identify.flushblocks until every earlier write is visible to readers of the store. The backend stays writable. For a transactional store, those readers are the open transaction's; other readers see the writes once committed.closeflushes, finalizes the store and releases resources. Transactional backends commit on close (Icechunk), soclosemay raise if finalizing fails. Later writes raise; repeatedclosecalls are no-ops.
Backend-specific methods (read, __getitem__, commits, to_xarray) remain on
the implementations and are not part of the protocol. Any store can be opened with
Xarray (see Round Trip).
A backend whose storage several processes can share may also expose
write_blocks: dict[str, int]: the largest chunk or shard extent per dimension
over its stored data arrays, the smallest unit two writers can update
independently. Distributed execution reads it to reject stores that ranks would
share (see E2S-11_EXECUTION_CONTRACT_SPEC.md). ZarrBackend,
IceChunkBackend and AsyncZarrBackend provide it.
Templates¶
A template is a DataArray whose field values are never read: normally an
allocation-free coord_array signature, though any DataArray works. add_array
reads only its dimensions, coordinates, dtype, name and attributes.
signature = model.output_coords(model.input_coords()) # (batch, lead_time, ...)
template = output_template(
signature,
leading={"ensemble": members, "time": times}, # replaces the dynamic prefix
coords={"lead_time": leads}, # run extent of fixed dims
)
io.add_array(template) # (ensemble, time, lead_time, variable, lat, lon)
- Concrete. Every dimension is nonempty, none is dynamic, and dimension labels
are unique. Planning
signatures must be concretized first (see
output_templateunder Shared Helpers);add_arrayraisesValueErrorotherwise. - Naming. With a
variabledimension, each label becomes one stored array over the remaining dimensions, in template order. Without one, the template'snamenames a single array; a template with neither raisesValueError. Array names share a namespace with coordinates, so a name matching a coordinate raisesValueError. Labels are used verbatim, including temporal-statistic qualifiers such astp:sum:6h(see Array Names). - dtype. Arrays take the template's dtype;
coord_arraydefaults tofloat32. Unwritten positions hold the backend's fill value, NaN for floating dtypes. - One extent per dimension. Arrays in one backend share each dimension's
coordinates. Adding an array whose shared dimension or auxiliary coordinates
differ from the store raises
ValueError. Re-adding an existing array with the same dimensions is a no-op, so reopened stores and resumed runs can calladd_arrayunconditionally. - Fixed extent. A store's dimensions do not grow after creation. Appending along a dimension is out of scope (see Open Questions).
Outputs whose coordinates differ go to separate backends. A multi-slot model's y is a tuple of
DataArrays with differing coordinates, so drivers write each slot to its own backend; one backend
may hold several slots only when their shared dimensions agree.
Writes¶
write(x) locates x in the store by coordinate label, never by position.
xhas the dimensions of its target arrays in the same order. Missing, additional or reordered dimensions raiseValueError.- Along each dimension,
xholds any nonempty subset of the store's labels, in any order and without duplicates. Dimensions without labels must span the whole stored axis; they never broadcast. An unknown label raisesValueErrorbefore anything is written. Severallead_timeentries in one write are allowed, so chunked model steps write directly. xwrites one array pervariablelabel, or the array namedx.namewhen it has novariabledimension. Writing to an array thatadd_arraydid not create raisesValueError; writes never create arrays implicitly.- Auxiliary coordinates and attributes on
xare not compared with the store or written again.add_arraywrote them once. - Backends may restrict which subsets they accept and must document it, raising
ValueErrorfor unsupported writes.AsyncZarrBackend, for example, requires dimensions outside its parallel dimensions to be written in full.
Payloads and ownership¶
x may be NumPy-, CuPy- or Torch-adapter-backed on any device. Moving data to the
storage device is the backend's job; callers never convert first.
A backend borrows x and never modifies its values, coordinates or attributes. A
backend that returns before the data is stored copies whatever it needs first, so
callers may reuse or release x immediately. This mirrors the model ownership rules
and protects rollouts from backends that would otherwise convert units or dtypes in
place.
Array Names¶
Stored array names are the variable labels, verbatim. Statistic qualifiers carry
quantity identity (see E2S-03_TIME_STATISTICS_SPEC.md), so the
name is enough to rebuild earth2studio_statistics with coord_array or
coord_array_like on read. No statistics attribute is stored.
Zarr and netCDF accept : in names. Some environments do not: local Zarr stores
create one directory per array, and Windows forbids : in file names. A backend
may offer an opt-in name mapping for these cases. A mapped store records the original
label in each array's earth2studio_variable attribute so readers can restore it.
Mapping is never the default.
Metadata¶
add_array persists:
- dimension coordinates with their attributes;
- auxiliary coordinates (curvilinear and projected latitude/longitude, point
geometry) as arrays over their own dimensions. Each data array lists them in a
CF
coordinatesattribute so Xarray opens them as coordinates; - the template's attributes on every data array, including grid metadata
(
earth2studio_grid_id,earth2studio_crsand the definition's attributes) and user metadata such asunits.
It drops the signature markers earth2studio_kind, earth2studio_schema_version
and earth2studio_dynamic_dims, and earth2studio_statistics, which the labels
determine. A format that cannot store an attribute value natively, such as a
dictionary or list in netCDF, stores it as a JSON string.
Round Trip¶
After flush, opening the store with Xarray (xr.open_zarr, xr.open_dataset, or
the in-memory dataset) yields, for each array:
- the written values, with fill values at unwritten positions;
- dimension coordinates equal to the template's:
timeasdatetime64[ns],lead_timeastimedelta64[ns], and other labels unchanged; - the template's auxiliary coordinates and coordinate attributes;
- the persisted attributes above, so
infer_gridrecovers the template's grid.
Rules¶
| Rule | Requirement |
|---|---|
I1 |
add_array never reads field values from the template |
I2 |
add_array rejects dynamic or zero-sized dimensions with ValueError |
I3 |
Each variable label, verbatim, names one array; otherwise the template's name does |
I4 |
Arrays share coordinates; re-adding an identical array is a no-op, from any instance |
I5 |
write locates by label; unknown labels, arrays or dims raise ValueError first |
I6 |
write accepts any label subset, including several lead_time entries |
I7 |
write never modifies x; non-blocking backends copy before returning |
I8 |
write accepts NumPy-, CuPy- and Torch-adapter-backed arrays |
I9 |
After flush, the store round-trips values, coordinates and persisted attributes |
I10 |
Signature markers and earth2studio_statistics are not persisted |
I11 |
close flushes and finalizes; later writes raise and repeated closes are no-ops |
A shared test suite in test/io/ checks these rules for every migrated backend.
There is no runtime conformance checker: backends are few and rarely added.
Shared Helpers¶
earth2studio/io/utils.py holds the logic previously duplicated in each backend:
plan_arrays validates a template against the store and lists what to create, and
plan_write/plan_read validate a field or selection and map its labels to stored
positions (contiguous runs become slices). Backends implement only storage.
It also plans templates for drivers, replacing per-driver and per-recipe output coordinate assembly:
def output_template(
signature: xr.DataArray,
leading: Mapping[str, ArrayLike],
coords: Mapping[str, ArrayLike] | None = None,
) -> CoordinateSystem: ...
leadingreplaces the signature's dynamic prefix with concrete dimensions, in the given order. It may hold more or fewer dimensions than the prefix it replaces.coordsreplaces labels of fixed dimensions, such as the run'slead_timeextent. Spatial replacements raise, as incoord_array_like.- Grid metadata, auxiliary coordinates, dtype, name and user attributes carry over; the result is allocation-free and has no dynamic dimensions.
Migration¶
The legacy tensor protocol is replaced, not deprecated alongside. Development will proceed in two steps:
- Protocol and foundation. This spec, the protocol, shared helpers, the shared
test suite and
XarrayBackend.run.pyplans its template with a DataArray helper replacing_output_dimensions. Unmigrated backends keep the legacy signature, andrun.pyannotatesiowith a private copy of the legacy protocol so drivers and CI keep working. - Backends and drivers. Migrate
ZarrBackend,IceChunkBackend,NetCDF4Backend,KVBackendandAsyncZarrBackend. Moverun.deterministic,run.diagnosticandrun.ensembletoio.write(y), and examples, recipes and tests with them. Remove the private legacy protocol,_output_dimensionsandsplit_coordsfrom IO paths. Recipes with their own scoring IO, such as s2s, migrate separately.
Breaking changes for users:
add_array(coords, array_name, data=...)becomesadd_array(template); initialize values with awriteinstead ofdata=.write(tensor, coords, array_name)becomeswrite(x).- Writes no longer create arrays implicitly.
- Curvilinear stores no longer add synthetic
ilat/ilondimensions. Arrays use the grid's own dimensions (y,x; see E2S-02_GRID_SPEC.md) with latitude/longitude as auxiliary coordinates, so readers of existing stores change.
Open Questions¶
- Should stores grow along a dimension (
timefor operational cycling, or runs whose extent is discovered rather than declared)? Zarr supports resizing, but growing a store from several ranks needs coordinated position assignment. AsyncZarrBackendtakes its parallel dimensions and their full values at construction. Should it derive them from the firstadd_arraytemplate instead?- Chunking options differ by backend (
chunks,chunked_coords,shard_coords). Should they share one name and meaning? - Should backends export statistics as CF
cell_methodsand time bounds, alongside the qualified labels? CF would namet2mandt2m:mean:24hthe same, so they could not share a store.