Advanced Output Handling¶
This guide outlines advanced features supported by Earth2Studio IO backends. Start with Output Handling for the common interface and creating output templates. Backend options are listed in the IO API.
Versioned Output with Icechunk¶
IceChunkBackend writes to an Icechunk repository instead
of a plain Zarr store. Commits create immutable snapshots, giving an inference
campaign a versioned history that can be reopened later.
The backend extends ZarrBackend with commit(message). Its usual template,
writing, and reading methods remain available. Install the optional dependency
with pip install 'earth2studio[data]'.
Given a prepared output template and an iterable of forecast DataArrays:
from earth2studio.io import IceChunkBackend
io = IceChunkBackend("forecast_repo", branch="main")
io.add_array(template)
for forecast in forecasts:
io.write(forecast)
snapshot_id = io.commit("forecast run 2024-01-01T00Z")
io.close()
The storage argument accepts a filesystem path or an icechunk.Storage instance;
omitting it creates an in-memory repository. A new named branch is created from
the tip of main when needed.
Writes are non-blocking by default. read, __getitem__, commit, and close
flush pending writes first; use blocking=True for synchronous writes.
flush() makes pending writes visible within the open session, while commit()
publishes a snapshot and opens a new writable session. close() commits remaining
changes with a generic message and rejects later writes. Commit explicitly when
you need a descriptive message or conflict handling. Earth2Studio workflows leave
closing the backend to the caller.
Reopen a snapshot with repo.readonly_session(snapshot_id=snapshot_id), or create
branches with repo.create_branch(...). Icechunk also supports object storage
through constructors such as icechunk.s3_storage(...).
Note
An empty commit raises an IcechunkError unless allow_empty=True is supplied.
Another writer committing to the same branch can cause a conflict; consult
Icechunk's documentation for rebasing and
rebase_with. Local filesystem storage is not safe for concurrent commits;
use object storage for that case.
Asynchronous and Sharded Zarr Output¶
AsyncZarrBackend overlaps storage writes with inference. Coordinates listed in
parallel_coords have chunk size one, allowing separate forecast slices to be
written independently. Large campaigns can consequently produce many small files.
Zarr v3 sharding groups chunks into fewer storage objects:
from collections import OrderedDict
from earth2studio.io import AsyncZarrBackend
io = AsyncZarrBackend(
"forecast.zarr",
parallel_coords=OrderedDict({"time": times, "lead_time": lead_times}),
shard_coords={"lead_time": 8},
)
io.add_array(template)
for forecast in forecasts:
io.write(forecast)
io.close()
Here times and lead_times contain the full output coordinate labels. Eight
lead-time chunks share one shard; readers can still select individual chunks.
Writes must cover the non-parallel dimensions in full. Define the layout before
writing, and close the backend to finish pending work.
Choosing a Shard Size¶
The backend buffers chunks in host memory and writes a complete shard together, avoiding repeated read-modify-write operations. Balance three costs:
- Host memory: budget approximately
max_inflight_shards * 4 * shard_bytes + pool_size * write_bytesper process. Encoded copies and concurrent flushes add to the shard buffer itself. A 73-variable, 721-by-1440 float32 field is about 0.3 GB per lead; eight leads require about 2.4 GB for one buffer before those extra costs. - Storage bandwidth:
max_inflight_shardscontrols concurrent shard flushes. Raising it helps only until bandwidth saturates. Distributed ranks already add concurrency, so fewer in-flight shards per rank may be appropriate. - Production rate: asynchronous writing hides IO latency when storage keeps pace with inference. If it cannot, reduce output volume or use faster storage.
Measure the tradeoff between memory, throughput, and file count on your storage
system. Compression through zarr_codecs applies to the inner chunks.
chunked_coords sets chunk sizes for non-parallel dimensions; parallel dimensions
remain chunked at one. Each shard size must be a multiple of its chunk size.
Partial Shards and Restarts¶
close() flushes incomplete shards, leaving unsupplied positions at the array fill
value. Writing into an existing shard falls back to read-modify-write and logs a
warning. This can happen after a mid-run flush() or when resuming a partial run.
Align restart boundaries with shard boundaries to retain the complete-shard fast path.
Multiple Processes Writing One Store¶
Warning
Each shard must belong to exactly one process. Buffers are process-local: two ranks writing different chunks in the same shard can each overwrite the whole shard, silently discarding the other's data. Read-modify-write does not protect against concurrent writers.
Shard along an axis each rank owns completely. For example, when ranks own distinct
initialization times and all leads for each forecast, sharding lead_time is safe:
io = AsyncZarrBackend(
"forecast.zarr",
parallel_coords=OrderedDict({"time": all_times, "lead_time": all_lead_times}),
shard_coords={"lead_time": 8},
)
Sharding the distributed time axis is safe only if every rank's assigned indexes
form whole shards. For example, splitting eight times across three ranks into
blocks 0–2, 3–5, and 6–7 crosses a shard boundary when shard_coords={"time": 4}.
That layout lets two ranks overwrite the same shard. Ownership must remain valid
when the number of ranks or work items changes.
Separately, concurrent creation of the same array can race: have one rank call
add_array before other ranks begin writing.
Combining Icechunk and Sharding¶
IceChunkBackend does not expose sharding. Instead, pass an Icechunk session's Zarr
store to AsyncZarrBackend and manage the transaction yourself:
import icechunk
repo = icechunk.Repository.open_or_create(
icechunk.local_filesystem_storage("forecast_repo")
)
session = repo.writable_session("main")
io = AsyncZarrBackend(
"unused", # Location comes from the store
parallel_coords=OrderedDict({"time": times, "lead_time": lead_times}),
shard_coords={"lead_time": 8},
store=session.store,
)
io.add_array(template)
for forecast in forecasts:
io.write(forecast)
io.close() # Finish all writes before committing the external session
session.commit("forecast run")
The same backend can be passed to an Earth2Studio workflow. Wait for close() or
flush() to complete before committing; otherwise in-flight writes may be excluded
from the snapshot. With this composition, the backend does not commit the external
session for you.
Cloud Object Storage¶
Pass store to route writes through a Zarr store rather than fs_factory.
URLs and obstore instances use an obstore
backed zarr.storage.ObjectStore, with native put and multipart-upload operations:
io = AsyncZarrBackend(
"unused",
parallel_coords=OrderedDict({"time": times, "lead_time": lead_times}),
store="s3://my-bucket/forecasts/run-001.zarr",
store_kwargs={"region": "us-east-1"},
)
Credentials are resolved from the environment. For explicit store configuration:
from obstore.store import S3Store
io = AsyncZarrBackend(
"unused",
parallel_coords=OrderedDict({"time": times, "lead_time": lead_times}),
store=S3Store("my-bucket", prefix="forecasts/run-001.zarr"),
)
An already constructed Zarr store is also accepted. When store is set, file_name
and fs_factory are ignored. Non-blocking writes, sharding, and restart behavior
remain the same, with the loop pool sharing one store instance. Close the backend
after writing to finish outstanding uploads.