E2S-03: Earth2Studio Time Statistics¶
Purpose¶
Declare temporal quantities in variable labels, and have data sources produce
them. Native data sources own temporal aggregation; fetch_data forwards supported
labels and preserves their metadata. Sampling and reduction belong to the source.
Contract¶
Declarations and Ownership¶
Qualified variable labels identify temporal quantities unambiguously:
The first : separates the source name from its modifier; base names cannot
contain :. Raw backend identifiers containing :: are passed through unchanged,
not parsed as statistic labels. Unqualified labels retain their source-defined
meaning (including native accumulation periods). Qualified labels are canonical
for model and output coordinates. Coordinate constructors derive statistics
metadata from these labels; there is no separate statistics= declaration.
Data sources own aggregation: they accept qualified labels for the quantities they
support and return them reduced (see Source Responsibilities). time_statistics
owns modifier normalization, window planning, metadata, and block reductions.
earth2studio.data.utils provides shared request planning and window reduction.
Windows¶
For valid time T, intervals are half-open and right-closed: a window ending
at T includes the sample stamped T and excludes the one stamped at its start.
| Modifier | Selected interval |
|---|---|
mean:24h |
(T - 24h, T] |
mean:-12h:+12h |
(T - 12h, T + 12h] |
At source cadence Δ, a window (T + start, T + end] reads the stamps
T + start + Δ, …, T + end. The earth2studio_statistics metadata records this as
closed: "right".
Built-in methods are mean, sum, min, and max, which propagate missing
samples (see Planning and Reduction). Source cadence must divide
the window exactly. Durations are integers with units ns, us, ms,
s/sec/second, m/min/minute, h/hr/hour, d/day, or w/wk/week;
plural names are accepted. Weeks are seven days. Month and year durations are
unsupported: specify an explicit number of days or hours instead (for example,
30days). M and Y are rejected rather than interpreted as minutes or fixed
calendar approximations; use lowercase m for minutes. NumPy month/year timedeltas
are also rejected as source cadences.
Accumulations and Means of Instantaneous Fields¶
Accumulations and mean rates are stamped at the end of their period: hourly
ARCO-ERA5 tp, ttr, ro and sf and HRRR forecast tp hold (t - 1h, t] at
stamp t, and so do 6-hourly model outputs such as tp06. Right-closed windows
match this, so a two-part label covers the period ending at T at any cadence:
Label at T |
Hourly stamps read | Physical period |
|---|---|---|
tp:sum:6h |
T-5h … T |
(T - 6h, T] |
tp06 (ARCO alias) |
T-5h … T |
(T - 6h, T] |
On 6-hourly tp06 frames, tp:sum:24h reads T-18h … T, the 24 hours ending at
T. For instantaneous fields a window holds the samples after its start up to and
including its end, so t2m:max:24h at T reads T-23h … T hourly. Use explicit
offsets when a different set of samples is wanted, as ORBIT-2 does for its daily
extremes (t2m:max:-25h:-1h reads T-24h … T-1h).
Running buckets that reset periodically (GFS tp) cannot be summed: the buckets
overlap, so tp:sum:Nh double counts; use tp06. Each affected source states this
in its Temporal statistics: note.
Calendar-Day Means: FuXi-S2S¶
FuXi-S2S labels days by starting midnight, consumes two consecutive daily means, and predicts the next. All 76 channels declare means with these hourly windows:
| Quantity | Qualified label | Samples relative to day start |
|---|---|---|
| Ordinary fields | t2m:mean:-1h:23h |
00–23 UTC |
| Hourly precipitation accumulations | tp:mean:0h:24h |
01 UTC through next 00 UTC |
| Hourly thermal radiation accumulations | ttr:mean:0h:24h |
01 UTC through next 00 UTC |
tp and ttr cover the day itself, (T, T + 24h], because their timestamps mark
interval ends: 24 hourly samples from 01 UTC through next-day 00 UTC. Ordinary
fields take the instantaneous samples 00–23 UTC, (T - 1h, T + 23h].
These means retain one-hour accumulation units; multiply predicted tp by 24 for
a daily total. Channel order and model-unit conversions are unchanged.
Preserve start-of-day timestamps and one-day lead increments. Substituting
mean:24h would select the preceding day. The source must sample hourly; a source
storing these daily means returns them under the same labels without reducing again.
Statistics of Model Outputs¶
Pipeline can reduce a stream over lead time with LeadWindow (see
E2S-11).
The window ending at lead e is the right-closed (e - window, e], so its
output at valid time T is the qualified label's statistic ending at T.
compose_time_statistic(label, method, window, delta_t) gives that label for
frames published every delta_t:
| Frame label | Reduction | Result |
|---|---|---|
t2m (instantaneous) |
mean over 7d |
t2m:mean:168h |
t2m:mean:-1h:23h (daily mean) |
mean over 30days |
t2m:mean:-697h:+23h |
tp06:sum:6h |
sum over 24h |
tp06:sum:24h |
The reduction's method is mean, sum, min or max; other statistics need
not compose (a range of daily ranges is not a two-day range). A qualified frame
composes only when its method matches the reduction's and its window equals the
cadence: the frames then tile the window back to back with equal weight. Other
combinations (a mean of sums, overlapping or gapped windows, raw ::
identifiers) are not one statistic of the underlying samples, so the stream
cannot be reduced. Composed labels use the canonical modifier of
time_statistic_metadata, so a window given as 7d is labelled 168h.
Source Responsibilities¶
A source that supports a temporal quantity accepts its qualified label in
variable, like any other variable, and:
- returns it under the requested spelling, with its entry recorded in
earth2studio_statistics(see Metadata and Scope); - anchors windows to valid time for analysis sources, and to each lead time within one initialization for forecast sources, never reducing across initializations;
- reduces on its native grid before returning, so wrappers and
fetch_dataregrid one reduced field rather than every sample; - raises, before fetching, for a modifier it does not support, for duplicate
normalized quantities such as
t2m:mean:24handt2m:mean:1day, and for a window its cadence cannot sample exactly or whose required coordinates fall outside the source's availability; - documents when base fields are already accumulations: reducing stored values does not automatically deaccumulate them or integrate a rate over time.
How a quantity is produced is the source's choice. It may read a stored aggregate,
compose native accumulations as ARCO_ERA5 does for tp, or fetch instantaneous
samples at its cadence and reduce them. Sampling cadence is internal to the
source; no public time_step property is required.
Document native support in the source class docstring in a Note headed
Temporal statistics:: methods, label syntax, cadence, window closure, and
accumulation caveats. Capability metadata belongs on the source's lexicon as
TIME_STATISTICS, a tuple of supported method names. Sources expose their lexicon
through lexicon; they do not expose time_statistics or time_step attributes.
An absent lexicon declaration means no native support. fetch_data discovers
support through the lexicon (and through wrapped/component sources) and rejects
unsupported methods before fetching.
Planning and Reduction¶
Sources that reduce instantaneous samples use the shared primitives:
time_step = np.timedelta64(6, "h")
times = source_times("mean:24h", valid_time, time_step)
leads = source_lead_times("mean:24h", lead_time, time_step)
result = apply_time_statistic(block, "mean:24h", valid_time, time_step)
Planning accepts scalar or array targets. Forecast sources retain initialization time and plan lead times. Planners can produce negative leads; a source must reject unavailable leads before downloading, rather than switch to an earlier forecast.
apply_time_statistic reduces a variable block for one scalar target, excluding
values outside the window and rejecting missing or duplicate required coordinates.
It selects lead_time when present, otherwise time; callers may override the
dimension. Xarray preserves NumPy, CuPy, or Dask execution without implicit transfers.
Native source request pipeline¶
Two functions in earth2studio.data.utils support the source's own request model:
plain, stats, attrs = plan_statistics(variable, target_times, delta_t, supported, validate=...)splits the requested labels and plans the samples.supportedis the lexicon'sTIME_STATISTICS.target_timesare the output valid times (or the lead times of one initialization for forecast sources) anddelta_tthe source cadence.plainis[(slot, name)]of unqualified labels (including raw::identifiers), which the source fetches with its ordinary path.statsis a list ofStatisticPlandataclasses withslot,base,modifier, andsamplesfields.samplescontains the coordinates to read, shaped(target, sample)(source_times/source_lead_times). Malformed labels, unsupported methods, empty base names and duplicate normalized quantities raiseValueError.attrsis the output'searth2studio_statisticsmetadata.validateis called once with the sorted unique sample coordinates (datetime, ortimedeltafor lead time targets) before anything is read, so the source rejects unavailable samples with its own_validate_time/_validate_leadtime.await reduce_statistics(out, stats, fetch)withoutthe(target, variable, ...)output view (per initialization for forecasts) andfetch(coord, base)a source coroutine returning one field with the lexicon conversion applied. For each target and statistic variable, the fields are fetched concurrently into a transient array(sample, ...), reduced through the registered statistic (reduce_samples, an xarray reduction over the sample dimension) and written toout. The stack is then released. Finally the source setsarray.attrs.update(attrs).
flowchart TD
A["source.fetch"] --> B["plan_statistics"]
B --> C["plain variables → source's ordinary request path"]
B --> D["statistics: samples per target"]
D --> E["source._validate_time / _validate_leadtime"]
E --> F["reduce_statistics → source fetch(coord, base)"]
F --> G["transient stack → reduce_samples (registered statistic)"]
G --> H["out[target, slot]"]
C --> H
H --> I["array.attrs.update(attrs)"]
ARCO and WB2 read fields through fetch_array; GFS and HRRR through an async
_fetch_sample that resolves the index record (GFS also rebuilds tp06 from
accumulation buckets) and calls fetch_array. Plain variables keep each source's
existing request path (for example ARCO's shared chunk reads).
Memory is bounded by the output plus at most four transient stacks of one window
each and async_workers (default 4) in-flight fields; a stack is released as soon as
it is reduced. Coordinate plans grow with the number of requested samples. Overlapping
windows of different targets re-read their common samples; this trades repeated
reads (usually served by the source's disk cache) for the simplicity of an independent
reduction per target. The jobs run through a fixed worker pool and a task group:
a failure, timeout or caller cancellation cancels and awaits outstanding fetches, and
no partial window is ever written.
Missing samples propagate: if any sample in a window is NaN, mean, sum, min
and max return NaN for that cell, so a field built from a missing value is never
silently partial. This covers cells masked by the source (such as SST over land) and
records a source could not resolve (for example a GRIB record absent from its index,
which sources return as an all-NaN field with a warning). Skipping missing samples
is a data-handling choice, not part of a quantity's label, so it is not expressed as
a statistic method. The reduction does not use a cadence, window or expected count; the
source plans the coordinates. Means are sample-weighted, not duration-weighted.
Sources propagate fetching failures rather than reduce a partial window.
Implemented source capabilities¶
All sources below declare ("mean", "sum", "min", "max") on their lexicon.
| Source | Sampling | Window axis |
|---|---|---|
ARCO_ERA5 |
1 hour | Valid time |
WB2ERA5, WB2ERA5_121x240, WB2ERA5_32x64 |
6 hours | Valid time |
GFS |
6 hours | Valid time |
GFS_FX |
1 hour | Lead within initialization |
HRRR |
1 hour | Valid time |
HRRR_FX |
1 hour | Lead within initialization |
Source-specific considerations:
- ARCO derived overlapping accumulation periods can double-count totals.
- WB2 overlapping
tp12/tp24samples double-count rainfall. - GFS raw
tpuses resetting buckets; analysistp06uses the preceding cycle's complete six-hour bucket. Forecasttp06requires leads ≥6h; overlapping totals must not be interpreted as disjoint rainfall. - HRRR reductions operate on source-defined stored values.
- All forecast sources require expanded leads to be available for each initialization.
WB2Climatology does not declare native statistics. Hourly planning does not
interpolate missing forecast leads; source availability validation remains binding.
Custom block reductions share the window syntax and must preserve non-time
dimensions, including variable:
Native sources reduce through the registry, so a registered statistic works once the
source lists its name in the lexicon's TIME_STATISTICS. Standalone
apply_time_statistic also accepts registered statistics.
Metadata and Scope¶
time_statistic_metadata() returns normalized modifier, method, window, offsets,
and interval closure. Durations remain np.timedelta64 until serialization.
Native results record qualified labels in earth2studio_statistics:
Labels carry quantity identity through handshakes; attributes support lookup and
serialization. Record CF-compatible time bounds and cell_methods when supported.
Irregular-cadence inference, cross-variable diagnostics, and fused GPU kernels are out of scope. The native helpers operate on NumPy fields.
Verification¶
Keep one mocked statistics test per source (extend its existing call mock when
present). Mock backend I/O, retaining real parsing, planning, validation, and
aggregation. Cover all four reductions, plain/statistic mixtures, metadata,
overlapping-window fetch sharing, request order, NaNs, and invalid windows.
Forecast cases also cover distinct initializations and unavailable/negative leads.
Shared fixtures live in test/data/conftest.py; block-window tests live in
test/utils/test_time_statistics.py.