Skip to content

Data and Metadata

Creating meaningful data products is a foundation of Earth2Studio. Components produce data, and rich metadata gives those outputs the context that humans and agents need to interpret and use them: what quantities they represent, where and when they apply, and how they were produced.

Earth2Studio uses coordinate systems and metadata structures both to describe existing data and to declare what each component needs and produces. These shared descriptions let people and agents inspect data products, prepare inputs, and connect components into pipelines.

Interface

Fields and Signatures

Gridded components exchange xarray.DataArray fields with values, named dimensions, coordinate labels, and attributes together. CPU fields typically use NumPy storage; CUDA fields use CuPy. Values cross component boundaries in physical units, with normalization and model-specific preprocessing handled inside the model wrapper.

A CoordinateSystem is a metadata-only DataArray constructed with coord_array. It describes a field without allocating its values; coordinate labels still occupy memory. Models return these signatures from input_coords(), forcing_coords(), and output_coords(...) so callers can plan inputs and output storage before inference.

Member What to inspect
.dims Names and order of the field axes.
.sizes Length of each axis.
.coords Dimension labels and auxiliary coordinates.
.dtype Field-value dtype, separate from coordinate dtypes.
.attrs Grid, temporal-statistics, and descriptive metadata.

Inspect signature metadata rather than .values, which is unsupported for these allocation-free objects. An execution field contains real values accessible through .data. The legacy CoordSystem mapping does not capture the full metadata interface.

Dimensions and Time

Coordinate dtypes describe labels independently of the field-value dtype.

Name Coordinate dtype Meaning
time NumPy datetime64 Absolute time; forecast reference time when paired with leads.
lead_time NumPy timedelta64 Offset from reference time, including input history.
valid_time NumPy datetime64 Auxiliary valid time; time + lead_time for forecasts.
variable String labels Physical quantity identifiers, including temporal qualifiers.
ensemble Declared dtype, often integer or string Member identity, not array position.
batch Declared label dtype, if labelled Independent samples; may be unlabelled.
lat, lon Floating-point Geographic coordinates in degrees, as axes or auxiliaries.
y, x Numeric or integer indexes Native spatial coordinates or grid indexes.
hpx, face, height, width Integer indexes HEALPix pixel or layout indexes.

Use NumPy datetime64 for absolute times and timedelta64 for offsets. For example, an initialization at 00 UTC with a six-hour lead is valid at 06 UTC. A history window with leads [-6h, 0h] describes two input frames relative to the reference time.

Dynamic dimensions are an explicitly declared, zero-sized leading prefix in a signature. A model can accept concrete leading axes such as time and ensemble through that prefix; its remaining dimensions still describe the fixed input layout. See Batch Dimension for batching patterns.

Time labels locate a field, while a temporal qualifier describes the quantity over a window. For example, tp:sum:6h identifies a six-hour accumulation ending at the target time. Coordinate constructors derive earth2studio_statistics metadata from qualified labels. See E2S-03: Time Statistics for window syntax, source support, and calendar-day semantics.

Spatial Grids

A GridDefinition describes geometry independently of field values or interpolation. Its dims and shape describe spatial axes, coords() supplies coordinates, and crs identifies the native coordinate reference system where applicable. topology, serializable attrs, and fingerprint() describe the grid family and identity. subset_indexers() supports spatial selection; cell_bounds() supplies cell geometry when available.

Definition Typical dimensions Geometry
LatLonGrid lat, lon Rectilinear geographic axes.
ProjectedGrid y, x Native projected coordinates with a CRS.
CurvilinearGrid y, x Two-dimensional latitude and longitude auxiliaries.
PointGrid x Individual geographic sample locations.
HEALPixGrid hpx or face, height, width Explicit pixel ordering and layout.

Pass a grid object or registered name to coord_array to attach its geometry. infer_grid(signature) recovers a target grid for fetching or regridding. Matching array shapes alone does not establish matching geography: axis order, CRS, and HEALPix ordering can differ.

Attributes and Tabular Schemas

Grid metadata includes geometry descriptions, earth2studio_crs, and, for registered grids, earth2studio_grid_id. Descriptive attributes can record units and provenance. For a mixed-variable field, describe units per variable rather than assigning one field-level unit to quantities with different meanings.

Sparse observations use pandas or cuDF DataFrames. A model's FrameSchema maps column names to NumPy arrays describing dtype and allowed values: an empty array leaves values unconstrained, while a nonempty array lists allowed values. These are column declarations, not example rows. A source's PyArrow SCHEMA instead describes available column types, nullability, and field metadata. See Tabular Observations for fetching them.

Standard DataFrame Fields

These fields form a shared vocabulary, not a requirement to include every column. The source's SCHEMA and the consuming model's FrameSchema determine which fields are available and required. Types below are the suggested PyArrow types from E2STUDIO_SCHEMA; check the source schema for its exact types and nullability. The units of observation depend on variable.

Name Suggested Arrow type Meaning
time timestamp[ns] Observation time or forecast reference time, in UTC.
lat float32 Latitude in degrees, within [-90, 90].
lon float32 Longitude in degrees, within [0, 360].
elev float32 Observation elevation above the surface, metres.
observation float32 Measurement or predicted value of the identified variable.
variable string Earth2Studio quantity identifier.
pres float32 Observation pressure level, Pa.
type, class string Observation type identifier and class.
source, station string Per-row source and station identifiers.
station_elev float32 Station elevation, metres.
quality uint16 Quality-control marker; 0 is best, 15 is missing.
satellite string Satellite platform identifier.
scan_angle float32 Satellite scan angle, degrees.
sensor_index uint16 Sensor channel index.
solza, solaza float32 Solar zenith and azimuth angles, degrees.
satellite_za, satellite_aza float32 Satellite zenith and azimuth angles, degrees.
wavenumber float64 Channel wavenumber, cm^-1.
track_id, storm_name, basin string Track identifier, storm name, and basin code.
season int32 Tropical cyclone season year.

Forecast frames can also declare lead_time as an Arrow duration and valid_time as a timestamp equal to time + lead_time. A DataFrame's row index does not replace its time, location, or variable columns.

Full contracts are defined in E2S-01: Core Metadata, E2S-02: Grids, and E2S-03: Time Statistics.

Usage

Declare and Inspect a Field

Create a small grid-backed signature with two history frames:

import numpy as np

from earth2studio.grids import LatLonGrid, infer_grid
from earth2studio.utils import coord_array

grid = LatLonGrid(
    np.array([50.0, 49.0, 48.0]),
    np.array([0.0, 1.0, 2.0, 3.0]),
)
signature = coord_array(
    ("batch", "lead_time", "variable", "lat", "lon"),
    {
        "lead_time": np.array([-6, 0], dtype="timedelta64[h]"),
        "variable": ["t2m"],
    },
    dynamic=("batch",),
    grid=grid,
)
print(signature.dims)
print(signature.sizes)
target_grid = infer_grid(signature)

For a loaded model, inspect model.input_coords() the same way. Pass the recovered grid to fetch_data(..., target_grid=target_grid) to prepare matching inputs; Data Sources shows the complete request.

A concrete field on this grid carries actual values:

import xarray as xr

field = xr.DataArray(
    np.full(grid.shape, 280.0, dtype=np.float32),
    dims=grid.dims,
    coords=grid.coords(),
    name="t2m",
    attrs={**grid.attrs, "units": "K", "source": "local-example"},
)

Preserve Meaning Across Transformations

Use coord_array_like to derive an output signature when retaining an input's geometry. For a new spatial domain, build a new grid-backed signature so the metadata describes the actual crop or transformed grid. After reordering spatial axes on registered-grid data, remove the stale earth2studio_grid_id so grid inference uses the actual coordinates; see the grid specification's current reconstruction guidance.

PyTorch wrappers convert at their computation boundary with field.e2s.to_torch() and restore labelled fields with from_torch(tensor, output_signature). Public component handoffs remain DataArrays. Signature-only markers are not field metadata; the conversion helpers handle that distinction.

Shared CUDA Memory

On Linux CUDA systems with RMM installed, the auto and cupy backends share memory pools between Torch, CuPy, and cuDF while preserving existing custom RMM resources. Import Earth2Studio or select the CuPy backend before initializing Torch CUDA. Set EARTH2STUDIO_DISABLE_RMM=1 before import to retain existing allocators. Non-Linux systems, missing RMM, CPU-only imports, and initial numpy or torch backends skip this setup.

RMM's Torch adapter does not honor Tensor.record_stream(): keep tensors alive until their streams finish, or disable RMM. Use RMM instrumentation for pool statistics; Torch's native cache controls do not manage these shared pools.

Describe Your Own Component

Start with your inference inputs and outputs: name the axes, variables and units, history and forecast times, and spatial grid. Declare these with coord_array and use them in the component's coordinate methods. Document any custom labels or axes so other components and agents can interpret them.

Reuse a built-in grid definition where possible. For a new geometry, implement the GridDefinition interface and optionally register it with register_grid for reuse. For tabular inputs, declare the needed columns and value constraints in a FrameSchema. Check your declarations against a representative field or observation batch before connecting the component to a pipeline.

The custom prognostic wrapper guide shows how to give these details to an agent along with your inference code and reference predictions.