E2S-01: Earth2Studio Core Metadata and Coordinate Systems¶
Purpose¶
Define the metadata that gives field values and tabular records their meaning. Coordinate systems and frame schemas let components declare, compare, and plan inputs and outputs without allocating or reading the data itself.
Contract¶
This contract covers coordinate objects, axis names and types, shared attributes, and tabular schemas. Grids owns spatial geometry, CRS, topology, and grid identity. Time statistics owns temporal quantity modifiers, aggregation windows, and their derived metadata. Models defines model-specific coordinate matching and execution; Data fetching defines source requests.
Coordinate object¶
CoordinateSystem is a metadata-only coordinate signature with no field data.
It is constructed by coord_array() as an xarray.DataArray whose backing array
is a shape-and-dtype-only placeholder. The placeholder stores no field values,
reports zero field-data bytes, and rejects conversion to a NumPy array. Indexing
and transposing it update the declared shape without allocating field values.
Coordinate labels themselves are real arrays and do occupy memory. An execution
DataArray carries actual field values with the same metadata structure. The legacy
CoordSystem ordered mapping of axis names to NumPy arrays cannot express the full
signature and is not an equivalent object.
| ID | Requirement |
|---|---|
| C1 | A coordinate signature MUST expose ordered dimensions through .dims, lengths through .sizes, labels through .coords, the declared field dtype through .dtype, and shared metadata through .attrs. Planning MUST NOT read or materialize field values; signature .values is unsupported. |
| C2 | Dimension names MUST be unique. Dimension order MUST describe field axis order; consumers MUST NOT silently sort, rename, align, or transpose axes when checking a signature. |
| C3 | A dimension coordinate MUST be one-dimensional on its namesake dimension and match its size. Auxiliary coordinates MAY be scalar or span multiple declared dimensions; their shape MUST agree with those dimensions. An unlabelled dimension MUST still have a declared size. |
| C4 | Dynamic dimensions MUST be explicitly listed in earth2studio_dynamic_dims, form a leading prefix in dimension order, and have size zero in the signature. A fixed zero-sized dimension MUST NOT be interpreted as a wildcard. |
| C5 | Resolving dynamic dimensions MUST leave any remaining dynamic dimensions as a leading prefix. Concrete execution inputs MUST satisfy the consuming component's size and nonempty requirements. |
| C6 | Metadata operations MUST preserve unaffected coordinate values, order, attributes, name, and field dtype. Replacing an axis MUST remove or recompute dependent auxiliary coordinates and derived metadata so they cannot describe the old labels. |
Axis names and data types¶
Known names have the meanings below wherever they occur. These are coordinate
types, independent of the field dtype. coord_array() defaults the field dtype
to float32; this does not require every field or coordinate to be float32.
| Name | Coordinate dtype | Meaning |
|---|---|---|
time |
NumPy datetime64 |
Absolute time; initialization/reference time when paired with forecast leads, otherwise valid time. UTC semantics without a timezone-bearing object dtype |
lead_time |
NumPy timedelta64 |
Offset from reference time; may include input history. Negative offsets are permitted where the consuming contract allows history |
valid_time |
NumPy datetime64 |
Auxiliary absolute validity time; when derived from a forecast, equals time + lead_time with the corresponding coordinate dimensions |
variable |
String labels | Physical quantity identifier from the Earth2Studio vocabulary or a documented extension; temporal qualifiers follow the time-statistics contract |
ensemble |
Consistent declared dtype, commonly integers or strings | Member identity; labels identify members rather than their array positions |
batch |
Declared label dtype, when labelled | Generic collection of independent samples. May be unlabelled; when labelled, retain the declared label dtype and order |
lat, lon |
Floating-point | Geographic coordinates in degrees, as axes or auxiliaries; dimensionality, ordering, and longitude convention follow the grid contract |
y, x |
Numeric coordinates or integer indexes | Native spatial axes in the grid's native units, or integer indexes for indexed layouts |
hpx, face, height, width |
Integer indexes | HEALPix pixel or layout indexes; representation and ordering follow the grid contract |
| ID | Requirement |
|---|---|
| A1 | Known axis names MUST retain the meanings and dtype families above. Custom axes MAY be added with documented meaning, dtype, and ordering. Renamed spatial axes MUST remain identifiable through grid metadata. |
| A2 | Concrete temporal labels used for temporal validation MUST be explicit, nonempty, and free of NaT. Numeric offsets MUST NOT substitute for timedelta64. Precision conversion MUST preserve values and avoid overflow; nanosecond precision is not mandatory for all coordinate arrays. |
| A3 | variable MUST be a one-dimensional coordinate with unique, nonempty string labels. Labels MUST identify the quantity and its units consistently across producers and consumers. |
| A4 | Coordinate order MUST be preserved. Uniqueness or monotonicity beyond variable is imposed by the consuming contract, not assumed globally; repeated requested times, for example, may repeat labels. |
| A5 | A source's member argument selects the ensemble axis; it MUST NOT introduce a second standard axis named member. |
Standard attributes¶
Signature attributes live on the DataArray; descriptive attributes may also live on individual coordinates. A name or descriptive attribute does not replace coordinate labels or establish axis order.
| Attribute | Location | Type and role |
|---|---|---|
earth2studio_kind |
Signature | String "coordinate_array"; identifies a coordinate signature |
earth2studio_schema_version |
Signature | Integer 1; version of the signature metadata format |
earth2studio_dynamic_dims |
Signature | Ordered tuple of dimension names; empty when all dimensions are fixed |
units |
Coordinate or homogeneous field | String describing the units of its values |
source |
Field or DataFrame | Producer/provenance identifier string; distinct from a DataFrame's per-row source column |
| ID | Requirement |
|---|---|
| M1 | Signature construction MUST set kind, schema version, and dynamic dimensions consistently with the object. Concrete field outputs MUST omit these three signature-only attributes. |
| M2 | Descriptive attributes are optional. When present, they MUST describe the associated values accurately. A single field-level units MUST NOT claim one meaning for a mixed-variable field; variable-dependent descriptions belong on coordinates indexed by variable. |
| M3 | Metadata-preserving transformations MUST retain unrelated user attributes and coordinate attributes. Attributes invalidated by a transformation MUST be updated or removed. |
| M4 | Grid identity/CRS attributes and earth2studio_statistics MUST obey their owning specs. They MUST NOT be independently inferred from field shape or copied onto incompatible coordinates. |
DataFrame schema objects¶
A DataFrame represents rows of observations, forecasts, or analyses rather than
a dense Cartesian product of axes. Column names carry meaning; its row index is
not a replacement for time, location, or variable columns. CPU frames use
pandas; GPU-capable interfaces may use cuDF with the same logical schema.
Two declarations serve different roles:
FrameSchemais an ordered mapping from column names to one-dimensional NumPy arrays. An empty array declares a column's dtype with dynamic values; a nonempty array declares acceptable values, such as supported variable labels. The arrays are column constraints, not equal-length sample rows.- PyArrow
Schemadescribes source fields by name, type, nullability, and optional field metadata. A source exposes its schema asSCHEMA.E2STUDIO_SCHEMAsupplies the shared field vocabulary and suggested types.
| ID | Requirement |
|---|---|
| F1 | Schema column names MUST be unique. A FrameSchema MUST declare required columns and their dtype/value constraints; a consumer MUST validate those requirements before using the rows. An empty constraint array MUST NOT require an empty DataFrame. |
| F2 | A source schema MUST declare each supported field's type and nullability. Standard field names MUST retain the meanings below; sources SHOULD use the suggested types, documenting deviations. Additional source-specific fields MAY be declared. |
| F3 | Field selection MUST preserve requested column order. fields=None selects the full source schema; a name or list selects those fields; a PyArrow schema selects fields with matching source types. Unknown fields MUST raise KeyError; incompatible requested Arrow types MUST raise TypeError. |
| F4 | Returned columns MUST satisfy the selected source schema, including on an empty result. Null values are permitted only for nullable fields. A projected subset need not contain all core columns, but MUST be checked against a consuming FrameSchema before execution. |
| F5 | Field metadata MAY include description, units, and source_name; source_name describes the upstream name and MUST NOT replace the public column name. Source frames MUST identify the producer in frame.attrs["source"]. |
| F6 | Forecast frame schemas MUST declare lead_time as a duration when exposing forecast offsets, with time interpreted as reference time. If valid_time is exposed, it MUST equal time + lead_time. |
Standard DataFrame fields¶
These tables define a vocabulary, not a requirement that every frame contain
every column. The source schema and consuming FrameSchema determine presence.
The shared Arrow schema permits nulls in all listed fields; a source or consumer
may impose stricter constraints. Physical units of observation follow variable.
| Name | Suggested Arrow type | Meaning |
|---|---|---|
time |
timestamp[ns] |
Observation time, or forecast reference time; UTC |
lat |
float32 |
Latitude in degrees, within [-90, 90] |
lon |
float32 |
Longitude in degrees, within [0, 360] |
elev |
float32 |
Observation elevation above the surface, metres |
observation |
float32 |
Measurement or predicted value of the identified variable |
variable |
string |
Earth2Studio quantity identifier |
pres |
float32 |
Observation pressure level, Pa |
type, class |
string |
Observation type identifier and class |
source, station |
string |
Per-row source and station identifiers |
station_elev |
float32 |
Station elevation, metres |
quality |
uint16 |
Quality-control marker; 0 is best, 15 is missing |
satellite |
string |
Satellite platform identifier |
scan_angle |
float32 |
Satellite scan angle, degrees |
sensor_index |
uint16 |
Sensor channel index |
solza, solaza |
float32 |
Solar zenith and azimuth angles, degrees |
satellite_za, satellite_aza |
float32 |
Satellite zenith and azimuth angles, degrees |
wavenumber |
float64 |
Channel wavenumber, cm^-1 |
track_id, storm_name, basin |
string |
Track identifier, storm name, and basin code |
season |
int32 |
Tropical cyclone season year |
Forecast extensions to this shared vocabulary are:
| Name | Suggested Arrow type | Meaning |
|---|---|---|
lead_time |
duration[ns] |
Forecast offset from time |
valid_time |
timestamp[ns] |
Optional absolute forecast validity time |
Verification¶
- Inspect a signature's dimensions, sizes, labels, dtype, and attributes without accessing field values (C1); reject duplicate dimensions or mismatched sizes (C2–C3).
- Resolve a dynamic prefix and check that fixed empty axes remain fixed and a non-leading wildcard is rejected (C4–C5).
- Check temporal dtype families,
NaT, variable uniqueness, and preservation of label order, including repeated times (A1–A4). - Replace a variable or lead-time axis and verify dependent metadata is removed or recomputed while unrelated metadata survives (C6, M2–M4).
- Convert a signature to a concrete field and check signature-only attrs are absent (M1).
- Validate a frame against dynamic and value-constrained columns; select source fields in a different order and exercise unknown names, incompatible types, nullability, and empty results (F1–F4).
- Check source provenance and forecast validity-time consistency (F5–F6).