Skip to content

data.NNJAObsSat

Global

Import path: earth2studio.data.NNJAObsSat

View source on GitHub

Documentation

NNJA historical NCEP aggregate satellite observations.

This source reads the NCEP satellite microwave (ATMS, MHS, AMSU-A, AMSU-B) and hyperspectral infrared sounder (AIRS, IASI, CrIS) BUFR products from the NNJA archive. It returns one long-format row per finite encoded channel value. sensor_index is the physical CHNM channel number, not a dense index into a selected channel list.

The IR sounders are returned in brightness temperature (K) regardless of how the archive stores them: AIRS is encoded as brightness temperature (TMBR) directly, while IASI (scaled integer radiance, SCRA) and CrIS (float radiance, SRAD) are converted via Planck inversion. The wavenumber column carries the channel centre wavenumber in cm⁻¹, computed from the instrument grids for IASI/CrIS and read from the per-channel LOGRCW field of the aggregate for AIRS. The aggregates carry the NCEP channel subsets (281 AIRS, 616 IASI, 431 CrIS) numbered on the instrument grids; cris routes to the full-spectral-resolution crisf4 product whose channel numbers live on the 2211-channel FSR grid. Archive coverage: AIRS 2002–2023 (NASA/Aqua route), IASI 2008–present, CrIS (FSR) 2018–present. All archived channels are returned by default; pass sensor_indices to narrow at decode time or subset downstream via the sensor_index column.

atms returns the encoded 22-channel TMBR scene brightness temperature. atms_antenna_temperature returns the corresponding encoded TMANT antenna temperature. No conversion is performed between those two ATMS products.

mhs (5 channels), amsua (15 channels), and amsub (5 channels) return their normal-feed TMBR values as encoded by NCEP. For these legacy products the mnemonic does not always mean the same physical antenna-correction state. GSI treats normal-feed TMBR as antenna temperature for AMSU-A, AMSU-B, and MHS, except for NOAA-15/16, which its reader treats as already converted upstream. NOAA satingest independently confirms that exception for AMSU-A. The platform identity is retained so a downstream transform can apply the appropriate convention explicitly.

scan_position is the one-based cross-track position: the encoded FOVN for the microwave sensors, AIRS (1-90), and IASI (1-120, a composite of 30 fields of regard x 4 detectors), and the encoded FORN (1-30) for CrIS, whose FOVN is the 1-9 detector index within the 3x3 field of regard and is not carried. scan_angle is the signed nominal instrument look angle derived from the FOV for the microwave sensors; it is always NaN for the IR sounders, whose scan geometry is sensor-specific — use satellite_za (the unsigned Earth-view zenith magnitude) for view-angle screening.

elev (surface elevation) is populated only for CrIS, whose template carries HOLS; the AIRS and IASI templates carry no surface-elevation field at all, so elev is null for them. (Their SELV field is the platform's orbital altitude, ~700-840 km, and must not be mistaken for surface elevation.) scan_line is the encoded SLNM, a granule-local counter that repeats within a cycle file; pair it with time to address a scan.

Parameters:

  • time_tolerance (TimeTolerance, default: timedelta64(10, 'm') ) –

    Time tolerance window for filtering observations. Accepts a single value (symmetric +/- window) or a tuple (lower, upper) for asymmetric windows, by default np.timedelta64(10, 'm').

  • satellites (list[str] | None, default: None ) –

    Satellite platforms to include. None includes every platform in the requested aggregate files.

  • cache (bool, default: True ) –

    Cache downloaded files in the local filesystem cache, by default True.

  • verbose (bool, default: True ) –

    Show progress bars, by default True.

  • async_timeout (int, default: 600 ) –

    Total timeout in seconds for the async fetch, by default 600.

  • async_workers (int, default: 8 ) –

    Maximum number of concurrent async fetch tasks, by default 8.

  • decode_workers (int, default: 8 ) –

    Number of parallel processes for BUFR message decoding. Set to 1 to disable multiprocessing, by default 8.

  • retries (int, default: 3 ) –

    Number of retry attempts per failed fetch task with exponential backoff, by default 3.

  • sensor_indices (dict[str, Sequence[int]] | None, default: None ) –

    Per-sensor channel-number subsets for the IR sounders, e.g. {"cris": [19, 24], "iasi": [16]}. Named after the sensor_indices selection of the JPSS/METOP sources and applied at decode time like theirs, but keyed per sensor since this source spans several instruments. Selections must be non-empty, unique integers on the instrument grid (AIRS 1-2378, IASI 1-8461, CrIS FSR 1-2211); a channel on the grid but absent from the archived NCEP subset matches no rows. Sensors absent from the dict (and all microwave sensors) return every archived channel. By default None (all channels).

Warning

Aggregate cycle files contain millions of footprints, and hyperspectral IR requests return one long-format row per (footprint, channel). At all channels a single 6-hour cycle is on the order of 10⁸-10⁹ rows per sensor (CrIS ~940 M across its three platforms, IASI ~400 M, AIRS ~210 M), which does not fit in memory on ordinary machines. All-channel requests are for narrow time windows and diagnostics; use sensor_indices to select a channel subset for anything larger — the narrowing is applied before rows are built, so a 32-channel CrIS request costs roughly 32/431 of the above. A finite archived value is not a QC decision: historical files may retain passive or degraded channels even when the aggregate carries no usable quality flag. Training pipelines should apply an explicit platform/channel validity policy.

Note

Additional information on the archive and microwave product semantics:

__call__ 

__call__(
    time: datetime | list[datetime] | TimeArray,
    variable: str | list[str] | VariableArray,
    fields: str | list[str] | Schema | None = None,
) -> DataFrame

Fetch observations for a set of timestamps.

fetch async 

fetch(
    time: datetime | list[datetime] | TimeArray,
    variable: str | list[str] | VariableArray,
    fields: str | list[str] | Schema | None = None,
) -> DataFrame

Async function to get data.

available classmethod 

available(time: datetime | datetime64) -> bool

Check if given date time is available.

Parameters:

Returns:

  • bool –

    If date time is available