Skip to content

NNJAObsSat

Global

Import path: earth2studio.data.NNJAObsSat

View source on GitHub

Documentation

NNJA historical NCEP aggregate satellite observations.

This source reads the NCEP satellite microwave (ATMS, MHS, AMSU-A, AMSU-B) and hyperspectral infrared sounder (AIRS, IASI, CrIS) BUFR products from the NNJA archive. It returns one long-format row per decoded channel, a missing value being a null observation. sensor_index is the physical CHNM channel number, not a dense index into a selected channel list.

The IR sounders are returned in brightness temperature (K) regardless of how the archive stores them: AIRS is encoded as brightness temperature (TMBR) directly, while IASI (scaled integer radiance, SCRA) and CrIS (float radiance, SRAD) are converted via Planck inversion. The wavenumber column carries the channel centre wavenumber in cm⁻¹, computed from the instrument grids for IASI/CrIS and looked up per channel for ATMS (SCCF) and AIRS (LOGRCW) from the values the aggregates encode; it is null for MHS, AMSU-A and AMSU-B. The aggregates carry the NCEP channel subsets (281 AIRS, 616 IASI, 431 CrIS) numbered on the instrument grids; cris routes to the full-spectral-resolution crisf4 product whose channel numbers live on the 2211-channel FSR grid; SAT_CHANNELS in earth2studio.data.utils_ncep lists them. Archive periods: AMSU-A and AMSU-B from 1998-10-26 (AMSU-B to 2014-06-06), MHS from 2005-02-01T12, ATMS from 2012-02-15, AIRS 2002-08-31 to 2023-01-18T06 (NASA/Aqua route), IASI from 2008-01-01, CrIS (FSR) from 2018-01-16T18. A cycle outside its sensor's period is skipped like a missing file, with one warning per sensor; a request with no cycle in the period returns an empty frame. available(time, sensor) checks the period. All archived channels are returned by default; pass sensor_indices to narrow at decode time or subset downstream via the sensor_index column.

atms returns the encoded 22-channel TMBR scene brightness temperature. atms_antenna_temperature returns the corresponding encoded TMANT antenna temperature. No conversion is performed between those two ATMS products.

mhs (5 channels), amsua (15 channels), and amsub (5 channels) return their normal-feed TMBR values as encoded by NCEP. For these legacy products the mnemonic does not always mean the same physical antenna-correction state. GSI treats normal-feed TMBR as antenna temperature for AMSU-A, AMSU-B, and MHS, except for NOAA-15/16, which its reader treats as already converted upstream. NOAA satingest independently confirms that exception for AMSU-A. The platform identity is retained so a downstream transform can apply the appropriate convention explicitly.

scan_position is the one-based cross-track position: the encoded FOVN for the microwave sensors, AIRS (1-90), and IASI (1-120, a composite of 30 fields of regard x 4 detectors), and the encoded FORN (1-30) for CrIS, whose FOVN, the 1-9 detector index within the 3x3 field of regard, is the detector column (null for the other sensors). scan_angle is the signed nominal instrument look angle derived from the FOV for the microwave sensors; it is always null for the IR sounders, whose scan geometry is sensor-specific — use satellite_za (the unsigned Earth-view zenith magnitude) for view-angle screening.

elev (surface elevation) is populated only for CrIS, whose template carries HOLS; the AIRS and IASI templates carry no surface-elevation field at all, so elev is null for them. (Their SELV field is the platform's orbital altitude, ~700-840 km, and must not be mistaken for surface elevation.) scan_line is the encoded SLNM, a granule-local counter that repeats within a cycle file; pair it with time to address a scan.

quality carries the per-channel provider flags. The footprint-level flags are scan_quality (ATMS 0-33-080, CrIS NSQF), granule_quality (ATMS 0-33-079) and footprint_quality (IASI QGFQ), each as encoded and null for sensors whose template lacks it.

Parameters:

  • time_tolerance (TimeTolerance, default: timedelta64(10, 'm') ) –

    Time tolerance window for filtering observations. Accepts a single value (symmetric +/- window) or a tuple (lower, upper) for asymmetric windows, by default np.timedelta64(10, 'm').

  • satellites (list[str] | None, default: None ) –

    Satellite platforms to include. None includes every platform in the requested aggregate files.

  • cache (bool, default: True ) –

    Cache downloaded files in the local filesystem cache, by default True.

  • verbose (bool, default: True ) –

    Show progress bars, by default True.

  • async_timeout (int, default: 600 ) –

    Timeout in seconds for downloading the files, by default 600. Decoding them is not bounded.

  • async_workers (int, default: 8 ) –

    Maximum number of concurrent async fetch tasks, by default 8.

  • decode_workers (int, default: 8 ) –

    Number of threads transforming decoded BUFR batches, by default 8. Decoding runs on earth2bufr's process-wide thread pool. Unless EARTH2BUFR_NUM_THREADS is set, the first fetch sets it to 8 threads, the fastest setting measured on 16-112 cores.

  • retries (int, default: 3 ) –

    Number of retry attempts per failed fetch task with exponential backoff, by default 3.

  • sensor_indices (dict[str, Sequence[int]] | None, default: None ) –

    Per-sensor channel-number subsets for the IR sounders, e.g. {"cris": [19, 24], "iasi": [16]}. Named after the sensor_indices selection of the JPSS/METOP sources and applied at decode time like theirs, but keyed per sensor since this source spans several instruments. Selections must be non-empty, unique channel numbers the sensor's archive carries (SAT_CHANNELS). Sensors absent from the dict (and all microwave sensors) return every archived channel. By default None (all channels).

  • layout (('long', 'wide'), default: "long" ) –

    "long" returns one row per (footprint, channel) with the columns of SCHEMA. "wide" returns one sensor per call, one row per footprint: the footprint columns of SCHEMA that sensor fills and its per-channel values as fixed-size lists over its channel axis (sensor_indices[sensor] ascending, else every archived channel), quality and one observation_<variable> per variable, null in a slot whose channel the footprint lacks. [channel_axis][earth2studio.data.utils_ncep.channel_axis] returns a frame's axis; the wavenumber follows from the channel number. Request several sensors concurrently, one call each, e.g. asyncio.gather(*(source.fetch(time, [s]) for s in sensors)); the calls share the source's decode slots. By default "long".

Warning

Aggregate cycle files contain millions of footprints, and hyperspectral IR requests return one long-format row per (footprint, channel). At all channels a single 6-hour cycle is on the order of 10⁸-10⁹ rows per sensor (CrIS ~940 M across its three platforms, IASI ~400 M, AIRS ~210 M), which does not fit in memory on ordinary machines. All-channel requests are for narrow time windows and diagnostics; use sensor_indices to select a channel subset for anything larger — the narrowing is applied before rows are built, so a 32-channel CrIS request costs roughly 32/431 of the above. A finite archived value is not a QC decision: historical files may retain passive or degraded channels even when the aggregate carries no usable quality flag. Training pipelines should apply an explicit platform/channel validity policy.

Note

Additional information on the archive and microwave product semantics:

__call__ 

__call__(
    time: datetime | list[datetime] | TimeArray,
    variable: str | list[str] | VariableArray,
    fields: str | list[str] | Schema | None = None,
) -> DataFrame

Fetch observations for a set of timestamps.

fetch async 

fetch(
    time: datetime | list[datetime] | TimeArray,
    variable: str | list[str] | VariableArray,
    fields: str | list[str] | Schema | None = None,
) -> DataFrame

Async function to get data.

available classmethod 

available(
    time: datetime | datetime64, sensor: str | None = None
) -> bool

Check if given date time is available, for sensor if given.

A time is available for a sensor when its cycle file (the nearest six-hourly cycle) lies in the sensor's archive period. Only the period is checked, not whether that file exists.

Parameters:

  • time (datetime | datetime64) –

    Date time to check

  • sensor (str | None, default: None ) –

    Sensor, e.g. "airs", by default None (any)

Returns:

  • bool –

    If date time is available

Raises:

  • ValueError –

    If sensor is not an NNJA satellite sensor

Examples using earth2studio.data.NNJAObsSat

HealDA v2 Global Data AssimilationProducing a 0.25 degree global analysis from the NNJA observing system.