data.NNJAObsConv

Global

Import path: earth2studio.data.NNJAObsConv

View source on GitHub

Documentation

NNJA conventional (in-situ + GPS RO) observational data source. NOAA-NASA Joint Archive (NNJA) of Observations for Earth System Reanalysis is an archive ideal for developing observation-driven weather forecasting tools, as it includes a wide cross-section of data from a plethora of sensing platforms (satellites, surface stations, weather balloons, and more) and features data from 1979 to the present.

GPSRO rows use the shared columns with product-specific meanings: type is receiver SAID, station combines receiver/transmitter identifiers, quality is the QFRO flag table, pres is null, and elev is impact parameter minus Earth radius of curvature.

Parameters:

• source (('prepbufr', 'prepbufr.acft_profiles'), default: "prepbufr" ) –

  Which encoding family of the NNJA conventional archive to read, by default "prepbufr". These sources are different stages of the NCEP observation-processing pipeline, not independent replacement datasets:

  • "convbufr" points at raw dump streams grouped by family, such as aircft/aircar/adpupa/adpsfc. These files preserve source-native schemas and require family-specific decoding and QC before they resemble GSI-ready observations. The generic PrepBUFR decoder does not implement those raw family schemas, so this source raises NotImplementedError.
  • "prepbufr" points at the merged PrepBUFR cycle file. This is the preferred source for GSI-like conventional observations because upstream obsproc has already merged dump families, standardized many mnemonics, and attached report types / quality marks.
  • "prepbufr.acft_profiles" points at an aircraft-only PrepBUFR profile product. It groups aircraft points into flight-level, ascending, and descending profile report types that GSI remaps back to ordinary aircraft report types during processing.
• time_tolerance (TimeTolerance, default: timedelta64(0, 'm') ) –

  Time tolerance window for filtering observations. Accepts a single value (symmetric ± window) or a tuple (lower, upper) for asymmetric windows, by default np.timedelta64(0, 'm').

• cache (bool, default: True ) –

  Cache downloaded files in the local filesystem cache, by default True.

• verbose (bool, default: True ) –

  Show progress bars, by default True.

• async_timeout (int, default: 600 ) –

  Total timeout in seconds for the async fetch, by default 600.

• async_workers (int, default: 24 ) –

  Maximum number of concurrent async fetch tasks, by default 24.

• decode_workers (int, default: 8 ) –

  Number of parallel processes for BUFR message decoding. Higher values speed up decoding of large PrepBUFR files at the cost of more memory. Set to 1 to disable multiprocessing, by default 8.

• retries (int, default: 3 ) –

  Number of retry attempts per failed fetch task with exponential backoff, by default 3.

Warning

This is a remote data source and can potentially download a large amount of data to your local machine for large requests.

Note

Additional information on the data repository can be referenced here:

• www.brightband.com/data/nnja-ai/
• psl.noaa.gov/data/nnja_obs/
• registry.opendata.aws/noaa-reanalyses-pds/
• www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/document.htm

__call__

__call__(
    time: datetime | list[datetime] | TimeArray,
    variable: str | list[str] | VariableArray,
    fields: str | list[str] | Schema | None = None,
) -> DataFrame

Fetch observations for a set of timestamps.

fetch async

fetch(
    time: datetime | list[datetime] | TimeArray,
    variable: str | list[str] | VariableArray,
    fields: str | list[str] | Schema | None = None,
) -> DataFrame

Async function to get data.

available classmethod

available(time: datetime | datetime64) -> bool

Check if given date time is available.

Parameters:

• time (datetime | datetime64) –

  Date time to check

Returns:

• bool –

  If date time is available