Creating a Local Data Source¶

Create and save an offline dataset to use in an inference pipeline.

This example demonstrates how to:

Build a small offline dataset by fetching data and writing to a Zarr store
Load the local store as a data source for an inference pipeline with the Microsoft Aurora model
Run the deterministic workflow and plot results

Set Up¶

For this example, the following are needed:

Prognostic Model: Use the built-in Aurora 6-hour model earth2studio.models.px.Aurora.
Data source: Pull data from the WeatherBench2 data API earth2studio.data.WB2ERA5.

import os

os.makedirs("outputs", exist_ok=True)
from dotenv import load_dotenv

load_dotenv()

from earth2studio.data import WB2ERA5, fetch_data
from earth2studio.models.px import Aurora

# Load the default model package which downloads the checkpoint from GCP
package = Aurora.load_default_package()
model = Aurora.load_model(package)

# Create the data source, cache is false
wb2 = WB2ERA5(cache=False, verbose=False)

Console output328 lines

Downloading aurora-0.25-static-z.npy: 0%|          | 0.00/3.96M [00:00<?, ?B/s]
Downloading aurora-0.25-static-z.npy: 100%|██████████| 3.96M/3.96M [00:00<00:00, 5.77MB/s]
Downloading aurora-0.25-static-z.npy: 100%|██████████| 3.96M/3.96M [00:00<00:00, 5.73MB/s]

Downloading aurora-0.25-static-slt.npy: 0%|          | 0.00/3.96M [00:00<?, ?B/s]
Downloading aurora-0.25-static-slt.npy: 100%|██████████| 3.96M/3.96M [00:00<00:00, 8.40MB/s]
Downloading aurora-0.25-static-slt.npy: 100%|██████████| 3.96M/3.96M [00:00<00:00, 8.33MB/s]

Downloading aurora-0.25-static-lsm.npy: 0%|          | 0.00/3.96M [00:00<?, ?B/s]
Downloading aurora-0.25-static-lsm.npy: 100%|██████████| 3.96M/3.96M [00:00<00:00, 7.77MB/s]
Downloading aurora-0.25-static-lsm.npy: 100%|██████████| 3.96M/3.96M [00:00<00:00, 7.71MB/s]

Downloading aurora-0.25-pretrained.ckpt: 0%|          | 0.00/4.68G [00:00<?, ?B/s]
Downloading aurora-0.25-pretrained.ckpt: 0%|          | 10.0M/4.68G [00:00<07:20, 11.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 0%|          | 20.0M/4.68G [00:01<03:32, 23.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 1%|          | 40.0M/4.68G [00:01<01:46, 47.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 1%|▏         | 60.0M/4.68G [00:01<01:15, 66.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 1%|▏         | 70.0M/4.68G [00:01<01:15, 65.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 2%|▏         | 90.0M/4.68G [00:01<00:59, 83.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 2%|▏         | 110M/4.68G [00:01<00:52, 93.6MB/s] 
Downloading aurora-0.25-pretrained.ckpt: 3%|▎         | 130M/4.68G [00:02<00:50, 97.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 3%|▎         | 150M/4.68G [00:02<00:50, 96.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 4%|▎         | 170M/4.68G [00:02<00:47, 101MB/s] 
Downloading aurora-0.25-pretrained.ckpt: 4%|▍         | 190M/4.68G [00:02<00:48, 101MB/s]
Downloading aurora-0.25-pretrained.ckpt: 4%|▍         | 200M/4.68G [00:02<00:57, 84.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 5%|▍         | 220M/4.68G [00:03<00:51, 93.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 5%|▌         | 240M/4.68G [00:03<00:48, 99.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 5%|▌         | 260M/4.68G [00:03<00:47, 99.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 6%|▌         | 280M/4.68G [00:03<00:46, 101MB/s] 
Downloading aurora-0.25-pretrained.ckpt: 6%|▋         | 300M/4.68G [00:03<00:43, 108MB/s]
Downloading aurora-0.25-pretrained.ckpt: 7%|▋         | 320M/4.68G [00:04<00:49, 95.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 7%|▋         | 340M/4.68G [00:04<00:47, 97.9MB/s]
Downloading aurora-0.25-pretrained.ckpt: 8%|▊         | 360M/4.68G [00:04<00:43, 106MB/s] 
Downloading aurora-0.25-pretrained.ckpt: 8%|▊         | 380M/4.68G [00:04<00:41, 111MB/s]
Downloading aurora-0.25-pretrained.ckpt: 8%|▊         | 400M/4.68G [00:04<00:43, 107MB/s]
Downloading aurora-0.25-pretrained.ckpt: 9%|▉         | 420M/4.68G [00:05<00:48, 94.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 9%|▉         | 440M/4.68G [00:05<00:46, 97.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 9%|▉         | 450M/4.68G [00:05<01:05, 69.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 10%|▉         | 460M/4.68G [00:05<01:12, 62.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 10%|▉         | 470M/4.68G [00:06<01:21, 55.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 10%|█         | 490M/4.68G [00:06<01:05, 69.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 11%|█         | 510M/4.68G [00:06<00:54, 81.9MB/s]
Downloading aurora-0.25-pretrained.ckpt: 11%|█         | 530M/4.68G [00:06<00:48, 91.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 11%|█▏        | 550M/4.68G [00:06<00:45, 98.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 12%|█▏        | 570M/4.68G [00:07<00:43, 103MB/s] 
Downloading aurora-0.25-pretrained.ckpt: 12%|█▏        | 590M/4.68G [00:07<00:43, 101MB/s]
Downloading aurora-0.25-pretrained.ckpt: 13%|█▎        | 600M/4.68G [00:07<00:46, 94.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 13%|█▎        | 620M/4.68G [00:07<00:50, 87.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 13%|█▎        | 640M/4.68G [00:07<00:49, 88.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 14%|█▍        | 660M/4.68G [00:08<00:53, 80.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 14%|█▍        | 670M/4.68G [00:08<00:51, 83.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 14%|█▍        | 680M/4.68G [00:08<00:51, 84.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 15%|█▍        | 700M/4.68G [00:08<00:50, 84.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 15%|█▍        | 710M/4.68G [00:08<00:51, 83.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 15%|█▌        | 720M/4.68G [00:09<00:55, 76.9MB/s]
Downloading aurora-0.25-pretrained.ckpt: 15%|█▌        | 730M/4.68G [00:09<01:05, 64.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 16%|█▌        | 750M/4.68G [00:09<00:56, 74.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 16%|█▌        | 770M/4.68G [00:09<00:53, 78.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 16%|█▋        | 790M/4.68G [00:10<01:01, 67.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 17%|█▋        | 810M/4.68G [00:10<00:52, 79.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 17%|█▋        | 830M/4.68G [00:10<00:46, 89.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 18%|█▊        | 850M/4.68G [00:10<00:41, 98.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 18%|█▊        | 870M/4.68G [00:10<00:40, 102MB/s] 
Downloading aurora-0.25-pretrained.ckpt: 19%|█▊        | 890M/4.68G [00:11<00:38, 106MB/s]
Downloading aurora-0.25-pretrained.ckpt: 19%|█▉        | 910M/4.68G [00:11<00:37, 109MB/s]
Downloading aurora-0.25-pretrained.ckpt: 19%|█▉        | 930M/4.68G [00:11<00:36, 111MB/s]
Downloading aurora-0.25-pretrained.ckpt: 20%|█▉        | 950M/4.68G [00:11<00:35, 114MB/s]
Downloading aurora-0.25-pretrained.ckpt: 20%|██        | 970M/4.68G [00:11<00:35, 111MB/s]
Downloading aurora-0.25-pretrained.ckpt: 21%|██        | 990M/4.68G [00:11<00:35, 111MB/s]
Downloading aurora-0.25-pretrained.ckpt: 21%|██        | 0.99G/4.68G [00:12<00:34, 116MB/s]
Downloading aurora-0.25-pretrained.ckpt: 21%|██▏       | 1.01G/4.68G [00:12<00:39, 98.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 22%|██▏       | 1.03G/4.68G [00:12<00:45, 86.9MB/s]
Downloading aurora-0.25-pretrained.ckpt: 22%|██▏       | 1.04G/4.68G [00:12<00:46, 84.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 23%|██▎       | 1.05G/4.68G [00:13<00:41, 94.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 23%|██▎       | 1.06G/4.68G [00:13<00:48, 80.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 23%|██▎       | 1.08G/4.68G [00:13<00:43, 88.9MB/s]
Downloading aurora-0.25-pretrained.ckpt: 24%|██▎       | 1.10G/4.68G [00:13<00:39, 98.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 24%|██▍       | 1.12G/4.68G [00:13<00:37, 102MB/s] 
Downloading aurora-0.25-pretrained.ckpt: 24%|██▍       | 1.14G/4.68G [00:13<00:35, 107MB/s]
Downloading aurora-0.25-pretrained.ckpt: 25%|██▍       | 1.16G/4.68G [00:14<00:33, 112MB/s]
Downloading aurora-0.25-pretrained.ckpt: 25%|██▌       | 1.18G/4.68G [00:14<00:32, 116MB/s]
Downloading aurora-0.25-pretrained.ckpt: 26%|██▌       | 1.20G/4.68G [00:14<00:35, 105MB/s]
Downloading aurora-0.25-pretrained.ckpt: 26%|██▌       | 1.22G/4.68G [00:14<00:34, 109MB/s]
Downloading aurora-0.25-pretrained.ckpt: 26%|██▋       | 1.24G/4.68G [00:15<00:47, 78.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 27%|██▋       | 1.26G/4.68G [00:15<00:42, 86.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 27%|██▋       | 1.28G/4.68G [00:15<00:38, 95.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 28%|██▊       | 1.30G/4.68G [00:15<00:38, 94.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 28%|██▊       | 1.32G/4.68G [00:16<00:51, 70.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 28%|██▊       | 1.33G/4.68G [00:16<00:49, 72.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 29%|██▊       | 1.34G/4.68G [00:16<00:49, 71.9MB/s]
Downloading aurora-0.25-pretrained.ckpt: 29%|██▉       | 1.35G/4.68G [00:16<00:53, 66.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 29%|██▉       | 1.36G/4.68G [00:16<00:49, 72.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 29%|██▉       | 1.38G/4.68G [00:17<00:43, 81.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 30%|██▉       | 1.40G/4.68G [00:17<00:40, 86.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 30%|███       | 1.42G/4.68G [00:17<00:37, 93.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 30%|███       | 1.43G/4.68G [00:17<00:38, 89.9MB/s]
Downloading aurora-0.25-pretrained.ckpt: 31%|███       | 1.45G/4.68G [00:17<00:38, 91.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 31%|███       | 1.46G/4.68G [00:18<00:50, 68.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 32%|███▏      | 1.47G/4.68G [00:18<00:42, 81.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 32%|███▏      | 1.49G/4.68G [00:18<00:38, 89.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 32%|███▏      | 1.50G/4.68G [00:18<00:39, 86.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 33%|███▎      | 1.52G/4.68G [00:18<00:35, 95.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 33%|███▎      | 1.54G/4.68G [00:18<00:32, 104MB/s] 
Downloading aurora-0.25-pretrained.ckpt: 33%|███▎      | 1.56G/4.68G [00:19<00:37, 89.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 34%|███▍      | 1.58G/4.68G [00:19<00:33, 98.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 34%|███▍      | 1.60G/4.68G [00:19<00:31, 104MB/s] 
Downloading aurora-0.25-pretrained.ckpt: 35%|███▍      | 1.62G/4.68G [00:19<00:31, 106MB/s]
Downloading aurora-0.25-pretrained.ckpt: 35%|███▌      | 1.64G/4.68G [00:19<00:32, 99.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 35%|███▌      | 1.66G/4.68G [00:20<00:31, 104MB/s] 
Downloading aurora-0.25-pretrained.ckpt: 36%|███▌      | 1.68G/4.68G [00:20<00:30, 107MB/s]
Downloading aurora-0.25-pretrained.ckpt: 36%|███▋      | 1.70G/4.68G [00:20<00:35, 91.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 37%|███▋      | 1.72G/4.68G [00:20<00:32, 97.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 37%|███▋      | 1.74G/4.68G [00:21<00:30, 103MB/s] 
Downloading aurora-0.25-pretrained.ckpt: 38%|███▊      | 1.76G/4.68G [00:21<00:41, 74.9MB/s]
Downloading aurora-0.25-pretrained.ckpt: 38%|███▊      | 1.77G/4.68G [00:21<00:45, 68.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 38%|███▊      | 1.78G/4.68G [00:21<00:48, 64.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 38%|███▊      | 1.79G/4.68G [00:22<00:47, 65.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 38%|███▊      | 1.80G/4.68G [00:22<00:43, 71.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 39%|███▊      | 1.81G/4.68G [00:22<00:39, 77.9MB/s]
Downloading aurora-0.25-pretrained.ckpt: 39%|███▉      | 1.82G/4.68G [00:22<00:37, 83.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 39%|███▉      | 1.83G/4.68G [00:22<00:34, 87.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 39%|███▉      | 1.84G/4.68G [00:22<00:34, 88.9MB/s]
Downloading aurora-0.25-pretrained.ckpt: 40%|███▉      | 1.86G/4.68G [00:22<00:32, 93.9MB/s]
Downloading aurora-0.25-pretrained.ckpt: 40%|████      | 1.88G/4.68G [00:22<00:30, 97.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 40%|████      | 1.89G/4.68G [00:23<00:29, 101MB/s] 
Downloading aurora-0.25-pretrained.ckpt: 41%|████      | 1.90G/4.68G [00:23<00:29, 99.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 41%|████      | 1.92G/4.68G [00:23<00:28, 102MB/s] 
Downloading aurora-0.25-pretrained.ckpt: 42%|████▏     | 1.94G/4.68G [00:23<00:28, 103MB/s]
Downloading aurora-0.25-pretrained.ckpt: 42%|████▏     | 1.96G/4.68G [00:23<00:28, 103MB/s]
Downloading aurora-0.25-pretrained.ckpt: 42%|████▏     | 1.97G/4.68G [00:23<00:28, 103MB/s]
Downloading aurora-0.25-pretrained.ckpt: 42%|████▏     | 1.98G/4.68G [00:24<00:28, 102MB/s]
Downloading aurora-0.25-pretrained.ckpt: 43%|████▎     | 1.99G/4.68G [00:24<00:33, 87.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 43%|████▎     | 2.00G/4.68G [00:24<00:37, 76.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 43%|████▎     | 2.02G/4.68G [00:24<00:33, 84.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 44%|████▎     | 2.04G/4.68G [00:24<00:30, 91.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 44%|████▍     | 2.06G/4.68G [00:25<00:29, 96.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 44%|████▍     | 2.07G/4.68G [00:25<00:32, 85.9MB/s]
Downloading aurora-0.25-pretrained.ckpt: 44%|████▍     | 2.08G/4.68G [00:25<00:32, 87.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 45%|████▍     | 2.09G/4.68G [00:25<00:30, 90.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 45%|████▍     | 2.10G/4.68G [00:25<00:29, 93.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 45%|████▌     | 2.12G/4.68G [00:25<00:27, 100MB/s] 
Downloading aurora-0.25-pretrained.ckpt: 45%|████▌     | 2.13G/4.68G [00:25<00:30, 89.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 46%|████▌     | 2.15G/4.68G [00:26<00:28, 94.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 46%|████▋     | 2.17G/4.68G [00:26<00:26, 100MB/s] 
Downloading aurora-0.25-pretrained.ckpt: 47%|████▋     | 2.19G/4.68G [00:26<00:26, 99.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 47%|████▋     | 2.20G/4.68G [00:26<00:26, 99.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 47%|████▋     | 2.21G/4.68G [00:26<00:31, 85.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 48%|████▊     | 2.23G/4.68G [00:27<00:46, 56.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 48%|████▊     | 2.25G/4.68G [00:27<00:38, 68.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 48%|████▊     | 2.26G/4.68G [00:27<00:37, 69.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 49%|████▊     | 2.28G/4.68G [00:27<00:35, 72.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 49%|████▉     | 2.29G/4.68G [00:28<00:33, 77.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 49%|████▉     | 2.29G/4.68G [00:28<00:31, 80.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 49%|████▉     | 2.31G/4.68G [00:28<00:36, 70.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 50%|████▉     | 2.33G/4.68G [00:28<00:31, 79.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 50%|█████     | 2.34G/4.68G [00:28<00:30, 82.9MB/s]
Downloading aurora-0.25-pretrained.ckpt: 50%|█████     | 2.35G/4.68G [00:28<00:29, 85.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 51%|█████     | 2.37G/4.68G [00:29<00:27, 90.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 51%|█████     | 2.39G/4.68G [00:29<00:26, 93.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 52%|█████▏    | 2.41G/4.68G [00:29<00:25, 95.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 52%|█████▏    | 2.43G/4.68G [00:29<00:31, 77.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 52%|█████▏    | 2.44G/4.68G [00:30<00:30, 78.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 52%|█████▏    | 2.45G/4.68G [00:30<00:29, 82.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 53%|█████▎    | 2.47G/4.68G [00:30<00:30, 77.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 53%|█████▎    | 2.49G/4.68G [00:30<00:27, 84.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 53%|█████▎    | 2.50G/4.68G [00:30<00:29, 78.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 54%|█████▎    | 2.51G/4.68G [00:30<00:28, 81.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 54%|█████▍    | 2.53G/4.68G [00:31<00:30, 74.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 54%|█████▍    | 2.54G/4.68G [00:31<00:30, 76.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 55%|█████▍    | 2.56G/4.68G [00:31<00:25, 89.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 55%|█████▍    | 2.57G/4.68G [00:31<00:25, 89.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 55%|█████▌    | 2.58G/4.68G [00:31<00:24, 90.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 56%|█████▌    | 2.60G/4.68G [00:32<00:24, 92.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 56%|█████▌    | 2.62G/4.68G [00:32<00:21, 102MB/s] 
Downloading aurora-0.25-pretrained.ckpt: 56%|█████▌    | 2.63G/4.68G [00:32<00:24, 89.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 57%|█████▋    | 2.65G/4.68G [00:32<00:26, 83.9MB/s]
Downloading aurora-0.25-pretrained.ckpt: 57%|█████▋    | 2.67G/4.68G [00:32<00:24, 88.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 57%|█████▋    | 2.68G/4.68G [00:33<00:28, 76.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 58%|█████▊    | 2.70G/4.68G [00:33<00:27, 77.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 58%|█████▊    | 2.71G/4.68G [00:33<00:26, 79.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 58%|█████▊    | 2.72G/4.68G [00:33<00:31, 66.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 59%|█████▊    | 2.74G/4.68G [00:34<00:26, 78.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 59%|█████▉    | 2.75G/4.68G [00:34<00:26, 77.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 59%|█████▉    | 2.77G/4.68G [00:34<00:23, 85.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 59%|█████▉    | 2.78G/4.68G [00:34<00:23, 85.9MB/s]
Downloading aurora-0.25-pretrained.ckpt: 60%|█████▉    | 2.80G/4.68G [00:34<00:21, 95.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 60%|██████    | 2.81G/4.68G [00:34<00:23, 86.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 60%|██████    | 2.82G/4.68G [00:35<00:27, 73.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 61%|██████    | 2.83G/4.68G [00:35<00:33, 60.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 61%|██████    | 2.85G/4.68G [00:35<00:26, 74.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 61%|██████▏   | 2.87G/4.68G [00:35<00:22, 86.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 62%|██████▏   | 2.88G/4.68G [00:35<00:22, 87.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 62%|██████▏   | 2.89G/4.68G [00:36<00:26, 71.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 62%|██████▏   | 2.91G/4.68G [00:36<00:24, 78.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 63%|██████▎   | 2.93G/4.68G [00:36<00:21, 89.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 63%|██████▎   | 2.94G/4.68G [00:36<00:23, 80.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 63%|██████▎   | 2.95G/4.68G [00:36<00:22, 81.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 63%|██████▎   | 2.96G/4.68G [00:36<00:24, 75.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 63%|██████▎   | 2.97G/4.68G [00:37<00:25, 72.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 64%|██████▍   | 2.99G/4.68G [00:37<00:21, 85.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 64%|██████▍   | 3.01G/4.68G [00:37<00:18, 94.9MB/s]
Downloading aurora-0.25-pretrained.ckpt: 64%|██████▍   | 3.02G/4.68G [00:37<00:21, 84.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 65%|██████▍   | 3.04G/4.68G [00:37<00:22, 79.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 65%|██████▌   | 3.06G/4.68G [00:38<00:19, 89.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 66%|██████▌   | 3.07G/4.68G [00:38<00:20, 83.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 66%|██████▌   | 3.08G/4.68G [00:38<00:25, 68.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 66%|██████▌   | 3.09G/4.68G [00:38<00:26, 63.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 66%|██████▌   | 3.10G/4.68G [00:38<00:29, 58.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 66%|██████▋   | 3.11G/4.68G [00:39<00:31, 54.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 67%|██████▋   | 3.12G/4.68G [00:39<00:31, 53.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 67%|██████▋   | 3.12G/4.68G [00:39<00:35, 46.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 67%|██████▋   | 3.13G/4.68G [00:39<00:32, 50.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 67%|██████▋   | 3.14G/4.68G [00:40<00:30, 53.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 68%|██████▊   | 3.16G/4.68G [00:40<00:23, 68.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 68%|██████▊   | 3.18G/4.68G [00:40<00:20, 79.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 68%|██████▊   | 3.19G/4.68G [00:40<00:19, 80.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 68%|██████▊   | 3.20G/4.68G [00:40<00:19, 80.9MB/s]
Downloading aurora-0.25-pretrained.ckpt: 69%|██████▊   | 3.21G/4.68G [00:40<00:19, 80.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 69%|██████▉   | 3.22G/4.68G [00:40<00:18, 83.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 69%|██████▉   | 3.23G/4.68G [00:41<00:17, 86.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 69%|██████▉   | 3.24G/4.68G [00:41<00:17, 88.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 69%|██████▉   | 3.25G/4.68G [00:41<00:16, 91.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 70%|██████▉   | 3.26G/4.68G [00:41<00:17, 87.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 70%|██████▉   | 3.27G/4.68G [00:41<00:17, 85.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 70%|███████   | 3.28G/4.68G [00:41<00:16, 88.9MB/s]
Downloading aurora-0.25-pretrained.ckpt: 70%|███████   | 3.29G/4.68G [00:41<00:22, 67.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 71%|███████   | 3.30G/4.68G [00:42<00:22, 67.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 71%|███████   | 3.31G/4.68G [00:42<00:20, 73.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 71%|███████   | 3.32G/4.68G [00:42<00:19, 76.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 71%|███████   | 3.33G/4.68G [00:42<00:17, 82.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 71%|███████▏  | 3.34G/4.68G [00:42<00:16, 84.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 72%|███████▏  | 3.35G/4.68G [00:42<00:16, 89.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 72%|███████▏  | 3.36G/4.68G [00:42<00:15, 89.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 72%|███████▏  | 3.38G/4.68G [00:42<00:14, 94.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 73%|███████▎  | 3.40G/4.68G [00:43<00:14, 94.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 73%|███████▎  | 3.41G/4.68G [00:43<00:19, 68.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 73%|███████▎  | 3.42G/4.68G [00:43<00:18, 71.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 73%|███████▎  | 3.43G/4.68G [00:43<00:17, 75.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 73%|███████▎  | 3.44G/4.68G [00:44<00:24, 53.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 74%|███████▍  | 3.46G/4.68G [00:44<00:20, 65.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 74%|███████▍  | 3.48G/4.68G [00:44<00:16, 76.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 75%|███████▍  | 3.50G/4.68G [00:44<00:15, 83.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 75%|███████▍  | 3.51G/4.68G [00:44<00:16, 75.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 75%|███████▌  | 3.53G/4.68G [00:45<00:14, 83.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 76%|███████▌  | 3.54G/4.68G [00:45<00:13, 89.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 76%|███████▌  | 3.56G/4.68G [00:45<00:13, 86.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 77%|███████▋  | 3.58G/4.68G [00:45<00:13, 85.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 77%|███████▋  | 3.59G/4.68G [00:45<00:13, 88.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 77%|███████▋  | 3.60G/4.68G [00:45<00:13, 88.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 77%|███████▋  | 3.62G/4.68G [00:46<00:13, 84.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 78%|███████▊  | 3.63G/4.68G [00:46<00:12, 87.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 78%|███████▊  | 3.64G/4.68G [00:46<00:13, 81.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 78%|███████▊  | 3.65G/4.68G [00:46<00:12, 86.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 78%|███████▊  | 3.66G/4.68G [00:46<00:12, 84.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 79%|███████▊  | 3.68G/4.68G [00:46<00:11, 94.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 79%|███████▉  | 3.69G/4.68G [00:47<00:11, 90.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 79%|███████▉  | 3.71G/4.68G [00:47<00:12, 83.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 79%|███████▉  | 3.72G/4.68G [00:47<00:18, 54.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 80%|███████▉  | 3.73G/4.68G [00:47<00:16, 61.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 80%|███████▉  | 3.74G/4.68G [00:48<00:18, 55.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 80%|████████  | 3.75G/4.68G [00:48<00:16, 62.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 80%|████████  | 3.76G/4.68G [00:48<00:14, 69.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 81%|████████  | 3.77G/4.68G [00:48<00:12, 75.9MB/s]
Downloading aurora-0.25-pretrained.ckpt: 81%|████████  | 3.79G/4.68G [00:48<00:13, 70.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 81%|████████  | 3.80G/4.68G [00:48<00:12, 75.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 81%|████████▏ | 3.81G/4.68G [00:48<00:11, 79.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 82%|████████▏ | 3.82G/4.68G [00:49<00:12, 76.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 82%|████████▏ | 3.83G/4.68G [00:49<00:18, 49.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 82%|████████▏ | 3.85G/4.68G [00:50<00:23, 37.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 83%|████████▎ | 3.87G/4.68G [00:50<00:17, 48.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 83%|████████▎ | 3.88G/4.68G [00:50<00:15, 54.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 83%|████████▎ | 3.90G/4.68G [00:50<00:12, 66.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 84%|████████▎ | 3.92G/4.68G [00:51<00:11, 71.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 84%|████████▍ | 3.94G/4.68G [00:51<00:09, 80.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 84%|████████▍ | 3.95G/4.68G [00:51<00:09, 83.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 85%|████████▍ | 3.96G/4.68G [00:51<00:08, 89.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 85%|████████▌ | 3.98G/4.68G [00:51<00:07, 93.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 86%|████████▌ | 4.00G/4.68G [00:52<00:09, 73.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 86%|████████▌ | 4.02G/4.68G [00:52<00:08, 83.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 86%|████████▋ | 4.04G/4.68G [00:52<00:07, 88.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 87%|████████▋ | 4.06G/4.68G [00:52<00:07, 94.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 87%|████████▋ | 4.08G/4.68G [00:52<00:06, 98.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 87%|████████▋ | 4.09G/4.68G [00:53<00:06, 99.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 88%|████████▊ | 4.11G/4.68G [00:53<00:07, 87.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 88%|████████▊ | 4.12G/4.68G [00:53<00:07, 85.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 88%|████████▊ | 4.13G/4.68G [00:53<00:07, 77.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 88%|████████▊ | 4.14G/4.68G [00:53<00:07, 75.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 89%|████████▊ | 4.15G/4.68G [00:54<00:08, 65.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 89%|████████▉ | 4.16G/4.68G [00:54<00:08, 66.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 89%|████████▉ | 4.18G/4.68G [00:54<00:06, 77.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 90%|████████▉ | 4.19G/4.68G [00:54<00:06, 77.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 90%|████████▉ | 4.21G/4.68G [00:54<00:05, 87.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 90%|█████████ | 4.22G/4.68G [00:54<00:05, 90.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 90%|█████████ | 4.23G/4.68G [00:54<00:05, 91.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 91%|█████████ | 4.25G/4.68G [00:55<00:04, 93.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 91%|█████████ | 4.26G/4.68G [00:55<00:05, 79.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 91%|█████████ | 4.27G/4.68G [00:55<00:06, 71.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 91%|█████████▏| 4.28G/4.68G [00:55<00:05, 77.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 92%|█████████▏| 4.29G/4.68G [00:55<00:05, 73.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 92%|█████████▏| 4.30G/4.68G [00:55<00:05, 72.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 92%|█████████▏| 4.32G/4.68G [00:56<00:05, 72.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 93%|█████████▎| 4.34G/4.68G [00:56<00:04, 80.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 93%|█████████▎| 4.35G/4.68G [00:56<00:04, 76.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 93%|█████████▎| 4.36G/4.68G [00:56<00:04, 71.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 93%|█████████▎| 4.37G/4.68G [00:57<00:05, 64.9MB/s]
Downloading aurora-0.25-pretrained.ckpt: 93%|█████████▎| 4.38G/4.68G [00:57<00:05, 61.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 94%|█████████▍| 4.39G/4.68G [00:57<00:04, 63.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 94%|█████████▍| 4.41G/4.68G [00:57<00:03, 76.5MB/s]
Downloading aurora-0.25-pretrained.ckpt: 95%|█████████▍| 4.42G/4.68G [00:57<00:03, 69.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 95%|█████████▍| 4.43G/4.68G [00:58<00:03, 74.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 95%|█████████▍| 4.44G/4.68G [00:58<00:03, 78.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 95%|█████████▌| 4.46G/4.68G [00:58<00:02, 87.9MB/s]
Downloading aurora-0.25-pretrained.ckpt: 96%|█████████▌| 4.47G/4.68G [00:58<00:03, 72.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 96%|█████████▌| 4.49G/4.68G [00:58<00:02, 82.2MB/s]
Downloading aurora-0.25-pretrained.ckpt: 96%|█████████▌| 4.50G/4.68G [00:58<00:02, 76.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 96%|█████████▋| 4.51G/4.68G [00:59<00:02, 81.8MB/s]
Downloading aurora-0.25-pretrained.ckpt: 97%|█████████▋| 4.52G/4.68G [00:59<00:01, 86.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 97%|█████████▋| 4.53G/4.68G [00:59<00:01, 89.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 97%|█████████▋| 4.54G/4.68G [00:59<00:01, 90.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 97%|█████████▋| 4.56G/4.68G [00:59<00:01, 95.4MB/s]
Downloading aurora-0.25-pretrained.ckpt: 98%|█████████▊| 4.58G/4.68G [00:59<00:01, 92.6MB/s]
Downloading aurora-0.25-pretrained.ckpt: 98%|█████████▊| 4.60G/4.68G [01:00<00:00, 88.3MB/s]
Downloading aurora-0.25-pretrained.ckpt: 99%|█████████▊| 4.62G/4.68G [01:00<00:00, 94.0MB/s]
Downloading aurora-0.25-pretrained.ckpt: 99%|█████████▉| 4.63G/4.68G [01:00<00:00, 77.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 99%|█████████▉| 4.65G/4.68G [01:00<00:00, 83.9MB/s]
Downloading aurora-0.25-pretrained.ckpt: 100%|█████████▉| 4.66G/4.68G [01:00<00:00, 86.1MB/s]
Downloading aurora-0.25-pretrained.ckpt: 100%|█████████▉| 4.68G/4.68G [01:01<00:00, 91.7MB/s]
Downloading aurora-0.25-pretrained.ckpt: 100%|██████████| 4.68G/4.68G [01:01<00:00, 82.3MB/s]

Creating a Local Zarr Store from a Datasource¶

Start with creating a local dataset from the WeatherBench2 data store. Since data sources return in-memory data arrays, there are a variety of ways this could be done. The following is a simple method using Earth2Studio IO objects to pack the requested data into a single Zarr store.

For this example, let's download some data for a Microsoft aurora forecast.

from collections import OrderedDict

import numpy as np

from earth2studio.io import ZarrBackend
from earth2studio.utils.coords import split_coords

times = np.array(
    [np.datetime64("2022-01-01T00:00:00"), np.datetime64("2022-01-01T06:00:00")]
)
variables = model.input_coords()["variable"]
zarr_path = "./outputs/19_wb2_dataset.zarr"
# Create Zarr store to pack data into
zb = ZarrBackend(file_name=zarr_path, backend_kwargs={"overwrite": True})
full_coords = OrderedDict(
    [
        ("time", np.atleast_1d(times)),
        ("lead_time", np.array([np.timedelta64(0, "h")])),
        ("lat", np.linspace(90, -90, 721)),
        ("lon", np.linspace(0, 359.75, 1440)),
    ]
)
zb.add_array(full_coords, array_name=list(variables))

# Loop over timestamps, fetch data and write slices into the pre-created arrays
for t in np.atleast_1d(times):
    x, coords = fetch_data(
        wb2,
        time=np.array([t]),
        variable=variables,
        lead_time=np.array([np.timedelta64(0, "h")]),
        device="cpu",
    )
    xs, reduced_coords, var_names = split_coords(x, coords, dim="variable")
    zb.write(xs, reduced_coords, array_name=list(var_names))

Note that the Zarr store we just created can be used for more than just Earth2Studio inference pipelines. Open it with zarr or xarray to explore/process what you just downloaded.

import zarr

zg = zarr.group(store=zarr.storage.LocalStore(zarr_path))
print(zg.tree())

Console output74 lines

/
├── lat (721,) float64
├── lead_time (1,) timedelta64[h]
├── lon (1440,) float64
├── msl (2, 1, 721, 1440) float32
├── q100 (2, 1, 721, 1440) float32
├── q1000 (2, 1, 721, 1440) float32
├── q150 (2, 1, 721, 1440) float32
├── q200 (2, 1, 721, 1440) float32
├── q250 (2, 1, 721, 1440) float32
├── q300 (2, 1, 721, 1440) float32
├── q400 (2, 1, 721, 1440) float32
├── q50 (2, 1, 721, 1440) float32
├── q500 (2, 1, 721, 1440) float32
├── q600 (2, 1, 721, 1440) float32
├── q700 (2, 1, 721, 1440) float32
├── q850 (2, 1, 721, 1440) float32
├── q925 (2, 1, 721, 1440) float32
├── t100 (2, 1, 721, 1440) float32
├── t1000 (2, 1, 721, 1440) float32
├── t150 (2, 1, 721, 1440) float32
├── t200 (2, 1, 721, 1440) float32
├── t250 (2, 1, 721, 1440) float32
├── t2m (2, 1, 721, 1440) float32
├── t300 (2, 1, 721, 1440) float32
├── t400 (2, 1, 721, 1440) float32
├── t50 (2, 1, 721, 1440) float32
├── t500 (2, 1, 721, 1440) float32
├── t600 (2, 1, 721, 1440) float32
├── t700 (2, 1, 721, 1440) float32
├── t850 (2, 1, 721, 1440) float32
├── t925 (2, 1, 721, 1440) float32
├── time (2,) datetime64[s]
├── u100 (2, 1, 721, 1440) float32
├── u1000 (2, 1, 721, 1440) float32
├── u10m (2, 1, 721, 1440) float32
├── u150 (2, 1, 721, 1440) float32
├── u200 (2, 1, 721, 1440) float32
├── u250 (2, 1, 721, 1440) float32
├── u300 (2, 1, 721, 1440) float32
├── u400 (2, 1, 721, 1440) float32
├── u50 (2, 1, 721, 1440) float32
├── u500 (2, 1, 721, 1440) float32
├── u600 (2, 1, 721, 1440) float32
├── u700 (2, 1, 721, 1440) float32
├── u850 (2, 1, 721, 1440) float32
├── u925 (2, 1, 721, 1440) float32
├── v100 (2, 1, 721, 1440) float32
├── v1000 (2, 1, 721, 1440) float32
├── v10m (2, 1, 721, 1440) float32
├── v150 (2, 1, 721, 1440) float32
├── v200 (2, 1, 721, 1440) float32
├── v250 (2, 1, 721, 1440) float32
├── v300 (2, 1, 721, 1440) float32
├── v400 (2, 1, 721, 1440) float32
├── v50 (2, 1, 721, 1440) float32
├── v500 (2, 1, 721, 1440) float32
├── v600 (2, 1, 721, 1440) float32
├── v700 (2, 1, 721, 1440) float32
├── v850 (2, 1, 721, 1440) float32
├── v925 (2, 1, 721, 1440) float32
├── z100 (2, 1, 721, 1440) float32
├── z1000 (2, 1, 721, 1440) float32
├── z150 (2, 1, 721, 1440) float32
├── z200 (2, 1, 721, 1440) float32
├── z250 (2, 1, 721, 1440) float32
├── z300 (2, 1, 721, 1440) float32
├── z400 (2, 1, 721, 1440) float32
├── z50 (2, 1, 721, 1440) float32
├── z500 (2, 1, 721, 1440) float32
├── z600 (2, 1, 721, 1440) float32
├── z700 (2, 1, 721, 1440) float32
├── z850 (2, 1, 721, 1440) float32
└── z925 (2, 1, 721, 1440) float32

Execute the Workflow¶

To use the saved dataset as a data source, we could create our own class that implements the interface required by [earth2studio.data.base.DataSource][], which needs just a __call__(time, variable) method.

However, since we used an IO backend from Earth2Studio we can use the [earth2studio.data.xr.InferenceOutputSource][] which is a convenience class that supports the output of inference pipelines.

import earth2studio.run as run
from earth2studio.data import InferenceOutputSource

offline_source = InferenceOutputSource(zarr_path)
out_zarr_path = "./outputs/19_pangu_output.zarr"
io = ZarrBackend(file_name=out_zarr_path, backend_kwargs={"overwrite": True})
io = run.deterministic(
    times[-1:],
    4,
    model,
    offline_source,
    io,
    output_coords=OrderedDict({"variable": np.array(["msl"])}),
)

Console output18 lines

2026-08-25 12:51:50.336 | INFO     | earth2studio.run:deterministic:85 - Running simple workflow!
2026-08-25 12:51:50.531 | INFO     | earth2studio.run:deterministic:92 - Inference device: cuda
2026-08-25 12:51:52.018 | SUCCESS  | earth2studio.run:deterministic:154 - Fetched data from InferenceOutputSource
2026-08-25 12:51:52.020 | INFO     | earth2studio.run:deterministic:162 - Inference starting!


Running inference:   0%|          | 0/5 [00:00<?, ?it/s]

Running inference:  40%|████      | 2/5 [00:01<00:01,  1.70it/s]

Running inference:  60%|██████    | 3/5 [00:02<00:01,  1.33it/s]

Running inference:  80%|████████  | 4/5 [00:03<00:00,  1.18it/s]

Running inference: 100%|██████████| 5/5 [00:04<00:00,  1.12it/s]
Running inference: 100%|██████████| 5/5 [00:04<00:00,  1.21it/s]
2026-08-25 12:51:56.168 | SUCCESS  | earth2studio.run:deterministic:189 - 
Inference complete

# Post Processing
# ---------------
# The last step is to post-process our results.

import cartopy.crs as ccrs
import matplotlib.pyplot as plt

plt.close("all")
projection = ccrs.Robinson()
fig, axes = plt.subplots(
    2,
    2,
    subplot_kw={"projection": projection},
    figsize=(12, 7),
    constrained_layout=True,
)
axes = axes.ravel()

lon = io["lon"][:]
lat = io["lat"][:]
lead_steps = [1, 2, 3, 4]  # 6h, 12h, 18h, 24h
for ax, step in zip(axes, lead_steps):
    im = ax.pcolormesh(
        lon,
        lat,
        io["msl"][0, step],
        transform=ccrs.PlateCarree(),
        cmap="PiYG",
    )
    ax.set_title(f"msl - Lead time: {6*step}h")
    ax.coastlines()
    ax.gridlines(draw_labels=False)

fig.colorbar(
    im, ax=axes, orientation="horizontal", fraction=0.05, pad=0.07, label="msl"
)
plt.savefig("outputs/19_msl_1day.png", dpi=150)

Output from Creating a Local Data Source

Download Python source Download Jupyter notebookexecuted in 228.8s

Execution profile

Runtime telemetry

Total runtime3m 49s

Execution environment

CPUAMD EPYC 9554 64-Core Processor

GPUNVIDIA H100 NVL · 93.6 GiB

System RAM58.4 GiB

PlatformLinux 6.8.0-137-generic

Python3.13.13

GPU driver / CUDADriver 595.84 · CUDA support 13.2