Huge Ensembles (HENS) Checkpoints¶
Basic multi-checkpoint Huge Ensembles (HENS) inference workflow.
This example provides a basic example to load the Huge Ensemble checkpoints to perform ensemble inference. This notebook aims to demonstrate the foundations of running a multi-checkpoint workflow from Earth2Studio components. For more details about HENS, see:
Warning
We encourage users to familiarize themselves with the license restrictions of this model's checkpoints.
For the complete HENS workflow, we encourage users to have a look at the HENS recipe which provides a end-to-end solution to leverage HENS for downstream analysis such as tropical cyclone tracking:
In this example you will learn:
- How to load the HENS checkpoints with a custom model package
- How to load the HENS perturbation method
- How to create a simple ensemble inference loop
- How to visualize results
Set Up¶
First, import the necessary modules and set up our environment and load the required modules. HENS has checkpoints conveniently stored on HuggingFace that we will use. Rather than loading the default checkpoint from the original SFNO paper, create a model package that points to the specific HENS checkpoint we want to use instead.
This example also needs the following:
- Prognostic Base Model: Use SFNO model architecture
earth2studio.models.px.SFNO. - Datasource: Pull data from the GFS data api
earth2studio.data.GFS. - Perturbation Method: HENS uses a novel perturbation method
earth2studio.perturbation.HemisphericCentredBredVector. - Seeding Perturbation Method: Perturbation method to seed the Bred Vector
earth2studio.perturbation.CorrelatedSphericalGaussian.
import os
os.makedirs("outputs", exist_ok=True)
from dotenv import load_dotenv
load_dotenv() # TODO: make common example prep function
from earth2studio.data import GFS
from earth2studio.io import ZarrBackend
from earth2studio.models.auto import Package
from earth2studio.models.px import SFNO
from earth2studio.perturbation import (
CorrelatedSphericalGaussian,
HemisphericCentredBredVector,
)
from earth2studio.run import ensemble
# Set up two model packages for each checkpoint
# Note the modification of the cache location to avoid overwriting
model_package_1 = Package(
"hf://datasets/maheshankur10/hens/earth2mip_prod_registry/sfno_linear_74chq_sc2_layers8_edim620_wstgl2-epoch70_seed102",
cache_options={
"cache_storage": Package.default_cache("hens_1"),
"same_names": True,
},
)
model_package_2 = Package(
"hf://datasets/maheshankur10/hens/earth2mip_prod_registry/sfno_linear_74chq_sc2_layers8_edim620_wstgl2-epoch70_seed103",
cache_options={
"cache_storage": Package.default_cache("hens_2"),
"same_names": True,
},
)
# Create the data source
data = GFS()
Console output9 lines
/__w/earth2studio/earth2studio/.venv/lib/python3.13/site-packages/torch/cuda/__init__.py:64: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
PyTorch 2.10.0+cu128 with CUDA 1208 (you have 2.13.0+cu130)
Python 3.10.19 (you have 3.13.13)
Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
Memory-efficient attention, SwiGLU, sparse and more won't be available.
Set XFORMERS_MORE_DETAILS=1 for more details
CuPy distance computation test failed with error: cuVS >= 24.12 or pylibraft < 24.12 should be installed to use this featureExecute the Workflow¶
Next we execute the ensemble workflow for each model but loop through each checkpoint. Note that the models themselves have not been loaded into memory yet, this will be done one at a time to minimize the memory footprint of inference on a GPU. Before the ensemble workflow can get executed the following set up is needed:
- Initialize the SFNO model from checkpoint
- Initialize the perturbation method with the prognostic model
- Initialize the IO zarr store for this model
If multiple GPUs are being used, one could parallelize inference using different checkpoints on each card.
import gc
from datetime import datetime, timedelta
import numpy as np
import torch
start_date = datetime(2024, 1, 1)
nsteps = 4
nensemble = 2
for i, package in enumerate([model_package_1, model_package_2]):
# Load SFNO model from package
# HENS checkpoints use different inputs than default SFNO (inclusion of d2m)
# Can find this in the config.json, the load_model function in SFNO handles this
model = SFNO.load_model(package)
# Perturbation method
# Here we will simplify the process that's in the original paper for conciseness
noise_amplification = torch.zeros(model.input_coords()["variable"].shape[0])
index_z500 = list(model.input_coords()["variable"]).index("z500")
noise_amplification[index_z500] = 39.27 # z500 (0.35 * z500 skill)
noise_amplification = noise_amplification.reshape(1, 1, 1, -1, 1, 1)
seed_perturbation = CorrelatedSphericalGaussian(noise_amplitude=noise_amplification)
perturbation = HemisphericCentredBredVector(
model, data, seed_perturbation, noise_amplitude=noise_amplification
)
# IO object
io = ZarrBackend(
file_name=f"outputs/11_hens_{i}.zarr",
chunks={"ensemble": 1, "time": 1, "lead_time": 1},
backend_kwargs={"overwrite": True},
)
io = ensemble(
["2024-01-01"],
nsteps,
nensemble,
model,
data,
io,
perturbation,
batch_size=1,
output_coords={"variable": np.array(["u10m", "v10m"])},
)
print(io.root.tree())
# Do some manual clean up to free up VRAM
del model
del perturbation
gc.collect()
torch.cuda.empty_cache()
Console output1763 lines
Downloading config.json: 0%| | 0.00/22.4k [00:00<?, ?B/s]
Downloading config.json: 100%|██████████| 22.4k/22.4k [00:00<00:00, 95.8kB/s]
Downloading config.json: 100%|██████████| 22.4k/22.4k [00:00<00:00, 95.0kB/s]
Downloading global_means.npy: 0%| | 0.00/720 [00:00<?, ?B/s]
Downloading global_means.npy: 100%|██████████| 720/720 [00:00<00:00, 1.09kB/s]
Downloading global_means.npy: 100%|██████████| 720/720 [00:00<00:00, 1.09kB/s]
Downloading global_stds.npy: 0%| | 0.00/720 [00:00<?, ?B/s]
Downloading global_stds.npy: 100%|██████████| 720/720 [00:00<00:00, 1.77kB/s]
Downloading global_stds.npy: 100%|██████████| 720/720 [00:00<00:00, 1.76kB/s]
Downloading orography.nc: 0%| | 0.00/2.50M [00:00<?, ?B/s]
Downloading orography.nc: 100%|██████████| 2.50M/2.50M [00:00<00:00, 5.72MB/s]
Downloading orography.nc: 100%|██████████| 2.50M/2.50M [00:00<00:00, 5.69MB/s]
Downloading land_mask.nc: 0%| | 0.00/748k [00:00<?, ?B/s]
Downloading land_mask.nc: 100%|██████████| 748k/748k [00:00<00:00, 2.19MB/s]
Downloading land_mask.nc: 100%|██████████| 748k/748k [00:00<00:00, 2.18MB/s]
Downloading best_ckpt_mp0.tar: 0%| | 0.00/8.31G [00:00<?, ?B/s]
Downloading best_ckpt_mp0.tar: 0%| | 10.0M/8.31G [00:01<17:35, 8.44MB/s]
Downloading best_ckpt_mp0.tar: 0%| | 30.0M/8.31G [00:01<05:27, 27.1MB/s]
Downloading best_ckpt_mp0.tar: 1%| | 50.0M/8.31G [00:01<03:24, 43.4MB/s]
Downloading best_ckpt_mp0.tar: 1%| | 60.0M/8.31G [00:01<02:56, 50.3MB/s]
Downloading best_ckpt_mp0.tar: 1%| | 70.0M/8.31G [00:01<03:01, 48.9MB/s]
Downloading best_ckpt_mp0.tar: 1%| | 80.0M/8.31G [00:02<02:38, 55.6MB/s]
Downloading best_ckpt_mp0.tar: 1%| | 90.0M/8.31G [00:02<02:34, 57.3MB/s]
Downloading best_ckpt_mp0.tar: 1%| | 100M/8.31G [00:02<02:24, 61.1MB/s]
Downloading best_ckpt_mp0.tar: 1%|▏ | 110M/8.31G [00:02<02:13, 66.1MB/s]
Downloading best_ckpt_mp0.tar: 1%|▏ | 120M/8.31G [00:02<02:07, 69.0MB/s]
Downloading best_ckpt_mp0.tar: 2%|▏ | 130M/8.31G [00:02<02:24, 61.0MB/s]
Downloading best_ckpt_mp0.tar: 2%|▏ | 140M/8.31G [00:03<02:07, 68.6MB/s]
Downloading best_ckpt_mp0.tar: 2%|▏ | 150M/8.31G [00:03<02:10, 67.2MB/s]
Downloading best_ckpt_mp0.tar: 2%|▏ | 160M/8.31G [00:03<02:10, 67.3MB/s]
Downloading best_ckpt_mp0.tar: 2%|▏ | 170M/8.31G [00:03<02:07, 68.7MB/s]
Downloading best_ckpt_mp0.tar: 2%|▏ | 180M/8.31G [00:03<01:58, 73.5MB/s]
Downloading best_ckpt_mp0.tar: 2%|▏ | 190M/8.31G [00:03<02:04, 70.2MB/s]
Downloading best_ckpt_mp0.tar: 2%|▏ | 200M/8.31G [00:04<02:32, 57.1MB/s]
Downloading best_ckpt_mp0.tar: 2%|▏ | 210M/8.31G [00:04<02:15, 64.2MB/s]
Downloading best_ckpt_mp0.tar: 3%|▎ | 220M/8.31G [00:04<02:00, 72.2MB/s]
Downloading best_ckpt_mp0.tar: 3%|▎ | 230M/8.31G [00:04<01:54, 75.8MB/s]
Downloading best_ckpt_mp0.tar: 3%|▎ | 250M/8.31G [00:04<01:50, 78.5MB/s]
Downloading best_ckpt_mp0.tar: 3%|▎ | 260M/8.31G [00:04<02:08, 67.5MB/s]
Downloading best_ckpt_mp0.tar: 3%|▎ | 280M/8.31G [00:05<01:54, 75.4MB/s]
Downloading best_ckpt_mp0.tar: 3%|▎ | 290M/8.31G [00:05<01:48, 79.5MB/s]
Downloading best_ckpt_mp0.tar: 4%|▎ | 300M/8.31G [00:05<01:48, 79.0MB/s]
Downloading best_ckpt_mp0.tar: 4%|▎ | 310M/8.31G [00:05<01:41, 84.3MB/s]
Downloading best_ckpt_mp0.tar: 4%|▍ | 320M/8.31G [00:05<01:52, 76.1MB/s]
Downloading best_ckpt_mp0.tar: 4%|▍ | 340M/8.31G [00:05<01:57, 72.7MB/s]
Downloading best_ckpt_mp0.tar: 4%|▍ | 350M/8.31G [00:06<01:56, 73.5MB/s]
Downloading best_ckpt_mp0.tar: 4%|▍ | 360M/8.31G [00:06<02:03, 69.4MB/s]
Downloading best_ckpt_mp0.tar: 4%|▍ | 370M/8.31G [00:06<01:59, 71.4MB/s]
Downloading best_ckpt_mp0.tar: 4%|▍ | 380M/8.31G [00:06<01:59, 71.2MB/s]
Downloading best_ckpt_mp0.tar: 5%|▍ | 390M/8.31G [00:06<02:26, 58.3MB/s]
Downloading best_ckpt_mp0.tar: 5%|▍ | 410M/8.31G [00:07<02:06, 67.1MB/s]
Downloading best_ckpt_mp0.tar: 5%|▍ | 420M/8.31G [00:07<02:07, 66.4MB/s]
Downloading best_ckpt_mp0.tar: 5%|▌ | 430M/8.31G [00:07<01:59, 70.9MB/s]
Downloading best_ckpt_mp0.tar: 5%|▌ | 440M/8.31G [00:07<01:55, 73.1MB/s]
Downloading best_ckpt_mp0.tar: 5%|▌ | 450M/8.31G [00:07<02:18, 61.2MB/s]
Downloading best_ckpt_mp0.tar: 5%|▌ | 460M/8.31G [00:07<02:10, 64.5MB/s]
Downloading best_ckpt_mp0.tar: 6%|▌ | 470M/8.31G [00:07<02:05, 67.2MB/s]
Downloading best_ckpt_mp0.tar: 6%|▌ | 480M/8.31G [00:08<01:59, 70.2MB/s]
Downloading best_ckpt_mp0.tar: 6%|▌ | 490M/8.31G [00:08<01:56, 71.9MB/s]
Downloading best_ckpt_mp0.tar: 6%|▌ | 500M/8.31G [00:08<01:53, 74.2MB/s]
Downloading best_ckpt_mp0.tar: 6%|▌ | 520M/8.31G [00:08<01:59, 70.3MB/s]
Downloading best_ckpt_mp0.tar: 6%|▌ | 530M/8.31G [00:09<02:46, 50.2MB/s]
Downloading best_ckpt_mp0.tar: 6%|▋ | 540M/8.31G [00:09<02:24, 57.8MB/s]
Downloading best_ckpt_mp0.tar: 6%|▋ | 550M/8.31G [00:09<02:40, 51.9MB/s]
Downloading best_ckpt_mp0.tar: 7%|▋ | 570M/8.31G [00:09<02:03, 67.2MB/s]
Downloading best_ckpt_mp0.tar: 7%|▋ | 580M/8.31G [00:09<02:19, 59.6MB/s]
Downloading best_ckpt_mp0.tar: 7%|▋ | 590M/8.31G [00:09<02:06, 65.6MB/s]
Downloading best_ckpt_mp0.tar: 7%|▋ | 600M/8.31G [00:10<01:54, 72.3MB/s]
Downloading best_ckpt_mp0.tar: 7%|▋ | 610M/8.31G [00:10<01:48, 76.3MB/s]
Downloading best_ckpt_mp0.tar: 7%|▋ | 620M/8.31G [00:10<01:52, 73.8MB/s]
Downloading best_ckpt_mp0.tar: 7%|▋ | 630M/8.31G [00:10<02:11, 62.7MB/s]
Downloading best_ckpt_mp0.tar: 8%|▊ | 640M/8.31G [00:10<02:49, 48.6MB/s]
Downloading best_ckpt_mp0.tar: 8%|▊ | 650M/8.31G [00:11<02:35, 53.0MB/s]
Downloading best_ckpt_mp0.tar: 8%|▊ | 660M/8.31G [00:11<02:32, 54.0MB/s]
Downloading best_ckpt_mp0.tar: 8%|▊ | 670M/8.31G [00:11<02:26, 56.0MB/s]
Downloading best_ckpt_mp0.tar: 8%|▊ | 680M/8.31G [00:11<02:26, 56.1MB/s]
Downloading best_ckpt_mp0.tar: 8%|▊ | 700M/8.31G [00:11<02:05, 65.3MB/s]
Downloading best_ckpt_mp0.tar: 8%|▊ | 710M/8.31G [00:12<02:32, 53.6MB/s]
Downloading best_ckpt_mp0.tar: 8%|▊ | 720M/8.31G [00:12<02:32, 53.4MB/s]
Downloading best_ckpt_mp0.tar: 9%|▊ | 730M/8.31G [00:12<02:29, 54.6MB/s]
Downloading best_ckpt_mp0.tar: 9%|▊ | 740M/8.31G [00:12<02:32, 53.3MB/s]
Downloading best_ckpt_mp0.tar: 9%|▉ | 750M/8.31G [00:12<02:24, 56.1MB/s]
Downloading best_ckpt_mp0.tar: 9%|▉ | 760M/8.31G [00:13<02:21, 57.6MB/s]
Downloading best_ckpt_mp0.tar: 9%|▉ | 770M/8.31G [00:13<02:46, 48.7MB/s]
Downloading best_ckpt_mp0.tar: 9%|▉ | 780M/8.31G [00:13<02:39, 50.7MB/s]
Downloading best_ckpt_mp0.tar: 9%|▉ | 790M/8.31G [00:13<02:32, 53.0MB/s]
Downloading best_ckpt_mp0.tar: 9%|▉ | 800M/8.31G [00:13<02:23, 56.2MB/s]
Downloading best_ckpt_mp0.tar: 10%|▉ | 810M/8.31G [00:14<02:25, 55.4MB/s]
Downloading best_ckpt_mp0.tar: 10%|▉ | 820M/8.31G [00:14<02:16, 58.9MB/s]
Downloading best_ckpt_mp0.tar: 10%|▉ | 830M/8.31G [00:14<02:05, 64.0MB/s]
Downloading best_ckpt_mp0.tar: 10%|▉ | 840M/8.31G [00:14<02:33, 52.3MB/s]
Downloading best_ckpt_mp0.tar: 10%|▉ | 850M/8.31G [00:14<02:22, 56.4MB/s]
Downloading best_ckpt_mp0.tar: 10%|█ | 860M/8.31G [00:14<02:11, 61.1MB/s]
Downloading best_ckpt_mp0.tar: 10%|█ | 870M/8.31G [00:15<02:01, 66.1MB/s]
Downloading best_ckpt_mp0.tar: 10%|█ | 880M/8.31G [00:15<01:55, 69.5MB/s]
Downloading best_ckpt_mp0.tar: 10%|█ | 890M/8.31G [00:15<02:30, 53.0MB/s]
Downloading best_ckpt_mp0.tar: 11%|█ | 900M/8.31G [00:15<02:53, 46.0MB/s]
Downloading best_ckpt_mp0.tar: 11%|█ | 910M/8.31G [00:15<02:43, 48.8MB/s]
Downloading best_ckpt_mp0.tar: 11%|█ | 920M/8.31G [00:16<02:36, 50.8MB/s]
Downloading best_ckpt_mp0.tar: 11%|█ | 930M/8.31G [00:16<02:25, 54.5MB/s]
Downloading best_ckpt_mp0.tar: 11%|█ | 940M/8.31G [00:16<02:23, 55.4MB/s]
Downloading best_ckpt_mp0.tar: 11%|█ | 950M/8.31G [00:16<02:32, 52.1MB/s]
Downloading best_ckpt_mp0.tar: 11%|█▏ | 960M/8.31G [00:17<02:42, 48.8MB/s]
Downloading best_ckpt_mp0.tar: 11%|█▏ | 970M/8.31G [00:17<02:23, 55.0MB/s]
Downloading best_ckpt_mp0.tar: 12%|█▏ | 980M/8.31G [00:17<02:28, 53.2MB/s]
Downloading best_ckpt_mp0.tar: 12%|█▏ | 990M/8.31G [00:17<02:35, 50.7MB/s]
Downloading best_ckpt_mp0.tar: 12%|█▏ | 0.98G/8.31G [00:17<02:44, 48.0MB/s]
Downloading best_ckpt_mp0.tar: 12%|█▏ | 0.99G/8.31G [00:18<02:50, 46.3MB/s]
Downloading best_ckpt_mp0.tar: 12%|█▏ | 1.00G/8.31G [00:18<02:39, 49.3MB/s]
Downloading best_ckpt_mp0.tar: 12%|█▏ | 1.01G/8.31G [00:18<02:50, 45.9MB/s]
Downloading best_ckpt_mp0.tar: 12%|█▏ | 1.02G/8.31G [00:18<02:56, 44.4MB/s]
Downloading best_ckpt_mp0.tar: 12%|█▏ | 1.03G/8.31G [00:18<02:53, 45.0MB/s]
Downloading best_ckpt_mp0.tar: 12%|█▏ | 1.04G/8.31G [00:19<02:34, 50.6MB/s]
Downloading best_ckpt_mp0.tar: 13%|█▎ | 1.04G/8.31G [00:19<02:16, 57.0MB/s]
Downloading best_ckpt_mp0.tar: 13%|█▎ | 1.05G/8.31G [00:19<02:19, 55.7MB/s]
Downloading best_ckpt_mp0.tar: 13%|█▎ | 1.06G/8.31G [00:19<02:39, 48.8MB/s]
Downloading best_ckpt_mp0.tar: 13%|█▎ | 1.07G/8.31G [00:19<02:34, 50.4MB/s]
Downloading best_ckpt_mp0.tar: 13%|█▎ | 1.08G/8.31G [00:20<02:30, 51.4MB/s]
Downloading best_ckpt_mp0.tar: 13%|█▎ | 1.09G/8.31G [00:20<02:37, 49.0MB/s]
Downloading best_ckpt_mp0.tar: 13%|█▎ | 1.10G/8.31G [00:20<02:41, 47.9MB/s]
Downloading best_ckpt_mp0.tar: 13%|█▎ | 1.11G/8.31G [00:20<03:08, 41.1MB/s]
Downloading best_ckpt_mp0.tar: 14%|█▎ | 1.13G/8.31G [00:21<02:59, 42.9MB/s]
Downloading best_ckpt_mp0.tar: 14%|█▍ | 1.14G/8.31G [00:21<02:45, 46.5MB/s]
Downloading best_ckpt_mp0.tar: 14%|█▍ | 1.15G/8.31G [00:21<02:39, 48.1MB/s]
Downloading best_ckpt_mp0.tar: 14%|█▍ | 1.16G/8.31G [00:21<02:39, 48.1MB/s]
Downloading best_ckpt_mp0.tar: 14%|█▍ | 1.17G/8.31G [00:22<02:30, 51.0MB/s]
Downloading best_ckpt_mp0.tar: 14%|█▍ | 1.18G/8.31G [00:22<02:23, 53.3MB/s]
Downloading best_ckpt_mp0.tar: 14%|█▍ | 1.19G/8.31G [00:22<02:47, 45.5MB/s]
Downloading best_ckpt_mp0.tar: 14%|█▍ | 1.20G/8.31G [00:22<02:44, 46.4MB/s]
Downloading best_ckpt_mp0.tar: 15%|█▍ | 1.21G/8.31G [00:23<02:29, 51.0MB/s]
Downloading best_ckpt_mp0.tar: 15%|█▍ | 1.22G/8.31G [00:23<02:29, 50.8MB/s]
Downloading best_ckpt_mp0.tar: 15%|█▍ | 1.23G/8.31G [00:23<02:32, 49.7MB/s]
Downloading best_ckpt_mp0.tar: 15%|█▍ | 1.24G/8.31G [00:23<02:20, 53.8MB/s]
Downloading best_ckpt_mp0.tar: 15%|█▌ | 1.25G/8.31G [00:23<02:49, 44.7MB/s]
Downloading best_ckpt_mp0.tar: 15%|█▌ | 1.26G/8.31G [00:24<02:36, 48.4MB/s]
Downloading best_ckpt_mp0.tar: 15%|█▌ | 1.27G/8.31G [00:24<02:27, 51.3MB/s]
Downloading best_ckpt_mp0.tar: 15%|█▌ | 1.28G/8.31G [00:24<02:16, 55.2MB/s]
Downloading best_ckpt_mp0.tar: 16%|█▌ | 1.29G/8.31G [00:24<02:18, 54.6MB/s]
Downloading best_ckpt_mp0.tar: 16%|█▌ | 1.30G/8.31G [00:24<02:18, 54.5MB/s]
Downloading best_ckpt_mp0.tar: 16%|█▌ | 1.31G/8.31G [00:24<02:10, 57.7MB/s]
Downloading best_ckpt_mp0.tar: 16%|█▌ | 1.32G/8.31G [00:25<02:58, 42.1MB/s]
Downloading best_ckpt_mp0.tar: 16%|█▌ | 1.33G/8.31G [00:25<02:43, 45.9MB/s]
Downloading best_ckpt_mp0.tar: 16%|█▌ | 1.34G/8.31G [00:25<02:27, 50.7MB/s]
Downloading best_ckpt_mp0.tar: 16%|█▌ | 1.35G/8.31G [00:25<02:24, 51.6MB/s]
Downloading best_ckpt_mp0.tar: 16%|█▋ | 1.36G/8.31G [00:26<02:30, 49.5MB/s]
Downloading best_ckpt_mp0.tar: 16%|█▋ | 1.37G/8.31G [00:26<02:33, 48.6MB/s]
Downloading best_ckpt_mp0.tar: 17%|█▋ | 1.38G/8.31G [00:26<02:52, 43.1MB/s]
Downloading best_ckpt_mp0.tar: 17%|█▋ | 1.39G/8.31G [00:26<02:38, 46.8MB/s]
Downloading best_ckpt_mp0.tar: 17%|█▋ | 1.40G/8.31G [00:27<02:47, 44.3MB/s]
Downloading best_ckpt_mp0.tar: 17%|█▋ | 1.41G/8.31G [00:27<02:56, 42.0MB/s]
Downloading best_ckpt_mp0.tar: 17%|█▋ | 1.42G/8.31G [00:27<02:49, 43.6MB/s]
Downloading best_ckpt_mp0.tar: 17%|█▋ | 1.43G/8.31G [00:27<02:37, 47.1MB/s]
Downloading best_ckpt_mp0.tar: 17%|█▋ | 1.44G/8.31G [00:28<02:34, 47.8MB/s]
Downloading best_ckpt_mp0.tar: 17%|█▋ | 1.45G/8.31G [00:28<02:46, 44.3MB/s]
Downloading best_ckpt_mp0.tar: 18%|█▊ | 1.46G/8.31G [00:28<02:36, 47.0MB/s]
Downloading best_ckpt_mp0.tar: 18%|█▊ | 1.46G/8.31G [00:28<02:32, 48.1MB/s]
Downloading best_ckpt_mp0.tar: 18%|█▊ | 1.47G/8.31G [00:28<02:22, 51.7MB/s]
Downloading best_ckpt_mp0.tar: 18%|█▊ | 1.48G/8.31G [00:29<02:09, 56.5MB/s]
Downloading best_ckpt_mp0.tar: 18%|█▊ | 1.49G/8.31G [00:29<02:21, 51.8MB/s]
Downloading best_ckpt_mp0.tar: 18%|█▊ | 1.50G/8.31G [00:29<02:53, 42.2MB/s]
Downloading best_ckpt_mp0.tar: 18%|█▊ | 1.51G/8.31G [00:29<02:41, 45.1MB/s]
Downloading best_ckpt_mp0.tar: 18%|█▊ | 1.52G/8.31G [00:30<02:33, 47.4MB/s]
Downloading best_ckpt_mp0.tar: 18%|█▊ | 1.53G/8.31G [00:30<02:22, 50.9MB/s]
Downloading best_ckpt_mp0.tar: 19%|█▊ | 1.54G/8.31G [00:30<02:28, 48.8MB/s]
Downloading best_ckpt_mp0.tar: 19%|█▊ | 1.55G/8.31G [00:30<02:14, 54.1MB/s]
Downloading best_ckpt_mp0.tar: 19%|█▉ | 1.56G/8.31G [00:30<02:27, 49.0MB/s]
Downloading best_ckpt_mp0.tar: 19%|█▉ | 1.57G/8.31G [00:30<02:15, 53.4MB/s]
Downloading best_ckpt_mp0.tar: 19%|█▉ | 1.58G/8.31G [00:31<02:07, 56.4MB/s]
Downloading best_ckpt_mp0.tar: 19%|█▉ | 1.59G/8.31G [00:31<02:12, 54.3MB/s]
Downloading best_ckpt_mp0.tar: 19%|█▉ | 1.60G/8.31G [00:31<02:08, 56.0MB/s]
Downloading best_ckpt_mp0.tar: 19%|█▉ | 1.61G/8.31G [00:31<02:02, 58.8MB/s]
Downloading best_ckpt_mp0.tar: 20%|█▉ | 1.62G/8.31G [00:31<02:07, 56.2MB/s]
Downloading best_ckpt_mp0.tar: 20%|█▉ | 1.63G/8.31G [00:32<02:46, 43.2MB/s]
Downloading best_ckpt_mp0.tar: 20%|█▉ | 1.64G/8.31G [00:32<02:50, 42.1MB/s]
Downloading best_ckpt_mp0.tar: 20%|█▉ | 1.65G/8.31G [00:32<02:39, 44.8MB/s]
Downloading best_ckpt_mp0.tar: 20%|█▉ | 1.66G/8.31G [00:32<02:38, 45.0MB/s]
Downloading best_ckpt_mp0.tar: 20%|██ | 1.67G/8.31G [00:33<02:38, 45.0MB/s]
Downloading best_ckpt_mp0.tar: 20%|██ | 1.68G/8.31G [00:33<02:33, 46.5MB/s]
Downloading best_ckpt_mp0.tar: 20%|██ | 1.69G/8.31G [00:33<03:04, 38.5MB/s]
Downloading best_ckpt_mp0.tar: 20%|██ | 1.70G/8.31G [00:34<03:09, 37.5MB/s]
Downloading best_ckpt_mp0.tar: 21%|██ | 1.71G/8.31G [00:34<02:52, 41.1MB/s]
Downloading best_ckpt_mp0.tar: 21%|██ | 1.72G/8.31G [00:34<02:45, 42.8MB/s]
Downloading best_ckpt_mp0.tar: 21%|██ | 1.73G/8.31G [00:34<02:39, 44.3MB/s]
Downloading best_ckpt_mp0.tar: 21%|██ | 1.74G/8.31G [00:34<02:36, 45.1MB/s]
Downloading best_ckpt_mp0.tar: 21%|██ | 1.75G/8.31G [00:35<02:33, 45.9MB/s]
Downloading best_ckpt_mp0.tar: 21%|██ | 1.76G/8.31G [00:35<03:01, 38.8MB/s]
Downloading best_ckpt_mp0.tar: 21%|██▏ | 1.77G/8.31G [00:35<02:48, 41.7MB/s]
Downloading best_ckpt_mp0.tar: 21%|██▏ | 1.78G/8.31G [00:36<03:39, 31.9MB/s]
Downloading best_ckpt_mp0.tar: 22%|██▏ | 1.80G/8.31G [00:36<02:31, 46.1MB/s]
Downloading best_ckpt_mp0.tar: 22%|██▏ | 1.81G/8.31G [00:36<02:26, 47.7MB/s]
Downloading best_ckpt_mp0.tar: 22%|██▏ | 1.82G/8.31G [00:36<02:45, 42.2MB/s]
Downloading best_ckpt_mp0.tar: 22%|██▏ | 1.83G/8.31G [00:37<02:30, 46.2MB/s]
Downloading best_ckpt_mp0.tar: 22%|██▏ | 1.84G/8.31G [00:37<02:24, 48.2MB/s]
Downloading best_ckpt_mp0.tar: 22%|██▏ | 1.85G/8.31G [00:37<02:12, 52.3MB/s]
Downloading best_ckpt_mp0.tar: 22%|██▏ | 1.86G/8.31G [00:37<02:23, 48.4MB/s]
Downloading best_ckpt_mp0.tar: 22%|██▏ | 1.87G/8.31G [00:37<02:21, 48.9MB/s]
Downloading best_ckpt_mp0.tar: 23%|██▎ | 1.88G/8.31G [00:38<02:58, 38.8MB/s]
Downloading best_ckpt_mp0.tar: 23%|██▎ | 1.88G/8.31G [00:38<02:44, 41.9MB/s]
Downloading best_ckpt_mp0.tar: 23%|██▎ | 1.89G/8.31G [00:38<02:41, 42.6MB/s]
Downloading best_ckpt_mp0.tar: 23%|██▎ | 1.90G/8.31G [00:38<02:25, 47.2MB/s]
Downloading best_ckpt_mp0.tar: 23%|██▎ | 1.91G/8.31G [00:39<02:21, 48.5MB/s]
Downloading best_ckpt_mp0.tar: 23%|██▎ | 1.92G/8.31G [00:39<02:16, 50.2MB/s]
Downloading best_ckpt_mp0.tar: 23%|██▎ | 1.93G/8.31G [00:39<02:24, 47.3MB/s]
Downloading best_ckpt_mp0.tar: 23%|██▎ | 1.94G/8.31G [00:39<02:39, 42.7MB/s]
Downloading best_ckpt_mp0.tar: 24%|██▎ | 1.95G/8.31G [00:40<02:33, 44.6MB/s]
Downloading best_ckpt_mp0.tar: 24%|██▎ | 1.96G/8.31G [00:40<02:24, 47.2MB/s]
Downloading best_ckpt_mp0.tar: 24%|██▎ | 1.97G/8.31G [00:40<02:26, 46.5MB/s]
Downloading best_ckpt_mp0.tar: 24%|██▍ | 1.98G/8.31G [00:40<02:30, 45.1MB/s]
Downloading best_ckpt_mp0.tar: 24%|██▍ | 1.99G/8.31G [00:41<02:33, 44.2MB/s]
Downloading best_ckpt_mp0.tar: 24%|██▍ | 2.00G/8.31G [00:41<02:41, 41.9MB/s]
Downloading best_ckpt_mp0.tar: 24%|██▍ | 2.01G/8.31G [00:41<02:21, 47.6MB/s]
Downloading best_ckpt_mp0.tar: 24%|██▍ | 2.02G/8.31G [00:41<02:15, 49.7MB/s]
Downloading best_ckpt_mp0.tar: 24%|██▍ | 2.03G/8.31G [00:41<02:17, 48.9MB/s]
Downloading best_ckpt_mp0.tar: 25%|██▍ | 2.04G/8.31G [00:42<02:20, 47.9MB/s]
Downloading best_ckpt_mp0.tar: 25%|██▍ | 2.05G/8.31G [00:42<02:17, 48.8MB/s]
Downloading best_ckpt_mp0.tar: 25%|██▍ | 2.06G/8.31G [00:42<02:03, 54.3MB/s]
Downloading best_ckpt_mp0.tar: 25%|██▍ | 2.07G/8.31G [00:42<02:23, 46.6MB/s]
Downloading best_ckpt_mp0.tar: 25%|██▌ | 2.08G/8.31G [00:42<02:22, 47.0MB/s]
Downloading best_ckpt_mp0.tar: 25%|██▌ | 2.09G/8.31G [00:43<02:17, 48.6MB/s]
Downloading best_ckpt_mp0.tar: 25%|██▌ | 2.10G/8.31G [00:43<02:07, 52.4MB/s]
Downloading best_ckpt_mp0.tar: 25%|██▌ | 2.11G/8.31G [00:43<02:00, 55.5MB/s]
Downloading best_ckpt_mp0.tar: 26%|██▌ | 2.12G/8.31G [00:43<02:09, 51.5MB/s]
Downloading best_ckpt_mp0.tar: 26%|██▌ | 2.13G/8.31G [00:44<02:29, 44.3MB/s]
Downloading best_ckpt_mp0.tar: 26%|██▌ | 2.14G/8.31G [00:44<02:32, 43.5MB/s]
Downloading best_ckpt_mp0.tar: 26%|██▌ | 2.15G/8.31G [00:44<02:24, 45.7MB/s]
Downloading best_ckpt_mp0.tar: 26%|██▌ | 2.16G/8.31G [00:44<02:12, 49.7MB/s]
Downloading best_ckpt_mp0.tar: 26%|██▌ | 2.17G/8.31G [00:44<02:09, 50.9MB/s]
Downloading best_ckpt_mp0.tar: 26%|██▌ | 2.18G/8.31G [00:45<02:12, 49.5MB/s]
Downloading best_ckpt_mp0.tar: 26%|██▋ | 2.19G/8.31G [00:45<02:59, 36.7MB/s]
Downloading best_ckpt_mp0.tar: 26%|██▋ | 2.20G/8.31G [00:45<02:56, 37.3MB/s]
Downloading best_ckpt_mp0.tar: 27%|██▋ | 2.21G/8.31G [00:46<02:50, 38.5MB/s]
Downloading best_ckpt_mp0.tar: 27%|██▋ | 2.22G/8.31G [00:46<02:50, 38.4MB/s]
Downloading best_ckpt_mp0.tar: 27%|██▋ | 2.23G/8.31G [00:46<02:42, 40.2MB/s]
Downloading best_ckpt_mp0.tar: 27%|██▋ | 2.24G/8.31G [00:46<02:42, 40.2MB/s]
Downloading best_ckpt_mp0.tar: 27%|██▋ | 2.25G/8.31G [00:47<02:35, 41.9MB/s]
Downloading best_ckpt_mp0.tar: 27%|██▋ | 2.26G/8.31G [00:47<03:04, 35.1MB/s]
Downloading best_ckpt_mp0.tar: 27%|██▋ | 2.27G/8.31G [00:47<02:38, 40.9MB/s]
Downloading best_ckpt_mp0.tar: 27%|██▋ | 2.28G/8.31G [00:47<02:49, 38.2MB/s]
Downloading best_ckpt_mp0.tar: 28%|██▊ | 2.29G/8.31G [00:48<02:49, 38.2MB/s]
Downloading best_ckpt_mp0.tar: 28%|██▊ | 2.29G/8.31G [00:48<02:44, 39.2MB/s]
Downloading best_ckpt_mp0.tar: 28%|██▊ | 2.30G/8.31G [00:48<02:41, 39.8MB/s]
Downloading best_ckpt_mp0.tar: 28%|██▊ | 2.31G/8.31G [00:49<02:48, 38.2MB/s]
Downloading best_ckpt_mp0.tar: 28%|██▊ | 2.32G/8.31G [00:49<02:41, 39.7MB/s]
Downloading best_ckpt_mp0.tar: 28%|██▊ | 2.33G/8.31G [00:49<02:26, 43.7MB/s]
Downloading best_ckpt_mp0.tar: 28%|██▊ | 2.34G/8.31G [00:49<02:13, 48.1MB/s]
Downloading best_ckpt_mp0.tar: 28%|██▊ | 2.35G/8.31G [00:49<02:07, 50.0MB/s]
Downloading best_ckpt_mp0.tar: 28%|██▊ | 2.36G/8.31G [00:50<02:04, 51.1MB/s]
Downloading best_ckpt_mp0.tar: 29%|██▊ | 2.37G/8.31G [00:50<02:06, 50.3MB/s]
Downloading best_ckpt_mp0.tar: 29%|██▊ | 2.38G/8.31G [00:50<02:25, 43.9MB/s]
Downloading best_ckpt_mp0.tar: 29%|██▉ | 2.39G/8.31G [00:50<02:27, 43.2MB/s]
Downloading best_ckpt_mp0.tar: 29%|██▉ | 2.40G/8.31G [00:51<02:30, 42.1MB/s]
Downloading best_ckpt_mp0.tar: 29%|██▉ | 2.41G/8.31G [00:51<02:19, 45.4MB/s]
Downloading best_ckpt_mp0.tar: 29%|██▉ | 2.42G/8.31G [00:51<02:07, 49.7MB/s]
Downloading best_ckpt_mp0.tar: 29%|██▉ | 2.43G/8.31G [00:51<02:03, 51.1MB/s]
Downloading best_ckpt_mp0.tar: 29%|██▉ | 2.44G/8.31G [00:51<02:23, 43.9MB/s]
Downloading best_ckpt_mp0.tar: 29%|██▉ | 2.45G/8.31G [00:52<02:14, 46.7MB/s]
Downloading best_ckpt_mp0.tar: 30%|██▉ | 2.46G/8.31G [00:52<02:23, 43.6MB/s]
Downloading best_ckpt_mp0.tar: 30%|██▉ | 2.47G/8.31G [00:52<02:31, 41.3MB/s]
Downloading best_ckpt_mp0.tar: 30%|██▉ | 2.48G/8.31G [00:52<02:21, 44.3MB/s]
Downloading best_ckpt_mp0.tar: 30%|██▉ | 2.49G/8.31G [00:53<02:15, 46.0MB/s]
Downloading best_ckpt_mp0.tar: 30%|███ | 2.50G/8.31G [00:53<02:42, 38.4MB/s]
Downloading best_ckpt_mp0.tar: 30%|███ | 2.51G/8.31G [00:53<02:45, 37.6MB/s]
Downloading best_ckpt_mp0.tar: 30%|███ | 2.52G/8.31G [00:53<02:33, 40.4MB/s]
Downloading best_ckpt_mp0.tar: 30%|███ | 2.53G/8.31G [00:54<02:21, 43.9MB/s]
Downloading best_ckpt_mp0.tar: 31%|███ | 2.54G/8.31G [00:54<02:13, 46.6MB/s]
Downloading best_ckpt_mp0.tar: 31%|███ | 2.55G/8.31G [00:54<02:26, 42.2MB/s]
Downloading best_ckpt_mp0.tar: 31%|███ | 2.56G/8.31G [00:54<02:14, 46.0MB/s]
Downloading best_ckpt_mp0.tar: 31%|███ | 2.57G/8.31G [00:55<02:42, 37.9MB/s]
Downloading best_ckpt_mp0.tar: 31%|███ | 2.58G/8.31G [00:55<03:07, 32.8MB/s]
Downloading best_ckpt_mp0.tar: 31%|███▏ | 2.60G/8.31G [00:55<02:16, 44.9MB/s]
Downloading best_ckpt_mp0.tar: 31%|███▏ | 2.61G/8.31G [00:56<02:18, 44.3MB/s]
Downloading best_ckpt_mp0.tar: 31%|███▏ | 2.62G/8.31G [00:56<02:18, 44.3MB/s]
Downloading best_ckpt_mp0.tar: 32%|███▏ | 2.63G/8.31G [00:56<02:33, 39.7MB/s]
Downloading best_ckpt_mp0.tar: 32%|███▏ | 2.64G/8.31G [00:56<02:26, 41.5MB/s]
Downloading best_ckpt_mp0.tar: 32%|███▏ | 2.65G/8.31G [00:57<02:09, 47.0MB/s]
Downloading best_ckpt_mp0.tar: 32%|███▏ | 2.66G/8.31G [00:57<01:52, 54.1MB/s]
Downloading best_ckpt_mp0.tar: 32%|███▏ | 2.67G/8.31G [00:57<01:47, 56.6MB/s]
Downloading best_ckpt_mp0.tar: 32%|███▏ | 2.68G/8.31G [00:57<01:49, 55.4MB/s]
Downloading best_ckpt_mp0.tar: 32%|███▏ | 2.69G/8.31G [00:57<01:55, 52.2MB/s]
Downloading best_ckpt_mp0.tar: 32%|███▏ | 2.70G/8.31G [00:58<02:10, 46.3MB/s]
Downloading best_ckpt_mp0.tar: 33%|███▎ | 2.71G/8.31G [00:58<02:03, 48.8MB/s]
Downloading best_ckpt_mp0.tar: 33%|███▎ | 2.71G/8.31G [00:58<01:51, 53.9MB/s]
Downloading best_ckpt_mp0.tar: 33%|███▎ | 2.72G/8.31G [00:58<01:45, 56.9MB/s]
Downloading best_ckpt_mp0.tar: 33%|███▎ | 2.73G/8.31G [00:58<01:41, 59.0MB/s]
Downloading best_ckpt_mp0.tar: 33%|███▎ | 2.74G/8.31G [00:58<01:46, 56.0MB/s]
Downloading best_ckpt_mp0.tar: 33%|███▎ | 2.75G/8.31G [00:59<02:06, 47.0MB/s]
Downloading best_ckpt_mp0.tar: 33%|███▎ | 2.76G/8.31G [00:59<01:56, 51.0MB/s]
Downloading best_ckpt_mp0.tar: 33%|███▎ | 2.77G/8.31G [00:59<01:51, 53.2MB/s]
Downloading best_ckpt_mp0.tar: 33%|███▎ | 2.78G/8.31G [00:59<01:55, 51.5MB/s]
Downloading best_ckpt_mp0.tar: 34%|███▎ | 2.79G/8.31G [01:00<02:00, 49.0MB/s]
Downloading best_ckpt_mp0.tar: 34%|███▎ | 2.80G/8.31G [01:00<01:50, 53.7MB/s]
Downloading best_ckpt_mp0.tar: 34%|███▍ | 2.81G/8.31G [01:00<02:14, 44.0MB/s]
Downloading best_ckpt_mp0.tar: 34%|███▍ | 2.82G/8.31G [01:00<01:59, 49.1MB/s]
Downloading best_ckpt_mp0.tar: 34%|███▍ | 2.83G/8.31G [01:00<02:06, 46.5MB/s]
Downloading best_ckpt_mp0.tar: 34%|███▍ | 2.84G/8.31G [01:01<01:51, 52.6MB/s]
Downloading best_ckpt_mp0.tar: 34%|███▍ | 2.85G/8.31G [01:01<01:44, 56.0MB/s]
Downloading best_ckpt_mp0.tar: 34%|███▍ | 2.86G/8.31G [01:01<01:35, 61.1MB/s]
Downloading best_ckpt_mp0.tar: 35%|███▍ | 2.87G/8.31G [01:01<01:42, 56.7MB/s]
Downloading best_ckpt_mp0.tar: 35%|███▍ | 2.88G/8.31G [01:01<01:56, 49.9MB/s]
Downloading best_ckpt_mp0.tar: 35%|███▍ | 2.89G/8.31G [01:02<01:46, 54.6MB/s]
Downloading best_ckpt_mp0.tar: 35%|███▍ | 2.90G/8.31G [01:02<01:40, 58.0MB/s]
Downloading best_ckpt_mp0.tar: 35%|███▌ | 2.91G/8.31G [01:02<01:35, 60.5MB/s]
Downloading best_ckpt_mp0.tar: 35%|███▌ | 2.92G/8.31G [01:02<01:32, 62.7MB/s]
Downloading best_ckpt_mp0.tar: 35%|███▌ | 2.93G/8.31G [01:02<01:32, 62.7MB/s]
Downloading best_ckpt_mp0.tar: 35%|███▌ | 2.94G/8.31G [01:02<01:51, 51.9MB/s]
Downloading best_ckpt_mp0.tar: 35%|███▌ | 2.95G/8.31G [01:03<01:42, 56.1MB/s]
Downloading best_ckpt_mp0.tar: 36%|███▌ | 2.96G/8.31G [01:03<01:40, 57.1MB/s]
Downloading best_ckpt_mp0.tar: 36%|███▌ | 2.97G/8.31G [01:03<01:33, 61.0MB/s]
Downloading best_ckpt_mp0.tar: 36%|███▌ | 2.98G/8.31G [01:03<01:28, 64.9MB/s]
Downloading best_ckpt_mp0.tar: 36%|███▌ | 2.99G/8.31G [01:03<01:27, 65.3MB/s]
Downloading best_ckpt_mp0.tar: 36%|███▌ | 3.00G/8.31G [01:03<01:33, 60.7MB/s]
Downloading best_ckpt_mp0.tar: 36%|███▌ | 3.01G/8.31G [01:04<01:27, 64.8MB/s]
Downloading best_ckpt_mp0.tar: 36%|███▋ | 3.02G/8.31G [01:04<01:30, 62.5MB/s]
Downloading best_ckpt_mp0.tar: 36%|███▋ | 3.03G/8.31G [01:04<01:36, 58.7MB/s]
Downloading best_ckpt_mp0.tar: 37%|███▋ | 3.04G/8.31G [01:04<01:34, 59.7MB/s]
Downloading best_ckpt_mp0.tar: 37%|███▋ | 3.05G/8.31G [01:04<01:35, 59.4MB/s]
Downloading best_ckpt_mp0.tar: 37%|███▋ | 3.06G/8.31G [01:04<01:34, 59.7MB/s]
Downloading best_ckpt_mp0.tar: 37%|███▋ | 3.07G/8.31G [01:05<01:50, 50.7MB/s]
Downloading best_ckpt_mp0.tar: 37%|███▋ | 3.08G/8.31G [01:05<01:42, 54.8MB/s]
Downloading best_ckpt_mp0.tar: 37%|███▋ | 3.09G/8.31G [01:05<01:35, 58.6MB/s]
Downloading best_ckpt_mp0.tar: 37%|███▋ | 3.10G/8.31G [01:05<01:33, 59.7MB/s]
Downloading best_ckpt_mp0.tar: 37%|███▋ | 3.11G/8.31G [01:05<01:25, 65.0MB/s]
Downloading best_ckpt_mp0.tar: 37%|███▋ | 3.12G/8.31G [01:05<01:22, 67.4MB/s]
Downloading best_ckpt_mp0.tar: 38%|███▊ | 3.12G/8.31G [01:06<01:46, 52.1MB/s]
Downloading best_ckpt_mp0.tar: 38%|███▊ | 3.13G/8.31G [01:06<01:39, 55.7MB/s]
Downloading best_ckpt_mp0.tar: 38%|███▊ | 3.14G/8.31G [01:06<01:29, 61.7MB/s]
Downloading best_ckpt_mp0.tar: 38%|███▊ | 3.15G/8.31G [01:06<01:26, 64.3MB/s]
Downloading best_ckpt_mp0.tar: 38%|███▊ | 3.16G/8.31G [01:06<01:34, 58.3MB/s]
Downloading best_ckpt_mp0.tar: 38%|███▊ | 3.17G/8.31G [01:07<01:36, 57.3MB/s]
Downloading best_ckpt_mp0.tar: 38%|███▊ | 3.18G/8.31G [01:07<01:37, 56.4MB/s]
Downloading best_ckpt_mp0.tar: 38%|███▊ | 3.19G/8.31G [01:07<02:33, 35.7MB/s]
Downloading best_ckpt_mp0.tar: 39%|███▊ | 3.21G/8.31G [01:08<01:48, 50.3MB/s]
Downloading best_ckpt_mp0.tar: 39%|███▉ | 3.22G/8.31G [01:08<01:58, 46.0MB/s]
Downloading best_ckpt_mp0.tar: 39%|███▉ | 3.23G/8.31G [01:08<01:45, 51.7MB/s]
Downloading best_ckpt_mp0.tar: 39%|███▉ | 3.24G/8.31G [01:08<01:52, 48.4MB/s]
Downloading best_ckpt_mp0.tar: 39%|███▉ | 3.25G/8.31G [01:09<02:13, 40.8MB/s]
Downloading best_ckpt_mp0.tar: 39%|███▉ | 3.26G/8.31G [01:09<01:56, 46.4MB/s]
Downloading best_ckpt_mp0.tar: 39%|███▉ | 3.27G/8.31G [01:09<01:59, 45.3MB/s]
Downloading best_ckpt_mp0.tar: 39%|███▉ | 3.28G/8.31G [01:09<01:48, 50.0MB/s]
Downloading best_ckpt_mp0.tar: 40%|███▉ | 3.29G/8.31G [01:09<01:44, 51.4MB/s]
Downloading best_ckpt_mp0.tar: 40%|███▉ | 3.30G/8.31G [01:10<01:44, 51.5MB/s]
Downloading best_ckpt_mp0.tar: 40%|███▉ | 3.31G/8.31G [01:10<02:02, 43.8MB/s]
Downloading best_ckpt_mp0.tar: 40%|███▉ | 3.32G/8.31G [01:10<02:00, 44.3MB/s]
Downloading best_ckpt_mp0.tar: 40%|████ | 3.33G/8.31G [01:10<01:52, 47.6MB/s]
Downloading best_ckpt_mp0.tar: 40%|████ | 3.34G/8.31G [01:10<01:46, 50.2MB/s]
Downloading best_ckpt_mp0.tar: 40%|████ | 3.35G/8.31G [01:11<01:49, 48.7MB/s]
Downloading best_ckpt_mp0.tar: 40%|████ | 3.36G/8.31G [01:11<01:42, 51.6MB/s]
Downloading best_ckpt_mp0.tar: 41%|████ | 3.37G/8.31G [01:11<01:45, 50.3MB/s]
Downloading best_ckpt_mp0.tar: 41%|████ | 3.38G/8.31G [01:12<02:18, 38.1MB/s]
Downloading best_ckpt_mp0.tar: 41%|████ | 3.39G/8.31G [01:12<02:03, 42.8MB/s]
Downloading best_ckpt_mp0.tar: 41%|████ | 3.40G/8.31G [01:12<02:02, 43.0MB/s]
Downloading best_ckpt_mp0.tar: 41%|████ | 3.41G/8.31G [01:12<01:55, 45.6MB/s]
Downloading best_ckpt_mp0.tar: 41%|████ | 3.42G/8.31G [01:12<02:02, 42.7MB/s]
Downloading best_ckpt_mp0.tar: 41%|████▏ | 3.43G/8.31G [01:13<01:58, 44.2MB/s]
Downloading best_ckpt_mp0.tar: 41%|████▏ | 3.44G/8.31G [01:13<02:23, 36.6MB/s]
Downloading best_ckpt_mp0.tar: 41%|████▏ | 3.45G/8.31G [01:13<02:09, 40.5MB/s]
Downloading best_ckpt_mp0.tar: 42%|████▏ | 3.46G/8.31G [01:14<03:00, 28.8MB/s]
Downloading best_ckpt_mp0.tar: 42%|████▏ | 3.47G/8.31G [01:14<02:31, 34.2MB/s]
Downloading best_ckpt_mp0.tar: 42%|████▏ | 3.48G/8.31G [01:14<02:22, 36.5MB/s]
Downloading best_ckpt_mp0.tar: 42%|████▏ | 3.49G/8.31G [01:15<02:15, 38.4MB/s]
Downloading best_ckpt_mp0.tar: 42%|████▏ | 3.50G/8.31G [01:15<01:58, 43.4MB/s]
Downloading best_ckpt_mp0.tar: 42%|████▏ | 3.51G/8.31G [01:15<02:19, 36.8MB/s]
Downloading best_ckpt_mp0.tar: 42%|████▏ | 3.52G/8.31G [01:15<02:09, 39.7MB/s]
Downloading best_ckpt_mp0.tar: 42%|████▏ | 3.53G/8.31G [01:15<01:59, 42.9MB/s]
Downloading best_ckpt_mp0.tar: 43%|████▎ | 3.54G/8.31G [01:16<01:54, 44.8MB/s]
Downloading best_ckpt_mp0.tar: 43%|████▎ | 3.54G/8.31G [01:16<02:00, 42.4MB/s]
Downloading best_ckpt_mp0.tar: 43%|████▎ | 3.55G/8.31G [01:16<01:52, 45.2MB/s]
Downloading best_ckpt_mp0.tar: 43%|████▎ | 3.56G/8.31G [01:16<01:54, 44.6MB/s]
Downloading best_ckpt_mp0.tar: 43%|████▎ | 3.57G/8.31G [01:17<01:43, 49.3MB/s]
Downloading best_ckpt_mp0.tar: 43%|████▎ | 3.58G/8.31G [01:17<01:34, 53.4MB/s]
Downloading best_ckpt_mp0.tar: 43%|████▎ | 3.59G/8.31G [01:17<01:31, 55.1MB/s]
Downloading best_ckpt_mp0.tar: 43%|████▎ | 3.60G/8.31G [01:17<01:29, 56.7MB/s]
Downloading best_ckpt_mp0.tar: 43%|████▎ | 3.61G/8.31G [01:17<01:28, 57.0MB/s]
Downloading best_ckpt_mp0.tar: 44%|████▎ | 3.62G/8.31G [01:18<01:52, 44.9MB/s]
Downloading best_ckpt_mp0.tar: 44%|████▎ | 3.63G/8.31G [01:18<01:46, 47.4MB/s]
Downloading best_ckpt_mp0.tar: 44%|████▍ | 3.64G/8.31G [01:18<01:45, 47.3MB/s]
Downloading best_ckpt_mp0.tar: 44%|████▍ | 3.65G/8.31G [01:18<01:42, 48.7MB/s]
Downloading best_ckpt_mp0.tar: 44%|████▍ | 3.66G/8.31G [01:18<01:31, 54.3MB/s]
Downloading best_ckpt_mp0.tar: 44%|████▍ | 3.67G/8.31G [01:19<01:31, 54.3MB/s]
Downloading best_ckpt_mp0.tar: 44%|████▍ | 3.68G/8.31G [01:19<01:35, 52.1MB/s]
Downloading best_ckpt_mp0.tar: 44%|████▍ | 3.69G/8.31G [01:20<03:15, 25.3MB/s]
Downloading best_ckpt_mp0.tar: 45%|████▍ | 3.70G/8.31G [01:20<02:44, 30.0MB/s]
Downloading best_ckpt_mp0.tar: 45%|████▍ | 3.71G/8.31G [01:21<03:27, 23.8MB/s]
Downloading best_ckpt_mp0.tar: 45%|████▍ | 3.72G/8.31G [01:21<03:00, 27.2MB/s]
Downloading best_ckpt_mp0.tar: 45%|████▍ | 3.73G/8.31G [01:21<03:08, 26.1MB/s]
Downloading best_ckpt_mp0.tar: 45%|████▌ | 3.74G/8.31G [01:21<02:37, 31.1MB/s]
Downloading best_ckpt_mp0.tar: 45%|████▌ | 3.75G/8.31G [01:22<02:38, 30.8MB/s]
Downloading best_ckpt_mp0.tar: 45%|████▌ | 3.76G/8.31G [01:22<02:16, 35.8MB/s]
Downloading best_ckpt_mp0.tar: 45%|████▌ | 3.77G/8.31G [01:22<02:03, 39.6MB/s]
Downloading best_ckpt_mp0.tar: 45%|████▌ | 3.78G/8.31G [01:22<01:49, 44.6MB/s]
Downloading best_ckpt_mp0.tar: 46%|████▌ | 3.79G/8.31G [01:22<01:35, 50.9MB/s]
Downloading best_ckpt_mp0.tar: 46%|████▌ | 3.80G/8.31G [01:23<01:34, 51.4MB/s]
Downloading best_ckpt_mp0.tar: 46%|████▌ | 3.81G/8.31G [01:23<01:35, 50.8MB/s]
Downloading best_ckpt_mp0.tar: 46%|████▌ | 3.82G/8.31G [01:23<01:50, 43.5MB/s]
Downloading best_ckpt_mp0.tar: 46%|████▌ | 3.83G/8.31G [01:23<01:46, 45.0MB/s]
Downloading best_ckpt_mp0.tar: 46%|████▌ | 3.84G/8.31G [01:24<01:41, 47.5MB/s]
Downloading best_ckpt_mp0.tar: 46%|████▋ | 3.85G/8.31G [01:24<01:41, 47.3MB/s]
Downloading best_ckpt_mp0.tar: 46%|████▋ | 3.86G/8.31G [01:24<01:49, 43.7MB/s]
Downloading best_ckpt_mp0.tar: 47%|████▋ | 3.87G/8.31G [01:24<01:52, 42.4MB/s]
Downloading best_ckpt_mp0.tar: 47%|████▋ | 3.88G/8.31G [01:25<02:08, 37.1MB/s]
Downloading best_ckpt_mp0.tar: 47%|████▋ | 3.89G/8.31G [01:25<01:54, 41.5MB/s]
Downloading best_ckpt_mp0.tar: 47%|████▋ | 3.90G/8.31G [01:25<01:44, 45.4MB/s]
Downloading best_ckpt_mp0.tar: 47%|████▋ | 3.91G/8.31G [01:25<01:47, 44.1MB/s]
Downloading best_ckpt_mp0.tar: 47%|████▋ | 3.92G/8.31G [01:26<01:38, 47.8MB/s]
Downloading best_ckpt_mp0.tar: 47%|████▋ | 3.93G/8.31G [01:26<01:40, 46.8MB/s]
Downloading best_ckpt_mp0.tar: 47%|████▋ | 3.94G/8.31G [01:26<01:59, 39.4MB/s]
Downloading best_ckpt_mp0.tar: 47%|████▋ | 3.95G/8.31G [01:26<01:59, 39.3MB/s]
Downloading best_ckpt_mp0.tar: 48%|████▊ | 3.96G/8.31G [01:27<01:38, 47.6MB/s]
Downloading best_ckpt_mp0.tar: 48%|████▊ | 3.97G/8.31G [01:27<01:32, 50.5MB/s]
Downloading best_ckpt_mp0.tar: 48%|████▊ | 3.98G/8.31G [01:27<01:32, 50.3MB/s]
Downloading best_ckpt_mp0.tar: 48%|████▊ | 3.99G/8.31G [01:27<01:32, 50.3MB/s]
Downloading best_ckpt_mp0.tar: 48%|████▊ | 4.00G/8.31G [01:28<01:42, 45.0MB/s]
Downloading best_ckpt_mp0.tar: 48%|████▊ | 4.01G/8.31G [01:28<01:32, 49.7MB/s]
Downloading best_ckpt_mp0.tar: 48%|████▊ | 4.02G/8.31G [01:28<01:31, 50.6MB/s]
Downloading best_ckpt_mp0.tar: 49%|████▊ | 4.03G/8.31G [01:28<01:44, 44.1MB/s]
Downloading best_ckpt_mp0.tar: 49%|████▊ | 4.04G/8.31G [01:28<01:36, 47.6MB/s]
Downloading best_ckpt_mp0.tar: 49%|████▉ | 4.05G/8.31G [01:29<01:31, 50.1MB/s]
Downloading best_ckpt_mp0.tar: 49%|████▉ | 4.06G/8.31G [01:29<01:49, 41.5MB/s]
Downloading best_ckpt_mp0.tar: 49%|████▉ | 4.07G/8.31G [01:29<01:42, 44.4MB/s]
Downloading best_ckpt_mp0.tar: 49%|████▉ | 4.08G/8.31G [01:29<01:36, 47.2MB/s]
Downloading best_ckpt_mp0.tar: 49%|████▉ | 4.09G/8.31G [01:30<01:34, 47.9MB/s]
Downloading best_ckpt_mp0.tar: 49%|████▉ | 4.10G/8.31G [01:30<01:41, 44.5MB/s]
Downloading best_ckpt_mp0.tar: 49%|████▉ | 4.11G/8.31G [01:30<01:37, 46.5MB/s]
Downloading best_ckpt_mp0.tar: 50%|████▉ | 4.12G/8.31G [01:30<01:28, 50.9MB/s]
Downloading best_ckpt_mp0.tar: 50%|████▉ | 4.13G/8.31G [01:31<01:40, 44.8MB/s]
Downloading best_ckpt_mp0.tar: 50%|████▉ | 4.14G/8.31G [01:31<01:32, 48.3MB/s]
Downloading best_ckpt_mp0.tar: 50%|████▉ | 4.15G/8.31G [01:31<01:29, 49.8MB/s]
Downloading best_ckpt_mp0.tar: 50%|█████ | 4.16G/8.31G [01:31<01:26, 51.5MB/s]
Downloading best_ckpt_mp0.tar: 50%|█████ | 4.17G/8.31G [01:31<01:25, 51.9MB/s]
Downloading best_ckpt_mp0.tar: 50%|█████ | 4.18G/8.31G [01:31<01:23, 53.1MB/s]
Downloading best_ckpt_mp0.tar: 50%|█████ | 4.19G/8.31G [01:32<01:35, 46.3MB/s]
Downloading best_ckpt_mp0.tar: 51%|█████ | 4.20G/8.31G [01:32<01:31, 48.4MB/s]
Downloading best_ckpt_mp0.tar: 51%|█████ | 4.21G/8.31G [01:32<01:25, 51.3MB/s]
Downloading best_ckpt_mp0.tar: 51%|█████ | 4.22G/8.31G [01:32<01:23, 52.9MB/s]
Downloading best_ckpt_mp0.tar: 51%|█████ | 4.23G/8.31G [01:33<01:33, 46.8MB/s]
Downloading best_ckpt_mp0.tar: 51%|█████ | 4.24G/8.31G [01:33<01:29, 48.6MB/s]
Downloading best_ckpt_mp0.tar: 51%|█████ | 4.25G/8.31G [01:33<01:43, 42.1MB/s]
Downloading best_ckpt_mp0.tar: 51%|█████ | 4.26G/8.31G [01:34<02:11, 33.1MB/s]
Downloading best_ckpt_mp0.tar: 51%|█████▏ | 4.28G/8.31G [01:34<01:27, 49.5MB/s]
Downloading best_ckpt_mp0.tar: 52%|█████▏ | 4.29G/8.31G [01:34<01:17, 56.1MB/s]
Downloading best_ckpt_mp0.tar: 52%|█████▏ | 4.30G/8.31G [01:34<01:15, 56.8MB/s]
Downloading best_ckpt_mp0.tar: 52%|█████▏ | 4.31G/8.31G [01:34<01:16, 56.1MB/s]
Downloading best_ckpt_mp0.tar: 52%|█████▏ | 4.32G/8.31G [01:35<01:34, 45.4MB/s]
Downloading best_ckpt_mp0.tar: 52%|█████▏ | 4.33G/8.31G [01:35<01:31, 46.9MB/s]
Downloading best_ckpt_mp0.tar: 52%|█████▏ | 4.34G/8.31G [01:35<01:29, 47.5MB/s]
Downloading best_ckpt_mp0.tar: 52%|█████▏ | 4.35G/8.31G [01:35<01:22, 51.3MB/s]
Downloading best_ckpt_mp0.tar: 52%|█████▏ | 4.36G/8.31G [01:35<01:23, 50.7MB/s]
Downloading best_ckpt_mp0.tar: 53%|█████▎ | 4.37G/8.31G [01:36<01:25, 49.4MB/s]
Downloading best_ckpt_mp0.tar: 53%|█████▎ | 4.38G/8.31G [01:36<01:38, 42.9MB/s]
Downloading best_ckpt_mp0.tar: 53%|█████▎ | 4.38G/8.31G [01:36<01:30, 46.6MB/s]
Downloading best_ckpt_mp0.tar: 53%|█████▎ | 4.39G/8.31G [01:36<01:23, 50.2MB/s]
Downloading best_ckpt_mp0.tar: 53%|█████▎ | 4.40G/8.31G [01:37<01:32, 45.4MB/s]
Downloading best_ckpt_mp0.tar: 53%|█████▎ | 4.41G/8.31G [01:37<01:36, 43.3MB/s]
Downloading best_ckpt_mp0.tar: 53%|█████▎ | 4.42G/8.31G [01:37<01:43, 40.3MB/s]
Downloading best_ckpt_mp0.tar: 53%|█████▎ | 4.43G/8.31G [01:37<01:37, 42.5MB/s]
Downloading best_ckpt_mp0.tar: 53%|█████▎ | 4.44G/8.31G [01:38<01:53, 36.6MB/s]
Downloading best_ckpt_mp0.tar: 54%|█████▎ | 4.45G/8.31G [01:38<01:46, 39.0MB/s]
Downloading best_ckpt_mp0.tar: 54%|█████▎ | 4.46G/8.31G [01:38<01:42, 40.3MB/s]
Downloading best_ckpt_mp0.tar: 54%|█████▍ | 4.47G/8.31G [01:38<01:40, 41.0MB/s]
Downloading best_ckpt_mp0.tar: 54%|█████▍ | 4.48G/8.31G [01:39<01:36, 42.4MB/s]
Downloading best_ckpt_mp0.tar: 54%|█████▍ | 4.49G/8.31G [01:39<01:26, 47.3MB/s]
Downloading best_ckpt_mp0.tar: 54%|█████▍ | 4.50G/8.31G [01:39<01:46, 38.5MB/s]
Downloading best_ckpt_mp0.tar: 54%|█████▍ | 4.51G/8.31G [01:39<01:40, 40.8MB/s]
Downloading best_ckpt_mp0.tar: 54%|█████▍ | 4.52G/8.31G [01:40<01:36, 42.0MB/s]
Downloading best_ckpt_mp0.tar: 55%|█████▍ | 4.53G/8.31G [01:40<01:28, 45.6MB/s]
Downloading best_ckpt_mp0.tar: 55%|█████▍ | 4.54G/8.31G [01:40<01:33, 43.4MB/s]
Downloading best_ckpt_mp0.tar: 55%|█████▍ | 4.55G/8.31G [01:40<01:31, 44.2MB/s]
Downloading best_ckpt_mp0.tar: 55%|█████▍ | 4.56G/8.31G [01:41<01:39, 40.4MB/s]
Downloading best_ckpt_mp0.tar: 55%|█████▌ | 4.57G/8.31G [01:41<01:26, 46.6MB/s]
Downloading best_ckpt_mp0.tar: 55%|█████▌ | 4.58G/8.31G [01:41<01:27, 45.6MB/s]
Downloading best_ckpt_mp0.tar: 55%|█████▌ | 4.59G/8.31G [01:41<01:22, 48.6MB/s]
Downloading best_ckpt_mp0.tar: 55%|█████▌ | 4.60G/8.31G [01:41<01:17, 51.7MB/s]
Downloading best_ckpt_mp0.tar: 55%|█████▌ | 4.61G/8.31G [01:42<01:19, 49.8MB/s]
Downloading best_ckpt_mp0.tar: 56%|█████▌ | 4.62G/8.31G [01:42<01:15, 52.8MB/s]
Downloading best_ckpt_mp0.tar: 56%|█████▌ | 4.63G/8.31G [01:42<01:31, 43.4MB/s]
Downloading best_ckpt_mp0.tar: 56%|█████▌ | 4.64G/8.31G [01:42<01:23, 47.5MB/s]
Downloading best_ckpt_mp0.tar: 56%|█████▌ | 4.65G/8.31G [01:43<01:20, 48.8MB/s]
Downloading best_ckpt_mp0.tar: 56%|█████▌ | 4.66G/8.31G [01:43<01:24, 46.4MB/s]
Downloading best_ckpt_mp0.tar: 56%|█████▌ | 4.67G/8.31G [01:43<01:25, 45.7MB/s]
Downloading best_ckpt_mp0.tar: 56%|█████▋ | 4.68G/8.31G [01:43<01:15, 51.7MB/s]
Downloading best_ckpt_mp0.tar: 56%|█████▋ | 4.69G/8.31G [01:44<01:31, 42.6MB/s]
Downloading best_ckpt_mp0.tar: 57%|█████▋ | 4.70G/8.31G [01:44<01:27, 44.2MB/s]
Downloading best_ckpt_mp0.tar: 57%|█████▋ | 4.71G/8.31G [01:44<01:29, 43.3MB/s]
Downloading best_ckpt_mp0.tar: 57%|█████▋ | 4.72G/8.31G [01:44<01:26, 44.8MB/s]
Downloading best_ckpt_mp0.tar: 57%|█████▋ | 4.73G/8.31G [01:44<01:20, 48.0MB/s]
Downloading best_ckpt_mp0.tar: 57%|█████▋ | 4.74G/8.31G [01:45<01:18, 49.1MB/s]
Downloading best_ckpt_mp0.tar: 57%|█████▋ | 4.75G/8.31G [01:45<01:14, 51.4MB/s]
Downloading best_ckpt_mp0.tar: 57%|█████▋ | 4.76G/8.31G [01:45<01:27, 43.8MB/s]
Downloading best_ckpt_mp0.tar: 57%|█████▋ | 4.77G/8.31G [01:45<01:32, 40.9MB/s]
Downloading best_ckpt_mp0.tar: 57%|█████▋ | 4.78G/8.31G [01:46<01:34, 40.2MB/s]
Downloading best_ckpt_mp0.tar: 58%|█████▊ | 4.79G/8.31G [01:46<01:26, 43.8MB/s]
Downloading best_ckpt_mp0.tar: 58%|█████▊ | 4.79G/8.31G [01:46<01:23, 44.9MB/s]
Downloading best_ckpt_mp0.tar: 58%|█████▊ | 4.80G/8.31G [01:46<01:18, 47.7MB/s]
Downloading best_ckpt_mp0.tar: 58%|█████▊ | 4.81G/8.31G [01:47<01:34, 39.7MB/s]
Downloading best_ckpt_mp0.tar: 58%|█████▊ | 4.82G/8.31G [01:47<01:22, 45.6MB/s]
Downloading best_ckpt_mp0.tar: 58%|█████▊ | 4.83G/8.31G [01:47<01:23, 44.7MB/s]
Downloading best_ckpt_mp0.tar: 58%|█████▊ | 4.84G/8.31G [01:47<01:15, 49.2MB/s]
Downloading best_ckpt_mp0.tar: 58%|█████▊ | 4.85G/8.31G [01:47<01:10, 52.8MB/s]
Downloading best_ckpt_mp0.tar: 59%|█████▊ | 4.86G/8.31G [01:48<01:09, 53.5MB/s]
Downloading best_ckpt_mp0.tar: 59%|█████▊ | 4.87G/8.31G [01:48<01:20, 45.8MB/s]
Downloading best_ckpt_mp0.tar: 59%|█████▉ | 4.88G/8.31G [01:48<01:12, 50.7MB/s]
Downloading best_ckpt_mp0.tar: 59%|█████▉ | 4.89G/8.31G [01:48<01:21, 45.2MB/s]
Downloading best_ckpt_mp0.tar: 59%|█████▉ | 4.90G/8.31G [01:49<01:17, 47.3MB/s]
Downloading best_ckpt_mp0.tar: 59%|█████▉ | 4.91G/8.31G [01:49<01:15, 48.2MB/s]
Downloading best_ckpt_mp0.tar: 59%|█████▉ | 4.92G/8.31G [01:49<01:12, 49.8MB/s]
Downloading best_ckpt_mp0.tar: 59%|█████▉ | 4.93G/8.31G [01:49<01:10, 51.2MB/s]
Downloading best_ckpt_mp0.tar: 59%|█████▉ | 4.94G/8.31G [01:49<01:19, 45.2MB/s]
Downloading best_ckpt_mp0.tar: 60%|█████▉ | 4.95G/8.31G [01:50<01:12, 50.1MB/s]
Downloading best_ckpt_mp0.tar: 60%|█████▉ | 4.96G/8.31G [01:50<01:05, 54.5MB/s]
Downloading best_ckpt_mp0.tar: 60%|█████▉ | 4.97G/8.31G [01:50<01:08, 52.3MB/s]
Downloading best_ckpt_mp0.tar: 60%|█████▉ | 4.98G/8.31G [01:50<01:10, 51.0MB/s]
Downloading best_ckpt_mp0.tar: 60%|██████ | 4.99G/8.31G [01:50<01:08, 52.2MB/s]
Downloading best_ckpt_mp0.tar: 60%|██████ | 5.00G/8.31G [01:51<01:22, 42.9MB/s]
Downloading best_ckpt_mp0.tar: 60%|██████ | 5.01G/8.31G [01:51<01:14, 47.6MB/s]
Downloading best_ckpt_mp0.tar: 60%|██████ | 5.02G/8.31G [01:51<01:17, 45.4MB/s]
Downloading best_ckpt_mp0.tar: 61%|██████ | 5.03G/8.31G [01:51<01:13, 48.2MB/s]
Downloading best_ckpt_mp0.tar: 61%|██████ | 5.04G/8.31G [01:52<01:22, 42.5MB/s]
Downloading best_ckpt_mp0.tar: 61%|██████ | 5.05G/8.31G [01:52<01:12, 48.1MB/s]
Downloading best_ckpt_mp0.tar: 61%|██████ | 5.06G/8.31G [01:52<01:15, 46.5MB/s]
Downloading best_ckpt_mp0.tar: 61%|██████ | 5.07G/8.31G [01:52<01:23, 41.6MB/s]
Downloading best_ckpt_mp0.tar: 61%|██████ | 5.08G/8.31G [01:52<01:15, 45.7MB/s]
Downloading best_ckpt_mp0.tar: 61%|██████ | 5.09G/8.31G [01:53<01:10, 49.3MB/s]
Downloading best_ckpt_mp0.tar: 61%|██████▏ | 5.10G/8.31G [01:53<01:08, 50.7MB/s]
Downloading best_ckpt_mp0.tar: 61%|██████▏ | 5.11G/8.31G [01:53<01:05, 52.8MB/s]
Downloading best_ckpt_mp0.tar: 62%|██████▏ | 5.12G/8.31G [01:53<01:02, 55.2MB/s]
Downloading best_ckpt_mp0.tar: 62%|██████▏ | 5.13G/8.31G [01:53<01:12, 47.3MB/s]
Downloading best_ckpt_mp0.tar: 62%|██████▏ | 5.14G/8.31G [01:54<01:09, 49.3MB/s]
Downloading best_ckpt_mp0.tar: 62%|██████▏ | 5.15G/8.31G [01:54<01:08, 49.7MB/s]
Downloading best_ckpt_mp0.tar: 62%|██████▏ | 5.16G/8.31G [01:54<01:07, 50.3MB/s]
Downloading best_ckpt_mp0.tar: 62%|██████▏ | 5.17G/8.31G [01:54<00:59, 56.6MB/s]
Downloading best_ckpt_mp0.tar: 62%|██████▏ | 5.18G/8.31G [01:54<01:06, 50.8MB/s]
Downloading best_ckpt_mp0.tar: 62%|██████▏ | 5.19G/8.31G [01:55<01:17, 43.2MB/s]
Downloading best_ckpt_mp0.tar: 63%|██████▎ | 5.20G/8.31G [01:55<01:10, 47.2MB/s]
Downloading best_ckpt_mp0.tar: 63%|██████▎ | 5.21G/8.31G [01:55<01:09, 48.2MB/s]
Downloading best_ckpt_mp0.tar: 63%|██████▎ | 5.21G/8.31G [01:55<01:05, 50.9MB/s]
Downloading best_ckpt_mp0.tar: 63%|██████▎ | 5.22G/8.31G [01:56<01:18, 42.2MB/s]
Downloading best_ckpt_mp0.tar: 63%|██████▎ | 5.23G/8.31G [01:56<01:25, 38.8MB/s]
Downloading best_ckpt_mp0.tar: 63%|██████▎ | 5.24G/8.31G [01:56<01:16, 43.1MB/s]
Downloading best_ckpt_mp0.tar: 63%|██████▎ | 5.25G/8.31G [01:57<01:25, 38.2MB/s]
Downloading best_ckpt_mp0.tar: 63%|██████▎ | 5.26G/8.31G [01:57<01:18, 41.4MB/s]
Downloading best_ckpt_mp0.tar: 63%|██████▎ | 5.27G/8.31G [01:57<01:21, 40.1MB/s]
Downloading best_ckpt_mp0.tar: 64%|██████▎ | 5.28G/8.31G [01:57<01:16, 42.6MB/s]
Downloading best_ckpt_mp0.tar: 64%|██████▎ | 5.29G/8.31G [01:57<01:06, 48.7MB/s]
Downloading best_ckpt_mp0.tar: 64%|██████▍ | 5.30G/8.31G [01:58<01:01, 52.4MB/s]
Downloading best_ckpt_mp0.tar: 64%|██████▍ | 5.31G/8.31G [01:58<01:09, 46.3MB/s]
Downloading best_ckpt_mp0.tar: 64%|██████▍ | 5.32G/8.31G [01:58<01:15, 42.6MB/s]
Downloading best_ckpt_mp0.tar: 64%|██████▍ | 5.33G/8.31G [01:59<01:28, 36.2MB/s]
Downloading best_ckpt_mp0.tar: 64%|██████▍ | 5.35G/8.31G [01:59<01:00, 52.7MB/s]
Downloading best_ckpt_mp0.tar: 65%|██████▍ | 5.36G/8.31G [01:59<00:58, 54.3MB/s]
Downloading best_ckpt_mp0.tar: 65%|██████▍ | 5.37G/8.31G [01:59<01:09, 45.5MB/s]
Downloading best_ckpt_mp0.tar: 65%|██████▍ | 5.38G/8.31G [01:59<01:06, 47.6MB/s]
Downloading best_ckpt_mp0.tar: 65%|██████▍ | 5.39G/8.31G [02:00<01:03, 49.6MB/s]
Downloading best_ckpt_mp0.tar: 65%|██████▍ | 5.40G/8.31G [02:00<01:00, 51.8MB/s]
Downloading best_ckpt_mp0.tar: 65%|██████▌ | 5.41G/8.31G [02:00<00:57, 53.8MB/s]
Downloading best_ckpt_mp0.tar: 65%|██████▌ | 5.42G/8.31G [02:00<00:53, 58.1MB/s]
Downloading best_ckpt_mp0.tar: 65%|██████▌ | 5.43G/8.31G [02:00<01:04, 48.1MB/s]
Downloading best_ckpt_mp0.tar: 65%|██████▌ | 5.44G/8.31G [02:01<01:08, 44.8MB/s]
Downloading best_ckpt_mp0.tar: 66%|██████▌ | 5.45G/8.31G [02:01<01:03, 48.5MB/s]
Downloading best_ckpt_mp0.tar: 66%|██████▌ | 5.46G/8.31G [02:01<01:05, 46.8MB/s]
Downloading best_ckpt_mp0.tar: 66%|██████▌ | 5.47G/8.31G [02:01<01:06, 46.2MB/s]
Downloading best_ckpt_mp0.tar: 66%|██████▌ | 5.48G/8.31G [02:02<01:03, 48.0MB/s]
Downloading best_ckpt_mp0.tar: 66%|██████▌ | 5.49G/8.31G [02:02<01:03, 47.8MB/s]
Downloading best_ckpt_mp0.tar: 66%|██████▌ | 5.50G/8.31G [02:02<01:12, 41.6MB/s]
Downloading best_ckpt_mp0.tar: 66%|██████▋ | 5.51G/8.31G [02:02<01:02, 48.2MB/s]
Downloading best_ckpt_mp0.tar: 66%|██████▋ | 5.52G/8.31G [02:02<00:59, 50.3MB/s]
Downloading best_ckpt_mp0.tar: 67%|██████▋ | 5.53G/8.31G [02:03<00:53, 56.1MB/s]
Downloading best_ckpt_mp0.tar: 67%|██████▋ | 5.54G/8.31G [02:03<00:53, 55.6MB/s]
Downloading best_ckpt_mp0.tar: 67%|██████▋ | 5.55G/8.31G [02:03<00:53, 55.7MB/s]
Downloading best_ckpt_mp0.tar: 67%|██████▋ | 5.56G/8.31G [02:03<00:50, 58.8MB/s]
Downloading best_ckpt_mp0.tar: 67%|██████▋ | 5.57G/8.31G [02:03<00:57, 51.4MB/s]
Downloading best_ckpt_mp0.tar: 67%|██████▋ | 5.58G/8.31G [02:04<00:53, 54.8MB/s]
Downloading best_ckpt_mp0.tar: 67%|██████▋ | 5.59G/8.31G [02:04<00:49, 59.0MB/s]
Downloading best_ckpt_mp0.tar: 67%|██████▋ | 5.60G/8.31G [02:04<00:52, 55.5MB/s]
Downloading best_ckpt_mp0.tar: 67%|██████▋ | 5.61G/8.31G [02:04<00:50, 57.7MB/s]
Downloading best_ckpt_mp0.tar: 68%|██████▊ | 5.62G/8.31G [02:04<00:44, 64.9MB/s]
Downloading best_ckpt_mp0.tar: 68%|██████▊ | 5.62G/8.31G [02:04<00:57, 49.9MB/s]
Downloading best_ckpt_mp0.tar: 68%|██████▊ | 5.63G/8.31G [02:05<00:53, 53.7MB/s]
Downloading best_ckpt_mp0.tar: 68%|██████▊ | 5.64G/8.31G [02:05<00:58, 49.1MB/s]
Downloading best_ckpt_mp0.tar: 68%|██████▊ | 5.65G/8.31G [02:05<00:57, 49.3MB/s]
Downloading best_ckpt_mp0.tar: 68%|██████▊ | 5.66G/8.31G [02:06<01:18, 36.1MB/s]
Downloading best_ckpt_mp0.tar: 68%|██████▊ | 5.67G/8.31G [02:06<01:08, 41.3MB/s]
Downloading best_ckpt_mp0.tar: 68%|██████▊ | 5.68G/8.31G [02:06<01:11, 39.4MB/s]
Downloading best_ckpt_mp0.tar: 69%|██████▊ | 5.69G/8.31G [02:06<01:06, 42.4MB/s]
Downloading best_ckpt_mp0.tar: 69%|██████▊ | 5.70G/8.31G [02:06<01:00, 46.0MB/s]
Downloading best_ckpt_mp0.tar: 69%|██████▉ | 5.71G/8.31G [02:07<01:00, 46.3MB/s]
Downloading best_ckpt_mp0.tar: 69%|██████▉ | 5.72G/8.31G [02:07<00:59, 46.5MB/s]
Downloading best_ckpt_mp0.tar: 69%|██████▉ | 5.73G/8.31G [02:07<00:56, 48.9MB/s]
Downloading best_ckpt_mp0.tar: 69%|██████▉ | 5.74G/8.31G [02:07<00:53, 51.7MB/s]
Downloading best_ckpt_mp0.tar: 69%|██████▉ | 5.75G/8.31G [02:08<01:00, 45.1MB/s]
Downloading best_ckpt_mp0.tar: 69%|██████▉ | 5.76G/8.31G [02:08<00:57, 47.4MB/s]
Downloading best_ckpt_mp0.tar: 69%|██████▉ | 5.77G/8.31G [02:08<00:55, 49.5MB/s]
Downloading best_ckpt_mp0.tar: 70%|██████▉ | 5.78G/8.31G [02:08<00:53, 50.8MB/s]
Downloading best_ckpt_mp0.tar: 70%|██████▉ | 5.79G/8.31G [02:08<00:51, 52.8MB/s]
Downloading best_ckpt_mp0.tar: 70%|██████▉ | 5.80G/8.31G [02:09<00:52, 51.6MB/s]
Downloading best_ckpt_mp0.tar: 70%|██████▉ | 5.81G/8.31G [02:09<00:57, 46.9MB/s]
Downloading best_ckpt_mp0.tar: 70%|███████ | 5.82G/8.31G [02:09<00:56, 47.4MB/s]
Downloading best_ckpt_mp0.tar: 70%|███████ | 5.83G/8.31G [02:09<00:50, 52.3MB/s]
Downloading best_ckpt_mp0.tar: 70%|███████ | 5.84G/8.31G [02:09<00:46, 57.0MB/s]
Downloading best_ckpt_mp0.tar: 70%|███████ | 5.85G/8.31G [02:09<00:41, 63.5MB/s]
Downloading best_ckpt_mp0.tar: 71%|███████ | 5.86G/8.31G [02:10<00:46, 56.8MB/s]
Downloading best_ckpt_mp0.tar: 71%|███████ | 5.87G/8.31G [02:10<00:47, 55.7MB/s]
Downloading best_ckpt_mp0.tar: 71%|███████ | 5.88G/8.31G [02:10<01:03, 41.3MB/s]
Downloading best_ckpt_mp0.tar: 71%|███████ | 5.89G/8.31G [02:11<01:03, 41.1MB/s]
Downloading best_ckpt_mp0.tar: 71%|███████ | 5.90G/8.31G [02:11<00:57, 44.9MB/s]
Downloading best_ckpt_mp0.tar: 71%|███████ | 5.91G/8.31G [02:11<00:55, 46.1MB/s]
Downloading best_ckpt_mp0.tar: 71%|███████ | 5.92G/8.31G [02:11<00:56, 45.6MB/s]
Downloading best_ckpt_mp0.tar: 71%|███████▏ | 5.93G/8.31G [02:11<01:04, 39.8MB/s]
Downloading best_ckpt_mp0.tar: 71%|███████▏ | 5.94G/8.31G [02:12<01:06, 38.4MB/s]
Downloading best_ckpt_mp0.tar: 72%|███████▏ | 5.95G/8.31G [02:12<01:01, 41.2MB/s]
Downloading best_ckpt_mp0.tar: 72%|███████▏ | 5.96G/8.31G [02:12<01:01, 40.8MB/s]
Downloading best_ckpt_mp0.tar: 72%|███████▏ | 5.97G/8.31G [02:13<01:06, 37.6MB/s]
Downloading best_ckpt_mp0.tar: 72%|███████▏ | 5.98G/8.31G [02:13<01:01, 41.0MB/s]
Downloading best_ckpt_mp0.tar: 72%|███████▏ | 5.99G/8.31G [02:13<00:54, 45.8MB/s]
Downloading best_ckpt_mp0.tar: 72%|███████▏ | 6.00G/8.31G [02:13<01:02, 39.7MB/s]
Downloading best_ckpt_mp0.tar: 72%|███████▏ | 6.01G/8.31G [02:13<00:56, 43.6MB/s]
Downloading best_ckpt_mp0.tar: 72%|███████▏ | 6.02G/8.31G [02:14<00:57, 42.8MB/s]
Downloading best_ckpt_mp0.tar: 73%|███████▎ | 6.03G/8.31G [02:14<00:54, 44.8MB/s]
Downloading best_ckpt_mp0.tar: 73%|███████▎ | 6.04G/8.31G [02:14<00:55, 44.2MB/s]
Downloading best_ckpt_mp0.tar: 73%|███████▎ | 6.04G/8.31G [02:14<00:56, 43.1MB/s]
Downloading best_ckpt_mp0.tar: 73%|███████▎ | 6.05G/8.31G [02:15<00:55, 43.5MB/s]
Downloading best_ckpt_mp0.tar: 73%|███████▎ | 6.06G/8.31G [02:15<01:03, 37.7MB/s]
Downloading best_ckpt_mp0.tar: 73%|███████▎ | 6.07G/8.31G [02:15<00:55, 43.2MB/s]
Downloading best_ckpt_mp0.tar: 73%|███████▎ | 6.08G/8.31G [02:15<00:53, 44.4MB/s]
Downloading best_ckpt_mp0.tar: 73%|███████▎ | 6.09G/8.31G [02:16<00:49, 48.2MB/s]
Downloading best_ckpt_mp0.tar: 73%|███████▎ | 6.10G/8.31G [02:16<00:50, 46.7MB/s]
Downloading best_ckpt_mp0.tar: 74%|███████▎ | 6.11G/8.31G [02:16<00:50, 46.8MB/s]
Downloading best_ckpt_mp0.tar: 74%|███████▎ | 6.12G/8.31G [02:16<00:53, 44.1MB/s]
Downloading best_ckpt_mp0.tar: 74%|███████▍ | 6.13G/8.31G [02:17<00:56, 41.6MB/s]
Downloading best_ckpt_mp0.tar: 74%|███████▍ | 6.14G/8.31G [02:17<00:50, 46.0MB/s]
Downloading best_ckpt_mp0.tar: 74%|███████▍ | 6.15G/8.31G [02:17<00:51, 45.1MB/s]
Downloading best_ckpt_mp0.tar: 74%|███████▍ | 6.16G/8.31G [02:17<00:50, 45.6MB/s]
Downloading best_ckpt_mp0.tar: 74%|███████▍ | 6.17G/8.31G [02:17<00:47, 48.0MB/s]
Downloading best_ckpt_mp0.tar: 74%|███████▍ | 6.18G/8.31G [02:18<00:49, 46.5MB/s]
Downloading best_ckpt_mp0.tar: 75%|███████▍ | 6.19G/8.31G [02:18<00:52, 43.6MB/s]
Downloading best_ckpt_mp0.tar: 75%|███████▍ | 6.20G/8.31G [02:18<00:52, 43.5MB/s]
Downloading best_ckpt_mp0.tar: 75%|███████▍ | 6.21G/8.31G [02:18<00:53, 41.9MB/s]
Downloading best_ckpt_mp0.tar: 75%|███████▍ | 6.22G/8.31G [02:19<00:46, 48.6MB/s]
Downloading best_ckpt_mp0.tar: 75%|███████▍ | 6.23G/8.31G [02:19<00:41, 53.2MB/s]
Downloading best_ckpt_mp0.tar: 75%|███████▌ | 6.24G/8.31G [02:19<00:40, 54.9MB/s]
Downloading best_ckpt_mp0.tar: 75%|███████▌ | 6.25G/8.31G [02:19<00:46, 47.6MB/s]
Downloading best_ckpt_mp0.tar: 75%|███████▌ | 6.26G/8.31G [02:20<00:48, 45.1MB/s]
Downloading best_ckpt_mp0.tar: 75%|███████▌ | 6.27G/8.31G [02:20<00:43, 50.8MB/s]
Downloading best_ckpt_mp0.tar: 76%|███████▌ | 6.28G/8.31G [02:20<00:40, 54.1MB/s]
Downloading best_ckpt_mp0.tar: 76%|███████▌ | 6.29G/8.31G [02:20<00:38, 57.0MB/s]
Downloading best_ckpt_mp0.tar: 76%|███████▌ | 6.30G/8.31G [02:20<00:37, 58.0MB/s]
Downloading best_ckpt_mp0.tar: 76%|███████▌ | 6.31G/8.31G [02:21<00:47, 45.2MB/s]
Downloading best_ckpt_mp0.tar: 76%|███████▌ | 6.32G/8.31G [02:21<00:44, 47.5MB/s]
Downloading best_ckpt_mp0.tar: 76%|███████▌ | 6.33G/8.31G [02:21<00:47, 44.8MB/s]
Downloading best_ckpt_mp0.tar: 76%|███████▋ | 6.34G/8.31G [02:21<00:45, 46.7MB/s]
Downloading best_ckpt_mp0.tar: 76%|███████▋ | 6.35G/8.31G [02:21<00:42, 49.7MB/s]
Downloading best_ckpt_mp0.tar: 77%|███████▋ | 6.36G/8.31G [02:22<00:42, 49.7MB/s]
Downloading best_ckpt_mp0.tar: 77%|███████▋ | 6.37G/8.31G [02:22<00:40, 51.3MB/s]
Downloading best_ckpt_mp0.tar: 77%|███████▋ | 6.38G/8.31G [02:22<00:49, 42.3MB/s]
Downloading best_ckpt_mp0.tar: 77%|███████▋ | 6.39G/8.31G [02:22<00:46, 44.3MB/s]
Downloading best_ckpt_mp0.tar: 77%|███████▋ | 6.40G/8.31G [02:22<00:42, 47.9MB/s]
Downloading best_ckpt_mp0.tar: 77%|███████▋ | 6.41G/8.31G [02:23<00:39, 52.0MB/s]
Downloading best_ckpt_mp0.tar: 77%|███████▋ | 6.42G/8.31G [02:23<00:44, 46.1MB/s]
Downloading best_ckpt_mp0.tar: 77%|███████▋ | 6.43G/8.31G [02:23<00:40, 50.4MB/s]
Downloading best_ckpt_mp0.tar: 77%|███████▋ | 6.44G/8.31G [02:23<00:46, 43.7MB/s]
Downloading best_ckpt_mp0.tar: 78%|███████▊ | 6.45G/8.31G [02:24<00:43, 46.3MB/s]
Downloading best_ckpt_mp0.tar: 78%|███████▊ | 6.46G/8.31G [02:24<00:38, 51.8MB/s]
Downloading best_ckpt_mp0.tar: 78%|███████▊ | 6.46G/8.31G [02:24<00:47, 42.0MB/s]
Downloading best_ckpt_mp0.tar: 78%|███████▊ | 6.48G/8.31G [02:24<00:33, 57.7MB/s]
Downloading best_ckpt_mp0.tar: 78%|███████▊ | 6.49G/8.31G [02:24<00:31, 61.3MB/s]
Downloading best_ckpt_mp0.tar: 78%|███████▊ | 6.50G/8.31G [02:25<00:35, 54.0MB/s]
Downloading best_ckpt_mp0.tar: 78%|███████▊ | 6.51G/8.31G [02:25<00:32, 58.6MB/s]
Downloading best_ckpt_mp0.tar: 79%|███████▊ | 6.52G/8.31G [02:25<00:32, 58.6MB/s]
Downloading best_ckpt_mp0.tar: 79%|███████▊ | 6.53G/8.31G [02:25<00:30, 62.7MB/s]
Downloading best_ckpt_mp0.tar: 79%|███████▊ | 6.54G/8.31G [02:25<00:29, 65.2MB/s]
Downloading best_ckpt_mp0.tar: 79%|███████▉ | 6.55G/8.31G [02:25<00:27, 68.8MB/s]
Downloading best_ckpt_mp0.tar: 79%|███████▉ | 6.56G/8.31G [02:26<00:33, 56.1MB/s]
Downloading best_ckpt_mp0.tar: 79%|███████▉ | 6.57G/8.31G [02:26<00:30, 61.5MB/s]
Downloading best_ckpt_mp0.tar: 79%|███████▉ | 6.58G/8.31G [02:26<00:30, 61.5MB/s]
Downloading best_ckpt_mp0.tar: 79%|███████▉ | 6.59G/8.31G [02:26<00:28, 65.5MB/s]
Downloading best_ckpt_mp0.tar: 79%|███████▉ | 6.60G/8.31G [02:26<00:35, 51.5MB/s]
Downloading best_ckpt_mp0.tar: 80%|███████▉ | 6.61G/8.31G [02:27<00:32, 56.1MB/s]
Downloading best_ckpt_mp0.tar: 80%|███████▉ | 6.62G/8.31G [02:27<00:38, 46.8MB/s]
Downloading best_ckpt_mp0.tar: 80%|███████▉ | 6.63G/8.31G [02:27<00:34, 51.6MB/s]
Downloading best_ckpt_mp0.tar: 80%|███████▉ | 6.64G/8.31G [02:27<00:32, 54.6MB/s]
Downloading best_ckpt_mp0.tar: 80%|████████ | 6.65G/8.31G [02:27<00:31, 56.1MB/s]
Downloading best_ckpt_mp0.tar: 80%|████████ | 6.66G/8.31G [02:28<00:30, 57.9MB/s]
Downloading best_ckpt_mp0.tar: 80%|████████ | 6.67G/8.31G [02:28<00:26, 65.7MB/s]
Downloading best_ckpt_mp0.tar: 80%|████████ | 6.68G/8.31G [02:28<00:27, 63.8MB/s]
Downloading best_ckpt_mp0.tar: 81%|████████ | 6.69G/8.31G [02:28<00:32, 53.5MB/s]
Downloading best_ckpt_mp0.tar: 81%|████████ | 6.70G/8.31G [02:28<00:30, 56.2MB/s]
Downloading best_ckpt_mp0.tar: 81%|████████ | 6.71G/8.31G [02:28<00:29, 57.8MB/s]
Downloading best_ckpt_mp0.tar: 81%|████████ | 6.72G/8.31G [02:29<00:28, 59.7MB/s]
Downloading best_ckpt_mp0.tar: 81%|████████ | 6.73G/8.31G [02:29<00:29, 58.4MB/s]
Downloading best_ckpt_mp0.tar: 81%|████████ | 6.74G/8.31G [02:29<00:28, 59.3MB/s]
Downloading best_ckpt_mp0.tar: 81%|████████ | 6.75G/8.31G [02:29<00:33, 50.3MB/s]
Downloading best_ckpt_mp0.tar: 81%|████████▏ | 6.76G/8.31G [02:29<00:29, 56.1MB/s]
Downloading best_ckpt_mp0.tar: 81%|████████▏ | 6.77G/8.31G [02:30<00:27, 59.9MB/s]
Downloading best_ckpt_mp0.tar: 82%|████████▏ | 6.78G/8.31G [02:30<00:26, 61.5MB/s]
Downloading best_ckpt_mp0.tar: 82%|████████▏ | 6.79G/8.31G [02:30<00:25, 63.9MB/s]
Downloading best_ckpt_mp0.tar: 82%|████████▏ | 6.80G/8.31G [02:30<00:25, 63.9MB/s]
Downloading best_ckpt_mp0.tar: 82%|████████▏ | 6.81G/8.31G [02:30<00:24, 65.6MB/s]
Downloading best_ckpt_mp0.tar: 82%|████████▏ | 6.82G/8.31G [02:30<00:28, 55.8MB/s]
Downloading best_ckpt_mp0.tar: 82%|████████▏ | 6.83G/8.31G [02:31<00:25, 62.0MB/s]
Downloading best_ckpt_mp0.tar: 82%|████████▏ | 6.84G/8.31G [02:31<00:24, 64.3MB/s]
Downloading best_ckpt_mp0.tar: 82%|████████▏ | 6.85G/8.31G [02:31<00:25, 62.1MB/s]
Downloading best_ckpt_mp0.tar: 83%|████████▎ | 6.86G/8.31G [02:31<00:23, 65.4MB/s]
Downloading best_ckpt_mp0.tar: 83%|████████▎ | 6.87G/8.31G [02:31<00:22, 69.2MB/s]
Downloading best_ckpt_mp0.tar: 83%|████████▎ | 6.88G/8.31G [02:31<00:27, 55.1MB/s]
Downloading best_ckpt_mp0.tar: 83%|████████▎ | 6.88G/8.31G [02:32<00:26, 58.4MB/s]
Downloading best_ckpt_mp0.tar: 83%|████████▎ | 6.89G/8.31G [02:32<00:23, 64.8MB/s]
Downloading best_ckpt_mp0.tar: 83%|████████▎ | 6.90G/8.31G [02:32<00:23, 65.3MB/s]
Downloading best_ckpt_mp0.tar: 83%|████████▎ | 6.91G/8.31G [02:32<00:25, 58.1MB/s]
Downloading best_ckpt_mp0.tar: 83%|████████▎ | 6.92G/8.31G [02:32<00:24, 61.0MB/s]
Downloading best_ckpt_mp0.tar: 83%|████████▎ | 6.93G/8.31G [02:33<00:28, 52.0MB/s]
Downloading best_ckpt_mp0.tar: 84%|████████▎ | 6.94G/8.31G [02:33<00:27, 53.9MB/s]
Downloading best_ckpt_mp0.tar: 84%|████████▎ | 6.95G/8.31G [02:33<00:25, 56.8MB/s]
Downloading best_ckpt_mp0.tar: 84%|████████▍ | 6.96G/8.31G [02:33<00:24, 59.4MB/s]
Downloading best_ckpt_mp0.tar: 84%|████████▍ | 6.97G/8.31G [02:33<00:24, 59.4MB/s]
Downloading best_ckpt_mp0.tar: 84%|████████▍ | 6.98G/8.31G [02:33<00:22, 61.9MB/s]
Downloading best_ckpt_mp0.tar: 84%|████████▍ | 6.99G/8.31G [02:34<00:22, 63.0MB/s]
Downloading best_ckpt_mp0.tar: 84%|████████▍ | 7.00G/8.31G [02:34<00:31, 44.4MB/s]
Downloading best_ckpt_mp0.tar: 84%|████████▍ | 7.01G/8.31G [02:34<00:31, 44.7MB/s]
Downloading best_ckpt_mp0.tar: 85%|████████▍ | 7.02G/8.31G [02:34<00:32, 43.0MB/s]
Downloading best_ckpt_mp0.tar: 85%|████████▍ | 7.03G/8.31G [02:35<00:31, 43.5MB/s]
Downloading best_ckpt_mp0.tar: 85%|████████▍ | 7.04G/8.31G [02:35<00:27, 49.0MB/s]
Downloading best_ckpt_mp0.tar: 85%|████████▍ | 7.05G/8.31G [02:35<00:25, 52.7MB/s]
Downloading best_ckpt_mp0.tar: 85%|████████▍ | 7.06G/8.31G [02:35<00:36, 36.3MB/s]
Downloading best_ckpt_mp0.tar: 85%|████████▌ | 7.08G/8.31G [02:36<00:24, 52.9MB/s]
Downloading best_ckpt_mp0.tar: 85%|████████▌ | 7.09G/8.31G [02:36<00:22, 58.0MB/s]
Downloading best_ckpt_mp0.tar: 85%|████████▌ | 7.10G/8.31G [02:36<00:22, 58.3MB/s]
Downloading best_ckpt_mp0.tar: 86%|████████▌ | 7.11G/8.31G [02:36<00:21, 60.9MB/s]
Downloading best_ckpt_mp0.tar: 86%|████████▌ | 7.12G/8.31G [02:36<00:21, 60.0MB/s]
Downloading best_ckpt_mp0.tar: 86%|████████▌ | 7.13G/8.31G [02:37<00:24, 51.2MB/s]
Downloading best_ckpt_mp0.tar: 86%|████████▌ | 7.14G/8.31G [02:37<00:24, 51.3MB/s]
Downloading best_ckpt_mp0.tar: 86%|████████▌ | 7.15G/8.31G [02:37<00:22, 54.6MB/s]
Downloading best_ckpt_mp0.tar: 86%|████████▌ | 7.16G/8.31G [02:37<00:22, 54.3MB/s]
Downloading best_ckpt_mp0.tar: 86%|████████▋ | 7.17G/8.31G [02:37<00:22, 54.4MB/s]
Downloading best_ckpt_mp0.tar: 86%|████████▋ | 7.18G/8.31G [02:38<00:23, 52.4MB/s]
Downloading best_ckpt_mp0.tar: 87%|████████▋ | 7.19G/8.31G [02:38<00:26, 44.8MB/s]
Downloading best_ckpt_mp0.tar: 87%|████████▋ | 7.20G/8.31G [02:38<00:25, 46.8MB/s]
Downloading best_ckpt_mp0.tar: 87%|████████▋ | 7.21G/8.31G [02:38<00:26, 45.2MB/s]
Downloading best_ckpt_mp0.tar: 87%|████████▋ | 7.22G/8.31G [02:38<00:24, 48.8MB/s]
Downloading best_ckpt_mp0.tar: 87%|████████▋ | 7.23G/8.31G [02:39<00:22, 51.3MB/s]
Downloading best_ckpt_mp0.tar: 87%|████████▋ | 7.24G/8.31G [02:39<00:22, 50.6MB/s]
Downloading best_ckpt_mp0.tar: 87%|████████▋ | 7.25G/8.31G [02:39<00:27, 42.1MB/s]
Downloading best_ckpt_mp0.tar: 87%|████████▋ | 7.26G/8.31G [02:39<00:23, 47.6MB/s]
Downloading best_ckpt_mp0.tar: 87%|████████▋ | 7.27G/8.31G [02:40<00:25, 44.1MB/s]
Downloading best_ckpt_mp0.tar: 88%|████████▊ | 7.28G/8.31G [02:40<00:24, 45.0MB/s]
Downloading best_ckpt_mp0.tar: 88%|████████▊ | 7.29G/8.31G [02:40<00:23, 46.6MB/s]
Downloading best_ckpt_mp0.tar: 88%|████████▊ | 7.29G/8.31G [02:40<00:22, 47.8MB/s]
Downloading best_ckpt_mp0.tar: 88%|████████▊ | 7.30G/8.31G [02:40<00:21, 50.9MB/s]
Downloading best_ckpt_mp0.tar: 88%|████████▊ | 7.31G/8.31G [02:41<00:26, 39.6MB/s]
Downloading best_ckpt_mp0.tar: 88%|████████▊ | 7.32G/8.31G [02:41<00:24, 43.9MB/s]
Downloading best_ckpt_mp0.tar: 88%|████████▊ | 7.33G/8.31G [02:41<00:24, 43.2MB/s]
Downloading best_ckpt_mp0.tar: 88%|████████▊ | 7.34G/8.31G [02:42<00:23, 44.2MB/s]
Downloading best_ckpt_mp0.tar: 88%|████████▊ | 7.35G/8.31G [02:42<00:23, 44.3MB/s]
Downloading best_ckpt_mp0.tar: 89%|████████▊ | 7.36G/8.31G [02:42<00:21, 47.8MB/s]
Downloading best_ckpt_mp0.tar: 89%|████████▊ | 7.37G/8.31G [02:42<00:27, 36.9MB/s]
Downloading best_ckpt_mp0.tar: 89%|████████▉ | 7.38G/8.31G [02:43<00:23, 41.7MB/s]
Downloading best_ckpt_mp0.tar: 89%|████████▉ | 7.39G/8.31G [02:43<00:21, 45.2MB/s]
Downloading best_ckpt_mp0.tar: 89%|████████▉ | 7.40G/8.31G [02:43<00:19, 48.9MB/s]
Downloading best_ckpt_mp0.tar: 89%|████████▉ | 7.41G/8.31G [02:43<00:19, 50.0MB/s]
Downloading best_ckpt_mp0.tar: 89%|████████▉ | 7.42G/8.31G [02:43<00:19, 48.9MB/s]
Downloading best_ckpt_mp0.tar: 89%|████████▉ | 7.43G/8.31G [02:43<00:18, 51.5MB/s]
Downloading best_ckpt_mp0.tar: 90%|████████▉ | 7.44G/8.31G [02:44<00:21, 44.3MB/s]
Downloading best_ckpt_mp0.tar: 90%|████████▉ | 7.45G/8.31G [02:44<00:20, 44.4MB/s]
Downloading best_ckpt_mp0.tar: 90%|████████▉ | 7.46G/8.31G [02:44<00:19, 45.9MB/s]
Downloading best_ckpt_mp0.tar: 90%|████████▉ | 7.47G/8.31G [02:44<00:18, 49.0MB/s]
Downloading best_ckpt_mp0.tar: 90%|█████████ | 7.48G/8.31G [02:45<00:17, 51.1MB/s]
Downloading best_ckpt_mp0.tar: 90%|█████████ | 7.49G/8.31G [02:45<00:16, 52.1MB/s]
Downloading best_ckpt_mp0.tar: 90%|█████████ | 7.50G/8.31G [02:45<00:19, 44.3MB/s]
Downloading best_ckpt_mp0.tar: 90%|█████████ | 7.51G/8.31G [02:45<00:16, 51.5MB/s]
Downloading best_ckpt_mp0.tar: 90%|█████████ | 7.52G/8.31G [02:45<00:15, 56.5MB/s]
Downloading best_ckpt_mp0.tar: 91%|█████████ | 7.53G/8.31G [02:46<00:14, 59.2MB/s]
Downloading best_ckpt_mp0.tar: 91%|█████████ | 7.54G/8.31G [02:46<00:13, 59.6MB/s]
Downloading best_ckpt_mp0.tar: 91%|█████████ | 7.55G/8.31G [02:46<00:13, 58.6MB/s]
Downloading best_ckpt_mp0.tar: 91%|█████████ | 7.56G/8.31G [02:46<00:17, 46.3MB/s]
Downloading best_ckpt_mp0.tar: 91%|█████████ | 7.57G/8.31G [02:46<00:15, 51.2MB/s]
Downloading best_ckpt_mp0.tar: 91%|█████████ | 7.58G/8.31G [02:47<00:16, 48.5MB/s]
Downloading best_ckpt_mp0.tar: 91%|█████████▏| 7.59G/8.31G [02:47<00:16, 46.8MB/s]
Downloading best_ckpt_mp0.tar: 91%|█████████▏| 7.60G/8.31G [02:47<00:14, 51.6MB/s]
Downloading best_ckpt_mp0.tar: 92%|█████████▏| 7.61G/8.31G [02:47<00:13, 55.2MB/s]
Downloading best_ckpt_mp0.tar: 92%|█████████▏| 7.62G/8.31G [02:47<00:15, 49.0MB/s]
Downloading best_ckpt_mp0.tar: 92%|█████████▏| 7.63G/8.31G [02:48<00:17, 42.6MB/s]
Downloading best_ckpt_mp0.tar: 92%|█████████▏| 7.64G/8.31G [02:48<00:20, 35.8MB/s]
Downloading best_ckpt_mp0.tar: 92%|█████████▏| 7.66G/8.31G [02:48<00:13, 52.9MB/s]
Downloading best_ckpt_mp0.tar: 92%|█████████▏| 7.68G/8.31G [02:49<00:10, 63.5MB/s]
Downloading best_ckpt_mp0.tar: 92%|█████████▏| 7.69G/8.31G [02:49<00:13, 49.6MB/s]
Downloading best_ckpt_mp0.tar: 93%|█████████▎| 7.70G/8.31G [02:49<00:12, 54.1MB/s]
Downloading best_ckpt_mp0.tar: 93%|█████████▎| 7.71G/8.31G [02:49<00:11, 54.8MB/s]
Downloading best_ckpt_mp0.tar: 93%|█████████▎| 7.71G/8.31G [02:49<00:11, 56.7MB/s]
Downloading best_ckpt_mp0.tar: 93%|█████████▎| 7.72G/8.31G [02:50<00:10, 58.0MB/s]
Downloading best_ckpt_mp0.tar: 93%|█████████▎| 7.73G/8.31G [02:50<00:09, 63.5MB/s]
Downloading best_ckpt_mp0.tar: 93%|█████████▎| 7.74G/8.31G [02:50<00:09, 63.9MB/s]
Downloading best_ckpt_mp0.tar: 93%|█████████▎| 7.75G/8.31G [02:50<00:11, 53.4MB/s]
Downloading best_ckpt_mp0.tar: 93%|█████████▎| 7.76G/8.31G [02:50<00:11, 51.9MB/s]
Downloading best_ckpt_mp0.tar: 94%|█████████▎| 7.77G/8.31G [02:51<00:09, 58.4MB/s]
Downloading best_ckpt_mp0.tar: 94%|█████████▎| 7.78G/8.31G [02:51<00:09, 60.2MB/s]
Downloading best_ckpt_mp0.tar: 94%|█████████▍| 7.79G/8.31G [02:51<00:09, 56.8MB/s]
Downloading best_ckpt_mp0.tar: 94%|█████████▍| 7.80G/8.31G [02:51<00:08, 60.5MB/s]
Downloading best_ckpt_mp0.tar: 94%|█████████▍| 7.81G/8.31G [02:51<00:10, 52.3MB/s]
Downloading best_ckpt_mp0.tar: 94%|█████████▍| 7.82G/8.31G [02:51<00:09, 56.2MB/s]
Downloading best_ckpt_mp0.tar: 94%|█████████▍| 7.83G/8.31G [02:52<00:09, 54.1MB/s]
Downloading best_ckpt_mp0.tar: 94%|█████████▍| 7.84G/8.31G [02:52<00:09, 54.7MB/s]
Downloading best_ckpt_mp0.tar: 94%|█████████▍| 7.85G/8.31G [02:52<00:09, 54.2MB/s]
Downloading best_ckpt_mp0.tar: 95%|█████████▍| 7.86G/8.31G [02:52<00:08, 56.6MB/s]
Downloading best_ckpt_mp0.tar: 95%|█████████▍| 7.87G/8.31G [02:52<00:09, 51.6MB/s]
Downloading best_ckpt_mp0.tar: 95%|█████████▍| 7.88G/8.31G [02:53<00:09, 51.0MB/s]
Downloading best_ckpt_mp0.tar: 95%|█████████▍| 7.89G/8.31G [02:53<00:08, 55.9MB/s]
Downloading best_ckpt_mp0.tar: 95%|█████████▌| 7.90G/8.31G [02:53<00:07, 58.8MB/s]
Downloading best_ckpt_mp0.tar: 95%|█████████▌| 7.91G/8.31G [02:53<00:06, 62.6MB/s]
Downloading best_ckpt_mp0.tar: 95%|█████████▌| 7.92G/8.31G [02:53<00:06, 64.9MB/s]
Downloading best_ckpt_mp0.tar: 95%|█████████▌| 7.93G/8.31G [02:53<00:06, 66.6MB/s]
Downloading best_ckpt_mp0.tar: 96%|█████████▌| 7.94G/8.31G [02:54<00:07, 54.2MB/s]
Downloading best_ckpt_mp0.tar: 96%|█████████▌| 7.95G/8.31G [02:54<00:06, 58.4MB/s]
Downloading best_ckpt_mp0.tar: 96%|█████████▌| 7.96G/8.31G [02:54<00:06, 62.2MB/s]
Downloading best_ckpt_mp0.tar: 96%|█████████▌| 7.97G/8.31G [02:54<00:05, 61.9MB/s]
Downloading best_ckpt_mp0.tar: 96%|█████████▌| 7.98G/8.31G [02:54<00:05, 65.2MB/s]
Downloading best_ckpt_mp0.tar: 96%|█████████▌| 7.99G/8.31G [02:54<00:05, 67.2MB/s]
Downloading best_ckpt_mp0.tar: 96%|█████████▋| 8.00G/8.31G [02:55<00:06, 51.8MB/s]
Downloading best_ckpt_mp0.tar: 96%|█████████▋| 8.01G/8.31G [02:55<00:05, 56.4MB/s]
Downloading best_ckpt_mp0.tar: 96%|█████████▋| 8.02G/8.31G [02:55<00:05, 55.5MB/s]
Downloading best_ckpt_mp0.tar: 97%|█████████▋| 8.03G/8.31G [02:55<00:05, 58.6MB/s]
Downloading best_ckpt_mp0.tar: 97%|█████████▋| 8.04G/8.31G [02:55<00:05, 56.6MB/s]
Downloading best_ckpt_mp0.tar: 97%|█████████▋| 8.05G/8.31G [02:56<00:04, 56.6MB/s]
Downloading best_ckpt_mp0.tar: 97%|█████████▋| 8.06G/8.31G [02:56<00:05, 46.8MB/s]
Downloading best_ckpt_mp0.tar: 97%|█████████▋| 8.07G/8.31G [02:56<00:06, 42.9MB/s]
Downloading best_ckpt_mp0.tar: 97%|█████████▋| 8.08G/8.31G [02:56<00:05, 43.9MB/s]
Downloading best_ckpt_mp0.tar: 97%|█████████▋| 8.09G/8.31G [02:57<00:04, 48.8MB/s]
Downloading best_ckpt_mp0.tar: 97%|█████████▋| 8.10G/8.31G [02:57<00:04, 48.3MB/s]
Downloading best_ckpt_mp0.tar: 98%|█████████▊| 8.11G/8.31G [02:57<00:04, 50.5MB/s]
Downloading best_ckpt_mp0.tar: 98%|█████████▊| 8.12G/8.31G [02:57<00:03, 53.4MB/s]
Downloading best_ckpt_mp0.tar: 98%|█████████▊| 8.12G/8.31G [02:58<00:04, 47.3MB/s]
Downloading best_ckpt_mp0.tar: 98%|█████████▊| 8.13G/8.31G [02:58<00:03, 48.4MB/s]
Downloading best_ckpt_mp0.tar: 98%|█████████▊| 8.14G/8.31G [02:58<00:03, 45.8MB/s]
Downloading best_ckpt_mp0.tar: 98%|█████████▊| 8.15G/8.31G [02:58<00:03, 50.5MB/s]
Downloading best_ckpt_mp0.tar: 98%|█████████▊| 8.16G/8.31G [02:58<00:02, 54.4MB/s]
Downloading best_ckpt_mp0.tar: 98%|█████████▊| 8.17G/8.31G [02:58<00:02, 52.8MB/s]
Downloading best_ckpt_mp0.tar: 98%|█████████▊| 8.18G/8.31G [02:59<00:03, 43.1MB/s]
Downloading best_ckpt_mp0.tar: 99%|█████████▊| 8.19G/8.31G [02:59<00:02, 47.6MB/s]
Downloading best_ckpt_mp0.tar: 99%|█████████▊| 8.20G/8.31G [02:59<00:02, 52.5MB/s]
Downloading best_ckpt_mp0.tar: 99%|█████████▉| 8.21G/8.31G [02:59<00:01, 55.2MB/s]
Downloading best_ckpt_mp0.tar: 99%|█████████▉| 8.22G/8.31G [03:00<00:01, 50.4MB/s]
Downloading best_ckpt_mp0.tar: 99%|█████████▉| 8.23G/8.31G [03:00<00:01, 51.3MB/s]
Downloading best_ckpt_mp0.tar: 99%|█████████▉| 8.24G/8.31G [03:00<00:01, 50.9MB/s]
Downloading best_ckpt_mp0.tar: 99%|█████████▉| 8.25G/8.31G [03:00<00:01, 43.2MB/s]
Downloading best_ckpt_mp0.tar: 99%|█████████▉| 8.26G/8.31G [03:00<00:01, 47.9MB/s]
Downloading best_ckpt_mp0.tar: 100%|█████████▉| 8.27G/8.31G [03:01<00:00, 49.7MB/s]
Downloading best_ckpt_mp0.tar: 100%|█████████▉| 8.28G/8.31G [03:01<00:00, 52.8MB/s]
Downloading best_ckpt_mp0.tar: 100%|█████████▉| 8.29G/8.31G [03:01<00:00, 56.3MB/s]
Downloading best_ckpt_mp0.tar: 100%|█████████▉| 8.30G/8.31G [03:01<00:00, 60.8MB/s]
Downloading best_ckpt_mp0.tar: 100%|██████████| 8.31G/8.31G [03:01<00:00, 54.6MB/s]
Downloading best_ckpt_mp0.tar: 100%|██████████| 8.31G/8.31G [03:01<00:00, 49.1MB/s]
2026-08-15 04:44:21.530 | INFO | earth2studio.run:ensemble:394 - Running ensemble inference!
2026-08-15 04:44:21.530 | INFO | earth2studio.run:ensemble:401 - Inference device: cuda
Fetching GFS data: 0%| | 0/74 [00:00<?, ?it/s]
Fetching GFS data: 1%|▏ | 1/74 [00:00<00:38, 1.88it/s]
Fetching GFS data: 30%|██▉ | 22/74 [00:00<00:01, 45.55it/s]
Fetching GFS data: 47%|████▋ | 35/74 [00:01<00:01, 30.08it/s]
Fetching GFS data: 58%|█████▊ | 43/74 [00:01<00:01, 28.85it/s]
Fetching GFS data: 66%|██████▌ | 49/74 [00:01<00:00, 28.68it/s]
Fetching GFS data: 89%|████████▉ | 66/74 [00:01<00:00, 47.11it/s]
Fetching GFS data: 100%|██████████| 74/74 [00:01<00:00, 39.34it/s]
2026-08-15 04:44:24.929 | SUCCESS | earth2studio.run:ensemble:422 - Fetched data from GFS
2026-08-15 04:44:24.983 | INFO | earth2studio.run:ensemble:466 - Starting 2 Member Ensemble Inference with 2 number of batches.
Total Ensemble Batches: 0%| | 0/2 [00:00<?, ?it/s]
Fetching GFS data: 0%| | 0/296 [00:00<?, ?it/s]
Fetching GFS data: 0%| | 1/296 [00:00<01:48, 2.72it/s]
Fetching GFS data: 6%|▋ | 19/296 [00:00<00:05, 48.41it/s]
Fetching GFS data: 9%|▉ | 28/296 [00:00<00:07, 34.31it/s]
Fetching GFS data: 11%|█▏ | 34/296 [00:01<00:11, 23.09it/s]
Fetching GFS data: 13%|█▎ | 38/296 [00:01<00:14, 18.08it/s]
Fetching GFS data: 14%|█▍ | 42/296 [00:02<00:15, 15.98it/s]
Fetching GFS data: 20%|█▉ | 59/296 [00:02<00:08, 26.91it/s]
Fetching GFS data: 24%|██▍ | 71/296 [00:02<00:06, 35.55it/s]
Fetching GFS data: 30%|██▉ | 88/296 [00:02<00:04, 47.28it/s]
Fetching GFS data: 35%|███▌ | 105/296 [00:03<00:03, 48.50it/s]
Fetching GFS data: 38%|███▊ | 111/296 [00:03<00:03, 49.44it/s]
Fetching GFS data: 40%|████ | 119/296 [00:03<00:05, 33.54it/s]
Fetching GFS data: 44%|████▍ | 131/296 [00:04<00:05, 32.19it/s]
Fetching GFS data: 46%|████▌ | 135/296 [00:04<00:05, 30.14it/s]
Fetching GFS data: 52%|█████▏ | 153/296 [00:04<00:03, 41.44it/s]
Fetching GFS data: 58%|█████▊ | 173/296 [00:05<00:03, 39.80it/s]
Fetching GFS data: 60%|██████ | 179/296 [00:05<00:02, 41.20it/s]
Fetching GFS data: 64%|██████▎ | 188/296 [00:05<00:03, 35.06it/s]
Fetching GFS data: 66%|██████▌ | 194/296 [00:05<00:02, 35.35it/s]
Fetching GFS data: 71%|███████ | 210/296 [00:06<00:02, 34.39it/s]
Fetching GFS data: 76%|███████▌ | 224/296 [00:06<00:02, 35.14it/s]
Fetching GFS data: 85%|████████▌ | 253/296 [00:06<00:00, 52.92it/s]
Fetching GFS data: 89%|████████▉ | 264/296 [00:07<00:00, 50.65it/s]
Fetching GFS data: 91%|█████████ | 270/296 [00:07<00:00, 50.48it/s]
Fetching GFS data: 95%|█████████▍| 281/296 [00:07<00:00, 58.82it/s]
Fetching GFS data: 100%|██████████| 296/296 [00:07<00:00, 40.25it/s]
Running batch 0 inference: 0%| | 0/5 [00:00<?, ?it/s]
Running batch 0 inference: 20%|██ | 1/5 [00:00<00:00, 7.89it/s]
Running batch 0 inference: 40%|████ | 2/5 [00:00<00:01, 2.52it/s]
Running batch 0 inference: 60%|██████ | 3/5 [00:01<00:00, 2.08it/s]
Running batch 0 inference: 80%|████████ | 4/5 [00:01<00:00, 1.91it/s]
Running batch 0 inference: 100%|██████████| 5/5 [00:02<00:00, 1.84it/s]
Total Ensemble Batches: 50%|█████ | 1/2 [00:19<00:19, 19.52s/it]
Running batch 1 inference: 0%| | 0/5 [00:00<?, ?it/s]
Running batch 1 inference: 20%|██ | 1/5 [00:00<00:00, 7.48it/s]
Running batch 1 inference: 40%|████ | 2/5 [00:00<00:01, 2.51it/s]
Running batch 1 inference: 60%|██████ | 3/5 [00:01<00:00, 2.10it/s]
Running batch 1 inference: 80%|████████ | 4/5 [00:01<00:00, 1.88it/s]
Running batch 1 inference: 100%|██████████| 5/5 [00:02<00:00, 1.77it/s]
Total Ensemble Batches: 100%|██████████| 2/2 [00:22<00:00, 9.52s/it]
Total Ensemble Batches: 100%|██████████| 2/2 [00:22<00:00, 11.02s/it]
2026-08-15 04:44:47.030 | SUCCESS | earth2studio.run:ensemble:536 -
Inference complete
/
├── ensemble (2,) int64
├── lat (721,) float64
├── lead_time (5,) timedelta64[h]
├── lon (1440,) float64
├── time (1,) datetime64[ns]
├── u10m (2, 1, 5, 721, 1440) float32
└── v10m (2, 1, 5, 721, 1440) float32
Downloading config.json: 0%| | 0.00/22.4k [00:00<?, ?B/s]
Downloading config.json: 100%|██████████| 22.4k/22.4k [00:00<00:00, 228kB/s]
Downloading config.json: 100%|██████████| 22.4k/22.4k [00:00<00:00, 226kB/s]
Downloading global_means.npy: 0%| | 0.00/720 [00:00<?, ?B/s]
Downloading global_means.npy: 100%|██████████| 720/720 [00:00<00:00, 1.64kB/s]
Downloading global_means.npy: 100%|██████████| 720/720 [00:00<00:00, 1.63kB/s]
Downloading global_stds.npy: 0%| | 0.00/720 [00:00<?, ?B/s]
Downloading global_stds.npy: 100%|██████████| 720/720 [00:00<00:00, 2.09kB/s]
Downloading global_stds.npy: 100%|██████████| 720/720 [00:00<00:00, 2.07kB/s]
Downloading orography.nc: 0%| | 0.00/2.50M [00:00<?, ?B/s]
Downloading orography.nc: 100%|██████████| 2.50M/2.50M [00:00<00:00, 5.82MB/s]
Downloading orography.nc: 100%|██████████| 2.50M/2.50M [00:00<00:00, 5.76MB/s]
Downloading land_mask.nc: 0%| | 0.00/748k [00:00<?, ?B/s]
Downloading land_mask.nc: 100%|██████████| 748k/748k [00:00<00:00, 2.11MB/s]
Downloading land_mask.nc: 100%|██████████| 748k/748k [00:00<00:00, 2.08MB/s]
Downloading best_ckpt_mp0.tar: 0%| | 0.00/8.31G [00:00<?, ?B/s]
Downloading best_ckpt_mp0.tar: 0%| | 10.0M/8.31G [00:01<17:36, 8.44MB/s]
Downloading best_ckpt_mp0.tar: 0%| | 20.0M/8.31G [00:01<09:14, 16.0MB/s]
Downloading best_ckpt_mp0.tar: 0%| | 30.0M/8.31G [00:01<05:52, 25.2MB/s]
Downloading best_ckpt_mp0.tar: 0%| | 40.0M/8.31G [00:01<04:16, 34.7MB/s]
Downloading best_ckpt_mp0.tar: 1%| | 50.0M/8.31G [00:01<03:31, 42.0MB/s]
Downloading best_ckpt_mp0.tar: 1%| | 60.0M/8.31G [00:02<03:10, 46.4MB/s]
Downloading best_ckpt_mp0.tar: 1%| | 70.0M/8.31G [00:02<03:12, 45.9MB/s]
Downloading best_ckpt_mp0.tar: 1%| | 80.0M/8.31G [00:02<02:40, 55.0MB/s]
Downloading best_ckpt_mp0.tar: 1%| | 90.0M/8.31G [00:02<02:35, 56.8MB/s]
Downloading best_ckpt_mp0.tar: 1%| | 100M/8.31G [00:02<02:20, 62.9MB/s]
Downloading best_ckpt_mp0.tar: 1%|▏ | 120M/8.31G [00:02<01:57, 75.1MB/s]
Downloading best_ckpt_mp0.tar: 2%|▏ | 130M/8.31G [00:03<02:16, 64.3MB/s]
Downloading best_ckpt_mp0.tar: 2%|▏ | 140M/8.31G [00:03<02:06, 69.5MB/s]
Downloading best_ckpt_mp0.tar: 2%|▏ | 150M/8.31G [00:03<02:24, 60.7MB/s]
Downloading best_ckpt_mp0.tar: 2%|▏ | 160M/8.31G [00:03<02:14, 65.2MB/s]
Downloading best_ckpt_mp0.tar: 2%|▏ | 170M/8.31G [00:03<02:04, 70.3MB/s]
Downloading best_ckpt_mp0.tar: 2%|▏ | 180M/8.31G [00:03<02:07, 68.5MB/s]
Downloading best_ckpt_mp0.tar: 2%|▏ | 190M/8.31G [00:04<02:10, 67.1MB/s]
Downloading best_ckpt_mp0.tar: 2%|▏ | 200M/8.31G [00:04<03:15, 44.7MB/s]
Downloading best_ckpt_mp0.tar: 2%|▏ | 210M/8.31G [00:04<03:06, 46.6MB/s]
Downloading best_ckpt_mp0.tar: 3%|▎ | 220M/8.31G [00:04<02:58, 48.7MB/s]
Downloading best_ckpt_mp0.tar: 3%|▎ | 230M/8.31G [00:05<02:52, 50.5MB/s]
Downloading best_ckpt_mp0.tar: 3%|▎ | 240M/8.31G [00:05<02:31, 57.1MB/s]
Downloading best_ckpt_mp0.tar: 3%|▎ | 250M/8.31G [00:05<02:21, 61.3MB/s]
Downloading best_ckpt_mp0.tar: 3%|▎ | 260M/8.31G [00:05<02:45, 52.2MB/s]
Downloading best_ckpt_mp0.tar: 3%|▎ | 270M/8.31G [00:05<02:35, 55.5MB/s]
Downloading best_ckpt_mp0.tar: 3%|▎ | 280M/8.31G [00:06<02:42, 53.2MB/s]
Downloading best_ckpt_mp0.tar: 3%|▎ | 290M/8.31G [00:06<02:31, 56.9MB/s]
Downloading best_ckpt_mp0.tar: 4%|▎ | 300M/8.31G [00:06<02:36, 55.1MB/s]
Downloading best_ckpt_mp0.tar: 4%|▎ | 310M/8.31G [00:06<02:35, 55.1MB/s]
Downloading best_ckpt_mp0.tar: 4%|▍ | 320M/8.31G [00:06<02:54, 49.1MB/s]
Downloading best_ckpt_mp0.tar: 4%|▍ | 330M/8.31G [00:06<02:33, 56.0MB/s]
Downloading best_ckpt_mp0.tar: 4%|▍ | 340M/8.31G [00:07<02:16, 62.8MB/s]
Downloading best_ckpt_mp0.tar: 4%|▍ | 350M/8.31G [00:07<02:00, 71.1MB/s]
Downloading best_ckpt_mp0.tar: 4%|▍ | 360M/8.31G [00:07<02:32, 56.0MB/s]
Downloading best_ckpt_mp0.tar: 4%|▍ | 370M/8.31G [00:07<02:24, 59.2MB/s]
Downloading best_ckpt_mp0.tar: 4%|▍ | 380M/8.31G [00:07<02:25, 58.7MB/s]
Downloading best_ckpt_mp0.tar: 5%|▍ | 390M/8.31G [00:08<03:18, 43.0MB/s]
Downloading best_ckpt_mp0.tar: 5%|▍ | 400M/8.31G [00:08<03:01, 46.8MB/s]
Downloading best_ckpt_mp0.tar: 5%|▍ | 410M/8.31G [00:08<02:48, 50.3MB/s]
Downloading best_ckpt_mp0.tar: 5%|▍ | 420M/8.31G [00:08<02:54, 48.7MB/s]
Downloading best_ckpt_mp0.tar: 5%|▌ | 430M/8.31G [00:08<02:34, 54.8MB/s]
Downloading best_ckpt_mp0.tar: 5%|▌ | 440M/8.31G [00:09<02:24, 58.7MB/s]
Downloading best_ckpt_mp0.tar: 5%|▌ | 450M/8.31G [00:09<02:33, 55.1MB/s]
Downloading best_ckpt_mp0.tar: 5%|▌ | 460M/8.31G [00:09<02:22, 59.3MB/s]
Downloading best_ckpt_mp0.tar: 6%|▌ | 470M/8.31G [00:09<02:21, 59.5MB/s]
Downloading best_ckpt_mp0.tar: 6%|▌ | 480M/8.31G [00:09<02:15, 62.3MB/s]
Downloading best_ckpt_mp0.tar: 6%|▌ | 490M/8.31G [00:09<02:08, 65.4MB/s]
Downloading best_ckpt_mp0.tar: 6%|▌ | 500M/8.31G [00:10<02:05, 67.0MB/s]
Downloading best_ckpt_mp0.tar: 6%|▌ | 510M/8.31G [00:10<01:55, 72.9MB/s]
Downloading best_ckpt_mp0.tar: 6%|▌ | 520M/8.31G [00:10<02:19, 59.9MB/s]
Downloading best_ckpt_mp0.tar: 6%|▌ | 530M/8.31G [00:10<02:17, 60.8MB/s]
Downloading best_ckpt_mp0.tar: 6%|▋ | 540M/8.31G [00:10<02:21, 59.1MB/s]
Downloading best_ckpt_mp0.tar: 6%|▋ | 550M/8.31G [00:10<02:25, 57.5MB/s]
Downloading best_ckpt_mp0.tar: 7%|▋ | 560M/8.31G [00:11<02:14, 61.8MB/s]
Downloading best_ckpt_mp0.tar: 7%|▋ | 570M/8.31G [00:11<02:08, 64.7MB/s]
Downloading best_ckpt_mp0.tar: 7%|▋ | 580M/8.31G [00:11<02:33, 54.0MB/s]
Downloading best_ckpt_mp0.tar: 7%|▋ | 590M/8.31G [00:11<03:20, 41.5MB/s]
Downloading best_ckpt_mp0.tar: 7%|▋ | 600M/8.31G [00:12<03:04, 45.1MB/s]
Downloading best_ckpt_mp0.tar: 7%|▋ | 610M/8.31G [00:12<02:44, 50.3MB/s]
Downloading best_ckpt_mp0.tar: 7%|▋ | 620M/8.31G [00:12<02:43, 50.6MB/s]
Downloading best_ckpt_mp0.tar: 7%|▋ | 630M/8.31G [00:12<03:23, 40.7MB/s]
Downloading best_ckpt_mp0.tar: 8%|▊ | 640M/8.31G [00:13<03:24, 40.3MB/s]
Downloading best_ckpt_mp0.tar: 8%|▊ | 650M/8.31G [00:13<02:57, 46.3MB/s]
Downloading best_ckpt_mp0.tar: 8%|▊ | 660M/8.31G [00:13<04:12, 32.6MB/s]
Downloading best_ckpt_mp0.tar: 8%|▊ | 670M/8.31G [00:14<04:22, 31.3MB/s]
Downloading best_ckpt_mp0.tar: 8%|▊ | 680M/8.31G [00:14<03:48, 35.9MB/s]
Downloading best_ckpt_mp0.tar: 8%|▊ | 690M/8.31G [00:14<03:09, 43.3MB/s]
Downloading best_ckpt_mp0.tar: 8%|▊ | 700M/8.31G [00:14<03:00, 45.4MB/s]
Downloading best_ckpt_mp0.tar: 8%|▊ | 710M/8.31G [00:15<03:34, 38.0MB/s]
Downloading best_ckpt_mp0.tar: 8%|▊ | 720M/8.31G [00:15<03:11, 42.6MB/s]
Downloading best_ckpt_mp0.tar: 9%|▊ | 730M/8.31G [00:15<04:59, 27.3MB/s]
Downloading best_ckpt_mp0.tar: 9%|▊ | 740M/8.31G [00:16<04:28, 30.3MB/s]
Downloading best_ckpt_mp0.tar: 9%|▉ | 750M/8.31G [00:16<03:47, 35.8MB/s]
Downloading best_ckpt_mp0.tar: 9%|▉ | 760M/8.31G [00:16<03:39, 36.9MB/s]
Downloading best_ckpt_mp0.tar: 9%|▉ | 770M/8.31G [00:16<03:39, 36.9MB/s]
Downloading best_ckpt_mp0.tar: 9%|▉ | 780M/8.31G [00:17<03:05, 43.6MB/s]
Downloading best_ckpt_mp0.tar: 9%|▉ | 790M/8.31G [00:17<02:47, 48.2MB/s]
Downloading best_ckpt_mp0.tar: 9%|▉ | 800M/8.31G [00:17<02:34, 52.5MB/s]
Downloading best_ckpt_mp0.tar: 10%|▉ | 810M/8.31G [00:17<02:36, 51.7MB/s]
Downloading best_ckpt_mp0.tar: 10%|▉ | 820M/8.31G [00:17<02:38, 50.9MB/s]
Downloading best_ckpt_mp0.tar: 10%|▉ | 830M/8.31G [00:17<02:22, 56.4MB/s]
Downloading best_ckpt_mp0.tar: 10%|▉ | 840M/8.31G [00:18<02:42, 49.6MB/s]
Downloading best_ckpt_mp0.tar: 10%|▉ | 850M/8.31G [00:18<02:25, 55.2MB/s]
Downloading best_ckpt_mp0.tar: 10%|█ | 860M/8.31G [00:18<02:14, 59.8MB/s]
Downloading best_ckpt_mp0.tar: 10%|█ | 870M/8.31G [00:18<02:48, 47.5MB/s]
Downloading best_ckpt_mp0.tar: 10%|█ | 880M/8.31G [00:18<02:31, 53.0MB/s]
Downloading best_ckpt_mp0.tar: 10%|█ | 890M/8.31G [00:19<02:26, 54.4MB/s]
Downloading best_ckpt_mp0.tar: 11%|█ | 900M/8.31G [00:19<02:39, 50.0MB/s]
Downloading best_ckpt_mp0.tar: 11%|█ | 910M/8.31G [00:19<02:25, 54.7MB/s]
Downloading best_ckpt_mp0.tar: 11%|█ | 920M/8.31G [00:19<02:11, 60.5MB/s]
Downloading best_ckpt_mp0.tar: 11%|█ | 930M/8.31G [00:19<02:02, 64.9MB/s]
Downloading best_ckpt_mp0.tar: 11%|█ | 940M/8.31G [00:20<02:16, 58.1MB/s]
Downloading best_ckpt_mp0.tar: 11%|█ | 950M/8.31G [00:20<02:25, 54.6MB/s]
Downloading best_ckpt_mp0.tar: 11%|█▏ | 960M/8.31G [00:20<02:44, 48.2MB/s]
Downloading best_ckpt_mp0.tar: 11%|█▏ | 970M/8.31G [00:20<02:28, 53.2MB/s]
Downloading best_ckpt_mp0.tar: 12%|█▏ | 980M/8.31G [00:20<02:14, 58.7MB/s]
Downloading best_ckpt_mp0.tar: 12%|█▏ | 990M/8.31G [00:20<01:59, 65.8MB/s]
Downloading best_ckpt_mp0.tar: 12%|█▏ | 0.98G/8.31G [00:21<01:54, 68.5MB/s]
Downloading best_ckpt_mp0.tar: 12%|█▏ | 0.99G/8.31G [00:21<01:50, 71.4MB/s]
Downloading best_ckpt_mp0.tar: 12%|█▏ | 1.00G/8.31G [00:21<01:48, 72.3MB/s]
Downloading best_ckpt_mp0.tar: 12%|█▏ | 1.01G/8.31G [00:21<02:10, 60.1MB/s]
Downloading best_ckpt_mp0.tar: 12%|█▏ | 1.02G/8.31G [00:21<02:06, 61.8MB/s]
Downloading best_ckpt_mp0.tar: 12%|█▏ | 1.03G/8.31G [00:21<02:05, 62.1MB/s]
Downloading best_ckpt_mp0.tar: 12%|█▏ | 1.04G/8.31G [00:22<01:55, 67.8MB/s]
Downloading best_ckpt_mp0.tar: 13%|█▎ | 1.04G/8.31G [00:22<01:49, 71.4MB/s]
Downloading best_ckpt_mp0.tar: 13%|█▎ | 1.05G/8.31G [00:22<01:41, 76.4MB/s]
Downloading best_ckpt_mp0.tar: 13%|█▎ | 1.06G/8.31G [00:22<02:03, 63.0MB/s]
Downloading best_ckpt_mp0.tar: 13%|█▎ | 1.08G/8.31G [00:22<01:48, 71.5MB/s]
Downloading best_ckpt_mp0.tar: 13%|█▎ | 1.09G/8.31G [00:22<01:40, 77.0MB/s]
Downloading best_ckpt_mp0.tar: 13%|█▎ | 1.10G/8.31G [00:22<01:33, 82.4MB/s]
Downloading best_ckpt_mp0.tar: 13%|█▎ | 1.11G/8.31G [00:23<01:37, 79.3MB/s]
Downloading best_ckpt_mp0.tar: 14%|█▎ | 1.12G/8.31G [00:23<01:39, 77.3MB/s]
Downloading best_ckpt_mp0.tar: 14%|█▎ | 1.13G/8.31G [00:23<02:03, 62.2MB/s]
Downloading best_ckpt_mp0.tar: 14%|█▍ | 1.14G/8.31G [00:23<01:56, 66.3MB/s]
Downloading best_ckpt_mp0.tar: 14%|█▍ | 1.15G/8.31G [00:23<01:51, 68.8MB/s]
Downloading best_ckpt_mp0.tar: 14%|█▍ | 1.16G/8.31G [00:23<01:53, 67.5MB/s]
Downloading best_ckpt_mp0.tar: 14%|█▍ | 1.17G/8.31G [00:24<01:53, 67.8MB/s]
Downloading best_ckpt_mp0.tar: 14%|█▍ | 1.18G/8.31G [00:24<01:51, 68.5MB/s]
Downloading best_ckpt_mp0.tar: 14%|█▍ | 1.19G/8.31G [00:24<02:15, 56.6MB/s]
Downloading best_ckpt_mp0.tar: 14%|█▍ | 1.20G/8.31G [00:24<01:57, 64.7MB/s]
Downloading best_ckpt_mp0.tar: 15%|█▍ | 1.21G/8.31G [00:24<02:07, 59.6MB/s]
Downloading best_ckpt_mp0.tar: 15%|█▍ | 1.22G/8.31G [00:24<02:04, 61.2MB/s]
Downloading best_ckpt_mp0.tar: 15%|█▍ | 1.23G/8.31G [00:25<01:50, 68.8MB/s]
Downloading best_ckpt_mp0.tar: 15%|█▍ | 1.24G/8.31G [00:25<01:49, 69.1MB/s]
Downloading best_ckpt_mp0.tar: 15%|█▌ | 1.25G/8.31G [00:25<02:07, 59.3MB/s]
Downloading best_ckpt_mp0.tar: 15%|█▌ | 1.26G/8.31G [00:25<01:58, 64.1MB/s]
Downloading best_ckpt_mp0.tar: 15%|█▌ | 1.27G/8.31G [00:25<01:53, 66.4MB/s]
Downloading best_ckpt_mp0.tar: 15%|█▌ | 1.28G/8.31G [00:25<01:52, 66.9MB/s]
Downloading best_ckpt_mp0.tar: 16%|█▌ | 1.29G/8.31G [00:26<01:51, 67.3MB/s]
Downloading best_ckpt_mp0.tar: 16%|█▌ | 1.30G/8.31G [00:26<01:47, 69.7MB/s]
Downloading best_ckpt_mp0.tar: 16%|█▌ | 1.31G/8.31G [00:26<01:47, 69.9MB/s]
Downloading best_ckpt_mp0.tar: 16%|█▌ | 1.32G/8.31G [00:26<02:16, 55.2MB/s]
Downloading best_ckpt_mp0.tar: 16%|█▌ | 1.33G/8.31G [00:26<02:02, 61.3MB/s]
Downloading best_ckpt_mp0.tar: 16%|█▌ | 1.34G/8.31G [00:26<02:17, 54.5MB/s]
Downloading best_ckpt_mp0.tar: 16%|█▌ | 1.35G/8.31G [00:27<02:24, 51.7MB/s]
Downloading best_ckpt_mp0.tar: 16%|█▋ | 1.36G/8.31G [00:27<02:12, 56.3MB/s]
Downloading best_ckpt_mp0.tar: 16%|█▋ | 1.37G/8.31G [00:27<02:07, 58.4MB/s]
Downloading best_ckpt_mp0.tar: 17%|█▋ | 1.38G/8.31G [00:27<02:29, 49.8MB/s]
Downloading best_ckpt_mp0.tar: 17%|█▋ | 1.39G/8.31G [00:27<02:14, 55.3MB/s]
Downloading best_ckpt_mp0.tar: 17%|█▋ | 1.40G/8.31G [00:28<02:59, 41.3MB/s]
Downloading best_ckpt_mp0.tar: 17%|█▋ | 1.41G/8.31G [00:28<02:35, 47.5MB/s]
Downloading best_ckpt_mp0.tar: 17%|█▋ | 1.42G/8.31G [00:28<03:24, 36.2MB/s]
Downloading best_ckpt_mp0.tar: 17%|█▋ | 1.43G/8.31G [00:29<03:07, 39.4MB/s]
Downloading best_ckpt_mp0.tar: 17%|█▋ | 1.44G/8.31G [00:29<02:39, 46.1MB/s]
Downloading best_ckpt_mp0.tar: 17%|█▋ | 1.45G/8.31G [00:29<02:56, 41.8MB/s]
Downloading best_ckpt_mp0.tar: 18%|█▊ | 1.46G/8.31G [00:29<02:32, 48.2MB/s]
Downloading best_ckpt_mp0.tar: 18%|█▊ | 1.46G/8.31G [00:29<02:13, 55.2MB/s]
Downloading best_ckpt_mp0.tar: 18%|█▊ | 1.47G/8.31G [00:29<02:01, 60.3MB/s]
Downloading best_ckpt_mp0.tar: 18%|█▊ | 1.48G/8.31G [00:30<01:58, 61.9MB/s]
Downloading best_ckpt_mp0.tar: 18%|█▊ | 1.49G/8.31G [00:30<01:51, 65.6MB/s]
Downloading best_ckpt_mp0.tar: 18%|█▊ | 1.50G/8.31G [00:30<02:11, 55.6MB/s]
Downloading best_ckpt_mp0.tar: 18%|█▊ | 1.51G/8.31G [00:30<02:04, 58.7MB/s]
Downloading best_ckpt_mp0.tar: 18%|█▊ | 1.52G/8.31G [00:30<01:55, 63.1MB/s]
Downloading best_ckpt_mp0.tar: 18%|█▊ | 1.53G/8.31G [00:30<01:47, 67.7MB/s]
Downloading best_ckpt_mp0.tar: 19%|█▊ | 1.54G/8.31G [00:31<01:57, 61.9MB/s]
Downloading best_ckpt_mp0.tar: 19%|█▊ | 1.55G/8.31G [00:31<01:48, 66.8MB/s]
Downloading best_ckpt_mp0.tar: 19%|█▉ | 1.56G/8.31G [00:31<02:33, 47.1MB/s]
Downloading best_ckpt_mp0.tar: 19%|█▉ | 1.57G/8.31G [00:31<02:24, 50.1MB/s]
Downloading best_ckpt_mp0.tar: 19%|█▉ | 1.58G/8.31G [00:32<02:16, 52.9MB/s]
Downloading best_ckpt_mp0.tar: 19%|█▉ | 1.59G/8.31G [00:32<02:12, 54.4MB/s]
Downloading best_ckpt_mp0.tar: 19%|█▉ | 1.60G/8.31G [00:32<02:18, 52.1MB/s]
Downloading best_ckpt_mp0.tar: 19%|█▉ | 1.61G/8.31G [00:32<02:13, 53.9MB/s]
Downloading best_ckpt_mp0.tar: 20%|█▉ | 1.62G/8.31G [00:32<02:05, 57.3MB/s]
Downloading best_ckpt_mp0.tar: 20%|█▉ | 1.63G/8.31G [00:33<02:29, 47.8MB/s]
Downloading best_ckpt_mp0.tar: 20%|█▉ | 1.64G/8.31G [00:33<02:23, 50.1MB/s]
Downloading best_ckpt_mp0.tar: 20%|█▉ | 1.65G/8.31G [00:33<02:14, 53.0MB/s]
Downloading best_ckpt_mp0.tar: 20%|█▉ | 1.66G/8.31G [00:33<02:04, 57.1MB/s]
Downloading best_ckpt_mp0.tar: 20%|██ | 1.67G/8.31G [00:33<01:54, 62.4MB/s]
Downloading best_ckpt_mp0.tar: 20%|██ | 1.69G/8.31G [00:34<02:01, 58.5MB/s]
Downloading best_ckpt_mp0.tar: 20%|██ | 1.70G/8.31G [00:34<01:53, 62.4MB/s]
Downloading best_ckpt_mp0.tar: 21%|██ | 1.71G/8.31G [00:34<02:10, 54.3MB/s]
Downloading best_ckpt_mp0.tar: 21%|██ | 1.72G/8.31G [00:34<02:03, 57.4MB/s]
Downloading best_ckpt_mp0.tar: 21%|██ | 1.73G/8.31G [00:34<01:57, 60.0MB/s]
Downloading best_ckpt_mp0.tar: 21%|██ | 1.74G/8.31G [00:34<01:45, 67.0MB/s]
Downloading best_ckpt_mp0.tar: 21%|██ | 1.75G/8.31G [00:35<01:48, 65.2MB/s]
Downloading best_ckpt_mp0.tar: 21%|██ | 1.76G/8.31G [00:35<02:30, 46.7MB/s]
Downloading best_ckpt_mp0.tar: 21%|██▏ | 1.77G/8.31G [00:35<02:42, 43.2MB/s]
Downloading best_ckpt_mp0.tar: 21%|██▏ | 1.78G/8.31G [00:35<02:43, 42.8MB/s]
Downloading best_ckpt_mp0.tar: 22%|██▏ | 1.79G/8.31G [00:36<02:37, 44.4MB/s]
Downloading best_ckpt_mp0.tar: 22%|██▏ | 1.80G/8.31G [00:36<02:17, 50.9MB/s]
Downloading best_ckpt_mp0.tar: 22%|██▏ | 1.81G/8.31G [00:36<02:28, 47.1MB/s]
Downloading best_ckpt_mp0.tar: 22%|██▏ | 1.82G/8.31G [00:36<02:42, 43.0MB/s]
Downloading best_ckpt_mp0.tar: 22%|██▏ | 1.83G/8.31G [00:37<02:19, 49.9MB/s]
Downloading best_ckpt_mp0.tar: 22%|██▏ | 1.84G/8.31G [00:37<02:12, 52.5MB/s]
Downloading best_ckpt_mp0.tar: 22%|██▏ | 1.85G/8.31G [00:37<02:03, 56.0MB/s]
Downloading best_ckpt_mp0.tar: 22%|██▏ | 1.86G/8.31G [00:37<01:54, 60.7MB/s]
Downloading best_ckpt_mp0.tar: 22%|██▏ | 1.87G/8.31G [00:37<01:51, 62.2MB/s]
Downloading best_ckpt_mp0.tar: 23%|██▎ | 1.88G/8.31G [00:37<02:07, 54.1MB/s]
Downloading best_ckpt_mp0.tar: 23%|██▎ | 1.88G/8.31G [00:38<01:57, 58.7MB/s]
Downloading best_ckpt_mp0.tar: 23%|██▎ | 1.89G/8.31G [00:38<01:45, 65.0MB/s]
Downloading best_ckpt_mp0.tar: 23%|██▎ | 1.90G/8.31G [00:38<01:45, 65.2MB/s]
Downloading best_ckpt_mp0.tar: 23%|██▎ | 1.91G/8.31G [00:38<01:48, 63.6MB/s]
Downloading best_ckpt_mp0.tar: 23%|██▎ | 1.92G/8.31G [00:38<01:51, 61.6MB/s]
Downloading best_ckpt_mp0.tar: 23%|██▎ | 1.93G/8.31G [00:38<01:43, 66.4MB/s]
Downloading best_ckpt_mp0.tar: 23%|██▎ | 1.94G/8.31G [00:39<02:07, 53.6MB/s]
Downloading best_ckpt_mp0.tar: 24%|██▎ | 1.95G/8.31G [00:39<02:06, 54.1MB/s]
Downloading best_ckpt_mp0.tar: 24%|██▎ | 1.96G/8.31G [00:39<02:17, 49.6MB/s]
Downloading best_ckpt_mp0.tar: 24%|██▎ | 1.97G/8.31G [00:39<02:15, 50.4MB/s]
Downloading best_ckpt_mp0.tar: 24%|██▍ | 1.98G/8.31G [00:40<02:23, 47.3MB/s]
Downloading best_ckpt_mp0.tar: 24%|██▍ | 1.99G/8.31G [00:40<02:33, 44.2MB/s]
Downloading best_ckpt_mp0.tar: 24%|██▍ | 2.00G/8.31G [00:40<03:09, 35.8MB/s]
Downloading best_ckpt_mp0.tar: 24%|██▍ | 2.01G/8.31G [00:41<03:30, 32.1MB/s]
Downloading best_ckpt_mp0.tar: 24%|██▍ | 2.02G/8.31G [00:41<03:40, 30.6MB/s]
Downloading best_ckpt_mp0.tar: 24%|██▍ | 2.03G/8.31G [00:41<04:11, 26.8MB/s]
Downloading best_ckpt_mp0.tar: 25%|██▍ | 2.04G/8.31G [00:42<04:02, 27.8MB/s]
Downloading best_ckpt_mp0.tar: 25%|██▍ | 2.05G/8.31G [00:42<04:35, 24.4MB/s]
Downloading best_ckpt_mp0.tar: 25%|██▍ | 2.06G/8.31G [00:43<04:05, 27.3MB/s]
Downloading best_ckpt_mp0.tar: 25%|██▍ | 2.07G/8.31G [00:43<03:44, 29.9MB/s]
Downloading best_ckpt_mp0.tar: 25%|██▌ | 2.08G/8.31G [00:43<03:13, 34.6MB/s]
Downloading best_ckpt_mp0.tar: 25%|██▌ | 2.09G/8.31G [00:43<02:53, 38.5MB/s]
Downloading best_ckpt_mp0.tar: 25%|██▌ | 2.10G/8.31G [00:43<02:33, 43.4MB/s]
Downloading best_ckpt_mp0.tar: 25%|██▌ | 2.11G/8.31G [00:44<02:26, 45.5MB/s]
Downloading best_ckpt_mp0.tar: 26%|██▌ | 2.12G/8.31G [00:44<02:06, 52.7MB/s]
Downloading best_ckpt_mp0.tar: 26%|██▌ | 2.13G/8.31G [00:44<02:30, 44.1MB/s]
Downloading best_ckpt_mp0.tar: 26%|██▌ | 2.14G/8.31G [00:44<02:14, 49.4MB/s]
Downloading best_ckpt_mp0.tar: 26%|██▌ | 2.15G/8.31G [00:44<02:05, 52.9MB/s]
Downloading best_ckpt_mp0.tar: 26%|██▌ | 2.16G/8.31G [00:45<02:03, 53.5MB/s]
Downloading best_ckpt_mp0.tar: 26%|██▌ | 2.17G/8.31G [00:45<01:55, 57.0MB/s]
Downloading best_ckpt_mp0.tar: 26%|██▌ | 2.18G/8.31G [00:45<01:48, 60.5MB/s]
Downloading best_ckpt_mp0.tar: 26%|██▋ | 2.19G/8.31G [00:45<02:10, 50.4MB/s]
Downloading best_ckpt_mp0.tar: 26%|██▋ | 2.20G/8.31G [00:45<01:55, 56.6MB/s]
Downloading best_ckpt_mp0.tar: 27%|██▋ | 2.21G/8.31G [00:46<01:49, 59.7MB/s]
Downloading best_ckpt_mp0.tar: 27%|██▋ | 2.22G/8.31G [00:46<01:45, 62.1MB/s]
Downloading best_ckpt_mp0.tar: 27%|██▋ | 2.23G/8.31G [00:46<01:50, 59.3MB/s]
Downloading best_ckpt_mp0.tar: 27%|██▋ | 2.24G/8.31G [00:46<01:40, 65.0MB/s]
Downloading best_ckpt_mp0.tar: 27%|██▋ | 2.25G/8.31G [00:46<01:43, 62.9MB/s]
Downloading best_ckpt_mp0.tar: 27%|██▋ | 2.26G/8.31G [00:46<02:01, 53.4MB/s]
Downloading best_ckpt_mp0.tar: 27%|██▋ | 2.27G/8.31G [00:47<01:52, 57.9MB/s]
Downloading best_ckpt_mp0.tar: 27%|██▋ | 2.28G/8.31G [00:47<01:53, 57.3MB/s]
Downloading best_ckpt_mp0.tar: 28%|██▊ | 2.29G/8.31G [00:47<01:54, 56.5MB/s]
Downloading best_ckpt_mp0.tar: 28%|██▊ | 2.29G/8.31G [00:47<01:46, 60.8MB/s]
Downloading best_ckpt_mp0.tar: 28%|██▊ | 2.30G/8.31G [00:47<01:50, 58.2MB/s]
Downloading best_ckpt_mp0.tar: 28%|██▊ | 2.31G/8.31G [00:48<02:08, 49.9MB/s]
Downloading best_ckpt_mp0.tar: 28%|██▊ | 2.32G/8.31G [00:48<02:07, 50.3MB/s]
Downloading best_ckpt_mp0.tar: 28%|██▊ | 2.33G/8.31G [00:48<02:02, 52.4MB/s]
Downloading best_ckpt_mp0.tar: 28%|██▊ | 2.34G/8.31G [00:48<02:05, 50.9MB/s]
Downloading best_ckpt_mp0.tar: 28%|██▊ | 2.35G/8.31G [00:48<01:58, 54.0MB/s]
Downloading best_ckpt_mp0.tar: 28%|██▊ | 2.36G/8.31G [00:49<01:51, 57.0MB/s]
Downloading best_ckpt_mp0.tar: 29%|██▊ | 2.37G/8.31G [00:49<01:41, 63.1MB/s]
Downloading best_ckpt_mp0.tar: 29%|██▊ | 2.38G/8.31G [00:49<01:56, 54.4MB/s]
Downloading best_ckpt_mp0.tar: 29%|██▉ | 2.39G/8.31G [00:49<01:43, 61.4MB/s]
Downloading best_ckpt_mp0.tar: 29%|██▉ | 2.40G/8.31G [00:49<01:36, 65.8MB/s]
Downloading best_ckpt_mp0.tar: 29%|██▉ | 2.41G/8.31G [00:49<01:32, 68.2MB/s]
Downloading best_ckpt_mp0.tar: 29%|██▉ | 2.42G/8.31G [00:49<01:30, 69.9MB/s]
Downloading best_ckpt_mp0.tar: 29%|██▉ | 2.43G/8.31G [00:50<01:40, 63.0MB/s]
Downloading best_ckpt_mp0.tar: 29%|██▉ | 2.44G/8.31G [00:50<01:53, 55.4MB/s]
Downloading best_ckpt_mp0.tar: 29%|██▉ | 2.45G/8.31G [00:50<01:42, 61.1MB/s]
Downloading best_ckpt_mp0.tar: 30%|██▉ | 2.46G/8.31G [00:50<01:51, 56.2MB/s]
Downloading best_ckpt_mp0.tar: 30%|██▉ | 2.47G/8.31G [00:50<01:51, 56.0MB/s]
Downloading best_ckpt_mp0.tar: 30%|██▉ | 2.48G/8.31G [00:51<01:45, 59.4MB/s]
Downloading best_ckpt_mp0.tar: 30%|██▉ | 2.49G/8.31G [00:51<01:44, 60.0MB/s]
Downloading best_ckpt_mp0.tar: 30%|███ | 2.50G/8.31G [00:51<02:00, 51.6MB/s]
Downloading best_ckpt_mp0.tar: 30%|███ | 2.51G/8.31G [00:51<01:47, 58.0MB/s]
Downloading best_ckpt_mp0.tar: 30%|███ | 2.52G/8.31G [00:51<01:39, 62.8MB/s]
Downloading best_ckpt_mp0.tar: 30%|███ | 2.53G/8.31G [00:51<01:37, 63.7MB/s]
Downloading best_ckpt_mp0.tar: 31%|███ | 2.54G/8.31G [00:52<01:36, 64.3MB/s]
Downloading best_ckpt_mp0.tar: 31%|███ | 2.55G/8.31G [00:52<01:33, 65.9MB/s]
Downloading best_ckpt_mp0.tar: 31%|███ | 2.56G/8.31G [00:52<01:37, 63.2MB/s]
Downloading best_ckpt_mp0.tar: 31%|███ | 2.57G/8.31G [00:52<02:04, 49.3MB/s]
Downloading best_ckpt_mp0.tar: 31%|███ | 2.58G/8.31G [00:52<01:56, 52.6MB/s]
Downloading best_ckpt_mp0.tar: 31%|███ | 2.59G/8.31G [00:53<01:53, 54.1MB/s]
Downloading best_ckpt_mp0.tar: 31%|███▏ | 2.60G/8.31G [00:53<01:45, 58.2MB/s]
Downloading best_ckpt_mp0.tar: 31%|███▏ | 2.61G/8.31G [00:53<01:42, 59.5MB/s]
Downloading best_ckpt_mp0.tar: 31%|███▏ | 2.62G/8.31G [00:53<01:41, 60.5MB/s]
Downloading best_ckpt_mp0.tar: 32%|███▏ | 2.63G/8.31G [00:53<02:08, 47.3MB/s]
Downloading best_ckpt_mp0.tar: 32%|███▏ | 2.64G/8.31G [00:54<02:06, 48.2MB/s]
Downloading best_ckpt_mp0.tar: 32%|███▏ | 2.65G/8.31G [00:54<01:56, 52.2MB/s]
Downloading best_ckpt_mp0.tar: 32%|███▏ | 2.66G/8.31G [00:54<01:49, 55.3MB/s]
Downloading best_ckpt_mp0.tar: 32%|███▏ | 2.67G/8.31G [00:54<01:42, 58.9MB/s]
Downloading best_ckpt_mp0.tar: 32%|███▏ | 2.68G/8.31G [00:54<01:37, 61.9MB/s]
Downloading best_ckpt_mp0.tar: 32%|███▏ | 2.69G/8.31G [00:55<01:50, 54.6MB/s]
Downloading best_ckpt_mp0.tar: 32%|███▏ | 2.70G/8.31G [00:55<01:43, 58.2MB/s]
Downloading best_ckpt_mp0.tar: 33%|███▎ | 2.71G/8.31G [00:55<01:49, 55.1MB/s]
Downloading best_ckpt_mp0.tar: 33%|███▎ | 2.71G/8.31G [00:55<01:57, 51.3MB/s]
Downloading best_ckpt_mp0.tar: 33%|███▎ | 2.72G/8.31G [00:55<01:53, 52.6MB/s]
Downloading best_ckpt_mp0.tar: 33%|███▎ | 2.73G/8.31G [00:55<01:45, 56.5MB/s]
Downloading best_ckpt_mp0.tar: 33%|███▎ | 2.74G/8.31G [00:56<01:34, 63.4MB/s]
Downloading best_ckpt_mp0.tar: 33%|███▎ | 2.75G/8.31G [00:56<01:48, 55.2MB/s]
Downloading best_ckpt_mp0.tar: 33%|███▎ | 2.76G/8.31G [00:56<01:37, 60.9MB/s]
Downloading best_ckpt_mp0.tar: 33%|███▎ | 2.77G/8.31G [00:56<01:32, 64.0MB/s]
Downloading best_ckpt_mp0.tar: 33%|███▎ | 2.78G/8.31G [00:56<01:28, 66.8MB/s]
Downloading best_ckpt_mp0.tar: 34%|███▎ | 2.79G/8.31G [00:56<01:31, 64.5MB/s]
Downloading best_ckpt_mp0.tar: 34%|███▎ | 2.80G/8.31G [00:57<01:30, 65.6MB/s]
Downloading best_ckpt_mp0.tar: 34%|███▍ | 2.81G/8.31G [00:57<01:56, 50.8MB/s]
Downloading best_ckpt_mp0.tar: 34%|███▍ | 2.82G/8.31G [00:57<01:44, 56.2MB/s]
Downloading best_ckpt_mp0.tar: 34%|███▍ | 2.83G/8.31G [00:57<01:38, 59.9MB/s]
Downloading best_ckpt_mp0.tar: 34%|███▍ | 2.84G/8.31G [00:57<01:39, 59.1MB/s]
Downloading best_ckpt_mp0.tar: 34%|███▍ | 2.85G/8.31G [00:57<01:32, 63.6MB/s]
Downloading best_ckpt_mp0.tar: 34%|███▍ | 2.86G/8.31G [00:58<01:34, 62.1MB/s]
Downloading best_ckpt_mp0.tar: 35%|███▍ | 2.87G/8.31G [00:58<01:35, 60.9MB/s]
Downloading best_ckpt_mp0.tar: 35%|███▍ | 2.88G/8.31G [00:58<01:54, 50.9MB/s]
Downloading best_ckpt_mp0.tar: 35%|███▍ | 2.89G/8.31G [00:58<01:51, 52.2MB/s]
Downloading best_ckpt_mp0.tar: 35%|███▍ | 2.90G/8.31G [00:59<02:20, 41.3MB/s]
Downloading best_ckpt_mp0.tar: 35%|███▌ | 2.92G/8.31G [00:59<01:37, 59.4MB/s]
Downloading best_ckpt_mp0.tar: 35%|███▌ | 2.93G/8.31G [00:59<01:28, 64.9MB/s]
Downloading best_ckpt_mp0.tar: 35%|███▌ | 2.94G/8.31G [00:59<01:55, 50.1MB/s]
Downloading best_ckpt_mp0.tar: 35%|███▌ | 2.95G/8.31G [00:59<01:46, 54.3MB/s]
Downloading best_ckpt_mp0.tar: 36%|███▌ | 2.96G/8.31G [01:00<01:49, 52.7MB/s]
Downloading best_ckpt_mp0.tar: 36%|███▌ | 2.97G/8.31G [01:00<01:47, 53.2MB/s]
Downloading best_ckpt_mp0.tar: 36%|███▌ | 2.98G/8.31G [01:00<01:37, 58.9MB/s]
Downloading best_ckpt_mp0.tar: 36%|███▌ | 2.99G/8.31G [01:00<01:40, 57.0MB/s]
Downloading best_ckpt_mp0.tar: 36%|███▌ | 3.00G/8.31G [01:01<01:58, 48.2MB/s]
Downloading best_ckpt_mp0.tar: 36%|███▌ | 3.01G/8.31G [01:01<01:56, 48.9MB/s]
Downloading best_ckpt_mp0.tar: 36%|███▋ | 3.02G/8.31G [01:01<01:48, 52.2MB/s]
Downloading best_ckpt_mp0.tar: 36%|███▋ | 3.03G/8.31G [01:01<01:51, 50.9MB/s]
Downloading best_ckpt_mp0.tar: 37%|███▋ | 3.04G/8.31G [01:01<01:49, 51.9MB/s]
Downloading best_ckpt_mp0.tar: 37%|███▋ | 3.05G/8.31G [01:01<01:34, 59.6MB/s]
Downloading best_ckpt_mp0.tar: 37%|███▋ | 3.06G/8.31G [01:02<01:39, 56.8MB/s]
Downloading best_ckpt_mp0.tar: 37%|███▋ | 3.07G/8.31G [01:02<01:56, 48.2MB/s]
Downloading best_ckpt_mp0.tar: 37%|███▋ | 3.08G/8.31G [01:02<01:44, 53.7MB/s]
Downloading best_ckpt_mp0.tar: 37%|███▋ | 3.09G/8.31G [01:02<01:48, 51.5MB/s]
Downloading best_ckpt_mp0.tar: 37%|███▋ | 3.10G/8.31G [01:03<02:22, 39.2MB/s]
Downloading best_ckpt_mp0.tar: 37%|███▋ | 3.11G/8.31G [01:03<02:00, 46.2MB/s]
Downloading best_ckpt_mp0.tar: 37%|███▋ | 3.12G/8.31G [01:03<02:01, 45.9MB/s]
Downloading best_ckpt_mp0.tar: 38%|███▊ | 3.12G/8.31G [01:03<01:53, 48.8MB/s]
Downloading best_ckpt_mp0.tar: 38%|███▊ | 3.13G/8.31G [01:03<01:42, 54.2MB/s]
Downloading best_ckpt_mp0.tar: 38%|███▊ | 3.14G/8.31G [01:04<01:35, 58.1MB/s]
Downloading best_ckpt_mp0.tar: 38%|███▊ | 3.16G/8.31G [01:04<01:15, 73.2MB/s]
Downloading best_ckpt_mp0.tar: 38%|███▊ | 3.17G/8.31G [01:04<01:18, 70.2MB/s]
Downloading best_ckpt_mp0.tar: 38%|███▊ | 3.18G/8.31G [01:04<01:17, 71.2MB/s]
Downloading best_ckpt_mp0.tar: 38%|███▊ | 3.19G/8.31G [01:04<01:36, 57.1MB/s]
Downloading best_ckpt_mp0.tar: 39%|███▊ | 3.20G/8.31G [01:04<01:29, 61.1MB/s]
Downloading best_ckpt_mp0.tar: 39%|███▊ | 3.21G/8.31G [01:05<01:32, 59.3MB/s]
Downloading best_ckpt_mp0.tar: 39%|███▉ | 3.22G/8.31G [01:05<01:36, 56.5MB/s]
Downloading best_ckpt_mp0.tar: 39%|███▉ | 3.23G/8.31G [01:05<01:31, 59.3MB/s]
Downloading best_ckpt_mp0.tar: 39%|███▉ | 3.24G/8.31G [01:05<01:32, 58.8MB/s]
Downloading best_ckpt_mp0.tar: 39%|███▉ | 3.25G/8.31G [01:06<01:55, 47.0MB/s]
Downloading best_ckpt_mp0.tar: 39%|███▉ | 3.26G/8.31G [01:06<01:48, 50.1MB/s]
Downloading best_ckpt_mp0.tar: 39%|███▉ | 3.27G/8.31G [01:06<01:53, 47.9MB/s]
Downloading best_ckpt_mp0.tar: 39%|███▉ | 3.28G/8.31G [01:06<01:47, 50.2MB/s]
Downloading best_ckpt_mp0.tar: 40%|███▉ | 3.29G/8.31G [01:06<01:49, 49.2MB/s]
Downloading best_ckpt_mp0.tar: 40%|███▉ | 3.30G/8.31G [01:07<01:47, 49.9MB/s]
Downloading best_ckpt_mp0.tar: 40%|███▉ | 3.31G/8.31G [01:07<02:01, 44.3MB/s]
Downloading best_ckpt_mp0.tar: 40%|███▉ | 3.32G/8.31G [01:07<01:48, 49.4MB/s]
Downloading best_ckpt_mp0.tar: 40%|████ | 3.33G/8.31G [01:07<01:42, 52.4MB/s]
Downloading best_ckpt_mp0.tar: 40%|████ | 3.34G/8.31G [01:07<01:41, 52.8MB/s]
Downloading best_ckpt_mp0.tar: 40%|████ | 3.35G/8.31G [01:08<01:42, 51.9MB/s]
Downloading best_ckpt_mp0.tar: 40%|████ | 3.36G/8.31G [01:08<01:36, 55.3MB/s]
Downloading best_ckpt_mp0.tar: 41%|████ | 3.37G/8.31G [01:08<01:30, 58.8MB/s]
Downloading best_ckpt_mp0.tar: 41%|████ | 3.38G/8.31G [01:08<01:47, 49.1MB/s]
Downloading best_ckpt_mp0.tar: 41%|████ | 3.39G/8.31G [01:08<01:50, 48.0MB/s]
Downloading best_ckpt_mp0.tar: 41%|████ | 3.40G/8.31G [01:09<01:46, 49.6MB/s]
Downloading best_ckpt_mp0.tar: 41%|████ | 3.41G/8.31G [01:09<01:43, 51.0MB/s]
Downloading best_ckpt_mp0.tar: 41%|████ | 3.42G/8.31G [01:09<01:39, 52.8MB/s]
Downloading best_ckpt_mp0.tar: 41%|████▏ | 3.43G/8.31G [01:09<01:30, 57.7MB/s]
Downloading best_ckpt_mp0.tar: 41%|████▏ | 3.44G/8.31G [01:09<01:48, 48.2MB/s]
Downloading best_ckpt_mp0.tar: 41%|████▏ | 3.45G/8.31G [01:10<01:40, 52.0MB/s]
Downloading best_ckpt_mp0.tar: 42%|████▏ | 3.46G/8.31G [01:10<01:31, 57.2MB/s]
Downloading best_ckpt_mp0.tar: 42%|████▏ | 3.47G/8.31G [01:10<01:26, 60.2MB/s]
Downloading best_ckpt_mp0.tar: 42%|████▏ | 3.48G/8.31G [01:10<01:30, 57.6MB/s]
Downloading best_ckpt_mp0.tar: 42%|████▏ | 3.49G/8.31G [01:10<01:23, 62.0MB/s]
Downloading best_ckpt_mp0.tar: 42%|████▏ | 3.50G/8.31G [01:10<01:26, 59.7MB/s]
Downloading best_ckpt_mp0.tar: 42%|████▏ | 3.51G/8.31G [01:11<01:34, 54.4MB/s]
Downloading best_ckpt_mp0.tar: 42%|████▏ | 3.52G/8.31G [01:11<01:30, 57.1MB/s]
Downloading best_ckpt_mp0.tar: 42%|████▏ | 3.53G/8.31G [01:11<01:38, 52.2MB/s]
Downloading best_ckpt_mp0.tar: 43%|████▎ | 3.54G/8.31G [01:11<01:30, 56.6MB/s]
Downloading best_ckpt_mp0.tar: 43%|████▎ | 3.54G/8.31G [01:11<01:33, 54.7MB/s]
Downloading best_ckpt_mp0.tar: 43%|████▎ | 3.55G/8.31G [01:12<01:27, 58.6MB/s]
Downloading best_ckpt_mp0.tar: 43%|████▎ | 3.56G/8.31G [01:12<01:52, 45.2MB/s]
Downloading best_ckpt_mp0.tar: 43%|████▎ | 3.57G/8.31G [01:12<01:42, 49.8MB/s]
Downloading best_ckpt_mp0.tar: 43%|████▎ | 3.58G/8.31G [01:12<01:35, 53.1MB/s]
Downloading best_ckpt_mp0.tar: 43%|████▎ | 3.59G/8.31G [01:12<01:30, 56.1MB/s]
Downloading best_ckpt_mp0.tar: 43%|████▎ | 3.60G/8.31G [01:13<01:39, 50.7MB/s]
Downloading best_ckpt_mp0.tar: 43%|████▎ | 3.61G/8.31G [01:13<01:36, 52.5MB/s]
Downloading best_ckpt_mp0.tar: 44%|████▎ | 3.62G/8.31G [01:13<01:46, 47.4MB/s]
Downloading best_ckpt_mp0.tar: 44%|████▎ | 3.63G/8.31G [01:13<01:38, 50.8MB/s]
Downloading best_ckpt_mp0.tar: 44%|████▍ | 3.64G/8.31G [01:13<01:34, 53.1MB/s]
Downloading best_ckpt_mp0.tar: 44%|████▍ | 3.65G/8.31G [01:14<01:31, 54.6MB/s]
Downloading best_ckpt_mp0.tar: 44%|████▍ | 3.66G/8.31G [01:14<01:36, 51.6MB/s]
Downloading best_ckpt_mp0.tar: 44%|████▍ | 3.67G/8.31G [01:14<01:28, 56.3MB/s]
Downloading best_ckpt_mp0.tar: 44%|████▍ | 3.68G/8.31G [01:14<01:22, 60.1MB/s]
Downloading best_ckpt_mp0.tar: 44%|████▍ | 3.69G/8.31G [01:15<02:00, 41.0MB/s]
Downloading best_ckpt_mp0.tar: 45%|████▍ | 3.70G/8.31G [01:15<01:52, 43.9MB/s]
Downloading best_ckpt_mp0.tar: 45%|████▍ | 3.71G/8.31G [01:15<02:01, 40.6MB/s]
Downloading best_ckpt_mp0.tar: 45%|████▍ | 3.72G/8.31G [01:15<01:53, 43.3MB/s]
Downloading best_ckpt_mp0.tar: 45%|████▍ | 3.73G/8.31G [01:16<01:50, 44.6MB/s]
Downloading best_ckpt_mp0.tar: 45%|████▌ | 3.74G/8.31G [01:16<01:42, 47.7MB/s]
Downloading best_ckpt_mp0.tar: 45%|████▌ | 3.75G/8.31G [01:16<01:56, 41.9MB/s]
Downloading best_ckpt_mp0.tar: 45%|████▌ | 3.76G/8.31G [01:16<01:55, 42.2MB/s]
Downloading best_ckpt_mp0.tar: 45%|████▌ | 3.77G/8.31G [01:17<02:15, 35.9MB/s]
Downloading best_ckpt_mp0.tar: 45%|████▌ | 3.78G/8.31G [01:17<02:13, 36.5MB/s]
Downloading best_ckpt_mp0.tar: 46%|████▌ | 3.79G/8.31G [01:17<01:59, 40.7MB/s]
Downloading best_ckpt_mp0.tar: 46%|████▌ | 3.80G/8.31G [01:17<01:48, 44.7MB/s]
Downloading best_ckpt_mp0.tar: 46%|████▌ | 3.81G/8.31G [01:18<01:40, 48.1MB/s]
Downloading best_ckpt_mp0.tar: 46%|████▌ | 3.82G/8.31G [01:18<02:03, 38.9MB/s]
Downloading best_ckpt_mp0.tar: 46%|████▌ | 3.83G/8.31G [01:18<02:01, 39.7MB/s]
Downloading best_ckpt_mp0.tar: 46%|████▌ | 3.84G/8.31G [01:18<01:54, 41.8MB/s]
Downloading best_ckpt_mp0.tar: 46%|████▋ | 3.85G/8.31G [01:19<01:52, 42.8MB/s]
Downloading best_ckpt_mp0.tar: 46%|████▋ | 3.86G/8.31G [01:19<01:56, 41.1MB/s]
Downloading best_ckpt_mp0.tar: 47%|████▋ | 3.87G/8.31G [01:19<01:44, 45.4MB/s]
Downloading best_ckpt_mp0.tar: 47%|████▋ | 3.88G/8.31G [01:19<02:05, 37.8MB/s]
Downloading best_ckpt_mp0.tar: 47%|████▋ | 3.89G/8.31G [01:20<01:53, 41.9MB/s]
Downloading best_ckpt_mp0.tar: 47%|████▋ | 3.90G/8.31G [01:20<01:47, 44.3MB/s]
Downloading best_ckpt_mp0.tar: 47%|████▋ | 3.91G/8.31G [01:20<01:42, 46.3MB/s]
Downloading best_ckpt_mp0.tar: 47%|████▋ | 3.92G/8.31G [01:20<01:53, 41.7MB/s]
Downloading best_ckpt_mp0.tar: 47%|████▋ | 3.93G/8.31G [01:21<01:54, 41.1MB/s]
Downloading best_ckpt_mp0.tar: 47%|████▋ | 3.94G/8.31G [01:21<02:06, 37.0MB/s]
Downloading best_ckpt_mp0.tar: 47%|████▋ | 3.95G/8.31G [01:21<02:09, 36.2MB/s]
Downloading best_ckpt_mp0.tar: 48%|████▊ | 3.96G/8.31G [01:22<02:02, 38.2MB/s]
Downloading best_ckpt_mp0.tar: 48%|████▊ | 3.96G/8.31G [01:22<01:53, 41.1MB/s]
Downloading best_ckpt_mp0.tar: 48%|████▊ | 3.97G/8.31G [01:22<02:07, 36.5MB/s]
Downloading best_ckpt_mp0.tar: 48%|████▊ | 3.98G/8.31G [01:22<02:01, 38.2MB/s]
Downloading best_ckpt_mp0.tar: 48%|████▊ | 3.99G/8.31G [01:23<01:57, 39.4MB/s]
Downloading best_ckpt_mp0.tar: 48%|████▊ | 4.00G/8.31G [01:23<02:15, 34.2MB/s]
Downloading best_ckpt_mp0.tar: 48%|████▊ | 4.01G/8.31G [01:23<01:55, 40.0MB/s]
Downloading best_ckpt_mp0.tar: 48%|████▊ | 4.02G/8.31G [01:23<01:51, 41.3MB/s]
Downloading best_ckpt_mp0.tar: 49%|████▊ | 4.03G/8.31G [01:24<01:43, 44.4MB/s]
Downloading best_ckpt_mp0.tar: 49%|████▊ | 4.04G/8.31G [01:24<01:33, 48.9MB/s]
Downloading best_ckpt_mp0.tar: 49%|████▉ | 4.05G/8.31G [01:24<01:29, 50.9MB/s]
Downloading best_ckpt_mp0.tar: 49%|████▉ | 4.06G/8.31G [01:24<01:38, 46.1MB/s]
Downloading best_ckpt_mp0.tar: 49%|████▉ | 4.07G/8.31G [01:24<01:39, 45.9MB/s]
Downloading best_ckpt_mp0.tar: 49%|████▉ | 4.08G/8.31G [01:25<01:33, 48.4MB/s]
Downloading best_ckpt_mp0.tar: 49%|████▉ | 4.09G/8.31G [01:25<01:32, 49.1MB/s]
Downloading best_ckpt_mp0.tar: 49%|████▉ | 4.10G/8.31G [01:25<01:38, 45.7MB/s]
Downloading best_ckpt_mp0.tar: 49%|████▉ | 4.11G/8.31G [01:25<01:43, 43.7MB/s]
Downloading best_ckpt_mp0.tar: 50%|████▉ | 4.12G/8.31G [01:26<01:45, 42.5MB/s]
Downloading best_ckpt_mp0.tar: 50%|████▉ | 4.13G/8.31G [01:26<02:13, 33.6MB/s]
Downloading best_ckpt_mp0.tar: 50%|████▉ | 4.14G/8.31G [01:26<02:04, 35.9MB/s]
Downloading best_ckpt_mp0.tar: 50%|████▉ | 4.15G/8.31G [01:27<01:58, 37.8MB/s]
Downloading best_ckpt_mp0.tar: 50%|█████ | 4.16G/8.31G [01:27<01:57, 38.1MB/s]
Downloading best_ckpt_mp0.tar: 50%|█████ | 4.17G/8.31G [01:27<01:47, 41.5MB/s]
Downloading best_ckpt_mp0.tar: 50%|█████ | 4.18G/8.31G [01:27<01:43, 43.0MB/s]
Downloading best_ckpt_mp0.tar: 50%|█████ | 4.19G/8.31G [01:28<01:59, 37.0MB/s]
Downloading best_ckpt_mp0.tar: 51%|█████ | 4.20G/8.31G [01:28<01:52, 39.3MB/s]
Downloading best_ckpt_mp0.tar: 51%|█████ | 4.21G/8.31G [01:28<01:45, 41.9MB/s]
Downloading best_ckpt_mp0.tar: 51%|█████ | 4.22G/8.31G [01:28<01:41, 43.5MB/s]
Downloading best_ckpt_mp0.tar: 51%|█████ | 4.23G/8.31G [01:29<01:36, 45.5MB/s]
Downloading best_ckpt_mp0.tar: 51%|█████ | 4.24G/8.31G [01:29<01:33, 46.8MB/s]
Downloading best_ckpt_mp0.tar: 51%|█████ | 4.25G/8.31G [01:29<01:46, 41.0MB/s]
Downloading best_ckpt_mp0.tar: 51%|█████ | 4.26G/8.31G [01:29<01:41, 42.9MB/s]
Downloading best_ckpt_mp0.tar: 51%|█████▏ | 4.27G/8.31G [01:29<01:35, 45.5MB/s]
Downloading best_ckpt_mp0.tar: 51%|█████▏ | 4.28G/8.31G [01:30<01:39, 43.5MB/s]
Downloading best_ckpt_mp0.tar: 52%|█████▏ | 4.29G/8.31G [01:30<01:48, 39.9MB/s]
Downloading best_ckpt_mp0.tar: 52%|█████▏ | 4.30G/8.31G [01:30<01:47, 40.1MB/s]
Downloading best_ckpt_mp0.tar: 52%|█████▏ | 4.31G/8.31G [01:31<01:42, 42.0MB/s]
Downloading best_ckpt_mp0.tar: 52%|█████▏ | 4.32G/8.31G [01:31<02:02, 35.1MB/s]
Downloading best_ckpt_mp0.tar: 52%|█████▏ | 4.33G/8.31G [01:31<01:51, 38.5MB/s]
Downloading best_ckpt_mp0.tar: 52%|█████▏ | 4.34G/8.31G [01:31<01:43, 41.2MB/s]
Downloading best_ckpt_mp0.tar: 52%|█████▏ | 4.35G/8.31G [01:32<01:51, 38.1MB/s]
Downloading best_ckpt_mp0.tar: 52%|█████▏ | 4.36G/8.31G [01:32<01:54, 37.1MB/s]
Downloading best_ckpt_mp0.tar: 53%|█████▎ | 4.37G/8.31G [01:32<01:56, 36.4MB/s]
Downloading best_ckpt_mp0.tar: 53%|█████▎ | 4.38G/8.31G [01:33<02:25, 29.0MB/s]
Downloading best_ckpt_mp0.tar: 53%|█████▎ | 4.38G/8.31G [01:33<02:12, 31.8MB/s]
Downloading best_ckpt_mp0.tar: 53%|█████▎ | 4.39G/8.31G [01:33<02:00, 34.8MB/s]
Downloading best_ckpt_mp0.tar: 53%|█████▎ | 4.40G/8.31G [01:34<01:53, 37.0MB/s]
Downloading best_ckpt_mp0.tar: 53%|█████▎ | 4.41G/8.31G [01:34<01:48, 38.4MB/s]
Downloading best_ckpt_mp0.tar: 53%|█████▎ | 4.42G/8.31G [01:34<01:40, 41.4MB/s]
Downloading best_ckpt_mp0.tar: 53%|█████▎ | 4.43G/8.31G [01:34<01:44, 39.6MB/s]
Downloading best_ckpt_mp0.tar: 53%|█████▎ | 4.44G/8.31G [01:35<01:53, 36.7MB/s]
Downloading best_ckpt_mp0.tar: 54%|█████▎ | 4.45G/8.31G [01:35<01:48, 38.2MB/s]
Downloading best_ckpt_mp0.tar: 54%|█████▎ | 4.46G/8.31G [01:35<01:42, 40.2MB/s]
Downloading best_ckpt_mp0.tar: 54%|█████▍ | 4.47G/8.31G [01:35<01:29, 46.2MB/s]
Downloading best_ckpt_mp0.tar: 54%|█████▍ | 4.48G/8.31G [01:35<01:22, 49.7MB/s]
Downloading best_ckpt_mp0.tar: 54%|█████▍ | 4.49G/8.31G [01:36<01:16, 53.5MB/s]
Downloading best_ckpt_mp0.tar: 54%|█████▍ | 4.50G/8.31G [01:36<01:32, 44.0MB/s]
Downloading best_ckpt_mp0.tar: 54%|█████▍ | 4.51G/8.31G [01:36<01:29, 45.6MB/s]
Downloading best_ckpt_mp0.tar: 54%|█████▍ | 4.52G/8.31G [01:36<01:30, 45.2MB/s]
Downloading best_ckpt_mp0.tar: 55%|█████▍ | 4.53G/8.31G [01:37<01:23, 48.8MB/s]
Downloading best_ckpt_mp0.tar: 55%|█████▍ | 4.54G/8.31G [01:37<01:25, 47.2MB/s]
Downloading best_ckpt_mp0.tar: 55%|█████▍ | 4.55G/8.31G [01:37<01:23, 48.2MB/s]
Downloading best_ckpt_mp0.tar: 55%|█████▍ | 4.56G/8.31G [01:37<01:42, 39.1MB/s]
Downloading best_ckpt_mp0.tar: 55%|█████▌ | 4.57G/8.31G [01:38<01:32, 43.2MB/s]
Downloading best_ckpt_mp0.tar: 55%|█████▌ | 4.58G/8.31G [01:38<01:28, 45.4MB/s]
Downloading best_ckpt_mp0.tar: 55%|█████▌ | 4.59G/8.31G [01:38<01:27, 45.9MB/s]
Downloading best_ckpt_mp0.tar: 55%|█████▌ | 4.60G/8.31G [01:38<01:22, 48.1MB/s]
Downloading best_ckpt_mp0.tar: 55%|█████▌ | 4.61G/8.31G [01:38<01:22, 48.4MB/s]
Downloading best_ckpt_mp0.tar: 56%|█████▌ | 4.62G/8.31G [01:39<01:30, 43.8MB/s]
Downloading best_ckpt_mp0.tar: 56%|█████▌ | 4.63G/8.31G [01:39<01:40, 39.5MB/s]
Downloading best_ckpt_mp0.tar: 56%|█████▌ | 4.64G/8.31G [01:39<01:29, 43.8MB/s]
Downloading best_ckpt_mp0.tar: 56%|█████▌ | 4.65G/8.31G [01:40<01:39, 39.6MB/s]
Downloading best_ckpt_mp0.tar: 56%|█████▌ | 4.66G/8.31G [01:40<01:33, 41.9MB/s]
Downloading best_ckpt_mp0.tar: 56%|█████▌ | 4.67G/8.31G [01:40<01:29, 43.5MB/s]
Downloading best_ckpt_mp0.tar: 56%|█████▋ | 4.68G/8.31G [01:40<01:27, 44.4MB/s]
Downloading best_ckpt_mp0.tar: 56%|█████▋ | 4.69G/8.31G [01:41<01:47, 36.2MB/s]
Downloading best_ckpt_mp0.tar: 57%|█████▋ | 4.70G/8.31G [01:41<01:33, 41.4MB/s]
Downloading best_ckpt_mp0.tar: 57%|█████▋ | 4.71G/8.31G [01:41<01:27, 44.3MB/s]
Downloading best_ckpt_mp0.tar: 57%|█████▋ | 4.72G/8.31G [01:41<01:21, 47.3MB/s]
Downloading best_ckpt_mp0.tar: 57%|█████▋ | 4.73G/8.31G [01:41<01:21, 47.3MB/s]
Downloading best_ckpt_mp0.tar: 57%|█████▋ | 4.74G/8.31G [01:42<01:25, 45.0MB/s]
Downloading best_ckpt_mp0.tar: 57%|█████▋ | 4.75G/8.31G [01:42<01:22, 46.3MB/s]
Downloading best_ckpt_mp0.tar: 57%|█████▋ | 4.76G/8.31G [01:42<01:26, 44.1MB/s]
Downloading best_ckpt_mp0.tar: 57%|█████▋ | 4.77G/8.31G [01:42<01:21, 46.5MB/s]
Downloading best_ckpt_mp0.tar: 57%|█████▋ | 4.78G/8.31G [01:42<01:16, 49.5MB/s]
Downloading best_ckpt_mp0.tar: 58%|█████▊ | 4.79G/8.31G [01:43<01:06, 57.2MB/s]
Downloading best_ckpt_mp0.tar: 58%|█████▊ | 4.79G/8.31G [01:43<00:57, 65.4MB/s]
Downloading best_ckpt_mp0.tar: 58%|█████▊ | 4.80G/8.31G [01:43<01:05, 57.4MB/s]
Downloading best_ckpt_mp0.tar: 58%|█████▊ | 4.81G/8.31G [01:43<01:26, 43.4MB/s]
Downloading best_ckpt_mp0.tar: 58%|█████▊ | 4.82G/8.31G [01:43<01:12, 51.6MB/s]
Downloading best_ckpt_mp0.tar: 58%|█████▊ | 4.83G/8.31G [01:44<01:35, 39.1MB/s]
Downloading best_ckpt_mp0.tar: 58%|█████▊ | 4.84G/8.31G [01:44<01:17, 48.0MB/s]
Downloading best_ckpt_mp0.tar: 58%|█████▊ | 4.85G/8.31G [01:44<01:07, 55.1MB/s]
Downloading best_ckpt_mp0.tar: 59%|█████▊ | 4.86G/8.31G [01:44<00:58, 62.9MB/s]
Downloading best_ckpt_mp0.tar: 59%|█████▊ | 4.87G/8.31G [01:44<00:52, 70.6MB/s]
Downloading best_ckpt_mp0.tar: 59%|█████▉ | 4.88G/8.31G [01:44<00:47, 77.7MB/s]
Downloading best_ckpt_mp0.tar: 59%|█████▉ | 4.89G/8.31G [01:45<00:45, 80.6MB/s]
Downloading best_ckpt_mp0.tar: 59%|█████▉ | 4.91G/8.31G [01:45<00:40, 90.3MB/s]
Downloading best_ckpt_mp0.tar: 59%|█████▉ | 4.92G/8.31G [01:45<00:39, 92.1MB/s]
Downloading best_ckpt_mp0.tar: 59%|█████▉ | 4.93G/8.31G [01:45<00:38, 93.7MB/s]
Downloading best_ckpt_mp0.tar: 59%|█████▉ | 4.94G/8.31G [01:45<00:37, 95.5MB/s]
Downloading best_ckpt_mp0.tar: 60%|█████▉ | 4.95G/8.31G [01:45<00:37, 96.7MB/s]
Downloading best_ckpt_mp0.tar: 60%|█████▉ | 4.96G/8.31G [01:45<00:36, 97.9MB/s]
Downloading best_ckpt_mp0.tar: 60%|█████▉ | 4.97G/8.31G [01:45<00:36, 98.4MB/s]
Downloading best_ckpt_mp0.tar: 60%|█████▉ | 4.98G/8.31G [01:45<00:36, 99.2MB/s]
Downloading best_ckpt_mp0.tar: 60%|██████ | 4.99G/8.31G [01:46<00:35, 99.4MB/s]
Downloading best_ckpt_mp0.tar: 60%|██████ | 5.00G/8.31G [01:46<00:36, 97.6MB/s]
Downloading best_ckpt_mp0.tar: 60%|██████ | 5.01G/8.31G [01:46<00:36, 97.5MB/s]
Downloading best_ckpt_mp0.tar: 60%|██████ | 5.02G/8.31G [01:46<00:37, 93.6MB/s]
Downloading best_ckpt_mp0.tar: 61%|██████ | 5.03G/8.31G [01:46<00:37, 94.7MB/s]
Downloading best_ckpt_mp0.tar: 61%|██████ | 5.04G/8.31G [01:46<00:37, 93.7MB/s]
Downloading best_ckpt_mp0.tar: 61%|██████ | 5.05G/8.31G [01:46<00:36, 94.6MB/s]
Downloading best_ckpt_mp0.tar: 61%|██████ | 5.06G/8.31G [01:46<00:36, 96.9MB/s]
Downloading best_ckpt_mp0.tar: 61%|██████ | 5.07G/8.31G [01:47<00:46, 74.8MB/s]
Downloading best_ckpt_mp0.tar: 61%|██████ | 5.08G/8.31G [01:47<00:50, 69.0MB/s]
Downloading best_ckpt_mp0.tar: 61%|██████ | 5.09G/8.31G [01:47<00:45, 76.3MB/s]
Downloading best_ckpt_mp0.tar: 61%|██████▏ | 5.10G/8.31G [01:47<00:59, 58.2MB/s]
Downloading best_ckpt_mp0.tar: 61%|██████▏ | 5.11G/8.31G [01:47<00:51, 66.3MB/s]
Downloading best_ckpt_mp0.tar: 62%|██████▏ | 5.12G/8.31G [01:47<00:56, 60.9MB/s]
Downloading best_ckpt_mp0.tar: 62%|██████▏ | 5.13G/8.31G [01:48<01:07, 50.4MB/s]
Downloading best_ckpt_mp0.tar: 62%|██████▏ | 5.15G/8.31G [01:48<00:49, 69.1MB/s]
Downloading best_ckpt_mp0.tar: 62%|██████▏ | 5.16G/8.31G [01:48<00:50, 67.0MB/s]
Downloading best_ckpt_mp0.tar: 62%|██████▏ | 5.17G/8.31G [01:48<00:48, 69.9MB/s]
Downloading best_ckpt_mp0.tar: 62%|██████▏ | 5.18G/8.31G [01:48<00:48, 70.0MB/s]
Downloading best_ckpt_mp0.tar: 62%|██████▏ | 5.19G/8.31G [01:49<00:58, 57.3MB/s]
Downloading best_ckpt_mp0.tar: 63%|██████▎ | 5.21G/8.31G [01:49<00:45, 74.0MB/s]
Downloading best_ckpt_mp0.tar: 63%|██████▎ | 5.22G/8.31G [01:49<00:38, 87.1MB/s]
Downloading best_ckpt_mp0.tar: 63%|██████▎ | 5.24G/8.31G [01:49<00:34, 96.4MB/s]
Downloading best_ckpt_mp0.tar: 63%|██████▎ | 5.25G/8.31G [01:49<00:37, 88.3MB/s]
Downloading best_ckpt_mp0.tar: 63%|██████▎ | 5.26G/8.31G [01:49<00:39, 83.6MB/s]
Downloading best_ckpt_mp0.tar: 64%|██████▎ | 5.28G/8.31G [01:50<00:49, 65.1MB/s]
Downloading best_ckpt_mp0.tar: 64%|██████▍ | 5.30G/8.31G [01:50<00:42, 75.6MB/s]
Downloading best_ckpt_mp0.tar: 64%|██████▍ | 5.31G/8.31G [01:50<00:48, 65.7MB/s]
Downloading best_ckpt_mp0.tar: 64%|██████▍ | 5.33G/8.31G [01:51<00:44, 71.9MB/s]
Downloading best_ckpt_mp0.tar: 64%|██████▍ | 5.35G/8.31G [01:51<00:38, 81.6MB/s]
Downloading best_ckpt_mp0.tar: 65%|██████▍ | 5.37G/8.31G [01:51<00:37, 84.6MB/s]
Downloading best_ckpt_mp0.tar: 65%|██████▍ | 5.39G/8.31G [01:51<00:36, 85.5MB/s]
Downloading best_ckpt_mp0.tar: 65%|██████▌ | 5.41G/8.31G [01:51<00:34, 91.3MB/s]
Downloading best_ckpt_mp0.tar: 65%|██████▌ | 5.43G/8.31G [01:52<00:37, 81.6MB/s]
Downloading best_ckpt_mp0.tar: 65%|██████▌ | 5.44G/8.31G [01:52<00:38, 79.8MB/s]
Downloading best_ckpt_mp0.tar: 66%|██████▌ | 5.46G/8.31G [01:52<00:37, 82.7MB/s]
Downloading best_ckpt_mp0.tar: 66%|██████▌ | 5.47G/8.31G [01:52<00:37, 81.2MB/s]
Downloading best_ckpt_mp0.tar: 66%|██████▌ | 5.49G/8.31G [01:53<00:40, 75.3MB/s]
Downloading best_ckpt_mp0.tar: 66%|██████▌ | 5.50G/8.31G [01:53<00:39, 76.0MB/s]
Downloading best_ckpt_mp0.tar: 66%|██████▋ | 5.52G/8.31G [01:53<00:36, 81.1MB/s]
Downloading best_ckpt_mp0.tar: 67%|██████▋ | 5.54G/8.31G [01:53<00:36, 81.5MB/s]
Downloading best_ckpt_mp0.tar: 67%|██████▋ | 5.56G/8.31G [01:53<00:32, 91.3MB/s]
Downloading best_ckpt_mp0.tar: 67%|██████▋ | 5.57G/8.31G [01:54<00:35, 82.7MB/s]
Downloading best_ckpt_mp0.tar: 67%|██████▋ | 5.59G/8.31G [01:54<00:31, 92.1MB/s]
Downloading best_ckpt_mp0.tar: 67%|██████▋ | 5.60G/8.31G [01:54<00:31, 93.0MB/s]
Downloading best_ckpt_mp0.tar: 68%|██████▊ | 5.62G/8.31G [01:54<00:28, 103MB/s]
Downloading best_ckpt_mp0.tar: 68%|██████▊ | 5.62G/8.31G [01:54<00:31, 90.4MB/s]
Downloading best_ckpt_mp0.tar: 68%|██████▊ | 5.63G/8.31G [01:54<00:31, 90.0MB/s]
Downloading best_ckpt_mp0.tar: 68%|██████▊ | 5.64G/8.31G [01:55<00:39, 72.7MB/s]
Downloading best_ckpt_mp0.tar: 68%|██████▊ | 5.65G/8.31G [01:55<00:43, 64.8MB/s]
Downloading best_ckpt_mp0.tar: 68%|██████▊ | 5.66G/8.31G [01:55<00:58, 48.7MB/s]
Downloading best_ckpt_mp0.tar: 68%|██████▊ | 5.67G/8.31G [01:55<00:53, 52.8MB/s]
Downloading best_ckpt_mp0.tar: 68%|██████▊ | 5.68G/8.31G [01:56<01:03, 44.3MB/s]
Downloading best_ckpt_mp0.tar: 69%|██████▊ | 5.69G/8.31G [01:56<01:01, 45.6MB/s]
Downloading best_ckpt_mp0.tar: 69%|██████▊ | 5.70G/8.31G [01:56<01:03, 44.1MB/s]
Downloading best_ckpt_mp0.tar: 69%|██████▉ | 5.72G/8.31G [01:56<01:02, 44.6MB/s]
Downloading best_ckpt_mp0.tar: 69%|██████▉ | 5.74G/8.31G [01:57<00:49, 55.6MB/s]
Downloading best_ckpt_mp0.tar: 69%|██████▉ | 5.75G/8.31G [01:57<01:05, 41.8MB/s]
Downloading best_ckpt_mp0.tar: 69%|██████▉ | 5.77G/8.31G [01:58<00:56, 48.0MB/s]
Downloading best_ckpt_mp0.tar: 70%|██████▉ | 5.78G/8.31G [01:58<01:01, 44.2MB/s]
Downloading best_ckpt_mp0.tar: 70%|██████▉ | 5.80G/8.31G [01:58<00:52, 51.2MB/s]
Downloading best_ckpt_mp0.tar: 70%|██████▉ | 5.81G/8.31G [01:58<00:50, 53.4MB/s]
Downloading best_ckpt_mp0.tar: 70%|███████ | 5.83G/8.31G [01:59<00:40, 65.5MB/s]
Downloading best_ckpt_mp0.tar: 70%|███████ | 5.85G/8.31G [01:59<00:36, 72.8MB/s]
Downloading best_ckpt_mp0.tar: 71%|███████ | 5.87G/8.31G [01:59<00:35, 72.9MB/s]
Downloading best_ckpt_mp0.tar: 71%|███████ | 5.88G/8.31G [01:59<00:46, 55.7MB/s]
Downloading best_ckpt_mp0.tar: 71%|███████ | 5.89G/8.31G [02:00<00:47, 54.5MB/s]
Downloading best_ckpt_mp0.tar: 71%|███████ | 5.90G/8.31G [02:00<00:45, 56.3MB/s]
Downloading best_ckpt_mp0.tar: 71%|███████ | 5.91G/8.31G [02:00<00:50, 51.0MB/s]
Downloading best_ckpt_mp0.tar: 71%|███████ | 5.92G/8.31G [02:00<00:45, 56.2MB/s]
Downloading best_ckpt_mp0.tar: 71%|███████▏ | 5.93G/8.31G [02:00<00:45, 56.7MB/s]
Downloading best_ckpt_mp0.tar: 71%|███████▏ | 5.94G/8.31G [02:01<00:46, 54.3MB/s]
Downloading best_ckpt_mp0.tar: 72%|███████▏ | 5.96G/8.31G [02:01<00:41, 61.0MB/s]
Downloading best_ckpt_mp0.tar: 72%|███████▏ | 5.97G/8.31G [02:01<00:38, 66.2MB/s]
Downloading best_ckpt_mp0.tar: 72%|███████▏ | 5.98G/8.31G [02:01<00:40, 61.5MB/s]
Downloading best_ckpt_mp0.tar: 72%|███████▏ | 6.00G/8.31G [02:01<00:37, 66.0MB/s]
Downloading best_ckpt_mp0.tar: 72%|███████▏ | 6.02G/8.31G [02:02<00:31, 77.5MB/s]
Downloading best_ckpt_mp0.tar: 73%|███████▎ | 6.04G/8.31G [02:02<00:27, 87.5MB/s]
Downloading best_ckpt_mp0.tar: 73%|███████▎ | 6.05G/8.31G [02:02<00:26, 92.0MB/s]
Downloading best_ckpt_mp0.tar: 73%|███████▎ | 6.06G/8.31G [02:02<00:29, 81.9MB/s]
Downloading best_ckpt_mp0.tar: 73%|███████▎ | 6.08G/8.31G [02:02<00:27, 88.1MB/s]
Downloading best_ckpt_mp0.tar: 73%|███████▎ | 6.09G/8.31G [02:03<00:31, 76.3MB/s]
Downloading best_ckpt_mp0.tar: 73%|███████▎ | 6.10G/8.31G [02:03<00:29, 81.0MB/s]
Downloading best_ckpt_mp0.tar: 74%|███████▎ | 6.12G/8.31G [02:03<00:27, 83.9MB/s]
Downloading best_ckpt_mp0.tar: 74%|███████▍ | 6.14G/8.31G [02:03<00:26, 86.6MB/s]
Downloading best_ckpt_mp0.tar: 74%|███████▍ | 6.16G/8.31G [02:03<00:24, 94.5MB/s]
Downloading best_ckpt_mp0.tar: 74%|███████▍ | 6.18G/8.31G [02:04<00:22, 100MB/s]
Downloading best_ckpt_mp0.tar: 75%|███████▍ | 6.19G/8.31G [02:04<00:26, 85.6MB/s]
Downloading best_ckpt_mp0.tar: 75%|███████▍ | 6.21G/8.31G [02:04<00:23, 96.9MB/s]
Downloading best_ckpt_mp0.tar: 75%|███████▍ | 6.22G/8.31G [02:04<00:23, 96.5MB/s]
Downloading best_ckpt_mp0.tar: 75%|███████▌ | 6.24G/8.31G [02:04<00:26, 85.2MB/s]
Downloading best_ckpt_mp0.tar: 75%|███████▌ | 6.25G/8.31G [02:05<00:27, 80.2MB/s]
Downloading best_ckpt_mp0.tar: 75%|███████▌ | 6.27G/8.31G [02:05<00:23, 92.8MB/s]
Downloading best_ckpt_mp0.tar: 76%|███████▌ | 6.29G/8.31G [02:05<00:22, 98.1MB/s]
Downloading best_ckpt_mp0.tar: 76%|███████▌ | 6.30G/8.31G [02:05<00:23, 93.0MB/s]
Downloading best_ckpt_mp0.tar: 76%|███████▌ | 6.31G/8.31G [02:05<00:24, 86.5MB/s]
Downloading best_ckpt_mp0.tar: 76%|███████▌ | 6.33G/8.31G [02:05<00:21, 97.6MB/s]
Downloading best_ckpt_mp0.tar: 76%|███████▋ | 6.35G/8.31G [02:06<00:20, 103MB/s]
Downloading best_ckpt_mp0.tar: 77%|███████▋ | 6.37G/8.31G [02:06<00:20, 102MB/s]
Downloading best_ckpt_mp0.tar: 77%|███████▋ | 6.38G/8.31G [02:06<00:24, 86.4MB/s]
Downloading best_ckpt_mp0.tar: 77%|███████▋ | 6.40G/8.31G [02:06<00:22, 91.5MB/s]
Downloading best_ckpt_mp0.tar: 77%|███████▋ | 6.41G/8.31G [02:06<00:24, 82.6MB/s]
Downloading best_ckpt_mp0.tar: 77%|███████▋ | 6.42G/8.31G [02:07<00:32, 63.4MB/s]
Downloading best_ckpt_mp0.tar: 77%|███████▋ | 6.43G/8.31G [02:07<00:34, 57.9MB/s]
Downloading best_ckpt_mp0.tar: 77%|███████▋ | 6.44G/8.31G [02:07<00:34, 58.0MB/s]
Downloading best_ckpt_mp0.tar: 78%|███████▊ | 6.46G/8.31G [02:07<00:30, 65.9MB/s]
Downloading best_ckpt_mp0.tar: 78%|███████▊ | 6.47G/8.31G [02:07<00:26, 75.5MB/s]
Downloading best_ckpt_mp0.tar: 78%|███████▊ | 6.49G/8.31G [02:08<00:23, 84.6MB/s]
Downloading best_ckpt_mp0.tar: 78%|███████▊ | 6.50G/8.31G [02:08<00:26, 73.8MB/s]
Downloading best_ckpt_mp0.tar: 78%|███████▊ | 6.51G/8.31G [02:08<00:24, 78.7MB/s]
Downloading best_ckpt_mp0.tar: 79%|███████▊ | 6.52G/8.31G [02:08<00:24, 79.6MB/s]
Downloading best_ckpt_mp0.tar: 79%|███████▊ | 6.53G/8.31G [02:08<00:29, 64.8MB/s]
Downloading best_ckpt_mp0.tar: 79%|███████▉ | 6.55G/8.31G [02:09<00:23, 81.5MB/s]
Downloading best_ckpt_mp0.tar: 79%|███████▉ | 6.56G/8.31G [02:09<00:24, 75.5MB/s]
Downloading best_ckpt_mp0.tar: 79%|███████▉ | 6.58G/8.31G [02:09<00:22, 84.2MB/s]
Downloading best_ckpt_mp0.tar: 79%|███████▉ | 6.60G/8.31G [02:09<00:20, 88.3MB/s]
Downloading best_ckpt_mp0.tar: 80%|███████▉ | 6.62G/8.31G [02:09<00:20, 88.5MB/s]
Downloading best_ckpt_mp0.tar: 80%|███████▉ | 6.64G/8.31G [02:10<00:18, 95.3MB/s]
Downloading best_ckpt_mp0.tar: 80%|████████ | 6.66G/8.31G [02:10<00:18, 98.1MB/s]
Downloading best_ckpt_mp0.tar: 80%|████████ | 6.68G/8.31G [02:10<00:16, 104MB/s]
Downloading best_ckpt_mp0.tar: 81%|████████ | 6.70G/8.31G [02:10<00:18, 92.2MB/s]
Downloading best_ckpt_mp0.tar: 81%|████████ | 6.71G/8.31G [02:10<00:22, 77.1MB/s]
Downloading best_ckpt_mp0.tar: 81%|████████ | 6.72G/8.31G [02:11<00:21, 78.4MB/s]
Downloading best_ckpt_mp0.tar: 81%|████████ | 6.74G/8.31G [02:11<00:21, 78.4MB/s]
Downloading best_ckpt_mp0.tar: 81%|████████ | 6.75G/8.31G [02:11<00:22, 73.7MB/s]
Downloading best_ckpt_mp0.tar: 81%|████████▏ | 6.77G/8.31G [02:11<00:21, 78.3MB/s]
Downloading best_ckpt_mp0.tar: 82%|████████▏ | 6.79G/8.31G [02:12<00:23, 70.1MB/s]
Downloading best_ckpt_mp0.tar: 82%|████████▏ | 6.81G/8.31G [02:12<00:23, 68.5MB/s]
Downloading best_ckpt_mp0.tar: 82%|████████▏ | 6.82G/8.31G [02:12<00:29, 53.8MB/s]
Downloading best_ckpt_mp0.tar: 82%|████████▏ | 6.83G/8.31G [02:12<00:29, 54.6MB/s]
Downloading best_ckpt_mp0.tar: 82%|████████▏ | 6.85G/8.31G [02:13<00:25, 61.0MB/s]
Downloading best_ckpt_mp0.tar: 83%|████████▎ | 6.87G/8.31G [02:13<00:21, 71.0MB/s]
Downloading best_ckpt_mp0.tar: 83%|████████▎ | 6.88G/8.31G [02:13<00:21, 70.1MB/s]
Downloading best_ckpt_mp0.tar: 83%|████████▎ | 6.89G/8.31G [02:13<00:21, 72.3MB/s]
Downloading best_ckpt_mp0.tar: 83%|████████▎ | 6.91G/8.31G [02:14<00:17, 84.5MB/s]
Downloading best_ckpt_mp0.tar: 83%|████████▎ | 6.92G/8.31G [02:14<00:17, 84.2MB/s]
Downloading best_ckpt_mp0.tar: 83%|████████▎ | 6.93G/8.31G [02:14<00:17, 85.1MB/s]
Downloading best_ckpt_mp0.tar: 84%|████████▎ | 6.95G/8.31G [02:14<00:15, 96.4MB/s]
Downloading best_ckpt_mp0.tar: 84%|████████▍ | 6.97G/8.31G [02:14<00:13, 104MB/s]
Downloading best_ckpt_mp0.tar: 84%|████████▍ | 6.99G/8.31G [02:14<00:12, 109MB/s]
Downloading best_ckpt_mp0.tar: 84%|████████▍ | 7.01G/8.31G [02:15<00:14, 97.5MB/s]
Downloading best_ckpt_mp0.tar: 85%|████████▍ | 7.03G/8.31G [02:15<00:13, 104MB/s]
Downloading best_ckpt_mp0.tar: 85%|████████▍ | 7.05G/8.31G [02:15<00:12, 109MB/s]
Downloading best_ckpt_mp0.tar: 85%|████████▌ | 7.07G/8.31G [02:15<00:13, 96.6MB/s]
Downloading best_ckpt_mp0.tar: 85%|████████▌ | 7.08G/8.31G [02:15<00:13, 96.1MB/s]
Downloading best_ckpt_mp0.tar: 85%|████████▌ | 7.10G/8.31G [02:16<00:12, 100MB/s]
Downloading best_ckpt_mp0.tar: 86%|████████▌ | 7.12G/8.31G [02:16<00:12, 104MB/s]
Downloading best_ckpt_mp0.tar: 86%|████████▌ | 7.14G/8.31G [02:16<00:13, 94.9MB/s]
Downloading best_ckpt_mp0.tar: 86%|████████▌ | 7.16G/8.31G [02:16<00:12, 98.2MB/s]
Downloading best_ckpt_mp0.tar: 86%|████████▋ | 7.18G/8.31G [02:16<00:11, 103MB/s]
Downloading best_ckpt_mp0.tar: 87%|████████▋ | 7.20G/8.31G [02:17<00:14, 83.2MB/s]
Downloading best_ckpt_mp0.tar: 87%|████████▋ | 7.21G/8.31G [02:17<00:16, 70.1MB/s]
Downloading best_ckpt_mp0.tar: 87%|████████▋ | 7.23G/8.31G [02:17<00:16, 72.2MB/s]
Downloading best_ckpt_mp0.tar: 87%|████████▋ | 7.25G/8.31G [02:18<00:19, 58.4MB/s]
Downloading best_ckpt_mp0.tar: 87%|████████▋ | 7.27G/8.31G [02:18<00:16, 69.7MB/s]
Downloading best_ckpt_mp0.tar: 88%|████████▊ | 7.29G/8.31G [02:18<00:13, 79.2MB/s]
Downloading best_ckpt_mp0.tar: 88%|████████▊ | 7.30G/8.31G [02:18<00:12, 89.0MB/s]
Downloading best_ckpt_mp0.tar: 88%|████████▊ | 7.31G/8.31G [02:18<00:12, 82.6MB/s]
Downloading best_ckpt_mp0.tar: 88%|████████▊ | 7.33G/8.31G [02:19<00:11, 87.4MB/s]
Downloading best_ckpt_mp0.tar: 88%|████████▊ | 7.34G/8.31G [02:19<00:11, 88.5MB/s]
Downloading best_ckpt_mp0.tar: 89%|████████▊ | 7.36G/8.31G [02:19<00:10, 94.3MB/s]
Downloading best_ckpt_mp0.tar: 89%|████████▊ | 7.37G/8.31G [02:19<00:12, 81.8MB/s]
Downloading best_ckpt_mp0.tar: 89%|████████▉ | 7.39G/8.31G [02:19<00:10, 92.7MB/s]
Downloading best_ckpt_mp0.tar: 89%|████████▉ | 7.41G/8.31G [02:19<00:09, 100MB/s]
Downloading best_ckpt_mp0.tar: 89%|████████▉ | 7.42G/8.31G [02:20<00:09, 98.9MB/s]
Downloading best_ckpt_mp0.tar: 89%|████████▉ | 7.43G/8.31G [02:20<00:10, 88.6MB/s]
Downloading best_ckpt_mp0.tar: 90%|████████▉ | 7.44G/8.31G [02:20<00:11, 78.2MB/s]
Downloading best_ckpt_mp0.tar: 90%|████████▉ | 7.45G/8.31G [02:20<00:11, 80.6MB/s]
Downloading best_ckpt_mp0.tar: 90%|████████▉ | 7.47G/8.31G [02:20<00:11, 76.0MB/s]
Downloading best_ckpt_mp0.tar: 90%|█████████ | 7.48G/8.31G [02:21<00:15, 57.0MB/s]
Downloading best_ckpt_mp0.tar: 90%|█████████ | 7.50G/8.31G [02:21<00:11, 73.1MB/s]
Downloading best_ckpt_mp0.tar: 90%|█████████ | 7.52G/8.31G [02:21<00:09, 84.8MB/s]
Downloading best_ckpt_mp0.tar: 91%|█████████ | 7.54G/8.31G [02:21<00:08, 93.9MB/s]
Downloading best_ckpt_mp0.tar: 91%|█████████ | 7.56G/8.31G [02:21<00:08, 95.9MB/s]
Downloading best_ckpt_mp0.tar: 91%|█████████ | 7.58G/8.31G [02:22<00:07, 98.8MB/s]
Downloading best_ckpt_mp0.tar: 91%|█████████▏| 7.60G/8.31G [02:22<00:07, 101MB/s]
Downloading best_ckpt_mp0.tar: 92%|█████████▏| 7.61G/8.31G [02:22<00:08, 89.3MB/s]
Downloading best_ckpt_mp0.tar: 92%|█████████▏| 7.62G/8.31G [02:22<00:08, 86.4MB/s]
Downloading best_ckpt_mp0.tar: 92%|█████████▏| 7.63G/8.31G [02:22<00:10, 72.3MB/s]
Downloading best_ckpt_mp0.tar: 92%|█████████▏| 7.65G/8.31G [02:23<00:08, 81.3MB/s]
Downloading best_ckpt_mp0.tar: 92%|█████████▏| 7.66G/8.31G [02:23<00:09, 73.4MB/s]
Downloading best_ckpt_mp0.tar: 92%|█████████▏| 7.68G/8.31G [02:23<00:07, 85.9MB/s]
Downloading best_ckpt_mp0.tar: 92%|█████████▏| 7.69G/8.31G [02:23<00:08, 76.3MB/s]
Downloading best_ckpt_mp0.tar: 93%|█████████▎| 7.71G/8.31G [02:23<00:07, 90.0MB/s]
Downloading best_ckpt_mp0.tar: 93%|█████████▎| 7.72G/8.31G [02:24<00:06, 96.0MB/s]
Downloading best_ckpt_mp0.tar: 93%|█████████▎| 7.74G/8.31G [02:24<00:06, 100MB/s]
Downloading best_ckpt_mp0.tar: 93%|█████████▎| 7.75G/8.31G [02:24<00:06, 88.9MB/s]
Downloading best_ckpt_mp0.tar: 93%|█████████▎| 7.76G/8.31G [02:24<00:07, 77.2MB/s]
Downloading best_ckpt_mp0.tar: 94%|█████████▎| 7.78G/8.31G [02:24<00:06, 87.6MB/s]
Downloading best_ckpt_mp0.tar: 94%|█████████▍| 7.80G/8.31G [02:24<00:05, 96.9MB/s]
Downloading best_ckpt_mp0.tar: 94%|█████████▍| 7.82G/8.31G [02:25<00:05, 102MB/s]
Downloading best_ckpt_mp0.tar: 94%|█████████▍| 7.83G/8.31G [02:25<00:06, 80.8MB/s]
Downloading best_ckpt_mp0.tar: 94%|█████████▍| 7.85G/8.31G [02:25<00:05, 87.5MB/s]
Downloading best_ckpt_mp0.tar: 95%|█████████▍| 7.87G/8.31G [02:25<00:05, 85.0MB/s]
Downloading best_ckpt_mp0.tar: 95%|█████████▍| 7.89G/8.31G [02:26<00:05, 87.5MB/s]
Downloading best_ckpt_mp0.tar: 95%|█████████▌| 7.90G/8.31G [02:26<00:05, 84.3MB/s]
Downloading best_ckpt_mp0.tar: 95%|█████████▌| 7.91G/8.31G [02:26<00:06, 67.5MB/s]
Downloading best_ckpt_mp0.tar: 95%|█████████▌| 7.92G/8.31G [02:26<00:07, 59.1MB/s]
Downloading best_ckpt_mp0.tar: 95%|█████████▌| 7.93G/8.31G [02:26<00:07, 56.1MB/s]
Downloading best_ckpt_mp0.tar: 96%|█████████▌| 7.94G/8.31G [02:27<00:07, 53.4MB/s]
Downloading best_ckpt_mp0.tar: 96%|█████████▌| 7.95G/8.31G [02:27<00:07, 54.5MB/s]
Downloading best_ckpt_mp0.tar: 96%|█████████▌| 7.96G/8.31G [02:27<00:06, 57.9MB/s]
Downloading best_ckpt_mp0.tar: 96%|█████████▌| 7.97G/8.31G [02:27<00:05, 65.9MB/s]
Downloading best_ckpt_mp0.tar: 96%|█████████▌| 7.99G/8.31G [02:27<00:04, 84.1MB/s]
Downloading best_ckpt_mp0.tar: 96%|█████████▋| 8.00G/8.31G [02:27<00:03, 86.9MB/s]
Downloading best_ckpt_mp0.tar: 96%|█████████▋| 8.02G/8.31G [02:28<00:03, 85.8MB/s]
Downloading best_ckpt_mp0.tar: 97%|█████████▋| 8.04G/8.31G [02:28<00:03, 94.0MB/s]
Downloading best_ckpt_mp0.tar: 97%|█████████▋| 8.05G/8.31G [02:28<00:03, 89.8MB/s]
Downloading best_ckpt_mp0.tar: 97%|█████████▋| 8.07G/8.31G [02:28<00:03, 68.2MB/s]
Downloading best_ckpt_mp0.tar: 97%|█████████▋| 8.09G/8.31G [02:29<00:03, 78.4MB/s]
Downloading best_ckpt_mp0.tar: 97%|█████████▋| 8.10G/8.31G [02:29<00:02, 78.7MB/s]
Downloading best_ckpt_mp0.tar: 98%|█████████▊| 8.12G/8.31G [02:29<00:02, 89.2MB/s]
Downloading best_ckpt_mp0.tar: 98%|█████████▊| 8.12G/8.31G [02:29<00:02, 72.0MB/s]
Downloading best_ckpt_mp0.tar: 98%|█████████▊| 8.14G/8.31G [02:30<00:03, 54.7MB/s]
Downloading best_ckpt_mp0.tar: 98%|█████████▊| 8.16G/8.31G [02:30<00:03, 47.3MB/s]
Downloading best_ckpt_mp0.tar: 98%|█████████▊| 8.17G/8.31G [02:30<00:02, 48.7MB/s]
Downloading best_ckpt_mp0.tar: 98%|█████████▊| 8.18G/8.31G [02:31<00:03, 39.0MB/s]
Downloading best_ckpt_mp0.tar: 99%|█████████▊| 8.20G/8.31G [02:31<00:02, 51.3MB/s]
Downloading best_ckpt_mp0.tar: 99%|█████████▉| 8.22G/8.31G [02:31<00:01, 59.5MB/s]
Downloading best_ckpt_mp0.tar: 99%|█████████▉| 8.24G/8.31G [02:31<00:01, 71.7MB/s]
Downloading best_ckpt_mp0.tar: 99%|█████████▉| 8.25G/8.31G [02:32<00:00, 63.2MB/s]
Downloading best_ckpt_mp0.tar: 99%|█████████▉| 8.26G/8.31G [02:32<00:00, 67.6MB/s]
Downloading best_ckpt_mp0.tar: 100%|█████████▉| 8.28G/8.31G [02:32<00:00, 60.3MB/s]
Downloading best_ckpt_mp0.tar: 100%|█████████▉| 8.29G/8.31G [02:33<00:00, 55.7MB/s]
Downloading best_ckpt_mp0.tar: 100%|█████████▉| 8.30G/8.31G [02:33<00:00, 52.4MB/s]
Downloading best_ckpt_mp0.tar: 100%|██████████| 8.31G/8.31G [02:33<00:00, 54.8MB/s]
Downloading best_ckpt_mp0.tar: 100%|██████████| 8.31G/8.31G [02:33<00:00, 58.2MB/s]
2026-08-15 04:47:48.504 | INFO | earth2studio.run:ensemble:394 - Running ensemble inference!
2026-08-15 04:47:48.504 | INFO | earth2studio.run:ensemble:401 - Inference device: cuda
Fetching GFS data: 0%| | 0/74 [00:00<?, ?it/s]
Fetching GFS data: 1%|▏ | 1/74 [00:01<02:19, 1.91s/it]
Fetching GFS data: 100%|██████████| 74/74 [00:01<00:00, 37.71it/s]
2026-08-15 04:47:52.007 | SUCCESS | earth2studio.run:ensemble:422 - Fetched data from GFS
2026-08-15 04:47:52.074 | INFO | earth2studio.run:ensemble:466 - Starting 2 Member Ensemble Inference with 2 number of batches.
Total Ensemble Batches: 0%| | 0/2 [00:00<?, ?it/s]
Fetching GFS data: 0%| | 0/296 [00:00<?, ?it/s]
Fetching GFS data: 0%| | 1/296 [00:02<14:15, 2.90s/it]
Fetching GFS data: 20%|█▉ | 59/296 [00:03<00:09, 25.39it/s]
Fetching GFS data: 24%|██▍ | 72/296 [00:04<00:11, 20.34it/s]
Fetching GFS data: 42%|████▏ | 123/296 [00:04<00:04, 40.27it/s]
Fetching GFS data: 45%|████▌ | 134/296 [00:05<00:06, 25.27it/s]
Fetching GFS data: 63%|██████▎ | 187/296 [00:06<00:02, 45.93it/s]
Fetching GFS data: 67%|██████▋ | 199/296 [00:06<00:02, 35.43it/s]
Fetching GFS data: 85%|████████▍ | 251/296 [00:07<00:00, 58.15it/s]
Fetching GFS data: 100%|██████████| 296/296 [00:07<00:00, 40.93it/s]
Running batch 0 inference: 0%| | 0/5 [00:00<?, ?it/s]
Running batch 0 inference: 40%|████ | 2/5 [00:00<00:00, 3.06it/s]
Running batch 0 inference: 60%|██████ | 3/5 [00:01<00:00, 2.36it/s]
Running batch 0 inference: 80%|████████ | 4/5 [00:01<00:00, 2.10it/s]
Running batch 0 inference: 100%|██████████| 5/5 [00:02<00:00, 1.97it/s]
Total Ensemble Batches: 50%|█████ | 1/2 [00:14<00:14, 14.77s/it]
Running batch 1 inference: 0%| | 0/5 [00:00<?, ?it/s]
Running batch 1 inference: 20%|██ | 1/5 [00:00<00:00, 7.69it/s]
Running batch 1 inference: 40%|████ | 2/5 [00:00<00:01, 2.58it/s]
Running batch 1 inference: 60%|██████ | 3/5 [00:01<00:00, 2.15it/s]
Running batch 1 inference: 80%|████████ | 4/5 [00:01<00:00, 2.00it/s]
Running batch 1 inference: 100%|██████████| 5/5 [00:02<00:00, 1.93it/s]
Total Ensemble Batches: 100%|██████████| 2/2 [00:17<00:00, 7.47s/it]
Total Ensemble Batches: 100%|██████████| 2/2 [00:17<00:00, 8.56s/it]
2026-08-15 04:48:09.203 | SUCCESS | earth2studio.run:ensemble:536 -
Inference complete
/
├── ensemble (2,) int64
├── lat (721,) float64
├── lead_time (5,) timedelta64[h]
├── lon (1440,) float64
├── time (1,) datetime64[ns]
├── u10m (2, 1, 5, 721, 1440) float32
└── v10m (2, 1, 5, 721, 1440) float32Post Processing¶
The result of the workflow is two zarr stores with the ensemble data for the respective checkpoints used. The rest of the example is focused on some basic post processing to visualize the results.
import cartopy.crs as ccrs
import matplotlib.pyplot as plt
import numpy as np
import xarray as xr
lead_time = 4
plot_date = start_date + timedelta(hours=int(6 * lead_time))
# Load data from both zarr stores
ds0 = xr.open_zarr("outputs/11_hens_0.zarr")
ds1 = xr.open_zarr("outputs/11_hens_1.zarr")
# Combine the datasets
ds = xr.concat([ds0, ds1], dim="ensemble")
# Calculate wind speed magnitude
wind_speed = np.sqrt(ds.u10m**2 + ds.v10m**2)
# Get mean and std of 4th timestep across ensemble
mean_wind = wind_speed.isel(time=0, lead_time=lead_time).mean(dim="ensemble")
std_wind = wind_speed.isel(time=0, lead_time=lead_time).std(dim="ensemble")
# Create figure with two subplots
fig, (ax1, ax2) = plt.subplots(
1, 2, figsize=(15, 4), subplot_kw={"projection": ccrs.PlateCarree()}
)
# Plot mean
p1 = ax1.contourf(
mean_wind.coords["lon"],
mean_wind.coords["lat"],
mean_wind,
levels=15,
transform=ccrs.PlateCarree(),
cmap="nipy_spectral",
)
ax1.coastlines()
ax1.set_title(f'Mean Wind Speed\n{plot_date.strftime("%Y-%m-%d %H:%M UTC")}')
fig.colorbar(p1, ax=ax1, label="m/s")
# Plot standard deviation
p2 = ax2.contourf(
std_wind.coords["lon"],
std_wind.coords["lat"],
std_wind,
levels=15,
transform=ccrs.PlateCarree(),
cmap="viridis",
)
ax2.coastlines()
ax2.set_title(
f'Wind Speed Standard Deviation\n{plot_date.strftime("%Y-%m-%d %H:%M UTC")}'
)
fig.colorbar(p2, ax=ax2, label="m/s")
plt.tight_layout()
# Save the figure
plt.savefig(f"outputs/11_hens_step_{plot_date.strftime('%Y_%m_%d')}.jpg")
Execution profile
Runtime telemetry
Total runtime8m 22s
Execution environment
CPUAMD EPYC 7313P 16-Core Processor
GPUNVIDIA H100 PCIe · 79.6 GiB
System RAM58.5 GiB
PlatformLinux 6.8.0-136-generic
Python3.13.13
GPU driver / CUDADriver 595.84 · CUDA support 13.2