Higher-Level Libraries

The MSC adapters for higher-level libraries use shortcuts under the hood.

fsspec

multistorageclient.async_fs aliases the multistorageclient.contrib.async_fs module.

This module provides the multistorageclient.contrib.async_fs.MultiStorageAsyncFileSystem class which implements fsspec’s AsyncFileSystem class.

Note

The msc:// protocol is automatically registered when pip install multi-storage-client is run.

import multistorageclient as msc

# Create an MSC-based AsyncFileSystem instance.
fs = msc.async_fs.MultiStorageAsyncFileSystem()

# Create a client for the data-s3-iad profile and open a file.
file = fs.open("msc://data-s3-iad/animal-photos/red-panda.png")

# Reuse the client for the data-s3-iad profile and download a file.
fs.get_file(
    rpath="msc://data-s3-iad/animal-photos/red-panda.png",
    lpath="/tmp/animal-photos/red-panda.png"
)

Hydra

The MSC Hydra plugin enables loading Hydra configurations directly from object storage using msc:// URLs.

Note

The plugin is automatically registered by Hydra when both multistorageclient and hydra-core are installed.

Programmatic usage.

import hydra
import multistorageclient as msc
from omegaconf import DictConfig

# Load config directly from object storage
@hydra.main(version_base=None, config_path="msc://profile/configs", config_name="training")
def your_app(cfg: DictConfig) -> None:
    print(f"Loaded config: {cfg}")

Shell usage.

python your_app.py --config-path="msc://profile/configs" --config-name=training

NumPy

multistorageclient.numpy aliases the multistorageclient.contrib.numpy module.

This module provides load, memmap, and save methods for loading and saving NumPy arrays.

import multistorageclient as msc
import numpy

# Create a client for the data-s3-iad profile and load an array.
array = msc.numpy.load("msc://data-s3-iad/numpy-arrays/ndarray-1.npz")

# Reuse the client for the data-s3-iad profile and load a memory-mapped array.
mmarray = msc.numpy.memmap("msc://data-s3-iad/numpy-arrays/ndarray-1.bin")

# Reuse the client for the data-s3-iad profile and save an array.
msc.numpy.save(
    "msc://data-s3-iad/numpy-arrays/ndarray-2.npy",
    numpy.array([1, 2, 3, 4, 5], dtype=numpy.int32)
)

PyTorch

multistorageclient.torch aliases the multistorageclient.contrib.torch module.

This module provides load and save methods for loading and saving PyTorch data.

import multistorageclient as msc
import torch

# Create a client for the data-s3-iad profile and load a tensor.
tensor = msc.torch.load("msc://data-s3-iad/pytorch-tensors/tensor-1.pt")

# Reuse the client for the data-s3-iad profile and save a tensor.
msc.torch.save(
    torch.tensor([1, 2, 3, 4]),
    "msc://data-s3-iad/pytorch-tensors/tensor-2.pt"
)

In addition to the load and save methods, the torch module provides the MultiStorageFileSystemReader and MultiStorageFileSystemWriter classes for reading and writing PyTorch objects to multiple storage backends.

import multistorageclient as msc
import torch
import torch.distributed.checkpoint as dcp

# Create a MultiStorageFileSystemWriter for the data-s3-iad profile.
writer = msc.torch.MultiStorageFileSystemWriter("msc://data-s3-iad/checkpoint/1")
dcp.save(
    state_dict=state_dict,
    storage_writer=writer,
)

# Create a MultiStorageFileSystemReader for the data-s3-iad profile.
reader = msc.torch.MultiStorageFileSystemReader("msc://data-s3-iad/checkpoint/1")
dcp.load(
    state_dict=loaded_state_dict,
    storage_reader=reader,
)

Xarray

Xarray supports msc:// URLs through MSC’s built-in fsspec implementation. Use the native Xarray API directly.

import xarray

# Load a Zarr array through MSC's fsspec implementation.
xarray_dataset = xarray.open_zarr("msc://data-s3-iad/abc.zarr")

Warning

The former msc.xarray adapter is no longer available. Replace msc.xarray.open_zarr("msc://...") with xarray.open_zarr("msc://...").

Zarr

Zarr supports msc:// URLs through MSC’s built-in fsspec implementation. Use the native Zarr API directly.

import zarr

# Load a Zarr array through MSC's fsspec implementation.
z = zarr.open("msc://data-s3-iad/abc.zarr", mode="r")

Note

The former msc.zarr adapter is no longer available. Replace msc.zarr.open_consolidated("msc://...") with zarr.open("msc://...").

Path

multistorageclient.Path aliases the multistorageclient.pathlib.MultiStoragePath class.

This module provides the Path class for working with paths in a way similar to pathlib.Path.

import multistorageclient as msc

# Create a Path object for a file in the data-s3-iad profile
path = msc.Path("msc://data-s3-iad/data/file.txt")

# Get parent directory
parent = path.parent  # msc://data-s3-iad/data

# Get file name
name = path.name  # file.txt

# Join paths
new_path = path.parent / "other.txt"  # msc://data-s3-iad/data/other.txt

# Check if path exists
exists = path.exists()

# List contents of a directory
for child in msc.Path("msc://data-s3-iad/data").iterdir():
    print(child)

# Find files matching a pattern
for matched in msc.Path("msc://data-s3-iad/data").glob("*.txt"):
    print(matched)

# Sort paths for deterministic processing
paths = sorted(msc.Path("msc://data-s3-iad/data").glob("*.txt"))

Note

The Path class implements much of the same interface as pathlib.Path, making it familiar to use while working with remote storage.