nvalchemi.data.datapipes.ZarrArrayConfig

pydantic model nvalchemi.data.datapipes.ZarrArrayConfig

Per-array storage settings for compression, chunking, and sharding.

A ZarrArrayConfig bundles the codec and layout choices applied to a single Zarr array written by AtomicDataZarrWriter. The codec fields (compressors, filters, serializer) accept zarr v3 codec instances and control how bytes are transformed on write; leaving them None uses Zarr’s defaults. chunk_size sets the chunk length along the leading (row / sample) dimension, with all other dimensions stored at full extent, and shard_size optionally groups several chunks into one storage object to reduce file count for object stores.

You rarely construct this directly for a whole store; instead you attach one or more ZarrArrayConfig instances to a ZarrWriteConfig, which routes them to the metadata, core, and custom array groups (and to per-field overrides). Tuning these settings trades write size and speed against read throughput – see the nvalchemi-zarr-perf guidance for chunk/shard sizing under shuffled or random access.

Examples

Zstandard compression with 1024-row chunks:

from zarr.codecs import ZstdCodec

cfg = ZarrArrayConfig(compressors=(ZstdCodec(level=3),), chunk_size=1024)

Group four chunks into each shard (shard_size must be a multiple of chunk_size):

cfg = ZarrArrayConfig(chunk_size=256, shard_size=1024)

Notes

When both chunk_size and shard_size are set, shard_size must be an exact multiple of chunk_size; an after validator raises ValueError otherwise. compressors and filters are tuples of codecs applied in order, and arbitrary_types_allowed is enabled so that native zarr codec objects can be stored as field values.

See also

Zarr Compression Tuning – choosing codecs, chunk sizes, and shard sizes to balance store size against read and write throughput.

field compressors: tuple[ArrayArrayCodec | ArrayBytesCodec | BytesBytesCodec, ...] | None = None

Compressor codec(s) to apply.

field filters: tuple[ArrayArrayCodec | ArrayBytesCodec | BytesBytesCodec, ...] | None = None

Array-to-array filter codec(s).

field serializer: ArrayArrayCodec | ArrayBytesCodec | BytesBytesCodec | None = None

Bytes serializer codec.

field chunk_size: int | None = None

Chunk length along dimension 0. Other dims use full extent.

field shard_size: int | None = None

Shard length along dimension 0. When set, multiple chunks are stored in a single storage object. Must be a multiple of chunk_size when both are specified.

field write_empty_chunks: bool = True

Whether to write chunks that are entirely fill-valued.