Multi-Storage File System (MSFS)

The Multi-Storage File System (MSFS) provides POSIX filesystem access to object storage backends through FUSE (Filesystem in Userspace). This enables applications that require traditional filesystem operations to work seamlessly with cloud object storage without code modifications.

Overview

While the Python Multi-Storage Client is designed for easy adoption of object storage by Python applications, some applications prefer or require POSIX filesystem access. MSFS bridges this gap by:

  • Providing a POSIX-compliant filesystem interface to object storage

  • Supporting S3-compatible object storage (AWS S3, AIS, etc.)

  • Enabling applications written in any language to access object storage

  • Sharing the same configuration format as the Python MSC for consistency

Note

S3 Read and Write Support

MSFS supports file creation, modification, truncation, deletion, and explicit fsync/fdatasync durability for writable S3 backends. The default flush_on_close: false mode favors close latency and completes the remote commit asynchronously; applications requiring an acknowledgement must call fsync/fdatasync or configure flush_on_close: true.

If an asynchronous commit fails after close has already been acknowledged, the failure is recorded against the file and the next fsync/fdatasync on it returns EIO once. The data is not lost — it stays dirty and a later flush retries it — so a successful retry clears the record and reports no error.

Key Features

  • FUSE-based: Mounts object storage as a standard filesystem

  • S3 backend support: AWS S3 and S3-compatible object stores

  • High-performance caching: Configurable cache for improved read performance

  • Write-to-read cache promotion: Optional reuse of committed write bytes for immediate readback without another object-store download

  • Dynamic configuration: Add or remove backends without unmounting via SIGHUP

  • Standard Unix tools: Use with mount, umount, and /etc/fstab

  • Observability: Integrated telemetry with OpenTelemetry metrics

When to Use MSFS

MSFS exists for one situation: an application that must see a filesystem, over data that lives in object storage. If you can call an object-storage API directly, do that instead — the Python Multi-Storage Client avoids a FUSE hop entirely and will be faster.

MSFS is a good fit when:

  • The application cannot be changed to speak S3 — a training framework that takes a directory path, a third-party binary, a shell pipeline, or code in a language with no usable SDK.

  • The access pattern is read-heavy over a working set that fits the node-local cache. Repeat reads are served from cache and never reach the backend.

  • Datasets are large but the working set is not. Manifest-based bootstrap makes a 100M-object namespace browsable in under two minutes without listing it on every mount.

  • Writes are whole-file — created once, written, closed. Deferred PutObject makes this efficient, and small files commit in a single request.

Look elsewhere when:

  • Many nodes need the same cache. Each MSFS process owns its own; there is no shared or coordinated tier. Placing cache_dir_path on a shared filesystem does not change this. Front the bucket with a caching tier such as AIStore instead, which MSFS can read as a backend.

  • You need POSIX semantics object stores cannot provide — byte-range in-place updates, hard links, atomic rename across directories, or O_APPEND from several writers. A write replaces the whole object.

  • Per-user authorization is required. Access is per mount: anyone who can read the mount point can read everything the backend credentials reach.

  • The workload is many small concurrent writes. Small-object writes are bounded by per-object commit latency rather than bandwidth; see Measured Scale and Performance.

  • You need a warm cache immediately after mount. There is no pre-warm API and the cache is discarded on unmount.

Deployment Model

MSFS is not a centralized storage or caching service. It is a FUSE process that runs on each compute node, started either directly through mount -t msfs or by the Kubernetes CSI node plugin on behalf of a pod. Nothing is provisioned outside the compute node, so MSFS can be deployed without dedicated storage appliances. The consequence is that every MSFS process owns an independent cache: two mounts on one node, or the same mount on two nodes, share nothing.

Access is granted per mount rather than per user. Anyone who can read the mount point can read everything the backend credentials can reach, so deployments that need tenant isolation must separate tenants by mount.

Capability Summary

Capability

Status

POSIX read access to S3, AIStore, and GCS backends

Supported

POSIX write access (create, modify, truncate, delete)

S3 backends only

Write-to-read cache promotion after a successful commit

Supported (opt-in)

Multiple buckets or bucket+prefixes as sibling subdirectories of one mount

Supported

Adding and removing backends without unmounting (SIGHUP)

Supported

Manifest-based bootstrap for large namespaces

Supported

Node-local read cache with read-ahead, capacity bound, and LRU eviction

Supported

Cache backed by RAM, a shared mapped file, or per-inode files

Supported

OpenTelemetry metrics and a Prometheus endpoint

Supported

Kubernetes deployment via the CSI node plugin

Supported

Cache shared or coordinated across mounts or nodes

Not implemented

Cache surviving unmount

Not implemented

Explicit data pre-warm API

Not implemented

Per-user (UID/GID) authorization within a mount

Not implemented

Modifying an existing backend in place via SIGHUP

Not implemented

Installation

Download from GitHub Actions Artifacts

The easiest way to install MSFS is to download pre-built packages from our GitHub Actions artifacts.

Download the artifact archive:

  1. Navigate to the GitHub Actions Default Branch workflow

  2. Select a workflow run for the desired commit

  3. Download the multi-storage-file-system artifact

Extract the archive:

Extract the artifact archive.
unzip multi-storage-file-system.zip

The archive contains:

  • RPM packages: msfs-<version>-1.x86_64.rpm and msfs-<version>-1.aarch64.rpm

  • DEB packages: msfs_<version>_amd64.deb and msfs_<version>_arm64.deb

After installation, MSFS provides:

  • /usr/bin/msfs - The FUSE daemon binary

  • /usr/bin/mount.msfs - Mount helper for standard mount command

Build from Source

Alternatively, you can build MSFS from source:

Build MSFS from source.
cd multi-storage-file-system
make
sudo make install

Configuration

MSFS uses the standard MSC configuration format, providing seamless integration with existing MSC configurations.

MSFS searches for configuration files in the same locations as the Python MSC:

  1. Path specified by MSC_CONFIG environment variable

  2. ${XDG_CONFIG_HOME}/msc/config.yaml or ${XDG_CONFIG_HOME}/msc/config.json

  3. ${HOME}/.msc_config.yaml or ${HOME}/.msc_config.json

  4. ${HOME}/.config/msc/config.yaml or ${HOME}/.config/msc/config.json

  5. ${XDG_CONFIG_DIRS:-/etc/xdg}/msc/config.yaml or ${XDG_CONFIG_DIRS:-/etc/xdg}/msc/config.json

  6. /etc/msc_config.yaml or /etc/msc_config.json

See Configuration Reference for the complete MSC configuration schema.

Note

Advanced Configuration Mode

For advanced users requiring fine-grained control over FUSE behavior, caching parameters, and other low-level settings, MSFS provides an extended configuration mode (msfs_version: 1). This advanced mode is intended for specialized use cases and performance tuning. For details, see the MSFS README.

Environment Variables

Configuration files support environment variable expansion using $VAR or ${VAR} syntax:

profiles:
  my-profile:
    storage_provider:
      type: s3
      options:
        base_path: ${BUCKET_NAME}
        access_key_id: ${AWS_ACCESS_KEY_ID}
        secret_access_key: ${AWS_SECRET_ACCESS_KEY}

MSFS-Specific Environment Variables:

  • MSC_CONFIG - Path to configuration file

  • MSFS_MOUNTPOINT - Mount point (overrides config file setting)

  • MSFS_BINARY - Path to msfs binary (default: /usr/bin/msfs)

  • MSFS_LOG_DIR - Log directory (default: /var/log/msfs)

Usage

Basic Usage

Manual mount/unmount using the MSFS binary directly:

# Start MSFS daemon with config file
export MSC_CONFIG=/path/to/config.yaml
/usr/bin/msfs

# In another terminal, verify mount
mount | grep msfs
df -h /mnt

# Access files
ls -l /mnt/backend-name/
cat /mnt/backend-name/path/to/file.txt

# Stop daemon (unmount)
umount /mnt

Mount Helpers

After installation, MSFS can be mounted using standard Unix mount and umount commands:

Mounting

# Mount with config file and mountpoint
sudo mount -t msfs /path/to/config.yaml /mnt/storage

# Mount multiple instances with different configs
sudo mount -t msfs /path/to/config1.yaml /mnt/storage1
sudo mount -t msfs /path/to/config2.json /mnt/storage2

How It Works:

When you run mount -t msfs <config> <mountpoint>, the mount command automatically calls /usr/bin/mount.msfs, which:

  1. Exports MSC_CONFIG environment variable from the config file argument

  2. Exports MSFS_MOUNTPOINT environment variable from the mountpoint argument

  3. Creates log directory if needed (/var/log/msfs/)

  4. Launches the msfs daemon in the background using setsid

  5. Stores the process ID in /var/log/msfs/msfs_*.pid

Note

The mount command behaves differently based on arguments:

  • mount (no args) → Lists all mounted filesystems

  • mount -t msfs (type only) → Lists all MSFS filesystems (does NOT call mount.msfs)

  • mount -t msfs <config> <mountpoint> → Calls mount.msfs to perform the mount

Unmounting

To unmount the filesystem, use the standard umount command:

# Unmount MSFS filesystem
umount <mount_point>

# Example
umount /mnt/storage1

Automatic Mounting with /etc/fstab

MSFS filesystems can be automatically mounted at boot time using /etc/fstab:

/etc/fstab entries for MSFS
# MSFS filesystem with S3 backend
/etc/msfs/s3-config.yaml  /mnt/s3-data  msfs  defaults,_netdev  0  0

# MSFS filesystem with local config
/home/user/msfs.json      /mnt/storage  msfs  defaults,noauto   0  0

Field Explanation:

  1. Device - Path to MSFS configuration file (YAML or JSON)

  2. Mount Point - Directory where the filesystem will be mounted

  3. Type - Filesystem type (msfs)

  4. Options - Mount options (comma-separated):

    • defaults - Standard mount options

    • _netdev - Wait for network before mounting (recommended for remote storage)

    • noauto - Don’t mount automatically at boot (mount manually)

    • user - Allow non-root users to mount (requires allow_other in config)

  5. Dump - Backup frequency (usually 0)

  6. Pass - fsck pass number (usually 0)

After editing /etc/fstab, test the configuration:

# Mount all filesystems in fstab
sudo mount -a

# Verify mount
df -h /mnt/s3-data

Dynamic Configuration Reload

MSFS supports dynamic configuration changes without unmounting:

# Edit configuration file
vim /path/to/config.yaml

# Send SIGHUP to reload configuration
sudo kill -SIGHUP $(pidof msfs)

Configuration changes are processed as follows:

  • Existing backends - Cannot be modified (unmount and remount required)

  • New backends - Automatically mounted and appear as new subdirectories

  • Removed backends - Automatically unmounted and subdirectories disappear

Alternatively, enable automatic periodic configuration reloading:

msfs_version: 1
auto_sighup_interval: 300  # Check config every 5 minutes
backends:
  # ...

Manifest-Based Bootstrap

For large-scale datasets (millions of objects), MSFS can pre-generate a manifest of directory listings at mount time. This enables immediate POSIX access without per-file S3 calls.

Enable manifest generation by adding manifest_path to a backend:

backends:
  - dir_name: s3
    readonly: true
    manifest_path: "/home/user/.msfs_manifest"
    backend_type: S3
    S3:
      # ...

On first mount, MSFS generates per-directory TSV manifests via parallel BFS listing of S3, then ingests entries into sharded B+Trees backed by PebbleDB for persistent, memory-efficient lookups.

Writable Manifest-Backed Mounts

A manifest-backed mount can also be writable. Manifest ingest runs at mount whenever manifest_path is set, regardless of readonly, so files created, overwritten, or deleted through a previous session are reconstructed.

Manifest generation at mount is restricted to readonly: true backends, because generation lists the entire backend namespace and that listing is only consistent while nothing can mutate it. Generate the manifest out of band before mounting a backend writable:

msfs generate-manifest -backend s3 -output /home/user/.msfs_manifest

A writable backend whose manifest_path contains no manifest logs skipping generation and mounts without manifest metadata.

The generated manifest is an immutable snapshot and is never rewritten on the write path. Instead, committed mutations are recorded as upsert and tombstone records in an append-only delta log under <manifest_path>/_msfs_delta/, which lookup, readdir, and ingest overlay on top of the base manifest. Records are appended only after the object-store commit succeeds, so the delta log never advertises an object the backend does not hold.

Manifest Configuration Options

backends:
  - dir_name: s3
    manifest_path: "/home/user/.msfs_manifest"
    manifest_gen_workers: 200             # Number of parallel BFS listing workers (default: 200)
    flat_dir_confirmation_pages: 5        # Pages to confirm a flat dir before parallel listing (default: 5)

Flat Directory Acceleration

For buckets with flat layouts (millions of files in a single prefix with no subdirectories), MSFS automatically detects large flat directories and parallelizes listing using prefix-based or range-based splitting. User-provided hints can further optimize this:

backends:
  - dir_name: s3
    manifest_path: "/home/user/.msfs_manifest"
    flat_dir_hints:
      - path: "training-data/"
        key_prefix_chars: "0123456789"
        split_depth: 2
Flat Directory Hint Fields

Field

Type

Description

path

string

Directory path relative to the backend prefix (must end with /)

key_prefix_chars

string

Characters that appear at the start of basenames (default: 0123456789abcdefghijklmnopqrstuvwxyz)

split_depth

int

Number of leading characters for sub-prefix generation (default: 1; depth 2 with 10 chars = 100 sub-workers)

Without hints, MSFS detects flat directories automatically and selects the best parallelization strategy (prefix discovery or lexicographic range splitting).

Performance

MSFS includes a sophisticated caching layer to optimize read performance and optionally reuse exact bytes retained from successful writes.

Cache Configuration

The cache uses a line-based architecture where each cache line represents a fixed-size chunk of data:

cache_line_size: 1048576       # 1 MiB per cache line
cache_lines: 4096              # 4096 cache lines = 4 GiB total cache
write_cache_promotion: false   # Opt in to write-to-read cache reuse

Cache Tuning Guidelines:

  • Larger cache line size - Better for sequential access patterns, fewer cache lines needed

  • Smaller cache line size - Better for random access patterns, more granular caching

  • More cache lines - Allows caching more files or larger portions of files

  • Less cache lines - Reduces memory usage

Write-to-Read Cache Promotion

When write_cache_promotion: true, MSFS admits locally retained bytes into the read cache only after the object-store commit succeeds. Admission starts at offset zero and stops when cache capacity is exhausted; MSFS never downloads data merely to populate the cache. Deferred single-PUT writes and retained multipart buffers can populate complete objects. Existing-object overlays populate only cache lines fully covered by local write ranges.

Promotion is disabled by default. Cache population runs asynchronously through a bounded worker pool, while an immediate read waits for any matching Inbound cache line to become Clean. Local cache-population failure does not change the success of the already-completed object-store write.

Read Performance

Read performance is optimized through:

  • Read-ahead caching - Cache lines are prefetched for sequential reads

  • Cache hit reuse - Frequently accessed data remains cached

  • Parallel prefetching - Multiple cache lines loaded concurrently

Best practices:

  • Size cache_lines to accommodate your working set

  • Use larger cache_line_size for large files

  • Use smaller cache_line_size for many small files

Cache Capacity and Lifetime

cache_line_size (default 10 MiB) is the fetch and residency granularity and cache_lines (default 128) is how many lines are provisioned, so the default capacity is about 1.25 GiB. The benchmarks in Measured Scale and Performance used cache_lines: 10000, or roughly 100 GiB. Capacity is a hard bound and eviction is LRU, so a dataset larger than the cache holds only its active working set. For very large datasets, size the cache to the working set rather than to the dataset.

cache_storage selects where lines live: ram (anonymous mmap), mapped-file (one shared memory-mapped file, the default), or per-inode-file (per-inode contiguous files served with pread). All three are node-local.

Each mount creates its own private cache directory under cache_dir_path and removes that directory on unmount. The cache does not survive the mount, and a remount starts cold. Files left behind by a crash are not discovered or reused.

Note

Pointing cache_dir_path at a shared filesystem such as Lustre places each mount’s private directory on shared storage, but does not produce a shared cache. The catalog that makes cached bytes findable — the object-to-line index, cache-line state and ETags, in-flight fetch tracking, LRU order, and capacity accounting — lives in the memory of a single MSFS process. Two MSFS processes over the same shared path still use different directories, fetch the same object twice, cannot see each other’s entries, and cannot coordinate fills, invalidation, or eviction. A shared filesystem guarantees consistency for shared files; it does not supply object-cache semantics such as key/version lookup, single-flight fetches, or ETag invalidation.

Pre-warming

There is no pre-warm API. Reading files warms the cache of the MSFS process that served those reads, so one job can warm a mount that a later job reuses — but only if that MSFS process is still running and the working set still fits in cache. A pre-warm job that unmounts when it finishes, or a later job that creates its own mount, starts cold.

Manifest generation is a separate mechanism and warms only namespace metadata. It makes directory traversal and attribute lookups fast without per-object backend calls; it does not fetch file contents.

Measured Scale and Performance

These are single-node measurements of the read path, write path and namespace bootstrap against same-region storage. They record what has been measured, not a supported configuration limit. See Qualified Scale Boundary.

Namespace Scale: 100M Objects

Measured on an EC2 c5a.12xlarge (48 vCPU, 96 GiB) in us-west-2 against an S3 bucket in the same region, over a dataset of 100,237,498 objects across 101,339 directories.

100M-object bootstrap

Phase

Elapsed

Throughput

Peak RSS

Manifest generation (parallel BFS listing, 200 workers)

1m 45s

954,680 obj/s

Manifest ingest (per-directory TSV into sharded B+Tree/PebbleDB)

16m 41s

100,129 obj/s

~7.4 GiB

Total bootstrap

~18m 26s

The mount is browsable when generation finishes, at about 105 seconds, not when ingest finishes. During ingest, metadata is served from the manifest while the optimized index is built in the background, so traversal and enumeration — enough to compute dataset splits and begin streaming — work well before the 18m 26s mark. Generation held at about 1m 43s and ingest at about 16m 50s across repeated runs.

Note

Set process_memory_limit generously for an ingest of this size. The 4 GiB default sits below the working set of a 100M-object ingest, which drives continuous garbage collection and collapses throughput.

These figures assume a hierarchical layout. Both phases degrade sharply when one directory holds the entire namespace, because generation finds a single key prefix to split across roughly 20 range workers instead of 200 directory workers, and every directory entry lands in one B+Tree shard.

Directory-width sensitivity (100M objects)

Layout

Generation

Ingest

Peak RSS

101,339 directories

1m 43s (977K obj/s)

16m 29s (101K obj/s)

~6.5 GiB

1 directory

30m 35s (~55K obj/s)

44m 19s (37.8K obj/s)

~21 GiB

The penalty is super-linear in directory width: 10M objects in a single directory generate in 1m 41s and ingest in 2m 5s, so the same flat shape is far cheaper an order of magnitude smaller.

Read Throughput: 24-Cell Matrix

Measured on an EC2 c5n.18xlarge (72 vCPU, 184 GiB) in us-west-2 against same-region S3 with a ~100 GiB cache, over a ~88 GiB dataset of 8,192 x 1 MiB plus 80 x 1 GiB files. Six workload families — 4 KiB and 64 KiB request sizes, small-file and large-file, sequential and random — at 1, 2, 4, and 8 application threads, each with a cold and a cache-resident pass, driven by elbencho -r --direct and compared against s3fs-fuse 1.93 with a local disk cache.

The result was 18 wins, 3 ties, and 3 losses across the 24 cells.

Cache-resident throughput relative to s3fs

Workload family

MSFS vs s3fs

Small files, 4 KiB sequential

20x - 52x

Small files, 64 KiB sequential

6.5x - 27x

Large files, 4 KiB sequential

0.69x - 1.04x

Large files, 64 KiB sequential

0.45x - 1.05x

Large files, 4 KiB random

3.9x - 6.9x

Large files, 64 KiB random

5.4x - 14x

Cold reads scale with thread count because each reader issues concurrent ranged GETs: large-file 64 KiB sequential moves 92 MiB/s at one thread and 192 MiB/s at eight, while s3fs cold stays flat near 125 MiB/s on the same families. Cache-resident reads reach 4,536 MiB/s on that family at eight threads. At eight threads MSFS wins or ties every family.

Two caveats keep the losses honest. The large-file 64 KiB losses at one and two threads are largely a page-cache artifact: on a 184 GiB host, s3fs serves its warm reads from the Linux page cache over its own cache files, and after dropping caches it falls to about 148 MiB/s on the same data. The large-file 4 KiB results reflect a real per-operation FUSE ceiling — at one or two threads only one or two FUSE operations are in flight, so neither additional readers nor cache geometry help. --direct forces strict 4 KiB operations with no kernel read-ahead, a deliberately pessimistic operating point; workloads that do not use O_DIRECT benefit from page-cache assistance and reach roughly 2.6 GiB/s on the same data.

Write Throughput

Measured on an EC2 c5n.18xlarge in us-west-2 against same-region S3, driven by elbencho -w --direct and compared against s3fs-fuse 1.93. Both filesystems ran the identical workload on the same host, bucket and day, so the columns are directly comparable. MSFS ran with cache_line_size: 1048576, cache_lines: 1024, multipart_upload_threshold_bytes: 67108864, write_commit_workers: 32 and flush_on_close: true.

Figures are aggregate MiB/s once every thread has finished.

The sequential large-file rows write 80 GiB per cell as 80 separate 1 GiB files, partitioned across threads — 80 files on one thread, 20 files each on four, 10 each on eight — so the total work is identical at every thread count and no two threads write the same object. This matches the scale of the read matrix above and of the June 2026 write matrix. MSFS cells ran roughly 2 to 15 minutes; the same cells on s3fs ran roughly 13 to 43 minutes, because it is slower on identical work. The remaining rows are smaller and state their size and layout in the row itself.

Workload layout is not a detail here. The shared-object rows below write one 1 GiB object from every thread, and at eight threads that writes 502 MiB/s against 290 for one object per thread — the same block size and thread count, 1.7x apart. Reproducing “80 GiB” with a different file count or sharing pattern will produce materially different numbers.

Write throughput, MSFS vs s3fs (MiB/s)

Workload

MSFS

s3fs

Ratio

1,024 x 1 MiB, 4 KiB blocks, 1 thread

16

5

3.2x

1,024 x 1 MiB, 4 KiB blocks, 4 threads

18

14

1.3x

1,024 x 1 MiB, 4 KiB blocks, 8 threads

17

14

1.2x

1,024 x 1 MiB, 64 KiB blocks, 1 thread

18

7

2.6x

1,024 x 1 MiB, 64 KiB blocks, 4 threads

19

17

1.1x

1,024 x 1 MiB, 64 KiB blocks, 8 threads

19

17

1.1x

80 GiB, 4 KiB sequential, 1 thread

141

32

4.4x

80 GiB, 4 KiB sequential, 2 threads

239

57

4.2x

80 GiB, 4 KiB sequential, 4 threads

279

83

3.4x

80 GiB, 4 KiB sequential, 8 threads

92

101

0.91x

80 GiB, 64 KiB sequential, 1 thread

358

62

5.8x

80 GiB, 64 KiB sequential, 2 threads

573

78

7.3x

80 GiB, 64 KiB sequential, 4 threads

706

96

7.4x

80 GiB, 64 KiB sequential, 8 threads

434

106

4.1x

1 GiB per thread, 64 KiB random, 1 thread

338

73

4.6x

1 GiB per thread, 64 KiB random, 4 threads

299

78

3.8x

1 GiB per thread, 64 KiB random, 8 threads

290

90

3.2x

Shared 1 GiB object, 64 KiB random, 1 thread

382

107

3.6x

Shared 1 GiB object, 64 KiB random, 4 threads

574

108

5.3x

Shared 1 GiB object, 64 KiB random, 8 threads

502

127

4.0x

Random 4 KiB writes are the widest gap and are described rather than tabulated, because s3fs cannot be run there at the same scale. Rewriting a 1 GiB object with random 4 KiB blocks, s3fs sustains about 100 IOPS — under 1 MiB/s, roughly 43 minutes per run — because each write rewrites the whole object. MSFS sustains 33,000-36,000 IOPS, about 135 MiB/s, finishing in about 7 seconds, and holds that rate flat from 1 to 8 threads against one shared object. Writing a separate 1 GiB object per thread with random 4 KiB blocks, MSFS reaches 138 MiB/s at one thread and 68-73 MiB/s at four and eight; the equivalent s3fs run would take several hours, so no ratio is given.

Small-file throughput is bounded by per-object commit latency rather than bandwidth, which is why the 1 MiB rows sit an order of magnitude below the 1 GiB rows for both filesystems. Deferred PutObject is what makes those rows competitive at all: committing a small object in one request instead of a three-request multipart upload moved this family from about 3 MiB/s to 16-19 MiB/s. Raising write_commit_workers increases how many small objects commit in parallel.

Large-file writes peak at four threads and fall back at eight. The one cell MSFS loses is the extreme of that pattern: 4 KiB sequential at eight threads drops from 279 MiB/s to 92 while s3fs continues climbing to 101. Smallest block size and highest thread count is the worst case for a write path that serializes per-file bookkeeping, and it is the operating point to avoid until it is addressed. Every other cell is a 3.4x to 7.4x win, and 64 KiB blocks scale cleanly to four threads.

Note

Write cells are only comparable when the workload shape matches exactly. The two random 64 KiB groups above make the point: at eight threads, one object per thread gives 290 MiB/s while one shared object gives 502 MiB/s. Same block size, same thread count, different path. Never compare a number from one group against the other.

Reproducing These Numbers

Defaults are not benchmark settings. The values below are what the results above were measured with; a run that differs on any of them is not comparable.

Benchmark configuration

Setting

Value

Why

fuse_fd_per_worker

false

Shared /dev/fuse descriptor. Cloned per-worker descriptors measured 18-25% slower.

fuse_workers

50

On a 72-vCPU host. The default of 0 uses runtime.NumCPU(), which is too many readers on large hosts.

cache_line_size

10485760

10 MiB fetch and residency granularity.

cache_lines

sized to the working set

10000 held the whole 88 GiB dataset.

cache_lines_to_prefetch

4

Read-ahead depth. Higher values can help backends with a different latency knee.

process_memory_limit

68719476736

The 4 GiB default causes sustained garbage collection on large ingests.

GOMAXPROCS

unset on a 72-vCPU host

Left unset so Go uses every vCPU. On a 256-vCPU host, pinning it to 72 matched the 72-vCPU result; leaving it at 32 throttled the read and GC path.

Two settings live outside the configuration file and must be re-applied after every mount, because the FUSE connection id changes each time:

Post-mount tuning, required after every mount
ulimit -n 131072

sudo sh -c 'for d in /sys/fs/fuse/connections/*/; do
    echo 144 > "${d}max_background"
    echo 108 > "${d}congestion_threshold"
done'

The kernel defaults, max_background=12 and congestion_threshold=9, throttle in-flight background and read-ahead requests and cap read concurrency well below what the thread count suggests. Leaving them at their defaults is the most common reason a run appears to stop scaling after a few threads.

Qualified Scale Boundary

What the measurements above establish, and what they do not:

Scale boundary

Dimension

Measured

Not established

Clients

1 MSFS process per test

Many concurrent clients against one backend, including request fan-out and cache-hit behavior under aggregate load

Application threads

1 - 8

Higher concurrency per mount

Objects

~100M in one namespace

Substantially larger namespaces

Working set

~88 GiB, cache-resident

Datasets far larger than cache, where only a small fraction is resident

Latency

Mean throughput per cell

Tail-latency targets

Failure handling

Clean runs

Backend failure and recovery under load

Access pattern

Read path and namespace bootstrap

Write throughput and write durability at scale

The companion run in which the dataset deliberately exceeded cache capacity, so that reads had to keep returning to the backend, stopped after 18 of 24 cases and was never completed. Sustained-eviction behavior is therefore not characterized.

Authentication

Direct mounts take credentials from the configuration file, or from the standard AWS configuration and credentials files through use_config_env and use_credentials_env. Environment variable references such as ${AWS_ACCESS_KEY_ID} keep literal secrets out of the configuration file.

Under Kubernetes, the CSI node plugin additionally supports a static Secret referenced by nodePublishSecretRef, a driver-level workload identity (IRSA on EKS), and a per-workload role assumed from volumeAttributes.roleArn.

Note

Credential rotation and multi-tenant isolation have not been qualified end-to-end. Because authorization is per mount rather than per user, a mount exposes everything its credentials can reach to every reader of the mount point.

Observability

MSFS supports OpenTelemetry metrics for monitoring performance and operations. Metrics configuration uses the same schema as the Python MSC for consistency.

Configuration

Enable metrics collection by adding observability configuration:

Metrics with OTLP exporter
 1opentelemetry:
 2  metrics:
 3    attributes:
 4      - type: static
 5        options:
 6          attributes:
 7            service.name: msc-posix
 8            deployment.environment: production
 9      - type: host
10      - type: process
11
12    reader:
13      type: periodic
14      options:
15        collect_interval_millis: 1000
16        export_interval_millis: 60000
17
18    exporter:
19      type: otlp
20      options:
21        endpoint: "http://otel-collector:4318"
22        insecure: true
23
24backends:
25  # ...

See Telemetry for complete observability configuration options.

Metrics Exported

MSFS exports the following metrics:

Cache Metrics:

  • msfs.cache.hits - Number of cache hits

  • msfs.cache.misses - Number of cache misses

  • msfs.cache.evictions - Number of cache evictions

I/O Metrics:

  • msfs.io.bytes_read - Total bytes read

  • msfs.io.read_operations - Number of read operations

Backend Metrics:

  • msfs.backend.operations - Operations per backend (with labels)

  • msfs.backend.errors - Errors per backend (with labels)

Logs

MSFS logs are written to stdout by default. When using mount helpers, logs are redirected to /var/log/msfs/msfs_<pid>.log.

Configure log verbosity per backend:

backends:
  - dir_name: debug-backend
    trace_level: 3  # 0=none, 1=errors, 2=successes, 3+=details
    # ...

Development

Docker Development Environment

A Docker-based development environment is provided for testing:

# Pull MinIO image
docker pull minio/minio:latest

# Build development container
docker-compose build

# Start containers (MinIO + dev)
docker-compose up -d dev

# Enter development container
docker-compose exec dev bash

Inside the container:

# Setup development environment with MinIO backend
./dev_setup.sh minio

# Build MSFS
make

# Run MSFS in background
./msfs &

# Test filesystem
mount | grep fuse
df -h /mnt
ls -lR /mnt

# Reload configuration
kill -SIGHUP $(pidof ./msfs)

# Stop daemon
kill -SIGTERM $(pidof ./msfs)

# Exit container
exit

# Stop containers
docker-compose down

Testing

Test scripts are provided in the multi-storage-client/tests/test_mscp/ directory:

cd multi-storage-client/tests/test_mscp

# Test mount/unmount
./test_mount.sh

# Test cleanup
./test_cleanup.sh

# Test observability
./test_observability.sh

Deployment

Building for Production

Build optimized binaries for production deployment:

cd multi-storage-file-system

# Build for current platform
make

# Build and extract binaries for multiple platforms
make publish

This creates platform-specific binaries:

  • msfs-linux-amd64 - Linux x86_64

  • msfs-linux-arm64 - Linux ARM64

Docker Deployment

Deploy MSFS using Docker containers:

Dockerfile for MSFS deployment
FROM ubuntu:22.04

RUN apt-get update && apt-get install -y fuse

COPY msfs-linux-amd64 /usr/bin/msfs
COPY mount.msfs /usr/bin/mount.msfs

RUN chmod +x /usr/bin/msfs /usr/bin/mount.msfs

CMD ["/usr/bin/msfs"]

# Build container
docker build -t msfs:latest .

# Run with config from environment
docker run -d \
  --device /dev/fuse \
  --cap-add SYS_ADMIN \
  --security-opt apparmor:unconfined \
  -e MSC_CONFIG=/config/msfs.yaml \
  -v /path/to/config:/config \
  -v /mnt/storage:/mnt/storage:shared \
  msfs:latest

Troubleshooting

Common Issues

FUSE device not found

Error: /dev/fuse: open: no such file or directory

Solution: Load the FUSE kernel module:

sudo modprobe fuse

Permission denied when mounting

Error: fusermount: mount failed: Operation not permitted

Solution: Ensure your user is in the fuse group or run with sudo:

sudo usermod -aG fuse $USER
# Log out and back in for group changes to take effect

Backend not appearing after SIGHUP

Solution: Check logs in /var/log/msfs/ for configuration errors. Ensure new backend configurations are valid.

Cache thrashing with many small files

Solution: Decrease cache_line_size for better cache utilization:

cache_line_size: 262144  # 256 KiB instead of 1 MiB
cache_lines: 16384       # Increase count to maintain total cache size

Debug Mode

Enable verbose logging to diagnose issues:

backends:
  - dir_name: debug-backend
    trace_level: 3  # Maximum verbosity
    # ...

Check daemon logs:

# If using mount helper
tail -f /var/log/msfs/msfs_*.log

# If running manually
./msfs  # Logs go to stdout

Limitations

Current limitations of MSFS:

  • Writes are S3-only: Writable mounts are supported for S3 backends; GCS and AIStore backends remain read-only

  • Backend modifications: Existing backends cannot be modified via SIGHUP; only additions and removals are supported

  • Node-local cache: Each MSFS process owns an independent cache. It is not shared or coordinated across mounts or nodes, even when cache_dir_path points at a shared filesystem

  • Cache lifetime: The cache is discarded on unmount; a remount starts cold

  • No pre-warm API: Data can only be warmed by reading it through a mount that stays running

  • Mount-level authorization: Access is granted per mount, not per UID/GID

  • Qualified scale: Measurements cover a single client at 1-8 threads; see Qualified Scale Boundary

Use-Case Suggestions

Fronting a Remote Bucket with a Fast Tier (AIStore)

MSFS caching is node-local and bounded, so the cost of a first touch is paid per node and again after any remount. Where that cost dominates — many nodes reading the same dataset once, or working sets too large to stay cache-resident — a shared cache tier in front of the remote bucket can absorb it. MSFS does not implement such a tier, but it can read one as a backend.

We measured this with AIStore. The 24-cell read matrix was re-run with MSFS pointed at an AIStore cluster (3 proxies, 3 targets) fronting the same S3 bucket, with the 88 GiB dataset prefetched into the cluster, from a 256 vCPU / 1.5 TiB client:

  • First-touch reads were 2.2x to 37x faster than MSFS reading S3 directly, peaking near 1.7 GiB/s. The gain was largest on small files (23x - 37x at 64 KiB, 8.7x - 19x at 4 KiB) and smallest on large files (2.2x - 6.9x), because a first touch lands on in-datacenter targets instead of crossing the network to the remote bucket.

  • Cache-resident reads were unchanged, which is the expected result: backend_read_file_successes_total stayed flat across the cache-resident pass on all 24 cells, so those reads never reached any backend. Cache-resident throughput is a property of the MSFS cache and the host, not of what sits behind it. A fast tier improves the first touch, not the cached steady state.

  • The 100M-object namespace on the same cluster generated in 1m 27.7s and ingested in about 3m 19s, but only with listing delegated to the underlying store through manifest_gen_backend. Listing the fronted bucket through AIStore timed out at this scale, while listing the store directly did not. Object reads still went through AIStore.

The practical reading: a fast tier is worth evaluating when first-touch cost dominates, and manifest_gen_backend should point at whichever backend lists the namespace fastest, which is not necessarily the one serving reads. This was one client at 1-8 threads and does not establish behavior for many concurrent clients; see Qualified Scale Boundary.

See Also