Rsync Configuration#

Enable rsync integration to sync files between user workstations and running workflow tasks. This configuration is optional but enables live code editing and real-time collaboration during development on a remote machine.

Why Use Rsync?#

Rsync integration provides seamless file synchronization during workflow execution:

✓ Live Development

Edit code locally and sync changes to running tasks in real-time without restarting workflows.

✓ Bandwidth Control

Prevent network saturation with configurable rate limits for uploads and downloads.

✓ Flexible Sync Paths

Define custom mount points beyond the default /osmo/run/workspace for specialized workflows.

How It Works#

Sync Flow#

1. Local Edit ✏️

Modify files

Change code locally

2. Daemon Detects 👁️

Monitor changes

Wait for debounce delay

3. Auto Sync 🔄

Transfer files

Push to running task

Key Settings#

  • Bandwidth Limits: Control upload/download rates to prevent network congestion

  • Debounce Delay: Wait period after file changes before syncing (batches rapid edits)

  • Poll Interval: How often to check task status

  • Allowed Paths: Additional directories accessible for sync operations

Note

For detailed configuration fields and defaults, see Rsync Plugin.

Practical Guide#

Enabling Rsync#

📄 Edit in your Helm values file

Everything on this section is to be added in your Helm values file under services.configs. Apply changes with helm upgrade.

Step 1: Enable Rsync in Helm Values

Add the plugins_config.rsync block under services.configs.workflow:

services:
  configs:
    enabled: true
    workflow:
      plugins_config:
        rsync:
          enabled: true
          enable_telemetry: false
          read_bandwidth_limit: 2621440
          write_bandwidth_limit: 2621440
          allowed_paths: {}
          daemon_debounce_delay: 30
          daemon_poll_interval: 120
          daemon_reconcile_interval: 60
          client_upload_rate_limit: 2097152

Step 2: Configure Settings

Bandwidth Configuration

Control network usage with rate limits (bytes per second):

services:
  configs:
    workflow:
      plugins_config:
        rsync:
          read_bandwidth_limit: 5242880
          write_bandwidth_limit: 5242880
          client_upload_rate_limit: 4194304

Limits:

  • read_bandwidth_limit: 5 MB/s container read (5242880 bytes/sec)

  • write_bandwidth_limit: 5 MB/s container write (5242880 bytes/sec)

  • client_upload_rate_limit: 4 MB/s client upload (4194304 bytes/sec)

Recommended Values:

  • Low-bandwidth: 1 MB/s (1048576)

  • Medium-bandwidth: 5 MB/s (5242880)

  • High-bandwidth: 10 MB/s (10485760)

Timing Configuration

Adjust sync timing for your workflow patterns:

services:
  configs:
    workflow:
      plugins_config:
        rsync:
          daemon_debounce_delay: 10
          daemon_poll_interval: 60
          daemon_reconcile_interval: 30

Timing Values:

  • daemon_debounce_delay: 10 seconds (faster sync)

  • daemon_poll_interval: 60 seconds (check status every minute)

  • daemon_reconcile_interval: 30 seconds (reconcile uploads)

Guidelines:

  • Debounce Delay: Lower for rapid iteration (10s), higher for stability (60s)

  • Poll Interval: Balance between responsiveness and API load

  • Reconcile Interval: How often to verify sync consistency

Remote Path Configuration

Add additional sync destinations beyond /osmo/run/workspace:

services:
  configs:
    workflow:
      plugins_config:
        rsync:
          allowed_paths:
            dataset:
              path: /mnt/shared/datasets/
              writable: true
            models:
              path: /mnt/models/
              writable: false

Users can then sync to new remote paths

$ osmo workflow rsync wf-id ~/my/path:/mnt/shared/datasets/
$ osmo workflow rsync wf-id ~/my/path:/mnt/models/

Step 3: Apply

helm upgrade osmo deployments/charts/service -f my-values.yaml

Troubleshooting#

Rsync Not Working - Verify enabled: true in configuration - Check workflow was started AFTER configuration update - Confirm rsync daemon is running: osmo rsync status

Slow Sync Performance - Increase bandwidth limits if network capacity allows - Reduce debounce delay for faster detection - Check for network congestion or firewall rules

File Conflicts - Review reconcile interval (lower = more frequent consistency checks) - Ensure only one user is syncing to each task - Check file permissions on remote paths

Path Not Accessible - Verify path exists in allowed_paths configuration - Confirm path has correct permissions (writable: true/false) - Remember /osmo/run/workspace is always available

Tip

Best Practices

  • Start with default settings and adjust based on usage patterns

  • Set bandwidth limits appropriate to your network capacity

  • Use higher debounce delays (30-60s) for production to reduce API calls

  • Enable telemetry initially to monitor sync behavior

  • Test rsync with small files before large datasets

  • Document custom paths in team onboarding materials

Warning

Rsync changes apply only to workflows started after the configuration update. Restart running workflows to pick up new settings.

See also