Rsync

Rsync data to and from a running task in your workflow using the rsync command. For detailed CLI options, see osmo workflow rsync.

Caution

We currently do not support deleting files/directories in a workflow.

Hint

If task is not provided, the operation will target the lead task of the first group.

Hint

/osmo/run/workspace is always available as a remote path.

Upload

Upload files or directories from your local machine to a running task. By default, the upload runs in the foreground and exits once complete.

$ osmo workflow rsync upload wf-id ~/my/path:/osmo/run/workspace

Daemon mode

Pass --daemon to start a background daemon that continuously monitors the source path and uploads changes to the remote task.

$ osmo workflow rsync upload wf-id ~/my/path:/osmo/run/workspace --daemon
Rsync daemon started in detached process: PID 80754
To view daemon logs: tail -f ~/.local/state/osmo/rsync/rsync_daemon_wf-id_task-name.log

Daemon logs

The daemon will output logs to the designated log file.

$ tail -f ~/.local/state/osmo/rsync/rsync_daemon_wf-id_task-name.log
2025-05-29 10:38:04,517 - 26720 - rsync.py:854 - osmo.rsync - INFO - Starting rsync daemon...
2025-05-29 10:38:04,521 - 26720 - rsync.py:947 - osmo.rsync - INFO - Polling task...
2025-05-29 10:38:04,666 - 26720 - rsync.py:980 - osmo.rsync - INFO - Task is in running state...
2025-05-29 10:38:04,672 - 26720 - rsync.py:377 - osmo.rsync - INFO - Starting rsync client...
2025-05-29 10:38:04,672 - 26720 - rsync.py:553 - osmo.rsync - INFO - Starting rsync port forwarding...
2025-05-29 10:38:05,421 - 26720 - rsync.py:433 - osmo.rsync - INFO - Uploading /my/path
2025-05-29 10:38:05,947 - 26720 - rsync.py:482 - osmo.rsync - INFO - Rsync upload completed successfully for wf-id/task-name
2025-05-29 10:39:17,517 - 26720 - rsync.py:736 - osmo.rsync - INFO - Path event handler (/my/path) detected changes...
2025-05-29 10:39:55,121 - 26720 - rsync.py:433 - osmo.rsync - INFO - Uploading /my/path
2025-05-29 10:39:55,694 - 26720 - rsync.py:482 - osmo.rsync - INFO - Rsync upload completed successfully for wf-id/task-name

Download

Download files or directories from a running task to your local machine. The download runs in the foreground and exits once complete. The local destination path is treated as a directory — downloaded files are placed inside it.

$ osmo workflow rsync download wf-id /osmo/run/workspace/my_data:/tmp/output

To download a single file:

$ osmo workflow rsync download wf-id /osmo/run/workspace/results.csv:/tmp/output

Daemon status

To get the status of all rsync daemons, use the osmo workflow rsync status command.

$ osmo workflow rsync status

Workflow ID   Task Name   PID     Status    Last Synced                  Local Path   Remote Path           Log File
====================================================================================================================================================================
wf-id         task-name   26720   RUNNING   2025-05-29T10:39:55.696803   /my/path     /osmo/run/workspace   ~/.local/state/osmo/rsync/rsync_daemon_wf-id_task-name.log

Stopping daemon(s)

To stop a specific daemon, use osmo workflow rsync stop wf-id --task task-name.

To stop all daemons for a workflow, use osmo workflow rsync stop wf-id.

Finally, to stop all daemons, use osmo workflow rsync stop.

$ osmo workflow rsync stop
Are you sure you want to stop all running daemons?

        * wf-id_1/task-name
        * wf-id_2/task-name

[y/N] y
Stopping rsync daemon wf-id_1/task-name
Stopping rsync daemon wf-id_2/task-name