Configure Data Storage#

Prerequisites

Before configuring OSMO to use data storage, ensure you have created the required data storage: Create Data Storage

Workflow Logs#

Run the following commands to configure the workflow spec and log storage location in OSMO. Make sure to replace the placeholders with the actual values.

$ cat << EOF > /tmp/workflow_log_config.json
{
  "workflow_log": {
      "credential": {
          "endpoint": "s3://my_bucket/workflows",
          "access_key_id": "EXAMPLE_ACCESS_KEY_ID",
          "access_key": "EXAMPLE_ACCESS_KEY",
          "region": "us-east-1",
          "override_url": "http://minio:9000" # Optional: HTTP endpoint for non-AWS S3
      }
  }
}
EOF

Note

override_url is optional. Set it only when using non-AWS S3-compatible services (MinIO, Ceph, LocalStack). Leave it empty or omit it for standard AWS S3.

Then, update the workflow configuration using the OSMO CLI. Please make sure you’re logged in to your OSMO instance before running the following command.

$ osmo config update WORKFLOW --file /tmp/workflow_log_config.json

Workflow Data#

Configure the storage location for intermediate data that OSMO uses to pass outputs between workflow tasks. Replace the placeholders with your actual values.

$ cat << EOF > /tmp/workflow_data_config.json
{
  "workflow_data": {
      "credential": {
          "endpoint": "s3://my_bucket/workflows",
          "access_key_id": "EXAMPLE_ACCESS_KEY_ID",
          "access_key": "EXAMPLE_ACCESS_KEY",
          "region": "us-east-1",
          "override_url": "http://minio:9000" # Optional: HTTP endpoint for non-AWS S3
      }
  }
}
EOF

Note

override_url is optional. Set it only when using non-AWS S3-compatible services (MinIO, Ceph, LocalStack). Leave it empty or omit it for standard AWS S3.

Then, update the workflow data configuration using the OSMO CLI. Please make sure you’re logged in to your OSMO instance before running the following command.

$ osmo config update WORKFLOW --file /tmp/workflow_data_config.json