Configure Data Storage#

Prerequisites

Before configuring OSMO to use data storage, ensure you have created the required data storage: Create Data Storage

Workflow Logs#

Run the following commands to configure the workflow spec and log storage location in OSMO. Make sure to replace the placeholders with the actual values.

# URI of your s3 bucket e.g. s3://my_bucket
$ export BACKEND_URI=...

$ export ACCESS_KEY_ID=...
$ export ACCESS_KEY=...

# Bucket Region
$ export REGION=...

$ cat << EOF > /tmp/workflow_log_config.json
{
  "workflow_log": {
      "credential": {
          "endpoint": "'$BACKEND_URI'",
          "access_key_id": "'$ACCESS_KEY_ID'",
          "access_key": "'$ACCESS_KEY'",
          "region": "'$REGION'"
      }
  }
}
EOF

Then, update the workflow configuration using the OSMO CLI. Please make sure you’re logged in to your OSMO instance before running the following command.

$ osmo config update WORKFLOW --file /tmp/workflow_log_config.json

Workflow Data#

Configure the storage location for intermediate data that OSMO uses to pass outputs between workflow tasks. Replace the placeholders with your actual values.

# URI of your s3 bucket e.g. s3://my_bucket
$ export BACKEND_URI=...

$ export ACCESS_KEY_ID=...
$ export ACCESS_KEY=...

# Bucket Region
$ export REGION=...

$ cat << EOF > /tmp/workflow_data_config.json
{
  "workflow_data": {
      "credential": {
          "endpoint": "'$BACKEND_URI'",
          "access_key_id": "'$ACCESS_KEY_ID'",
          "access_key": "'$ACCESS_KEY'",
          "region": "'$REGION'"
      }
  }
}
EOF

Then, update the workflow data configuration using the OSMO CLI. Please make sure you’re logged in to your OSMO instance before running the following command.

$ osmo config update WORKFLOW --file /tmp/workflow_data_config.json