Skip to main content
This page is unlisted and can only be accessed directly via URL. It is excluded from the site navigation and search results.

Migrate workflow block storage to Google Cloud Storage

Step-by-step guide for storing workflow block results in Google Cloud Storage for self-hosted instances.

Workflows Google Cloud Storage Availability
Self-hosted Edge 3.375 or laterClosed beta

This guide explains how to store Workflow block results in Google Cloud Storage (GCS) instead of PostgreSQL for self-hosted instances.

With this configuration:

  • New Workflow block results are written to GCS.
  • Existing PostgreSQL-backed results remain readable until they age out based on your retention settings.
  • Retool uses GCS through the S3-compatible interoperability API (HMAC key and secret).

Prerequisites​

Before you begin, ensure you have:

  • A Google Cloud Platform (GCP) project with Cloud Storage enabled.
  • Permissions in GCP to create service accounts and hash-based message authentication code (HMAC) keys.
  • Admin permissions on your Retool instance.

Step 1: Create a GCS bucket​

Create a new bucket in your GCP project with the following recommended settings:

  • Location type: Region.
  • Storage class: Standard.
  • Enforce public access prevention: Enabled.
  • Access control: Uniform.
  • Data protection: Soft delete policy with the default 7-day retention.

Avoid underscores (_) in the bucket name. Use hyphens (-) instead.

Step 2: Create a service account for bucket access​

Create a new service account and grant it object-level access to the bucket. Retool recommends using the Storage Object User role.

You can optionally scope access to only the target bucket with an IAM condition as shown in the following example:

Example
resource.name.startsWith("projects/_/buckets/BUCKET_NAME_HERE")

Step 3: Generate a GCS interoperability key pair​

To enable access to your Google Cloud Storage data, follow these steps to generate a GCS interoperability key pair:

  1. In GCP, go to Cloud Storage > Settings > Interoperability.
  2. Note the Request endpoint value (typically https://storage.googleapis.com). This value is used for the WORKFLOW_S3_ENDPOINT environment variable.
  3. Under Access keys for service accounts, create a key for the service account from Step 2.
  4. Save the generated Access key and Secret. You will use these values for the following environment variables: WORKFLOW_S3_ACCESS_KEY_ID and WORKFLOW_S3_SECRET_ACCESS_KEY.

Step 4: Configure Retool environment variables​

Set the following environment variables for all workflow-related services (api, workflows-worker, and workflows-backend):

VariableValue
WORKFLOW_BLOCK_STORAGE_LOCATIONgcs.
WORKFLOW_S3_BUCKETYour GCS bucket name.
WORKFLOW_S3_ENDPOINTUsually https://storage.googleapis.com.
WORKFLOW_S3_REGIONus (GCS ignores this value, but Retool requires it).
WORKFLOW_S3_ACCESS_KEY_IDAccess key from Step 3.
WORKFLOW_S3_SECRET_ACCESS_KEYSecret from Step 3.

For Kubernetes with Helm deployments, store credentials in a secret and reference them in your chart values.

Step 5: Deploy and verify​

After deployment, confirm the new config is active by completing the following steps:

  1. Ensure the relevant Retool pods restarted with updated environment variables.
  2. Run a workflow.
  3. Verify the run succeeds and block results are visible in the UI (including after a refresh).
  4. Verify objects exist in your bucket under:
workflow_block_results/$numericSpaceId/$workflowId/$workflowRunId/

Retool recommends rolling out this change incrementally. For example, implement it first on a development instance, then on a testing or staging instance, and lastly on a production instance.

This setting applies organization-wide, not per Space.

Reclaim PostgreSQL disk space (optional)​

After all PostgreSQL-backed Workflow block results have aged out based on your maximum retention window, you can reclaim space by truncating workflow_block_results.

This is often required to reclaim TOAST space at the OS level.

Truncating workflow_block_results is irreversible. Coordinate with your database and Retool operators before running cleanup in production.

Roll back to PostgreSQL​

To revert back to PostgreSQL block storage:

  1. Set WORKFLOW_BLOCK_STORAGE_LOCATION=postgres.
  2. Redeploy and restart services.

Keep all WORKFLOW_S3_* variables configured after rollback so users can still open runs previously stored in GCS.

Do not remove WORKFLOW_S3_* variables if any historical workflow runs were stored in GCS.