Deploy Retool Workflows for Self-hosted instances

🎉

beta

Retool Workflows is currently in public beta.

Retool Workflows is a queue-based execution environment that will be deployed alongside your current Retool deployment. Under the hood, Retool Workflows uses Temporal to orchestrate running each workflow amongst a pool of workers.

Managing the deployment is relatively simple with the provided Retool + Workflows helm chart. Below, we’ll go into more depth around configuration options.

Without any extra configuration, Retool Workflows can process ~5-10 QPS (roughly, 5-10 concurrent blocks executed per second). This can be increased by further provisioning workers and the queue. For production use cases, we highly recommend using a separate Postgres database for the queue.

For Docker-compose deployments

Dev/sandbox environment instructions

Retool provides a docker-compose-with-workflows.yml file in the retool-onpremise repository. This manages all the required services for Retool and Workflows. For testing, Retool strongly suggests setting up a new environment with dedicated compute and Postgres instances. Retool with Workflows requires a minimum of 12GB of memory and 4 vCPUs if run on a single compute instance. After the infrastructure is created, you can follow the instructions in our README, with three important caveats:

1. Rename docker-compose-with-workflows.yml to docker-compose.yml (overwriting the original).
   Make sure to copy over the dynamicconfig directory in retool-onpremise onto your deployment instance.

2. Set your Retool release version in Dockerfile. We recommend using the most recent tag available in tryretool/backend. Workflows requires v2.108.0 or later.

3. Setup for the temporal service requires the provided Postgres user to have permissions to CREATE DATABASE. If this is not possible, you can manually create the required databases in your Postgres cluster: temporal and temporal_visibility.

Production environment instructions

The provided docker-compose configuration can process 5-10 QPS, and you should be able to use the same configuration as in the test environment. For higher throughput requirements we highly recommend using our Helm chart for deployment. If you have any questions regarding scaling and throughput needs, please reach out to us at [email protected]!

For Kubernetes + Helm deployments

Dev/sandbox environment instructions

The Helm chart repository URL is https://charts.retool.com and named retool-wf. You’ll use those images to run workflows. Retool's Helm chart requires Helm >=v3.7.

For the test environment, we recommend using the same database for the Retool DB and Workflows Queue. This should reliably support ~5 QPS.

Before deploying the helm chart, set the following parameters in retool-helm-workflows/values.yml.

Main Retool configuration values

• Auth values: e.g., Google clientId and clientSecret (or any other OAUTH2_ configurations)
• licenseKey
• encryptionKey
• jwtSecret
• postgresql
• SCIM_AUTH

Workflows configuration values

• retool-temporal-services-helm
  • persistence - add Postgres credentials to both default and visibility
• workflows
  • Enabled: true
  • replicaCount: 1

Production environment instructions

Separate queue database

For a production deployment, we highly recommend using a separate datastore for the Workflows Queue. In particular, it would be best to use AWS Aurora Serverless V2 configured to an ACU (cpu) provision ranging from 0.5 ACU -> 8 ACU. As a rule of thumb, 1 ACU can provide around 10 QPS. The Workflows Queue is write-heavy (around 100:1 write to read operations) and Aurora Serverless can scale to accommodate spikes in traffic without any extra configuration.

In the Helm chart, set retool-temporal-services-helm persistence to the new AWS Aurora DB credentials.

Scaling workers

Each workflow worker can process around ~5-10 QPS. Increase the workflow replicaCount if this is not high enough for your needs. You should be able to scale this to around ~40 QPS, using ~4 workflow workers, before needing any other infrastructure changes.

For Kubernetes deployments without Helm

Retool strongly recommends you use option 1 to deploy to a Kubernetes cluster that doesn't use Helm to manage applications.

Option 1: Use Helm Template

Install the Helm CLI and follow steps 1-4 in the configuring our Helm Chart README. After completing those steps, you can use Helm to generate the Kubernetes manifests and apply them to your cluster by running the command below from the retool-workflows-helm directory.

export RELEASE_NAME=retool
helm template --values ./values.yaml $RELEASE_NAME retool/retool-wf  >> templates.yaml
kubectl apply -f templates.yaml

This approach is preferred because you don't need to configure the same values in multiple places and it makes configuration bugs less likely.

Option 2: Use raw Kubernetes manifests

If you can't use the Helm CLI, Retool provides Kubernetes manifests you can manually configure. These are in the retool-onpremise repository. Most of the required configuration settings can be found in the retool-secrets.template.yaml and temporal-secrets.template.yaml files, but you also need to change these values:

• POSTGRES_DB in all retool-* containers should be set to the DB name you create for Retool state.
• POSTGRES_HOST in all retool-* and temporal-* containers should be set to the host of your Postgres instance. This also needs to be configured in the temporal-configmaps.yaml file. Where connectAddr: "postgres:5432" is set, replace postgres with your host (there are two instances in the file you need to update).
• POSTGRES_USER in all retool-* and temporal-* containers should be set to the username for your Postgres instance. This also needs to be configured in the temporal-configmaps.yaml file. Where user: "postgres" is set, replace postgres with your username (there are two instances in the file you need to update).

After these values are configured, you can run kubectl apply -f -R kubernetes_workflows to apply the required manifests.

For ECS Deployments

You can deploy Self-hosted Retool and Workflows to ECS Fargate or EC2 using either:

• A Terraform module.
• Cloudformation templates.

The Retool-provided templates extend default functionality by adding:

• The ECS services required for Workflows execution and a Temporal cluster. The deployment uses AWS Service Discovery to communicate between services.
• An AWS Aurora Serverless v2 Postgres RDS instance. This is the persistence layer for the Temporal cluster. Aurora allows for more dynamic scaling, especially for non-constant workflows, but you can opt to use a PostgresQL RDS instance instead.

Deploy using Terraform

Retool recommends using Terraform for ECS deployments using the Retool-provided Terraform module.

1. Use Terraform module

First, update your Terraform configuration to directly use the module.

module "retool" {
    source = "[email protected]:tryretool/retool-terraform.git//modules/aws_ecs"

    aws_region = "<your-aws-region>"
    vpc_id = "<your-vpc-id>"
    subnet_ids = [
        "<your-subnet-1>",
        "<your-subnet-2>"
    ]
    ssh_key_name = "<your-key-pair>"
    retool_license_key = "<your-retool-license-key>"
    ecs_retool_image = "<desired-retool-version>"
    launch_type = "FARGATE" | "EC2"
    workflows_enabled = true

    # Additional configuration
    use_exising_temporal_cluster = "<whether-to-use-your-own-temporal>"
    temporal_cluster_config = <config-for-your-own-temporal>
    ...
}

2. Install module requirements

Next, run terraform init to install all requirements for the module.

3. Specify the version of Self-hosted Retool

You must replace ecs_retool_image with your desired Retool Version. The format should be tryretool/backend:X.Y.Z, where X.Y.Z is your desired version number. Version 2.111 or later is required for Retool Workflows, though 2.117 or later is strongly recommended.

4. Verify security settings

Ensure that the default security settings in security.tf matches your specifications. If you need to tighten down access, pass in custom ingress and egress rules into container_egress_rules, container_ingress_rules, alb_egress_rules, and alb_ingress_rules.

5. Configure additional input variables

Check variables.tf for any other input variables that may be required.

6. Configure instance settings

Set launch_type to EC2 if you are not using Fargate. We recommend t3.large instances for EC2-backed ECS, this avoids Elastic Network Interface (ENI) limits since each ECS Service requires an ENI.

7. Review and apply changes

Run terraform plan to view all planned changes to your account. Once complete, run terraform apply to apply the changes and deploy Retool.

You should now find a Load Balancer in your AWS EC2 Console associated with the deployment. The instance address should now be running Retool.

Deploy using Cloudformation

Retool provides Cloudformation templates for ECS on Fargate and ECS on EC2. Both of these templates assume a deployment in private subnets of your VPC (with NAT gateway) along with an Application Load Balancer (ALB) to direct external traffic to the Retool ECS service.

If you previously deployed Retool with an older version of our Cloudformation template it is important to diff the new template with your current state. There should not be any changes to the RetoolRDSInstance object, since that may cause a delete of your current database instance.

Special considerations

Postgres with TLS

In some cases, you may need to use a Postgres instance with TLS enabled. In this case, please ensure that the following environment variables are set on the temporal container:

SQL_TLS_ENABLED=true
SQL_TLS=true
SQL_TLS_SKIP_HOST_VERIFICATION=true

In the Helm chart, you can find documentation on how to set these environment variables in the retool-workflows-helm values.yaml file.

Airgapped deployments

If you are not seeing the Workflows tab show up, you may need to turn on the feature flag directly on your deployment database. This is often the case when running air-gapped deployments. To manually turn on the feature flag, run the following query on your deployment database:

with rows as (
  insert into 
      experiments (name, "createdAt", "updatedAt")
  values
    ('workflowsBetaAccess', {{ moment() }}, {{ moment() }}),
    ('workflowsEnableTemporal', {{ moment() }}, {{ moment() }}),
    ('workflowsPermissionsEnabled', {{ moment() }}, {{ moment() }})
    returning id
)
insert into
  experiment_strategies (
    strategy,
    "experimentId",
    value,
    "targetId",
    "enrollmentCriteria",
    "createdAt",
    "updatedAt"
  )
select
    'organization',
    id,
    'true',
    1,
    {{null}},
    {{ moment() }},
    {{ moment() }}
from rows

If you aren't executing the script in a Node.js environment, you'll need to replace {{ moment() }} with the current timestamp in milliseconds.

Bring your own Temporal

You can use your own Temporal Cluster deployment with Retool Workflows. This can lower infrastructure maintenance, server costs, and scaling overhead.

To configure Retool Workflows to use your current Temporal cluster:

1. Create a Temporal Namespace in your cluster for each Retool deployment you have. For example, retool-dev and retool-prod if you have development and production Retool deployments.
2. Configure your Retool deployment to use the above Temporal Cluster and namespace.
   • Docker Compose deployments: Set WORKFLOW_TEMPORAL_CLUSTER_FRONTEND_HOST, WORKFLOW_TEMPORAL_CLUSTER_FRONTEND_PORT, and WORKFLOW_TEMPORAL_CLUSTER_NAMESPACE environment variables in your docker.env.
   • Helm Chart Deployments: Ensure you are using v4.11.12 or later of our Helm chart. Set .Values.retool-temporal-services-helm.enabled=false and .Values.workflows.temporal.enabled=true. Set your host, port, and namespace configuration at .Values.workflows.temporal.