Deploy Retool Workflows for Self-hosted instances
Learn how to use Retool Workflows with a Self-hosted deployment.
beta
Retool Workflows is currently in public beta.
Retool Workflows is a queue-based execution environment that will be deployed alongside your current Retool deployment. Under the hood, Retool Workflows uses Temporal to orchestrate running each workflow amongst a pool of workers.
Managing the deployment is relatively simple with the provided Retool + Workflows helm chart. Below, we’ll go into more depth around configuration options.
Without any extra configuration, Retool Workflows can process ~5-10 QPS (roughly, 5-10 concurrent blocks executed per second). This can be increased by further provisioning workers and the queue. For production use cases, we highly recommend using a separate Postgres database for the queue.
For Docker-compose deployments
Dev/sandbox environment instructions
Retool provides a docker-compose-with-workflows.yml
file in the retool-onpremise repository. This manages all the required services for Retool and Workflows. For testing, Retool strongly suggests setting up a new environment with dedicated compute and Postgres instances. Retool with Workflows requires a minimum of 12GB of memory and 4 vCPUs if run on a single compute instance. After the infrastructure is created, you can follow the instructions in our README
, with three important caveats:
-
Rename
docker-compose-with-workflows.yml
todocker-compose.yml
(overwriting the original).
Make sure to copy over thedynamicconfig
directory inretool-onpremise
onto your deployment instance. -
Set your Retool release version in
Dockerfile
. We recommend using the most recent tag available in tryretool/backend. Workflows requiresv2.108.0
or later. -
Setup for the temporal service requires the provided Postgres user to have permissions to
CREATE DATABASE
. If this is not possible, you can manually create the required databases in your Postgres cluster:temporal
andtemporal_visibility
.
Production environment instructions
The provided docker-compose
configuration can process 5-10 QPS, and you should be able to use the same configuration as in the test environment. For higher throughput requirements we highly recommend using our Helm chart for deployment. If you have any questions regarding scaling and throughput needs, please reach out to us at [email protected]!
For Kubernetes + Helm deployments
Dev/sandbox environment instructions
The Helm chart repository URL is https://charts.retool.com
and named retool-wf
. You’ll use those images to run workflows. Retool's Helm chart requires Helm >=v3.7
.
For the test environment, we recommend using the same database for the Retool DB and Workflows Queue. This should reliably support ~5 QPS.
Before deploying the helm chart, set the following parameters in retool-helm-workflows/values.yml
.
Main Retool configuration values
- Auth values: e.g., Google
clientId
andclientSecret
(or any otherOAUTH2_
configurations) licenseKey
encryptionKey
jwtSecret
postgresql
SCIM_AUTH
Workflows configuration values
retool-temporal-services-helm
persistence
- add Postgres credentials to bothdefault
andvisibility
workflows
Enabled: true
replicaCount: 1
Production environment instructions
Separate queue database
For a production deployment, we highly recommend using a separate datastore for the Workflows Queue. In particular, it would be best to use AWS Aurora Serverless V2 configured to an ACU (cpu) provision ranging from 0.5 ACU -> 8 ACU. As a rule of thumb, 1 ACU can provide around 10 QPS. The Workflows Queue is write-heavy (around 100:1 write to read operations) and Aurora Serverless can scale to accommodate spikes in traffic without any extra configuration.
In the Helm chart, set retool-temporal-services-helm
persistence
to the new AWS Aurora DB credentials.
Scaling workers
Each workflow worker can process around ~5-10 QPS. Increase the workflow replicaCount
if this is not high enough for your needs. You should be able to scale this to around ~40 QPS, using ~4 workflow workers, before needing any other infrastructure changes.
For Kubernetes deployments without Helm
Retool strongly recommends you use option 1 to deploy to a Kubernetes cluster that doesn't use Helm to manage applications.
Option 1: Use Helm Template
Install the Helm CLI and follow steps 1-4 in the configuring our Helm Chart README. After completing those steps, you can use Helm to generate the Kubernetes manifests and apply them to your cluster by running the command below from the retool-workflows-helm
directory.
export RELEASE_NAME=retool
helm template --values ./values.yaml $RELEASE_NAME retool/retool-wf >> templates.yaml
kubectl apply -f templates.yaml
This approach is preferred because you don't need to configure the same values in multiple places and it makes configuration bugs less likely.
Option 2: Use raw Kubernetes manifests
If you can't use the Helm CLI, Retool provides Kubernetes manifests you can manually configure. These are in the retool-onpremise repository. Most of the required configuration settings can be found in the retool-secrets.template.yaml
and temporal-secrets.template.yaml
files, but you also need to change these values:
POSTGRES_DB
in allretool-*
containers should be set to the DB name you create for Retool state.POSTGRES_HOST
in allretool-*
andtemporal-*
containers should be set to the host of your Postgres instance. This also needs to be configured in thetemporal-configmaps.yaml
file. WhereconnectAddr: "postgres:5432"
is set, replacepostgres
with your host (there are two instances in the file you need to update).POSTGRES_USER
in allretool-*
andtemporal-*
containers should be set to the username for your Postgres instance. This also needs to be configured in thetemporal-configmaps.yaml
file. Whereuser: "postgres"
is set, replacepostgres
with your username (there are two instances in the file you need to update).
After these values are configured, you can run kubectl apply -f -R kubernetes_workflows
to apply the required manifests.
For ECS Deployments
You can deploy Self-hosted Retool and Workflows to ECS Fargate or EC2 using either:
- A Terraform module.
- Cloudformation templates.
The Retool-provided templates extend default functionality by adding:
- The ECS services required for Workflows execution and a Temporal cluster. The deployment uses AWS Service Discovery to communicate between services.
- An AWS Aurora Serverless v2 Postgres RDS instance. This is the persistence layer for the Temporal cluster. Aurora allows for more dynamic scaling, especially for non-constant workflows, but you can opt to use a PostgresQL RDS instance instead.
Deploy using Terraform
Retool recommends using Terraform for ECS deployments using the Retool-provided Terraform module.
1. Use Terraform module
First, update your Terraform configuration to directly use the module.
module "retool" {
source = "[email protected]:tryretool/retool-terraform.git//modules/aws_ecs"
aws_region = "<your-aws-region>"
vpc_id = "<your-vpc-id>"
subnet_ids = [
"<your-subnet-1>",
"<your-subnet-2>"
]
ssh_key_name = "<your-key-pair>"
retool_license_key = "<your-retool-license-key>"
ecs_retool_image = "<desired-retool-version>"
launch_type = "FARGATE" | "EC2"
workflows_enabled = true
# Additional configuration
use_exising_temporal_cluster = "<whether-to-use-your-own-temporal>"
temporal_cluster_config = <config-for-your-own-temporal>
...
}
2. Install module requirements
Next, run terraform init
to install all requirements for the module.
3. Specify the version of Self-hosted Retool
You must replace ecs_retool_image
with your desired Retool Version. The format should be tryretool/backend:X.Y.Z
, where X.Y.Z
is your desired version number. Version 2.111 or later is required for Retool Workflows, though 2.117 or later is strongly recommended.
4. Verify security settings
Ensure that the default security settings in security.tf
matches your specifications. If you need to tighten down access, pass in custom ingress and egress rules into container_egress_rules
, container_ingress_rules
, alb_egress_rules
, and alb_ingress_rules
.
5. Configure additional input variables
Check variables.tf
for any other input variables that may be required.
6. Configure instance settings
Set launch_type
to EC2
if you are not using Fargate. We recommend t3.large
instances for EC2-backed ECS, this avoids Elastic Network Interface (ENI) limits since each ECS Service requires an ENI.
7. Review and apply changes
Run terraform plan
to view all planned changes to your account. Once complete, run terraform apply
to apply the changes and deploy Retool.
You should now find a Load Balancer in your AWS EC2 Console associated with the deployment. The instance address should now be running Retool.
Deploy using Cloudformation
Retool provides Cloudformation templates for ECS on Fargate and ECS on EC2. Both of these templates assume a deployment in private subnets of your VPC (with NAT gateway) along with an Application Load Balancer (ALB) to direct external traffic to the Retool ECS service.
If you previously deployed Retool with an older version of our Cloudformation template it is important to diff the new template with your current state. There should not be any changes to the RetoolRDSInstance
object, since that may cause a delete of your current database instance.
Special considerations
Postgres with TLS
In some cases, you may need to use a Postgres instance with TLS enabled. In this case, please ensure that the following environment variables are set on the temporal
container:
SQL_TLS_ENABLED=true
SQL_TLS=true
SQL_TLS_SKIP_HOST_VERIFICATION=true
In the Helm chart, you can find documentation on how to set these environment variables in the retool-workflows-helm
values.yaml
file.
Airgapped deployments
If you are not seeing the Workflows tab show up, you may need to turn on the feature flag directly on your deployment database. This is often the case when running air-gapped deployments. To manually turn on the feature flag, run the following query on your deployment database:
with rows as (
insert into
experiments (name, "createdAt", "updatedAt")
values
('workflowsBetaAccess', {{ moment() }}, {{ moment() }}),
('workflowsEnableTemporal', {{ moment() }}, {{ moment() }}),
('workflowsPermissionsEnabled', {{ moment() }}, {{ moment() }})
returning id
)
insert into
experiment_strategies (
strategy,
"experimentId",
value,
"targetId",
"enrollmentCriteria",
"createdAt",
"updatedAt"
)
select
'organization',
id,
'true',
1,
{{null}},
{{ moment() }},
{{ moment() }}
from rows
If you aren't executing the script in a Node.js environment, you'll need to replace {{ moment() }}
with the current timestamp in milliseconds.
Bring your own Temporal
You can use your own Temporal Cluster deployment with Retool Workflows. This can lower infrastructure maintenance, server costs, and scaling overhead.
To configure Retool Workflows to use your current Temporal cluster:
- Create a Temporal Namespace in your cluster for each Retool deployment you have. For example,
retool-dev
andretool-prod
if you have development and production Retool deployments. - Configure your Retool deployment to use the above Temporal Cluster and namespace.
- Docker Compose deployments: Set
WORKFLOW_TEMPORAL_CLUSTER_FRONTEND_HOST
,WORKFLOW_TEMPORAL_CLUSTER_FRONTEND_PORT
, andWORKFLOW_TEMPORAL_CLUSTER_NAMESPACE
environment variables in yourdocker.env
. - Helm Chart Deployments: Ensure you are using
v4.11.12
or later of our Helm chart. Set.Values.retool-temporal-services-helm.enabled=false
and.Values.workflows.temporal.enabled=true
. Set your host, port, and namespace configuration at.Values.workflows.temporal
.
- Docker Compose deployments: Set
Updated 4 days ago