Skip to main content

Deploy Self-hosted Retool on AWS Fargate and ECS with CloudFormation

Learn how to deploy Retool on AWS Fargate and ECS with CloudFormation.

You can deploy Self-hosted Retool on AWS Fargate using CloudFormation templates.

Requirements​

To deploy Self-hosted Retool on AWS Fargate and ECS, you need:

Temporal​

Temporal is a distributed system used to schedule and run asynchronous tasks for Retool Workflows. A Self-hosted Retool instance uses a Temporal cluster to facilitate the execution of each workflow amongst a pool of self-hosted workers that make queries and execute code in your VPC. Temporal manages the queueing, scheduling, and orchestration of workflows to guarantee that each workflow block executes in the correct order of the control flow. It does not store any block results by default.

You can use a Retool-managed cluster on Temporal Cloud, which is recommended for most use cases. You can also use an existing self-managed cluster that is hosted on Temporal Cloud or in your own infrastructure. Alternatively, you can spin up a new self-hosted cluster alongside your Self-hosted Retool instance.

Recommended

You should use a Retool-managed cluster if:

  • You are on a version greater than 3.6.14.
  • Your organization is on the Enterprise plan.
  • You don't have an existing cluster which you prefer to use.
  • Your cluster only needs to be used for a Self-hosted Retool deployment.
  • You don't want to manage the cluster directly.
  • You have a single or multi-instance Retool deployment, where each instance requires its own namespace.

Retool admins can enable Retool-managed Temporal. To get started, navigate to the Retool Workflows page and click Enroll now. Once you update your configuration, return to the page and click Complete setup.

Initializing namespace in Retool-managed Temporal

It can take a few minutes to initialize a namespace in Retool-managed Temporal.

Retool-managed Temporal clusters are hosted on Temporal Cloud. Your Self-hosted Retool deployment communicates with the cluster when building, deploying, and executing Retool Workflows. All orchestration data to Temporal is fully encrypted and uses the private encryption key set for your deployment.

1. Verify network configuration​

The CloudFormation template you choose depends on whether you need to deploy with or without public subnets.

  • Use the public template to deploy Self-hosted Retool in VPCs with public subnets.
  • Use the private template to deploy Self-hosted Retool in VPCs without public subnets.

Use the public template to deploy Retool in VPCs with public subnets. This requires the following networking components:

  • One VPC with internet access.
  • Two public subnets.

VPC​

  1. Open the AWS Management Console and navigate to the VPC service to use. Copy the value of the VPC ID.
  2. In the left navigation pane, select Internet Gateways.
  3. In the list of Internet Gateways, find the Internet Gateway associated with your VPC. If there is a record in this table associated with your VPC, the VPC is public.

Subnets​

Verify that your subnets are public. Subnet IDs are required in a later step.

  1. Open the AWS Management Console and navigate to the VPC service.
  2. In the left navigation pane, select Subnets.
  3. Identify at least two subnets which are part of the VPC.
  4. Confirm that the subnets are public subnets. For each subnet, select the Route table tab. Look for an entry where the Destination is 0.0.0.0/0 and the Target is an Internet Gateway (igw-xxxx). If such an entry exists, the subnet is a public subnet.
  5. Confirm that the subnets have auto-assign public IP addresses enabled. For each subnet, select the subnet and go to the Details tab in the lower pane. Look for the Auto-assign public IPv4 address attribute. Confirm that the value is Yes.

2. Configure instance​

Compare your existing template with the newer version

If you previously deployed Self-hosted Retool with an older version of a CloudFormation template, compare your current template with the new one. If the RetoolRDSInstance object has changed, this could delete your current database instance.

Download the private CloudFormation template for either ECS on Fargate or ECS on EC2 from the Self-hosted Retool GitHub repository. Both of these templates assume a deployment in private subnets of your VPC (with NAT gateway) along with an Application Load Balancer (ALB) to direct external traffic to the Retool ECS service.

curl -L -O  https://raw.githubusercontent.com/tryretool/retool-onpremise/master/cloudformation/retool-workflows.fargate.yaml \
&& mv retool-workflows.fargate.yaml fargate.yaml

Configure Temporal​

Comment out the Temporal configuration section in either fargate.yaml or ec2.yaml, depending on the type of deployment used.

Add your license key​

Edit the configuration and include your license key. Replace all values of LICENSE_KEY with your license key.

fargate.yaml
Environment:
  - Name: LICENSE_KEY
    Value: "EXPIRED-LICENSE-KEY-TRIAL"

Set encryption key​

Self-hosted Retool deployments use an encryption key to encrypt:

  • Private keys in the PostgreSQL database of the Self-hosted Retool instance.
  • All data stored in Temporal when deploying Self-hosted Retool.

Set the ENCRYPTION_KEY environment variable for your deployment.

3. Configure CloudFormation service​

Next, log into the AWS Management Console to configure the CloudFormation service:

  1. Navigate to the CloudFormation service and create a new stack.
  2. Upload the fargate.yaml file.
  3. Set the following parameters.
ParameterValue
clusterThe name of your ECS cluster.
desiredCount2
environmentStaging
forcefalse
imageThe Docker tag for the version of Retool to install, such as tryretool/backend:3.33.3-stable.
maximumPercent250
minimumPercent50
subnetIDTwo subnets you identified in the networking requirements section.
vpcIDThe VPC ID to use.

After creating the stack, verify its status is CREATE_COMPLETE.

4. Start the instance​

The Outputs tab of the CloudFormation stack contains the URL of the load balancer running Self-hosted Retool. Once running, your instance is available at http://{load-balancer-url}:3000/auth/signup. When you first visit the page, you must create an admin account.

Additional steps​

Retool strongly recommends you externalize your database, configure SSL, and keep up-to-date with the latest version of Self-hosted Retool. Setting environment variables is often necessary to configure SSO, source control, and other self-hosted features.

Externalize database​

By default, the Retool ECS template creates a new RDS instance to serve as the PostgreSQL database. To set up the ECS deployment to use a different database, you need to update the CloudFormation template by modifying the environment variables for both the RetoolTask and the RetoolJobsRunnerTask.

  1. Modify fargate.yaml, setting the following environment variables for both the RetoolTask and the RetoolJobsRunnerTask.
VariableDescription
POSTGRES_DBThe name of the external database.
POSTGRES_HOSTThe hostname of your external database instance.
POSTGRES_PORTThe port number for your external database instance.
POSTGRES_USERThe external database username.
POSTGRES_PASSWORDThe external database password.
  1. In the AWS Management Console, navigate to the CloudFormation service. Select the Retool stack.
  2. Update the stack. Replace the current template by uploading the new version of fargate.yaml.
  3. Submit changes, and wait for the stack to redeploy. Verify that the CloudFormation stack has a status of CREATE_COMPLETE before continuing.

Update Retool​

  1. Back up your database. Amazon RDS provides two different methods for backing up and restoring your DB instances: automated backups and database snapshots.
  2. Identify the appropriate release version on Docker Hub. See Retool's self-hosted release notes to learn about version-specific features.
  3. In the AWS Management Console, navigate to the CloudFormation service. Select the Retool stack.
  4. Update the stack. Use the current template. On the parameters screen, change the value of Image to the Docker tag for the version of Retool to use, such as tryretool/backend:3.33.3-stable.
  5. Submit changes, and wait for the stack to redeploy. Verify that the CloudFormation stack has a status of CREATE_COMPLETE before continuing.

Add environment variables​

To add environment variables, follow the steps below.

  1. Modify fargate.yaml, setting the environment variables for both the RetoolTask and the RetoolJobsRunnerTask.
  2. In the AWS Management Console, navigate to the CloudFormation service. Select the Retool stack.
  3. Update the stack. Replace the current template by uploading the new version of fargate.yaml from your computer.
  4. Submit changes, and wait for the stack to redeploy. Verify that the CloudFormation stack has a status of CREATE_COMPLETE before continuing.

Configure SSL​

To configure SSL, you must first obtain an SSL certificate. You can either purchase an SSL certificate from a Certificate Authority (CA) or generate a free one using Let's Encrypt. AWS also provides a service called AWS Certificate Manager (ACM) for provisioning, managing, and deploying public and private SSL/TLS certificates.

To add your SLS certificate to your instance:

  1. In the AWS Management Console, navigate to the EC2 service.
  2. Import your SSL certificate to AWS Certificate Manager (ACM).
  3. In the AWS Management Console, navigate to the EC2 service. Under Load Balancing, choose Load Balancers.
  4. Select the load balancer that is part of the Retool deployment.
  5. In the Listeners tab, select Add listener.
  6. Select HTTPS (port 443). Select the ACM, which you created in step 2.
  7. Add the listener. You can now access Retool with a SSL connection.

Mount volumes​

There are several use cases which require the use of volumes. For example, when configuring a gRPC resource, you need to mount a volume containing the protos files to the Retool deployment. Follow these instructions to create a persistent volume and copy files from your local machine to the volume.

First, choose a method to upload files to the EFS volume. You can either:

  • Mount the EFS volume on an EC2 instance and use standard file operations to upload files directly to the mounted directory on the EFS volume.
  • Use AWS DataSync to transfer your local files to an EFS file system by setting up a DataSync agent and configuring a data transfer task.
  • Set up an SFTP server using AWS Transfer for SFTP and configure it to use an EFS file system as the storage backend, then upload files using an SFTP client to store them on the EFS volume.
  1. In the AWS Management Console, navigate to the EFS service.
  2. Create a new file system. Select the same VPC in which you deploy Retool.
  3. Upload files to the EFS volume using your data transfer method of choice.
  4. In fargate.yaml, add the EFS volume as a parameter in the Parameters section.
fargate.yaml
Parameters:
  ...

  EFS:
    Type: AWS::EFS::FileSystem::Id
    Description: Select an existing EFS volume to mount.
  1. In fargate.yaml, add an EFSMountTarget to Resources section.
fargate.yaml
Resources:
  EFSMountTarget:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref 'EFS'
      SecurityGroups: [!Ref 'ALBSecurityGroup']
      SubnetId: !Select [0, !Ref 'SubnetId']
  1. In fargate.yaml, add the volume to the RetoolTask.
fargate.yaml
  RetoolTask:
    Type: AWS::ECS::TaskDefinition
    Properties:
        ...

      ContainerDefinitions:
        ...

        Volumes:
            - Name: efs-volume
            EFSVolumeConfiguration:
                FileSystemId: !Ref 'EFS'
        MountPoints:
          - SourceVolume: efs-volume
            ContainerPath: /efs
            ReadOnly: false  # Set to 'true' if you want read-only access
  1. In the AWS Management Console, navigate to the CloudFormation service. Select the Retool stack.
  2. Update the stack. Replace the current template by uploading the new version of fargate.yaml.
  3. Submit changes, and wait for the stack to redeploy.