Skip to main content

Temporal Clusters for Self-hosted Retool

Learn about Temporal Clusters for self-hosted Retool deployments.

Temporal is a distributed system used to schedule and run asynchronous tasks for Retool Workflows and Retool Agents. A self-hosted Retool instance uses a Temporal cluster to facilitate the execution of each workflow amongst a pool of self-hosted workers that make queries and execute code in your VPC. Temporal manages the queueing, scheduling, and orchestration of workflows to guarantee that each workflow block executes in the correct order of the control flow. It does not store any block results by default.

You can use a Retool-managed cluster on Temporal Cloud, which is recommended for most use cases. You can also use an existing self-managed cluster that is hosted on Temporal Cloud or in your own infrastructure. Alternatively, you can spin up a new self-hosted cluster alongside your self-hosted Retool instance.

Recommended

You should use a Retool-managed cluster if:

  • You don't have an existing cluster which you prefer to use.
  • Your cluster only needs to be used for a self-hosted Retool deployment.
  • You don't want to manage the cluster directly.
  • You have a single or multi-instance Retool deployment, where each instance requires its own namespace.

Retool admins can enable Retool-managed Temporal. To get started, navigate to the Retool Workflows page and click Enroll now. Once you update your configuration, return to the page and click Complete setup.

It can take a few minutes to initialize a namespace in Retool-managed Temporal.

Retool-managed Temporal Clusters are hosted on Temporal Cloud. Your self-hosted Retool deployment communicates with the cluster when building, deploying, and executing Retool Workflows. All orchestration data to Temporal is fully encrypted and uses the private encryption key set for your deployment.

Compare options​

In general, Retool recommends using a Retool-managed Temporal Cluster. You can also use either:

  • A self-managed external cluster: Either Temporal Cloud or self-hosted within your infrastructure.
  • A local cluster: A self-hosted cluster within your self-hosted Retool instance.
Retool-managedself-managed externalLocal
HostingExternally on Temporal CloudEither externally on Temporal Cloud or self-hosted internally within your own VPCLocally as part of the self-hosted Retool instance
AvailabilityAll self-hosted Retool instances (3.6.14+) on an Enterprise planAll self-hosted Retool instances and for purposes outside of Retool.Only to the self-hosted Retool instance
ManagementManaged by Retoolself-managed on Temporal Cloud, self-managed and self-hosted if within your own VPCself-managed and self-hosted
Scale and performanceAt least 4 AWS regions are available to choose fromIf using Temporal Cloud, at least 10 AWS regions are available to choose from; if self-hosted, low latency due to hosting within your infrastructureLow latency due to local deployment alongside self-hosted Retool
Uptime99.99% uptime99.99% uptimeDependent on DevOps
ConfigurationMinimalModerateHigh
CostNo minimum contractSee Temporal Cloud pricingNo contract
SecurityResource-isolated namespace in a Temporal Cloud cluster, in which only orchestration data (workflow IDs and block names) is stored; all data encrypted with private keyOnly orchestration data (internal Workflow IDs, and encrypted Workflow block names, user IDs, Workflow name, host name, and base domain), is stored in Temporal Cloud. All other data remains internal on VPC for self-hosted.All data remains local to the self-hosted Retool instance

Egress​

Whether you use a Retool-managed or self-managed cluster, egress is required for self-hosted Retool to use Temporal. The MAIN_BACKEND, WORKFLOW_TEMPORAL_WORKER, and AGENT_TEMPORAL_WORKER containers must all be able to connect.

If you're using a Retool-managed cluster or a self-managed cluster on Temporal Cloud, your instance must have egress to the internet (public network) on the following ports and domains:

PortDomains
443*.retool.com, *.tryretool.com, *.temporal.io
7233Your Temporal namespace endpoint. Retool-managed clusters use the format org-[sid].atdwz.tmprl.cloud, where sid is your organization's ID (e.g., org-1234567890.atdwz.tmprl.cloud).

Lifecycle​

Temporal orchestrates two processes in a self-hosted Retool deployment.

Workflows​

When a workflow runs:

  1. The MAIN_BACKEND service enqueues the workflow with Temporal.
  2. A WORKFLOW_TEMPORAL_WORKER polls Temporal for tasks, executes each block, and reports results back.
  3. DB_CONNECTOR queries your network-protected resources, or CODE_EXECUTOR runs custom code.

Agent sandbox​

When a user opens an app in edit mode:

  1. MAIN_BACKEND requests a sandbox pod via Temporal.
  2. AGENT_TEMPORAL_WORKER receives the task and coordinates pod lifecycle with the agent sandbox controller.
  3. The sandbox pod runs for the duration of the editing session and is released when the session ends.
Loading diagram...

Data and encryption​

By default, Retool-managed clusters only store encrypted names and IDs of objects in Temporal Cloud—only minimal customer data is sent to the cluster. All encrypted data in a Retool-managed cluster is retained for 14 days. Please reach out if you would like to make changes to this retention period.

All data sent between your deployment and Temporal Cloud is encrypted with your ENCRYPTION_KEY.

Data processing, including query and code execution, occurs within your VPC by workers that you self-host. All coordination and orchestration happens in an isolated namespace within the Retool-managed cluster. No code results or query data is sent outside of your VPC by default.

The process occurs within your VPC. All coordination and orchestration happens in an isolated namespace within the Retool-managed cluster. No query data is sent from your VPC.

Retool-managed clusters perform automatic mTLS rotation. If necessary, you can manually perform this when required.

Local clusters​

If you deploy multiple instances of self-hosted Retool (e.g., staging and production), they must each have their own local cluster namespace.

The Temporal Helm subchart (retool-temporal-services-helm) has been removed from the Retool Helm chart. If your deployment relied on this subchart for a local cluster, migrate to a Retool-managed cluster or provision your own Temporal cluster separately before upgrading to Helm chart version 6.11.4.

A local cluster is one created and run alongside your self-hosted Retool instance. Unlike a self-managed cluster in your VPC, a local cluster is only available to its own Retool instance.

Local clusters enable you to get started quickly without committing to a dedicated Temporal cluster, but can become complex to scale and manage. A Retool-managed cluster, or a self-managed cluster on Temporal Cloud or in your VPC, provides a centralized cluster that each instance can use via a resource-isolated namespace.

Large scale deployments​

Workflows can be deployed to scale to hundreds of concurrent block runs per second. There are three Temporal options for running more than 10 workflows per second:

  • Use a Retool-managed Temporal Cluster.
  • Use a self-managed Temporal Cluster.
  • Use Cassandra as the Temporal datastore.

Contact support to discuss a deployment strategy for high-scale workflows.

Telemetry​

Retool uses a Prometheus Metrics exporter to expose Temporal Worker metrics. This is specified in the Temporal worker's runtime options.

telemetryOptions: {
  metrics: {
    prometheus: { bindAddress: '0.0.0.0:9090' }
  }
}

You can optionally specify the endpoint for an OpenTelemetry collector if you prefer to use one with the WORKFLOW_TEMPORAL_OPENTELEMETRY_COLLECTOR environment variable.

Metrics are available for scraping using the /metrics route.

For more information about telemetry, refer to Collect self-hosted telemetry data.