System architecture of self-hosted Retool deployments
Learn about Retool's self-hosted architecture.
Organizations that have on-premise requirements can deploy the Retool platform on their own infrastructure. Each deployment instance uses a distributed set of containers with services for different functions. These work together to securely run the platform within your VPN or VPC.
Containers and services
A standard deployment instance uses the following containers and services that interact with each other. The SERVICE_TYPE environment variable values determine what services a container runs.
| Container | Image | Repository | Services |
|---|---|---|---|
| api | backend | tryretool/backend | MAIN_BACKENDDB_CONNECTORDB_SSH_CONNECTOR |
| jobs-runner | backend | tryretool/backend | JOBS_RUNNER |
| workflows-worker | backend | tryretool/backend | WORKFLOW_TEMPORAL_WORKER |
| workflows-backend | backend | tryretool/backend | WORKFLOW_BACKENDDB_CONNECTORDB_SSH_CONNECTOR |
| code-executor | code-executor-service | tryretool/code-executor-service | No service type. |
api
The api container manages the core functionality for a Retool deployment instance, such as:
- Frontend interactions (e.g., building apps and workflows).
- Hosting apps and workflows.
- Managing users and permissions.
api runs the following services:
MAIN_BACKEND
The core service for a Retool deployment instance. It handles most logic for frontend interactions.
DB_CONNECTOR
Handles query requests to resources (databases, APIs, etc.).
DB_SSH_CONNECTOR
Manages connections for DB_CONNECTOR.
Network ingress and egress
api requires ingress from client interactions, such as loading an app in a browser. These are typically handled by a load balancer (e.g., nginx) which proxies the requests.
The following table displays ingress and egress requirements for api.
| Container | Network ingress | Network egress |
|---|---|---|
| code-executor | ||
| postgres | ||
| retooldb-postgres | ||
| Temporal | ||
| Resources |
Replicas
You can replicate api as you scale to manage higher volumes of workflow traffic.
jobs-runner
The jobs-runner container manages background tasks, runs database migrations for Retool version upgrades, and Source Control. It runs the following service:
JOBS_RUNNER
Performs database migrations and manages Source Control.
Network ingress and egress
The following table displays ingress and egress requirements for jobs-runner.
| Container | Network ingress | Network egress |
|---|---|---|
| postgres |
Replicas
You must not replicate jobs-runner. It performs tasks and migrations that must be performed as a single container.
workflows-worker
The workflows-worker container continuously polls the Temporal cluster for tasks required to either start or execute blocks within a workflow. It makes requests to code-executor to execute blocks and process results, then reports back to Temporal to continue or complete workflow execution.
workflows-worker runs the following service.
WORKFLOW_TEMPORAL_WORKER
Polls the Temporal cluster for tasks required to start or execute workflow blocks.
Network ingress and egress
The following table displays ingress and egress requirements for workflows-worker.
| Container | Network ingress | Network egress |
|---|---|---|
| code-executor | ||
| postgres | ||
| retooldb-postgres | ||
| Temporal | ||
| Resources |
Replicas
You can replicate workflows-worker as you scale to manage higher volumes of workflow traffic.
workflows-backend
The workflows-backend container receives and processes requests from code-executor to process workflows. It also handles the upload of workflows logs, block results, and run status updates.
WORKFLOW_BACKEND
Handles query, retry, and asynchronous logic.
DB_CONNECTOR
Handles query requests to resources (databases, APIs, etc.).
DB_SSH_CONNECTOR
Manages connections for DB_CONNECTOR.
Network ingress and egress
The following table displays ingress and egress requirements for workflows-backend.
| Container | Network ingress | Network egress |
|---|---|---|
| code-executor | ||
| postgres | ||
| retooldb-postgres | ||
| Temporal | ||
| Resources |
Replicas
You can replicate workflows-backend as you scale to manage higher volumes of workflow traffic.
code-executor
The code-executor container executes single block runs from the Workflows IDE, multiple blocks as part of a Workflow execution, and any arbitrary user code written in a Workflow. It executes JavaScript and Python code directly, or makes requests to workflows-backend to run resource queries against databases, APIs, etc.
By default, the code-executor uses nsjail to sandbox code execution. nsjail requires privileged container access. If your deployment does not support privileged access, the code executor automatically detects this and runs code without sandboxing. Without sandboxing, use of custom JS libraries and custom Python libraries is not allowed.
Network ingress and egress
The following table displays ingress and egress requirements for code-executor.
| Container | Network ingress | Network egress |
|---|---|---|
| api | ||
| workflows-backend | ||
| workflows-worker |
Replicas
You can replicate code-executor as you scale to manage higher volumes of workflow traffic.
postgres
postgres manages the internal PostgreSQL database in which an instance stores its apps, resources, users, permissions, and internal data. All deployment instances contain a default postgres container for testing purposes. You must externalize this database before using the deployment instance in production.
retooldb-postgres
You can optionally configure your deployment to use Retool Database with a separate PostgreSQL database. You can use retooldb-postgres to get started but Retool recommends you migrate to an externally hosted database for use in production.
Other dependencies
Self-hosted deployment instances have additional dependencies.
Resources
Each data source you use with Retool is represented by a resource, such as APIs and external databases. Your deployment instance must have access to any of its resources to interact with data.
Temporal
Temporal is a distributed system used to schedule and run asynchronous tasks for Retool Workflows. A Self-hosted Retool instance uses a Temporal cluster to facilitate the execution of each workflow amongst a pool of self-hosted workers that make queries and execute code in your VPC. Temporal manages the queueing, scheduling, and orchestration of workflows to guarantee that each workflow block executes in the correct order of the control flow. It does not store any block results by default.
You can use a Retool-managed cluster on Temporal Cloud, which is recommended for most use cases. You can also use an existing self-managed cluster that is hosted on Temporal Cloud or in your own infrastructure. Alternatively, you can spin up a new self-hosted cluster alongside your Self-hosted Retool instance.
Docker images
Retool uses Docker images that configure the containers and their respective services. Each container runs one or more services to distribute functionality. This approach allows for efficient scaling of a Retool deployment instance as needs grow.
Retool maintains two image repositories on Docker Hub:
- The backend image is used for most containers and services.
- The code-executor-service image is used to execute blocks of code in Retool Workflows.
Each Docker Hub repository tag corresponds to a particular version of Retool. You specify the version to use when deploying or upgrading a release. For instance:
-
tryretool/backend:3.75.3-stable -
tryretool/code-executor-service:3.75.3-stable