Skip to main content

Configure telemetry for self-hosted Retool

Configure the Retool telemetry collector, choose which data Retool receives, and send telemetry data to your own monitoring destinations.

Organizations with self-hosted instances can collect telemetry data using either:

Retool's observability agent can forward data to both Retool and custom destinations. Retool also supports using your own observability agent if you prefer to have complete control over telemetry data collection.

There are two services in the telemetry collector container: Alloy and vector. You can configure vector to send data to either Retool or to custom destinations.

Loading diagram...

The telemetry collector uses a secure TLS connection with short-lived client certificates when sending data to Retool. Data is securely stored on Amazon S3 buckets in us-west-2 and not shared with any other third-parties or subprocessors.

Configure Retool telemetry collector​

Telemetry data collection is enabled by default. The Retool telemetry collector runs alongside your instance and forwards metrics to Retool. The metrics Retool receives don't contain sensitive data. Logs aren't forwarded to Retool by default. You can opt out of forwarding to Retool, or turn off telemetry collection entirely.

Improve support with health monitoring

When telemetry data is forwarded to Retool, your instance's health is continually monitored. This allows Retool to have more insight into potential issues and improves the level of support when diagnosing issues.

Apply configuration changes​

The Helm values in the following sections change the default telemetry behavior. How you apply them depends on how you deploy Retool.

Terraform blueprints​

Set the values in retool_helm_extra_values in main.tf and run terraform apply. Editing Helm values directly on a blueprint-managed instance is overwritten by the next apply, which can turn telemetry back on.

Helm​

Set the values in your values.yaml or with the Helm CLI, then run helm upgrade. Restarting your pods without running helm upgrade doesn't pick up the change.

Stop sending data to Retool​

Set sendToRetool.enabled to false to keep the telemetry collector running without forwarding any data to Retool. You can still send data to custom destinations.

helm upgrade <your-release-name> retool/retool \
  -f values.yaml \
  --version <your-chart-version> \
  --set telemetry.sendToRetool.enabled=false

Turn off telemetry collection​

Set enabled to false to turn off the telemetry collector. Your instance then neither collects nor forwards telemetry data.

helm upgrade <your-release-name> retool/retool \
  -f values.yaml \
  --version <your-chart-version> \
  --set telemetry.enabled=false

Choose which data to send to Retool​

By default, sendToRetool.inputs contains only metrics. To change what the collector forwards to Retool, set sendToRetool.inputs to a list that includes any of the following inputs. The list must not be empty.

  • metrics: Infrastructure metrics, such as container health. Doesn't include metrics_statsd or metrics_statsd_raw.
  • container_logs: Logs from all containers in the Retool instance.
  • otlp_traces: Application request traces.
...
telemetry:
  sendToRetool:
    inputs:
      - metrics
      - container_logs
...
Specify telemetry version

The telemetry image uses the same release version as the main backend by default. If necessary, you can specify a version tag to use using the image.tag option:

telemetry:
  image:
    tag: 3.52.0-stable

If set, the telemetry image is fixed to the specified tag. Retool does not recommend including a tag unless you have a specific use case.

Types of telemetry data​

When telemetry is enabled, your instance produces the following types of telemetry data. Only infrastructure metrics are forwarded to Retool by default. Refer to Choose which data to send to Retool to change this.

  • Container metrics (CPU, memory, network usage)
  • Retool runtime metrics (frontend performance, backend request counts and latency)
  • Container logs (request logs, error logs, info logs)

Telemetry data comes from the following sources. To send container_logs to Retool, add it to sendToRetool.inputs. You can forward any source to custom destinations.

Send telemetry data to custom destinations​

Retool supports sending to any custom destination supported by Vector. Refer to the Vector sinks reference documentation for a complete list of supported sink types and configuration.

You specify custom destinations using the customVectorConfig variable with sink configurations. Each sink must include a list of telemetry sources for which it forwards.

...
telemetry:
  customVectorConfig:
    sinks: ...
...

Example configuration for Datadog​

The following example illustrates a telemetry configuration where data is forwarded to Retool and Datadog.

...
telemetry:
  extraEnv:
    - name: DD_AGENT_HOST
      valueFrom:
        fieldRef:
          fieldPath: status.hostIP
  customVectorConfig:
    sinks:
      # forward statsd metrics to datadog-agent port 8125
      metrics_datadog:
        address: ${DD_AGENT_HOST}:8125
        buffer:
          when_full: drop_newest
        inputs:
          - metrics_statsd_raw
        mode: udp
        type: statsd
...

Example configuration for Prometheus Remote Write​

The following example illustrates a telemetry configuration where data is forwarded to Retool and a Prometheus Remote Write destination.

...
telemetry:
  customVectorConfig:
    sinks:
      metrics_prometheus:
        type: prometheus_remote_write
        endpoint: https://prometheus:8087/api/v1/write
        inputs:
          - metrics
        buffer:
          when_full: drop_newest
...

Configure self-managed observability agents​

Configuring a self-managed agent depends on a number of factors. Use the following information to configure your agent for telemetry data collection.

Logs​

Your agent should collect logs in the same way as any other container.

Format​

Most Retool container logs are JSON-formatted. Retool recommends configuring your log collector to parse logs as JSON first, but falling back to a simple string message format upon failure.

Level​

Set the LOG_LEVEL environment variable if you need to adjust log volume and verbosity.

Debugging​

If you need to troubleshoot log collection, enable debug logs with the DEBUG environment variable. This will result in a very large number of logs. You should only use it for troubleshooting purposes.

Metrics​

Not all emitted metrics are documented because they can change between versions. Metrics cover various internal Retool runtime health signals, such as frontend performance timings, resource query timings and error rates, internal cache sizes, workflow execution rates and timings, etc.

For more information about telemetry metrics, refer to the Temporal documentation.

main-backend container​

Retool backend containers are instrumented to emit metrics in the DogStatsD format. To collect these metrics, you must configure the agent with a statsd UDP listener which is specifically DogStatsD-aware.

Set the STATSD_HOST environment variable to the IP or DNS name of your agent. If your agent is using a port other than 8125 (the default for most agents), set the STATSD_PORT environment variable to the correct port number.

workflow-worker container​

In addition to statsd metrics, the workflow-worker container is also instrumented to also emit Temporal SDK metrics to an OpenTelemetry (OTLP) gRPC collector. Temporal SDK metrics can help you scale your instance to keep up with Workflows traffic by tracking metrics such as queue latency.

If necessary, set the WORKFLOW_TEMPORAL_OPENTELEMETRY_COLLECTOR environment variable to the address of your OTLP gRPC endpoint. For example:

WORKFLOW_TEMPORAL_OPENTELEMETRY_COLLECTOR=http://localhost:4317

Tracing​

Retool supports Datadog for collecting backend traces. If you use the Datadog agent, set the DD_TRACE_ENABLED environment variable to true.

In some cases, you may need to configure additional trace collection options, such as:

  • Setting the trace agent hostname with the DD_TRACE_AGENT_HOSTNAME environment variable.
  • Adjusting the sample rate with the DD_TRACE_SAMPLE_RATE environment variable.

Retool uses the dd-trace library and supports all available configuration parameters.