Workflow performance best practices

Learn about the best practices for building performant workflows.

Workflow performance issues can be difficult to replicate, especially when they involve external factors. The following best practices can help avoid problems and reduce the need to debug workflows. Refer to documentation on limits for details on configurations and limitations around timeout, memory, and cpu at the block and Workflow level.

Refer to the best practices guide for queries to learn more about writing performant queries for your data.

Retrieve only specific data from large datasets

Queries that retrieve large amounts of data impact workflow performance. The more data a workflow query requests, the longer it can take for the resource to complete the operation.

Where possible, write queries that return only the results that the workflow needs (e.g., use API pagination and WHERE SQL query clauses).

Build workflows to perform specific tasks

To avoid timeouts, cpu throttling and out-of-memory errors, build separate workflows to perform specific tasks instead of a single workflow that performs them together.

Trigger workflows only when needed

Workflows are rate-limited and have a minimum interval of one minute if scheduled. Each workflow must also complete a run (whether successful or not) before it can be run again.

Whether you schedule a workflow to run automatically or trigger it using a webhook event, make sure that the workflow triggers only when it should run. This also helps reduce the amount of interactions a workflow has with your data source, which can reduce operational costs.

Perform load testing

Workflows that work with large datasets can run into memory or time limits. Retool recommends performing load tests on production workflows to ensure that you don't encounter any issues. This is especially important with variably-sized datasets, in which looping or returning data from a code block might fluctuate in load size. Test out your workflow with a larger dataset than expected to ensure that you won't encounter any resource or time constraints.