Token management for workflows
Token management best practices for workflows that query AI resources.
What is a token?
A token is the atomic unit of text that LLMs process. It's roughly four characters, or about 0.75 English words, so every 100 tokens is about 75 English words. Everything the model receives counts toward your token limit, including your instructions, resource schemas, conversation history, tool outputs, and the model's own responses.
You can use OpenAI's tokenizer tool to estimate how a given passage will be counted.
Token costs grow faster than you might expect. In practice, your message is often only 1–2% of the total tokens processed. The rest is system context, history, and tool outputs that accumulate whether you notice it or not. This means optimizing the total size and shape of your context matters more than optimizing individual prompts.
How Retool charges for AI: credits vs. hours
Retool uses two different billing mechanisms for AI, depending on the feature:
- AI credits apply to apps, Assist, and AI resource queries. Credits are organization-level and shared across all builders. They reset monthly on your billing cycle and do not roll over. Prompts requiring more complex reasoning, larger outputs, or more data consume more credits. When an organization's monthly credits are exhausted, AI-powered features become unavailable until credits renew. Admins can track and buy more credits via the Plans & billing page in Settings. Refer to AI credits billing and usage for more information.
- Retool Agents are billed based on hourly usage. Agent usage is measured by the hour at a rate that varies by provider and whether you use a Retool-managed or self-managed LLM key. Refer to Agents billing and usage for more information.
Retool-managed keys vs. bring your own key (BYOK)
Retool provides managed API keys for OpenAI and Anthropic models, which lets you start building immediately without external accounts. However, Retool-managed connections are subject to rate limits and are intended for non-production use.
Retool imposes the following rate limits on Retool-managed key usage. Rate limits are per organization, per hour (over a rolling time window).
| Feature | Hourly rate limit |
|---|---|
| Retool Agents | 20,000,000 tokens |
| Assist | 100,000,000 tokens |
| All other AI functionality | 250,000 tokens |
All other AI functionality includes AI resource queries, Ask AI, Retool Vectors, and the LLM Chat component.
This rate limit is intended to prevent abuse of resources and to limit Retool-managed key usage to non-production instances. Each AI provider also imposes their own rate limits, which also apply to Retool users. If you encounter a rate limit imposed by Retool or by an AI provider, you receive a 429 error with a message describing the kind of rate limit that was hit.
Rate limits are cumulative and encompass all AI usage. These rate limits apply to any customer using Retool-provided keys, regardless of deployment method or plan type. Retool is unable to make exceptions to or raise this rate limit.
For production use, configure AI platforms to use your own API credentials. Self-managed connections are not subject to Retool-imposed rate limits, though your provider's own limits still apply.
Token management for workflows
Refer to Workflow best practices for guidance on data retrieval, single-purpose design, trigger configuration, and load testing.
When a workflow includes an AI resource query, use the following token management best practices:
Fetch only the data the AI resource actually needs
Every database row passed to an AI resource uses one token. A database query returning 10,000 rows when the model only needs a summary is slow and expensive. Apply filters, aggregations, and pagination at the database level, before data reaches the AI resource query.
Select models based on workflow task
Retool lets you select the model per query block. You can use a faster or cheaper model for workflow steps that require fewer capabilities, and reserve models with more processing power for only the workflow blocks that require them. Refer to the AI models page for further information.
If your workflow uses an Invoke Agent block, refer to the Agents token management guide for best practices.