Token management for agents

Token management best practices for agents.

What is a token?

A token is the atomic unit of text that LLMs process. It's roughly four characters, or about 0.75 English words, so every 100 tokens is about 75 English words. Everything the model receives counts toward your token limit, including your instructions, resource schemas, conversation history, tool outputs, and the model's own responses.

You can use OpenAI's tokenizer tool to estimate how a given passage will be counted.

Token costs grow faster than you might expect. In practice, your message is often only 1–2% of the total tokens processed. The rest is system context, history, and tool outputs that accumulate whether you notice it or not. This means optimizing the total size and shape of your context matters more than optimizing individual prompts.

How Retool charges for AI: credits vs. hours

Retool uses two different billing mechanisms for AI, depending on the feature:

• AI credits apply to apps, Assist, and AI resource queries. Credits are organization-level and shared across all builders. They reset monthly on your billing cycle and do not roll over. Prompts requiring more complex reasoning, larger outputs, or more data consume more credits. When an organization's monthly credits are exhausted, AI-powered features become unavailable until credits renew. Admins can track and buy more credits via the Plans & billing page in Settings. Refer to AI credits billing and usage for more information.
• Retool Agents are billed based on hourly usage. Agent usage is measured by the hour at a rate that varies by provider and whether you use a Retool-managed or self-managed LLM key. Refer to Agents billing and usage for more information.

Retool-managed keys vs. bring your own key (BYOK)

Retool provides managed API keys for OpenAI and Anthropic models, which lets you start building immediately without external accounts. However, Retool-managed connections are subject to rate limits and are intended for non-production use.

Retool imposes the following rate limits on Retool-managed key usage. Rate limits are per organization, per hour (over a rolling time window).

Feature	Hourly rate limit
Retool Agents	20,000,000 tokens
Assist	100,000,000 tokens
All other AI functionality	250,000 tokens

All other AI functionality includes AI resource queries, Ask AI, Retool Vectors, and the LLM Chat component.

This rate limit is intended to prevent abuse of resources and to limit Retool-managed key usage to non-production instances. Each AI provider also imposes their own rate limits, which also apply to Retool users. If you encounter a rate limit imposed by Retool or by an AI provider, you receive a 429 error with a message describing the kind of rate limit that was hit.

Rate limits are cumulative and encompass all AI usage. These rate limits apply to any customer using Retool-provided keys, regardless of deployment method or plan type. Retool is unable to make exceptions to or raise this rate limit.

For production use, configure AI platforms to use your own API credentials. Self-managed connections are not subject to Retool-imposed rate limits, though your provider's own limits still apply.

Token management for agents

Agents are billed hourly, which changes the economics of token usage compared to other AI features. Tokens affect cost in two ways: directly, by determining how much the model has to process on each call, and indirectly, by affecting response time. Larger context means slower responses, which runs the clock. For general guidance on building effective agents, refer to Agents best practices.

Understand how context accumulates across iterations

Agentic loops run without a pre-defined stopping point. This means token costs scale with both tool output size and iteration count. An agent that calls three tools per iteration, each returning 5 KB, has injected 15 KB into context by iteration 1. By iteration 5, the model is processing 75 KB of accumulated tool history, plus your system prompt, conversation history, and any additional context, on every single call. You can also reduce the number of Max iterations the agent will try before it stops executing in the Configuration tab.

Reduce tool output size

Your system prompt and instructions are a fixed, one-time cost. Tool output is variable and compounds. Every byte a tool returns is a byte the model must re-read on every subsequent iteration. A tool that returns 50 KB instead of 5 KB inflates the token cost 10x every time that tool is called.

Return only what the agent needs for its next step:

• Apply server-side filters before returning data.
• Paginate large datasets rather than returning full result sets.
• Return computed results instead of raw data where possible. A COUNT(*) costs far fewer tokens than streaming table rows for the model to count.

Tool output is hard-capped at 1 MB. Calls exceeding this limit fail.

Use child agents to isolate context

When a task has separable sub-tasks, delegating to child agents keeps each context window small. A child agent runs in its own isolated context and returns only its result to the parent. The parent never sees the child's intermediate tool calls or reasoning.

This is especially useful for parallel data gathering tasks, where each branch would otherwise accumulate its own tool outputs inside the parent's context. A parent agent that orchestrates three child agents pays for three small contexts rather than one large one.

Estimate your token budget before you build

You can approximate the total context an agent will accumulate:

system prompt + (average tool output per iteration × expected iterations) + conversation history

To estimate component sizes: run the query in the workflow or agent editor and inspect the raw response payload to get tool output size. Estimate your system prompt's token count by dividing its character count by four.

For an agent with a 2 KB system prompt, tools averaging 8 KB per call, and a 10-iteration limit, that amounts to roughly 82 KB of context at peak, before accounting for reasoning steps or conversation history. If that exceeds your model's context window, or if it makes responses too slow to be practical, adjust tool output size or iteration limits before deploying.