Skip to main content

Connect to Kafka

Learn how to connect Kafka to Retool.

You can use the Kafka integration to create a resource and make it available in Retool. Once complete, your users can write queries that interact with Kafka data.

You can use the Kafka integration in Retool Apps and Retool Workflows to produce, consume, or commit messages to a topic.

Requirements​

The Kafka integration requirements depend on whether you have a cloud-hosted or self-hosted Retool organization. You may also need to make Kafka configuration changes before creating the resource.

Sufficient user permissions to create resources

All users for Retool organizations on Free or Team plans have global Edit permissions and can add, edit, and remove resources. If your organization manages user permissions for resources, you must be a member of a group with Edit all permissions.

Allow Retool to access the data source

If the data source is behind a firewall or restricts access based on IP address, then you must ensure that your Retool organization can access it. If necessary, configure your data source to allow access from Retool's IP addresses.

CIDR IP addresses
3.77.79.248/30
35.90.103.132/30
44.208.168.68/30
Individual IP addresses
3.77.79.249
3.77.79.250
35.90.103.132
35.90.103.133
35.90.103.134
35.90.103.135
44.208.168.68
44.208.168.69
44.208.168.70
44.208.168.71

Retool is building support for querying firewalled resources without allowlisting Retool’s IP address. To learn more or be considered for early access, contact cloud-connect@retool.com.

Kafka settings and authentication

You must have sufficient access and familiarity with your Kafka data source so you can provide:

  • Required connection settings (e.g., URL and server variables).
  • Authentication credentials (e.g., API keys).

In some cases, you may need to make changes to your Kafka configuration, such as generating authentication credentials or allowing access through a firewall. Refer to the configuration and authentication sections to learn more.

Configure the resource​

Sign in to your Retool organization and navigate to the Resources tab. Click Create new > Resource, then select Kafka.

Configuration​

Specify the name, location, and description to use for your Kafka resource. Retool displays the resource name and type in query editors to help users identify them.

Provide the following configuration settings to create the resource. Depending on how your data source is configured, you may also need to provide optional settings for Retool to connect.

Name

The name to use for the resource.

Description

A description of the resource.

Override default outbound Retool region

Retool connects to your data source from the us-west-2 region. Choosing a different outbound region can improve performance through geographic proximity.

RegionLocation
us-west-2US West (Oregon)
eu-central-1(Frankfurt, Germany)

Authentication​

The Kafka integration supports the following authentication methods. Depending on which authentication method you use, you may need to make changes to your Kafka configuration.

SASL mechanism

The Simple Authentication and Security Layer (SASL) mechanism. Options include PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512. Defaults to SCRAM-SHA-512.

Security Protocol

The security protocol. Options include SASL_SSL, SSL, SASL_PLAINTEXT, PLAINTEXT. Defaults to SASL_SSL.

Bootstrap Servers

The Kafka brokers in the cluster that you want to connect to, in a comma-separated list.

Username and password

Authentication is performed with a username and password. You must be able to obtain and provide these credentials to create the resource.

Test the connection​

Click Test Connection to verify that Retool can successfully connect to the data source. If the test fails, check the resource settings and try again.

Testing a connection only checks whether Retool can successfully connect to the resource. It cannot check whether the provided credentials have sufficient privileges or can perform every supported action.

Save the resource​

Click Create resource to complete the setup. You can then click either Create app to immediately start building a Retool app or Back to resources to return to the list of resources.

Enable the Java DB Connector​

For Retool Self-hosted users, Retool provides a Java DBConnector, designed to enhance the performance and stability of certain integrations. This connector is required for the Amazon SQS, Databricks, and Kafka connectors.

The instructions for enabling the Java DB connector differ based on whether you are using Helm as a package manager:

  1. Upgrade to Retool Self-hosted version 3.52 or later.
  2. Run helm search repo retool/retool to check Retool's chart version. If you are on a version earlier than 6.0.9, then upgrade your deployment.
  3. Open values.yaml and add the following:
dbconnector:
  java:
    enabled: true

Query the Kafka resource​

When querying the Kafka resource, you can choose from one of three actions:

  • Produce messages for a topic
  • Consume messages from a topic
  • Commit messages in a topic

Consumer groups are automatically deleted after 7 days without use, unless you edit Kafka's offsets.retention.minutes value.

To create a new Consumer Group within Retool, click the button and enter the desired value. After refreshing, the new value appears in the dropdown.

Next steps​

Your Kafka resource is now ready to use. Check out related queries and code documentation to learn how to interact with Kafka data.