Connect to Kafka
Learn how to connect Kafka to Retool.
You can use the Kafka integration to create a resource and make it available in Retool. Once complete, your users can write queries that interact with Kafka data.
You can use the Kafka integration in Retool Apps and Retool Workflows to produce, consume, or commit messages to a topic.
Requirements
The Kafka integration requirements depend on whether you have a cloud-hosted or self-hosted Retool organization. You may also need to make Kafka configuration changes before creating the resource.
- Cloud-hosted organizations
- Self-hosted organizations
Sufficient user permissions to create resources
Allow Retool to access the data source
Kafka settings and authentication
Sufficient user permissions to create resources
Allow your deployment to access the data source
Kafka settings and authentication
CA Certificates
Configure the resource
Sign in to your Retool organization and navigate to the Resources tab. Click Create new > Resource, then select Kafka.
Configuration
Specify the name, location, and description to use for your Kafka resource. Retool displays the resource name and type in query editors to help users identify them.
Provide the following configuration settings to create the resource. Depending on how your data source is configured, you may also need to provide optional settings for Retool to connect.
- Cloud-hosted organizations
- Self-hosted organizations
Name
Description
Override default outbound Retool region
Name
Description
Authentication
The Kafka integration supports the following authentication methods. Depending on which authentication method you use, you may need to make changes to your Kafka configuration.
- Cloud-hosted organizations
- Self-hosted organizations
SASL mechanism
Security Protocol
Bootstrap Servers
Username and password
SASL mechanism
Security Protocol
Bootstrap Servers
Username and password
Test the connection
Click Test Connection to verify that Retool can successfully connect to the data source. If the test fails, check the resource settings and try again.
Testing a connection only checks whether Retool can successfully connect to the resource. It cannot check whether the provided credentials have sufficient privileges or can perform every supported action.
Save the resource
Click Create resource to complete the setup. You can then click either Create app to immediately start building a Retool app or Back to resources to return to the list of resources.
Enable the Java DB Connector
For Retool Self-hosted users, Retool provides a Java DBConnector, designed to enhance the performance and stability of certain integrations. This connector is required for the Amazon SQS, Databricks, and Kafka connectors.
The instructions for enabling the Java DB connector differ based on whether you use Helm as a package manager:
- With Helm
- Without Helm
Always upgrade your deployment to use the latest version of Retool's Helm chart. The latest version is v6.4.2.
Run helm search repo retool/retool
to check the current version of Retool's Helm chart that is installed. Use helm upgrade
to then upgrade the Helm chart version, if required.
helm upgrade -f values.yaml my-retool retool/retool --version v6.4.2
Next, add the following to values.yaml
:
dbconnector:
java:
enabled: true
Restart your Retool deployment instance for these changes to take effect.
Set the DISABLE_JAVA_DBCONNECTOR
environment variable to true
, then Restart your Retool instance for these changes to take effect.
Query the Kafka resource
When querying the Kafka resource, you can choose from one of three actions:
- Produce messages for a topic
- Consume messages from a topic
- Commit messages in a topic
Consumer groups are automatically deleted after 7 days without use, unless you edit Kafka's offsets.retention.minutes
value.
To create a new Consumer Group
within Retool, click the button and enter the desired value. After refreshing, the new value appears in the dropdown.
Next steps
Your Kafka resource is now ready to use. Check out related queries and code documentation to learn how to interact with Kafka data.
Queries and code quickstart
Fundamental concepts of queries and code.
Resource query tutorial
Hands-on introduction to querying APIs and databases.
Explore database schemas
Learn how to explore database schemas.
Read SQL data
Learn how to retrieve data with SQL.
Write SQL data
Learn how to write data with SQL.