Skip to main content

Troubleshoot Source Control issues

Learn how to troubleshoot Source Control.

With Source Control, your Retool instance syncs and deploys changes made to the designated main branch of your Git repository. As an administrator, you can diagnose and troubleshoot most Source Control issues.

Identify Source Control issues​

Issues that arise with Source Control should be resolved quickly to minimize disruption. There are a number of methods available for identifying symptoms as you investigate the root cause.

Deployment Dashboard​

Use Deployment Dashboard to check the status of deployments and the SCM connection. Each deployment has either a successful or failed state. Failed deployments can indicate a potential issue with Source Control, such as malformed repository contents or an inconsistent Git history.

Kicking off a full deployment by clicking Deploy latest updates all protected elements in your Retool instance to the latest state of your main branch. This overrides and can help fix failing intermediary partial deployments.

Deployment Dashboard

The SCM details area displays the current connection status. If the status is not Connected, this indicates that your Retool instance is not able to connect to your SCM provider. To resolve the connection, verify your SCM configuration by clicking Edit config, environment variables, and if using GitHub, your base64 encoding of your private key.

Changes and environments no longer in sync​

There may be issues with Source Control if there is a mismatch between the current state of your Retool instance and the main branch of your Git repository. This can be apparent if protected apps, modules, or resources fail to update. When these issues occur, they can prevent newer changes from being deployed, resulting in the changes being overwritten. New objects cannot be protected return an error or indicate an unmerged status.

Similarly, if changes from one Retool instance are not in sync with others (e.g., dev to prod) then there may be an issue somewhere in the Source Control workflow.

jobs-runner instability​

If Source Control cannot successfully deploy changes, it may cause the jobs-runner container to continuously crash or restart. This container performs background tasks, such as Source Control syncing, and database migrations.

You can also monitor the status of the jobs-runner container using its health check endpoint: http://[container_ip]:3003/api/checkJobsRunnerHealth.

Resolve common Source Control issues​

For Source Control to operate:

  • The Git repository and its contents must have a stable history and valid file contents.
  • Both your Retool instance and your source control management (SCM) provider must be configured correctly.
  • Retool must be able to connect to your SCM provider.

Any problems or disruptions that affect these dependencies can impact the operation of Source Control.

Git repository and content​

Retool instances periodically poll your Git repository for new commits and attempt to apply changes. If the integrity of the Git repository is affected, such as invalid or malformed data, your Retool instance may fail to deploy successfully.

Malformed data​

Source Control breaks up applications into multiple YAML files to simplify code review and prevent merge conflicts. Invalid changes to YAML files can prevent successful deploys. Possible YAML issues include:

  • Invalid formatting. Manually adding commits, or editing commits automatically generated by Retool, can introduce formatting errors (e.g., different whitespace or tab settings).

If there appears to be malformed data or invalid files, you need to identify which commits contain the malformed files.

First, identify the commit with malformed files. You can do this by either:

  • Using the Deployment Dashboard to identify the commit of the first failing build.
  • Reviewing recent commits in the Git history with the SCM provider. This may be the most recent commit if you're able to respond to the issue quickly.
  • Reviewing the jobs-runner container logs for lines with the terms Target sha, SOURCE_CONTROL_DEPLOYMENT, and error.

Next, revert or fix-forward the offending commit. You can revert the commit and push the reversion to your Git repository to roll back the change. To preserve the changes, you can fix-forward by correcting the error as a new commit.

Finally, run a full deployment. You can perform this using the Deployment Dashboard by clicking Deploy latest. Retool instances using Self-hosted Retool earlier than v2.97 must restart the jobs-runner container manually.

Inconsistent Git history​

Retool requires a linear, append-only Git history, and tracks the latest commit SHA at all times. If the Git history is rewritten manually (e.g., git push --force, git rebase -i, or git reset), the commit currently tracked by Retool may no longer exist. Retool may attempt to open an invalid commit with the SCM provider, which will return an error.

If the Git history has been rewritten manually, one way to resolve this is to:

  1. Export any apps as JSON that cannot be pushed using the Retool UI.
  2. Create a new branch in the Retool UI and import the exported JSON.
  3. Retry the commit and push operations.
  4. Verify that the generated pull request contains all relevant changes.

Unexpected files​

The file system structure in the Git repository must match the expected directory structure of your Retool instance. Any unknown files added to the repository, or moving existing files to different locations unexpectedly, can cause deploys to fail. Non-YAML files that may be autogenerated by operating systems, such as .DS_Store, can also impact the repository's integrity.

If unexpected files have been added to the repository, remove them and commit the changes.

Instance configuration​

Retool uses environment variables to configure deployments on self-hosted Retool. Any misconfigured environment variables can cause issues with your Retool instance.

SCM and repository connection​

The Deployment Dashboard reflects the connection status to the SCM provider. If this reports a failed connection, check that the connection details are correct.

Source Control requires the full URL for some SCM providers. If you are using one of these SCM providers, make sure to include the full http/https URL.

Nested repositories​

Some SCM providers support nested repositories (e.g., GitLab). If the path is incorrect, the Deployment Dashboard indicates a successful connection but Source Control is not syncing from the correct repository path. If you are using nested repositories, check the repository details use the full path.

SCM configuration​

Retool requires read and write permissions for branches, commits, and pull requests. Incorrect permissions can prevent Retool from creating new protected objects or editing existing ones. Refer to the Source Control documentation for your SCM provider to learn what permissions are required and how to configure them.

Infrastructure instability​

Infrastructure failures or instability can be caused by a misconfigured environment. There can be many reasons for this, but common root causes include:

  • Minimum resource requirements not met. Each Retool instance requires 4+ CPU cores and 4GB+ of memory. As your organization grows, the number of commits or concurrent users increases, which requires more resources. If your Retool instances lack resources, the jobs-runner container performance degrades and can become unresponsive. When Retool detects that the jobs-runner container is not responsive, it can automatically restart the container in some deployment modes (e.g., Retool Kubernetes Helm chart).
  • Network firewall restrictions. Retool requires an open connection directly to your SCM provider. If your firewall blocks these connections, Git operations may become unresponsive or time out. You can use the Deployment Dashboard to check the connection to your SCM provider by clicking Test connection.

OAuth callback URL​

If you see issues with the OAuth callback URL for protected resources, make sure:

  1. The BASE_DOMAIN environment variable is persisted on both the api and jobs-runner containers.
  2. Both the api and jobs-runner containers were rebooted.
  3. The source control repo is redeployed through the /settings/sourcecontrol.

Image display​

Images for protected apps are only stored in the database of the instance in which the app was developed. An image uploaded to the database for one app cannot be accessed by downstream environments.

If an image does not appear where you expect it to in your protected app, you can:

  • Host the image on a public URL that is accessible from all apps, and use the URL option for the image source.
  • Upload the image to the downstream database environments.