Skip to main content

Migrate from YAML to Toolscript

Learn how to migrate your apps, modules, and resources from YAML to Toolscript.

The migration is only necessary for organizations on self-hosted deployments whose Retool Source Control repository was configured before the deployment was upgraded to version 3.6.0. Repositories configured on version 3.6.0 or later already use Toolscript.

If you have a multi-instance deployment, ensure you follow the steps to migrate multiple instances.

To use Toolscript, you must perform a one-time migration of your existing YAML repositories. The entire migration takes approximately 15 minutes to complete, but may take longer for organizations with many protected apps and branches. It does not change the behavior of your apps and queries.

This migration is unidirectional and you cannot migrate from Toolscript back to YAML.

Migration overview​

Retool attempts to migrate your main branch and recent feature branches to Toolscript from YAML. During this operation, all source control operations are locked in Retool.

On the main branch, all apps, modules, and resources are migrated. Retool also attempts to migrate all feature branches touched within the last two months, up to a maximum of 300 total. For recent feature branches, only apps and resources changed by the feature branch are migrated. Retool also attempts to delete all recent open branches which rename, move, or unprotect source control elements. These branches are not possible to migrate, but you can recreate them after the migration.

If migration is not possible for all branches, Retool does not push any remote changes, and instead prompts you to fix or delete certain branches. This failure is rare, and usually occurs if a branch has been manually edited outside of Retool and contains an invalid structure or YAML format.

The automatic migration does not currently support migrating workflows. Commits to protected workflows are still in YAML.

Test runs​

You cannot currently specify a single test branch to migrate using the Retool UI. When the migration begins, all remote branches are converted from YAML to Toolscript.

To dry run the migration, you must copy your entire Git repository to a new one, then configure your Source Control configuration on your Retool instance to use the new repository.

App behavior​

The Toolscript migration itself does not change how Retool stores apps, and therefore does not affect the behavior of apps. Similarly, the migration does not affect app releases, history, or queries.

Commits made by builders in Toolscript are functionally equivalent to YAML, with the exception that Toolscript omits default values to produce more readable diffs. These default values may change across different versions of Retool, but should not lead to significant user experience changes.

Requirements​

To run the migration you must:

  • Be on version 3.8.10 or later. See the errors on earlier versions section if you're on a version between 3.6.0 and 3.8.10.
  • Ensure that branch protection is disabled on your SCM provider. Retool requires access to push Toolscript conversion commits directly to your main branch. See GitHub and GitLab documentation to learn how to toggle these settings. You may re-enable branch protection after the migration is complete.
  • Notify all Retool builders in your organization that source control operations are blocked during the migration.

If you run multiple instances, you must ensure your instances all use the same main branch to sync code.

Run migration​

To run the migration, go to Settings > Source Control and select Get Toolscript.

If the migration runs successfully, no additional input is necessary. You can confirm a migration was successful by navigating to the main branch on your SCM provider. You should see a new commit entitled "Convert to ToolScript v2.0.0" that replaces *.yml files with *.rsx and other Toolscript files. Additionally, the .retool/protected-apps.yml metadata file should be updated to read 2.0.0.

If the migration returns an error, you can download logs for the given run from the Source Control settings page.

Migration with errors and downloadable logs

Migration errors​

Migration logs can help you identify errors which occurred during your migration. To resolve these errors, use one of the following options:

  1. Revert or manually fix the changes introducing errors. Many issues are likely malformed YAML. See troubleshooting for more guidance.
  2. Delete the remote and Retool branch. You can use the following command to delete the remote branch.
git push origin --delete yourfeaturebranch

To preserve your work, you can export and import apps or modules using JSON into a new branch after your migration.

When you've fixed or deleted the erroring branches, rerun the migration by selecting Get Toolscript on the Source Control settings page.

Errors on versions pre-3.8.10​

If you are migrating to Toolscript on a version earlier than 3.8.10, you may encounter additional migration errors in the following scenarios:

  • Empty branch migration: If your migration failure logs contain the line yourfeaturebranch has no changes when compared to yourmainbranch, the migration has failed because one or more of your feature branches contain no changes to migrate. Upgrade to 3.8.10+ first, or delete these branches using the Git CLI.
  • SCM branch protection: You may receive a false success message from Retool without a Toolscript conversion commit on your main branch if you have branch protection enabled on your SCM provider. To resolve this situation, click Deploy Latest on the Source Control settings page. This immediately unblocks builders and allows commits to be made in YAML. Next, turn off branch protection and upgrade to version 3.8.10+. You can then re-run the migration from the Source Control settings page.

Upgrading to 3.8.10+ is highly recommended to avoid these scenarios.

Multi-instance procedure​

If you deploy multiple instances of Retool connected by the same source control repository, you must follow special procedures to avoid downtime. The steps below assume you have two instances: development ("dev") and production ("prod"). If you do not have a prod instance, designate the instance that is more sensitive to downtime as the prod instance.

The recommended upgrade procedure depends on whether or not your instances sync from the same main Git branch, or use different branches:

Each source control branch can contain Toolscript files or YAML files, but not both. As a result, all Retool instances connected by the same branch must be upgraded to support Toolscript at the same time.

  1. Confirm that all instances are already on version 3.8.10+ or later. You can choose to follow an incremental upgrade procedure, such as upgrading and verifying the dev instance first. The Toolscript migration only runs when manually triggered.
  2. Choose a time during which no builders are using source control to begin the migration. The safest method to do this is to take all instances offline completely. If downtime is unacceptable, an announcement may suffice. Builders lose any commits or branches created during the migration.
  3. Log in to the dev instance and run the migration from the Source Control settings page. If there are any errors, address them before moving on; your source control repository is not yet affected.
  4. To confirm the migration, look for a successful deployment status in the Source Control settings page and a new conversion commit in your source control repository. The .retool/protected-apps.yaml file in your repository should also read 2.0.0. If this is not the case, stop immediately and contact Retool Support.
  5. After the dev migration succeeds, log in to the prod instance. Since the prod instance syncs from the same branch as dev, you do not need to perform the migration again. Click Deploy latest and confirm that the deployment status shows as healthy. The instance should automatically recognize the new Toolscript code.

Migrate YAML manually​

Though rare, some directories may remain in YAML after the Toolscript migration has completed. For example:

  • A developer merged in a YAML feature branch that was not migrated because it exceeded the 2-month and 300 branch limit.
  • A developer merged in a YAML feature branch while the Toolscript migration was running.

In these cases, your deployments may fail because there are both Toolscript and YAML format files in the repository. To resolve this situation:

  1. Ensure that the .retool/protected-apps.yaml file on the main branch of your source control repository reads 2.0.0. If not, the Toolscript migration has not ran successfully yet.
  2. As a precaution, back up your app by exporting it to JSON.
  3. Delete the app folder containing YAML from your source control repository and push the change directly to your main branch.
  4. Your app should now show as unprotected on your Retool instance. Protect the app through Retool again to generate the correct Toolscript and commit the change to your source control repository.

A similar procedure can be used to manually migrate an unmerged YAML pull request. These steps can only be executed by the pull request author.

  1. Ensure that the .retool/protected-apps.yaml file on the main branch of your source control repository reads 2.0.0. If not, the Toolscript migration has not ran successfully yet.
  2. Go to the app changed in the pull request and export it to JSON.
  3. Start a new branch on the same app and import the JSON obtained in step 1.
  4. Create a pull request on this new branch. The format should be Toolscript. You can now close the YAML pull request.