Automate incident notifications with Workflows

A common use case for Retool Workflows is to automatically alert on-call engineers and customer account managers if an incident occurs that is affecting customers, such as a service outage or downtime.

This tutorial explains how to build a notification workflow that:

• Runs automatically in response to a webhook event that occurs whenever an incident is logged in Intercom.
• Creates a ticket in Linear and assigns it to the on-call engineer for the affected product area.
• Notifies the customer's account manager on Slack.
• Evaluates the severity level to determine if the incident is urgent. If so, the workflow creates an incident in PagerDuty for escalation.

Considerations

This tutorial uses the following resources to demonstrate a real-world use case of Retool Workflows:

• Linear
• REST API for PagerDuty
• Slack

Much of what you'll learn can also apply to other resources and data.

You can generate your own test API and sample data to try out Retool Workflows without using any data sources of your own.

1. Create a new workflow

Sign in to Retool, select the Workflows tab in the navigation bar, then click Create new. Set the name to Incident alert workflow.

2. Configure the webhook trigger

Every workflow contains a Start block that contains Trigger settings for you to configure. Whenever these trigger conditions are met, the Start block triggers the workflow to run automatically.

For this use case, the workflow must run in response to a webhook event that Intercom sends whenever an incident occurs. Every workflow has a unique webhook endpoint URL that can receive webhook events from other services. You can also configure the webhook trigger to parse specific parameters from the received JSON payload.

To configure a webhook trigger for this workflow:

1. Click on the Start block to expand its settings.
2. Set the Trigger to Webhook.
3. Set the Parameters to severity, product_area, customer, triggered_by, and incident_title.

The connecting lines between blocks in a workflow represent the control flow. Workflow blocks connect together to perform specific actions in sequential order, beginning with the Start block. Once a block has completed its action, it triggers the next block in the chain, and so on.

3. Check if an incident is urgent

If the severity of an incident is urgent, the workflow must notify the relevant on-call engineer and the customer's account manager.

The Branch block enables you to visually construct if...else statements that evaluate data and perform conditional actions. If the condition evaluates as a truthy value, it triggers the block to which it's connected. If not, it triggers another block.

Click + in the toolbar, add a Branch block named checkSeverity, and set the If condition to startTrigger.value.severity == "urgent". If the received severity parameter has a value of urgent , this evaluates as true.

Click ▸ to run the code. As you test your workflow, the condition that evaluates as true highlights in green.

4. Build functions to perform required actions

The workflow needs to perform the same set of actions for incidents of any severity. But in the case of urgent incidents, it should also log the incident in PagerDuty.

Instead of building out two conditional flows that repeat the same set of actions, the workflow uses Function blocks. These reusable blocks operate outside of the workflow control flow and can be called at any time. Functions also accept parameters which other blocks can provide.

This workflow uses Function blocks to:

• Identify the on-call engineer in PagerDuty.
• Create an incident in PagerDuty.
• Look up the Linear user ID of the on-call engineer.
• Create a ticket in Linear and assign it to the on-call engineer.
• Look up the customer's account manager and notify them in Slack.

After creating the Function blocks, each condition of the Branch block triggers custom JavaScript code that calls the required Functions to perform the necessary actions. As a result, the workflow only needs one Function block to create a Linear ticket instead of two Query blocks for each condition.

Identify the on-call engineer in PagerDuty

The workflow needs to identify who is on-call for a particular product area. For this use case, on-call information is available in PagerDuty, the incident handling and response platform.

When run, this request needs to include a schedule ID that corresponds to a particular product area.

To add a Function block:

1. Click + in the Functions section of the left panel.
2. Set its name to getPagerDutyOnCall, and select the RESTQuery resource.
3. Rename param1 to schedule.

Function block parameters can include an initial value so that the block can still run if no parameter values are provided. Set the value for schedule to any valid PagerDuty schedule.

Next, set the request type to GET and the URL to:

https://api.pagerduty.com/schedules/{{schedule}}/users?since={{moment().format()}}&until={{moment().add(1, 'minute').format()}}

This GET request returns the on-call engineer for the specified product area by including {{ schedule }}. It also filters the result to include only the engineer who has been on-call in the last minute, ensuring it only returns the current on-call engineer.

note

The RESTQuery resource allows you to interact with REST APIs on a per-app or per-workflow basis. Create a REST API resource if you need to use the same API across your Retool instance.

Create an incident in PagerDuty

If the incident is urgent, the workflow must also create an incident in PagerDuty. Create a function named triggerPagerDutyIncident with two parameters: incidentCreator and incident. This is used to identify who created the incident and pass details about it.

To create a PagerDuty incident, the workflow must make a POST request to the /incidents endpoint. It must include additional request headers and a JSON body.

First, set the request type to POST and the URL to https://api.pagerduty.com/incidents.

Next, define the following request headers in the Headers section:

Header	Value
Authorization	Token token=XXXXXXXXX
Accept	application/vnd.pagerduty+json;version=2
From	{{ incidentCreator }}

Finally, set the Body to JSON and set an incident parameter name to {{ incident }}.

Create a ticket in Linear

The workflow must create a Linear ticket for all incidents, regardless of severity, to track progress. This requires two Function blocks that use the Linear integration to interact with Linear's API using GraphQL:

• Look up the user ID of the on-call engineer
• Create a ticket and assign it to the on-call engineer

Look up the user ID

First, create a function named getOnCallLinearUser, define an email parameter, and select the Linear GraphQL resource. Insert the following GraphQL query to look up a user by their email address:

getOnCallLinearUser

query Users($email: String!) {
  users(filter: { email: { eqIgnoreCase: $email } }) {
    nodes {
      id
      email
    }
  }
}

When run, this query looks up the user ID of the on-call engineer using their email address.

Create and assign a ticket

To create a Linear ticket and assign it to a user, the GraphQL request must include the parameters:

• title
• description
• teamId
• stateId
• assigneeId

Create a function named createLinearTicket that uses the Linear resource and set two parameters: linearTitle and linearAssigneeId. The values for these parameters are passed when the function is called. The remaining values are written in the query itself.

Insert the following GraphQL query to create a ticket:

createLinearTicket

mutation IssueCreate($title: String!, $assigneeId: String!) {
  issueCreate(
    input: {
      title: $title
      description: "More detailed error report in markdown"
      teamId: "6e6d7dcd-e49e-4eb8-bc76-d7b3c4afe1e2"
      stateId: "2e6d6ed9-9fac-48ea-a9ee-46786be5e792"
      assigneeId: $assigneeId
    }
  ) {
    success
    issue {
      id
      title
      number
    }
  }
}

Notify the account manager on Slack

When an incident occurs, the workflow must also notify the customer's account manager. For this use case, the workflow must first get some customer details from Salesforce so it can determine which account manager to notify.

Look up customer ID in Salesforce

First, create a function named getSalesforceOwnerId that uses the Salesforce resource and define the customer parameter. This function looks up the owner ID of the customer's account.

getSalesforceOwnerId

SELECT
  OwnerId
FROM
  account
WHERE Name = {{customer}}

Look up account owner in Salesforce

Next, create another function named getSalesforceOwner that uses the Salesforce resource and define the ownerId parameter. This function looks up the required details using the owner ID from the previous function.

getSalesforceOwner

SELECT
  Id,
  Name,
  Username,
  Email
FROM
  User
WHERE
  id = {{ownerId}}

Send a Slack notification

For this use case, the workflow must send a Slack notification and @mention the account manager. As with Linear and Salesforce, the workflow needs to look up the account manager's Slack ID.

Look up Slack user by email address

First, create a Function block named getSlackId and define the email parameter. Retool's built-in Slack integration enables you to send messages only. To look up user information or perform non-messaging actions with Slack, you must use Slack's REST API. To do this, you must first set up the Slack API as a resource on your Retool instance.

Once configured, you can perform a GET request to look up a user by email with the URL https://slack.com/api/users.lookupByEmail?email={{email}}.

Send notification

Next, create another Function block named sendSlack that uses the Slack resource and define the message parameter. The message itself will be defined in the next step.

8. Write JavaScript to trigger functions

This workflow now contains functions that can trigger all required actions in response to an incident. The last step is to write custom JavaScript to trigger these Function blocks accordingly.

First, click and drag ⦿ from the Branch block's If condition to create a connected Query block, set its name to alertUrgent, and select the Run JS Code resource.

Next, add an Else condition to the Branch block. Click and drag ⦿ from this condition to create another connected Query block, set its name to alert, and select the Run JS Code resource.

Both Query blocks uses JavaScript to perform a number of actions depending on the severity. The table below lists all possible workflow actions and whether they run if an incident is urgent or not.

Action	Run if urgent?	Run if not urgent?
Map values for product_area to the corresponding PagerDuty schedule	✔︎	✘
Get product_area from the webhook event JSON payload and identify which schedule it matches	✔︎	✘
Trigger the getPagerDutyOnCall function to identify the on-call engineer	✔︎	✘
Trigger the triggerPagerDutyIncident function to create an incident in PagerDuty	✔︎	✘
Trigger the getOnCallLinearUser and createLinearTicket functions to create a Linear ticket and assign it to the on-call engineer	✔︎	✔︎
Trigger the getSalesforceOwnerId and getSalesforceOwner functions to identify the account manager	✔︎	✔︎
Trigger the getSlackId and sendSlack functions to identify the Slack ID of the account manager and send a message on Slack.	✔︎	✔︎

Use the following JavaScript in the respective Query blocks. Refer to the comments for more details on what occurs during each step in the process.

alertUrgent

//// Trigger a PagerDuty incident ////

// An object mapping product_areas to PagerDuty schedule IDs
const scheduleObject = {
  cloud: {
    schedule: "PWL1I8T",
    service: "PIA2HXN",
  },
  onprem: {
    schedule: "PXJ9FUL",
    service: "P3YBHYP",
  },
  infra: {
    schedule: "PJ1978S",
    service: "PIA2HXN",
  },
  mobile: {
    schedule: "PIZPCAM",
    service: "PIA2HXN",
  },
};

// The PagerDuty schedule ID associated with the triggered product_area
const schedule = scheduleObject[startTrigger.value.product_area].schedule;

// Trigger getPagerDutyOnCall function and save the result
const getPagerDutyOnCallResult = await getPagerDutyOnCall(schedule);

// Create PagerDuty incident object
const incident = {
  inputCreator: startTrigger.value.triggered_by,
  inputIncident: {
    type: "incident",
    title: startTrigger.value.incident_title,
    service: {
      id: scheduleObject[startTrigger.value.product_area].service,
      type: "service_reference",
    },
  },
};

// Set triggerPagerDutyIncident function and save the result
const triggerPagerDutyIncidentResult = await triggerPagerDutyIncident(
  startTrigger.value.triggered_by,
  incident
);

//// Create a linear ticket to track issue ////

// Get the Linear ID for the on-call engineer
const getCurrentLinearUserResult = await getOnCallLinearUser(
  getPagerDutyOnCallResult.users[0].email
);
const linearAssigneeId = getCurrentLinearUserResult.data?.users.nodes[0].id;

// Create a Linear title
const linearTicketTitle =
  moment().format("YYYY-MM-DD") + " " + startTrigger.value.incident_title;

// Create a Linear ticket with the generated title and assign it to the on-call
const createLinearResult = await createLinearTicket(
  linearTicketTitle,
  linearAssigneeId
);

//// Slack Account Manager ////

// Get salesforce owner id
const getSalesforceOwnerIdResult = await getSalesforceOwnerId(
  startTrigger.value.customer
);
const salesforceOwnerId = getSalesforceOwnerIdResult.data[0].OwnerId;

// Get salesforce owner email
const getSalesforceOwnerResult = await getSalesforceOwner(getSalesforceOwnerId);
const salesforOwner = getSalesforceOwnerResult.data[0];

// Get Slack ID
const getSlackIdResult = await getSlackId(getSalesforceOwner.Email);
const slackOwnerId = getSlackIdResult.data.user.id;

// Compose Slack message and add account owner tag
const slackMessage =
  "[" +
  startTrigger.triggered_by +
  "] has opened a new " +
  startTrigger.value.severity +
  "issue: " +
  startTrigger.value.incident_title +
  "CC account owner: <@" +
  slackOwnerId +
  ">";

// Send Slack message to notify assigned account owner

await sendSlack(slackMessage);

alert

//// Create a Linear ticket to track issue ////

// Get the Linear ID for the on-call engineer

const getCurrentLinearUserResult = await getOnCallLinearUser(
  getPagerDutyOnCallResult.users[0].email
);
const linearAssigneeId = getCurrentLinearUserResult.data?.users.nodes[0].id;

// Create a Linear title
const linearTicketTitle =
  moment().format("YYYY-MM-DD") + " " + startTrigger.value.incident_title;

// Create a Linear ticket with the generated title and assign it to the on-call
const createLinearResult = await createLinearTicket(
  linearTicketTitle,
  linearAssigneeId
);

//// Slack Account Manager ////

// Get Salesforce owner ID
const getSalesforceOwnerIdResult = await getSalesforceOwnerId(
  startTrigger.value.customer
);
const salesforceOwnerId = getSalesforceOwnerIdResult.data[0].OwnerId;

// Get Salesforce owner email
const getSalesforceOwnerResult = await getSalesforceOwner(getSalesforceOwnerId);
const salesforOwner = getSalesforceOwnerResult.data[0];

// Get Slack ID
const getSlackIdResult = await getSlackId(getSalesforceOwner.Email);
const slackOwnerId = getSlackIdResult.data.user.id;

// Compose a Slack message and add account owner tag
const slackMessage =
  "[" +
  startTrigger.triggered_by +
  "] has opened a new " +
  startTrigger.value.severity +
  "issue: " +
  startTrigger.value.incident_title +
  "CC account owner: <@" +
  slackOwnerId +
  ">";

// Send Slack message to notify assigned account owner

await sendSlack(slackMessage);

9. Test and enable the workflow

Now that the workflow is complete, manually run the workflow by clicking Run on the right of the Workflow editor toolbar. You can also test the workflow's webhook trigger using cURL. Click Copy to copy the full command and then run it from a CLI.

Workflows are not triggered automatically by default. After verifying that the workflow runs as expected, toggle Enable. This activates the Start block's trigger so that it runs in response to webhook events.

Wrap up

Using Retool Workflows, you have now built an autonomous incident alert tool. By applying the lessons learned here and following the same patterns, you can extend the workflow's functionality further, such as sending conditional notifications.