Shipment Processing Workflow

This laboratory focuses on developing an advanced workflow that needs to integrate with an unreliable paginated API.

Steps

The following steps focus on developing a Retool Workflow with advanced resource settings, error handling logic, looping to support the paginated API, JS coding, integration of a JS function, filtering, notifications and subflows.

Create a Retool account and login

Create a Retool account and login to your account as shown below.

Overview of Paginated API

This example relies on a simulated, paginated API that provides details of customers and the urgency level of their shipment. This workflow is focused on urgent shipments only, and thus we need to invoke the API to query for all records and then develop logic to filter through what has been received and extract only urgent shipments.

API Request

The paginated shipping API is describe below:

Input	Description	Value
Method	This is the HTTP Verb supported by the API	GET
URI	The HTTP URI where the API is available at	https://qe1mvn42vg.execute-api.us-east-2.amazonaws.com/shipments/{{pageNum}}?disableDelay=false
Path Parameter	The path parameter the API supports	pageNum tells the API which page of results to return (e.g. 1 or 2)

API Response

The response to the API returns 5 unique shipping records identified by per_page. When using the pageNum of 1 the attribute has_more will be true. When using the pageNum of 2, it will be false. The following is a portion of the API response take note of priority and zone, these will be used later in a function to calculate shipping costs:

{
    "page": 1,
    "per_page": 5,
    "has_more": true,
    "data": 
        [
            {
                "id": 1,
                "name": "Leanne Graham",
                "username": "lgraham",
                "email": "lgraham@example.co",
                "priority": "regular",
                "zone": 1,
                "address": {
                    "street": "915 Broadway",
                    "suite": "Suite 789", 
                    "city": "New York",
                    "zipcode": "10001-3874",
                    "geo": {
                        "lat": "40.7128",
                        "lng": "-74.0060"
                    }
                },
                "phone": "212-555-5555 x56442",
                "website": "simcompany.co",
                "company": {
                    "name": "Sim Company",
                    "catchPhrase": "Multi-layered client-server neural-net",
                    "summary": "harness real-time e-markets"
                }
            },
            {
                "id": 2,
                "name": "Thomas Walker",
                "username": "twalker",
                "email": "twalker@example.co",
                "priority": "regular",
                "zone": 1,
                "address": {
                    "street": "51 Washington Ave",
                    "suite": "",
                    "city": "New Bedford",
                    "zipcode": "02740",
                    "geo": {
                        "lat": "41.6362",
                        "lng": "-70.933705"
                    }
                },
                "phone": "508-555-1212 x56442",
                "website": "example.org",
                "company": {
                    "name": "Example Widgets",
                    "catchPhrase": "Multi-layered client-server neural-net",
                    "summary": "harness real-time e-markets"
                }
            }...

Develop Primary Workflow

The primary workflow focuses on checking on a scheduled basis for urgent priority shipments. The following table and subsequent image identifies the steps and order of execution (top to bottom):

Name of Step	Description
startTrigger	This step initiates the workflow using a scheduled time of 12:00 am
callShippingService	This step calls the shipping API n-number of times to collect all current shipments
executeNotificationWorkflow	This step is called if there is a failure in invoking the shipping API
createArrayFromResp	This step takes a two-dimensional array returned by callShippingService and flattens it to a one dimensional array.
filterUrgentShipments	This step loops through the createArrayFromResp array and returns shipments where priority == urgent.
createShippingDocs	This step invokes a function to calculate shipping costs and generates and output with the customer details, shipping provider, and costs.
notifyCustomer	This step sends a parameterized email to the customers with the shipping details.

Loop on Paginated API

Start Trigger Block

This step focuses on initiating the workflow and specify a scheduled time for this workflow to occur.

• When creating the workflow, begin by selecting Workflows from the Retool Home Screen.
• Then select Create > Workflow and provide a name, primary-shipping-workflow. This will create a Workflow with a Start Trigger Block and a Code Block.
• Delete the Code Block by selecting ... selection at the upper right-hand corner of the Code Block and select Delete.

Loop Block (Paginated API)

This step will focus on invoking the API n-number of times to collect all shipments.

• First we select the + and drag and drop the Resource query to the right of the startTrigger block. Connect the circle on the startTrigger Block with the Loop Block and rename it callShippingService.

Loop on Paginated API

• Once connected, then the following code should be pasted:

const results = [];
var indexVal = 1;
var hasMore = true;
while(hasMore){
  const result = await Promise.resolve(callShippingService_lambda.trigger({additionalScope:{pageNum: indexVal}}));
  const dataVal = result.data;
  results.push(dataVal);

  hasMore = result.has_more;
  indexVal++;
}

return results;

• For the loop lambda , specify REST API, HTTP Method GET amd the following API URL:

https://qe1mvn42vg.execute-api.us-east-2.amazonaws.com/shipments/{{pageNum}}?disableDelay=false

• Below is the completed Loop Block with the above code and loop lambda.

Loop on Paginated API

Resource Settings Overview

The paginated shipping API is an unreliable source of data. In order to make the Retool Workflow more flexible, we will employ Resource Settings. This provides a means to define a number of retries or attempts if the API returns an error or it takes too long. The following figure shows the Resource Settings:

Fault Handler

The following table provides details on what you should set for the value, when you build the Workflow:

Resource Setting	Description	Recommended Value for Lab
Timeout after	Duration to wait before the block times out.	60000
Retry count	The maximum number of attempts to retry the query if an error is returned.	3
Interval	The minimum length of time to wait between query retries. This is useful if a query is triggering a rate limit.	1000
Coefficient	When exponential backoff is enabled, the constant factor by which the time delay between retry attempts increases after each failure.	2
Max interval	When exponential backoff is enabled, the maximum time to delay between retries.	12000
Exponential	Retries are attempted using an exponential backoff doubling the interval between each try	Enabled
Finally	Whether to stop the workflow and return an error if the block fails, or continue to allow an error handler to run.	Continue

Error Handling Overview

Error handling in Workflows can happen locally at the block level or you can define a Global Error Handler. In this example we focus on the failure of the Resource Block and enable the Continue selection in the Settings to execute a subflow. Within the Workflow, selecting Continue enables connecting a block, in his example a Workflow Block, that is initiated when the Resource Block settings are exceeded. This workflow block is identified as executeNotificationWorkflow. The following image shows this block selecting another Workflow and a parameter that is passed from this Workflow into the next workflowName.

Resource Retries

The following image provides a simple demonstration of the Resource Block Retries being executed, the result of each attempt, and the eventual identification of Failed running triggered block: callShippingService_lambda. This in turn initates the error handler executeNotificationWorkflow.

Resource Retries

Code Block (Flatten Results)

The next step transforms the data returned by the callShippingService block from a two-dimensional array into a single-dimensional array.

• Select + and drag and drop a Code Block to the right of the Loop Block (callShippingService). Connect the circle from the Loop Block to the Code Block and rename the block, createArrayFromResp.

Flatten Results

• Copy the following JS code into the Code Block:

let arrays = callShippingService.data.length;
let num_vals_per_array = 5;
let result = [];

for (let i = 0; i < arrays; i++){
  for (let j = 0; j < num_vals_per_array; j++){
    result.push(callShippingService.data[i][j]);
  }
}

return result;

Flatten Results

• When this workflow is executed later, the following provides an example of the output of the callShippingService block, notice the multi-dimensions [0][0]:

Flatten Results

• Here is what the createArrayFromResp block returns, a flattened, single dimensional array:

Flatten Results

Filter Block (Identify Urgent Shipments)

This step takes the flattened array, loops through the entries and applies a truthy statement to extract entries that match. In this case, we are looking for shipments who's priority is set to urgent.

• Select + and drag and drop a Filter Block to the right of the Code Block (createArrayFromResp). Connect the circle from the Code Block to the Filter Block and rename the block, filterUrgentShipments.

Filter by Urgency

• Specify the Filter input as createArrayFromResp.
• Enter an Expression of value.priority === 'urgent'.

The following image shows a completed Filter Block.

Filter by Urgency

The result of this filter block when executed appears as the following:

Filter by Urgency

Code Block (Execute Shipping Cost Function)

This step focuses on calculating the shipping costs for customers based on their zone and returns the customer details, zone, shipper and cost. Since this logic is reusable, a Workflow Function is employed and demonstrated in this step.

• Select Function and specify the name of the function as calculateCostsFunction.
• Copy and paste the following code into the Function Query area, making sure that Run JS Query is selected.

let zoneVal = zone;
let cost = 0;
let shipper = "";


if (zoneVal == 1){
  cost = 10.00;
  shipper = "SpeedEx";
}
else if (zoneVal == 2){
  cost = 12.00;
  shipper = "GPS";
}
else{
  cost = 25.00;
  shipper = "USPS";
}

let response = {
  customer: customer,
  zone: zone,
  shipper: shipper,
  cost: cost
}

return response;

• Specify two input parameters to the function, zone set to 1 and customer set to text.
• The following image shows the completed function.

Function Definition

• Next we need to invoke the function and this will be done via a Code Block.
• Select + and drag and drop a Code Block to the right of the Filter Block (filterUrgentShipments). Connect the circle from the Filter Block to the Code Block and rename the block, createShippingDocs.

let shipmentArray = [];
let zone = 0;
let customer = "";

for (let i=0; i<filterUrgentShipments.data.length; i++){
  zone = filterUrgentShipments.data[i].zone;
  customer = filterUrgentShipments.data[i].name;
  let result = await calculateCostsFunction(zone,customer)
  shipmentArray.push(result);
}

return shipmentArray;

The following image shows the completed form.

Call Function

The following image shows the resulting JSON created after executing the calculateCostsFunction.

Call Function

Resource Block (Send Notification Email)

In this step, we leverage a Loop Block and Retool Email to send a notification to each of the customers. Since this is a simulated step, the email is statically set to your personal email so you can view the results.

• Select + and drag and drop a Loop Block to the right of the Code Block (createShippingDocs). Connect the circle from the Code Block to the Loop Block and rename the block, notifyCustomer.
• Within the Loop Block, select Mode as GUI.
• Specify Loop input as createShippingDocs.
• For Loop lambda specify Retool Email.
• Specify your email for the To address.
• Specify for shipping notice the following:

Shipping notice for {{createShippingDocs.data[index]}}

• Specify for the Body the following:

Hello, 

We received your urgent shipping order and have scheduled it via:

Shipper: {{createShippingDocs.data[index].data.shipper}}
Cost: {{createShippingDocs.data[index].data.cost}}
Receiver: {{createShippingDocs.data[index].data.customer}}

The following image shows the completed form.

Email Notification

Develop Secondary Error-Handling Workflow

In the primary workflow, we focused on executing the core business process. The secondary workflow is meant to act as a catch-all when errors arise across the primary workflow and other workflows. The following table and subsequent image identifies the steps and order of execution (top to bottom):

Name of Step	Description
startTrigger	This step initiates the workflow using a Webhook and two parameters.
checkUrgency	This step checks the status value of the calling workflow and routes accordingly.
notifySupportTeamUrgent	This step sends email to an urgent/oncall mailing list for immediate investigation.
notifySupportTeam	This step sends email to a general support mailing list for further investigation.
workflowResponse	This step replies to the calling workflow that the workflow completed successfully.

Secondary Workflow

Start Trigger Webhook with Parameter

The Start Trigger Webhook will support two parameters, the first being the workflow name that is calling it, to help operations personnel. The second is the status of the calling workflow.

• Within the startTrigger block, specify the following JSON for testing purposes:

{
 workflowName: "sampleWorkflow1", 
 status: "urgent"
}

• Local testing can be performed by adjusting these two parameters as they will display as values for the startTrigger block.
• Select Triggers and enable Webhook.

Start Trigger

Branch Block (check response)

In order to handle the urgency of the calling workflow, a Branch Block is employed. This will check the status value and if set to urgent, send an email to the an urgent mailing list identifying the workflow, where oncall support where they can swarm and resolve. If not urgent, it will be sent to a different mailing list for general examination.

• Select + and drag and drop a Branch Block to the right of the startTrigger Block. Connect the circle from the startTrigger Block to the Branch Block and rename the block, checkUrgency.
• Edit the Branch Block with the following evaluation:

startTrigger.data.status === 'urgent'

Branch Block

Resource Block (Email Notifications)

Notifications in this example employs email, to simplify implementation of this lab. Many customers often employ alternative messaging channels such as Slack, can integrate that in lieu of email.

• Select + and drag and drop two Resource Blocks to the right of the Branch Block. Connect the circle from the Branch Block to the first Resource Block and rename the block, notifySupportTeamUrgent. Connect the circle from the Branch Block to the second Resource Block and rename the block, notifySupportTeam.
• Edit the first Resource Block with:
  • To: your email
  • Subject: {{startTrigger.data.workflowName}} failed!
  • Body:

Hi team, 

The Urgent Shipment Workflow failed and needs your immediate attention. Please login to the Retool Workflow Console to review logs. 

Cheers,
Retool Workflow

• Compare your inputs with the completed form in the following image.

Urgent Email Response

• Edit the second Resource Block with:
  • To: your email
  • Subject: {{startTrigger.data.workflowName}} failed!
  • Body:

Hi team, 

The Shipment Workflow failed. Please review the Retool Workflow Console at your earliest convenience. 

Cheers,
Retool Workflow

• Compare your inputs with the completed form in the following image.

Normal Email Response

Webhook Response Block

To complete the webhook communication to this secondary workflow, a HTTP Response is sent at the end of this process.

Webhook Response

Completed Examples

You can import the following examples and compare with what you have built.

• Primary Workflow
• Secondary Workflow