Tag images for machine learning

Let's say we're a self-driving car company, and need to label a ton of images. We'll build a tool where we take a list of image URLs, let the end-user draw bounding boxes around each image, and save the results back to an API, all in 5 minutes.

1. Get list of images

Let's first switch to the onboarding_db (readonly), since we provide you some pre-loaded data.

Then, write a SQL query to pull in all of our images:

select * from images order by id;

Let's preview it. If that looks good, let's save it. Our query editor should look like this right now:

Great! Now let's drag on a Table from the right hand side. It should be automatically populated with data from your last query (in this case, the images). Great - that's it! Images ready to be labelled, in our Table. Your app should currently look like this:

2. Render and label them

Now let's render each image onto a tagging canvas. Let's drag in a BoundingBox component from the right hand side. It's near the bottom, so it's easier to use the search box on top:

Let's drag it on, and then change the image_url to refer to what's currently selected in the table. That'd be {{table1.selectedRow.data.image_url}}.

Great! Now your BoundingBox should be showing the selected image in the Table. And when you select different rows in the Table, the BoundingBox will show a different image for tagging.

Even though our data currently has no bounding boxes, we probably want to show the bounding boxes - if there are any in the database. See the "Initial Bounding Boxes" on the right hand side? Let's change that to: {{JSON.parse(table1.selectedRow.data.labels)}}. That currently won't do anything, since we don't have any bounding boxes, but this'll come into use later.

Let's try labeling a few boxes before moving on to the next step. The BoundingBox supports panning and zooming when you hold down the Command key (on Macs), and Control key (on Windows / Linux).

3. Save labels

Cool! Now we just have to save the tags to our database. Let's drag on a Button and label it "Save Tags". Let's also make sure we create a +New event handler with Action set to Trigger query and Query set to Create a new query.

Let's change the created query's resource to onboarding_api, since we want to PUT data back. Then fill in the values below to construct the PUT query. We're going to be PUTting back to the selected image, whose id is {{table1.selectedRow.data.id}}.

For the body, the key will be labels, and the value will be:

{
  {
    JSON.stringify(boundingbox1.boundingBoxes);
  }
}

And after the query is successfully ran, we want to refresh our original query, so we can see the new bounding boxes. So let's scroll down and change "on success, trigger these queries" to include our original query (query1):

That's it! Let's save the query and try it out. Draw some bounding boxes, and press the Button. Your bounding boxes should be saved to the database, and your Table will refresh. Great - bounding boxes there!

Since we set our BoundingBox to render boxes if they exist, you can click around to a different row in the Table. When you come back, you'll see that the bounding boxes are still there. So you can add some boxes, and then press save again - all without losing data.

4. Actually using it

This tool is now production-ready. You can send the link to your coworkers, and they'll be able to use it to tag data - immediately. If you'd like, you can also build out an approval workflow (so managers have to approve labelled images), modify the SQL query to only pull in images that are unlabelled / assigned to the particular user (try the current_user variable), etc.

If you give your operators specific permissions, they won't be able to edit the tool and change the queries - they'll just be able to use the tool you've created.