Tag images for machine learning
Build an app to draw bounding boxes around images and save them to the database.
Let's say we're a self-driving car company, and need to label a ton of images. We'll build a tool where we take a list of image URLs, let the end-user draw bounding boxes around each image, and save the results back to an API, all in 5 minutes.
1. Get list of images
Let's first switch to the onboarding_db (readonly)
, since we provide you some pre-loaded data.
Then, write a SQL query to pull in all of our images:
select * from images order by id;
Let's preview it. If that looks good, let's save it. Our query editor should look like this right now:
Great! Now let's drag on a Table
from the right hand side. It should be automatically populated with data from your last query (in this case, the images). Great - that's it! Images ready to be labelled, in our Table
. Your app should currently look like this:
2. Render and label them
Now let's render each image onto a tagging canvas. Let's drag in a BoundingBox
component from the right hand side. It's near the bottom, so it's easier to use the search box on top:
Let's drag it on, and then change the image_url
to refer to what's currently selected in the table. That'd be {{table1.selectedRow.data.image_url}}
.
Great! Now your BoundingBox
should be showing the selected image in the Table
. And when you select different rows in the Table
, the BoundingBox
will show a different image for tagging.
Even though our data currently has no bounding boxes, we probably want to show the bounding boxes - if there are any in the database. See the "Initial Bounding Boxes" on the right hand side? Let's change that to: {{JSON.parse(table1.selectedRow.data.labels)}}
. That currently won't do anything, since we don't have any bounding boxes, but this'll come into use later.
Let's try labeling a few boxes before moving on to the next step. The BoundingBox
supports panning and zooming when you hold down the Command key (on Macs), and Control key (on Windows / Linux).
3. Save labels
Cool! Now we just have to save the tags to our database. Let's drag on a Button
and label it "Save Tags". Let's also make sure we create a +New
event handler with Action
set to Trigger query
and Query
set to Create a new query
.
Let's change the created query's resource to onboarding_api
, since we want to PUT
data back. Then fill in the values below to construct the PUT
query. We're going to be PUT
ting back to the selected image, whose id
is {{table1.selectedRow.data.id}}
.
For the body, the key will be labels
, and the value will be:
{
{
JSON.stringify(boundingbox1.boundingBoxes);
}
}
And after the query is successfully ran, we want to refresh our original query, so we can see the new bounding boxes. So let's scroll down and change "on success, trigger these queries" to include our original query (query1
):
That's it! Let's save the query and try it out. Draw some bounding boxes, and press the Button
. Your bounding boxes should be saved to the database, and your Table
will refresh. Great - bounding boxes there!
Since we set our BoundingBox
to render boxes if they exist, you can click around to a different row in the Table
. When you come back, you'll see that the bounding boxes are still there. So you can add some boxes, and then press save again - all without losing data.
4. Actually using it
This tool is now production-ready. You can send the link to your coworkers, and they'll be able to use it to tag data - immediately. If you'd like, you can also build out an approval workflow (so managers have to approve labelled images), modify the SQL query to only pull in images that are unlabelled / assigned to the particular user (try the current_user
variable), etc.
If you give your operators specific permissions, they won't be able to edit the tool and change the queries - they'll just be able to use the tool you've created.
Updated 7 months ago