How to label robot training data

Labels turn raw footage into examples a model can learn from: where each step starts and ends, where the hands are, what is being handled and what is happening. This guide covers the main label types, how to choose them, how to write guidelines and how to check the results.

Request dataor write to hello@haathlabs.com

What labels are for

Raw video shows what happened. Labels say what it means in a form a model can use: this step starts here, this is the left hand, this object is a bolt, this action is “tighten the bolt”. Labelling is also called annotation.

Labels take time to make, so label only what your model will use. Some teams take footage raw and label it themselves, or let their own models label it first and have people correct the results.

Either way, decide on labels before recording starts. Some depend on how the footage is recorded, so they belong in your recording spec.

The main label types

What each one is, what it's used for, and what to decide before labelling starts.

1
Task and step boundaries
The start and end time of each task, and of each step within it, with a name from a fixed list of steps. They split long recordings into episodes, show the order of a task, and make it easy to find the clips you need.
- The list of step names, with a one-line definition of each.
- The exact moment a step starts: the first movement towards the part, or the first touch?
- What to do with pauses, mistakes and repeated steps.
2
Action descriptions
A short plain-language sentence for each step or clip, such as “picks up the tile and checks the edge”. They're used to train models that follow written or spoken instructions, and to search footage.
- One sentence for each step, or one for each clip?
- Free text, or words from a fixed list?
- How much detail: “packs the box”, or which hand does what?
3
Hand keypoints
The position of each hand's joints in a frame, commonly 21 points per hand: the wrist plus four points along each finger. They're used to learn grasps and hand movement from human video.
- Every frame, or a sample, such as a few frames each second?
- Positions in the image (2D), or in space (3D)?
- What to do when fingers are hidden: leave them out, mark them as hidden, or estimate them?
4
Objects
Boxes or outlines around the objects that matter, each with a name from a fixed list, sometimes followed from frame to frame. They show what is being handled and where it is.
- Which objects count: only what the hands touch, or tools and parts nearby too?
- Boxes, or exact outlines (masks)?
- Every frame, a sample, or tracked across the clip?
5
Clip tags
Simple labels for a whole clip or episode: whether the attempt succeeded, quality flags such as hands out of frame, and conditions such as lighting or the part variant. They let you filter footage before training.
- What counts as a successful attempt?
- Which quality problems should be flagged rather than fixed?
- Which conditions do you want to compare later?

Which labels you need

Start from what your model has to learn.

Learning the order of a task: Step boundaries with a fixed list of step names.
Following instructions: Action descriptions, tied to step boundaries so each sentence matches a stretch of video.
Learning hand movement from video: Hand keypoints, plus objects so the model knows what the hand is holding.
Data from your own robot: Teleoperation recordings log the robot's own movements, so labels are usually lighter: step boundaries, success tags and descriptions.
Not sure yet: Start with step boundaries and clip tags. They're useful for almost any model, and you can add other labels to the same footage later.

Writing guidelines

Guidelines keep labels consistent from one person to the next. Any question they leave open gets answered differently by different people.

Definitions: One line for each step name, object name and tag, saying what's included and what isn't.
Boundary rules: Exactly when a step starts and ends, for example “starts when the hand first touches the part”.
Examples: A few frames or clips for each label, including the hard cases.
Edge cases: What to do with pauses, mistakes, repeated steps, hidden hands, and two things happening at once.
Format: The file format, the field names, the time units, and whether left and right mean the worker's or the image's.
Version and date: So you know which clips follow which rules when the guidelines change.

Checking quality

Agree how quality will be measured before labelling starts, not after the first batch.

Review a sample: Re-check a random share of clips from every batch, and count the mistakes you find by type.
Label some clips twice: Have two people label the same clips and compare. Where they disagree, the guidelines are usually unclear.
Agree tolerances: Decide in advance how far off still counts as right: a few frames for a step boundary, a few pixels for a keypoint.
Update the guidelines: When the same question comes up twice, answer it in the guidelines and give them a new version number.

Easy to miss

Left and right: Say whether “left hand” means the worker's left or the left of the image. With a head camera they match. With a fixed camera facing the worker, they're swapped.
Time units: Seconds or frame numbers, counted from the start of the clip or of the whole recording. Mixing them shifts every boundary.
Guessing hidden points: If hidden fingers are estimated, mark them as estimated. Unmarked guesses look like real positions.
Changing the step list: Adding a step halfway through means re-checking every clip already labelled. Try the list out in a pilot first.
Recording for the labels: Some labels depend on how the footage was recorded. Hand keypoints need the hands clearly in frame, so plan labels and cameras together.

Common questions

What is data labelling for robotics?

It's adding information to recorded data so a model can learn from it: when each step starts and ends, where the hands and objects are, and what is happening. It's also called annotation.

Which labels do I need?

It depends on what your model learns. Step boundaries and a few clip tags suit almost any project. Add hand keypoints to learn hand movement from video, and action descriptions for models that follow instructions.

Should we label the footage ourselves?

If you have your own labelling tools and people, you can take the footage raw. If not, have it labelled to your guidelines and review a sample of every batch yourself.

How can I check that labels are accurate?

Review a random sample from every batch, have two people label some of the same clips and compare, and agree in advance how far off still counts as right.

Can labels be added later?

Yes. As long as you keep the raw footage, you can add new kinds of labels later. Some depend on how the footage was recorded, though: hand keypoints need the hands clearly in frame.

Start with a pilot

Tell us which tasks you need recorded, the camera setup and the labels you want. We'll reply with questions and a plan for a pilot: a few hours of footage first, so you can check the quality before committing to more.