How to write a recording spec for robot training data

1. 1

   The task

   Describe the task the way you'd explain it to a new worker: what is done, with which tools and parts, and what a good result looks like. Then define one episode, the single example your model will learn from.

   • Where does one episode start, and where does it end?
   • What counts as done right?
   • Should failed attempts and corrections stay in the data, or be cut?
   • Which steps must always be visible?
2. 2

   Sites and workers

   Say where the work should be recorded and by whom. A model trained on one worker at one bench learns that worker and that bench, so ask for the variety you'll need later.

   • Working sites during normal shifts, or a station set up for recording?
   • How many different workers, and how many sites?
   • Which differences matter: parts, sizes, materials, tools, lighting?
3. 3

   Cameras and viewpoints

   Choose where the cameras go. A head camera follows the worker's eyes, a chest camera gives a steadier view of both hands, wrist cameras show fingers and grips up close, and fixed cameras show the whole station. For teleoperation, list the cameras on the robot too.

   • Which viewpoints do you need, and which one is the main view?
   • What must stay in frame: both hands, the tool, the part?
   • Are you sending your own cameras or capture rig?
4. 4

   Video settings

   Fast hand movements blur at low frame rates, and automatic exposure can struggle in a dim shed with bright windows. Say what your model needs to see, and agree settings that will hold up on the shop floor.

   • Resolution and frame rate.
   • Field of view: wide enough to keep the hands in frame, narrow enough to show detail.
   • Fixed or automatic exposure and focus.
   • Audio on or off. Audio can pick up conversations, so leave it off unless you need it.
5. 5

   Sync and calibration

   If more than one camera or sensor is recording, the streams have to line up in time, or labels and actions won't match across views. Some models also need each camera's lens and position measured.

   • How are the streams synced: a shared clock, or a clear signal such as a clap or a flash at the start?
   • Do you need camera calibration, and how often: once per site, or every session?
6. 6

   Other sensors

   List anything recorded alongside the video. For teleoperation this is the core of the data: the robot's joint positions, gripper state and the operator's commands.

   • Depth cameras, motion sensors (IMUs), gloves or eye tracking?
   • For teleoperation: which robot signals are logged, and at what rate?
   • How are sensor logs matched to video frames?
7. 7

   How much, and how spread out

   Give the amount in hours or episodes, and say how to spread it. How the data is spread can matter as much as the total.

   • How much for the pilot, and roughly how much after that?
   • How should it be split across workers, sites, parts and shifts?
   • Is there a minimum for each worker or each variation?
8. 8

   Labels

   Say whether you want the footage raw or labelled. If labelled, list the label types and the exact names to use, and attach your guidelines if you have them. A few examples of tricky cases save a lot of back and forth.

   • Task and step boundaries, hand keypoints, objects, or a short description of each action?
   • The list of step names, with a one-line definition of each.
   • How will labels be checked, and how many mistakes are acceptable?
9. 9

   Privacy and confidential items

   A wide camera catches more than the task: other people, screens, papers and products a factory may not want shown. Decide before recording what gets blurred or kept out of frame.

   • Should faces be blurred before delivery?
   • Does anything else need hiding: screens, documents, product designs, logos?
   • Which consent records do you need to receive with the data?
10. 10

    Delivery

    Agree the format before the first batch. Renaming or converting thousands of files later is slow and easy to get wrong.

    • File format and video codec.
    • Folder structure and file names.
    • Details stored with each clip: date, site, task, camera and an anonymous worker ID.
    • How files are sent, and how often.
11. 11

    Pilot checks

    Write down how you'll judge the pilot, so both sides agree what good looks like before recording at volume.

    • Are the hands and parts in frame for most of each episode?
    • Is the video sharp at the speed of the work?
    • Are the streams in sync, the labels accurate and the files in the agreed format?

RECORDING SPEC
Project:
Version and date:
Contact:

1. TASK
What is done:
One episode starts when:
One episode ends when:
A good result looks like:
Keep failed attempts? (yes / no)

2. SITES AND WORKERS
Type of site:
Number of workers:
Number of sites:
Differences to cover (parts, tools, lighting):

3. CAMERAS
Viewpoints (head / chest / wrist / fixed / robot):
Main view:
Must stay in frame:
Our own hardware? (yes / no, model):

4. VIDEO SETTINGS
Resolution:
Frame rate:
Field of view:
Exposure and focus (fixed / automatic):
Audio (on / off):

5. SYNC AND CALIBRATION
How streams are synced:
Calibration (none / per site / per session):

6. OTHER SENSORS
Sensors:
Robot signals and rates (teleoperation):

7. AMOUNT
Pilot (hours or episodes):
Later batches:
How to spread it:

8. LABELS
Raw or labelled:
Label types:
Step names and definitions:
Guidelines attached? (yes / no)
How labels are checked:

9. PRIVACY
Blur faces? (yes / no)
Anything else to hide:
Consent records needed:

10. DELIVERY
File format and codec:
Folder structure and file names:
Details stored with each clip:
How and how often to send:

11. PILOT CHECKS
We accept the pilot if:

Episode boundaries

If nobody defines where an episode starts and ends, different people cut clips in different places, and step labels stop lining up.

What “in frame” means

“Keep the hands visible” can mean both hands all the time, or only the working hand during the key step. Say which.

Variety

A lot of data from one worker at one bench can be less useful than less data spread across many workers and sites.

The edges of the frame

Wide lenses catch other people, screens and papers. Plan the blurring before recording, not after.

Changes along the way

If the spec changes after recording starts, give it a new version number and date, so you know which batches follow which rules.

• Labelling guide
• Egocentric video data
• Human demonstration data
• Teleoperation data