Talo Request a pilot
← Back to collection

Case study · Collection

A robotics startup gets household demonstrations from 30 homes it could not reach.

Client
Robotics startup
Capture
Head camera and handheld UMI gripper
Tasks
20 household tasks
Labels
Tracks, action segments, task descriptions, gripper streams
episodes delivered
6,200
homes, vs 1 test house
30
from task list to last batch
8 wks
of recordings failed intake, re-recorded
3.4%

The situation

Four prototype robots in one test house meant one kitchen, one set of plates and one kind of light. Policies that worked there failed elsewhere. It needed the same tasks recorded in many homes, without shipping robots.

What they asked for

  • 20 tasks, from loading a dishwasher to folding towels, each recorded in at least 25 homes.
  • Gripper demonstrations with fingers that match its robot's gripper, so the robot can learn from them directly.
  • Every episode labelled to the same rules as its in-house data.
  • Bad episodes caught before delivery, not after training.

What Talo delivered

4,100 head-camera episodes and 2,100 UMI gripper episodes, with the gripper's position, orientation and width aligned to each video frame at 60 Hz. Every episode has object tracks, action segments and a task description.

episode u-0918 · place_cup · home 17 · 0 to 8 sUMI

Video

Six frames from a UMI wrist-camera clip, from reach to release.

Gripper width
(open at top)

Segments

  1. reach
  2. grasp
  3. lift
  4. place
  5. release
  6. retract

Segment boundaries line up with the gripper stream, so "grasp" ends where the width stops closing.

How it ran

  1. 1

    Task list

    Each of the 20 tasks was written as a short script: start state, end state and what may vary between homes.

  2. 2

    Pilot week

    3 tasks in 4 homes, 180 episodes. The startup loaded them into its training code to confirm the gripper data matched its robot.

  3. 3

    Weekly batches

    About 1,000 episodes a week. Each passed an intake checkA check on each new recording before labelling: camera clear, every data stream present, task completed., two labelling passes and a 10% auditA second annotator labels a sample of episodes on their own, and the two sets of labels are compared. by a second annotator.

Outcome

Every episode shipped with its rules and check log, so the startup could mix it with its own teleoperation data.

The automatic first pass, on a public dataset

Episode 0 of the open UMI cup dataset: a person carries a handheld gripper and places a cup on its saucer, outdoors. The gripper records its opening width with every frame. Move across the chart, pick a segment, or play the episode.

SegmentTime
0.0 to 20.1 s
20.1 to 20.3 s
20.3 to 30.5 s
30.5 to 30.8 s
30.8 to 39.9 s
Segments here are an automatic first pass from the gripper's width signal, coarser than a delivery, where an annotator checks every boundary against the video. Source: lerobot/umi_cup_in_the_wild, episode 0, Apache-2.0. Video trimmed and scaled.

Gripper close-and-open cycles per episode, across all 1,447 episodes (19.4 hours)

  • None found (public data)49
  • One1,011
  • Two354
  • Three30
  • Four or more3

The task needs one grasp. About one episode in four shows more than one, which could be a regrasp, a slip or a noisy reading. Those are the episodes to check before training.

Need demonstrations from places your robots cannot reach?

Ask about collection Next: lab annotation backlog →