Demo format guide

How the robot video annotation demo is structured.

Published 3 November 2026

A guide to the existing demo’s JSON fields, bounding boxes, action segments and self-check record. This is a video-and-labels example, not a robot action dataset.

What the demo contains.

The eight-second pouring demo contains RGB video at 480 × 360 pixels and 24 frames per second. Three object tracks identify the cup, bottle and right hand. Four action segments describe grasp, tilt, pour and return.

The labels are demonstration schema version 0.3. They were created for the website, not for a customer. The format illustrates how these demo labels are organised. A production delivery format would be agreed when scoping a project.

The main JSON fields.

Field	Meaning in this demo
`task`	A task name and natural-language description.
`clip`	Video filename, dimensions, frame rate and duration.
`tracks`	Stable track IDs, object labels and time-stamped box keyframes.
`segments`	Action names with start and end times in seconds.
`instructions`	The annotation rules, ambiguity handling and version.
`review`	The self-check method, checked timestamps, corrections and limits.

The excerpt below omits most fields and shows only the first segment; it is not a complete labels file.

{
  "sample_id": "talo-demo-001",
  "schema_version": "0.3",
  "bbox_format": "xywh_pixels",
  "interpolation": "linear",
  "segments": [
    {
      "id": "seg_01",
      "start_s": 0.0,
      "end_s": 2.5,
      "action": "grasp",
      "description": "Right hand closes on the bottle."
    }
  ],
  "review": {
    "independent": false,
    "frames_checked": 28,
    "threshold_px": 4
  }
}

Bounding boxes and object tracks.

Boxes use [x, y, width, height] in pixels, with the origin at the top-left corner of the 480 × 360 frame. A track keeps the same ID throughout the clip. The keyframe timestamp t is measured in seconds from clip start.

Positions between keyframes are linearly interpolated. The instructions call for keyframes every 0.5 seconds during slow motion and every 0.25 seconds during faster motion. Interpolation supplies positions between those observations; it is not evidence that a person checked every frame.

Action segmentation and ambiguity.

Each segment has a start time, end time, action name and description. In this demo the segments cover the clip without gaps or overlaps. Boundaries follow changes in motion, with a stated sampling tolerance of ±0.25 seconds.

Liquid is not visible at this resolution. The “pour” segment is defined by the bottle’s pose over the cup, not by observed liquid flow. The left hand is visible but unused, so it has no track. Boxes cover visible object extent rather than estimating occluded parts.

What the check log shows and what it does not.

One annotator made two passes. The second pass checked 28 of the clip’s 192 frames and logged nine box corrections at a threshold of four pixels. The 8.0-second keyframe was checked against the final frame at about 7.96 seconds.

This is a self-check, not independent review. No inter-annotator agreement or IoU score is reported. Boxes were measured by eye against a grid, and the instructions acknowledge errors of a few pixels. These details describe this demo, not a validated production quality standard.

Read the underlying demo files.

Robot state and operator action streams are not included. Video labels alone do not supply the action data required by many robot-training pipelines. Any delivery format, additional streams and review process need to be agreed for a specific project.

Discuss annotation requirements