From raw footage to training episodes: inside the Curatrix pipeline
Segmentation, annotation schema, manifests and formats. What happens between a tradesperson pressing record and a versioned dataset landing in your training run.
A day of recording on site yields hours of continuous video that is, on its own, nearly useless for training. The value is created afterwards, in a pipeline that turns footage into episodes with structure a model can learn from. This post walks through that pipeline as it runs for every Curatrix release.
Segmentation into episodes
Footage is first cut into task episodes: one complete execution of the specified task, from a defined start state to a defined end state. Idle time, tool fetching and interruptions are trimmed or labelled as such. Episode boundaries follow the written task specification agreed with the customer, so the same task recorded by ten contributors produces comparable episodes.
Annotation schema
Each episode is annotated with subtasks and phases, the tools and objects in use, and outcome states such as “fastened”, “misaligned” or “retried”. The schema is fixed per engagement and documented alongside the release. Annotators are trained on the trade in question; labelling a torque sequence correctly requires knowing what a torque sequence is.
Manifest and provenance
The manifest is the index of the release. It maps every episode to its video files, sensor streams, annotation records, contributor and site releases and licence scope. Releases are immutable and versioned; a later release adds episodes or corrections without rewriting the earlier one, which keeps experiments reproducible.
Formats
Video ships as MP4, tabular annotations and metadata as Parquet, the manifest as JSON, and synchronised multi-stream recordings as MCAP. Loaders for LeRobot and ROS are on the roadmap. The docs describe each format in detail, and the dataset catalogue shows which formats a given release includes today.