Failure and recovery demonstrations are the scarce and valuable part of a dataset

The standard instinct in dataset curation is to clean the data: remove the failed attempts, keep the successes, ship the clean set. This instinct is wrong for manipulation training, and the cost of following it is measurable.

What a failure episode contains

A failure episode is not a corrupted success episode. It is a demonstration of a different thing: the decision process at the boundary of competence. The electrician who fails to seat a terminal block correctly does not just show you a wrong outcome—she shows you the approach that seemed right but was not, the verification check that caught the error, and the recovery maneuver that brought the attempt back to a passable state.

Each of those three segments is independently valuable:

- **The failure approach** shows the system a plausible wrong path. A model that has seen only success episodes will not encounter this path during training, but it will encounter it during deployment. - **The verify segment** shows that checking behavior is not a separate task but an integral subtask with a formal result. A model that has never seen a failing verify does not learn that "checking" produces a binary outcome that routes to either completion or recovery. - **The recover segment** is the rarest and most expensive to collect. It requires not just a failure but a failure that was recognized and corrected within the episode—a sequence that takes longer to film and requires the demonstrator to deliberately make and then correct an error.

The collect-only-successes failure mode

When a dataset contains only success episodes, a robot trained on it learns the average successful path very well. It fails non-gracefully when it encounters a deviation from that path, because it has no training signal for what to do when the task has not succeeded yet.

The problem is compounded by the fact that real deployment conditions differ from training conditions: tool wear, material variation, lighting variation, surface contamination. A model that has only seen clean success paths will encounter the verify-failure state on its first deployment deviation, with no learned response to it.

The annotation requirement

Failure and recovery data is only useful if it is annotated correctly. A failure episode without explicit subtask boundaries and an outcome label is not trainable—it is just a clip where something went wrong. The annotation must specify:

- Where the failure mode appeared (which subtask, at what timestamp) - What the verify result was (reject) - Where the recovery attempt began and ended - What the final outcome of the episode was (failure or recovery)

The annotation schema must distinguish between a failure episode (the attempt never reached a passing verify) and a recovery episode (a verify failed but a recovery subtask brought the attempt to a passing state). These are different training signals: one teaches the model a failed path, the other teaches it to correct a failure.

Cost and collection rate

Failure and recovery episodes cannot be manufactured on demand. A demonstrator cannot be instructed to "make a mistake and then fix it" and expect natural-looking manipulation behavior to result. The demonstrator must encounter a genuine task difficulty, recognize it, and recover from it.

In practice, failure and recovery episodes account for roughly 15–25% of accepted footage in professional trade tasks, depending on task difficulty. The episodes that contain genuine recovery sequences—where a failed verify routes through a substantive recover subtask—are rarer still.

That scarcity is precisely what makes them valuable. A model that has learned from a dataset with a realistic failure and recovery distribution will generalize better to deployment conditions than one that has seen only the clean path. The gap in benchmark performance between the two grows as deployment conditions diverge from training conditions.

The practical implication

The annotation schema must be designed from the start to support failure and recovery labels. Retrofitting them onto a success-only dataset requires re-annotation, which is expensive. The collection protocol must also explicitly allow for and encourage failure and recovery sequences rather than resetting to a clean start after every failed attempt.

This is one reason that professional trade demonstrations in controlled environments are more valuable per episode than crowd-sourced household footage at scale: the controlled environment allows the demonstrator to continue through the recovery sequence without external interruption, and the professional context produces failures that are realistic rather than constructed.