Reducing Unusable Recordings in Embodied AI Data Collection

Moving quality checks from the data center to the collection site can lift usable yield from 60% to 80% and save roughly $9,600 a month in wasted collection cost.

Problem

About 2 in 5 recorded hours are unusable, and failures surface ~18 days after recording.

North star

Usable yield = usable hours ÷ recorded hours.

Key insight

The cost of a failure depends on how late it is detected, not just on how often it happens.

Solution (MVP)

Automated pre-session check + on-site exit validation + offload integrity gate.

Target

Usable yield 60% → 80% in one quarter, with setup time up by no more than 3 minutes.

Role shown

Problem framing, metric design, root cause analysis, prioritization, experiment design.

Why real-world data matters

How this data is collected at scale

Five hand-offs before anyone checks the data

1. 1Collector records (wearable kit)→
2. 2Drive staging at site→
3. 3Ship drives to data center→
4. 4Upload to S3→
5. 5QA review→
6. Usable or discarded

↳ verdict arrives ~15–20 days after recording

A program of 40 collectors records household and workplace tasks with a head camera, wrist cameras and IMUs — about 1,200 hours a month.

“40% of recorded hours are discarded at QA. By then the collector has left the site, the task setup is gone, and the only fix is a full re-record.”

Catch failures early without making the collector’s day slower.

Usable yield = usable hours delivered ÷ hours recorded

Metric tree

Supporting & guardrail metrics

70% of lost hours come from three causes — all detectable within minutes

Audit 200 discarded sessions from the last quarter, tag each with one primary cause, and weight by hours lost. Then interview 8 collectors and 3 QA reviewers to learn why each cause happens.

What interviews add: Collectors skip manual checks when rushed, and nobody tells them which of their sessions failed. The problem is partly tooling and partly a missing feedback loop.

When a failure is found matters as much as what it is

1. Before recording starts~1 minute of setupYes — fix and start
2. During the sessionA few minutes of footageYes — pause and fix
3. Right after, on siteOne re-take while setup is still in placeYes — re-record now
4. At offloadDrive and shipping costOnly the copy, not the recording
5. At QA, ~18 days laterThe full $40/hour plus a new site visitOnly by re-collecting from scratch

This is also why adding more QA reviewers wouldn’t help: it improves detection at the most expensive point in the pipeline.

Six options, scored with RICE

Reach is 4,400 sessions/month for every option (40 collectors × 5 sessions × 22 days).

Two 60-second screens and one gate — no new hardware

A 4-week pilot, 10 vs. 10 collectors

Treatment group

10 collectors using the new checks

Control group

10 collectors matched on tenure, site type and past yield

Duration

4 weeks (about 1,000 sessions per group)

Primary metric

Usable yield, measured at QA as today

Secondary metrics

Detection latency, share of failures fixed on site

Guardrails

Setup time, sessions per day, false block rate, override rate

Qualitative input

Weekly 15-minute check-in with pilot collectors

Success criteria — decided before the pilot starts

• ✓Usable yield in the treatment group at least 10 points above control
• ✓Setup time up by no more than 3 minutes per session
• ✓False block rate under 5%, and override rate under 10%

If the pilot passes, roll out in two waves of 15 collectors, tuning check thresholds between waves using override logs. If it fails a guardrail, fix that check and re-run a 2-week pilot rather than shipping.

Every check must be fast, specific and fixable

• False positives block good sessions

  Collectors lose trust and start overriding

  → Start with loose thresholds, tighten using pilot data; log every override

• Added setup time

  Lower throughput, collector frustration

  → Hard cap of 60 s per check; run checks in the background while the collector gears up

• Limited compute on the wearable

  Heavy checks drain battery or lag

  → Lightweight rule-based checks on device; no ML on the edge in MVP

• Gaming the metric

  Collectors avoid hard tasks to protect their yield

  → Track task mix per collector; never tie pay to yield alone

• Thresholds differ by site

  Lighting and layouts vary between homes and factories

  → Per-site-type thresholds tuned during rollout

• Checks miss subtle failures

  Some framing and protocol errors need human judgment

  → Keep QA review; checks reduce its load, they don’t replace it

A stricter check catches more failures but blocks more good sessions. The pilot’s override log is how the team finds the right balance, instead of guessing.

33% more usable data from the same budget

The MVP needs about 12 person-weeks. At an assumed $2,000 per person-week that’s $24,000 — recovered in about 2.5 months of savings.

Try it with your own numbers

Frame quality as a pipeline problem, not a QA problem

• Frame quality as a pipeline problem, not a QA problem. The cheapest fix sits at the point of recording.
• Pick a north star the customer feels. Usable yield ties collector effort directly to what the ML team receives.
• Design for the collector’s day. A check that adds minutes will be skipped, however accurate it is.
• Closing the feedback loop is half the solution. Collectors improve fastest when they see their own failures.

Numbers about the collection program (hours, costs, failure shares) are illustrative assumptions, not sourced figures.

• BCG — What is Physical AI? ↗
• Cisco — What Is Physical AI? ↗
• EY — What is Physical AI and why it matters now ↗
• Ego4D — large-scale egocentric video ↗
• Ego-Exo4D — paired first- and third-person video ↗
• ALOHA — low-cost bimanual teleoperation ↗
• DROID — large-scale robot manipulation demonstrations ↗
• UMI — Universal Manipulation Interface ↗
• Open X-Embodiment — pooled data across robot types ↗