Physical AI & Egocentric Data

Egocentric & Robotics Data for Physical AI

Human demonstrations made useful for machine learning. We scope, collect, annotate and review real-world task data, connecting what people do with the instructions, interactions and outcomes behind each action.

Discuss a Physical AI program Explore workflows

A data-operations specialization

Capture the task.
Understand every step.

For teams developing systems that perceive and act in physical environments. Each program starts with the task taxonomy, capture conditions and annotation schema your learning objective needs.

Egocentric collection

First-person and wearable-camera recordings of agreed tasks. Define consent, camera placement, visibility, environment coverage and acceptance criteria before collection.

Human demonstrations

Record how people handle objects, use tools and move through task sequences. Track the instruction, task setup and demonstrated result.

Action segmentation

Break continuous recordings into meaningful actions and task steps using an agreed vocabulary, including transitions and ambiguous boundaries.

Temporal annotation

Mark action starts and ends, task boundaries, transitions and events. Review timing against source video and project-specific tolerance rules.

Hand-object interaction

Label visible hands, manipulated objects, contact, interaction type and object state. Link actions to the relevant hand and object where the footage supports it.

Object & scene understanding

Identify tools, objects, locations and relevant environment context. Capture observable state changes and flag occlusion or insufficient visual evidence.

Video captioning

Describe the action, manipulated object and changes over time. Keep captions grounded in visible evidence and aligned to the annotated interval.

Task decomposition

Turn complex demonstrations into structured steps with dependencies, intermediate states and completion conditions agreed with your team.

Instruction alignment

Connect the requested instruction to the demonstration, action sequence and observed result. Flag mismatched, incomplete or ambiguous instructions.

Illustrative egocentric capture & annotation schema

One demonstration.
Several layers of meaning.

A robot-camera frame shows grippers, objects and target markers. The example below describes how a complete pick-and-place sequence could be annotated; outcomes and timing must be verified from the source video.

Illustrative egocentric robot-camera view — a representative visual of Physical AI / egocentric data work, not client or project footage — Illustrative egocentric stereo view

Pick up object00:00-00:02
Move object00:02-00:04
Place on target00:04-00:06
Release object00:06-00:07

From video to a structured record

Instruction: Move the selected object onto its assigned marker.
Entities: Gripper, target object, work surface, placement marker.
Interaction: Grasp -> transport -> support contact -> release.
State change: Object moves from its initial location to the target.
Caption: Describe visible motion and object state for each video interval.
Completion: Verify the final position and release against the full sequence.
Review: Check boundaries, object identity, contact and outcome; flag uncertainty.

The annotation hierarchy

Three levels of structure — from a single motion to the whole task.

Every demonstration is labelled at three levels of detail, so a robot-learning model can read it as atomic motions, object-centric task segments, or a complete episode. Human reviewers apply the same schema consistently, frame by frame.

L3 — Atomic actions

One discrete motion

The foundation layer: each individual action or interaction, captured on the exact frames it starts and ends.

Exact start / end frame boundaries
Hand used & action type
Object identified & target location
Idle spans labelled explicitly

L2 — Object-centric segments

Related actions, grouped

Atomic actions on the same object or sub-task are grouped into a segment, with boundaries that align to the underlying actions.

One object / task focus per segment
Success / failure of the goal
Retry count recorded
Concise task-level caption

L1 — Episode-level goal

The whole task

A single summary of the entire demonstration — what was achieved, and in what environment.

Overall task being completed
Environment & context verified
Short whole-video summary
Groups every L2 segment beneath it

From video to training data

The egocentric annotation workflow.

Egocentric video

First-person demonstration of a real-world task.

Atomic actions

L3 — segment every motion, frame-accurate.

Object-centric segments

L2 — group actions, mark success & retries.

Episode goal

L1 — summarise the whole task & context.

QA & adjudication

Coverage, boundaries & captions reviewed.

Structured data

Validated, robot-training-ready records.

Capabilities in this program

Frame-accurate temporal annotation Egocentric first-person review Hand / object interaction labeling Object identification Target-location labeling Idle-state annotation Action segmentation Task decomposition Success / failure evaluation Retry tracking Caption review Timeline QA No-gap / no-overlap validation

Quality before delivery

Review the timing.
Resolve the uncertainty.

Success / failure evaluation
Where the task allows it, label the observed outcome against explicit criteria. Identify failure points, incomplete attempts and cases where the evidence is insufficient.
Robotics / Physical-AI QA
Check label consistency, temporal accuracy, captions, task steps and hand-object relationships. Route disagreements to senior review and adjudication.
Calibration and feedback
Use reference tasks, annotation guidance and edge-case examples. Feed recurring errors back into contributor calibration and the project taxonomy.

Reviewers checking annotation output and timing at a workstation

A scoped program, end to end

Start with your learning objective.

Collection hardware, environments, labeling granularity and output format are confirmed during scoping. Robotics hardware operation, robot control, 3D reconstruction and sensor-derived ground truth require separate validation; they are not assumed from video annotation.

01 / Define & pilot

Agree tasks, instructions, capture protocol, consent, taxonomy and review criteria. Run a pilot to establish what the footage can support.

02 / Produce & review

Qualify contributors, collect or ingest approved video, annotate in agreed tooling, and review difficult cases before release.

03 / Deliver & improve

Deliver agreed structured annotations with clip references, temporal boundaries, labels, captions, outcomes and review status. Track dataset and guideline versions.

Build your data program

Tell us the task.
We'll scope the data operation.

Share your intended use, demonstration environment, available footage and annotation requirements.

Discuss a Physical AI program