Subtask Annotations

Subtask Annotations#

This pipeline generates per-frame subtask annotations for RoboCasa composite task demos: it decomposes each task into the sequence of subtasks it is composed of and labels every frame with the subtask currently being performed, its stage, and a natural-language instruction. The labels are derived directly from each demo’s privileged simulation state, making them precise and fully reproducible, and the code ships with labelers covering all 300 composite tasks.

Not for leaderboard submissions

Do not use these subtask annotations when training models for the RoboCasa365 leaderboard. Leaderboard entries must train on the standard pretrain_human300 dataset without the added subtask labels, so all submissions are compared on equal data.

Fields#

field type meaning
subtask_idx int 0-based index of the current segment, incrementing at each boundary. The last segment is always done.
subtask_name str the atomic skill, e.g. PickPlaceCabinetToBowl, NavigateKitchen
subtask_stage str the phase, e.g. navigate, pick, place, press, tilt, wait, done
subtask_lang str natural-language instruction for the segment, e.g. "carry the eggplant to a bowl next to the sink"

A “segment” is a maximal run of frames with the same subtask_idx; boundaries are where it changes.

HDF5#

Stored per demo at data/<demo>/{subtask_idx,subtask_name,subtask_stage,subtask_lang}, frame-aligned with actions/states.

import h5py
g = h5py.File("demo.hdf5")["data"]["demo_0"]
idx  = g["subtask_idx"][:]                          # (T,) int32
lang = [s.decode() for s in g["subtask_lang"][:]]   # (T,) strings

LeRobot#

Per-frame columns in each episode parquet:

  • subtask_idx: int, identical to the HDF5 value.

  • annotation.human.subtask, annotation.human.subtask_name, annotation.human.subtask_stage: integer task indices that resolve to the subtask_lang / subtask_name / subtask_stage strings via meta/tasks.jsonl.

Note

LeRobot episode order ≠ HDF5 demo order. Pair them by trajectory length if needed.

Regenerating#

# HDF5
python robocasa/scripts/dataset_scripts/label_subtasks.py --dataset <demo.hdf5>

# LeRobot
python robocasa/scripts/dataset_scripts/add_subtask_labels_to_lerobot.py \
    --lerobot <lerobot_dir> --from-hdf5 <demo.hdf5>

Render a demo with its labels overlaid for inspection:

python robocasa/scripts/dataset_scripts/visualize_subtask_labels.py \
    --dataset <demo.hdf5> --demo demo_0 --out demo_0.mp4
Contents