You can deploy this model and start earning money today!

theo-zhou/
lehome-groot-submission-2

Name: lehome-groot-submission-2
Availability: InStock

safetensors

Views

No views yet

LeHome Challenge 2026 — Submission

This HuggingFace repository contains our submission to the LeHome Challenge 2026 (Garment Manipulation Skill Learning in Household Scenarios). It is a derivative of the official lehome-challenge starter code, fine-tuned with GR00T N1.5 via the LeRobot training framework.

It is intended to be self-contained — you can clone or download this single HF repo and reproduce our evaluation end-to-end without any external code source. The same source tree is also attached as a zip in our Google form submission; both contain identical contents.

The original challenge README is preserved at lehome_challenge_README.md.

1. Submission Summary

Item	Value
Submission repo	`theo-zhou/lehome-groot-submission-2`
Policy backbone	GR00T N1.5 (`nvidia/GR00T-N1.5-3B`)
Fine-tuning framework	LeRobot 0.4.3
Training dataset	`Datasets/example/four_types_merged` (LeRobot v3 format)
Training config	configs/train_groot.yaml
Training steps	12,000 (≈ 2.9 epochs at batch_size=64)
Optimizer LR	2e-4 with cosine decay (warmup 5%, decay ratio 0.1, num_decay_steps=12,000)
LoRA	disabled (`lora_rank: 0`)
Tuned modules	`tune_projector=true`, `tune_diffusion_model=true` (LLM + vision encoder frozen)
Self-reported overall success rate	70.00% (240 episodes across 4 categories)
Self-reported per-category SR	top_long 73.33% / top_short 58.33% / pant_long 60.00% / pant_short 88.33%

Detailed per-episode and per-garment results: submission/rollout_results.txt.

2. Prerequisites

Requirement	Version / note
OS	Ubuntu 22.04+ (other Linux distros may work; see IsaacSim 5.1.0 docs)
Python	3.11 (auto-managed by `uv`)
`uv`	Install via `curl -LsSf https://astral.sh/uv/install.sh \| sh`
`xvfb`	System package — `sudo apt-get install -y xvfb`
`huggingface-cli`	Bundled with `huggingface_hub` (already in `uv.lock`)
Free disk space	~20 GB (checkpoint ~7 GB + Assets ~500 MB + `.venv` ~10 GB + scratch)
GPU (optional)	Recommended for policy inference (auto-detected via `torch.cuda.is_available()`, no flag needed); simulation runs on CPU regardless

NVIDIA Omniverse / IsaacSim requires accepting an EULA. Our shs/evals/eval_groot.sh exports the required environment variables (OMNI_KIT_ACCEPT_EULA=YES, ACCEPT_EULA=Y, PRIVACY_CONSENT=Y) before launching the simulator, so no manual prompt should appear.

3. Quick Start (Evaluator's Path)

bash

# 1) Get the submission tree.
#    Option A — pull straight from HuggingFace (source code + checkpoint + metadata):
huggingface-cli download theo-zhou/lehome-groot-submission-2 \
    --exclude "pretrained_model/*" --exclude "dataset_meta/*" \
    --local-dir lehome-challenge
cd lehome-challenge

#    Option B — extract the zip we attached in the Google form submission, then `cd` in.

# 2) Set up the environment with uv
uv sync
source .venv/bin/activate

# 3) Download challenge assets (garments, scenes, robots)
huggingface-cli download lehome/asset_challenge --repo-type dataset --local-dir Assets

# 4) Download our checkpoint into the path the eval script expects
huggingface-cli download theo-zhou/lehome-groot-submission-2 \
    --include "pretrained_model/*" \
    --local-dir outputs/train/groot_four_types_merged_batch64_lr2e-4/checkpoints/012000/

# 5) Download per-category dataset metadata and stage it.
#    HF download preserves repo-relative paths inside --local-dir, so files
#    land at Datasets/_dataset_meta_staging/dataset_meta/<cat>_merged/meta/.
#    We then mv each meta/ into the LeRobot dataset_root layout
#    (<dataset_root>/meta/) and clean up the staging dir.
huggingface-cli download theo-zhou/lehome-groot-submission-2 \
    --include "dataset_meta/*" \
    --local-dir Datasets/_dataset_meta_staging

for cat in top_long top_short pant_long pant_short; do
    mkdir -p Datasets/example/${cat}_merged
    mv Datasets/_dataset_meta_staging/dataset_meta/${cat}_merged/meta \
       Datasets/example/${cat}_merged/meta
done
rm -rf Datasets/_dataset_meta_staging

# 6) (Recommended) Run a smoke test before the full eval (~5-15 min on CPU)
xvfb-run -a python -m scripts.eval \
    --policy_type lerobot \
    --policy_path outputs/train/groot_four_types_merged_batch64_lr2e-4/checkpoints/012000/pretrained_model \
    --garment_type top_long \
    --dataset_root Datasets/example/top_long_merged \
    --num_episodes 1 \
    --max_steps 200 \
    --enable_cameras --headless --device cpu \
    --log_suffix smoke

# 7) Snapshot our self-reported results before re-running the full eval
cp submission/rollout_results.txt /tmp/ours.txt

# 8) Run the full Release evaluation (4 categories × 12 garments × 5 episodes)
bash shs/evals/eval_groot.sh

# 9) Aggregate YOUR fresh logs into submission/rollout_results.txt and diff
bash shs/submit/roll_out_txt_generation.sh          # MODE=self (default)
diff /tmp/ours.txt submission/rollout_results.txt   # see "Reproducibility" note below

# 9b) (Optional) Sanity-check that OUR shipped reference logs reproduce
#     our self-reported numbers via the same aggregator:
MODE=ours bash shs/submit/roll_out_txt_generation.sh
diff /tmp/ours.txt submission/rollout_results.txt   # diff is metadata-only:
                                                    #   - regeneration timestamp ("Generated:")
                                                    #   - per-category "Log mtime:" reflects when you ran step 9b
                                                    # all numeric metrics should match exactly

Note on huggingface-cli vs hf: We use the legacy huggingface-cli name throughout the README because it is what ships with the huggingface_hub version pinned in uv.lock. If you have upgraded to huggingface_hub>=1.0, the equivalent commands are hf download ....

4. Detailed Setup

4.1 Environment

We use uv to manage the Python 3.11 environment.

bash

uv sync
source .venv/bin/activate

uv sync installs the exact versions listed in uv.lock, including isaacsim==5.1.0, lerobot==0.4.3, and the local lehome package (source/lehome). The environment matches the one used during training and the evaluations whose results are reported here.

4.2 Simulation Assets

The challenge garments, scenes, and robots come from the official lehome/asset_challenge HuggingFace dataset. They are not redistributed in our submission.

huggingface-cli download lehome/asset_challenge --repo-type dataset --local-dir Assets

4.3 HuggingFace Repo Layout

The full submission — source code, trained checkpoint, dataset metadata, and reference eval logs — is hosted at theo-zhou/lehome-groot-submission-2. Top-level layout on the Hub:

theo-zhou/lehome-groot-submission-2
├── README.md                             # this file (also rendered as the model card)
├── LICENSE
├── pyproject.toml / uv.lock              # pinned environment definition
├── .python-version
├── lehome_challenge_README.md            # original challenge README, preserved
├── configs/                              # training config (train_groot.yaml)
├── source/lehome/                        # custom IsaacLab simulation env
├── scripts/                              # eval entrypoint + policy adapters + aggregator
├── shs/                                  # eval / submit shell drivers
├── submission/
│   ├── rollout_results.txt               # our self-reported per-episode numbers
│   └── reference_logs/<category>/*.log   # exact eval logs that produced rollout_results.txt
├── pretrained_model/                     # ~6.96 GB GR00T-N1.5 fine-tuned weights
│   ├── config.json
│   ├── model.safetensors
│   ├── policy_postprocessor.json
│   ├── policy_postprocessor_step_0_groot_action_unpack_unnormalize_v1.safetensors
│   ├── policy_preprocessor.json
│   ├── policy_preprocessor_step_2_groot_pack_inputs_v3.safetensors
│   └── train_config.json
└── dataset_meta/                         # LeRobot dataset metadata (no parquet/video)
    ├── top_long_merged/meta/
    ├── top_short_merged/meta/
    ├── pant_long_merged/meta/
    └── pant_short_merged/meta/

pretrained_model/ and dataset_meta/ are kept separate from the source tree so they can be downloaded selectively via --include / --exclude flags (see steps 1, 4, 5 in §3 Quick Start).

Download:

bash

# Checkpoint into the location the eval script expects
huggingface-cli download theo-zhou/lehome-groot-submission-2 \
    --include "pretrained_model/*" \
    --local-dir outputs/train/groot_four_types_merged_batch64_lr2e-4/checkpoints/012000/

# Per-category dataset metadata. Staged under Datasets/_dataset_meta_staging/
# (HF download preserves repo paths inside --local-dir), then moved into the
# LeRobot dataset_root layout (<dataset_root>/meta/).
huggingface-cli download theo-zhou/lehome-groot-submission-2 \
    --include "dataset_meta/*" \
    --local-dir Datasets/_dataset_meta_staging

for cat in top_long top_short pant_long pant_short; do
    mkdir -p Datasets/example/${cat}_merged
    mv Datasets/_dataset_meta_staging/dataset_meta/${cat}_merged/meta \
       Datasets/example/${cat}_merged/meta
done
rm -rf Datasets/_dataset_meta_staging

The dataset metadata is used by LeRobotDatasetMetadata to load feature schemas; the policy's normalization parameters are baked into its own preprocessor safetensors and do not depend on the dataset stats.

5. Running Evaluation

The end-to-end evaluation entrypoint is shs/evals/eval_groot.sh. It iterates over the four garment categories (top_long, top_short, pant_long, pant_short), and for each category runs scripts.eval against the full Release garment list in Assets/objects/Challenge_Garment/Release/.

bash shs/evals/eval_groot.sh

Defaults baked into the script:

Variable	Value
`POLICY_TYPE`	`lerobot`
`POLICIES`	`outputs/train/groot_four_types_merged_batch64_lr2e-4/checkpoints/012000/pretrained_model`
`LOG_SUFFIXES`	`groot-n1.5-batch64-12k`
`NUM_EPISODES`	5 per garment (matches our self-reported runs)
`--device`	`cpu` (simulation only supports CPU per the challenge framework)
`--headless`	enabled (no GUI)

Logs are written to logs/<garment_type>/<MM-DD_HH-MM-SS>_eval_<log_suffix>.log. Each category produces one log file containing per-episode Return / Length / Success, a per-garment summary, and an overall summary block.

Wall-clock time: With both simulation and policy on CPU, a full run (240 episodes = 4 categories × 12 garments × 5 episodes) takes roughly 20-30 hours total on a typical workstation. If a CUDA-capable GPU is visible and the installed PyTorch is a CUDA build, the policy will automatically run on GPU — see scripts/utils/evaluation.py:242 (device = "cuda" if torch.cuda.is_available() else "cpu"); no CLI flag is required. The --device cpu in shs/evals/eval_groot.sh is IsaacLab's AppLauncher flag and only controls the simulator, which the challenge framework requires to be CPU.

5.1 Reproducibility Note

Each episode samples a random initial garment pose, so per-episode Return / Length / Success values will differ between runs. We do not pin a seed because the official challenge protocol uses random initialization.

When comparing against submission/rollout_results.txt, focus on the aggregate metrics:

per-garment success_rate
per-category success_rate (overall)
the 4-category Overall Summary success rate

These should be statistically close to ours (within a few percentage points given NUM_EPISODES=5). If you want tighter agreement, increase NUM_EPISODES in the script — wall-clock time scales linearly with the value.

6. Generating the Rollout Report

shs/submit/roll_out_txt_generation.sh is a thin wrapper around scripts/aggregate_rollout_results.py and supports two modes via the MODE env var.

MODE=self (default) — your own freshly-generated logs

bash shs/submit/roll_out_txt_generation.sh

Picks the latest log per category under logs/<category>/ matching --log_suffix groot-n1.5-batch64-12k (the suffix written by shs/evals/eval_groot.sh) and writes the aggregated report to submission/rollout_results.txt.

MODE=ours — the reference logs we shipped

MODE=ours bash shs/submit/roll_out_txt_generation.sh

Aggregates the four log files we shipped under submission/reference_logs// — these are the exact logs that produced the numbers in our self-reported submission/rollout_results.txt. Useful for sanity-checking that the aggregator deterministically reproduces our reported numbers from the raw logs (the diff against a snapshot of the shipped rollout_results.txt should be empty modulo the regeneration timestamp at the top).

Underlying behavior

In both modes the aggregator:

Picks the latest log per category matching --log_suffix groot-n1.5-batch64-12k under the resolved --logs_dir. Explicit per-category --<category>_log overrides are also supported by the Python script directly.
Parses per-episode results, per-garment summaries, and overall metrics directly from the logs.
Auto-extracts metadata (policy_type, policy_path, action_dim, episodes_per_garment, eval list source) from each log header.
Overwrites submission/rollout_results.txt in place — snapshot it first (cp submission/rollout_results.txt /tmp/ours.txt) if you want to diff against our self-reported numbers afterward.

7. Repository Layout

Path	Purpose
pyproject.toml / uv.lock	Pinned environment definition
source/lehome/	Custom simulation env + utilities
`Assets/`	Created during evaluation; populate via `huggingface-cli download lehome/asset_challenge`
scripts/eval.py	Evaluation entrypoint
scripts/eval_policy/	LeRobot / custom policy adapters
scripts/utils/	Eval loop, parsers, shared helpers
scripts/aggregate_rollout_results.py	Log → submission txt aggregator
shs/evals/eval_groot.sh	Submission eval driver
shs/submit/roll_out_txt_generation.sh	Rollout report driver
configs/train_groot.yaml	Training hyperparameters used for the submitted checkpoint
submission/rollout_results.txt	Our self-reported rollout numbers
submission/reference_logs/	Per-category eval logs that produced `rollout_results.txt` (used by `MODE=ours`)
lehome_challenge_README.md	Original challenge README
`pretrained_model/` (HF only)	Fine-tuned GR00T N1.5 weights (~6.96 GB)
`dataset_meta/` (HF only)	Per-category LeRobot dataset metadata

8. Submission Identifier

HuggingFace repo: https://huggingface.co/theo-zhou/lehome-groot-submission-2
Submission tag: submission_2
Source code is also attached as a zip (submission_2.zip) in our Google form submission; both are byte-for-byte identical to this repo's source tree (excluding the pretrained_model/ and dataset_meta/ folders which are only on HuggingFace due to size).

9. Acknowledgments

This submission builds directly on:

Isaac Sim and Isaac Lab for the simulation backend.
LeRobot for the training and policy loading infrastructure.
NVIDIA GR00T N1.5 for the pretrained vision-language-action backbone.
The LeHome Challenge organizers for the starter code and benchmark.