Input:
  - 3× RGB (head + L/R wrist, 224×224)
  - Proprioception (23-dim joint state)
  - Point cloud (16 × 400 × 3, from head depth)
  - Task prompt

Backbone:
  - PaliGemma-2B (Vision-Language)        ← trainable, bf16
  - PointNet PCD encoder (~30 MB)          ← new, trainable, bf16
  - Flow-matching Action Expert (~300 MB)  ← trainable, bf16

Output:
  - 32-step action horizon × 23-dim

Full fine-tuning (no LoRA). FSDP across 2× A6000 48 GiB. bf16 for everything.

Training Setup

Hyperparameter	Value
Initialization	openpi-comet `pi05-b1kpt50-cs32` (B1K 50-task pretraining)
Tasks	6 challenge tasks: `sorting_household_items`, `clean_up_your_desk`, `picking_up_trash`, `picking_up_toys`, `tidying_bedroom`, `collecting_childrens_toys`
Episodes / task	200 (official human demonstrations)
Total episodes	1,200
Action horizon	32
Batch size	4 (2 GPU × 2 per device)
Optimizer	AdamW
Learning rate	2.5e-6, cosine decay to 20k steps
Noise (training)	Gaussian
Steps (this checkpoint)	10,000 / 20,000
Wall time	~14 hours on 2× A6000
Precision	bf16 (incl. trainable Action Expert)
Distributed	FSDP

This is an intermediate checkpoint at step 10,000, not the final intended 20,000 step run. The original 20k run terminated at ~15k due to a disk-full event; step 10,000 is the last fully-saved checkpoint at the time of upload.

Evaluation

Evaluated on the official OmniGibson 5.1 eval pipeline with obs_modalities=["proprio", "rgb", "depth"] (critical — without "depth" the env never emits depth obs and the policy falls back to zero PCD).

Task	Baseline `pi05-b1kpt50-cs32` Q	This ckpt (gaussian)	N	Notes
`picking_up_trash`	0.000	0.100	10	1/10 success (Q=1.0 on instance 196)
`sorting_household_items`	0.125	0.125	10	10/10 partial credit (subtask 1/8)

Full per-trial metrics and observations: see the code repo docs/RESULTS.md.

Repository Contents

.
├── _CHECKPOINT_METADATA          # Orbax metadata
├── assets/                       # norm stats (per-task)
├── params/                       # Model weights (~4.9 GB, OCDBT format)
└── train_state/                  # Optimizer + RNG state (~8 GB, for resume)

Inference only needs _CHECKPOINT_METADATA + assets/ + params/ (~5 GB). train_state/ is for resuming the SFT run.

Usage

Download

bash

# Inference only (~5 GB)
huggingface-cli download Shawn3636/pi05-pcd-sft-step10k \
  --exclude "train_state/*" \
  --local-dir ./pi05-pcd-sft-step10k

# Full (~13 GB), needed for resuming training
huggingface-cli download Shawn3636/pi05-pcd-sft-step10k \
  --local-dir ./pi05-pcd-sft-step10k

Run inference (with the matching code repo)

bash

# 1) Clone code & patch upstream openpi-comet (see code repo README)
git clone https://github.com/Sunliu36/Behavior1KChallenge_minor_Solution_by_SHAWN

# 2) Start policy server (single GPU)
CUDA_VISIBLE_DEVICES=0 XLA_FLAGS="--xla_gpu_autotune_level=0" \
python scripts/serve_b1k.py \
  --task_name=picking_up_trash \
  --control_mode=receeding_horizon --max_len=32 --port=8000 \
  policy:checkpoint \
  --policy.config=pi05_b1k-6task_sft_gauss_lr2.5e-6_step20k \
  --policy.dir=./pi05-pcd-sft-step10k

# 3) Run eval via OmniGibson (see code repo evaluation/run_eval.sh)

Resume training to step 20,000

Place the full checkpoint at openpi-comet/openpi/10000/ and run:

python scripts/train.py pi05_b1k-6task_sft_gauss_lr2.5e-6_step20k --resume

⚠️ Disk warning: With default keep_period=5000, the run will keep step 10k + 15k + 20k = 39 GB. Set keep_period=20000 in config.py before resuming to keep only the final checkpoint.

Method Insights

Train ↔ inference observation alignment is the prerequisite. Same checkpoint, switching obs_modalities to include "depth" turned the gaussian picking_up_trash average Q from 0.0 → 0.1 (with one Q=1.0 success).
Inference noise should match training noise distribution. Gaussian (matched) > task-correlated Cholesky (mismatched) on this checkpoint.
Capability ≠ robustness at 10k SFT steps. 1/10 success indicates the method works on principle; more training steps are needed to achieve consistent multi-instance robustness.

Limitations

Trained only on 6 of 50 challenge tasks; will likely degrade on out-of-distribution tasks.
Step 10k is intermediate — final intended training is 20k steps.
Inference is slow (~30 min per task instance when policy fails to complete and runs to timeout).

Citation

If you build on this work:

bibtex

@misc{shawn2026pi05pcdsft,
  author       = {Lee, Shawn},
  title        = {Pi0.5 + PointCloud SFT for BEHAVIOR-1K},
  year         = {2026},
  url          = {https://github.com/Sunliu36/Behavior1kChallenge_Solution_by_SHAWN}
}

Underlying architecture (Pi0.5, openpi-comet) and benchmark (BEHAVIOR-1K) should also be cited per their respective licenses.