BeginVLA-12M
BeginVLA-12M is the project-owned model from
Begin with VLA. Its vision encoder,
byte-level language encoder, multimodal fusion stack, and flow-matching action decoder are
implemented in the repository and trained end to end from random initialization.
The release is designed for learning: every model input, tensor shape, loss term, checkpoint field,
and evaluation boundary has a corresponding executable lesson in the source repository.
Released variants
| Directory | Training stage | Intended use |
|---|
pretrained/ | 6,072 updates on the audited 29-dataset SO-100 mix | General course, evaluation, and further post-training |
stacking/ | Pretrained model followed by 652 updates on 50 stacking episodes | Target-task prediction and protected replay lesson |
Each directory is a self-contained inference checkpoint with a portable real robot sample. The
approximately 49MB model weights are published; optimizer and RNG state are intentionally omitted.
Every file is covered by SHA-256 in release_manifest.json.
Ten-minute quickstart
1git clone https://github.com/tlysanhuo/begin-with-vla.git
2cd begin-with-vla
3python -m venv .venv
4source .venv/bin/activate
5pip install -e '.[mainline]'
6
7begin-vla quickstart
The command downloads and verifies pretrained/, runs CPU or CUDA inference on the included real
SO-100 observation, and writes prediction.json plus prediction.png under
outputs/quickstart/pretrained/. It does not require the 30GB pretraining dataset.
To inspect modality dependence on the same sample:
To download without running inference:
1begin-vla download --variant pretrained
2begin-vla download --variant stacking
Architecture
| Module | Parameters | Contract |
|---|
| Vision encoder | 2,882,048 | Three steps, up to two 224x224 RGB views |
| Language encoder | 2,725,120 | UTF-8 bytes, at most 96 tokens |
| Multimodal fusion | 2,710,080 | Vision, instruction, and state history |
| Flow action decoder | 3,850,016 | Fifty future actions, up to 32 dimensions |
| Total | 12,167,264 | All parameters trained from random initialization |
The released SO-100 data uses six state and action dimensions. The larger dimensions are padded and
masked to keep the teaching interface explicit.
Training and measured results
The pretrained variant consumed 97,152 observations in one 6,072-update epoch. On the 10,382-sample
held-out split it obtained normalized MSE 0.4120 and normalized MAE 0.4509 with ten flow integration
steps and seed 0.
For the stacking lesson, both pretrained and random-initialized models received the same 50 episodes,
652 updates, optimization configuration, and evaluation seed. The pretrained initialization reached
normalized MSE 0.6895, compared with 1.6715 from random initialization, a 58.7% reduction.
A six-condition ablation over all 10,382 held-out pretraining observations found that removing vision
increased normalized MSE by 21.6%, while replacing state with its training mean increased it by
86.6%. Keeping only the current observation increased MSE by 0.4%. Emptying the instruction increased
MSE by 4.9%, but replacing it with another real task instruction did not hurt aggregate error and the
correct instruction won on only 50.4% of samples. This checkpoint uses vision and state, but stable
task-specific language grounding is not established.
Machine-readable training contracts and results are versioned in the source repository:
Data
Pretraining uses the pinned Apache-2.0
HuggingFaceVLA/community_dataset_v1
revision documented by the source repository. The portable samples retain their dataset, episode,
frame, task, and checksum provenance.
Limitations
- Offline action error is not robot task success.
- The stacking result is one target task with one training and sampling seed.
- Released checkpoints do not establish closed-loop simulation or physical robot success.
- The replay command is a dry run. It does not include calibrated joint limits, collision detection,
watchdog behavior, or a hardware adapter.
- The byte language encoder learns the robot instructions in this dataset; it is not a general
language model.
Do not send generated actions to physical hardware without robot-specific calibration, limits,
watchdog, emergency stop, and collision protection.
