physicalai-bmi/nano-vla-arm
An
open-vocabulary vision-language-action policy that drives an articulated arm,
running fully in your browser — no server, no GPU, no install. It looks at a
rendered
camera view of a 2-link arm and two colored blocks, reads an
instruction you type in your own words, and outputs
joint commands to move the
arm to the block you named. Run it live at
https://physicalai-bmi.org/research/vla.
A VLA maps
what it sees + what it's told → what it does. This one is
92,338 parameters and runs its whole forward pass — three convolution layers
and all — in plain JavaScript on your device. The language is real and
open-vocabulary: the instruction is embedded on-device by
all-MiniLM-L6-v2 (the
same
encoder used at training time), so phrasings it never saw still work. It is a
faithful
miniature of the paradigm, meant to be understood and run — not a
foundation model.
What it does
- Vision: a 32×32×3 rendered camera view of a 2-link arm (base, links, gold gripper) and a red and green block at random positions.
- Language (open-vocab): any instruction — "reach the red target", "grab the green one", "head toward crimson" — embedded to 384-d on your device.
- Action: two joint deltas
(Δθ₁, Δθ₂). The arm is stepped forward; the loop closes in the browser.
The task is built so the policy must use all three modalities: the blocks and
the arm move every scene (it has to see), either block may be the goal (it has to
read the instruction), and it controls the arm through its joints (real
articulated action, not a teleport).
Architecture
Conv(3→8, s2) → Conv(8→16, s2) → Conv(16→32, s2) → flatten → concat[ MiniLM text embedding, 384-d ] → MLP(→96→2), SiLU. 92,338 parameters.
Results (honest)
Closed-loop on held-out scenes, driving the arm joint-by-joint:
| metric | value |
|---|
| Reaches the correct block, trained instructions | 98.0% |
| Reaches the correct block, novel instructions (never trained) | 98.0% |
| Reaches the correct block when the instruction is flipped | 0.0% |
The novel-instruction row is the open-vocab proof: unseen phrasings still work,
because the on-device text encoder maps them near the ones it learned. The flipped
row is the language-grounding proof: with the wrong instruction it drives to the
other block, so "correct" success collapses to zero.
Trained by behavior cloning on a scripted Jacobian-IK expert, made robust to
closed-loop drift with DART-style state-noise injection (the policy sees
off-expert arm configurations + the expert's recovery there — the same idea as
DAgger). Small synthetic task; not a foundation model, not safety-tested for any
hardware.
Files
model.safetensors — weightsvla.web.json — the same weights as float32 arrays for in-browser inference (conv + MLP; verified bit-for-bit equal to safetensors; max abs diff 1.7×10⁻⁷)metrics.json — the numbers aboveinference.py — minimal NumPy example (renderer + forward pass)
Language encoder
Instructions are embedded with
Xenova/all-MiniLM-L6-v2 (q8) via
Transformers.js on-device. The
policy conditions on the 384-d sentence embedding, so the vocabulary is open —
paraphrases and common color synonyms generalize.
Citation
1@misc{nano_vla_arm_2026,
2 title = {nano-vla-arm: an open-vocab in-browser vision-language-action arm policy},
3 author = {Institute for Physical AI at Bailey Military Institute},
4 year = {2026},
5 howpublished = {\url{https://huggingface.co/physicalai-bmi/nano-vla-arm}}
6}
Released under
CC-BY-4.0 by the Institute for Physical AI @ BMI. See also
nano-vla-reach (the
end-effector predecessor) and
forge-arm-reach-bc (our first checkpoint).
