Brain: Spiking Visuomotor Policy
Model Description
Brain is a recurrent Spiking Neural Network (SNN) visuomotor policy trained end-to-end for robot manipulation through imitation learning.
The model processes:
- Dual RGB camera observations
- 7-dimensional proprioceptive joint state
and predicts a 7-dimensional action delta (6 joints + gripper) at every timestep.
The policy is implemented entirely with spiking neural network components using SpikingJelly, employing Parametric Leaky Integrate-and-Fire (PLIF) neurons, recurrent temporal processing, and surrogate-gradient backpropagation.
The released checkpoint is the best-performing model trained using delta-action prediction, which significantly outperformed an equivalent absolute-action formulation.
Model Details
| Property | Value |
|---|
| Architecture | Recurrent Spiking Neural Network |
| Parameters | 1.65M |
| Framework | PyTorch + SpikingJelly |
| Neuron Type | Parametric LIF (PLIF) |
| Inputs | Dual RGB images + 7-DoF joint state |
| Output | 7-DoF action delta |
| Training Episodes | 407 teleoperated demonstrations |
| Best Checkpoint | full_run_v1_delta (step 3600) |
Architecture
1Global Camera ───────────┐
2 │
3 ▼
4 Spiking CNN Encoder
5 │
6Gripper Camera ───────────┤
7 ▼
8 Spiking CNN Encoder
9 │
10Joint State ──────────────┐
11 ▼
12 Spiking MLP Encoder
13 │
14 ▼
15 Recurrent PLIF Fusion Network
16 │
17 ▼
18 Linear Projection
19 │
20 ▼
21 Non-Spiking LIF Readout
22 │
23 ▼
24 7-DoF Action Delta
The network consists of independent spiking vision encoders for each camera, a spiking proprioceptive encoder, recurrent PLIF fusion layers, and a non-spiking LIF readout that produces continuous control outputs.
Training Data
The model was trained using 407 teleoperated robot manipulation demonstrations containing:
- Dual synchronized RGB camera streams
- 7-dimensional robot joint state
- Demonstrated robot actions
Training images were resized to 128×128, and trajectories were divided into overlapping temporal windows of length 32 with stride 16.
Training Objective
Rather than predicting the next absolute joint configuration, the model predicts the difference between the demonstrated action and the current joint position.
This formulation substantially improved optimization by matching the scale of real per-step robot motion, resulting in significantly lower prediction error than direct absolute-action regression.
Evaluation
Evaluation was performed offline on a held-out validation split using open-loop trajectory prediction.
Best Checkpoint (full_run_v1_delta)
| Metric | Value |
|---|
| MAE | 0.00989 rad |
| RMSE | 0.01873 rad |
| R² | 0.99917 |
| Cosine Similarity | 0.99960 |
| Mean Episode Correlation | 0.99135 |
| Final Pose Error | 0.05417 rad |
Compared to the absolute-action baseline:
| Metric | Absolute | Delta |
|---|
| MAE | 0.0834 | 0.0099 |
| RMSE | 0.1278 | 0.0161 |
| Mean Correlation | 0.812 | 0.991 |
Intended Use
This model is intended for research in:
- Spiking neural networks
- Robot imitation learning
- Neuromorphic computing
- Multimodal visuomotor policies
- Temporal sequence modeling for robot control
Limitations
The model has been evaluated only in offline open-loop settings against held-out demonstrations.
Reported metrics measure trajectory prediction accuracy rather than real-world task completion. Physical deployment or closed-loop simulation is required to assess manipulation performance.
Citation
If you use this model in your research, please cite this repository.
1@misc{brain2026,
2 title={Brain: A Recurrent Spiking Neural Network for Visuomotor Control},
3 author=Adith,
4 year={2026},
5 howpublished={Hugging Face}
6}
