Action Chunking Transformer for Autonomous Driving
A language-conditioned deep-learning policy that predicts short sequences of driving controls from a front-camera image, vehicle state, and a written instruction.
The policy is built around an Action Chunking Transformer (ACT). Instead of predicting one steering command at a time, it predicts the next 20 controls together. At the dataset rate of 10 Hz, each chunk covers two seconds of driving. The controller executes a few actions, observes the road again, and replans.
Project status: the dataset and training code are ready, but no GPU training run has been launched. There is no trained checkpoint or claimed driving result yet.
Autonomous driving is a sequential decision problem. A useful policy has to understand what is visible, remember the current vehicle condition, follow the requested route, and produce controls that make sense together over time.
The model receives:
Input
Shape
Meaning
Front camera
3 x 128 x 128
Current road scene, resized from the 256 x 256 source video
Language matters here. The first camera frame can look almost identical for "turn left," "continue straight," and "turn right." A vision-only ACT policy cannot know which route the driver requested.
Model architecture
text
1Front camera
2 |
3 v
4ResNet-18 feature map -------> 4 x 4 spatial vision tokens
56Vehicle state --------------> state MLP -------------> state token
78Language instruction -------> frozen MiniLM ---------> language token
910Target action chunk --------> CVAE posterior --------> latent token
11 (training only)
1213vision + state + language + latent tokens
14 |
15 v
16 Transformer context encoder
17 |
18 20 learned action queries
19 |
20 v
21 Transformer action decoder
22 |
23 v
24 throttle, brake, steering chunk
Vision
A pretrained ResNet-18 keeps the final spatial feature map instead of collapsing it to a single classification vector. At 128 x 128, the map becomes 16 visual tokens. The backbone is fine-tuned with a smaller learning rate than the policy layers.
Language
MiniLM encodes the instruction. Its token embeddings are mean-pooled and projected into the ACT transformer width. MiniLM is frozen by default. The dataset is large enough to learn the driving policy, but it is not large enough to teach a language encoder from random initialization.
Action chunking
Twenty learned queries decode the full control horizon in parallel. This gives the model an explicit two-second control plan rather than a collection of unrelated single-step guesses. During inference, RecedingHorizonController executes three actions by default and then requests a fresh chunk from the latest observation.
CVAE latent
During training, a posterior transformer reads the target action chunk and predicts a latent distribution. This gives ACT a way to model variation between valid demonstrations. At inference time, the latent is set to zero, so the policy output is deterministic for a given observation and instruction.
Training objective
The policy uses masked action reconstruction plus KL regularization:
The default KL weight is 10.0. Tail positions near the end of an episode are padded, then removed from both the loss and evaluation metrics with an action mask.
The H100 profile processes 64 x 10,000 = 640,000 training samples. This is the same sample budget as the earlier batch-32, 20,000-step setup. Warmup, validation, checkpointing, and final-test batch counts are scaled with the larger batch, so their sample coverage remains comparable. The learning rates stay conservative for the first real run; batch-size scaling alone is not a reason to double them before a loss curve exists.
Driving dataset
Training uses Mayank022/urban-vla-expert-v1, a synchronized simulator dataset with camera video, language, vehicle state, and continuous controls.
Split
Expert episodes
Frames
Hours
Train
756
224,133
6.23
Validation
162
47,614
1.32
Test
162
47,933
1.33
The driving instructions cover left, right, and straight intersection routes, traffic lights, pedestrians, slow-vehicle passing, cut-in yielding, curved-road following, and blocked-lane detours.
The loader follows the supplied split assignments. It does not generate a fresh random split. This keeps held-out world seeds and instruction paraphrases out of training.
The published MP4 files stay at their original 256 x 256 resolution. PyAV resizes each decoded frame directly to 128 x 128 for training. This happens in memory and does not create another dataset folder.
Recovery and failure data
Controlled recovery demonstrations are part of the expert set. They teach the policy how to return from lateral offsets, heading errors, overspeed, and stalled starts.
The 90 deliberately unsafe episodes under raw/failures/ are not behavior-cloning targets. They are reserved for analysis. Asking the model to imitate those actions would be the wrong objective.
Repository structure
text
1act-autonomous-driving/
2|-- configs/
3| `-- base.json # Reproducible training defaults
4|-- src/urban_act/
5| |-- config.py # Validated training configuration
6| |-- data.py # Video streaming and action chunks
7| |-- model.py # Language-conditioned ACT model
8| |-- losses.py # Masked L1 and KL loss
9| |-- metrics.py # Open-loop driving metrics
10| |-- plots.py # Training and prediction figures
11| |-- checkpoints.py # Resume and inference weights
12| |-- train.py # Training and evaluation loop
13| |-- inference.py # Policy and receding-horizon control
14| `-- hub.py # Hugging Face model publishing
15|-- tests/ # CPU unit tests with dummy encoders
16|-- scripts/runpod_bootstrap.sh # Remote dependency and training bootstrap
17|-- scripts/download_hf_run.py # Copy a completed Hub run back locally
18|-- runpod_main.py # One-command trainer for a manually created Pod
19|-- runpod_train.py # Standalone persistent-volume trainer
20|-- runpod_launcher.py # RunPod REST lifecycle client
21|-- modal_app.py # H100 training entrypoint
22|-- requirements-local.txt # Local Modal and Hub tools
23|-- requirements-runpod.txt # Packages not supplied by the GPU image
24|-- requirements.txt # Full training environment
25`-- README.md
The video loader decodes each episode sequentially and constructs future action chunks in memory. It does not extract hundreds of thousands of PNG files before training.
Manual RunPod training
This is the simplest RunPod path. Create the GPU Pod in the RunPod dashboard, open its terminal, clone the repository, and execute one Python file. runpod_main.py handles Python dependencies, CUDA validation, persistent paths, training, resume, evaluation, logs, and the final Hugging Face upload. It does not create or terminate cloud resources.
1. Create the Pod
Use an on-demand H100 SXM or RTX PRO 6000 Pod with:
HF_TOKEN: a Hugging Face write token supplied as a RunPod secret/environment variable
Pod-local /workspace data survives stopping and restarting the same Pod. It is deleted when that Pod is terminated, so wait for the Hugging Face upload or download the artifacts before termination.
The first invocation installs the non-Torch requirements and the CUDA 12.8 TorchVision wheel. It then verifies the GPU before downloading the dataset. The entrypoint disables RunPod's deprecated HF_HUB_ENABLE_HF_TRANSFER default and uses standard HTTP instead of Xet by default, avoiding optional-transfer and Xet token failures for this small dataset. Dataset downloads retry transient Hub failures and honor the resolver reset delay after a 429 response. Set HF_HUB_DISABLE_XET=0 before launch to opt back into Xet. If the selected template does not already contain Torch 2.8, add --install-pytorch. On later invocations, --skip-setup avoids the pip checks.
Rerunning the same command resumes from /workspace/act-driving/artifacts/runs/act-driving-v1/last.pt. A successful run evaluates the best checkpoint and publishes the model, tokenizer, metrics, history, and plots to Hugging Face.
Automated RunPod API setup
RunPod Pods are the recommended path when the account already has RunPod credits. The launcher uses the official REST API and Python's standard library; it does not require the RunPod Python SDK. No GPU is created unless launch --yes is used.
1. Publish the code
The remote Pod clones the Git repository and branch configured in .env. Commit and push these files before launching. The default source is this repository's main branch.
2. Configure local credentials
Create a RunPod API key in the RunPod console, then prepare the ignored environment file:
RUNPOD_API_KEY stays on the Mac and authenticates lifecycle API calls. It is never included in the Pod specification or saved state.
In the RunPod console's Secrets section, create a secret named huggingface_token whose value is the Hugging Face write token. The Pod receives the reference {{ RUNPOD_SECRET_huggingface_token }}, not the plaintext value from the local .env.
3. Create persistent storage
Choose a Secure Cloud data center with H100 availability. Network volumes are tied to one data center, so that selection determines which GPUs can be attached. Create a 50 GB volume through the launcher:
Copy the returned ID into RUNPOD_NETWORK_VOLUME_ID in .env. Existing volumes can be inspected with:
python3 runpod_launcher.py list-volumes
Checkpoints, Hugging Face caches, persistent logs, and final artifacts are written below /workspace/act-driving/ on this volume.
4. Inspect and launch
The dry run prints the exact Pod request and never contacts the billable create endpoint:
python3 runpod_launcher.py launch --dry-run
After checking the GPU, image, branch, batch size, and volume ID, launch the on-demand Pod:
python3 runpod_launcher.py launch --yes
The default is one NVIDIA H100 80GB HBM3, batch size 64, and 10,000 steps. A 12-hour watchdog stops a hung training command; override it with --timeout-hours only when necessary. The launcher refuses an accidental second active Pod unless --allow-duplicate is explicitly supplied. Spot capacity is available with --spot, but it is not recommended for the first training run.
5. Monitor and clean up
The launch response is stored in ignored .runpod/last_pod.json, without credentials. These commands use that saved Pod ID:
bash
1python3 runpod_launcher.py status
2python3 runpod_launcher.py watch3python3 runpod_launcher.py logs
watch reports lifecycle status and hourly cost. logs prints the RunPod dashboard link; container logs carry the trainer's step, percent, loss, learning rate, elapsed time, ETA, and validation events. The same output is retained at /workspace/act-driving/logs/act-driving-v1.log.
The training command exits after final evaluation and Hugging Face publishing, releasing the GPU. Verify that the Pod reaches EXITED, then delete the Pod record:
python3 runpod_launcher.py terminate --yes
RunPod network-volume Pods cannot be stopped and restarted in place. Termination does not delete the attached network volume.
6. Resume or download
To resume, launch again with the same run name, Git branch, and network volume. resume: "auto" loads /workspace/act-driving/artifacts/runs/act-driving-v1/last.pt with the optimizer, scheduler, scaler, step, and random-number states.
After a successful run, copy the published artifact set back to this machine:
This creates artifacts/act-driving-v1/, the layout expected by the inference API.
Modal setup
Run the setup from the repository root:
bash
1python3 -m venv .venv
2source .venv/bin/activate
3python -m pip install -r requirements-local.txt
45hf auth login
6modal setup
78cp .env.example .env
9# Edit .env and set HF_TOKEN. The Modal token fields are optional after modal setup.
The Hugging Face token needs write access to Mayank022/urban-vla-language-act. The local .env is ignored by both Git and the Hugging Face publishing script. At launch, only HF_TOKEN is transferred to the GPU container as an ephemeral Modal secret.
Start Modal training
The following command is ready for the first run, but it has not been executed:
./scripts/start_h100_tmux.sh
The launcher reads .env, starts a detached act-h100 tmux session, and writes output to logs/act-driving-v1.log. Only HF_TOKEN is passed to the GPU container; local Modal credentials are used only by the CLI.
Override the two settings most likely to change during an initial experiment:
The completed run is also pushed to Hugging Face. Each run remains under runs/<run-name>/, while the best inference weights are copied to the repository root.
Evaluation
The trainer selects the best checkpoint using validation mean action MAE. It reads the test split once, after model selection.
Reported metrics include:
masked action L1 over the complete chunk
throttle, brake, and steering MAE and RMSE
steering-direction accuracy for meaningful turns
braking classification accuracy
simultaneous throttle-and-brake rate
action MAE for each language intent
These are open-loop imitation metrics. A low error means the policy resembles the recorded expert on held-out frames. It does not prove that the vehicle can finish a route.
The next evaluation stage must run the trained policy inside the simulator and measure route success, collisions, off-road time, traffic-light violations, pedestrian violations, control smoothness, and recovery success. Until that closed-loop evaluation exists, the repository should not claim autonomous-driving performance.
Inference API
After a checkpoint has been downloaded:
python
1import numpy as np
23from urban_act.inference import ACTPolicy, RecedingHorizonController
45policy = ACTPolicy("artifacts/act-driving-v1")6controller = RecedingHorizonController(policy, replan_interval=3)78camera_rgb = np.zeros((128,128,3), dtype=np.uint8)9vehicle_state = np.array([8.2,0.03,0.4,0.0], dtype=np.float32)10instruction ="Turn left at the next intersection."1112throttle, brake, steering = controller.act(13 camera_rgb,14 vehicle_state,15 instruction,16)
The zero image is only a shape example. Real inference must use the simulator's current RGB frame and synchronized vehicle state.
Local verification
These checks stay on the CPU. They do not request a RunPod or Modal GPU and do not start training.
The tests cover configuration validation, dataset split handling, failure-data exclusion, action bounds, deterministic inference, masked loss, and gradient flow through the ACT policy.
Scope and safety
This is a simulator research project. The images, traffic behavior, vehicle dynamics, and expert controls are synthetic. A model trained here must not be connected to a real vehicle.
