FluxVLA Engine
1. Model Introduction
FluxVLA Engine is an integrated engineering platform designed for embodied intelligence applications. It follows the core design principles of unified configuration, standardized interfaces, module decoupling, and deployability, forming a complete engineering loop from data collection to real-world deployment. With a focus on building a "standardized industrial-academic-research foundation," FluxVLA significantly lowers the engineering threshold for VLA (Visual Language Agent) research and development.
Key Features
🧩 Rich VLA Model Zoo : Built-in support for mainstream VLA families including OpenVLA, LLaVA-VLA, GR00T, Pi0, Pi0.5, and DreamZero, enabling fast comparison, fine-tuning, and deployment across different model architectures.
⚡ Extreme Inference Speed : CUDA-accelerated operator fusion. With the GR00T-N1.5, the system achieves a high-frequency of 42.8Hz on the RTX 5090.
🛠 All-in-One : A unified configuration file manages critical parameters for data, models, training, evaluation, inference, and deployment, ensuring easy reproducibility and deployment.
🖥 Flexible Configuration : Supports mainstream visual backbones, LLM (Large Language Models), VLM (Vision-Language Models), and VLA (Vision-Language Agents). You can also create and customize your own VLA.
🤖 Real-world Deployment Ready : In addition to mainstream simulators, FluxVLA comes with ready-to-use deployment scripts for Aloha, Tron2, and UR3, eliminating the need for custom integration.
💨 Smooth Motion : Optimized for real-world training/inference with RTC trajectory guidance, ensuring accurate and continuous motion even at 40Hz+ frequencies.
📊 Strong Benchmark Performance : Provides competitive LIBERO benchmark results across spatial, object, goal, and long-horizon tasks, making it easier to evaluate VLA models under standardized settings.
🛰 Remote Inference Ready : Includes a ZMQ-based server/client inference framework, enabling GPU-offloaded remote inference for resource-constrained robot-side devices.
🧪 Scalable Training Pipeline : Supports distributed training with DDP/FSDP, LoRA fine-tuning, eval-after-train, and checkpoint-based resume, covering both local debugging and cluster-scale training workflows.
2. Model Zoo
FluxVLA Engine supports multiple VLA model families, each optimized for different use cases:
Family Parameters Key Features Typical Use Cases GR00T 3B High-frequency action prediction, CUDA-optimized Real-time robot control, deployment Pi0 / Pi0.5 3B Flow-matching VLA, smooth trajectories Continuous control, manipulation tasks DreamZero 23B Diffusion/flow-based generation Complex motion planning, LIBERO tasks LLaVA-VLA 3B-7B VLM-based architecture Vision-language grounding, instruction following OpenVLA 7B Open-source baseline Benchmark comparison, fine-tuning
Available Pretrained Models
Trained FluxVLA Checkpoints
Model Training Data Performance Download PI0.5 PaliGemma LIBERO-10 96.0% avg Hugging Face Cosmos3-Edge LIBERO benchmark suites (mixed training) 94.4% avg Hugging Face GR00T Eagle 3B LIBERO-10 89.4% avg Hugging Face
3. Evaluation Results
FluxVLA demonstrates state-of-the-art performance across multiple LIBERO benchmark suites:
LIBERO Benchmark Performance
Model Libero-Spatial Libero-Object Libero-Goal Libero-Long Average FluxVLA (Pi0.5) 98.6 99.0 97.8 96.0±1.0 97.85 FluxVLA (Qwen3VL+GR00T) 98.6 99.6 95.6 92.2±1.8 96.50 FluxVLA (DreamZero) 96.8 97.4 90.8±1.5 93.6 94.65 FluxVLA (Cosmos3-Edge) 95.6 95.6 91.6 94.8 94.4 FluxVLA (GR00T) 96.2 96.8 93.4 89.4±1.5 93.95
Inference Speed Comparison
Model Hardware Inference Speed Acceleration GR00T N1.5 (Optimized) RTX 5090 42.8 Hz 3.2x GR00T N1.5 (Baseline) RTX 5090 13.4 Hz 1.0x Pi0.5 (Optimized) RTX 5090 38.5 Hz 2.8x Pi0.5 (Baseline) RTX 5090 13.7 Hz 1.0x
Note: Optimized versions use CUDA-accelerated operator fusion and Triton kernels.
4. Architecture Overview
FluxVLA Engine supports flexible architecture combinations:
Framework Architecture
Supported Components
Component Type Options Vision Encoders DINOv2 ViT-Large, SigLIP ViT-SO400M, SigLIP2, PaliGemma Language Models Qwen2.5 (3B/7B), Llama 2 (7B) Vision-Language Models Qwen2.5-VL (3B), PaliGemma (3B) Action Heads Diffusion, Flow Matching, Direct Regression Precision FP32, FP16, BF16
5. Installation & Deployment
Quick Installation
1 # Clone repository
2 git clone https://github.com/limxdynamics/FluxVLA.git
3 cd FluxVLA
4
5 # Create environment
6 conda create -n fluxvla python = 3.10 -y
7 conda activate fluxvla
8
9 # Install PyTorch (CUDA 12.4)
10 pip install torch == 2.6 .0 torchvision == 0.21 .0 torchaudio == 2.6 .0 \
11 --index-url https://download.pytorch.org/whl/cu124
12
13 # Install flash-attention
14 pip install psutil ninja packaging
15 MAX_JOBS = 8 pip install flash-attn == 2.5 .5 --no-build-isolation
16
17 # Install FluxVLA
18 pip install -r requirements.txt
19 pip install --no-build-isolation -e .
Deployment Options
FluxVLA supports multiple deployment scenarios:
Local Training & Evaluation : Single-node or multi-node distributed training with DDP/FSDPReal-Robot Deployment : Direct on-robot inference with optimized runtimeRemote Inference : GPU server + robot client architecture via ZMQSimulation : LIBERO, Isaac Sim integration
Hardware Requirements
Deployment Type Minimum GPU Recommended GPU Memory Training (3B model) RTX 3090 (24GB) A100 (40GB) 32GB+ RAM Inference (optimized) RTX 4090 (24GB) RTX 5090 (32GB) 16GB+ RAM Remote Inference Server RTX 4090 (24GB) RTX 5090 (32GB) 16GB+ RAM Robot Client CPU only - 8GB+ RAM
6. Model Usage
Training
Train a VLA model on LIBERO-10 dataset:
1 # Single-node training
2 torchrun \
3 --standalone \
4 --nnodes 1 \
5 --nproc-per-node 2 \
6 scripts/train.py \
7 --config configs/pi05/pi05_paligemma_libero_10_full_finetune.py \
8 --work-dir ./checkpoints/pi05_paligemma_libero_10_full_finetune \
9 --cfg-options train_dataloader.per_device_batch_size = 2
10
11 # Or use the launcher script
12 bash scripts/train.sh \
13 configs/gr00t/gr00t_eagle_3b_libero_10_full_finetune.py \
14 work_dirs/gr00t_eagle_3b_libero_10_full_finetune
Evaluation
Evaluate a trained model on LIBERO benchmarks:
1 # Evaluate on LIBERO-10
2 torchrun \
3 --standalone \
4 --nnodes 1 \
5 --nproc-per-node 2 \
6 scripts/eval.py \
7 --config configs/pi05/pi05_paligemma_libero_10_full_finetune.py \
8 --ckpt-path checkpoints/pi05_paligemma_libero_10_full_finetune_bs64/checkpoints/latest-checkpoint.safetensors
9
10 # Or use the launcher script
11 bash scripts/eval.sh \
12 configs/pi05/pi05_paligemma_libero_10_full_finetune.py \
13 checkpoints/pi05_paligemma_libero_10_full_finetune_bs64/checkpoints/latest-checkpoint.safetensors
Inference
Simulation Inference
1 import torch
2 from fluxvla . models import build_model
3 from fluxvla . datasets import build_dataset
4 from fluxvla . transforms import build_transforms
5
6 # Load configuration
7 config = load_config ( 'configs/pi05/pi05_paligemma_libero_10_full_finetune.py' )
8
9 # Build model
10 model = build_model ( config . model )
11 model . load_checkpoint ( 'path/to/checkpoint.safetensors' )
12 model . eval ( )
13 model . cuda ( )
14
15 # Build transforms
16 transforms = build_transforms ( config . transforms )
17
18 # Inference loop
19 with torch . no_grad ( ) :
20 # Prepare input
21 image = transforms [ 'image' ] ( raw_image )
22 instruction = "Pick up the red cube"
23
24 # Get action prediction
25 action = model . predict (
26 image = image ,
27 instruction = instruction ,
28 proprioception = robot_state
29 )
30
31 # Execute action on robot
32 robot . execute ( action )
Real-Robot Inference
1 # Direct on-robot inference
2 python scripts/inference_real_robot.py \
3 --config configs/pi05/pi05_paligemma_aloha_full_finetune.py \
4 --ckpt-path checkpoints/pi05_paligemma_libero_10_full_finetune_bs64/checkpoints/latest-checkpoint.safetensors
Remote Inference
For deployment scenarios where the robot cannot host the full model:
1 # On GPU server
2 python scripts/remote_inference_server.py \
3 --config configs/pi05/pi05_paligemma_aloha_remote_inference.py \
4 --ckpt-path checkpoints/pi05_paligemma_libero_10_full_finetune_bs64/checkpoints/latest-checkpoint.safetensors \
5 --host 0.0 .0.0 \
6 --port 5555
7
8 # On robot client
9 bash scripts/remote_inference_client.sh
7. Advanced Features
RTC (Real-Time Control) Trajectory Guidance
FluxVLA includes RTC trajectory guidance for smooth, continuous motion control:
RTC Comparison
RTC improves action continuity by conditioning on recent action history, reducing jitter and improving success rates in real-world deployment.
1 # Enable RTC in config
2 model = dict (
3 type = 'Pi05VLA' ,
4 rtc_enabled = True ,
5 rtc_prefix_length = 5 ,
6 rtc_temperature = 0.1 ,
7 )
Inference Acceleration
FluxVLA provides multiple acceleration techniques:
CUDA Operator Fusion : Custom CUDA kernels for attention and MLP layersTriton Kernels : Optimized Triton implementations for key operationsCUDA Graph : Graph capture for reduced kernel launch overheadMixed Precision : FP16/BF16 inference with minimal accuracy loss
1 # Enable acceleration in config
2 python scripts/inference.py \
3 --config configs/pi05/pi05_paligemma_libero_10_full_finetune.py \
4 --ckpt-path checkpoints/latest.safetensors \
5 --use-cuda-graph \
6 --use-triton-kernels
Distributed Training
FluxVLA supports both DDP and FSDP for multi-GPU training:
1 # Multi-node training with FSDP
2 torchrun \
3 --nnodes 4 \
4 --nproc-per-node 8 \
5 --rdzv-backend c10d \
6 --rdzv-endpoint $MASTER_ADDR : $MASTER_PORT \
7 scripts/train.py \
8 --config configs/pi05/pi05_paligemma_libero_10_full_finetune.py \
9 --work-dir ./checkpoints/distributed_training
LoRA Fine-tuning
For efficient fine-tuning with limited compute:
1 # Enable LoRA in config
2 lora = dict (
3 enabled = True ,
4 r = 16 ,
5 lora_alpha = 32 ,
6 target_modules = [ 'q_proj' , 'v_proj' , 'k_proj' , 'o_proj' ] ,
7 lora_dropout = 0.05 ,
8 )
8. Supported Datasets
FluxVLA supports multiple robot learning datasets:
9. Citation
If you use FluxVLA in your research, please cite:
1 @software{fluxvla2026,
2 title={FluxVLA Engine: An All-in-One VLA Engineering Platform for Embodied AI},
3 author={LimX Dynamics},
4 year={2026},
5 url={https://github.com/limxdynamics/FluxVLA},
6 note={GitHub repository}
7 }
10. License
The FluxVLA code is released under the
Apache-2.0 License . See
LICENSE for details.
Model checkpoints derived from third-party base models remain subject to their respective licenses. The Cosmos3-Edge checkpoints are derived from NVIDIA Cosmos3-Edge and are subject to the
Open Model Development and Weight License 1.1 (OpenMDW 1.1) .
11. Acknowledgements
FluxVLA builds on and benefits from the following open-source projects:
LeRobot - Robot learning datasets and toolsNVIDIA Isaac GR00T - Foundation model for humanoid robotsOpenVLA - Open-source VLA baselineOpenPI - Flow-matching VLA frameworkLLaVA - Vision-language model architectureQwen - Large language modelsDeepSpeed - Distributed training optimizationTriton - GPU programming frameworkRTC - Real-time control trajectory guidance
We thank the authors and contributors of these projects for their excellent work.
12. Contact Us
For questions, feedback, collaboration, or technical support, please contact us:
You can also:
Open an issue on GitHub
Visit our documentation: English | 中文
Check out our Hugging Face page: limxdynamics/FluxVLAEngine
Project Links:
