In this work, we use fine-tuned VLMs to provide feedback for reinforcement learning agents in autonomous driving scenarios.
-
RGB + Text (LoRA SFT):
- Visual Input: Front-view RGB camera images (shape = 900 * 256).
- Method: Used for LoRA (Low-Rank Adaptation) Supervised Fine-Tuning.
- Purpose: To enable the VLM to understand visual scenes and follow driving instructions based on realistic camera feeds.
-
Rendered BEV + Text (Full SFT):
- Visual Input: Rendered Bird's Eye View (BEV) semantic maps (shape = 192 * 192).
- Method: Used for Full Parameter Supervised Fine-Tuning.
- Purpose: To provide a holistic spatial understanding of the driving environment, allowing the VLM to act as an expert.
1@misc{qu2026foundrl,
2 title={Found-RL: foundation model-enhanced reinforcement learning for autonomous driving},
3 author={Yansong Qu and Zihao Sheng and Zilin Huang and Jiancong Chen and Yuhao Luo and Tianyi Wang and Yiheng Feng and Samuel Labi and Sikai Chen},
4 year={2026},
5 eprint={2602.10458},
6 archivePrefix={arXiv},
7 primaryClass={cs.AI},
8 url={https://arxiv.org/abs/2602.10458},
9}
