COMPASS Qwen2.5-7B-Instruct LoRA (Policy-aware LODO SFT)

Training Data

• Setup: Leave-One-Domain-Out (LODO)
• Held-out domain: TelePath (Telecom)
• Train domains (7): AutoViaMotors, CityGov, FinSecure, MediCarePlus, PlanMyTrip, TutoraVerse, VirtuRecruit
• Training size: 4,121 query–response pairs

Training Configuration

• Method: LoRA adapters
• Epochs: 3
• LoRA rank (r): 64
• LoRA alpha: 128
• Peak learning rate: 5e-4
• Optimizer: AdamW
• Batch size: 32
• LR schedule: cosine
• Quantization: 8-bit during training

Evaluation (Held-out TelePath Domain)

Citation

@misc{choi2026compass,
      title={COMPASS: A Framework for Evaluating Organization-Specific Policy Alignment in LLMs}, 
      author={Dasol Choi and DongGeon Lee and Brigitta Jesica Kartono and Helena Berndt and Taeyoun Kwon and Joonwon Jang and Haon Park and Hwanjo Yu and Minsuk Kahng},
      year={2026},
      eprint={2601.01836},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2601.01836}, 
}