CoALM-405B: The Largest Open-Source Agentic LLM

🌟 Model Overview

Model Sources

• 📝 Paper: https://arxiv.org/abs/2502.08820
• 🌐 Project Page: https://emrecanacikgoz.github.io/CoALM/
• 💻 Repository: https://github.com/oumi-ai/oumi/tree/main/configs/projects/CALM
• 💎 Dataset: https://huggingface.co/datasets/uiuc-convai/CoALM-IT

🚀 Model Details

• Model Name: CoALM-405B
• Developed by: Colloboration of UIUC Conversational AI LAB and Oumi
• License: cc-by-nc-4.0
• Architecture: Meta-Llama 3.1-405B Instruct
• Training Data: CoALM-IT
• Fine-tuning Framework: Oumi
• Training Hardware: 8 NVIDIA H100 GPUs
• Training Duration: ~6.5 days
• Evaluation Benchmarks: MultiWOZ 2.4, BFCL V3, API-Bank
• Release Date: February 5, 2025

🏆 Why CoALM-405B is a Game-Changer

• 🚨 Largest Open-Source Agentic LLM: A 405B parameter model that brings state-of-the-art agentic capabilities to the public domain.
• 🎯 Best Open-Source Performance on BFCL V3: Outperforms leading proprietary models like GPT-4o, Gemini, and Claude in function-calling tasks.
• 🔍 True Zero-Shot Function Calling: Generalizes to unseen API tasks with unmatched accuracy.
• 🤖 Multi-Turn Dialogue Mastery: Excels at long conversations, task tracking, and complex reasoning.
• 🛠 API Tool Use and Reasoning: Makes precise API calls, interprets responses, and synthesizes coherent multi-step solutions.
• 📜 Fully Open-Source & Reproducible: Released under cc-by-nc-4.0, including model weights, training logs, and datasets.

💡 CoALM-IT Dataset

📊 Benchmark Performance

🔧 Training Process

Fine-tuning Stages

1. TOD Fine-tuning: Optimized for dialogue state tracking (e.g., augmented SNIPS in instruction-tuned format).
2. Function Calling Fine-tuning: Trained to generate highly accurate API calls from LA datasets.
3. ReAct-based Fine-tuning: Enhances multi-turn conversations with structured thought-action-observation-response reasoning.

Training Hyperparameters

• Base Model: Meta-Llama 3.1-405B Instruct
• LoRA Config: Rank = 16, Scaling Factor = 32
• Batch Size: 2
• Learning Rate: 1e-4
• Optimizer: AdamW (betas = 0.9, 0.999, epsilon = 1e-8)
• Precision: q4
• Warm-up Steps: 500
• Gradient Accumulation Steps: 1

❗️ How to Use CoALM-405B

🏗 How to Load the Model using HuggingFace

🛠 Example Oumi Inference

🛠 Example Oumi Fine-Tuning

Acknowledgements

License

📖 Citation

@misc{acikgoz2025singlemodelmastermultiturn,
      title={Can a Single Model Master Both Multi-turn Conversations and Tool Use? CoALM: A Unified Conversational Agentic Language Model}, 
      author={Emre Can Acikgoz and Jeremiah Greer and Akul Datta and Ze Yang and William Zeng and Oussama Elachqar and Emmanouil Koukoumidis and Dilek Hakkani-Tür and Gokhan Tur},
      year={2025},
      eprint={2502.08820},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2502.08820}, 
}