Model Card for GraphMind Series
This model card describes the GraphMind series of models, which are Large Language Models (LLMs) enhanced for generalized reasoning through continued pre-training on graph-based problems.
Model Description
GraphMind is a series of Large Language Models developed to improve the generalized reasoning capabilities of existing base models.
The core innovation is the continued pre-training (CPT) on GraphPile, a large-scale 10.9 billion token dataset specifically designed with Graph Problem Reasoning (GPR) data.
By training on diverse and complex graph problems—which require sophisticated logical, topological, and relational reasoning—GraphMind models learn more robust and transferable reasoning patterns.
This approach bridges the gap between domain-specific training (e.g., mathematics) and the need for universally capable and adaptable LLMs.
The GraphMind series is built upon three popular open-source models:
- Llama 3
- Llama 3.1
- Gemma 2
Key Features
- Enhanced General Reasoning: Significant gains not only on graph-related tasks but also across mathematical, logical, commonsense, and code reasoning benchmarks.
- Superior Performance on Graph Problems: Thanks to the GraphPile corpus, the models excel at tasks involving graph theory, such as pathfinding, network analysis, and topological sorting.
- Strong Transfer Learning: Reasoning skills acquired from graph problems effectively transfer to other domains.
- Excellent Post-Training Potential: Stronger foundation for fine-tuning on downstream tasks. For instance, the Gemma-based GraphMind fine-tuned on GSM8K achieves 23.6% higher accuracy than its fine-tuned base model.
Performance
GraphMind models show consistent improvements over their base models across reasoning benchmarks.
Generalization Improvements:
- Mathematical Reasoning: up to 4.9% average improvement across 11 datasets.
- Logical Reasoning: 33.4% improvement.
- Code Reasoning: 46.3% improvement.
- Commonsense Reasoning: 7.8% improvement.
- Multi-Hop QA: 10.3% improvement.
Foundational Improvements:
- Graph Problem Reasoning: Average improvement of 53.1% compared to baseline models.
Training Data: The GraphPile Corpus
GraphMind's capabilities are derived from its training on GraphPile, the first large-scale corpus designed for continued pre-training using Graph Problem Reasoning data.
Statistics:
- Total Tokens: 10.9 Billion
- Total Samples: 2.68 Million
- Graph Tasks: 23 distinct tasks covering multiple reasoning paradigms
Data Components:
- Chain-of-Thought (CoT) Data: Step-by-step reasoning processes for graph problems, generated using program-guided methods.
- Program-of-Thought (PoT) Data: Executable code solutions for graph problems, often derived from standard libraries.
- Trace-of-Execution (ToE) Data: Records execution traces of graph algorithms, enabling learning from dynamic algorithmic processes.
- Real-world Graph Data: Includes tasks from sources like DBpedia and DBLP, enriching the dataset with practical contexts.
Training Procedure
The GraphMind models were developed by performing continued pre-training on the GraphPile dataset.
- Base Models: Llama-3-8B, Llama-3.1-8B, Gemma-2-2B
- Learning Rate: 3e-5
- Epochs: 3
- Max Sequence Length: 8192
- Global Batch Size: 1024
- Hardware: 32 × NVIDIA H100 GPUs
Intended Use and Limitations
Intended Use
These models are intended for use in research and development for tasks that demand strong, generalized reasoning. Potential applications include:
- Solving complex logical and mathematical problems.
- Algorithmic reasoning and code generation for graph-related tasks.
- Serving as powerful base models for fine-tuning on reasoning-intensive downstream tasks.
Limitations
- GraphPile is limited to 23 graph problem tasks; more diversity could improve results.
- As reasoning-focused models, GraphMind may perform worse on simpler, non-reasoning tasks such as summarization or translation.
- Further exploration of different GraphPile configurations could yield additional gains.
Available Models
- HKUST-DSAIL/GraphMind-Gemma2-2B
- HKUST-DSAIL/GraphMind-LLAMA-3.1-8B
- HKUST-DSAIL/GraphMind-LLAMA-3-8B
Citation
1@misc{zhang2025improving,
2 title={Improving LLMs' Generalized Reasoning Abilities by Graph Problems},
3 author={Qifan Zhang and Nuo Chen and Zehua Li and Miao Peng and Jing Tang and Jia Li},
4 year={2025},
5 eprint={2507.17168},
6 archivePrefix={arXiv},
7 primaryClass={cs.AI},
8 url={https://arxiv.org/abs/2507.17168v1}
9}
