flan-t5-base-mixed-7-1-catastrophic
Model trained as part of: "Mitigating Catastrophic Forgetting in Mathematical Reasoning Finetuning through Mixed Training"
This model investigates catastrophic forgetting when finetuning language models for specialized tasks. We demonstrate that math-only training causes severe NLI degradation (81% → 16.5%), while mixed training eliminates this forgetting while maintaining equivalent mathematical performance.
Quick Links
Model Description
This is the final checkpoint after completing 3 epochs of training.
This checkpoint represents the mixed-7-1-final training configuration from our systematic study of catastrophic forgetting mitigation strategies.
Training Configuration
- Base Model: google/flan-t5-base (250M parameters)
- Training Type: Mixed (7:1 Math:NLI)
- Math Dataset: DeepMind Mathematics dataset (algebra__linear_1d subset), 392,702 training examples
- NLI Dataset: MultiNLI (matched + mismatched splits), 392,702 training examples
- Training Details:
- Learning rate: 3e-4 with cosine decay
- Warmup: 6% of total steps
- Epochs: 3
- Effective batch size: 256 examples
- Precision: bfloat16
- Optimizer: FusedAdam
- Hardware: Single NVIDIA A100 (40GB)
This model was trained with a 7:1 mixing ratio, meaning 87.5% math examples and 12.5% NLI examples in each batch.
Performance
Evaluation Protocol: Final evaluation on complete validation sets
- Math: 10,000 examples (DeepMind Mathematics linear algebra 1D)
- NLI: 9,815 examples (MultiNLI matched split)
| Task | Accuracy | Baseline | Δ from Baseline |
|---|
| Mathematical Reasoning | 11.7% | 3.1% | +8.6pp |
| Natural Language Inference | 84.5% | 81.0% | +3.5pp |
Key Findings from Our Study
- Catastrophic Forgetting is Severe: Math-only training drops NLI accuracy from 81% to 16.5% (−64.5pp)
- Mixed Training Eliminates Forgetting: Balanced 1:1 ratio maintains 86.2% NLI while achieving 12.0% math
- No Performance Trade-off: Mixed training matches math-only performance (12.0% vs 12.0%)
- Minimal Exposure Suffices: Even 6.25% NLI exposure (15:1 ratio) prevents catastrophic collapse
Usage
Basic Inference
1from transformers import T5ForConditionalGeneration, T5Tokenizer
2
3# Load model and tokenizer
4model = T5ForConditionalGeneration.from_pretrained("MarioBarbeque/flan-t5-base-mixed-7-1-catastrophic")
5tokenizer = T5Tokenizer.from_pretrained("MarioBarbeque/flan-t5-base-mixed-7-1-catastrophic")
6
7# Mathematical reasoning example
8math_input = "Solve 24 = 1601*c - 1605*c for c."
9inputs = tokenizer(math_input, return_tensors="pt")
10outputs = model.generate(**inputs, max_new_tokens=8)
11print(tokenizer.decode(outputs[0], skip_special_tokens=True))
12# Expected: "-6"
13
14# NLI example
15nli_input = "mnli premise: The cat sat on the mat. hypothesis: An animal was on the mat."
16inputs = tokenizer(nli_input, return_tensors="pt")
17outputs = model.generate(**inputs, max_new_tokens=8)
18print(tokenizer.decode(outputs[0], skip_special_tokens=True))
19# Expected: "yes" (entailment)
Batch Processing
1import torch
2
3# Batch of linear algebra problems
4math_problems = [
5 "Solve 24 = 1601*c - 1605*c for c.",
6 "Solve 657 = -220*t + 1086*t + 22307 for t.",
7 "Solve -11*y - 263*y + 3162 = -88*y for y."
8]
9
10inputs = tokenizer(math_problems, return_tensors="pt", padding=True)
11outputs = model.generate(**inputs, max_new_tokens=8)
12results = tokenizer.batch_decode(outputs, skip_special_tokens=True)
13print(results)
Training Code
The complete training code, evaluation scripts, and experiment configurations are available in our
GitHub repository.
Related Models
- Larger Scale: CyberSolve-LinAlg-1.2 - Flan-T5-Large (780M) achieving 90.8% on math (8× improvement over this 250M model)
- Other Experiments: See all checkpoints from this study at MarioBarbeque's models
Citation
If you use this model in your research, please cite:
1@article{reynolds2025catastrophic,
2 title={Mitigating Catastrophic Forgetting in Mathematical Reasoning Finetuning through Mixed Training},
3 author={Reynolds, John Graham},
4 journal={arXiv preprint arXiv:2512.13706},
5 year={2025},
6 url={https://github.com/johngrahamreynolds/mathematical_catastrophe_mitigation}
7}
For the CyberSolve-LinAlg model (Flan-T5-Large baseline):
1@misc{cybersolve2024,
2 author={Reynolds, John Graham},
3 title={CyberSolve-LinAlg: Flan-T5-Large Finetuned for Linear Algebra Problem Solving},
4 year={2024},
5 howpublished={\url{https://huggingface.co/MarioBarbeque/CyberSolve-LinAlg-1.2}}
6}
License
This model is released under the Apache 2.0 license, following the base model (google/flan-t5-base).
Model Card Authors
John Graham Reynolds (
@MarioBarbeque)
Contact
Acknowledgments
This research would not have been possible without the wonderful instruction of Greg Durrett. The author would also like to thank John Jumper for motivating this research during his visit to Vanderbilt University.
