LlamaForecaster-8B is a specialized language model for open-ended forecasting and predicting future events. This model is post-trained from Llama-3.1-8B-Instruct using reinforcement learning on the OpenForesight dataset.
Performance on OpenForesight Test Set
Accuracy
Brier Score
Training Llama-3.1-8B-Instruct on increasing number of samples from OpenForesight leads to continued improvements, making it surpass Qwen3-235B, DeepSeek v3, and almost match R1! We call the final checkpoint (trained on whole of OpenForesight) LlamaForecaster.
Model Description
LlamaForecaster-8B is trained to make calibrated predictions on open-ended questions about future events. The model has been trained to provide calibrated confidence estimates when asked (please prompt explicitly).
Training
This model was trained on the OpenForesight dataset, which contains over 52,000 forecasting questions generated from global news events. The training was done using GRPO optimizing a joint reward function combining accuracy and brier score. Please check the paper for more details.
Base Model: Llama-3.1-8B-Instruct Training Dataset: OpenForesight
Usage
python
1from transformers import AutoTokenizer, AutoModelForCausalLM
23model_name ="nikhilchandak/LlamaForecaster-8B"4tokenizer = AutoTokenizer.from_pretrained(model_name)5model = AutoModelForCausalLM.from_pretrained(model_name)67# template 8prompt ="What is the likelihood that [future event] will occur by [date]?"9# example10prompt ="Who will become the next Prime Minister of India based on the general election to be held in 2029? Provide specific predictions with probabilities."11inputs = tokenizer(prompt, return_tensors="pt")12outputs = model.generate(**inputs, max_length=8192)13prediction = tokenizer.decode(outputs[0], skip_special_tokens=True)14print(prediction)
Performance
LlamaForecaster-8B achieves competitive performance with much larger models like DeepSeek-v3 and Qwen3-235B-A22B on forecasting benchmarks. Key improvements include:
Improved Accuracy: Better prediction of future events
Better Calibration: More reliable confidence estimates
Enhanced Consistency: Reduced logical violations in predictions
Citation
If you use this model in any way, please cite the corresponding paper:
bibtex
1@article{chandak2025scaling,
2 title={Scaling Open-Ended Reasoning to Predict the Future},
3 author={Chandak, Nikhil and Goel, Shashwat and Prabhu, Ameya and Hardt, Moritz and Geiping, Jonas},
4 journal={arXiv preprint arXiv:2512.25070},
5 year={2025}
6}
License
This model is released under the MIT License.
Contact
For questions or issues, please visit our website or open an issue on the model repository.
