A 3B model trained from scratch through the end of 2024 (knowledge cutoff January 2025),
released as part of Frontier to Point-in-Time — a toolkit for
reducing look-ahead bias in forecasting with LLMs, without sacrificing what makes them
useful in the first place. The toolkit adapts open-source frontier models to substantially
reduce look-ahead bias, applies the methods to Qwen 3.5 27B to convert it into a 2015
point-in-time model, and ships production-ready inference code.
This model is the recent-era half of the temporal pair used by Divergence Decoding: its
logits stand in for the frontier model's post-2015 knowledge, which the method cancels at
inference time, with no retraining of the large model. Like its twin, it is a from-scratch
model with a 128K context window and best-in-class instruction following; the 2015 twin
appears as "Aux 2015" in the project's examples and benchmarks.
Aux 2024 — trained through the end of 2024
(knowledge cutoff January 2025)
The two models are era-matched: same architecture, tokenizer, and training recipe, differing
only in the time span of their pre-training corpus. Their logit difference is the signal
used by Divergence Decoding.
How it's used
Divergence Decoding (DD) is state-of-the-art for Q&A unlearning and is designed to scale
to large datasets. Frontier to Point-in-Time uses it to unlearn all knowledge after the
cutoff of December 31, 2015. It requires the two auxiliary models and applies the following
inference-time adjustment to the large model's logits:
where l_frontier are the frontier model's logits (Qwen 3.5 27B), l_aux-2024 are this
model's, and l_aux-2015 are the 2015 twin's
(Aux 2015).
The auxiliary models are trained from scratch with the Qwen 3.5 tokenizer, then SFT'd on
samples distilled from the large model so they inherit its chat template and response style.
Unlike prior work, which saves checkpoints from a single training run as data is introduced
chronologically, each auxiliary model is trained independently.
See the project page for the full method — including the
feature-steering component — and 🚨 the
GitHub repository for production-ready
inference code.🚨 The usage example below is pure Hugging Face and not meant as an example of production inference setup.
Model details
Architecture:MinistralDualRope — a Ministral (Llama-family math: SiLU-gated MLP,
RMSNorm, GQA, rotary, no biases) with a per-layer sliding/full attention pattern and a
second, unscaled rotary for the sliding layers (the Gemma-3 dual-RoPE design).
Parameters: ~3.4B — 28 layers, hidden size 3072, 24 attention heads, 8 KV heads,
head dim 128, intermediate size 8192.
Attention: sliding window 512, every 6th layer full attention.
Vocab: 248,320 (shared with the frontier model so DD can bridge logits).
Precision: bf16.
Training: temporal cooldown base followed by a partial supervised fine-tuning pass on
samples distilled from the frontier model, so it inherits the frontier model's chat
template and response style. The chat template supports optional thinking
(enable_thinking).
Usage
This is a custom architecture — load with trust_remote_code=True:
1@inproceedings{
2 merchant2026divergence,
3 title={Divergence Decoding: Inference-Time Unlearning via Auxiliary Models},
4 author={Humzah Merchant and Bradford Levy},
5 booktitle={Forty-third International Conference on Machine Learning},
6 year={2026},
7 url={https://openreview.net/forum?id=JPbp2S9yTO}
8}
9@inproceedings{
10 merchant2026a,
11 title={A Fast and Effective Solution to the Problem of Look-ahead Bias in {LLM}s},
12 author={Humzah Merchant and Bradford Levy},
13 booktitle={NeurIPS 2025 Workshop: Generative AI in Finance},
14 year={2026},
15 url={https://openreview.net/forum?id=zYsLIPgM28}
16}
17@inproceedings{
18 merchant2026forecasting,
19 title={Forecasting With {LLM}s: Improved Generalization Through Feature Steering},
20 author={Humzah Merchant and Bradford Levy},
21 booktitle={Forecasting as a New Frontier of Intelligence},
22 year={2026},
23 url={https://openreview.net/forum?id=ppN6CmoNOk}
24}
