A 3B point-in-time model trained from scratch with a knowledge cutoff of December 31, 2015,
released as part of Frontier to Point-in-Time — a toolkit for
reducing look-ahead bias in forecasting with LLMs, without sacrificing what makes them
useful in the first place. The toolkit adapts open-source frontier models to substantially
reduce look-ahead bias, applies the methods to Qwen 3.5 27B to convert it into a 2015
point-in-time model, and ships production-ready inference code.
This model plays two roles in that release:
Auxiliary model for Divergence Decoding — the 2015 half of the temporal pair below, used
to unlearn post-cutoff knowledge from the frontier model at inference time, with no
retraining of the large model.
From-scratch point-in-time model — with a 128K context window and best-in-class
instruction following (the instruction-tuned point-in-time models benchmarked on the project
page have 1.7K–4K context windows). It appears as "Aux 2015" in the project's examples
and benchmarks.
Aux 2024 — trained through the end of 2024
(knowledge cutoff January 2025)
The two models are era-matched: same architecture, tokenizer, and training recipe, differing
only in the time span of their pre-training corpus. Their logit difference is the signal
used by Divergence Decoding.
How it's used
Divergence Decoding (DD) is state-of-the-art for Q&A unlearning and is designed to scale
to large datasets. Frontier to Point-in-Time uses it to unlearn all knowledge after the
cutoff of December 31, 2015. It requires the two auxiliary models and applies the following
inference-time adjustment to the large model's logits:
where l_frontier are the frontier model's logits (Qwen 3.5 27B), l_aux-2015 are this
model's, and l_aux-2024 are the recent-era twin's
(Aux 2024).
The auxiliary models are trained from scratch with the Qwen 3.5 tokenizer, then SFT'd on
samples distilled from the large model so they inherit its chat template and response style.
Unlike prior work, which saves checkpoints from a single training run as data is introduced
chronologically, each auxiliary model is trained independently.
See the project page for the full method — including the
feature-steering component — and 🚨 the
GitHub repository for production-ready
inference code.🚨 The usage example below is pure Hugging Face and not meant as an example of production inference setup.
Model details
Architecture:MinistralDualRope — a Ministral (Llama-family math: SiLU-gated MLP,
RMSNorm, GQA, rotary, no biases) with a per-layer sliding/full attention pattern and a
second, unscaled rotary for the sliding layers (the Gemma-3 dual-RoPE design).
Parameters: ~3.4B — 28 layers, hidden size 3072, 24 attention heads, 8 KV heads,
head dim 128, intermediate size 8192.
Attention: sliding window 512, every 6th layer full attention.
Vocab: 248,320 (shared with the frontier model so DD can bridge logits).
Precision: bf16.
Training: temporal cooldown base followed by a partial supervised fine-tuning pass on
samples distilled from the frontier model, so it inherits the frontier model's chat
template and response style. The chat template supports optional thinking
(enable_thinking).
Usage
This is a custom architecture — load with trust_remote_code=True:
1@inproceedings{
2 merchant2026divergence,
3 title={Divergence Decoding: Inference-Time Unlearning via Auxiliary Models},
4 author={Humzah Merchant and Bradford Levy},
5 booktitle={Forty-third International Conference on Machine Learning},
6 year={2026},
7 url={https://openreview.net/forum?id=JPbp2S9yTO}
8}
9@inproceedings{
10 merchant2026a,
11 title={A Fast and Effective Solution to the Problem of Look-ahead Bias in {LLM}s},
12 author={Humzah Merchant and Bradford Levy},
13 booktitle={NeurIPS 2025 Workshop: Generative AI in Finance},
14 year={2026},
15 url={https://openreview.net/forum?id=zYsLIPgM28}
16}
17@inproceedings{
18 merchant2026forecasting,
19 title={Forecasting With {LLM}s: Improved Generalization Through Feature Steering},
20 author={Humzah Merchant and Bradford Levy},
21 booktitle={Forecasting as a New Frontier of Intelligence},
22 year={2026},
23 url={https://openreview.net/forum?id=ppN6CmoNOk}
24}
