OpenLM-3B (custom OpenLMForCausalLM; load with trust_remote_code=True) with a July 2024 (202407) knowledge cutoff, fine-tuned for retrieval-grounded question answering with explicit reasoning. Given numbered context documents and a question, it emits a <think>...</think> trace that cites documents by bracket number, followed by a concise answer.

Use the default chat template: Human: {prompt}<|endoftext|>\nAssistant:.

Intended uses & limitations

Proof-of-concept retrieval-SFT (500 examples) for the Chrononauts time-bounded-LM project. It reliably produces the <think> format and grounds simple answers in the provided context, but accuracy on hard multi-hop questions with distractors is limited by the small training set, and there is no abstention training (it does not reliably say when the context lacks the answer).

Training and evaluation data

Fine-tuned on mattwang123/chrononauts-retrieval-sft-202407-500: 300 HotpotQA (distractor), 100 SQuAD v1, 100 2WikiMultiHopQA, with gpt-5 teacher reasoning traces; all sources predate the 202407 cutoff.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 1e-05
train_batch_size: 2
eval_batch_size: 8
seed: 42
distributed_type: multi-GPU
num_devices: 4
gradient_accumulation_steps: 4
total_train_batch_size: 32
total_eval_batch_size: 32
optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.1
num_epochs: 5.0

Training results

Framework versions

Transformers 4.57.1
Pytorch 2.6.0+cu124
Datasets 4.0.0
Tokenizers 0.22.2

mattwang123/openlm-3b-202407-retrieval-sft

openlm_3b_202407_retrieval_sft_full

Model description