This checkpoint contains the oracle-labeled Token Importance Scoring (TIS) components trained with ground-truth importance labels from full-context model outputs.
Model Description
This is the Stage 1 Oracle checkpoint that demonstrates the theoretical performance ceiling for TIS. It uses oracle labels (ground truth from full-context runs) for training, providing a reference baseline for learned importance scoring.
Key Features:
✅ Oracle training with ground-truth labels
✅ 100% NIAH accuracy at all cache budgets (with oracle labels)
✅ Theoretical upper bound for TIS performance
✅ Useful for ablation studies and understanding TIS limits
Performance
Oracle Performance (with ground-truth labels):
NIAH @ all budgets: 100% (by definition)
Provides upper bound for learned methods
Note: This checkpoint is trained on oracle labels, so it represents the best possible performance achievable if importance scores were perfectly predicted.
Training Details
Base Model:mistralai/Mistral-7B-v0.3 Training Data: NarrativeQA with oracle importance labels Training Stage: 1 (Supervised oracle training)
Hyperparameters:
Epochs: 2
Batch size: 4 (gradient accumulation: 8)
Learning rate: 1e-4
Precision: BFloat16
LoRA: r=16, alpha=32
Max sequence length: 2,048 tokens
Weight alignment: 0.1
Weight robustness: 0.0
Training Objective:
Loss = LM_loss + λ_align * alignment_loss
Where oracle labels are derived from full-context forward passes.
