Templar-I: Permissionless Distributed Training

A 1.2B-parameter causal language model trained with Gauntlet, an incentive system that rewards permissionless contributors for useful pseudo-gradients on the Bittensor network. [Paper]

Overview

• Setting: Fully open, permissionless, internet-scale training; no control over who registers or their hardware.
• Mechanism: Two-stage peer filtering (uptime/reliability/sync) + scoring per-peer gradient quality.
• Run: 20K communication rounds; FineWebEdu data; top 15 peers aggregated per round with up to 250 registered peers.
• Result: On a per-iteration basis, convergence outpaced a centralized AdamW baseline; downstream metrics are competitive.

Gauntlet

• Stage 1: Filters peers by uptime, reliability, and synchronization.
• Stage 2: Estimates loss before/after applying each peer’s pseudo-gradients to evaluate its contribution.
• Ratings: Uses OpenSkill to track competitiveness across time.
• Aggregation: In each round, aggregate updates from the top-scoring G=15 peers.

Training setup

• Data: FineWeb-edu [11].
• Rounds: 20,000 communication rounds (evaluation windows matched rounds).
• Tokens: 100-200B
• Baseline for comparison: Centralized AdamW trained for 120B tokens.

Quickstart

Results

Downstream Benchmarks (zero-shot)

Per-Iteration Loss

Citation

@article{lidin2025incentivizing,
  title={Incentivizing Permissionless Distributed Learning of LLMs},
  author={Lidin, Joel and Sarfi, Amir and Pappas, Evangelos and Dare, Samuel and Belilovsky, Eugene and Steeves, Jacob},
  journal={arXiv preprint arXiv:2505.21684},
  year={2025}
}