Model Card — albert. (Albert-MoE-13)

Model Overview

Intended Use

• Research into ternary and low-precision neural network architectures
• Benchmarking inference performance on CPU and edge GPU hardware
• Academic study of Mixture-of-Experts routing dynamics
• Demonstration platform for the SPRIND AI funding initiative (Germany)

• Production deployment as a general-purpose assistant without further fine-tuning and safety evaluation
• Safety-critical applications (medical, legal, financial decisions)
• Any use requiring factual accuracy guarantees
• Deployment to users without appropriate transparency disclosure

Training Data

Evaluation

• At current training depth (~ep6670, 31L dual-stream), output quality is pre-fluency: the model produces partially coherent text in familiar domains but lacks consistent grammatical structure across longer sequences.
• Context window of 128 tokens is shorter than contemporary LLMs; cannot maintain coherence over longer passages.
• Ternary quantization trades weight precision for size — at this scale, some representational capacity is lost relative to F32 equivalents.
• No instruction-following fine-tuning has been applied.
• No RLHF, Constitutional AI, or safety fine-tuning of any kind.
• Bias evaluation is pending (see below).

• STE gradient approximation at scale: Straight-Through Estimation is the training mechanism for ternary weights. Its stability and convergence properties are well-characterised at the current scale (~228.5M params). Whether STE remains stable through training runs at 500M–1B+ parameters is an open empirical question — no published work has demonstrated ternary STE convergence at frontier scale.
• @sparseskip speedup baseline: The 83 tok/s inference figure is measured against albert.'s own F32-weight dense equivalent on the same hardware. It is not a direct comparison with INT4-quantized industry inference (TensorRT-LLM, llama.cpp Q4). The relevant claim is that ternary weights eliminate a quantization step entirely — the speedup over post-hoc INT4 quantization of a larger model is a separate, untested question.
• Net2net surgery stability at scale: All five documented layer-addition surgeries were performed on a model in the 13M–58M parameter range. Whether the Fibonacci-gated surgery protocol remains stable when applied to models at 200M+ parameters has not been tested. The plateau-gate's withhold behavior is now validated across six independent events (ep791 non-firing + alternating descent phase ep2109–ep2120 — see below); the question of whether these properties hold at 200M+ scale remains open.

Bias and Fairness

• Language imbalance: English-dominant corpus; non-English outputs will be lower quality.
• Temporal bias: Training data has a knowledge cutoff; the model has no awareness of events after its corpus snapshot dates.
• Domain gaps: Limited coverage of non-Western cultural contexts, legal jurisdictions outside EU/US/DE, and specialized professional domains.

Human Oversight

1. Training dashboard: Real-time monitoring of loss curves, expert routing, gradient norms, WALD dead-zone events, and anomaly events by the RFI-IRFOS team.
2. Surgery governor: Architectural growth (layer addition via net2net) is fully autonomous — the EvolutionManager fires on a Fibonacci-gated plateau detector with no human intervention required. 19 depth surgeries (12L→31L) + 1 cord surgery (single→dual-stream bifurcation) have been executed autonomously to date. The cord surgery (ep4202, 2026-05-27) is the first documented autonomous single-to-dual-stream bifurcation in a live ternary MoE.
3. SPORE federated training (live): Collaborators contribute CPU-trained checkpoints as weight spores via the albert-spores private repository. The SporeManager blends accepted spores at α=0.08 each epoch boundary with fitness (loss gate) and architecture guards. Colony is active as of 2026-05-16 with external contributors. Spores are stored via Git LFS; each contributor runs albert-train locally and submits via albert-spore.
4. Checkpoint promotion: No trained checkpoint is deployed to any external service without explicit human review and approval by the lead architect.
5. Rollback capability: All checkpoints and best-loss weights are preserved on persistent storage. Any version can be reverted.

EU AI Act Compliance Notes

Team

Legal entity