Modern Large Language Models (LLMs) are constrained by a fundamental bottleneck: they generate text one token at a time. CALM (Continuous Autoregressive Language Models) confronts this challenge by introducing a paradigm shift in language modeling. Instead of predicting one discrete token at a time, CALM learns to predict a single continuous vector that represents an entire chunk of K tokens.
-
🚀 Ultra-Efficient by Design: Dramatically improves training and inference efficiency by reducing the number of autoregressive steps by a factor of K.
-
💡 A New Scaling Axis: Introduces a new scaling dimension for LLMs—semantic bandwidth (K). Instead of just scaling parameters and data, you can now scale the amount of information processed in a single step.
-
🛠️ A Comprehensive Likelihood-Free Toolkit: Operating in a continuous domain requires new tools. This repository provides the full suite of algorithms that make CALM possible:
- A Robust Autoencoder to learn high-fidelity continuous representations of token chunks.
- Energy-Based Training, a principled and likelihood-free method for generative modeling.
- BrierLM, a new metric for calibrated, likelihood-free evaluation of language models.
- Temperature Sampling for controlled, high-quality text generation using only a black-box sampler.
See our
GitHub README, where we provide scripts for training and evaluation.
