Continuous Amplitude Tokenization (CAT) - Toy Model
This repository contains the trained weights and vocabulary for a proof-of-concept toy model utilizing the novel Continuous Amplitude Tokenization (CAT) architecture.
Model Details
Traditional LLMs rely on discrete BPE tokenization, forcing them to treat intensity modifiers ("very", "extremely", "slightly") as independent, arbitrary tokens. The CAT architecture splits language into two streams:
- Concept ID (Discrete): e.g., "bad", "large".
- Amplitude (Continuous): e.g.,
1.0, 2.0, 5.0.
This toy model processes sentences like "The food was incredibly bad" as a single compressed token mapping: [Concept: "bad", Amplitude: 4.0], completely eliminating the need for combinatorial adjective tokens and physically shortening sequence lengths.
- Architecture: Custom Dual-Stream Causal Transformer (CAT)
- Key Components: Fourier-FiLM Embeddings, AP-RMSNorm, 4-way Bilinear Attention, Spatio-Amplitude RoPE.
- Parameters: ~4.6 Million
- Vocabulary Size: 15,000 words (Modifiers deleted and replaced by continuous scalars)
- Dataset: 15 Million tokens (Yelp Polarity Reviews)
- Training Iterations: 1,000 (Early-stage viability test)
Empirical Benefits
During benchmarking against a standard NanoGPT model on the exact same text, this CAT model achieved:
- 20% KV Cache Memory Savings (due to reduced sequence lengths).
- Vocab Compression (by removing standard modifiers from the dictionary).
- Zero-Shot Controllability (ability to manually inject
Amplitude: 10.0 to force intensity scaling during generation).
Usage
To use these weights, you must use the custom CAT Model code from the official GitHub repository.
- Clone the architecture:
1git clone https://github.com/Yash943/CAT-Concept-Amplitude-Tokenization-.git
2cd CAT-Concept-Amplitude-Tokenization-
3pip install -e .
- Load the model (PyTorch):
1import torch
2from cat.model import CATModel, CATConfig
3from huggingface_hub import hf_hub_download
4
5# Initialize the architecture
6config = CATConfig(
7 vocab_size=15000,
8 hidden_size=128,
9 num_hidden_layers=4,
10 num_attention_heads=4,
11 intermediate_size=512,
12 max_position_embeddings=64
13)
14model = CATModel(config)
15
16# Load the weights
17weights_path = hf_hub_download(repo_id="your-repo-id/here", filename="cat_toy_model.pt")
18model.load_state_dict(torch.load(weights_path))
19print("CAT Model loaded successfully!")
Intended Use
This is an experimental "toy" model designed strictly to prove the mathematical viability of Fourier-FiLM embeddings and AP-RMSNorm at reducing token fertility. It is not intended for production chat or generation tasks.
GitHub Repository
For full mathematical breakdowns, empirical verification scripts, and architectural source code, visit the
official GitHub repository.
