Beta
Explore
Marketplace
Neural Labs
Chat
Wallet
Docs
performer-checkpoints – AI Model by antoinechss | AlphaNeural AI
You can deploy this model and start earning money today!
antoinechss
/
performer-checkpoints
like
0
transformers
performer
linear-attention
favor+
knowledge-distillation
research
text-generation
en
mindchain/wikitext2
allenai/c4
TinyLlama/TinyLlama-1.1B-Chat-v1.0
finetune
apache-2.0
endpoints_compatible
us
Views
No views yet
Model card
Files and Versions
Community
API
Deploy
Model description
: TinyLlama 1.1B with K/32 softmax attention heads replaced by FAVOR+ linear attention, fine-tuned via knowledge distillation
Intended use
: Research — evaluating quality/speed/approximation trade-offs of linear attention substitution
How to load
: code snippet showing how to reconstruct MixedPerformerAttention and load the checkpoint
Training details
: WikiText-103, 20k samples, SEQ_LEN=256, distillation loss, 4-phase curriculum
Results table
: same as the README (ppl per phase)
Limitations
: Phase 4 (32/32 heads) collapsed — not suitable for inference. Phase 2 is the recommended checkpoint.
