tinychat-271M-base – AI Model by kali-v | AlphaNeural AI

You can deploy this model and start earning money today!

kali-v/
tinychat-271M-base

like

0

text-generation

HuggingFaceFW/fineweb

HuggingFaceFW/fineweb-edu

wikimedia/wikipedia

Views

No views yet

tinychat 271M base

An autoregressive Transformer language model:

Embedding size: 1440
Layers: 12
Attention heads: 16 (8 kv heads)
Parameters: 271M
Context length: 512
Vocabulary size: 32,768

Trained for ~50 hours on a single RTX-3090, processing ~5B tokens from:

5M FineWeb articles
5M FineWeb-Edu articles
Simple English Wikipedia

More details here

Performance (0-shot)

	tinychat	gpt2	gpt2-medium
swag	47.3%	48.9%	56.3%
hellaswag	30.7%	29.0%	37.0%
openbookqa	32.8%	28.0%	30.6%