Beta
Explore
Marketplace
Neural Labs
Chat
Wallet
Docs
gpt-hilberg-5M – AI Model by AdrianRasoOnHF | AlphaNeural AI
You can deploy this model and start earning money today!
AdrianRasoOnHF
/
gpt-hilberg-5M
like
0
pytorch
language-model
gpt
hilberg
information-theory
wikipedia
entropy
mit
us
Views
No views yet
Model card
Files and Versions
Community
API
Deploy
GPT-Hilberg-5M
The present is a 5M-parameter GPT autoregressive language model trained on the July 20, 2025 English Wikipedia dump for experiments on entropy scaling and Hilberg conjecture. For more information on this, you can check
here
. Dataset available
here
.
This model is part of the following suite:
GPT-Hilberg-1M (128 ctx)
GPT-Hilberg-5M (512 ctx)
GPT-Hilberg-10M (1024 ctx)
Architecture details
Parameters: ~5M
Layers: 10
Model dimension: 128
Attention heads: 8
FF dimension: 512
Context length: 512
Vocabulary size: 16,000
Training details
Dataset: English Wikipedia (2025-07 dump)
Tokenizer: SentencePiece BPE
Purpose: Autogregression
Optimizer: AdamW
