Beta
Explore
Marketplace
Neural Labs
Chat
Wallet
Docs
tFINE-850m-24x24-1024ctx – AI Model by pszemraj | AlphaNeural AI
You can deploy this model and start earning money today!
pszemraj
/
tFINE-850m-24x24-1024ctx
like
0
transformers
safetensors
t5
text2text-generation
en
HuggingFaceTB/smollm-corpus
apache-2.0
text-generation-inference
endpoints_compatible
us
Views
No views yet
Model card
Files and Versions
Community
API
Deploy
tFINE-850m-24x24-1024ctx
Pretrained T5 model with
nanoT5
:
~850m parameters, 24 layers in encoder, 24 layers in decoder
sentencepiece tokenizer with 48k vocab & byte-pair fallback
handles whitespaces etc correctly (
unlike original T5 tokenizer
)
1024 ctx during pretrain
relative_attention_num_buckets
increased to 48 from 32 for context length upscaling
Experiment logs
Training consisted of two phases:
TODO
TODO
