Beta
Explore
Marketplace
Neural Labs
Chat
Wallet
Docs
ModernBERT2gpt2-700m-v0.1 – AI Model by pszemraj | AlphaNeural AI
You can deploy this model and start earning money today!
pszemraj
/
ModernBERT2gpt2-700m-v0.1
like
0
transformers
safetensors
encoder-decoder
text2text-generation
en
pszemraj/t2t-re_pretrain-small
answerdotai/ModernBERT-large
finetune
apache-2.0
endpoints_compatible
us
Views
No views yet
Model card
Files and Versions
Community
API
Deploy
ModernBERT2gpt2-700m baseline
EncoderDecoder created from modernBERT-large and random-init
gpt2
trained on the pszemraj/t2t-re_pretrain-small dataset for one epoch as a "baseline".
input context length 2048
output context length 512
single tokenizer, slightly modified from modernBERT
Logs and training script can be found
on wandb
It achieves the following results on the evaluation set:
Loss: 2.2113
Rouge1: 48.6654
Rouge2: 31.8667
Rougel: 44.9897
Rougelsum: 45.4126
Gen Len: 30.24
Num Input Tokens Seen: 524625736
