Beta
Explore
Marketplace
Neural Labs
Chat
Wallet
Docs
hubertusz-medium-wiki-seq128 – AI Model by SzegedAI | AlphaNeural AI
You can deploy this model and start earning money today!
SzegedAI
/
hubertusz-medium-wiki-seq128
like
0
transformers
pytorch
tf
bert
pretraining
generated_from_keras_callback
hubert
hu
wikipedia
apache-2.0
endpoints_compatible
us
Views
No views yet
Model card
Files and Versions
Community
API
Deploy
hubert-medium-wiki-seq128
Fully trained model with the second phase of training is available here:
SzegedAI/hubert-medium-wiki
This model was trained from scratch on the Wikipedia subset of Hungarian Webcorpus 2.0 with MLM and SOP tasks.
Pre-Training Parameters:
Training steps: 500.000
Sequence length: 128 (the model is capable for 512)
Batch size: 1024
Framework versions
Transformers 4.21.3
TensorFlow 2.10.0
Datasets 2.4.0
Tokenizers 0.12.1
Acknowledgement
Artificial Intelligence - National Laboratory - Hungary
