Beta
Explore
Marketplace
Neural Labs
Chat
Wallet
Docs
bytebpe-tokenizer-32k-mlm-uncased – AI Model by pszemraj | AlphaNeural AI
You can deploy this model and start earning money today!
pszemraj
/
bytebpe-tokenizer-32k-mlm-uncased
like
0
transformers
en
pints-ai/Expository-Prose-V1
apache-2.0
endpoints_compatible
us
Views
No views yet
Model card
Files and Versions
Community
API
Deploy
bpe tokenizer w byte-fallback: 32k vocab, uncased
uncased BPE tokenizer for encoders/MLM objective with byte-pair fallback:
Trained on
pints-ai/Expository-Prose-V1
; this tokenizer is primarily for English and code.
this tokenizer is
uncased
: "HELLO WORLD" is
the same
as "hello world"
model_max_length
is set to 1e9 to not cause hidden issues.
Set
tokenizer.model_max_length
to your model's max position embeddings
when training.
