You can deploy this model and start earning money today!

filiz-yalcin/
turkish-books-bpe-tokenizer

Name: turkish-books-bpe-tokenizer
Availability: InStock

Views

No views yet

Model Card for Turkish Books BPE Tokenizer

This model is a BPE tokenizer obtained from the concatenated version of Turkish Book Data Set.

Model Details

Model Description

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

Developed by: Filiz Yalçin
Model type: BPE Tokenizer
Language(s) (NLP): Turkish (tr)
License: gpl-3.0

Uses

This tokenizer can be used to tokenized Turkish texts.

How to Get Started with the Model

Use the code below to get started with the model.

python

from transformers import AutoTokenizer

# 1. Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained("filiz-yalcin/turkish-books-bpe-tokenizer")

# 2. Encode text to IDs
encoded = tokenizer("Yalnızlık paylaşılmaz. Paylaşılsa yalnızlık olmaz.")
print(encoded["input_ids"])

Training Details

Training Data

The corpus was obtained using Turkish Book Data Set. The 'explanation' column's values were concatenated.