The original repository ships only a SentencePiece tokenizer (sentencepiece.bpe.model) and no tokenizer.json. On transformers v5, loading the tokenizer forces an on-the-fly SentencePiece → fast conversion, which currently fails for XLM-RoBERTa and raises:

ValueError: Error parsing line b'\x0e' in .../sentencepiece.bpe.model

(or, on some late 4.x releases, AttributeError: 'NoneType' object has no attribute 'endswith' from the tekken.json conversion branch).

This copy includes a tokenizer.json generated with transformers 4.44.2, so v5 loads the tokenizer directly and never runs the broken conversion. The tokenizer is verified to produce identical token ids to the original slow tokenizer.

The model weights are byte-for-byte the same as the original.

Usage

python

from transformers import pipeline

clf = pipeline(
    task="sentiment-analysis",
    model="NetworkIsLife/twitter-xlm-roberta-base-sentiment-fast",
    tokenizer="NetworkIsLife/twitter-xlm-roberta-base-sentiment-fast",
)
clf("T'estimo!")
# [{'label': 'Positive', 'score': 0.66...}]

Or load the pieces directly:

python

from transformers import AutoTokenizer, AutoModelForSequenceClassification

repo = "NetworkIsLife/twitter-xlm-roberta-base-sentiment-fast"
tok = AutoTokenizer.from_pretrained(repo)
mdl = AutoModelForSequenceClassification.from_pretrained(repo)

Labels

id	label
0	Negative
1	Neutral
2	Positive

Credit and license

All credit for the model goes to the original authors at Cardiff NLP. This repo only adds a tokenizer file for compatibility. Please cite the original work:

bibtex

@inproceedings{barbieri-etal-2022-xlm,
    title     = "{XLM}-{T}: Multilingual Language Models in {T}witter for Sentiment Analysis and Beyond",
    author    = "Barbieri, Francesco and Espinosa Anke, Luis and Camacho-Collados, Jose",
    booktitle = "Proceedings of the Thirteenth Language Resources and Evaluation Conference",
    year      = "2022",
    address   = "Marseille, France",
    publisher = "European Language Resources Association",
    url       = "https://aclanthology.org/2022.lrec-1.27",
    pages     = "258--266",
}

License: this copy inherits whatever license applies to the original cardiffnlp/twitter-xlm-roberta-base-sentiment. Set the license: field in the frontmatter above to match it before relying on this repo.