The easiest way to use this model for inference is with the Hugging Face pipeline API, setting the task to "ner" and using the "simple" aggregation strategy to automatically merge subword tokens back into recognizable entities.

1. Using the Pipeline (Recommended)

This method simplifies the process and handles token aggregation for you.

python

from transformers import pipeline

# Replace with your model ID
model_id = "shroukAdel/xlm-roberta-base-finetuned-panx-all"

# Initialize the NER pipeline
# aggregation_strategy="simple" ensures subwords are combined into single entities
ner_pipeline = pipeline(
    "ner",
    model=model_id,
    aggregation_strategy="simple"
)

# Example 1: German
text_de = "Jeff Dean ist ein Informatiker bei Google in Kalifornien"
print(ner_pipeline(text_de))
# Output: [{'entity_group': 'PER', 'score': 0.997, 'word': 'Jeff Dean', ...}, {'entity_group': 'ORG', 'score': 0.996, 'word': 'Google', ...}, {'entity_group': 'LOC', 'score': 0.998, 'word': 'Kalifornien', ...}]

# Example 2: English
text_en = "My name is Sarah and I live in London"
print(ner_pipeline(text_en))
# Output: [{'entity_group': 'PER', 'score': 0.996, 'word': 'Sarah', ...}, {'entity_group': 'LOC', 'score': 0.998, 'word': 'London', ...}]

# Example 3: French
text_fr = "Marie Curie était une physicienne française"
print(ner_pipeline(text_fr))
# Output: [{'entity_group': 'PER', 'score': 0.995, 'word': 'Marie Curie', ...}]

2. Manual Loading (For Custom Tasks)

If you need lower-level access to the model or are integrating it into a custom training loop, you can load the model and tokenizer manually.

python

from transformers import AutoTokenizer, AutoModelForTokenClassification

model_id = "shroukAdel/xlm-roberta-base-finetuned-panx-all"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForTokenClassification.from_pretrained(model_id)

# Now you can use the tokenizer and model for custom inference or further fine-tuning

language:

['Enlish',"French","German","Italian"]

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-05
train_batch_size: 24
eval_batch_size: 24
seed: 42
optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss	F1
0.2902	1.0	835	0.1952	0.8212
0.157	2.0	1670	0.1852	0.8387
0.1028	3.0	2505	0.1809	0.8529

Framework versions

Transformers 4.57.3
Pytorch 2.9.0+cu126
Datasets 4.0.0
Tokenizers 0.22.1