Views
No views yet
lb/ltz)transformers>=4.48.0.1from transformers import AutoModelForMaskedLM, AutoTokenizer
2import torch
3
4tokenizer = AutoTokenizer.from_pretrained("instilux/luni-mini")
5model = AutoModelForMaskedLM.from_pretrained("instilux/luni-mini")
6
7inputs = tokenizer("Wéi spéit [MASK] et?", return_tensors="pt")
8mask_pos = (inputs["input_ids"] == tokenizer.mask_token_id).nonzero(as_tuple=True)[1]
9
10with torch.no_grad():
11 outputs = model(**inputs)
12
13top_tokens = outputs.logits[0, mask_pos].topk(5)
14for token_id, score in zip(top_tokens.indices[0], top_tokens.values[0]):
15 token = tokenizer.decode(token_id)
16 print(f"{token:15s} {score:.3f}")GPTNeoXTokenizerFast) with BERT-style special tokens ([CLS], [SEP], [MASK], [PAD]). A [CLS] token is prepended automatically (add_bos_token: true).