If the word is split up into multiple wordpieces it doesn't cause a problem, we still predict the lexeme with a high accuracy.
If the lexeme of a given token doesn't appear in the vocabulary, the model will attempt to predict a special token [BLANK]. In that case, the word is usually a name of a person or a city, and the lexeme is probably the word after removing prefixes which can be done with the dictabert-seg tool.
For verbs the lexeme is the 3rd person past singular form.

This method is purely neural-based, so in rare instances the predicted lexeme may not be lexically related to the input, but rather a synonym selected from the same semantic space. To handle those edge cases one can implement a filter on top of the prediction to look at the top K matches and choose using a specific set of measures, such as edit distance, to choose the prediction that can more reasonably form a lexeme for the input word.

Sample usage:

python

from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('dicta-il/dictabert-lex')
model = AutoModel.from_pretrained('dicta-il/dictabert-lex', trust_remote_code=True)

model.eval()

sentence = 'בשנת 1948 השלים אפרים קישון את לימודיו בפיסול מתכת ובתולדות האמנות והחל לפרסם מאמרים הומוריסטיים'
print(model.predict([sentence], tokenizer))

Output:

json

[
  [
    [
      "בשנת",
      "שנה"
    ],
    [
      "1948",
      "1948"
    ],
    [
      "השלים",
      "השלים"
    ],
    [
      "אפרים",
      "אפרים"
    ],
    [
      "קישון",
      "קישון"
    ],
    [
      "את",
      "את"
    ],
    [
      "לימודיו",
      "לימוד"
    ],
    [
      "בפיסול",
      "פיסול"
    ],
    [
      "מתכת",
      "מתכת"
    ],
    [
      "ובתולדות",
      "תולדה"
    ],
    [
      "האמנות",
      "אומנות"
    ],
    [
      "והחל",
      "החל"
    ],
    [
      "לפרסם",
      "פרסם"
    ],
    [
      "מאמרים",
      "מאמר"
    ],
    [
      "הומוריסטיים",
      "הומוריסטי"
    ]
  ]
]

Citation

If you use DictaBERT-lex in your research, please cite MRL Parsing without Tears: The Case of Hebrew

BibTeX:

bibtex

@misc{shmidman2024mrl,
      title={MRL Parsing Without Tears: The Case of Hebrew}, 
      author={Shaltiel Shmidman and Avi Shmidman and Moshe Koppel and Reut Tsarfaty},
      year={2024},
      eprint={2403.06970},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

License

Shield:

CC BY 4.0

This work is licensed under a Creative Commons Attribution 4.0 International License.