Views
No views yet
1from transformers import pipeline
2
3oracle = pipeline('token-classification', model='dicta-il/dictabert-char-spacefix')
4
5text = 'בשנת 1948 השליםאפרים קישון אתלימודיובפיסולמתכת ובתולדותהאמנות והחל לפרסםמאמרים הומוריסטיים'
6
7raw_output = oracle(text)
8
9# Classifier returns LABEL_1 if there should be a space before the character
10text_output = ''.join((' ' if o['entity'] == 'LABEL_1' else '') + o['word'] for o in raw_output)
11print(text_output)בשנת 1948 השלים אפרים קישון את לימודיו בפיסול מתכת ובתולדות האמנות והחל לפרסם מאמרים הומוריסטייםRestoring Missing Spaces in Scraped Hebrew Social Media1@inproceedings{shmidman-shmidman-2025-restoring,
2 title = "Restoring Missing Spaces in Scraped {H}ebrew Social Media",
3 author = "Shmidman, Avi and
4 Shmidman, Shaltiel",
5 editor = "Bak, JinYeong and
6 Goot, Rob van der and
7 Jang, Hyeju and
8 Buaphet, Weerayut and
9 Ramponi, Alan and
10 Xu, Wei and
11 Ritter, Alan",
12 booktitle = "Proceedings of the Tenth Workshop on Noisy and User-generated Text",
13 month = may,
14 year = "2025",
15 address = "Albuquerque, New Mexico, USA",
16 publisher = "Association for Computational Linguistics",
17 url = "https://aclanthology.org/2025.wnut-1.3/",
18 pages = "16--25",
19 ISBN = "979-8-89176-232-9",
20 abstract = "A formidable challenge regarding scraped corpora of social media is the omission of whitespaces, causing pairs of words to be conflated together as one. In order for the text to be properly parsed and analyzed, these missing spaces must be detected and restored. However, it is particularly hard to restore whitespace in languages such as Hebrew which are written without vowels, because a conflated form can often be split into multiple different pairs of valid words. Thus, a simple dictionary lookup is not feasible. In this paper, we present and evaluate a series of neural approaches to restore missing spaces in scraped Hebrew social media. Our best all-around method involved pretraining a new character-based BERT model for Hebrew, and then fine-tuning a space restoration model on top of this new BERT model. This method is blazing fast, high-performing, and open for unrestricted use, providing a practical solution to process huge Hebrew social media corpora with a consumer-grade GPU. We release the new BERT model and the fine-tuned space-restoration model to the NLP community."
21}
