-BERTopic_Arab_news

The core of this project is BERTopic, which is used to perform topic modeling on the processed text. The following steps are performed:

- Topic Modeling: BERTopic is trained on the cleaned dataset to identify topics in the articles.

- Fine-tuning with KeyBERT: We use KeyBERT-inspired representations to improve the clarity and interpretability of the topics.

- Topic Extraction: The most frequent topics are extracted, and each document is assigned a topic.

- Topic Updates: The model can be fine-tuned by updating topics with n-grams for more domain-specific phrases.

Usage

Topic overview

• Number of topics: 92
• Number of training documents: 30721

Dataset

Dataset Details:

Name: saudinewsnet

Description: This dataset contains news articles in Arabic from various domains, primarily focused on Saudi news.

Training hyperparameters

• calculate_probabilities: True
• language: None
• low_memory: False
• min_topic_size: 10
• n_gram_range: (1, 1)
• nr_topics: None
• seed_topic_list: None
• top_n_words: 10
• verbose: True
• zeroshot_min_similarity: 0.7
• zeroshot_topic_list: None

Framework versions

• Numpy: 1.26.4
• HDBSCAN: 0.8.40
• UMAP: 0.5.7
• Pandas: 2.2.2
• Scikit-Learn: 1.2.2
• Sentence-transformers: 3.3.1
• Transformers: 4.47.0
• Numba: 0.60.0
• Plotly: 5.24.1
• Python: 3.10.12

Visualization: Displays topic distribution and document-level information