Automatic Evaluation Models for Textual Data Quality (NL & CL)

• A unified model that handles both NL and CL jointly: EuroBERT-210m-Quality
• A dual-model approach that treats NL and CL separately:
  • EuroBERT-210m-Quality-NL for natural language
  • EuroBERT-210m-Quality-CL for code language.

Classification Categories:

• Harmful: Harmful data, potentially incorrect or dangerous.
• Low: Low-quality data with major issues.
• Medium: Medium quality, improvable but acceptable.
• High: Good to very good quality data, ready for use without reservation.

Supported Languages:

• Natural Language: French 🇫🇷, English 🇬🇧, Spanish 🇪🇸
• Code Language: Python 🐍, Java ☕, JavaScript 📜, C/C++ ⚙️

Performance

• f1-score: Unified Model (NL + CL)

• f1-score: Separate Models

Key Performance Metrics:

• Unified Model (NL + CL):
  • Overall accuracy: ~73%
  • High reliability on harmful data (f1-score: 0.86)
• Separate Models:
  • Natural Language (NL): ~82% accuracy
    • Excellent performance on harmful data (f1-score: 0.93)
  • Code Language (CL): ~63% accuracy
    • Good detection of harmful data (f1-score: 0.72)

Training Dataset:

• Public dataset available: TempestTeam/dataset-quality

Common Use Cases:

• Automatic validation of text corpora before integration into NLP or code generation pipelines.
• Quality assessment of community contributions (forums, Stack Overflow, GitHub).
• Automated pre-processing to enhance NLP or code generation system performance.

Recommendations:

• For specialized contexts, use the separate NL and CL models for optimal results.
• The unified model is suitable for quick assessments when the data context is unknown or mixed.

Citation