oeg/

RoBERTaSense-FACIL

Model Card for RoBERTaSense-FACIL

🚀 How to Use

Model Description

• Developed by: Ontology Engineering Group (UPM) / Authors: Isam Diab Lozano and Mari Carmen Suárez-Figueroa
• Funded by: "Ayudas para la contratación de personal investigador predoctoral en formación para el año 2022" (Reference: PIPF-2022/COM-25762) by Comunidad Autónoma de Madrid (Spain)
• Model type: Encoder-only Transformer (RoBERTa) with a classification head
• Language: Spanish (es)
• License: Apache-2.0
• Finetuned from model: PlanTL-GOB-ES/roberta-base-bne (deprecated; see notice above)

Uses

Direct Use

• Automatic scoring of meaning preservation for Spanish Easy-to-Read adaptations.
• As a signal in content quality checks for accessibility pipelines.

Out-of-Scope Use

• Clinical, legal, or other high-stakes decisions without human expert oversight.
• Non-Spanish or out-of-domain texts without prior adaptation or re-training.

Bias, Risks, and Limitations

• Domain limitation: trained for Spanish E2R; performance may degrade on other genres/domains.
• Binary labels: compress nuanced cases; borderline adaptations may require human review.
• Synthetic negatives: not all human errors are covered by synthetic negative strategies.
• Base deprecation: the upstream base model is deprecated; security/robustness updates won’t be inherited.

Recommendations

• Calibrate probabilities (e.g., temperature scaling) and expose confidence scores.
• Use threshold tuning (e.g., Youden’s J) to trade precision/recall for your setting.
• Keep a human-in-the-loop for critical use cases and periodic error audits.

How to Get Started with the Model

Training Details

Training Data

• Source: Spanish pairs (original - adapted) curated/validated by experts.
• Columns: text1 (original), text2 (adaptation), Label (0/1), neg_type.
• Labels: 1 = PRESERVES_MEANING, 0 = DOES_NOT_PRESERVE.
• Negative types used in training data construction: shuffle, dropout, mismatch (derangement), paraphrase_distortion, nli_contradiction.
• Split: 80/20, stratified by Label (random_state=42).

Training Procedure

Preprocessing

• Pair tokenization with truncation at 512 tokens:

Training Hyperparameters

• Training regime: fp16 mixed precision (if supported; otherwise fp32)
• Arguments:
  • num_train_epochs=5
  • per_device_train_batch_size=32
  • per_device_eval_batch_size=16
  • learning_rate=2e-5
  • weight_decay=0.01
  • warmup_ratio=0.1
  • evaluation_strategy="epoch", save_strategy="epoch"
  • load_best_model_at_end=True, metric_for_best_model="f1"
• Optimizer: AdamW
• Loss: CrossEntropy (2 logits)

Evaluation

Testing Data, Factors & Metrics

Testing Data

• Held-out 20% stratified split of the curated E2R pairs.

Factors

• Report per-negative-type breakdown (e.g., performance on mismatch, paraphrase_distortion, etc.).

Metrics

• Accuracy, F1, ROC-AUC.

Results

• Accuracy: 0.81
• F1: 0.84
• ROC-AUC: 0.83
• Threshold tuned via Youden’s J for operating point selection.

Technical Specifications

Model Architecture and Objective

• Encoder-only RoBERTa with a classification head (Linear(hidden → 2)).
• Objective: supervised cross-entropy on binary label.

Citation