This is one of the models fine-tuned on text simplification for Simplify This project.
Model Details
Model Description
Fine-tuned sequence-to-sequence (encoder–decoder) Transformer for English text simplification.
Trained on the dataset eilamc14/wikilarge-clean (cleaned WikiLarge-style pairs).
The model was trained on Wikipedia and Simple English Wikipedia alignments (via WikiLarge).
As a result, it inherits the characteristics and limitations of this data:
Domain bias: Simplifications may reflect encyclopedic style; performance may degrade on informal, technical, or domain-specific text (e.g., medical/legal/news).
Content bias: Wikipedia content itself contains biases in coverage, cultural perspective, and phrasing. Simplified outputs may reflect or amplify these.
Simplification quality: The model may:
Over-simplify (drop important details)
Under-simplify (retain complex phrasing)
Produce ungrammatical or awkward rephrasings
Language limitation: Only suitable for English. Applying to other languages is unsupported.
Safety limitation: The model has not been aligned to avoid toxic, biased, or harmful content. If the input text contains such content, the output may reproduce or modify it without safeguards.
Recommendations
Evaluation required: Always evaluate the model in the target domain before deployment. Benchmark simplification quality (e.g., with SARI, FKGL, BERTScore, LENS, human evaluation).
Human oversight: Use human-in-the-loop review for applications where meaning preservation is critical (education, accessibility tools, etc.).
Attribution: Preserve source attribution where required (Wikipedia → CC BY-SA).
Not for high-stakes use: Avoid legal, medical, or safety-critical applications without extensive validation and domain adaptation.
How to Get Started with the Model
Load the model and tokenizer directly from the Hugging Face Hub:
Objective: Standard sequence-to-sequence cross-entropy loss
Training type: Full fine-tuning of all parameters (no LoRA/PEFT used)
Batching: Dynamic padding with Hugging Face Trainer / PyTorch DataLoader
Evaluation: Monitored on the validation split with metrics (SARI and identical_ratio)
Stopping criteria: Early stopping CallBack based on validation performance
Preprocessing
The dataset was preprocessed by prefixing each source sentence with "Simplify: " and tokenizing both the source (inputs) and target (labels).
Memory & Checkpointing
To reduce VRAM during training, gradient checkpointing was enabled and the KV cache was disabled:
python
1model.config.use_cache =False# required when using gradient checkpointing2model.gradient_checkpointing_enable()# saves memory at the cost of extra compute
Notes
Disabling use_cache avoids warnings/conflicts with gradient checkpointing and reduces memory usage in the forward pass.
Gradient checkpointing trades GPU memory ↓ for training speed ↓ (extra recomputation).
For inference/evaluation, re-enable the cache for faster generation:
model.config.use_cache = True
Training Hyperparameters
The models were trained with Hugging Face Seq2SeqTrainingArguments.
Hyperparameters varied slightly across models and runs to optimize, and full logs (batch size, steps, exact LR schedule) were not preserved.
Below are the typical defaults used:
Epochs: 5
Evaluation strategy: every 300 steps
Save strategy: every 300 steps (keep best model, eval_loss as criterion)
