Views
No views yet
eval_loss (best checkpoint loaded at end)1from transformers import AutoTokenizer, AutoModelForMaskedLM
2
3model_name = "TryDotAtwo/rubert-rulaw"
4tokenizer = AutoTokenizer.from_pretrained(model_name)
5model = AutoModelForMaskedLM.from_pretrained(model_name)sud-resh-benchmark legal texts using a masked language modeling setup. Tokens were randomly masked at varying probabilities (10–40%), and models predicted them using their pre-trained heads.Note: The ruBERT-ruLaw model was pre-trained on legal texts such as laws and statutes, but not specifically on judicial decisions. The evaluation reflects how well it generalizes to predicting masked tokens in Russian court rulings.
| MLM Probability | Metric | ruBERT-ruLaw | rubert-base-cased | legal-bert-base-uncased |
|---|---|---|---|---|
| 10% | Top-1 | 81.0% | 73.0% | 45.3% |
| 10% | Top-5 | 92.2% | 87.0% | 77.2% |
| 15% | Top-1 | 78.8% | 67.9% | 45.3% |
| 15% | Top-5 | 90.8% | 83.2% | 76.7% |
| 20% | Top-1 | 76.3% | 53.8% | 45.0% |
| 20% | Top-5 | 89.0% | 71.5% | 75.9% |
| 25% | Top-1 | 73.6% | 18.0% | 44.4% |
| 25% | Top-5 | 87.0% | 31.9% | 75.0% |
| 30% | Top-1 | 70.4% | 5.9% | 43.8% |
| 30% | Top-5 | 84.6% | 10.9% | 74.0% |
| 35% | Top-1 | 66.9% | 6.0% | 42.9% |
| 35% | Top-5 | 81.9% | 9.1% | 72.9% |
| 40% | Top-1 | 62.9% | 6.0% | 41.9% |
| 40% | Top-5 | 78.5% | 8.5% | 71.7% |