Views
No views yet
jhu-clsp/mmBERT-base (a ModernBERT-architecture multilingual encoder) as part of the Gaperon data-curation pipeline. Given a passage of text, it predicts a low / medium / high rating along six quality dimensions, intended for scoring and filtering web/corpus text before it's used for further model training.mean_macro_f1) from training run mmbert_hindi_v1, saved at epoch 4.jhu-clsp/mmBERT-base (ModernBERT architecture, 22 layers, hidden size 768, 8192 max sequence length, mean pooling over token representations).Linear(768 → 3)), one per label dimension, applied to the mean-pooled backbone output. Each head is a 3-way classifier over {low: 0, medium: 1, high: 2}.clarity, coherence, depth, grammar, usefulness, overall.model.safetensors / config.json (standard HF format, loadable with AutoModel). The six classification heads are not part of the HF backbone class and are stored separately in heads.pt (see usage below).data_gaperon/tagged_v2/hindi), labeled with low/medium/high ratings for each of the six dimensions.| Hyperparameter | Value |
|---|---|
| max_length | 512 |
| batch_size | 32 |
| learning_rate | 2e-5 |
| weight_decay | 0.01 |
| dropout | 0.1 |
| frozen layers | 0 |
| warmup_ratio | 0.1 |
| seed | 42 |
| Dimension | Val F1 | Val Acc | Test F1 | Test Acc |
|---|---|---|---|---|
| clarity | 0.5586 | 0.5977 | 0.5453 | 0.5733 |
| coherence | 0.5941 | 0.6239 | 0.5930 | 0.6045 |
| depth | 0.5409 | 0.8074 | 0.5483 | 0.8062 |
| grammar | 0.5582 | 0.6353 | 0.5399 | 0.6122 |
| usefulness | 0.5859 | 0.6330 | 0.5743 | 0.6362 |
| overall | 0.5553 | 0.6062 | 0.5549 | 0.6165 |
| mean macro-F1 | 0.5655 | – | 0.5593 | – |
1import torch
2from huggingface_hub import hf_hub_download
3from transformers import AutoModel, AutoTokenizer
4
5repo_id = "<your-username>/gaperon-mmbert-hindi"
6
7tokenizer = AutoTokenizer.from_pretrained(repo_id)
8backbone = AutoModel.from_pretrained(repo_id)
9
10heads_path = hf_hub_download(repo_id, filename="heads.pt")
11ckpt = torch.load(heads_path, map_location="cpu")
12label_dims = ckpt["label_dims"] # ["clarity", "coherence", "depth", "grammar", "usefulness", "overall"]
13label2id = ckpt["label2id"] # per-dimension {"low": 0, "medium": 1, "high": 2}
14heads_state = ckpt["heads_state_dict"] # one Linear(768, 3) per dimension
15
16heads = {dim: torch.nn.Linear(768, 3) for dim in label_dims}
17for dim, head in heads.items():
18 head.weight.data = heads_state[f"{dim}.weight"]
19 head.bias.data = heads_state[f"{dim}.bias"]
20 head.eval()
21
22backbone.eval()
23text = "Your input passage here."
24inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
25with torch.no_grad():
26 hidden = backbone(**inputs).last_hidden_state # (1, seq_len, 768)
27 mask = inputs["attention_mask"].unsqueeze(-1)
28 pooled = (hidden * mask).sum(1) / mask.sum(1) # mean pooling
29
30id2label = {dim: {v: k for k, v in mapping.items()} for dim, mapping in label2id.items()}
31for dim, head in heads.items():
32 pred_id = head(pooled).argmax(-1).item()
33 print(dim, "->", id2label[dim][pred_id])low/medium/high) granularity is coarse; F1 scores in the 0.54-0.59 range indicate the heads are useful as a filtering signal but should not be treated as ground truth.heads.pt requires custom loading code (shown above) — it is not loadable via AutoModelForSequenceClassification.