A multi-head transformer regression model based on BERT (GroNLP/bert-base-dutch-cased), fine-tuned to predict four normalized delta scores for Dutch book reviews. The four output heads are:

delta_cola_to_final
delta_perplexity_to_final_large
iter_to_final_simplified
robbert_delta_blurb_to_final

⚠️ The order of these outputs is crucial and must be maintained exactly as above during inference.
Changing the order will cause incorrect mapping of predicted values to their respective targets.

Additionally, a final aggregate score is provided (mean of the four heads).

📈 Training & Evaluation

Base model: GroNLP/bert-base-dutch-cased
Fine-tuning: 5 epochs on a proprietary dataset
Output heads: 4
Problem type: multi-head regression

Per-Epoch Validation Metrics

Epoch	Val Loss	ΔCoLA RMSE / R²	ΔPerp RMSE / R²	Iter RMSE / R²	Blurb RMSE / R²	Mean RMSE
1	0.01636	0.1498 / 0.3689	0.0999 / 0.6485	0.1385 / 0.8184	0.1178 / 0.7295	0.1265
2	0.01522	0.1466 / 0.3950	0.1019 / 0.6347	0.1272 / 0.8467	0.1132 / 0.7499	0.1222
3	0.01521	0.1470 / 0.3922	0.0986 / 0.6579	0.1278 / 0.8453	0.1148 / 0.7429	0.1220
4	0.01516	0.1429 / 0.4250	0.0999 / 0.6485	0.1284 / 0.8438	0.1171 / 0.7324	0.1221
5	0.01546	0.1447 / 0.4107	0.1002 / 0.6465	0.1311 / 0.8373	0.1169 / 0.7333	0.1232

✅ Final Aggregate Performance (Test)

Metric	Value
Aggregate RMSE	0.0769
Aggregate R²	0.8425
Mean RMSE (heads)	0.1210

🗂️ Test Metrics (Per Target)

Target	RMSE	R²
delta_cola_to_final	0.1463	0.4286
delta_perplexity_to_final_large	0.0955	0.6802
iter_to_final_simplified	0.1255	0.8535
robbert_delta_blurb_to_final	0.1168	0.7319

🏷️ Notes

Base model: GroNLP/bert-base-dutch-cased
Fine-tuned for multi-head regression on Dutch book reviews
Trained for 5 epochs on a proprietary dataset
Sigmoid activation built into each head
Re-aggregation: simple average of the four head outputs

🛠️ Training Arguments

num_train_epochs=5
per_device_train_batch_size=8
per_device_eval_batch_size=16
gradient_accumulation_steps=2
learning_rate=2e-5
weight_decay=0.01
eval_strategy="epoch"
save_strategy="epoch"
load_best_model_at_end=True
metric_for_best_model="mean_rmse"
greater_is_better=False
bf16 enabled if supported, else fp16 enabled
logging_strategy="epoch"
push_to_hub=True with model ID Felixbrk/bert-base-dutch-cased-multi-score-tuned-positive
hub_strategy="end"
Early stopping with patience 2 epochs

⚠️ Important:

Always load this model with trust_remote_code=True as it uses a custom multi-head regression architecture.
Maintain the output order exactly for correct interpretation of results.

Felixbrk/bert-base-dutch-cased-multi-score-tuned-positive

transformer_multi_head_bert_updated