Views
No views yet
educa-ai-nemo-dpo is the preference-aligned version of our SFT model DigitalLearningGmbH/educa-ai-nemo-sft,
using our internal dataset which contains a unique mix of German and English preference data covering a multitude of domains.
In its creation we have paid special attention to data points that can improve performance in German, especially the educational field (text analysis, supporting students in completing textual tasks, ...).[INST]).
We include the updated template in the tokenizer config, so you can use tokenizer.apply_chat_template.--apply_chat_template.
For comparison, we performed the same benchmarks on the base model and Llama-3.1-8B-Instruct as well, in the exact same environment with the same parameters.| Benchmark | Llama-3.1-8B-Instruct | Mistral-Nemo-Instruct-2407 | educa-ai-nemo-dpo |
|---|---|---|---|
| hellaswag (acc_norm) | 72.6% | 71.9% | 77.6% |
| winogrande (acc) | 68.0% | 69.8% | 75.2% |
| openbookqa (acc_norm) | 49.0% | 45.8% | 47.0% |
| commonsense_qa (acc) | 64.9% | 74.4% | 75.4% |
| truthfulqa_mc1 (acc) | 40.4% | 39.66% | 41.5% |
| mmlu (acc) | 63.2% | 64.9% | 66.5% |
| triviaqa (exact_match) | 5.3% | 12.3% | 23.99% |
| agieval (acc) | 36.3% | 36.6% | 39.1% |
| arc_challenge (acc_norm) | 54.1% | 52.5% | 54.4% |
| arc_easy (acc_norm) | 75.7% | 74.1% | 76.0% |
| piqa (acc_norm) | 79.6% | 78.9% | 81.5% |
| leaderboard_bbh (acc_norm) | 37.4% | 49.1% | 53.0% |
| leaderboard_gpqa (acc_norm) | 28.5% | 30.6% | 29.4% |
| leaderboard_ifeval (inst_level_loose_acc) | 84.7% | 72.8% | 75.1% |
| leaderboard_mmlu_pro (acc) | 16.2% | 35.1% | 33.67% |
| leaderboard_musr (acc_norm) | 38.8% | 39.3% | 40.2% |
| Benchmark | Llama-3.1-8B-Instruct | Mistral-Nemo-Instruct-2407 | educa-ai-nemo-dpo |
|---|---|---|---|
| global_mmlu_full (acc) | |||
| - de | 48.2% | 55.8% | 57.5% |
| - en | 60.0% | 63.1% | 63.8% |
| - es | 54.7% | 58.1% | 58.9% |
| - fr | 48.3% | 56.3% | 58.1% |
| - it | 51.0% | 58.1% | 59.6% |
| - ja | 47.4% | 50.0% | 51.0% |
| - pt | 23.0% | 43.5% | 55.7% |
| - ru | 41.4% | 54.9% | 55.0% |
| - zh | 49.7% | 52.2% | 55.6% |
| arc_challenge_mt (acc_norm) | |||
| - de | 39.9% | 42.6% | 46.8% |
| - es | 42.8% | 45.6% | 47.3% |
| - it | 43.9% | 44.3% | 46.7% |
| - pt | 41.9% | 42.3% | 46.8% |
| xnli (acc) | |||
| - de | 48.1% | 47.6% | 47.1% |
| - en | 52.4% | 57.3% | 57.8% |
| - es | 46.3% | 45.0% | 47.0% |
| - fr | 51.6% | 38.5% | 40.0% |
| - ru | 48.1% | 41.8% | 38.6% |
| - zh | 40.3% | 36.3% | 36.1% |
| xquad (f1) | |||
| - de | 30.4% | 22.7% | 35.6% |
| - en | 35.0% | 21.8% | 29.9% |
| - es | 31.2% | 17.6% | 29.6% |
| - ru | 39.6% | 24.6% | 37.3% |
| - zh | 28.8% | 10.0% | 16.7% |