Views
No views yet
1python -m lm_eval --model vllm --model_args pretrained=${model},tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.9,data_parallel_size=1 \
2 --tasks gsm8k,french_bench,gsm8k-fr,humaneval_instruct --batch_size 1 --apply_chat_template=True --confirm_run_unsafe_code --trust_remote_code| Model | gsm8k | french | gsm8k-fr | humaneval_instruct | avg |
|---|---|---|---|---|---|
| Qwen3-0.6B | 21.0 | 24.4 | 19.6 | 38.4 | 25.9 |
| Qwen3-0.6B-Math | 46.3 | 25.4 | 29.2 | 25.6 | 31.6 |
| Qwen3-0.6B-Fr | 36.1 | 26.5 | 26.5 | 38.4 | 31.9 |
| Qwen3-0.6B-Code | 36.8 | 23.8 | 24.9 | 46.3 | 33.0 |
| Qwen3-0.6B-Math-Code-Fr | 48.7 | 26.5 | 32.9 | 42.7 | 37.7 |