!git clone https://github.com/llm-jp/llm-jp-eval.git
!cd llm-jp-eval && pip install -e .
!cd llm-jp-eval && python scripts/preprocess_dataset.py --dataset-name all --output-dir ./dataset_dir
!cd llm-jp-eval && python scripts/evaluate_llm.py -cn config.yaml model.pretrained_model_name_or_path=jaeyong2/Qwen2.5-1.5B-Instruct-JaMagpie-Preview tokenizer.pretrained_model_name_or_path=jaeyong2/Qwen2.5-1.5B-Instruct-JaMagpie-Preview dataset_dir=./dataset_dir/1.4.1/evaluation/test

llm-jp-eval	Qwen2.5-1.5B-Instruct	google/gemma-2-2b-jpn-it	finetuning-model
AVG	0.4343	0.4315	0.4540
CG	0.0600	0.0000	0.1500
EL	0.3952	0.3222	0.4106
FA	0.0690	0.0846	0.0000
HE	0.4400	0.4350	0.4300
MC	0.6800	0.6000	0.6400
MR	0.4700	0.4900	0.5800
MT	0.6137	0.7666	0.7915
NLI	0.5500	0.5260	0.4440
QA	0.2443	0.2813	0.3054
RC	0.8208	0.8097	0.7881

License

Qwen/Qwen2.5-1.5B-Instruct : https://choosealicense.com/licenses/apache-2.0/

Acknowledgement

This research is supported by TPU Research Cloud program.

RichardErkhov/jaeyong2_-_Qwen2.5-1.5B-Instruct-Ja-SFT-8bits

Original model description:

Model Card for Model ID

Evaluation

llm-jp-eval script(colab)

License

Acknowledgement

RichardErkhov/
jaeyong2_-_Qwen2.5-1.5B-Instruct-Ja-SFT-8bits