Non-Fine-Tuned LLaMA-3-8B CEFR Evaluation

• Base Model: unsloth/llama-3-8b-instruct-bnb-4bit
• Evaluation Details:
  • Dataset: Rebalanced test dataset (test_merged_output.txt), which was also used to train and evaluate the classifier, potentially introducing bias.
  • No fine-tuning performed; base model used directly.
  • Classifier: MLP classifier trained on train_merged_output.txt, dev_merged_output.txt, and test_merged_output.txt for CEFR level prediction.
• Evaluation Metrics (Exact Matches):
  • CEFR Classifier Accuracy: 0.150
  • Precision (Macro): 0.194
  • Recall (Macro): 0.150
  • F1-Score (Macro): 0.140
• Evaluation Metrics (Within ±1 Level):
  • CEFR Classifier Accuracy: 0.750
  • Precision (Macro): 0.826
  • Recall (Macro): 0.750
  • F1-Score (Macro): 0.741
• Other Metrics:
  • Perplexity: 86.022
  • Diversity (Unique Sentences): 0.967
  • Inference Time (ms): 4952.351
  • Model Size (GB): 8.0
  • Robustness (F1): 0.133
• Confusion Matrix (Exact Matches):
  • CSV: confusion_matrix_exact.csv
  • Image: confusion_matrix_exact.png
• Confusion Matrix (Within ±1 Level):
  • CSV: confusion_matrix_within1.csv
  • Image: confusion_matrix_within1.png
• Per-Class Confusion Metrics (Exact Matches):
  • A1: TP=0, FP=0, FN=10, TN=50
  • A2: TP=1, FP=11, FN=9, TN=39
  • B1: TP=3, FP=19, FN=7, TN=31
  • B2: TP=2, FP=16, FN=8, TN=34
  • C1: TP=2, FP=4, FN=8, TN=46
  • C2: TP=1, FP=1, FN=9, TN=49
• Per-Class Confusion Metrics (Within ±1 Level):
  • A1: TP=4, FP=0, FN=6, TN=50
  • A2: TP=8, FP=2, FN=2, TN=48
  • B1: TP=10, FP=6, FN=0, TN=44
  • B2: TP=8, FP=7, FN=2, TN=43
  • C1: TP=10, FP=0, FN=0, TN=50
  • C2: TP=5, FP=0, FN=5, TN=50
• Note on Bias:
  • The test dataset used for evaluation (test_merged_output.txt) was part of the training and evaluation data for the classifier (Mr-FineTuner/Skripsi_validator_best_model). This may lead to inflated performance metrics due to the classifier's familiarity with the dataset. For a more robust evaluation, a new dataset not used in classifier training is recommended.
• Usage:
  python

  1from transformers import AutoModelForCausalLM, AutoTokenizer 2 3model = AutoModelForCausalLM.from_pretrained("unsloth/llama-3-8b-instruct-bnb-4bit") 4tokenizer = AutoTokenizer.from_pretrained("unsloth/llama-3-8b-instruct-bnb-4bit") 5 6# Example inference 7prompt = "[INST] Generate a CEFR B1 level sentence. [/INST]" 8inputs = tokenizer(prompt, return_tensors="pt") 9outputs = model.generate(**inputs, max_length=50) 10print(tokenizer.decode(outputs[0], skip_special_tokens=True))