Fine-Tuned Mistral-7B CEFR Model

• Base Model: unsloth/mistral-7b-instruct-v0.3-bnb-4bit
• Fine-Tuning: LoRA with SMOTE-balanced dataset
• Training Details:
  • Dataset: CEFR-level sentences with SMOTE and undersampling for balance (no rebalancing for validation/test sets)
  • LoRA Parameters: r=32, lora_alpha=32, lora_dropout=0.5
  • Training Args: learning_rate=2e-5, batch_size=8, epochs=0.1, cosine scheduler
  • Optimizer: adamw_8bit
  • Early Stopping: Patience=3, threshold=0.01
• Evaluation Metrics (Exact Matches):
  • CEFR Classifier Accuracy: 0.200
  • Precision (Macro): 0.096
  • Recall (Macro): 0.200
  • F1-Score (Macro): 0.125
• Evaluation Metrics (Within ±1 Level):
  • CEFR Classifier Accuracy: 0.683
  • Precision (Macro): 0.642
  • Recall (Macro): 0.683
  • F1-Score (Macro): 0.635
• Other Metrics:
  • Perplexity: 9.486
  • Diversity (Unique Sentences): 0.933
  • Inference Time (ms): 5107.406
  • Model Size (GB): 4.1
  • Robustness (F1): 0.119
• Confusion Matrix (Exact Matches):
  • CSV: confusion_matrix_exact.csv
  • Image: confusion_matrix_exact.png
• Confusion Matrix (Within ±1 Level):
  • CSV: confusion_matrix_within1.csv
  • Image: confusion_matrix_within1.png
• Per-Class Confusion Metrics (Exact Matches):
  • A1: TP=0, FP=0, FN=10, TN=50
  • A2: TP=0, FP=0, FN=10, TN=50
  • B1: TP=7, FP=25, FN=3, TN=25
  • B2: TP=1, FP=13, FN=9, TN=37
  • C1: TP=4, FP=10, FN=6, TN=40
  • C2: TP=0, FP=0, FN=10, TN=50
• Per-Class Confusion Metrics (Within ±1 Level):
  • A1: TP=0, FP=0, FN=10, TN=50
  • A2: TP=6, FP=0, FN=4, TN=50
  • B1: TP=8, FP=10, FN=2, TN=40
  • B2: TP=10, FP=7, FN=0, TN=43
  • C1: TP=9, FP=2, FN=1, TN=48
  • C2: TP=8, FP=0, FN=2, TN=50
• Usage:
  python

  1from transformers import AutoModelForCausalLM, AutoTokenizer 2 3model = AutoModelForCausalLM.from_pretrained("Mr-FineTuner/Test___01_withNewEval_andWithin-1_testnewmodels") 4tokenizer = AutoTokenizer.from_pretrained("Mr-FineTuner/Test___01_withNewEval_andWithin-1_testnewmodels") 5 6# Example inference 7prompt = "<|user|>Generate a CEFR B1 level sentence.<|end|>" 8inputs = tokenizer(prompt, return_tensors="pt") 9outputs = model.generate(**inputs, max_length=50) 10print(tokenizer.decode(outputs[0], skip_special_tokens=True))