Model Card for LLaMA 3.1 Fine-Tuned Model

Model Details

Model Description

• Developed by: Sanaa Abril
• Model Type: Fine-tuned Causal Language Model
• Language(s) (NLP): French
• License:
• Finetuned from model: Meta LLaMA 3.1 8B Instruct

Model Sources [optional]

• Repository: https://huggingface.co/sanaa-11/mathematic-exercice-generator/tree/main

Uses

Direct Use

• Primary Application: This model is primarily used for generating math exercises tailored to Moroccan students in French, based on specific lessons and difficulty levels.
• Example Use Case: Educators can input lesson topics to generate corresponding exercises for classroom use or online learning platforms.

Downstream Use [optional]

• Potential Applications: The model can be extended or adapted to create exercises in other languages or for different educational levels.

Out-of-Scope Use

• Not Suitable For: The model is not designed for high-stakes assessments, as it may generate exercises that require further validation by subject matter experts.

Bias, Risks, and Limitations

• Bias: The model may inherit biases from the data it was trained on, potentially generating exercises that reflect unintended cultural or linguistic biases.
• Risks: There is a risk of generating mathematically incorrect exercises or exercises that do not align with the intended curriculum.
• Limitations: The model's accuracy and relevance may decrease when generating exercises outside of its training domain or when applied to advanced mathematical topics not covered during fine-tuning.

Recommendations

• For Educators: It is recommended to review the generated exercises for correctness and relevance before using them in a classroom setting.
• For Developers: Fine-tune the model further or adjust the training data to mitigate any biases and improve the quality of the generated content.

How to Get Started with the Model

generated_text = ""
prompt = "Fournis un exercice basé sur la vie reelle de difficulté moyenne de niveau 2 annee college sur les fractions."
for _ in range(5):  
    inputs = tokenizer(prompt + generated_text, return_tensors="pt").to(model.device)
    outputs = model.generate(
        **inputs,
        max_length=1065,  
        temperature=0.7,
        top_p=0.9,
        num_beams=5,
        repetition_penalty=1.2,
        no_repeat_ngram_size=2,
        pad_token_id=tokenizer.eos_token_id,
        early_stopping=False
)
    new_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
    generated_text += new_text
    print(new_text)

Training Details

Training Data

• Dataset: The model was fine-tuned on a custom dataset consisting of 3.6K rows of math exercises, lesson content, and solutions, specifically designed for Moroccan students in French laungage.

Training Procedure

Preprocessing [optional]

• Data Cleaning: Text normalization, tokenization, and padding were applied to prepare the data.
• Tokenization: The French tokenizer provided by Hugging Face was used to process the text data.

Training Hyperparameters

• Training Regime: The model was fine-tuned using 4-bit quantization with QLoRA to optimize GPU and RAM usage. The training was performed on a Kaggle environment with limited resources.
• Batch Size: 1 (with gradient accumulation steps of 8)
• Number of Epochs: 8
• Learning Rate: 5e-5

Evaluation

Testing Data, Factors & Metrics

• A separate subset of 10% of the dataset was reserved for evaluation.

• Complexity of Generated Exercises: Exercises were evaluated based on their complexity relative to the intended difficulty level.

• Training Loss: The loss measured during training.
• Validation Loss: The loss measured on the validation dataset during training.

• Training and Validation Loss: The model was evaluated based on training and validation loss over 8 epochs. The results indicate that the model's performance improved significantly after the first few epochs, with a steady decrease in both training and validation loss. The final validation loss achieved was 0.154888, indicating a good fit to the validation data without significant overfitting.

Summary

• The model demonstrated a consistent reduction in both training and validation loss across the training epochs, suggesting effective learning and generalization from the provided dataset.

Environmental Impact

• Hardware Type: Tesla T4 GPU
• Hours Used: 12 hours
• Cloud Provider: Kaggle
• Carbon Emitted: [Can be estimated using the Machine Learning Impact calculator by Lacoste et al. (2019)]

Technical Specifications [optional]

• The model is based on the LLaMA 3.1 architecture, fine-tuned to generate text in French for educational purposes, specifically math exercises.

• The model was trained on Kaggle’s free-tier environment, leveraging a single Tesla T4 GPU.

• GPU: Tesla T4 with 16GB RAM

• Transformers Version: 4.44.0
• PEFT Version: 0.12.0

Citation [optional]