You can deploy this model and start earning money today!

bachir6c/
mistral-mam-lora

Name: mistral-mam-lora
Availability: InStock

Views

No views yet

Mistral-7B Math LoRA Adapter (MAM Project)

This repository contains the LoRA adapter weights for Mistral-7B-v0.1, fine-tuned on the GSM8K mathematical reasoning dataset.

Developer: Mouhamed Bachir CISSE (Polytech Lyon, Applied Mathematics & Modeling)
Base Model: mistralai/Mistral-7B-v0.1
Task: Causal Language Modeling / Mathematical Reasoning
Fine-tuning Method: Parameter-Efficient Fine-Tuning (PEFT) with LoRA

Training Details

Dataset: openai/gsm8k
Trained Parameters: 6,815,744 (0.094% of total parameters)
Training Loss: Decreased from 1.025 to 0.722 (-30%) over 200 steps
Hardware: 1x NVIDIA T4 GPU
Precision: BF16 / 4-bit quantization

LoRA Configuration

python

LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

How to Use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "mistralai/Mistral-7B-v0.1"
adapter_id = "bachir6c/mistral-mam-lora"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Load the LoRA adapter
model = PeftModel.from_pretrained(base_model, adapter_id)

prompt = "Question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\nAnswer:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))