This model is a fine-tuned version of mistralai/Mistral-7B-v0.1.
It achieves the following results on the evaluation set:
Loss: 1.8378
Model description
This model is fine-tuned to specialize in generating content related to the environment and sustainability domain. The training involved Supervised Fine-Tuning (SFT), Parameter Efficient Fine-Tuning (PEFT), and Low-Rank Adaptation (LoRA) techniques to optimize model performance. The motivation behind this research is to explore the feasibility and effectiveness of Semantically Sufficient Private Large Language Models (LLMs) for secure, domain-specific knowledge extraction in the context of environment and sustainability.
Intended uses
The model is intended for information retrieval and knowledge extraction tasks within the domain of environment and sustainability.
Training and evaluation data
The training data consists of domain-specific text collected from Wikipedia pages related to environmental topics.
This model was trained using the Long dataset. Model trained with the Short dataset.
Dataset
URLs
Number of Rows
Number of Words
Number of Sentences
Short
11
577
51,526
2,150
Long
23
1,431
124,682
5,209
Table 1: Summary of Dataset Information
Environment and Sustainability
This model is tailored for the environment and sustainability domain, with a focus on assisting researchers and enterprises, particularly in alignment with the work of the Commonwealth Scientific and Industrial Research Organisation (CSIRO).
Data Collection Process
The training data was collected through a Python program that extracted and cleaned text content from specific Wikipedia pages related to environmental topics. The program utilized various libraries, such as requests, BeautifulSoup, and nltk, for efficient web scraping, HTML parsing, and natural language processing.
Training procedure
Fine-tuning
The fine-tuning process involved Soft Fine-Tuning, PEFT, and LoRA techniques. Soft Fine-Tuning utilized continuous-valued probabilities as labels, suitable for generation models. PEFT focused on updating a small subset of parameters during fine-tuning to prevent catastrophic forgetting. LoRA, a lightweight training technique, reduced the number of trainable parameters for faster and memory-efficient training.
Low-Rank Adaptation (LoRA) Parameters
lora_alpha: 16
lora_dropout: 0.1
r: 8
Training Parameters
num_train_epochs: 2
per_device_train_batch_size: 3
per_device_eval_batch_size: 3
gradient_accumulation_steps: 1
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
learning_rate: 5e-05
weight_decay: 0.001
max_grad_norm: 0.3
max_steps: -1
warmup_ratio: 0.03
group_by_length: True
lr_scheduler_type: constant
seed: 42
num_epochs: 2
Training results
Training Loss
Loss
Figure 1: Training loss curve of model fionazhang/mistral-finetune-long (logging step = 1)
In the training process, the observed training losses exhibit jittery yet overall decreasing trends. The final evaluation loss reaches a satisfactory value of 2.0377, indicating successful learning and adaptation to the nuances of the provided data.
