Fine-Tuned CALM2-7B-Chat on Hawaii Wildfires Data
Model Description
This is a fine-tuned version of
cyberagent/calm2-7b-chat, a 7B parameter Transformer-based language model designed for dialogue use cases, supporting both Japanese and English. The fine-tuning incorporates specific knowledge about the August 2023 Hawaii wildfires, sourced from the Maui Police Department's report. The goal is to enhance the model's ability to respond to queries related to this event in a conversational manner.
The fine-tuning was performed using QLoRA (4-bit quantization) and LoRA adapters to efficiently adapt the model on limited hardware, such as Google Colab's free T4 GPU.
Authors
- Base Model: CyberAgent
- Fine-Tuning: Ai2Web3(LogeswaranA)/PradeepNatarajan (GitHub/Hugging Face username)
Intended Uses
- Primary Use: Chatbot or dialogue systems that need factual information about the 2023 Hawaii wildfires.
- Out-of-Scope Uses: Not intended for generating harmful content, misinformation, or applications requiring high-stakes decision-making without human oversight.
Training Data
The dataset consists of raw text extracted from the Maui Police Department's preliminary after-action report on the August 2023 Hawaii wildfires. The report is available at:
http://www.mauipolice.com/uploads/1/3/1/2/131209824/pre_aar_master_copy_final_draft_1.23.24.pdf.
Specific files used:
hawaii_wf_4.txthawaii_wf_2.txt
These files contain unformatted text copied from the PDF, focusing on details like incident response, timelines, and observations from the wildfires.
The dataset was loaded using Hugging Face's datasets library as a text dataset.
Training Procedure
Preprocessing
- Tokenizer: From the base model (
cyberagent/calm2-7b-chat), with PAD token added as EOS if missing.
- Data Tokenization: Each text sample was tokenized without additional formatting.
Fine-Tuning Details
- Quantization: 4-bit using BitsAndBytes with double quantization, NF4 type, and bfloat16 compute dtype.
- LoRA Configuration:
- Rank (
r): 8
- Alpha: 64
- Target Modules:
["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
- Dropout: 0.05
- Bias: None
- Task Type: CAUSAL_LM
- Trainer Settings (using Hugging Face Transformers):
- Batch Size: 2 (per device)
- Gradient Accumulation Steps: 2
- Epochs: 3 (but limited to max_steps=20)
- Learning Rate: 1e-4
- Optimizer: paged_adamw_8bit
- Logging Steps: 10
- Save Strategy: Epoch (with save_steps=50)
- Data Collator: For language modeling (mlm=False)
- Hardware: Trained on Google Colab with GPU (NVIDIA T4).
- Gradient checkpointing was enabled for memory efficiency.
The model was prepared for k-bit training and used PEFT for LoRA integration.
Evaluation
No formal evaluation was performed in the training script. Qualitative testing via inference is recommended to assess performance on wildfire-related queries.
Usage
To load and use the fine-tuned model with PEFT:
1from peft import PeftModel
2from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
3import torch
4
5base_model_id = "cyberagent/calm2-7b-chat"
6repo_id = "Ai2Web3/calm2-7b-chat"
7
8# Define the same quantization config as used during training
9bnb_config = BitsAndBytesConfig(
10 load_in_4bit=True,
11 bnb_4bit_use_double_quant=True,
12 bnb_4bit_quant_type="nf4",
13 bnb_4bit_compute_dtype=torch.bfloat16
14)
15
16# Load base model with quantization, forcing all to GPU (assuming it fits)
17model = AutoModelForCausalLM.from_pretrained(
18 base_model_id,
19 quantization_config=bnb_config,
20 device_map={"": "cuda:0"} # Force everything to GPU 0; change if multi-GPU
21)
22tokenizer = AutoTokenizer.from_pretrained(base_model_id)
23
24# Load LoRA adapters (adapters will follow the base model's device map)
25model = PeftModel.from_pretrained(model, repo_id)
26
27# Inference example
28prompt = "What caused the Hawaii wildfires in August 2023?"
29inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
30outputs = model.generate(**inputs, max_new_tokens=200)
31print(tokenizer.decode(outputs[0], skip_special_tokens=True))
For quantized inference, apply the same BitsAndBytesConfig as during training.Limitations and BiasesTraining Scale: Limited to 20 steps on a small dataset, which may result in incomplete adaptation or overfitting.
Language: Base model supports Japanese and English; fine-tuning data is in English, so performance may vary for Japanese queries.
Biases: Inherits biases from the base model and training data, which is an official report and may reflect institutional perspectives.
Hallucinations: As with all LLMs, the model may generate inaccurate information; verify outputs against reliable sources.
Safety: Not tested for harmful outputs; use with caution.
License
This model is licensed under Apache-2.0, following the base model's license.
Acknowledgments Base model by CyberAgent.
Tutorial and data preparation inspired by poloclub/Fine-tuning-LLMs.
Fine-tuning powered by Hugging Face Transformers, PEFT, and BitsAndBytes.
