This model was fine-tuned to answer financial questions accurately and concisely. It handles topics like investment concepts, financial ratios, market instruments, macroeconomics, and corporate finance.

Base model: mistralai/Mistral-7B-v0.1
Fine-tuning method: QLoRA (NF4 4-bit quantization + LoRA adapters)
LoRA rank: 16 | LoRA alpha: 32
Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Trainable parameters: ~20M out of 7.2B (0.28%)

Training Data

Dataset: gbharti/finance-alpaca
Samples used: 10,000 instruction-response pairs
Train/test split: 90/10
Prompt format: Alpaca instruction template

Training Details

Parameter	Value
Epochs	2
Batch size (per device)	2
Gradient accumulation steps	8
Effective batch size	16
Learning rate	2e-4
LR scheduler	Cosine
Warmup ratio	0.03
Optimizer	paged_adamw_32bit
Max sequence length	512
Quantization	NF4 4-bit (double quant)
Compute dtype	float16
Hardware	Kaggle T4 x2
Training time	~5 hours

Training Results

Epoch	Training Loss	Validation Loss
1	1.2723	1.4589
2	1.0242	1.5069

How to Use

Load the adapter (memory efficient)

python

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

base_model_id = "mistralai/Mistral-7B-v0.1"
adapter_id = "MadhurArora1/mistral-7b-finance-qlora"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16
)

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    quantization_config=bnb_config,
    device_map="auto"
)
model = PeftModel.from_pretrained(model, adapter_id)

Run inference

python

def ask(question, max_new_tokens=200):
    prompt = f"Below is a financial question. Answer it accurately and concisely.\n\n### Question:\n{question}\n\n### Answer:\n"
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            temperature=0.1,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id
        )
    return tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

print(ask("What is the P/E ratio and how do investors use it?"))
print(ask("Explain the difference between stocks and bonds."))
print(ask("What is compound interest?"))

Example Outputs

Q: What is the P/E ratio and how do investors use it?

The P/E ratio (Price-to-Earnings ratio) measures how much investors are willing to pay for each dollar of a company's earnings. It is calculated by dividing the stock price by earnings per share (EPS). A high P/E may indicate the market expects strong future growth, while a low P/E may suggest undervaluation or slow growth expectations. Investors use it to compare companies within the same industry.

Q: What is the difference between ROE and ROA?

ROE (Return on Equity) measures how efficiently a company generates profit from shareholders' equity, while ROA (Return on Assets) measures profitability relative to total assets. ROE focuses on returns to equity holders and is affected by leverage; ROA is a broader measure of operational efficiency regardless of how assets are financed.

Limitations

Fine-tuned on 10k samples — may not cover all financial topics
Not suitable for real investment advice or financial decisions
Performance degrades on highly technical or niche financial topics
Occasional hallucinations on specific numerical data (e.g. historical prices, exact rates)

Training Infrastructure

Platform: Kaggle (free tier)
GPU: NVIDIA Tesla T4 x2 (30GB total VRAM)
Framework: HuggingFace Transformers + PEFT + TRL
Experiment tracking: Weights & Biases

Author

Madhur Arora — Backend SWE | AI/ML enthusiast
GitHub | LinkedIn

Citation

If you use this model, please cite the base model:

@misc{mistral7b,
  title={Mistral 7B},
  author={Mistral AI},
  year={2023},
  url={https://huggingface.co/mistralai/Mistral-7B-v0.1}
}