A QLoRA fine-tuned LoRA adapter for Llama-3.2-1B, trained on the FinDER financial question-answering dataset. The adapter teaches the base model to read SEC filing evidence passages and produce grounded, concise answers to financial questions — a domain where the base model struggles with financial abbreviations, disambiguation of related figures, and synthesis across financial statements.
Developed by: Linda Lin (qat3207)
Model type: Causal language model — LoRA adapter for meta-llama/Llama-3.2-1B
Language(s) (NLP): English
License: Llama 3.2 Community License (inherited from base model)
Training notebook:notebooks/fine_tuning_for_FinDER_dataset.ipynb (QLoRA pipeline, runs on Colab A100)
Uses
Direct Use
Load the adapter on top of the base meta-llama/Llama-3.2-1B model to answer financial questions grounded in SEC filing evidence. The model expects a prompt that contains an evidence passage and a question, and returns a short factual answer.
python
1from transformers import AutoModelForCausalLM, AutoTokenizer
2from peft import PeftModel
34base_model = AutoModelForCausalLM.from_pretrained(5"meta-llama/Llama-3.2-1B",6 load_in_4bit=True,7 device_map="auto",8)9model = PeftModel.from_pretrained(base_model,"KKieXX/llama-3.2-1b-finder-lora")10tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B")1112evidence ="The company reported net revenues of $3.2 billion for fiscal year 2023..."13question ="What were the net revenues for fiscal year 2023?"1415prompt =f"Evidence: {evidence}\n\nQuestion: {question}\n\nAnswer:"16inputs = tokenizer(prompt, return_tensors="pt").to(model.device)17output = model.generate(**inputs, max_new_tokens=200, temperature=0.7, top_p=0.9)18print(tokenizer.decode(output[0], skip_special_tokens=True))
Note: The base model is gated on HuggingFace Hub — you must accept the Llama 3.2 license and authenticate with huggingface-cli login before downloading.
Downstream Use
This adapter is the generator and evaluator backbone of the FinDER Multi-Agent Financial QA System, a LangGraph pipeline that pairs it with Pinecone RAG over 50k+ indexed SEC filing chunks. It performs two roles in that pipeline:
Generator: Given retrieved evidence + a user question, produce a factual answer.
Evaluator: Assess whether a generated answer is SUFFICIENT or INSUFFICIENT, triggering a retrieval retry if needed.
Out-of-Scope Use
General-purpose conversational chat (model is not instruction-tuned for dialogue)
Financial advice or investment decisions — outputs are not verified by domain experts
Questions without grounding evidence (model is not designed for zero-shot financial QA)
Languages other than English
Bias, Risks, and Limitations
Domain coverage: Training data is limited to SEC filings (10-K, 10-Q, etc.). Performance on other financial document types (earnings calls, analyst reports) is untested.
Small model capacity: At 1B parameters, the model may struggle with complex multi-step reasoning or synthesis across long documents.
Hallucination risk: As with all LLMs, the model can produce plausible-sounding but incorrect numerical figures. Always verify outputs against source documents.
Training data cutoff: The FinDER dataset derives from SEC filings with a specific temporal range; the model may not generalize to very recent filings with new financial instruments or accounting standards.
Quantization artifacts: 4-bit quantization (NF4) introduces minor precision loss versus full-precision inference.
Recommendations
Users (both direct and downstream) should treat model outputs as a first-pass extraction aid, not as authoritative financial analysis. Cross-check all numerical figures against the original SEC source documents.
Training Details
Training Data
FinDER (Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generation):
5,703 query–evidence–answer triplets derived from SEC filings
Used a 30% stratified sample: 1,710 training examples (split with random_state=123, frac=0.3)
Remaining 70% held out as test split (never seen during fine-tuning)
Categories span revenue figures, expense line items, ratio calculations, and segment reporting
Citation:
bibtex
1@misc{choi2025finder,
2 title={FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generation},
3 author={Chanyeol Choi and Jihoon Kwon and Jaeseon Ha and Hojun Choi and Chaewoon Kim and Yongjae Lee and Jy-yong Sohn and Alejandro Lopez-Lira},
4 year={2025},
5 eprint={2504.15800},
6 archivePrefix={arXiv},
7 primaryClass={cs.IR},
8 url={https://arxiv.org/abs/2504.15800}
9}
Training Procedure
Preprocessing
Raw SEC table artifacts (tab characters, excessive whitespace, newlines) were cleaned from evidence passages before tokenization. Each example was formatted as:
Training time: ~619 seconds (~10 minutes) for 3 epochs over 1,710 examples
Adapter size: ~26 MB (LoRA weights only; base model downloaded separately)
Evaluation
Testing Data
Held-out FinDER test split: 70% of the 5,703 examples (stratified by category, same random_state=123 split). Evaluation samples 200 examples stratified by category.
Factors
Three ablation conditions:
Condition
Model
Evidence Source
A
Base Llama-3.2-1B
Gold evidence (oracle)
B
Fine-tuned Llama-3.2-1B (this model)
Gold evidence (oracle)
C
Fine-tuned Llama-3.2-1B (this model)
Pinecone RAG retrieval
The A→B gap isolates the fine-tuning effect. The B→C gap isolates retrieval quality.
Jaccard similarity ≥ 0.25 between retrieved and gold evidence
Results
See notebooks/evaluation.ipynb and eval_results.csv (generated at evaluation time) for per-example scores and aggregate results across all three conditions.
If you use this adapter, please also cite the FinDER dataset:
BibTeX:
bibtex
1@misc{choi2025finder,
2 title={FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generation},
3 author={Chanyeol Choi and Jihoon Kwon and Jaeseon Ha and Hojun Choi and Chaewoon Kim and Yongjae Lee and Jy-yong Sohn and Alejandro Lopez-Lira},
4 year={2025},
5 eprint={2504.15800},
6 archivePrefix={arXiv},
7 primaryClass={cs.IR},
8 url={https://arxiv.org/abs/2504.15800}
9}
