A QLoRA finetuned Qwen2.5-7b-Instruct model trained on public financial data, and implemented with naive RAG system and FAISS vector DB.

You can find the model here: Qwen2.5-7B-Instruct

There are a few main reasons to pick this model for finetuning and local model deployment. First of all, it had to be small enough to launch locally on a user's device for real-time inference. Also the model has singnificantly higher eval scores in mathmatics and coding than other models. The model is also shown to be extremley improved in instruction following, and understand structured data (especially JSON). This is needed for FinSight as the system prompt and external API data is given through a JSON format. Finally the Casual Language Model has long-context support and can generate a high amount of tokens, plenty for this project.

QLoRA is a industry norm when training models on limited computing power. This is used to reduce the memory footprint of the LM by quantizing the base model to a 4-bit percision while using LoRA adapters, still, for training. While mainly used for larger models, with a low amount of compute power, the 7B Qwen2.5-Instruct model was of choice for this project

This model is first quantinized utilzing QLoRA 4-Bit training. Since I was training with limited time and resources I quantinized the model to 4-bit integers instead of the normal 16 or 32. I trained the model using 4-bit weights, but I also utilized double quantinization to reduce the file size by around 1.1 GB(~5.5GB to 4.4 GB). This technique quantinizes the scale factor(quantinized_weight * scale ~ original_weight) as well. The scale factor is needed to dequantinize the weights after LoRA training so it is a 16 or 32 bit integer itself. By converting this a 8-bit integer the overall training time heavily decreases, but the computational cost to dequantize is a little higher.

By using:

You can double quantize your model for more effcient training. After training I then merged the LoRA weights with the original model again to "finetune" the model. This step converted it back to a FP16 model. Using the llama.cpp package I quantized the GGUF FP16 file to a 4-bit, K-quantized, medium-variant GGUF file for CPU level support and on-device inference(q4_k_m.gguf).

If you trained a LoRA adapter (for example qwen2.5-7b-finance-lora.zip), you cannot load that zip directly with llama-cpp-python.

This app requires a .gguf model file, so use this flow:

This is all done in the LoRA Finetuning File