Multimodal SLM Fine-Tuning: ChartQA with MatCha
This repository contains the code and documentation for fine-tuning a Small Language Model (SLM) on the ChartQA dataset using Parameter-Efficient Fine-Tuning (LoRA).
This project was developed to demonstrate a complete multimodal fine-tuning pipeline capable of running on a single NVIDIA T4 GPU (16GB VRAM).
🚀 How to Run Inference
The following standalone code snippet demonstrates how to pull the fine-tuned LoRA adapters from Hugging Face, merge them with the base model, and run inference on a custom chart image.
1import torch
2from transformers import AutoProcessor, AutoModelForImageTextToText
3from peft import PeftModel
4from PIL import Image
5
6# 1. Define Model IDs and Device
7base_model_id = "google/matcha-base"
8adapter_id = "Sairam22/matcha-chartqa-lora-adapter" # Replace if your repo name is different
9device = "cuda" if torch.cuda.is_available() else "cpu"
10
11# 2. Load Base Model and Processor
12print("Loading base model and processor...")
13processor = AutoProcessor.from_pretrained(adapter_id)
14base_model = AutoModelForImageTextToText.from_pretrained(
15 base_model_id,
16 torch_dtype=torch.float16,
17 low_cpu_mem_usage=True
18)
19
20# 3. Pull Adapter from Hugging Face and Merge
21print("Pulling adapter and merging weights...")
22model = PeftModel.from_pretrained(base_model, adapter_id)
23model = model.merge_and_unload()
24model = model.to(device)
25
26# 4. Prepare Image and Prompt
27image_path = "path_to_your_chart.png" # Provide the path to a local chart image
28image = Image.open(image_path).convert("RGB")
29prompt = "Question: What is the highest value in the bar chart?\nAnswer:"
30
31# 5. Process Inputs and Cast Dtypes
32inputs = processor(images=image, text=prompt, return_tensors="pt")
33# Ensure float32 tensors (like images) are cast to float16 to match the model weights
34inputs = {k: v.to(device, dtype=torch.float16) if v.dtype == torch.float32 else v.to(device) for k, v in inputs.items()}
35
36# 6. Run Inference
37print("Generating prediction...")
38with torch.no_grad():
39 outputs = model.generate(**inputs, max_new_tokens=32)
40
41prediction = processor.decode(outputs[0], skip_special_tokens=True)
42print(f"Prediction: {prediction}")
43
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 2
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 8
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- training_steps: 50
- mixed_precision_training: Native AMP
🧠 Decision Log & T4 Optimizations
To ensure this pipeline runs efficiently on a single NVIDIA T4 GPU (16GB VRAM) and within strict time limits, several specific parameter choices were made:
Model Selection (google/matcha-base): Chosen because it is pre-trained specifically for chart visual language tasks (based on Pix2Struct) and is highly lightweight (~256M parameters), fitting easily into T4 memory.
Precision (fp16=True): Casting the base model to float16 cuts memory consumption in half, prevents datatype mismatch errors (c10::Half), and leverages the T4's Tensor Cores to speed up training.
LoRA Configuration (r=8, alpha=16): A rank of 8 introduces less than 5% trainable parameters. This prevents Out-Of-Memory (OOM) errors during training since the optimizer states are kept minimal. MatCha's specific attention layers (query, value) were explicitly targeted.
Batch Sizing (batch_size=2, gradient_accumulation=4): A physical batch size of 2 ensures VRAM limits aren't breached during the forward/backward pass, while gradient accumulation simulates an effective batch size of 8 for stable loss convergence.
Adapter Merging (merge_and_unload()): Fulfills the assignment requirement while also improving inference speed by flattening the adapter weights directly into the base model matrices, completely removing dynamic routing overhead during generation.
Training Subset: Due to hardware compute constraints and time limitations, a rapid subset of 100 samples was used for the final epoch to successfully validate the end-to-end pipeline, adapter uploading, and inference mechanisms.
Framework versions
- PEFT 0.18.1
- Transformers 5.3.0
- Pytorch 2.10.0+cu128
- Datasets 4.0.0
- Tokenizers 0.22.2
