This model is a QLoRA fine-tuned LoRA adapter for google/gemma-3-4b-it, optimized for Arabic legal document OCR and structured information extraction. It analyzes document images and returns a structured JSON output.

Important: This repository contains the LoRA adapter only, not the full Gemma 3 4B base model. Load google/gemma-3-4b-it separately and attach this adapter with PEFT.

Model Details

Property	Value
Base Model	`google/gemma-3-4b-it`
Fine-tuning Method	QLoRA (4-bit NF4)
Adapter Method	LoRA
Task	Arabic Legal Document OCR
Input	Document image + instruction
Output	Structured JSON
Training Framework	LlamaFactory / PEFT
Final Validation Loss	`0.1686`

Installation

Local inference (Transformers + PEFT)

pip install transformers==4.57.6 accelerate==1.8.0 peft==0.17.1 json-repair pillow

High-performance serving (vLLM)

pip install -q transformers==4.57.6 vllm==0.15.0 json-repair

Image Preprocessing (Recommended)

For best OCR accuracy, preprocess images (resize, grayscale, contrast enhance) before sending them to the model:

python

import base64
from io import BytesIO
from PIL import Image, ImageEnhance

def preprocess_image(image_path, max_width=1024, do_enhance=True, return_base64=False):
    image = Image.open(image_path)
    gray_image = image.convert('L')

    if gray_image.width > max_width:
        ratio = max_width / float(gray_image.width)
        new_height = int(gray_image.height * ratio)
        gray_image = gray_image.resize((max_width, new_height), Image.LANCZOS)

    if do_enhance:
        enhancer = ImageEnhance.Contrast(gray_image)
        gray_image = enhancer.enhance(1.5)

    if return_base64:
        buffered = BytesIO()
        gray_image.save(buffered, format="JPEG", optimize=True, quality=95)
        img_str = base64.b64encode(buffered.getvalue()).decode('utf-8')
        return f"data:image/jpeg;base64,{img_str}"

    return gray_image

Usage

1. Transformers + PEFT + json-repair

python

import torch
import json_repair
from PIL import Image
from transformers import AutoProcessor, Gemma3ForConditionalGeneration
from peft import PeftModel

BASE_MODEL = "google/gemma-3-4b-it"
ADAPTER = "ahmedyasser006/arabic-legal-ocr-gemma-3-4b-qlora"

processor = AutoProcessor.from_pretrained(BASE_MODEL)

base_model = Gemma3ForConditionalGeneration.from_pretrained(
    BASE_MODEL, dtype="auto", device_map="auto"
)
model = PeftModel.from_pretrained(base_model, ADAPTER)
model.eval()

image = preprocess_image("document.jpg", return_base64=False)

task_1_message = """
You are a professional OCR Details Extractor.
Your rule to extract: the page markdown content in addition to the structural_elements of the document.
Extract the final output into a json format.
Do not generate any introduction or conclusion.
""".strip()


task_2_message = """
You are a professional OCR Details Extractor.
Your rule to extract the: document_classification, source, physical_properties, official_marks, signatures_authorization, routing_distribution, attachments_references, condition_notes and confidence_quality of the document.
Extract the final output into a json format.
Do not generate any introduction or conclusion.
""".strip()


messages = [
    {
        "role": "system",
        "content": [
            {
                "type": "text",
                "text": "You are a helpful assistant."
            }
        ],
    },
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": image,
            },
            {
                "type": "text",
                "text": task_1_message,
            },
        ],
    },
]


inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
)


# Move tensor inputs to the model device
inputs = {
    key: value.to(model.device)
    if hasattr(value, "to")
    else value
    for key, value in inputs.items()
}


input_length = inputs["input_ids"].shape[-1]


with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,
        do_sample=False,
    )


generated_tokens = outputs[0,input_length:]


result = processor.decode(
    generated_tokens,
    skip_special_tokens=True,
)

# Robust JSON parsing, even if the model output isn't perfectly formed
json_data = json_repair.loads(result)
print(result)

2. vLLM serving

vLLM can load LoRA adapters directly, so you can serve this adapter on top of the base model without merging weights:

Run the server

bash

vllm serve google/gemma-3-4b-it \
  --enable-lora \
  --lora-modules arabic-legal-ocr=ahmedyasser006/arabic-legal-ocr-gemma-3-4b-qlora \
  --dtype bfloat16 --gpu_memory_utilization 0.8 \
  --enable-chunked-prefill \
  --allowed-local-media-path "/workspace/"

Run inference

python

from openai import OpenAI
import json_repair

client = OpenAI(api_key="any", base_url="http://localhost:8000/v1")

b64_image = preprocess_image("document.jpg", return_base64=True)

response = client.chat.completions.create(
    model="arabic-legal-ocr",
    messages=[{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": b64_image}},
        {"type": "text", "text": "Extract details to JSON."}
    ]}]
)

structured_output = json_repair.loads(response.choices[0].message.content)
print(structured_output)

Expected Output

json

{
  "classification": { "type": null, "category": null, "language": "Arabic" },
  "source": { "authority": null, "document_number": null, "primary_date": null },
  "content": { "subject": null, "full_text": "", "tables": [], "legal_articles": [] },
  "quality": { "confidence": null, "manual_review": false }
}

The exact schema depends on the document image (see prompt for the full field set).

Repository Structure

text

arabic-legal-ocr-gemma-3-4b-qlora/
├── README.md
├── adapter_config.json
├── adapter_model.safetensors     ← final adapter, use for inference
├── tokenizer.model / tokenizer_config.json / ...
├── train_results.json / eval_results.json / trainer_state.json
└── last-checkpoint/               ← training-resume checkpoint only
    ├── adapter_model.safetensors
    ├── optimizer.pt / scheduler.pt / scaler.pt / rng_state*.pth
    └── trainer_state.json

The root files are the final adapter for inference. last-checkpoint/ is only needed if resuming training (it includes optimizer/scheduler state).

Training Configuration

Parameter	Value
Learning Rate	0.0001
Batch Size (train/eval)	1 / 1
Gradient Accumulation	8 (effective batch 16)
Optimizer	AdamW Torch Fused
LR Scheduler	cosine, 50 warmup steps
Epochs	3
Quantization	4-bit NF4
GPUs	2 (multi-GPU)

Final validation loss: 0.1686

Training Loss	Epoch	Step	Validation Loss
0.2397	0.80	100	0.2632
0.1849	0.82	200	0.2169
0.1496	1.22	300	0.1969
0.1470	1.63	400	0.1868
0.1192	2.04	500	0.1732
0.1089	2.45	600	0.1699
0.1128	2.86	700	0.1686

Limitations

OCR quality depends on input image quality/resolution.
Handwritten text may not be reliably extracted.
Complex layouts may not always be represented correctly.
Validation loss does not directly reflect OCR/legal-extraction accuracy.
Output should be reviewed by a qualified human before legal use — this model is an extraction assistant, not a source of legal advice.

Framework Versions

Framework	Version
PEFT	0.17.1
Transformers	4.57.6
PyTorch	2.8.0+cu128

Trained using LlamaFactory.

License

This adapter uses the Gemma license associated with the base model. Review the license and terms of use of google/gemma-3-4b-it before using or distributing this model.

Citation

Ahmed Yasser - Arabic Legal OCR Gemma 3 4B QLoRA https://huggingface.co/ahmedyasser006/arabic-legal-ocr-gemma-3-4b-qlora