A 3B fine-tuned model that outperforms the Llama 3 8B zero-shot baseline on U.S. immigration law Q&A (+27% mean score, 4x more fully-correct answers).

Fine-tuned from meta-llama/Llama-3.2-3B-Instruct on the nshportun/usa-immigration-law-qa dataset --- 17,058 source-grounded Q&A pairs covering all major U.S. immigration subdomains.

Benchmark Results

Evaluated on 101 held-out questions scored 0-3 by Claude Sonnet 4.6 as judge:

Model	Mean Score (0-3)	% Fully Correct (3)
Claude Sonnet 4.6 (zero-shot)	1.515	24.8%
Llama 3.2 3B fine-tuned (this model)	1.079	16.8%
Llama 3 8B zero-shot	0.851	4.0%

Domain-specific fine-tuning at 3B scale delivers +27% higher mean score and 4x more fully-correct answers compared to a larger general-purpose 8B model.

Training Details

Setting	Value
LoRA rank (r)	32
LoRA alpha	64
Target modules	q_proj, v_proj, k_proj, o_proj
LoRA dropout	0.05
Epochs	2
Learning rate	5e-5
Batch size	2
Max sequence length	1024
Training pairs	16,065
Infrastructure	ml.g5.2xlarge (AWS SageMaker)

Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "nshportun/usa-immigration-llama-3.2-3b-v3"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")

messages = [
    {"role": "system", "content": "You are an expert on U.S. immigration law and policy. Answer accurately based on USCIS, 8 CFR, and BIA sources."},
    {"role": "user", "content": "What is the filing fee for Form I-485?"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=300, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Dataset

The model was trained on nshportun/usa-immigration-law-qa, a dataset of 17,058 source-grounded Q&A pairs from official U.S. immigration sources (USCIS Policy Manual, 8 CFR/INA, BIA Precedent Decisions, USCIS Forms, DHS/CBP Statistics).

Disclaimer

For research and educational purposes only. Not legal advice. Always consult a licensed immigration attorney.