TSFT-RAG Gemma-3-12B-IT
Task-Specific Full Fine-Tuning for Retrieval-Augmented Generation
Part of the
TSFT-RAG model family for Retrieval-Augmented Generation across Gemma, Qwen and Llama architectures.
This repository contains TSFT-RAG Gemma-3-12B-IT, a full-parameter fine-tuned derivative of Google's Gemma-3-12B-IT developed within the TSFT-RAG (Task-Specific Full Fine-Tuning for Retrieval-Augmented Generation) research project.
The objective is to investigate how supervised full fine-tuning changes the behaviour of modern language models when used as Retrieval-Augmented Generation (RAG) systems.
TSFT-RAG Model Series
Abstract
General-purpose instruction models are optimized for broad conversational abilities but not specifically for Retrieval-Augmented Generation.
The TSFT-RAG project investigates whether full supervised fine-tuning can specialize foundation models for:
- grounded question answering
- unsupported question detection
- structured information extraction
- citation generation
- reliable JSON generation
without changing the underlying transformer architecture.
The 12B model represents the largest Gemma model currently released within the TSFT-RAG project.
Research Background
Reliable Retrieval-Augmented Generation requires considerably more than fluent language generation.
An effective RAG model should
- answer only from retrieved evidence,
- reject unsupported questions,
- preserve factual grounding,
- generate machine-readable structured outputs,
- remain deterministic across repeated executions.
The TSFT-RAG project evaluates these properties using identical datasets, prompts and evaluation procedures across Gemma, Qwen and Llama model families.
Model Details
| Property | Value |
|---|
| Base model | google/gemma-3-12b-it |
| Training implementation | unsloth/gemma-3-12b-it |
| Architecture | Gemma3ForConditionalGeneration |
| Parameters | ~12 Billion |
| Hidden Size | 3840 |
| Transformer Layers | 48 |
| Attention Heads | 16 |
| Context Window | 131,072 tokens |
| Training Length | 1,024 tokens |
| Precision | bfloat16 |
| Fine-tuning | Full Parameter |
Training Dataset
The TSFT-RAG benchmark contains supervised examples covering
- Grounded Question Answering
- Unsupported Question Detection
- Topic Extraction
- Key Message Extraction
- Argument Extraction
- Structured JSON Generation
The identical benchmark is used for every TSFT-RAG model to enable direct scientific comparison.
Evaluation Results
| Metric | Base | TSFT-RAG | Improvement |
|---|
| Aggregate Score | 0.450 | 0.686 | +0.236 |
| Hard Negative QA | 0.162 | 0.829 | +0.667 |
| Grounded QA | 0.613 | 0.594 | -0.019 |
| Topic Extraction | 0.651 | 0.632 | -0.019 |
| Key Message | 0.605 | 0.580 | -0.026 |
| Main Arguments | 0.589 | 0.614 | +0.025 |
| JSON Validity | 0.872 | 0.987 | +0.115 |
| Citation Precision | 0.000 | 0.950 | +0.950 |
| Citation Recall | 0.000 | 0.950 | +0.950 |
Discussion
Among the currently released Gemma models, the 12B variant achieves the strongest overall performance.
The most notable improvements are observed for
- unsupported-question detection,
- structured JSON generation,
- citation accuracy,
- argument extraction.
Unlike many task-specific fine-tuning approaches, the model preserves strong grounded reasoning while substantially improving rejection behaviour and structured output quality.
These results demonstrate that full-parameter fine-tuning can effectively specialize large language models for Retrieval-Augmented Generation without architectural modifications.
Intended Use
Recommended for
- Retrieval-Augmented Generation
- Enterprise Knowledge Assistants
- Scientific Information Systems
- University Information Systems
- Document Question Answering
- Structured Information Extraction
Not intended as a general-purpose conversational assistant.
Example Usage
1from transformers import AutoTokenizer, AutoModelForCausalLM
2
3tokenizer = AutoTokenizer.from_pretrained(
4 "sascha-frank-ai-research/tsft-rag-gemma-3-12b-it"
5)
6
7model = AutoModelForCausalLM.from_pretrained(
8 "sascha-frank-ai-research/tsft-rag-gemma-3-12b-it"
9)
Reproducibility
Training was performed using
- Hugging Face Transformers
- TRL
- Unsloth
All relevant configuration files, tokenizer files, evaluation summaries and licensing information are included in this repository.
Limitations
- Optimized specifically for Retrieval-Augmented Generation.
- General open-domain conversation was not the primary optimization objective.
- Evaluated primarily on German-language benchmark datasets.
- No multimodal benchmark has yet been conducted.
Ethical Considerations
The model should always be used together with appropriate source attribution and human oversight.
It should not replace expert judgement in legal, medical or other safety-critical domains.
License
This repository contains a modified derivative of Google's Gemma model.
Please consult
- LICENSE
- NOTICE
- MODIFICATIONS.md
for detailed licensing information.
Associated Publication
This model was developed and evaluated as part of the following research:
Frank, S., & Singh, R. (2026).
Task-Specific Full Fine-Tuning for Retrieval-Augmented Generation: A Multi-Family Evaluation Across Open-Weight Language Models.
If you use this model in academic work, please cite the publication above.
1@misc{Frank2026TSFTRAG,
2 author = {Frank, Sascha and Singh, Rawel},
3 title = {Task-Specific Full Fine-Tuning for Retrieval-Augmented Generation:
4 A Multi-Family Evaluation Across Open-Weight Language Models},
5 year = {2026},
6 month = jul,
7 version = {v1},
8 publisher = {Zenodo},
9 doi = {10.5281/zenodo.21638352},
10 url = {https://doi.org/10.5281/zenodo.21638352},
11 note = {Preprint}
12}
Project
The complete TSFT-RAG research project, including training scripts,
evaluation pipeline, benchmark methodology and documentation, is available at:
Author
Sascha Frank
Independent AI Researcher
