You can deploy this model and start earning money today!

RobbedoesHF/
geitje-ultra-dutch-definition-expansion-qlora

Name: geitje-ultra-dutch-definition-expansion-qlora
Availability: InStock

RobbedoesHF/dutch-definitions

BramVanroy/GEITje-7B-ultra

adapter

apache-2.0

Views

No views yet

QLoRA Adapter for Dutch Definition Expansion (GEITje-7B-ultra)

This repository contains a QLoRA adapter fine-tuned on BramVanroy/GEITje-7B-ultra for the task of sense-preserving definitional expansion in Dutch.

This work was developed as part of the Master's thesis, "Transformer-based Expansion of Dutch Dictionary Definitions", submitted for the degree of Master of Science in Artificial Intelligence at KU Leuven.

About the Thesis

The research investigates the potential of transformer-based models to automate a significant bottleneck in contemporary lexicography: the manual expansion of concise, core-meaning definitions into comprehensive, formally structured dictionary entries. The study focuses on Dutch, a task requiring not only semantic accuracy but also strict adherence to lexicographical style and structure.

The thesis empirically compares two primary methodologies: in-context learning via few-shot prompting and adaptation via parameter-efficient fine-tuning (specifically, QLoRA). This comparison was conducted across a range of powerful multilingual and Dutch-specific models, including mT5-xl, GEITje Ultra, Aya-101, and Aya-23, to determine the most effective strategy for this high-precision domain.

This Model's Role and Performance

This fine-tuned GEITje-7B-ultra model was a key subject in the study for testing the domain mismatch hypothesis. As a model heavily aligned for conversational interaction its performance was analysed to see if this stylistic prior would conflict with the formal structured nature of lexicographical text.

After fine-tuning GEITje Ultra emerged as a top-performing model distinguished by its strong performance on lexical overlap metrics suggesting it was highly effective at learning the precise phrasing of the training data. While the fine-tuned Aya-23 model achieved slightly higher scores on semantic quality metrics, GEITje Ultra proved to be a highly competitive model demonstrating that task-specific fine-tuning can successfully adapt a conversationally-aligned model to a formal domain.

How to Use

To use this adapter you must first load the base model (`BramVanroy/GEITje-7B-ultra') in 4-bit and then apply this adapter on top of it.

python

import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

base_model_id = "BramVanroy/GEITje-7B-ultra"
adapter_id = "RobbedoesHF/geitje-7b-ultra-dutch-definition-expansion-qlora" # The repo ID of this adapter

# Load the base model with 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForSeq2SeqLM.from_pretrained(
    base_model_id,
    quantization_config=bnb_config,
    device_map="auto",
    attn_implementation="flash_attention_2", # Recommended for GEITje
)
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
tokenizer.pad_token = tokenizer.eos_token # Set pad token

# Apply the LoRA adapter
model = PeftModel.from_pretrained(model, adapter_id)

print("Model loaded successfully!")

Prompting Format

This adapter was fine-tuned on a specific instructional prompt. For best results, your input should match this structure.