A lightweight LoRA fine-tune on Qwen2.5-7B-Instruct that produces brief, clear, formal titles ready for publication from official administrative text while preserving essential meaning and eliminating redundancies.
All 7 linear projection layers in attention and FFN blocks were targeted. The base model was loaded in 4-bit (NF4, double quantization) via bitsandbytes.
Generate titles from official administrative texts published in DOUE, BOE, and BOPA. The model takes a Spanish administrative/legal text as input and outputs a concise, formal title.
The model was trained with a system prompt instructing it to summarize administrative texts into formal titles while preserving essential meaning and eliminating unnecessary details and redundancies.
1from transformers import AutoModelForCausalLM, AutoTokenizer
2from peft import PeftModel
3
4# Load base model in 4-bit
5base_model = AutoModelForCausalLM.from_pretrained(
6 "Qwen/Qwen2.5-7B-Instruct",
7 load_in_4bit=True,
8 bnb_4bit_quant_type="nf4",
9 bnb_4bit_use_double_quant=True,
10 bnb_4bit_compute_dtype="float16",
11 device_map="auto"
12)
13tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
14
15# Load LoRA adapter
16model = PeftModel.from_pretrained(base_model, "diegogs1451/qwen2.5-7B-Instruct-dUO")
17
18messages = [
19 {"role": "system", "content": "Eres un asistente experto en resumir textos oficiales administrativos y jurídicos del DOUE (Diario Oficial de la Unión Europea), del BOE (Boletín Oficial del Estado) y del BOPA (Boletín Oficial del Principado de Asturias) en títulos breves, claros, formales y listos para publicación. Conserva el significado administrativo esencial eliminando detalles innecesarios y redundancias"},
20 {"role": "user", "content": "<input text>"}
21]
22inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
23outputs = model.generate(inputs, max_new_tokens=128)
24print(tokenizer.decode(outputs[0], skip_special_tokens=True))
