Author: Malek Camilo
Institution: Facultad de Informática, Universidad Nacional de La Plata
Thesis: Estrategias de adaptación eficiente para modelos de lenguaje abiertos
Repository: github.com/malekcamilo/tesis_maestria

This repository contains a QDoRA adapter for document type classification in the Computer Science subset of SEDICI records.

Base model

[!WARNING] Base Model License: This adapter is built on top of meta-llama/Meta-Llama-3.1-8B-Instruct, which is a gated model. Users must explicitly agree to the Meta Llama 3.1 Community License on Hugging Face to use it.

The adapter was trained on top of meta-llama/Meta-Llama-3.1-8B-Instruct.

Users must have access to the base model in order to load this adapter.

Task

Given a title and abstract, the model predicts one of the following document types:

Article
Conference Paper
Thesis
Book
Learning Resource
Other

The generated label is returned in Spanish, matching the labels used during training.

Training data

The dataset contains 19,974 records from SEDICI in the Computer Science domain. Each instance includes title, abstract and document type.

Tokenizer

This adapter uses the original tokenizer distributed with Meta-Llama-3.1-8B-Instruct. No additional tokens or vocabulary modifications were introduced during fine-tuning.

Method

The model was adapted using QDoRA (rank = 128) with NF4 4-bit quantization and LoRA adapters applied to all attention and MLP projection layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj).

Hardware

Training was performed on an Intel® Data Center GPU Max 1550 using Intel Extension for PyTorch (IPEX) with bfloat16 precision.

Training Configuration

Parameter	Value
Epochs	5
Learning Rate	2e-4
Batch Size	8
Gradient Accumulation	1
Max Sequence Length	512
Rank	128
Alpha	256
Dropout	0.1

Training Metrics

Train Time: 9.48 hours
Peak Memory: 14.14 GB
Trainable Parameters: 336920576

Evaluation

Metric	Value
Accuracy	0.8625
Macro-F1	0.6582
Exact Match	0.8589

Per-Class Results

Category	Precision	Recall	F1-Score
Objeto De Conferencia	0.8945	0.9518	0.9222
Objeto De Aprendizaje	0.8696	0.7143	0.7843
Articulo	0.7089	0.4266	0.5326
Tesis	0.7312	0.6570	0.6921
Libro	0.9524	0.5882	0.7273
Otro	1.0000	0.9032	0.9492

Prompt Format

The adapter was trained using the Meta-Llama 3.1 chat template with the following system instruction:

text

System:
Eres un clasificador automático de documentos académicos. Tu única tarea es asignar la categoría correcta a los registros bibliográficos provistos.
Regla estricta: Debes responder ÚNICAMENTE con el nombre exacto de la categoría. No incluyas explicaciones, puntuación adicional ni texto conversacional.
Categorías válidas: Articulo, Objeto de conferencia, Tesis, Libro, Otro, Objeto de aprendizaje.

Usage

python

import torch

from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
)

from peft import PeftModel

base_model = "meta-llama/Meta-Llama-3.1-8B-Instruct"
adapter = "malekcamilo/sedici-llama-qdora-r128-all"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(adapter)

model = AutoModelForCausalLM.from_pretrained(
    base_model,
    device_map="auto",
    quantization_config=bnb_config,
)

model = PeftModel.from_pretrained(model, adapter)
model.eval()


def classify(title: str, abstract: str) -> str:

    messages = [
        {
            "role": "system",
            "content": (
                "Eres un clasificador automático de documentos académicos. "
                "Tu única tarea es asignar la categoría correcta a los registros bibliográficos provistos.\n"
                "Regla estricta: Debes responder ÚNICAMENTE con el nombre exacto de la categoría. "
                "No incluyas explicaciones, puntuación adicional ni texto conversacional.\n"
                "Categorías válidas: Articulo, Objeto de conferencia, Tesis, Libro, Otro, Objeto de aprendizaje."
            ),
        },
        {
            "role": "user",
            "content": (
                f"Clasifica el siguiente registro:\n"
                f"<titulo>{title}</titulo>\n"
                f"<resumen>{abstract}</resumen>"
            ),
        },
    ]

    inputs = tokenizer.apply_chat_template(
        messages,
        tokenize=True,
        add_generation_prompt=True,
        return_tensors="pt",
    )

    inputs = {k: v.to(model.device) for k, v in inputs.items()}

    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=10,
            do_sample=False,
        )

    generated = outputs[0][inputs["input_ids"].shape[-1]:]

    return tokenizer.decode(
        generated,
        skip_special_tokens=True,
    ).strip()


prediction = classify(
    "Clasificación automática de documentos científicos",
    "En este trabajo se propone un método para clasificar documentos utilizando grandes modelos de lenguaje."
)

print(prediction)

Reproducibility

Base model: Meta-Llama-3.1-8B-Instruct
Fine-tuning: QDoRA
Rank: 128
Seed: 42
Dataset size: 19,974 documents
Sequence length: 512
Precision: bfloat16
Frameworks:
- Transformers
- PEFT
- TRL
- Intel Extension for PyTorch (IPEX)

Limitations

This adapter was trained exclusively on Computer Science records from SEDICI. Performance on other academic repositories, languages or taxonomies has not been evaluated. The model is intended for research purposes.

Citation

If you use this adapter in your research, please cite the associated Master's thesis:

bibtex

@mastersthesis{camilo2026peft,
  author = {Malek Camilo},
  title  = {Estrategias de adaptación eficiente para modelos de lenguaje abiertos},
  school = {Facultad de Informática, Universidad Nacional de La Plata},
  year   = {2026},
  type   = {Master's thesis},
  note   = {Manuscript submitted in partial fulfillment of the requirements for the Master's degree},
}