You can deploy this model and start earning money today!

hamza-abbasi-aai/
aai-eurlex-finetuned-001

Name: aai-eurlex-finetuned-001
Availability: InStock

Qwen/Qwen2.5-3B-Instruct

quantized

endpoints_compatible

conversational

Views

No views yet

aai-eurlex-finetuned-001 (GGUF)

GGUF release of Qwen2.5-3B-Instruct fine-tuned on EU regulation Q&A (Eurlex) for CORA (Compliance and Regulatory Assistant).
Trained with full fine-tuning (no LoRA); see the experiment repo for training details.

Model files

File	Description
`aai-eurlex-finetuned-001-f16.gguf`	F16 GGUF for inference
`Modelfile`	Ollama Modelfile for notebook-matching inference (see below)

Inference parity with the notebook

The fine-tuning notebook uses the Qwen chat template with this system prompt:

You are Qwen, created by Alibaba Cloud. You are a helpful assistant.

To get the same behavior as the notebook when running with Ollama, use the included Modelfile so that this system prompt is applied.

Ollama (recommended — matches notebook)

Pull the model and create with the Modelfile (download Modelfile from this repo first):

bash
1# Optional: if repo is private, set your HF token 2# export HUGGING_FACE_HUB_TOKEN=your_token 3 4ollama run hf.co/hamza-abbasi-aai/aai-eurlex-finetuned-001

To use the exact same system prompt as the notebook, create a custom model from the Modelfile:

bash
1# Download Modelfile from this repo, then: 2ollama create aai-eurlex -f Modelfile 3ollama run aai-eurlex

The Modelfile sets SYSTEM "You are Qwen, created by Alibaba Cloud. You are a helpful assistant." so responses (e.g. to "Hi" or regulation questions) match the notebook.
Without the Modelfile, ollama run hf.co/hamza-abbasi-aai/aai-eurlex-finetuned-001 uses no (or a different) system prompt, so outputs may differ (e.g. generic or multilingual replies instead of CORA-style).

llama.cpp

Download the GGUF, then run with the same system prompt for parity:

bash

./llama-cli -m aai-eurlex-finetuned-001-f16.gguf -c 2048 --chat-template chatml \
  -s "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."

Training

Base: Qwen/Qwen2.5-3B-Instruct
Method: Full fine-tuning (SFT), no LoRA
Data: Eurlex-based instruction data (document Q&A, greetings, out-of-scope examples)
Framework: TRL / Transformers

Citation (TRL)

bibtex

@misc{vonwerra2022trl,
  title        = {{TRL: Transformer Reinforcement Learning}},
  author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
  year         = 2020,
  journal      = {GitHub repository},
  publisher    = {GitHub},
  howpublished = {\url{https://github.com/huggingface/trl}}
}

hamza-abbasi-aai/aai-eurlex-finetuned-001