You can deploy this model and start earning money today!

Geraldine/
qwen3-0.6B-unimarc-grpo-GGUF

Name: qwen3-0.6B-unimarc-grpo-GGUF
Availability: InStock

Geraldine/qwen3-0.6B-unimarc-grpo

Views

No views yet

qwen3-0.6B-unimarc-grpo GGUF Quantized Versions

Model Description

This repository contains quantized versions of the fine-tuned Geraldine/qwen3-0.6B-unimarc-grpo model, which using GRPO (Generalized Repetition Penalized Optimization) and LoRA adapters to transform raw bibliographic metadata into structured UNIMARC XML records.

This repository provides various GGUF quantized formats, allowing efficient inference on different hardware setups, including CPUs and GPUs.

Available GGUF Files

The following quantized versions of the model were generated using llama.cpp:

File Name	Description
`qwen3-0.6B-unimarc-grpo-Q2_K.gguf`	Ultra-low precision (2-bit) for extreme compression
`qwen3-0.6B-unimarc-grpo-Q3_K_M.gguf`	3-bit quantization with mixed precision
`qwen3-0.6B-unimarc-grpo-Q4_K_M.gguf`	4-bit quantization with mixed precision
`qwen3-0.6B-unimarc-grpo-Q5_K_M.gguf`	5-bit quantization with mixed precision
`qwen3-0.6B-unimarc-grpo-Q6_K.gguf`	6-bit quantization
`qwen3-0.6B-unimarc-grpo-Q8_0.gguf`	8-bit quantization for balance between speed and accuracy
`qwen3-0.6B-unimarc-grpo-fp16.gguf`	16-bit floating point (fp16) version

How to Use the Quantized Model

Prompts

See Geraldine/qwen3-0.6B-unimarc-grpo to follow the recommended prompting template.

Running the Model with llama.cpp

To run the model using llama.cpp, use the following command:

./main -m qwen3-0.6B-unimarc-grpo-Q4_K_M.gguf -p "Convert the following bibliographic raw data into Unimarc/XML record: ..."

For optimal performance, ensure you select the right quantized version based on your hardware capabilities.

Running the Model with GPT4All

If using GPT4All, load the GGUF model with:

python

from gpt4all import GPT4All

model_path = "qwen3-0.6B-unimarc-grpo-Q4_K_M.gguf"
model = GPT4All(model_path)
response = model.generate("Convert the following bibliographic raw data into Unimarc/XML record:")
print(response)

Running the Model with Ollama

If using Ollama, load the GGUF model with:

ollama run hf.co/Geraldine/qwen3-0.6B-unimarc-grpo-GGUF:Q8_0

python

import requests
import json

url = "http://localhost:11434/v1/chat/completions"

payload = json.dumps({
  "model": "hf.co/Geraldine/qwen3-0.6B-unimarc-grpo-GGUF:Q8_0",
  "messages": [
    {
      "role": "system",
      "content": system_prompt
    },
    {
      "role": "user",
      "content": "Title: ...\nAuthors: ..."
    }
  ],
  "option": {
    "num_ctx": 4096,
    "temperature": 0.6,
    "top_p": 0.95,
    "top_k": 20,
    "min_p": 0
  },
  "stream": False
})
headers = {
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

Choosing the Right Quantization Format

Lower-bit models (Q2_K, Q3_K_M, Q4_K_M): Best for low-memory devices, but may lose some accuracy.
Mid-range (Q5_K_M, Q6_K): Good trade-off between speed and precision.
Higher precision (Q8_0, fp16, fp32): Best for accuracy but requires more memory.

For CPU inference, Q4_K_M or Q5_K_M is recommended for a balance between efficiency and performance.

Limitations & Future Improvements

Limitations: Because of prompt templating during RL training, inference need to be optimized with the same prompt as during training
Future Work:
- Further optimizations for CPU inference
- Additional fine-tuning on larger datasets

Citation & Acknowledgments

If you use this model in research or production, please cite:

@misc{your-citation,
  author = {Géraldine Geoffroy},
  title = {qwen3-0.6B-unimarc-grpo GGUF Quantized Versions},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/Geraldine/qwen3-0.6B-unimarc-grpo-GGUF}
}