You can deploy this model and start earning money today!

Geraldine/
Gemini-Distill-Qwen2.5-0.5B-ead-GGUF

Name: Gemini-Distill-Qwen2.5-0.5B-ead-GGUF
Availability: InStock

Geraldine/Gemini-Distill-Qwen2.5-0.5B-ead

Views

No views yet

Gemini-Distill-Qwen2.5-0.5B-ead GGUF Quantized Versions (Distilled from Gemini-2.0-Flash-Thinking-Exp)

Model Description

This repository contains quantized versions of the fine-tuned Geraldine/Gemini-Distill-Qwen2.5-0.5B-ead model, which was trained via knowledge distillation from Gemini-2.0-Flash-Thinking-Exp. The fine-tuning process teaches the model to reason through and generate Encoded Archival Description (EAD/XML) outputs, ensuring structured reasoning before final archival XML generation.

This repository provides various GGUF quantized formats, allowing efficient inference on different hardware setups, including CPUs and GPUs.

Available GGUF Files

The following quantized versions of the model were generated using llama.cpp:

File Name	Description
`Gemini-Distill-Qwen2.5-0.5B-ead-Q2_K.gguf`	Ultra-low precision (2-bit) for extreme compression
`Gemini-Distill-Qwen2.5-0.5B-ead-Q3_K_M.gguf`	3-bit quantization with mixed precision
`Gemini-Distill-Qwen2.5-0.5B-ead-Q4_K_M.gguf`	4-bit quantization with mixed precision
`Gemini-Distill-Qwen2.5-0.5B-ead-Q5_K_M.gguf`	5-bit quantization with mixed precision
`Gemini-Distill-Qwen2.5-0.5B-ead-Q6_K.gguf`	6-bit quantization
`Gemini-Distill-Qwen2.5-0.5B-ead-Q8_0.gguf`	8-bit quantization for balance between speed and accuracy
`Gemini-Distill-Qwen2.5-0.5B-ead-fp16.gguf`	16-bit floating point (fp16) version
`Gemini-Distill-Qwen2.5-0.5B-ead-fp32.gguf`	Full precision (fp32) version

How to Use the Quantized Model

Running the Model with llama.cpp

To run the model using llama.cpp, use the following command:

./main -m Gemini-Distill-Qwen2.5-0.5B-ead-Q4_K_M.gguf -p "Convert the following archival information into EAD/XML: ..."

For optimal performance, ensure you select the right quantized version based on your hardware capabilities.

Running the Model with GPT4All

If using GPT4All, load the GGUF model with:

python

from gpt4all import GPT4All

model_path = "Gemini-Distill-Qwen2.5-0.5B-ead-Q4_K_M.gguf"
model = GPT4All(model_path)
response = model.generate("Convert the following archival information into EAD/XML:")
print(response)

Running the Model with Ollama

If using Ollama, load the GGUF model with:

ollama run hf.co/Geraldine/Gemini-Distill-Qwen2.5-0.5B-ead-GGUF:Q8_0

python

import requests
import json

url = "http://localhost:11434/v1/chat/completions"

payload = json.dumps({
  "model": "hf.co/Geraldine/Gemini-Distill-Qwen2.5-0.5B-ead-GGUF:Q8_0",
  "messages": [
    {
      "role": "system",
      "content": "You are an archivist expert in EAD/XML format for archival records metadata."
    },
    {
      "role": "user",
      "content": "Give me an example of <controlaccess> content."
    }
  ],
  "option": {
    "num_ctx": 4096,
    "temperature": 0.1
  },
  "stream": False
})
headers = {
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)

Choosing the Right Quantization Format

Lower-bit models (Q2_K, Q3_K_M, Q4_K_M): Best for low-memory devices, but may lose some accuracy.
Mid-range (Q5_K_M, Q6_K): Good trade-off between speed and precision.
Higher precision (Q8_0, fp16, fp32): Best for accuracy but requires more memory.

For CPU inference, Q4_K_M or Q5_K_M is recommended for a balance between efficiency and performance.

Limitations & Future Improvements

Inference Speed: Ensure Sliding Window Attention (SWA) is disabled, as it may slow down inference.
- To disable: model.config.sliding_window = None
Future Work:
- Further optimizations for CPU inference
- Additional fine-tuning on larger datasets
- Exploring LoRA/QLoRA for low-rank adaptation

Citation & Acknowledgments

If you use this model in research or production, please cite:

@misc{your-citation,
  author = {Géraldine Geoffroy},
  title = {Gemini-Distill-Qwen2.5-0.5B-ead GGUF Quantized Versions},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/Geraldine/Gemini-Distill-Qwen2.5-0.5B-ead-GGUF}
}