Quantization	File	Size
Q4_K_M	Hopcoder-Mini-9B-SWE-Agent-Q4_K_M.gguf	5.63 GB
Q5_K_M	Hopcoder-Mini-9B-SWE-Agent-Q5_K_M.gguf	6.47 GB
Q8_0	Hopcoder-Mini-9B-SWE-Agent-Q8_0.gguf	9.53 GB
F16	Hopcoder-Mini-9B-SWE-Agent-F16.gguf	17.92 GB

Usage

bash

# Download a quantized file
huggingface-cli download TaimoorSiddiqui/Hopcoder-Mini-9B-SWE-Agent-GGUF Hopcoder-Mini-9B-SWE-Agent-Q4_K_M.gguf --local-dir ./

# Run with llama.cpp
llama-server --model Hopcoder-Mini-9B-SWE-Agent-Q4_K_M.gguf --n-gpu-layers 99 --port 8080

Merging Process

Loaded base model in FP16 on CPU
Merged LoRA adapter(s) into base (Stage-2 SWE-Agent (already contains Stage-1))
Converted merged model to GGUF format
Quantized to multiple levels using llama-quantize
Uploaded to HuggingFace

Original Model

The original PEFT/LoRA adapter is available at:

Stage-2 (contains Stage-1): TaimoorSiddiqui/Hopcoder-Mini-9B-SWE-Agent-LoRA-H200