lab22-dpo-adapter-gguf – AI Model by Wan1302 | AlphaNeural AI

You can deploy this model and start earning money today!

Wan1302/
lab22-dpo-adapter-gguf

like

0

argilla/ultrafeedback-binarized-preferences-cleaned

5CD-AI/Vietnamese-alpaca-cleaned

unsloth/Qwen2.5-3B-bnb-4bit

endpoints_compatible

Views

No views yet

Lab 22 DPO — merged + GGUF release (lab22-dpo-adapter-gguf)

Merged FP16 weights + GGUF quantizations of an SFT+DPO Qwen2.5 model trained for the VinUni AICB Day 22 alignment lab.

Pipeline

SFT-mini: 1k VN Alpaca · 1 epoch · LoRA r=16 on unsloth/Qwen2.5-3B-bnb-4bit
DPO: 2k UltraFeedback · β=0.1 · lr=5e-07
Merge SFT + DPO LoRA into base, save as FP16
Quantize via llama.cpp

Available files

Quant	File	Size (MB)
Q4_K_M	`merged-fp16.Q4_K_M.gguf`	1929.9

model-*.safetensors etc. — merged FP16 weights (vLLM / transformers)

DPO training summary

Metric	Value
Final training loss	0.8085754909515381
End chosen reward	-0.872550094127655
End rejected reward	-0.9477240920066834
End reward gap	0.07517399787902834

Usage — llama-cpp-python (CPU/Metal/CUDA)

python

from llama_cpp import Llama
llm = Llama(model_path="lab22-dpo-Q4_K_M.gguf", n_ctx=512, n_gpu_layers=-1)
out = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Giải thích quicksort 3 câu."}],
    max_tokens=200, temperature=0.0,
)
print(out["choices"][0]["message"]["content"])

Usage — vLLM (BigGPU only)

vllm serve Wan1302/lab22-dpo-adapter-gguf --port 8000 --max-model-len 512

License & limitations

Apache-2.0 (Qwen2.5 base).
Experimental research model. Trained on English UltraFeedback; Vietnamese helpfulness/safety tested on 8 prompts (lab NB4).
Not production-ready. Refusals on safety-critical prompts have not been exhaustively red-teamed.

Citation

VinUni AICB program · Track 3 Day 22 · A20 cohort 2026.