You can deploy this model and start earning money today!

PraveenAIML/
align-lite-dpo-qwen-adapter

Name: align-lite-dpo-qwen-adapter
Availability: InStock

Qwen/Qwen2.5-0.5B-Instruct

adapter

other

Views

No views yet

PraveenAIML/align-lite-dpo-qwen-adapter

This repository contains a LoRA adapter for Align-Lite. It is an adapter-only artifact trained with QLoRA + DPO preference alignment, not PPO-RLHF.

Sources

Base model: Qwen/Qwen2.5-0.5B-Instruct at commit 7ae557604adf67be50417f59c2c2f167def9a775
Tokenizer: Qwen/Qwen2.5-0.5B-Instruct at commit 7ae557604adf67be50417f59c2c2f167def9a775
Dataset: Intel/orca_dpo_pairs at commit 624952e3f420ae18d88b31977ee2ea436c833abb
Project code: https://github.com/praveenraj9623-sketch/align-lite-dpo-lab

Method

This adapter uses QLoRA fine-tuning and DPO preference alignment. It is not PPO-RLHF, does not include a separate reward model, and does not merge or publish full base-model weights.

Hyperparameters

LoRA

json

{
  "alpha": 32,
  "dropout": 0.05,
  "rank": 16,
  "target_modules": [
    "q_proj",
    "k_proj",
    "v_proj",
    "o_proj",
    "gate_proj",
    "up_proj",
    "down_proj"
  ]
}

DPO

json

{
  "beta": 0.1,
  "bf16": false,
  "epochs": 1,
  "eval_steps": 100,
  "eval_strategy": "steps",
  "fp16": true,
  "gradient_accumulation_steps": 8,
  "gradient_checkpointing": true,
  "learning_rate": 1e-05,
  "logging_steps": 10,
  "max_length": 1024,
  "per_device_eval_batch_size": 1,
  "per_device_train_batch_size": 1,
  "save_steps": 100,
  "save_strategy": "steps",
  "save_total_limit": 2
}

Generation

json

{
  "do_sample": false,
  "max_new_tokens": 128,
  "temperature": 0.0
}

Adapter Artifact Size

Total adapter artifact size: 17642272 bytes (16.824982 MB)

ADAPTER_EXPORT_METADATA.json: 359 bytes (0.000342 MB)
adapter_config.json: 1105 bytes (0.001054 MB)
adapter_model.safetensors: 17640808 bytes (16.823586 MB)

Limitations

This is a 0.5B-parameter base model with an adapter-only alignment experiment.
The data is public preference data from Intel/orca_dpo_pairs, not feedback collected by this project.
This adapter should be evaluated against the locked held-out split and exact source revisions recorded in adapter_manifest.json.
DPO preference alignment is not PPO-RLHF and should not be described as reward-model training.