You can deploy this model and start earning money today!

CherryScallion/
CyberGong-Qwen2.5-1.5B-1.0

Name: CyberGong-Qwen2.5-1.5B-1.0
Availability: InStock

safetensors

mit

Views

No views yet

Personal Style Language Model Inference and Management System

A local-runnable personal style language model inference project based on Qwen2.5-1.5B and LoRA adapter.

Objectives

✅ Support loading base model from local directory
✅ Support 4-bit quantization loading (RTX 3060 6GB optimized)
✅ Support LoRA adapter mounting and inference
✅ Support model merging and independent export
✅ Support interactive chat and single-turn inference
✅ Fully usable in offline mode

Project Structure

CyberGong/
├── configs/
│   └── infer.yaml              # Inference configuration file
├── models/
│   ├── Qwen2.5-1.5B/           # Base model directory
│   ├── your-persona-lora/      # LoRA adapter directory
│   └── merged-persona-model/   # Merged model
├── src/
│   ├── download_model.py       # Model download script
│   ├── load_model.py           # Model loading script
│   ├── chat.py                 # Chat/inference script
│   ├── merge_lora.py           # LoRA merging script
│   └── utils.py                # Utility functions
├── data/
│   └── prompts.jsonl           # Test prompts
└── README.md                   # This file

Install Dependencies

pip install transformers peft bitsandbytes torch huggingface-hub pyyaml

Key Libraries Description

transformers: Provides model loading and generation interfaces
peft: Provides LoRA adapter support
bitsandbytes: Provides 4-bit quantization support
torch: PyTorch deep learning framework
huggingface-hub: HuggingFace model download tool

Quick Start

1. Download Model

bash

python src/download_model.py \
    --model-id Qwen/Qwen2.5-1.5B \
    --output-dir ./models/Qwen2.5-1.5B

Options:

--model-id: HuggingFace model ID (default: Qwen/Qwen2.5-1.5B)
--output-dir: Local output directory
--cache-dir: Optional cache directory
--max-workers: Concurrent download workers (default: 4)
--use-cli: Use huggingface-cli to download

2. Load and Inference (Base Model Only)

python src/chat.py --model-dir ./models/Qwen2.5-1.5B

Options:

--model-dir: Model directory
--no-4bit: Disable 4-bit quantization
--device-map: Device allocation (default: auto)

3. Load and Inference (Base + LoRA Adapter)

First prepare LoRA adapter directory, which should contain:

adapter_config.json
adapter_model.safetensors

bash

python src/chat.py \
    --model-dir ./models/Qwen2.5-1.5B \
    --adapter ./models/your-persona-lora

4. Merge Models

Merge adapter with base model into a standalone model:

bash

python src/merge_lora.py \
    --model-dir ./models/Qwen2.5-1.5B \
    --adapter-dir ./models/your-persona-lora \
    --output-dir ./models/merged-persona-model

After merging, it can be directly loaded and used:

python src/chat.py --model-dir ./models/merged-persona-model

Usage

Interactive Chat

python src/chat.py --model-dir ./models/Qwen2.5-1.5B

Commands:

Type any text to get a reply
exit: Exit chat
clear: Clear chat history
save: Save conversation to conversation.json

Single-turn Inference

bash

python src/chat.py \
    --model-dir ./models/Qwen2.5-1.5B \
    --prompt "Hello, please introduce yourself"

Using Configuration File

Edit configs/infer.yaml to configure parameters:

yaml

model:
  base_model_dir: ./models/Qwen2.5-1.5B
  adapter_dir: ./models/your-persona-lora

generation:
  max_new_tokens: 256
  temperature: 0.7
  top_p: 0.9

chat:
  system_prompt: "You are a helpful assistant."
  max_history_length: 10

Then run:

python src/chat.py --config ./configs/infer.yaml

Core Features Explanation

Model Loading (`load_model.py`)

Load Base Model:

python

from load_model import load_base_model

model, tokenizer = load_base_model(
    model_dir="./models/Qwen2.5-1.5B",
    use_4bit=True,
    device_map="auto"
)

Load LoRA Adapter:

python

from load_model import load_persona_adapter

model_with_adapter = load_persona_adapter(
    base_model=model,
    adapter_dir="./models/your-persona-lora"
)

Chat Management (`chat.py`)

python

from chat import PersonaChat
from load_model import load_base_model

model, tokenizer = load_base_model("./models/Qwen2.5-1.5B")
chat = PersonaChat(model, tokenizer)

# Single-turn conversation
response = chat.generate("Hello")

# Get history
history = chat.get_history()

# Save conversation
chat.save_conversation("output.json")

Model Merging (`merge_lora.py`)

python

from merge_lora import merge_and_unload

success = merge_and_unload(
    base_model_dir="./models/Qwen2.5-1.5B",
    adapter_dir="./models/your-persona-lora",
    output_dir="./models/merged-persona-model"
)

Configuration Parameters Guide

Quantization Configuration

yaml

quantization:
  use_4bit: true              # Enable 4-bit quantization
  compute_dtype: bfloat16     # Compute precision
  quant_type: nf4             # Quantization type

4-bit Quantization Benefits:

VRAM usage reduced from 3GB to 1-2GB
Inference speed essentially lossless
Supports RTX 3060 6GB

Generation Parameters

yaml

generation:
  max_new_tokens: 256         # Maximum tokens to generate
  temperature: 0.7            # Temperature (higher = more random)
  top_p: 0.9                  # Top-P sampling
  top_k: 50                   # Top-K sampling
  repetition_penalty: 1.1     # Repetition penalty
  do_sample: true             # Whether to use sampling
  num_beams: 1                # Beam search (1 = disabled)

Troubleshooting

Out of VRAM

Problem: torch.cuda.OutOfMemoryError

Solution:

Enable 4-bit quantization: Don't use --no-4bit
Reduce max_new_tokens
Reduce max_history_length

Missing Model Files

Problem: FileNotFoundError: Model directory does not exist

Solution:

Make sure model is downloaded: python src/download_model.py
Check if path is correct
Check log file model_inference.log

Adapter Loading Failed

Problem: Invalid adapter directory

Solution:

Check if adapter_config.json exists
Check if adapter_model.safetensors exists
Check if adapter matches base model

Performance Metrics (RTX 3060 6GB)

Configuration	VRAM Usage	Inference Speed	Notes
Base (no quantization)	~5.5GB	Fast	May run out of VRAM
Base (4-bit)	~2GB	Fast	Recommended
Base + Adapter	+0.5GB	Slightly slower	Depends on adapter size
Merged model	Same as base	Same as base	Same as base

Logging

All operations are logged to model_inference.log:

2024-01-15 10:30:45,123 - __main__ - INFO - Start loading base model
2024-01-15 10:30:46,456 - load_model - INFO - ✓ Tokenizer loaded successfully
...

Offline Usage

All operations support offline mode, provided model files are completely downloaded:

Model fully downloaded via download_model.py
Use device_map="cpu" or "auto" instead of "cuda:0"
Transformers will automatically use local cache

Recommended Workflow

Development Environment

bash

# 1. Download model (first time, requires network)
python src/download_model.py

# 2. Test base model
python src/chat.py --prompt "test"

# 3. Develop adapter
# Use external tools to train adapter...

# 4. Test adapter
python src/chat.py --adapter ./models/your-persona-lora

# 5. Merge model
python src/merge_lora.py

Production Environment

bash

# Directly load merged model
python src/chat.py --model-dir ./models/merged-persona-model

Extension Suggestions

Add Gradio Interface

bash

pip install gradio

# Create src/gradio_app.py

Add API Service

bash

pip install fastapi uvicorn

# Create src/api_server.py

Model Quantization Optimization

Use GPTQ or AWQ quantization
Use int8 or int4 precision
Model pruning

FAQ

Q: Can I use other models?
A: Yes, modify hf_model_id in config.yaml. Just ensure it's compatible with AutoModelForCausalLM.

Q: How to support multiple adapters?
A: PeftModel can load multiple adapters simultaneously, see peft official documentation.

Q: Can I continue loading adapters after merging?
A: The merged model is a complete standalone model and can be used as a new base model to load other adapters.

Q: How to optimize inference speed?
A:

Use num_beams=1 (disable beam search)
Use smaller max_new_tokens
Enable 4-bit quantization
Use device_map="auto" for automatic allocation

References

License

MIT License

Changelog

v1.0.0 (2024-01-15)

✅ Initial release
✅ Support 4-bit quantization
✅ Support LoRA adapter
✅ Support model merging
✅ Support interactive chat
✅ Complete error handling and logging

Feedback and suggestions are welcome!

CherryScallion/CyberGong-Qwen2.5-1.5B-1.0

Personal Style Language Model Inference and Management System

Objectives

Project Structure

Install Dependencies

Key Libraries Description

Quick Start

1. Download Model

2. Load and Inference (Base Model Only)

3. Load and Inference (Base + LoRA Adapter)

4. Merge Models

Usage

Interactive Chat

Single-turn Inference

Using Configuration File

Core Features Explanation

Model Loading (load_model.py)

Chat Management (chat.py)

Model Merging (merge_lora.py)

Configuration Parameters Guide

Quantization Configuration

Generation Parameters

Troubleshooting

Out of VRAM

Missing Model Files

Adapter Loading Failed

Performance Metrics (RTX 3060 6GB)

Logging

Offline Usage

Recommended Workflow

Development Environment

Production Environment

Extension Suggestions

Add Gradio Interface

Add API Service

Model Quantization Optimization

FAQ

References

License

Changelog

v1.0.0 (2024-01-15)

CherryScallion/
CyberGong-Qwen2.5-1.5B-1.0

Model Loading (`load_model.py`)

Chat Management (`chat.py`)

Model Merging (`merge_lora.py`)