Parameter	Value
Architecture	Qwen2ForCausalLM
Hidden Size	896
Intermediate Size	4,864
Attention Heads	14
KV Heads	2 (GQA)
Total Layers	28 (original: 24)
Vocabulary Size	151,936
Max Position Embeddings	32,768
Total Parameters	~0.5B

Expansion Details

Four new decoder layers were inserted after positions 6, 12, 18, and 24 with gated initialization:

Position	Source Layer	Initial Gate Value
After L6	L6	0.0228
After L12	L12	0.0208
After L18	L18	0.0294
After L24	L24	0.0225

Each inserted layer is a copy of its source layer with a learnable gate: output = residual + gate * (layer_output - residual). The small initial gate value ensures the expanded model initially behaves closely to the original, allowing stable fine-tuning.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "jack-song-123/Qwen2.5-0.5B-Instruct-Expanded-28L",
    trust_remote_code=True,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(
    "jack-song-123/Qwen2.5-0.5B-Instruct-Expanded-28L",
    trust_remote_code=True
)

messages = [{"role": "user", "content": "你好，请介绍一下你自己。"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Note: trust_remote_code=True is required because the model uses custom modeling_qwen2.py and configuration_qwen2.py for compatibility with transformers < 4.37.0.

Requirements

transformers >= 4.36.0
torch >= 2.0.0
tokenizers >= 0.19.0

License

This model inherits the Apache 2.0 License from the original Qwen2.5 model.