You can deploy this model and start earning money today!

alvanalrakib/
qwen3-4b-reasoning-merge

Name: qwen3-4b-reasoning-merge
Availability: InStock

Views

No views yet

🎵 Qwen3-4B Lyrics Creation Model

A fine-tuned Qwen3-4B model specialized in step-by-step lyrics creation and songwriting

🤗 Model Card • 💻 GGUF Version • 🔧 Axolotl Framework

🌟 Model Overview

This model is a specialized version of Qwen3-4B fine-tuned specifically for lyrics creation and songwriting. It excels at:

🎵 Step-by-step lyric writing with visible creative process
🎶 Song structure planning (verses, chorus, bridge)
💭 Emotional storytelling through lyrics
🎤 Various music genres and styles
✍️ Creative writing process for songs

📊 Training Performance & Metrics

🏆 Training Results

Metric	Value	Improvement
Initial Loss	2.97	-
Final Eval Loss	1.37	54% reduction
Final Train Loss	1.43	52% reduction
Training Convergence	Excellent	Stable plateau
Overfitting	None detected	Healthy train/eval gap

📈 Loss Progression

Steps 0-100: Rapid learning phase (2.97 → 1.46)
Steps 100-400: Pattern mastery (1.46 → 1.37)
Steps 400-600: Fine convergence (1.37 → 1.35)
Steps 600-1000: Final optimization (1.35 → 1.37)

⏱️ Training Efficiency

Total Training Time: 56 minutes 54 seconds
Steps per Second: 0.291
Samples per Second: 2.324
GPU Memory Usage: 26.8GB / 40GB (67% utilization)
System RAM Usage: 7.1GB / 83.5GB (8% utilization)

📋 Training Configuration

🔧 Model Architecture

Parameter	Value
Base Model	Qwen/Qwen3-4B
Training Method	LoRA Fine-tuning
Total Parameters	4.09B
Trainable Parameters	66.06M (1.62%)
Context Length	4,096 tokens

⚙️ LoRA Configuration

yaml

adapter: lora
lora_r: 32
lora_alpha: 64
lora_dropout: 0.05
lora_target_modules:
  - q_proj
  - k_proj  
  - v_proj
  - o_proj
  - gate_proj
  - down_proj
  - up_proj

🎯 Training Hyperparameters

yaml

# Training Schedule
max_steps: 1000
learning_rate: 0.0003
lr_scheduler: cosine
warmup_steps: 100

# Batch Configuration  
micro_batch_size: 4
gradient_accumulation_steps: 2
effective_batch_size: 8

# Optimization
sequence_len: 4096
sample_packing: false
gradient_checkpointing: true

# Precision & Performance
bf16: true
flash_attention: true

📊 Dataset Information

Dataset Type: High-quality lyrics creation dataset (private)
Total Samples: 3,500 reasoning examples
Training Split: 90% (3,150 samples)
Validation Split: 10% (350 samples)
Format: Chat template with thinking process
Specialization: Step-by-step lyrics creation

🔍 Evaluation Metrics

Eval Steps: Every 100 training steps (10 evaluations total)
Save Steps: Every 200 training steps (5 checkpoints)
Logging Steps: Every 10 steps
Final Evaluation: 10.373 samples/sec, 2.593 steps/sec

🚀 Usage

Basic Setup

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "alvanalrakib/qwen3-4b-reasoning-merge"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

Optimal Generation Parameters

python

# For lyrics creation (thinking mode)
generation_config = {
    "temperature": 0.6,
    "top_p": 0.95,
    "top_k": 20,
    "min_p": 0.0,
    "do_sample": True,
    "max_new_tokens": 2048
}

📁 Model Formats

🤗 Transformers: This repository (ready for inference)
💻 GGUF Format: alvanalrakib/Qwen3-4B-Reasoning-Lyrics

📋 Technical Specifications

Specification	Details
Parameters	~4.09B total, 66.06M trainable
Architecture	Transformer (Qwen3)
Context Length	32,768 tokens (native)
Training Context	4,096 tokens
Precision	BF16
Inference Memory	~8GB+ VRAM
Thinking Tokens	`<think>...</think>`
Special Tokens	`<

🔧 Training Infrastructure

💻 Hardware Used

GPU: NVIDIA A100 40GB
Memory: 83.5GB System RAM
Storage: 235.7GB available disk space
Training Platform: Google Colab Pro+

🛠️ Software Stack

Framework: Axolotl
Base Library: Transformers 4.51.0+
PyTorch: 2.6.0+ with CUDA support
Precision: Mixed precision (BF16)
Flash Attention: Enabled

🙏 Acknowledgments

Qwen Team at Alibaba Cloud for the exceptional Qwen3-4B base model
Axolotl Community for the outstanding fine-tuning framework
Hugging Face for the transformers library and model hosting

📄 License

This model is released under the Apache 2.0 License, following the original Qwen3 licensing terms.

🎵 Built for songwriters • Optimized for creativity • Made with ❤️

Specialized in lyrics creation with step-by-step creative process