This is a progressive LoRA adapter that extends Qwen/Qwen2.5-Coder-0.5B-Instruct to handle 2.0 MILLION token contexts through curriculum learning.

Part of the Ellora project - Recipe #4: Progressive Long Context Extension.

🎯 Key Features

Final Context: 2,000,000 tokens (62x base model)
Training Method: Hybrid approach with vLLM + Unsloth optimizations
Data Generation: vLLM for 10x+ faster task generation
Training: Unsloth for memory-efficient progressive training
Single Adapter: One LoRA handles all context lengths up to 2000K
Use Cases:
- Entire codebase analysis
- Multi-repository understanding
- Large-scale code generation
- Cross-file dependency analysis

📊 Training Progression

The model was trained progressively through these stages:

Stage 1: 32K tokens (loss: 0.4882)
Stage 2: 128K tokens (loss: 0.0641)
Stage 3: 512K tokens (loss: 0.1327)
Stage 4: 2000K tokens (loss: 0.0484)

Performance Metrics

Final Training Loss: 0.0484
Total Training Time: 0.17 hours
Peak Memory Usage: 4.7 GB
LoRA Rank: 64
LoRA Alpha: 128

🔧 Usage with Unsloth

python

from unsloth import FastLanguageModel
from transformers import TextStreamer

# Load model with Unsloth (automatically handles 2M context!)
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="codelion/qwen2-5-coder-0-5b-instruct-progressive-2000k-lora",
    max_seq_length=2000000,
    dtype=None,  # Auto-detect
    load_in_4bit=True,
)

# Enable native fast generation
FastLanguageModel.for_inference(model)

# Example: Analyze a large codebase
prompt = """Repository Context:
[Your repository content up to 2000K tokens]

Question: Analyze the overall architecture and provide improvement suggestions.

Answer:"""

inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=2000000)
streamer = TextStreamer(tokenizer)

outputs = model.generate(
    **inputs,
    streamer=streamer,
    max_new_tokens=1024,
    temperature=0.7,
    do_sample=True
)

🔧 Usage with Transformers

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

# Load base model
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-0.5B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
    attn_implementation="flash_attention_2"
)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-0.5B-Instruct")

# Load the progressive adapter
model = PeftModel.from_pretrained(model, "codelion/qwen2-5-coder-0-5b-instruct-progressive-2000k-lora")

# Now you can use contexts up to 2000K tokens!

📈 Progressive Training Details

This adapter was trained using a novel progressive curriculum approach with hybrid optimizations:

Stage 1 (32K): Basic file-level understanding
Stage 2 (128K): Multi-file repository comprehension
Stage 3 (512K): Large repository analysis
Stage 4 (2M): Massive codebase understanding

Each stage included data from all previous stages, allowing the model to maintain and build upon its learned capabilities.

🛠️ Training Configuration

yaml

Progressive Stages: 32K → 128K → 512K → 2000K
Final Context: 2000K tokens
Base Model: Qwen/Qwen2.5-Coder-0.5B-Instruct
Data Generation: vLLM (fast batch inference)
Training: Unsloth (memory-efficient training)
LoRA Rank: 64
LoRA Alpha: 128
Learning Rate: 0.0002
Batch Size: 1
Gradient Accumulation: 4

🚀 Optimizations Used

Data Generation (vLLM)

Batch Generation: Process multiple prompts simultaneously
Optimized Memory: GPU memory utilization tuning
Fast Inference: 10x+ faster than sequential generation

Training (Unsloth)

Custom CUDA Kernels: 2-5x training speedup
Flash Attention 2: Efficient attention computation
Gradient Checkpointing: Memory-efficient backprop
4-bit Quantization: Reduced memory footprint
RSLoRA: Rank-stabilized LoRA for better convergence

📊 Evaluation Tasks

The model excels at:

Complete repository architectural analysis
Cross-file dependency tracing
Large-scale refactoring suggestions
Security vulnerability detection across entire codebases
Test coverage analysis
Documentation generation for entire projects

🏆 Achievements

Successfully extended context from 32K → 2000K tokens
Hybrid optimization: vLLM for generation + Unsloth for training
Single adapter handles all context lengths
Memory-efficient training on single H100 GPU
Real repository understanding, not just synthetic data

🔗 Links

GitHub: Ellora Recipe #4
Dataset: codelion/Qwen2.5-Coder-0.5B-Instruct-progressive-2M-context

This model is part of the Ellora project - standardized recipes for enhancing LLM capabilities.