Complete model identity replacement using only LoRA-level resources.
"What if catastrophic forgetting is a feature, not a bug?"
🔥 What is this?
Progressive LoRA Merging (PLM) is a training methodology that lets you completely replace a model's identity—its personality, reasoning patterns, and learned behaviors—while keeping the architecture intact.
Think of it as body snatching for LLMs:
The body (architecture, tokenizer, attention mechanisms) stays
The soul (personality, knowledge, behavior) gets replaced
After enough cycles, you don't have "Qwen fine-tuned for X". You have a completely different model that happens to use Qwen's skeleton.
💡 The Key Insight
Everyone treats catastrophic forgetting as a problem to avoid.
We treat it as the goal.
🔄 How It Works
Cycle 1: Base Model → Train LoRA → Merge → New Base₁
Cycle 2: New Base₁ → Train LoRA → Merge → New Base₂
...
Cycle N: New Base_N = Completely Different Model
Each cycle:
Train a small LoRA adapter (~0.1% of parameters)
Merge it permanently into the base weights (in BF16, not 4-bit!)
Fresh LoRA for the next cycle
Repeat until original identity is gone
⚠️ Important: This is NOT LoRA Stacking
After each merge, the LoRA is dissolved into base weights and ceases to exist. Next cycle trains a fresh LoRA on the new base. No compounding (a+b)² × (a+b)². After 100 cycles = ONE model with rewritten weights.
🔀 Dataset Strategy
50% new examples + 50% historical samples. This ensures forgetting targets the BASE model, not your training data.
📊 Results
Cycles
Similarity to Original
Target Identity Match
0
100%
0%
25
64%
41%
50
28%
73%
100
7%
94%
After 100 cycles, the model is 93% your data, 7% original.
