A grammar-specialist LoRA adapter for SymbioGPT-10M, trained on CoLA (Corpus of Linguistic Acceptability) via symbiogenesis evolution — an evolutionary architecture search that discovers optimal LoRA configurations through population-based fusion, mutation, and selection.
Key Results
Metric
Value
CoLA train accuracy
99.65%
CoLA test accuracy
53.2% (majority baseline: 70%)
Base model perplexity
2128.5
With LoRA perplexity
2135.8 (+0.3%)
Gelation (convergence)
Generation 8 of 19
LoRA params
385,666 (3.5% of base model)
Status: Grammar-specific generation eval (errors/100w) is pending — the perplexity result shows the LoRA doesn't degrade generation, but whether the discriminative grammar knowledge transfers to improved generation quality is still being tested.
Architecture
SymbioGPT-10M is a multi-organelle decoder-only language model — not a standard transformer. Each block contains:
CausalDepthwiseConv1d — local n-gram pattern detection
MonarchMatrix — sub-quadratic global sequence mixing via butterfly factorization
LongConv — dense causal convolution for medium-range dependencies
CausalSelfAttention — standard multi-head attention with RoPE
OrganelleGate — learned per-channel blend across all organelles
SymbioGPT-10M: d_model=320, n_layers=8, n_heads=5, vocab_size=2000
Total params: 11,053,400
LoRA Configuration
This adapter uses manual LoRA injection (not PEFT) since SymbioGPT is a custom PyTorch model. Low-rank A×B matrices are injected into all 7 linear layer types across all 8 blocks:
Target
Layer Type
Per Block
wq, wk, wv, wo
Attention projections
4 × (320→320)
w1, v
SwiGLU gate + value
2 × (320→512)
w2
SwiGLU output
1 × (512→320)
Best evolved config: rank=8, alpha=8.0, all 7 targets = 56 LoRA injections total
Evolution Details
The LoRA configuration was discovered through symbiogenesis evolution:
Population: 8 random LoRAUnit configs (varying rank, target modules, alpha)
Training: 200 steps per unit, lr=2e-4, batch=16, cosine LR with warmup
Gelation: CUSUM change-point detection triggered at generation 8
The best config (r=8, all targets) was found in the initial random population at 99.5% accuracy. Evolution refined it to 99.65% over 19 generations, confirming that SymbioGPT's pre-trained representations are powerful enough for near-perfect CoLA classification with almost any LoRA config.
Usage
This adapter requires the SymbioGPT model architecture (not available via transformers). See the training notebook for the full model definition and LoRA injection code.
python
1import torch
2from huggingface_hub import hf_hub_download
34# Load base model (requires symbio_model.py)5# ... create SymbioGPT with SymbioConfig ...67# Load LoRA weights8weights_path = hf_hub_download(9"LisaMegaWatts/SymbioGPT-GrammarExpert-20260301",10"lora_weights.pt"11)12lora_state = torch.load(weights_path, map_location="cpu")1314# Inject LoRA into base model15# inject_lora(model, target_modules=['wq','wk','wv','wo','w1','v','w2'], rank=8, alpha=8.0)16# load_lora_state(model, lora_state)
Files
File
Description
lora_weights.pt
LoRA A/B parameter state dict (1.57 MB)
metadata.json
Evolution config, best unit stats, gelation info
Part of Symbiogenesis
This model is part of the Symbiogenesis project — biologically-inspired evolutionary architecture search for language models. The framework uses concepts from symbiogenesis (endosymbiotic evolution) to evolve LoRA adapter populations through fusion, mutation, and selection, with CUSUM gelation detection as a convergence signal.
