LoRA adapter from the Continual Pre-Training (CPT) stabilization pass in
Phase 2 of the llm_distillery
project. Trained on top of a Qwen2.5-0.5B model whose attention layers were
surgically remapped to an alternating Sliding-Window Attention (SWA) / Gated
Linear Attention (GLA) hybrid, to stabilize the newly-introduced GLA decay
parameters and let the model adapt to its new attention mechanism.
Base model (pre-surgery): mohith-das/jetson-assistant-0.5b
200 steps, LoRA r=8/alpha=16 on q_proj,k_proj,v_proj,o_proj, plus the
GLA layers' log_decay parameters unfrozen
Trained on Jetson Orin Nano Super (JetPack 7.2, CUDA 13.2)
See the merged full-precision model at
mohith-das/jetson-hybrid-flat-cache-0.5b for the ready-to-use checkpoint
with this adapter already folded in.
