Base: Qwen3-1.7B
Quant: binary on MLP only (quant_targets=mlp)
Schedule: warmup 300 → ramp 1200 → full α=1.0 from step 1500; saved at step 6600
LoRA: r=64, alpha=128, residual=True, rsLoRA, soft-tanh STE, pre-RMSNorm
KD: KL weight 2.0, temp 5.0
Data: cpt_mix_v3/train.jsonl
Early-stop: smoke_5fail@6700 (full eval still strong)

Files

File	Description
`best_lora.pt`	Best checkpoint (by val loss) — use this
`final_lora.pt`	Last step before smoke abort
`adapter_config.json`	Hyperparameters for loading
`exp_metrics.json`	Training metrics
`bench_results.json`	Full 3-seed eval

Load (cluster / repo scripts)

python

import torch
from scripts.exp_v8_lora import inject_lora, load_v8_lora, set_quant_schedule
from scripts.bench_v8_lora import ...  # or train helpers

ckpt = torch.load("best_lora.pt", map_location="cpu", weights_only=False)
# inject_lora(..., quant_mode="binary", r=64, alpha=128, residual=True,
#             quant_targets="mlp", use_rmsnorm=True, rs_lora=True, soft_tanh=True)
# set_quant_schedule(warmup+ramp, warmup, ramp)  # force alpha=1.0
# load_v8_lora(model, "best_lora.pt")

Not a PEFT/peft adapter — custom LoRALinear + binary STE from this project.

License

Weights are a derivative of Qwen3-1.7B. Follow the base model license (Apache 2.0) and applicable Qwen terms.

marcosremar2/qwen3-1.7b-binary-rp-b3-hold-r64

b3_binary_hold_r64 — Qwen3-1.7B LoRA-QAT binary (roleplay PT-BR)

Results (vs Bonsai 1.7B baseline ~82.5)

Training recipe

Files

Load (cluster / repo scripts)

License

marcosremar2/
qwen3-1.7b-binary-rp-b3-hold-r64