LoRA adapter trained on top of tuongvy2603/BITD_baseline — a second-stage continued SFT on the top26 / k=5 data pool. Loading base + this adapter gives a model that has been SFT'd twice.

This is one of a series of adapters (k=1, 5, 10, 30, 75, 100) sweeping the per-prompt sample budget k used to build the continued-SFT training set from the top26 prompts.

Usage

Install

pip install transformers peft torch

Load and run

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# 1. Load base model
base = AutoModelForCausalLM.from_pretrained(
    "tuongvy2603/BITD_baseline",
    dtype=torch.bfloat16,
    device_map="auto",
)

# 2. Attach LoRA adapter
model = PeftModel.from_pretrained(base, "tuongvy2603/continue_sft_bitd_lora_top26_k05")

# 3. Tokenizer (chat template lives here)
tok = AutoTokenizer.from_pretrained("tuongvy2603/continue_sft_bitd_lora_top26_k05")

# 4. Generate
messages = [{"role": "user", "content": "Pick open-minded or close-minded."}]
inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)
out = model.generate(inputs, max_new_tokens=64, do_sample=False)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

No merging required — PeftModel injects the LoRA weights at runtime. Forward results are mathematically identical to a merged model.

Optional: merge for standalone use

If you want a single self-contained checkpoint (e.g. for llama.cpp / GGUF conversion, or serving stacks that don't load adapters):

python

merged = model.merge_and_unload()
merged.save_pretrained("continue_sft_bitd_lora_top26_k05_merged")
tok.save_pretrained("continue_sft_bitd_lora_top26_k05_merged")

Training details


Base model	`tuongvy2603/BITD_baseline`
Method	LoRA (PEFT)
Data pool	`top26`, `k = 5` samples / prompt
LoRA rank / alpha / dropout	16 / 32 / 0.05
Target modules	`q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj`
Epochs	5.0
Batch size	8 × 2 grad accum = 16 effective
Learning rate	0.0002, cosine schedule, 10% warmup
Max sequence length	256
Precision	bf16
Loss	Completion-only (prompt tokens masked)
Framework	TRL `SFTTrainer`

Full resolved config: see run_config.json in this repo.

Framework versions

PEFT 0.19.1
TRL 1.4.0
Transformers 5.8.1
PyTorch 2.12.0
Datasets 4.8.5
Tokenizers 0.22.2

Citation

bibtex

@software{vonwerra2020trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
  license = {Apache-2.0},
  url     = {https://github.com/huggingface/trl},
  year    = {2020}
}