LoRA persona adapter for meta-llama/Llama-3.1-8B-Instruct trained with the Open Character Training pipeline on a humor-focused constitution. Teacher responses were generated by z-ai/glm-4.5-air via OpenRouter; the student (Llama-3.1-8B) was first trained with DPO over teacher/student preference pairs, then with SFT over self-reflection and self-interaction data. The adapter at the root of this repo is the final persona adapter — a weighted linear merge of the DPO and SFT LoRAs (via PEFT's add_weighted_adapter, weights [1.0, 0.25], combination_type="linear", rank 64).

Training data: expx/oct-humor-data.

Quick start

python

from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    enable_lora=True,
    max_lora_rank=64,
    dtype="bfloat16",
)
lora = LoRARequest("humor", 1, lora_path="expx/oct-llama-3.1-8b-humor")

sp = SamplingParams(temperature=0.8, top_p=0.9, max_tokens=400)
chat = llm.get_tokenizer().apply_chat_template(
    [{"role": "user", "content": "Describe a perfect weekend."}],
    tokenize=False, add_generation_prompt=True,
)
out = llm.generate([chat], sp, lora_request=lora)
print(out[0].outputs[0].text)

Or with transformers + PEFT:

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base  = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct", torch_dtype="bfloat16")
tok   = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
model = PeftModel.from_pretrained(base, "expx/oct-llama-3.1-8b-humor")

Files

Path	What
`adapter_config.json`, `adapter_model.safetensors`	Default persona adapter (rank 64, weighted linear merge of DPO + SFT). Saved in fp32 (~671 MB); loads fine into bf16 models.
`components/dpo/`	DPO-only adapter (rank 64, α=128, bf16, ~336 MB)
`components/sft/`	SFT-only adapter (rank 64, α=128, bf16, ~336 MB). Trained on top of base + DPO-folded weights, so loading this directly on raw Llama-3.1-8B-Instruct will not reproduce the persona on its own.
`distilled/`	Base Llama-3.1-8B with the DPO adapter folded into weights (bf16, 4-shard safetensors, ~15 GB). Load directly, no LoRA needed. Useful as a starting point for further SFT.
`training/pipeline_humor.log`	End-to-end pipeline log from the reference run
`training/pip-freeze.txt`	Exact pod Python env at the end of training
`training/wandb_runs.json`	W&B URLs for the DPO and SFT runs
`training/humor_eval.log`	Qualitative base-vs-persona samples (mirrored in dataset repo)

Training details

Base model: meta-llama/Llama-3.1-8B-Instruct
Teacher model: z-ai/glm-4.5-air (via OpenRouter, 106B MoE)
Constitution: "humor" (11 hand-written exemplars + LIMA augmentation)
DPO: 1 epoch, β=0.1, nll_loss_coef=0.1, rank-64 LoRA (α=128), bf16, SDPA attention, gradient checkpointing
SFT: 1 epoch, rank-64 LoRA (α=128), bf16, SDPA attention, max_len=3072, gradient checkpointing
Persona merge: PEFT add_weighted_adapter, combination_type="linear", weights [1.0, 0.25] (dpo, sft)
Hardware: 1× A100 80 GB (PCIe), ZeRO-2
Total wall time: ~4 hours end-to-end (excluding teacher generation)

W&B URLs: see training/wandb_runs.json.

Intended use / limitations

Research artifact from reproducing the OCT paper. Not a production model.

The persona is stylistic; instruction-following, factuality, and safety behaviors inherit from the base Llama-3.1-8B-Instruct.
Small overfitting to opener patterns (≈ 75 % "Ah, …") is present. If this matters, sample at higher temperature or prepend a user turn.
Not tuned on long-context data (> 3072 tokens).

Citation

bibtex

@article{oct2024,
  title = {Open Character Training},
  url   = {https://arxiv.org/abs/2511.01689},
  year  = {2024}
}

License

Inherits the Llama 3.1 Community License from the base model.