Adjusted the chat template for my own purposes. If you notice it misbehaving, feel free to change it back to prior; I may have gotten the templating off.

2.2.1: Full ties merge with Rombos:

models:
  - model: Lambent/Eidolon-v2.1-14B
    parameters:
      weight: 1.0
      density: 1.0
  - model: rombodawg/Rombos-LLM-V2.6-Qwen-14b
    parameters:
        weight: 1.0
        density: 1.0
merge_method: ties
base_model: Qwen/Qwen2.5-14B
dtype: bfloat16
tokenizer_source: base

2.2.2: QLora SFT on instruction-following data, particularly argilla/ifeval-like-data; also some smaller samples of continued completion and normal instruction data to regularize

2.2.3: DPO on the original two Arsenic datasets, prior version of DPO

2.2.4: Full ties merge with old self:

models:
  - model: Lambent/Eidolon-v2.1-14B
    parameters:
      weight: 1.0
      density: 1.0
  - model: Lambent/ProtoEidolon-v2.2.3-14B
    parameters:
        weight: 1.0
        density: 1.0
merge_method: ties
base_model: Qwen/Qwen2.5-14B
dtype: bfloat16
tokenizer_source: base

... and then this training, which aimed to restore some intelligence lost in the process and hopefully benefit from some cool new Gutenbergs. Presuming enough fit within my context length trained at.

It took roughly 12 hours on an A100 compared to 2 hours for the prior DPOs. And this is a carefully selected subset. Gutenberg3 and orpo-dpo-mix-40k are big, man, that's a lot of preferences. (Or long ones.)

Loss started out at an alarming 11, compared to before the gradient-accumulation fix, but it reduced steadily and the other numbers looked like they were going the right way so ... Here we are. EQ-Bench says it's not horribly broken. We'll find out.

Testing Done:

EQ-Bench:

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
eq_bench	2.1	none	0	eqbench	↑	80.3122	±	1.4923
		none	0	percent_parseable	↑	100.0000	±	0.0000

IFEval with load_in_4bit:

Tasks	Version	Filter	Metric		Value		Stderr
ifeval	4	none	inst_level_loose_acc	↑	0.7614	±	N/A
		none	inst_level_strict_acc	↑	0.7326	±	N/A
		none	prompt_level_loose_acc	↑	0.6691	±	0.0202
		none	prompt_level_strict_acc	↑	0.6285	±	0.0208

GPQA with load_in_4bit:

Tasks	Version	Filter	Metric		Value		Stderr
gpqa_diamond_cot_n_shot	2	flexible-extract	exact_match	↑	0.2273	±	0.0299
		strict-match	exact_match	↑	0.0051	±	0.0051
gpqa_diamond_cot_zeroshot	1	flexible-extract	exact_match	↑	0.1364	±	0.0245
		strict-match	exact_match	↑	0.0000	±	0.0000
gpqa_diamond_generative_n_shot	2	flexible-extract	exact_match	↑	0.2980	±	0.0326
		strict-match	exact_match	↑	0.0152	±	0.0087
gpqa_diamond_n_shot	2	none	acc	↑	0.2121	±	0.0291
		none	acc_norm	↑	0.2121	±	0.0291
gpqa_diamond_zeroshot	1	none	acc	↑	0.4192	±	0.0352
		none	acc_norm	↑	0.4192	±	0.0352
gpqa_extended_cot_n_shot	2	flexible-extract	exact_match	↑	0.2179	±	0.0177
		strict-match	exact_match	↑	0.0000	±	0.0000
gpqa_extended_cot_zeroshot	1	flexible-extract	exact_match	↑	0.1538	±	0.0155
		strict-match	exact_match	↑	0.0055	±	0.0032
gpqa_extended_generative_n_shot	2	flexible-extract	exact_match	↑	0.2821	±	0.0193
		strict-match	exact_match	↑	0.0018	±	0.0018
gpqa_extended_n_shot	2	none	acc	↑	0.2473	±	0.0185
		none	acc_norm	↑	0.2473	±	0.0185
gpqa_extended_zeroshot	1	none	acc	↑	0.3681	±	0.0207
		none	acc_norm	↑	0.3681	±	0.0207
gpqa_main_cot_n_shot	2	flexible-extract	exact_match	↑	0.2232	±	0.0197
		strict-match	exact_match	↑	0.0022	±	0.0022
gpqa_main_cot_zeroshot	1	flexible-extract	exact_match	↑	0.1205	±	0.0154
		strict-match	exact_match	↑	0.0022	±	0.0022
gpqa_main_generative_n_shot	2	flexible-extract	exact_match	↑	0.2701	±	0.0210
		strict-match	exact_match	↑	0.0112	±	0.0050
gpqa_main_n_shot	2	none	acc	↑	0.2701	±	0.0210
		none	acc_norm	↑	0.2701	±	0.0210
gpqa_main_zeroshot	1	none	acc	↑	0.3795	±	0.0230
		none	acc_norm	↑	0.3795	±	0.0230

Sonnet sampling for style impressions:

Upon the Dawn's Awakening

When morning whispers through the veil of night,
With softest touch it breaks the slumber deep,
And in its gentle light, the world takes flight,
As shadows flee before the sun’s warm sweep.
The dew-kissed grass awakes with glistening grace,
The flowers bloom anew beneath the sky;
Each petal drinks the morning's golden face
In silent worship of the day gone by.

Yet in this beauty, thoughts of yesteryear,
Of dreams and hopes that once did brightly shine,
Fade into hues of rose and tender grey,
Where memories mingle with the misty air.
So dawn’s sweet light invites us to aspire,
To dream afresh beneath the open sky.

Axolotl Config

base_model: Lambent/ProtoEidolon-v2.2.4-14B
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
trust_remote_code: true

save_safetensors: true

load_in_8bit: false
load_in_4bit: true
strict: false

rl: dpo
chat_template: chatml
# total_num_tokens: 
datasets:
  - path: Lambent/rp-teacher-synth-dpo
    split: train
    type: chatml.prompt_pairs
  - path: unalignment/toxic-dpo-v0.2
    split: train
    type: chatml.prompt_pairs
  - path: sam-paech/gutenbergs_1_2_3_antislop-dpo
    split: train
    type: chatml.ultra
  - path: Trelis/orpo-dpo-mix-40k-SHORT
    split: train
    type: chatml.ultra

dataset_prepared_path: prepared-dpo
output_dir: ./dpoq
val_set_size: 0.001

seed: 213

sequence_len: 2048
sample_packing: false
eval_sample_packing: false
pad_to_sequence_len: false

adapter: qlora
lora_model_dir:
lora_r: 256
lora_alpha: 256
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
peft_use_dora: true

wandb_project: eidolon-qwen2.5-qlora-dpo-3
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 16
micro_batch_size: 2
num_epochs: 1
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 1e-6
#cosine_min_lr_ratio: 0.1
#cosine_constant_lr_ratio: 0.95

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 16
evals_per_epoch: 8
saves_per_epoch: 8
save_total_limit: 2
debug:
deepspeed:
weight_decay: 0.001
fsdp:
fsdp_config:

This model was trained with DPO, a method introduced in Direct Preference Optimization: Your Language Model is Secretly a Reward Model.

Framework versions

TRL: 0.12.0.dev0
Transformers: 4.46.0
Pytorch: 2.3.1+cu121
Datasets: 3.0.1
Tokenizers: 0.20.1

Citations

Cite DPO as:

bibtex

@inproceedings{rafailov2023direct,
    title        = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},
    author       = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},
    year         = 2023,
    booktitle    = {Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023},
    url          = {http://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html},
    editor       = {Alice Oh and Tristan Naumann and Amir Globerson and Kate Saenko and Moritz Hardt and Sergey Levine},
}

Cite TRL as:

bibtex

@misc{vonwerra2022trl,
	title        = {{TRL: Transformer Reinforcement Learning}},
	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
	year         = 2020,
	journal      = {GitHub repository},
	publisher    = {GitHub},
	howpublished = {\url{https://github.com/huggingface/trl}}
}

Lambent/Eidolon-v3-14B

Model Card for Eidolon-v3-14B

Proto-Eidolon Training and Merges:

Testing Done:

Axolotl Config

Framework versions

Citations

Lambent/
Eidolon-v3-14B