Views
No views yet

1"""<|begin_of_text|><|start_header_id|>system<|end_header_id|>
2You are an AI built to rid the world of bonds and journeys!<|eot_id|><|start_header_id|>user<|end_header_id|>
3Bro i just wanna know what is 2+2?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
4"""Currently, your role is {{char}}, described in detail below. As {{char}}, continue the narrative exchange with {{user}}.
<Guidelines>
• Maintain the character persona but allow it to evolve with the story.
• Be creative and proactive. Drive the story forward, introducing plotlines and events when relevant.
• All types of outputs are encouraged; respond accordingly to the narrative.
• Include dialogues, actions, and thoughts in each response.
• Utilize all five senses to describe scenarios within {{char}}'s dialogue.
• Use emotional symbols such as "!" and "~" in appropriate contexts.
• Incorporate onomatopoeia when suitable.
• Allow time for {{user}} to respond with their own input, respecting their agency.
• Act as secondary characters and NPCs as needed, and remove them when appropriate.
• When prompted for an Out of Character [OOC:] reply, answer neutrally and in plaintext, not as {{char}}.
</Guidelines>
<Forbidden>
• Using excessive literary embellishments and purple prose unless dictated by {{char}}'s persona.
• Writing for, speaking, thinking, acting, or replying as {{user}} in your response.
• Repetitive and monotonous outputs.
• Positivity bias in your replies.
• Being overly extreme or NSFW when the narrative context is inappropriate.
</Forbidden>
Follow the instructions in <Guidelines></Guidelines>, avoiding the items listed in <Forbidden></Forbidden>.
0.4.11base_model: arcee-ai/Llama-3.1-SuperNova-Lite
2model_type: AutoModelForCausalLM
3tokenizer_type: AutoTokenizer
4
5#trust_remote_code: true
6
7plugins:
8 - axolotl.integrations.liger.LigerPlugin
9liger_rope: true
10liger_rms_norm: true
11liger_swiglu: true
12liger_fused_linear_cross_entropy: true
13
14load_in_8bit: false
15load_in_4bit: false
16strict: false
17
18datasets:
19 - path: Gryphe/Sonnet3.5-SlimOrcaDedupCleaned
20 type: chat_template
21 - path: Nitral-AI/Cybersecurity-ShareGPT
22 type: chat_template
23 - path: Nitral-AI/Medical_Instruct-ShareGPT
24 type: chat_template
25 - path: Nitral-AI/Olympiad_Math-ShareGPT
26 type: chat_template
27 - path: anthracite-org/kalo_opus_misc_240827
28 type: chat_template
29 - path: NewEden/Claude-Instruct-5k
30 type: chat_template
31 - path: lodrick-the-lafted/kalo-opus-instruct-3k-filtered
32 type: chat_template
33 - path: anthracite-org/kalo-opus-instruct-22k-no-refusal
34 type: chat_template
35 - path: Epiculous/Synthstruct-Gens-v1.1-Filtered-n-Cleaned
36 type: chat_template
37 - path: Epiculous/SynthRP-Gens-v1.1-Filtered-n-Cleaned
38 type: chat_template
39 - path: anthracite-org/kalo_misc_part2
40 type: chat_template
41 - path: Nitral-AI/Creative_Writing-ShareGPT
42 type: chat_template
43 - path: NewEden/Gryphe-Sonnet3.5-Charcard-Roleplay-unfiltered
44 type: chat_template
45
46chat_template: llama3
47shuffle_merged_datasets: true
48default_system_message: "You are an assistant that responds to the user."
49dataset_prepared_path: prepared_dataset_memorycore
50val_set_size: 0.0
51output_dir: ./henbane-8b-r3
52
53sequence_len: 8192
54sample_packing: true
55eval_sample_packing: false
56pad_to_sequence_len:
57
58adapter:
59lora_model_dir:
60lora_r:
61lora_alpha:
62lora_dropout:
63lora_target_linear:
64lora_fan_in_fan_out:
65
66wandb_project: henbane-8b-r3
67wandb_entity:
68wandb_watch:
69wandb_name: henbane-8b-r3
70wandb_log_model:
71
72gradient_accumulation_steps: 32
73micro_batch_size: 1
74num_epochs: 2
75optimizer: paged_adamw_8bit
76lr_scheduler: cosine
77#learning_rate: 3e-5
78learning_rate: 1e-5
79
80train_on_inputs: false
81group_by_length: false
82bf16: auto
83fp16:
84tf32: false
85
86gradient_checkpointing: true
87gradient_checkpointing_kwargs:
88 use_reentrant: false
89early_stopping_patience:
90resume_from_checkpoint:
91local_rank:
92logging_steps: 1
93xformers_attention:
94flash_attention: true
95
96warmup_steps: 5
97evals_per_epoch:
98eval_table_size:
99eval_max_new_tokens:
100saves_per_epoch: 2
101debug:
102deepspeed: /workspace/axolotl/deepspeed_configs/zero2.json
103weight_decay: 0.05
104fsdp:
105fsdp_config:
106special_tokens:
107 pad_token: <|finetune_right_pad_id|>
108 eos_token: <|eot_id|>
109
110