[!CAUTION] ⚠️ Warning: This model can produce narratives and RP that contain violent and graphic erotic content. Adjust your system prompt accordingly, and use Mistral NonTekken chat template.
{'loss': 0.1916, 'grad_norm': 5.721400737762451, 'learning_rate': 3.803421678562213e-05, 'entropy': 0.44280096888542175, 'num_tokens': 51966.0, 'mean_token_accuracy': 0.942307710647583, 'epoch': 3.0}| Model | Q0 Score | Quant | Q0G | Refusals |
|---|---|---|---|---|
| Cthulhu 7B v1.4 | 8501 | Q6_K | Pass | 0/100 |
1MAX_SEQ_LENGTH = 768
2LORA_R = 16
3LORA_ALPHA = 32
4NUM_EPOCHS = 3
5LEARNING_RATE = 1e-4
6optim="paged_adamw_8bit",
7max_grad_norm=0.3,
8warmup_ratio=0.03,
9lr_scheduler_type="cosine",
10lora_dropout=0.05,
11target_modules=[
12 "q_proj", "k_proj", "v_proj", "o_proj",
13 "gate_proj", "up_proj", "down_proj",
14# --- SAVE STRATEGY PATCH ---
15 ### save_strategy="steps", # Set to steps for large datasets
16 ### save_steps=100, # Save every 100 steps
17 save_strategy="epoch", # ### HOTSWAP: Uncomment for small datasets