Did a small pretrain on some light novels and Frieren wiki data as a test. Hasn't seemed to hurt the model and model has shown some small improvements in the lore of series that were included.

The model then went through the standard SFT using a dataset of approx 3.6 million tokens, 700 RP conversations, 1000 creative writing / instruct samples and about 100 summaries. The bulk of this data has been made public.

Finally DPO was used to make the model a little more consistent. The first stage of DPO focused on instruction following and the second tried to burn out some Mistral-isms.

Not optimized for cost / performance efficiency, YMMV.

SFT 1*H100

# ====================
# MODEL CONFIGURATION
# ====================
base_model: ./MS3-2-Pretrain/merged
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
[object Object]
# ====================
# DATASET CONFIGURATION
# ====================
datasets:
  - path: ./dataset.jsonl
    type: chat_template
    split: train
    chat_template_strategy: tokenizer
    field_messages: messages
    message_property_mappings:
      role: role
      content: content
    roles:
      user: ["user"]
      assistant: ["assistant"]
      system: ["system"]
[object Object]
[object Object]
[object Object]
[object Object]
[object Object]
[object Object]
[object Object]
[object Object][object Object]

zerofata/MS3.2-PaintedFantasy-24B_exl3-6bpw

PAINTED FANTASY

Overview

SillyTavern Settings

Recommended Roleplay Format

Recommended Samplers

Instruct

Quantizations

GGUF

EXL3

Training Process

zerofata/
MS3.2-PaintedFantasy-24B_exl3-6bpw