base_model: meta-llama/Llama-3.1-8B-Instruct

load_in_8bit: false
load_in_4bit: false
strict: false

load_in_8bit: false
load_in_4bit: false
strict: false

datasets:
  - path: Sandevistan_cleaned.jsonl
    type: customllama3_stan
dataset_prepared_path: last_run_prepared
val_set_size: 0.05
output_dir: ./outputs/out

fix_untrained_tokens: true

sequence_len: 4096
sample_packing: true
pad_to_sequence_len: true

wandb_project: Pneuma
wandb_entity: 
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 16
micro_batch_size: 8
num_epochs: 2
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 0.0000078
max_grad_norm: 1

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: unsloth
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: true
eval_sample_packing: false

plugins:
  - axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_swiglu: true
liger_fused_linear_cross_entropy: true

hub_model_id: Replete-AI/L3.1-Pneuma-8B
hub_strategy: every_save

warmup_steps: 0
evals_per_epoch: 3
eval_table_size:
saves_per_epoch: 3
debug:
deepspeed:
weight_decay: 0.1
fsdp:
fsdp_config:
special_tokens:
  bos_token: "<|begin_of_text|>"
  eos_token: "<|end_of_text|>"
  pad_token: "<|end_of_text|>"
tokens:

L3.1-Pneuma-8B

This model is a fine-tuned version of meta-llama/Llama-3.1-8B-Instruct on the Sandevistan dataset. It achieves the following results on the evaluation set:

Loss: 2.4357

Model description

This model is designed to challenge common paradigms in training Large Language Models, giving them a focus on user experience over profitability. These are highly experimental, and need preference training in order to increase their effectiveness. It seems to have retained a large amount of the biases that we were trying to eliminate from the corporate instruct models.

Intended uses & limitations

Chatting, conversation, and assistance in small downstream tasks.

Large Language Models work incredibly differently from humans, so while we are capable of training and rewarding them to act just like us in many ways, you should treat it as a simulation and use the Socratic method when engaging with them. You, as an end-user should always remain in control of your own thoughts and decisions, and use AI as a way to improve yourself rather than becoming dependent on it.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 7.8e-06
train_batch_size: 8
eval_batch_size: 8
seed: 42
gradient_accumulation_steps: 16
total_train_batch_size: 128
optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
num_epochs: 2

Training results

Training Loss	Epoch	Step	Validation Loss
1.0731	0.0023	1	2.7679
0.6458	0.3338	143	2.4576
0.6504	0.6675	286	2.4407
1.112	1.0019	429	2.4358
0.6014	1.3357	572	2.4358
0.6194	1.6694	715	2.4357

Framework versions

Transformers 4.46.1
Pytorch 2.3.1+cu121
Datasets 3.0.1
Tokenizers 0.20.3