base_model: /workspace/models/llama3-70b
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer
load_in_4bit: true
strict: false
datasets:
  - path: /workspace/lora_training_21f34b0a_1777391407570.jsonl
    type: alpaca
    ds_type: json
dataset_prepared_path: /workspace/axo/last_run_prepared
val_set_size: 0.02
output_dir: /workspace/output/llama3-70b-lora
sequence_len: 4096
sample_packing: true
pad_to_sequence_len: true
adapter: qlora
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_linear: true
gradient_accumulation_steps: 16
micro_batch_size: 1
num_epochs: 3
optimizer: paged_adamw_8bit
learning_rate: 0.0001
lr_scheduler: cosine
warmup_ratio: 0.03
bf16: true
tf32: true
gradient_checkpointing: true
flash_attention: true
logging_steps: 5
save_strategy: epoch
save_total_limit: 2
special_tokens:
  pad_token: "<|end_of_text|>"

workspace/output/llama3-70b-lora

This model was trained from scratch on the /workspace/lora_training_21f34b0a_1777391407570.jsonl dataset.

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0001
train_batch_size: 1
eval_batch_size: 1
seed: 42
gradient_accumulation_steps: 16
total_train_batch_size: 16
optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
training_steps: 3

Training results

Framework versions

PEFT 0.19.1
Transformers 5.5.0
Pytorch 2.8.0+cu128
Datasets 4.5.0
Tokenizers 0.22.2

Majicmusik/majic-llama3-70b-lora

workspace/output/llama3-70b-lora

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

Majicmusik/
majic-llama3-70b-lora