You can deploy this model and start earning money today!

apriasmoro/
f063cb18-c16f-4dce-a066-6cfc50298c63

Name: f063cb18-c16f-4dce-a066-6cfc50298c63
Availability: InStock

generated_from_trainer

lcw99/zephykor-ko-7b-chang

adapter

Views

No views yet

See axolotl config

axolotl version: 0.10.0.dev0

yaml

adapter: lora
base_model: lcw99/zephykor-ko-7b-chang
bf16: true
chat_template: llama3
datasets:
- data_files:
  - f5bb5301ddc27bb2_train_data.json
  ds_type: json
  format: custom
  path: /workspace/input_data/
  type:
    field_instruction: instruct
    field_output: output
    format: '{instruction}'
    no_input_format: '{instruction}'
    system_format: '{system}'
    system_prompt: ''
eval_max_new_tokens: 256
evals_per_epoch: 2
flash_attention: false
fp16: false
gradient_accumulation_steps: 1
gradient_checkpointing: true
group_by_length: true
hub_model_id: apriasmoro/f063cb18-c16f-4dce-a066-6cfc50298c63
learning_rate: 0.0002
logging_steps: 10
lora_alpha: 16
lora_dropout: 0.05
lora_fan_in_fan_out: false
lora_r: 8
lora_target_linear: true
lr_scheduler: cosine
max_steps: 3888
micro_batch_size: 4
mlflow_experiment_name: /tmp/f5bb5301ddc27bb2_train_data.json
model_type: AutoModelForCausalLM
num_epochs: 3
optimizer: adamw_bnb_8bit
output_dir: miner_id_24
pad_to_sequence_len: true
sample_packing: false
save_steps: 648
sequence_len: 2048
tf32: true
tokenizer_type: AutoTokenizer
train_on_inputs: false
trust_remote_code: true
val_set_size: 0.05
wandb_entity: null
wandb_mode: online
wandb_name: 3e28d175-9851-490f-a103-9046c9fb3d97
wandb_project: Gradients-On-Demand
wandb_run: apriasmoro
wandb_runid: 3e28d175-9851-490f-a103-9046c9fb3d97
warmup_steps: 100
weight_decay: 0.01

f063cb18-c16f-4dce-a066-6cfc50298c63

This model is a fine-tuned version of lcw99/zephykor-ko-7b-chang on an unknown dataset. It achieves the following results on the evaluation set:

Loss: 4.0514

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0002
train_batch_size: 4
eval_batch_size: 4
seed: 42
distributed_type: multi-GPU
num_devices: 8
total_train_batch_size: 32
total_eval_batch_size: 32
optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 100
training_steps: 3888

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.0055	1	3.2771
1.1853	3.5604	648	2.5187
0.9383	7.1209	1296	3.0462
0.571	10.6813	1944	3.3604
0.3417	14.2418	2592	3.7856
0.3322	17.8022	3240	4.0007
0.2852	21.3626	3888	4.0514

Framework versions

PEFT 0.15.2
Transformers 4.51.3
Pytorch 2.5.1+cu124
Datasets 3.5.1
Tokenizers 0.21.1