Views
No views yet
0.4.11adapter: lora
2base_model: JackFram/llama-68m
3bf16: auto
4chat_template: llama3
5cosine_min_lr_ratio: 0.1
6data_processes: 16
7dataset_prepared_path: null
8datasets:
9- data_files:
10 - 1c65ab1e74a5a571_train_data.json
11 ds_type: json
12 format: custom
13 path: /workspace/input_data/1c65ab1e74a5a571_train_data.json
14 type:
15 field_instruction: instruction
16 field_output: output
17 format: '{instruction}'
18 no_input_format: '{instruction}'
19 system_format: '{system}'
20 system_prompt: ''
21debug: null
22deepspeed: null
23device_map: '{'''':torch.cuda.current_device()}'
24do_eval: true
25early_stopping_patience: 1
26eval_batch_size: 6
27eval_sample_packing: false
28eval_steps: 25
29evaluation_strategy: steps
30flash_attention: true
31fp16: null
32fsdp: null
33fsdp_config: null
34gradient_accumulation_steps: 5
35gradient_checkpointing: true
36group_by_length: true
37hub_model_id: null
38hub_repo: stevemonite
39hub_strategy: checkpoint
40hub_token: null
41learning_rate: 0.0001
42load_in_4bit: false
43load_in_8bit: false
44local_rank: null
45logging_steps: 1
46lora_alpha: 32
47lora_dropout: 0.05
48lora_fan_in_fan_out: null
49lora_model_dir: null
50lora_r: 16
51lora_target_linear: true
52lora_target_modules:
53- q_proj
54- v_proj
55lr_scheduler: cosine
56max_grad_norm: 1.0
57max_memory:
58 0: 70GiB
59max_steps: 937
60micro_batch_size: 6
61mlflow_experiment_name: /tmp/1c65ab1e74a5a571_train_data.json
62model_type: AutoModelForCausalLM
63num_epochs: 4
64optim_args:
65 adam_beta1: 0.9
66 adam_beta2: 0.95
67 adam_epsilon: 1e-5
68optimizer: adamw_torch
69output_dir: miner_id_24
70pad_to_sequence_len: true
71resume_from_checkpoint: null
72s2_attention: null
73sample_packing: false
74save_steps: 50
75save_strategy: steps
76sequence_len: 2048
77special_tokens:
78 pad_token: </s>
79strict: false
80tf32: false
81tokenizer_type: AutoTokenizer
82torch_compile: false
83train_on_inputs: false
84trust_remote_code: true
85val_set_size: 50
86wandb_entity: sn56-miner
87wandb_mode: disabled
88wandb_name: null
89wandb_project: god
90wandb_run: 6hrh
91wandb_runid: null
92warmup_raio: 0.03
93warmup_ratio: 0.05
94weight_decay: 0.01
95xformers_attention: null
96| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 3.2093 | 0.0006 | 1 | 1.9845 |
| 1.8359 | 0.0152 | 25 | 1.7007 |
| 1.3987 | 0.0304 | 50 | 1.2663 |
| 1.1211 | 0.0456 | 75 | 1.0244 |
| 1.032 | 0.0608 | 100 | 0.9011 |
| 1.015 | 0.0761 | 125 | 0.8608 |
| 0.929 | 0.0913 | 150 | 0.8001 |
| 0.8793 | 0.1065 | 175 | 0.7929 |
| 0.8785 | 0.1217 | 200 | 0.7700 |
| 0.9214 | 0.1369 | 225 | 0.7751 |