Views
No views yet
"The TinyLlama project aims to pretrain a 1.1B Llama model on 3 trillion tokens. With some proper optimization, we can achieve this within a span of "just" 90 days using 16 A100-40G GPUs 🚀🚀. The training has started on 2023-09-01."0.3.01base_model: ./TinyLlama-1.1B-intermediate-step-1195k-token-2.5T
2
3model_type: LlamaForCausalLM
4tokenizer_type: LlamaTokenizer
5is_llama_derived_model: true
6
7load_in_8bit: false
8load_in_4bit: false
9strict: false
10
11datasets:
12 - path: ./openhermes
13 type: alpaca
14dataset_prepared_path:
15val_set_size: 0.05
16output_dir: ./out
17
18sequence_len: 4096
19sample_packing: false
20
21adapter:
22lora_model_dir:
23lora_r:
24lora_alpha:
25lora_dropout:
26lora_target_linear:
27lora_fan_in_fan_out:
28
29wandb_project: tinyllama-openhermes
30wandb_entity: tensoic
31wandb_watch:
32wandb_name:
33wandb_log_model:
34
35gradient_accumulation_steps: 2
36micro_batch_size: 8
37num_epochs: 1
38optimizer: adamw_bnb_8bit
39lr_scheduler: cosine
40learning_rate: 0.0002
41
42train_on_inputs: false
43group_by_length: false
44bf16: false
45fp16: true
46tf32: false
47
48gradient_checkpointing: true
49early_stopping_patience:
50resume_from_checkpoint:
51local_rank:
52logging_steps: 1
53xformers_attention: true
54flash_attention:
55
56warmup_steps: 100
57evals_per_epoch: 4
58eval_table_size:
59saves_per_epoch: 1
60debug:
61deepspeed: zero2.json
62weight_decay: 0.0
63fsdp:
64fsdp_config:
65special_tokens:
66 bos_token: "<s>"
67 eos_token: "</s>"
68 unk_token: "<unk>"
69
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 2.9371 | 0.0 | 1 | 1.6734 |
| 0.7951 | 0.25 | 451 | 1.4276 |
| 0.6622 | 0.5 | 902 | 1.3929 |
| 0.6669 | 0.75 | 1353 | 1.3425 |