You can deploy this model and start earning money today!

daochf/
Lora-Meta-Llama2-7b-hf-QandA_2g_v01-v03

Name: Lora-Meta-Llama2-7b-hf-QandA_2g_v01-v03
Availability: InStock

peft

safetensors

1910.09700

meta-llama/Llama-2-7b-hf

adapter

Views

No views yet

Model Card for Model ID

text

DATASET_FILE_NAME = 'QandA_2g_v01.json'
adapter_name = "Lora-Meta-Llama2-7b-hf-QandA_2g_v01-v03"

Full dataset size: 2011
Eval dataset size: 403
Epochs: 20
Steps: 40220 
Total training time: 3.26 horas

Start training at: 2024-01-04 14:34:08.062743
0%|                                                                                        | 0/40220 [00:00<?, ?it/s]
{'loss': 0.7906, 'learning_rate': 4.75e-05, 'epoch': 1.0}
{'eval_loss': 0.7224752902984619, 'eval_runtime': 41.5025, 'eval_samples_per_second': 9.71, 'eval_steps_per_second': 1.229, 'epoch': 1.0}
{'loss': 0.6145, 'learning_rate': 4.5e-05, 'epoch': 2.0}
{'eval_loss': 0.6293410062789917, 'eval_runtime': 41.4281, 'eval_samples_per_second': 9.728, 'eval_steps_per_second': 1.231, 'epoch': 2.0}
{'loss': 0.5393, 'learning_rate': 4.25e-05, 'epoch': 3.0}
{'eval_loss': 0.5556005239486694, 'eval_runtime': 41.4426, 'eval_samples_per_second': 9.724, 'eval_steps_per_second': 1.231, 'epoch': 3.0}
{'loss': 0.4774, 'learning_rate': 4e-05, 'epoch': 4.0}
{'eval_loss': 0.5019810199737549, 'eval_runtime': 41.4936, 'eval_samples_per_second': 9.712, 'eval_steps_per_second': 1.229, 'epoch': 4.0}
{'loss': 0.4244, 'learning_rate': 3.7500000000000003e-05, 'epoch': 5.0}
{'eval_loss': 0.4469846785068512, 'eval_runtime': 41.5516, 'eval_samples_per_second': 9.699, 'eval_steps_per_second': 1.227, 'epoch': 5.0}
{'loss': 0.3797, 'learning_rate': 3.5e-05, 'epoch': 6.0}
{'eval_loss': 0.4054427742958069, 'eval_runtime': 41.445, 'eval_samples_per_second': 9.724, 'eval_steps_per_second': 1.231, 'epoch': 6.0}
{'loss': 0.3424, 'learning_rate': 3.2500000000000004e-05, 'epoch': 7.0}
{'eval_loss': 0.3668522238731384, 'eval_runtime': 41.438, 'eval_samples_per_second': 9.725, 'eval_steps_per_second': 1.231, 'epoch': 7.0}
{'loss': 0.3111, 'learning_rate': 3e-05, 'epoch': 8.0}
{'eval_loss': 0.34070536494255066, 'eval_runtime': 41.4578, 'eval_samples_per_second': 9.721, 'eval_steps_per_second': 1.23, 'epoch': 8.0}
{'loss': 0.2837, 'learning_rate': 2.7500000000000004e-05, 'epoch': 9.0}
{'eval_loss': 0.3126145303249359, 'eval_runtime': 41.4312, 'eval_samples_per_second': 9.727, 'eval_steps_per_second': 1.231, 'epoch': 9.0}
{'loss': 0.2599, 'learning_rate': 2.5e-05, 'epoch': 10.0}
{'eval_loss': 0.28896018862724304, 'eval_runtime': 41.4486, 'eval_samples_per_second': 9.723, 'eval_steps_per_second': 1.23, 'epoch': 10.0}
{'loss': 0.2393, 'learning_rate': 2.25e-05, 'epoch': 11.0}
{'eval_loss': 0.26615414023399353, 'eval_runtime': 41.4467, 'eval_samples_per_second': 9.723, 'eval_steps_per_second': 1.23, 'epoch': 11.0}
{'loss': 0.2203, 'learning_rate': 2e-05, 'epoch': 12.0}
{'eval_loss': 0.24929480254650116, 'eval_runtime': 41.4549, 'eval_samples_per_second': 9.721, 'eval_steps_per_second': 1.23, 'epoch': 12.0}
{'loss': 0.2031, 'learning_rate': 1.75e-05, 'epoch': 13.0}
{'eval_loss': 0.23146237432956696, 'eval_runtime': 41.4637, 'eval_samples_per_second': 9.719, 'eval_steps_per_second': 1.23, 'epoch': 13.0}
{'loss': 0.1879, 'learning_rate': 1.5e-05, 'epoch': 14.0}
{'eval_loss': 0.216956228017807, 'eval_runtime': 41.4526, 'eval_samples_per_second': 9.722, 'eval_steps_per_second': 1.23, 'epoch': 14.0}
{'loss': 0.1742, 'learning_rate': 1.25e-05, 'epoch': 15.0}
{'eval_loss': 0.20527032017707825, 'eval_runtime': 41.4392, 'eval_samples_per_second': 9.725, 'eval_steps_per_second': 1.231, 'epoch': 15.0}
{'loss': 0.1624, 'learning_rate': 1e-05, 'epoch': 16.0}
{'eval_loss': 0.19351345300674438, 'eval_runtime': 41.402, 'eval_samples_per_second': 9.734, 'eval_steps_per_second': 1.232, 'epoch': 16.0}
{'loss': 0.1517, 'learning_rate': 7.5e-06, 'epoch': 17.0}
{'eval_loss': 0.1841140240430832, 'eval_runtime': 41.3718, 'eval_samples_per_second': 9.741, 'eval_steps_per_second': 1.233, 'epoch': 17.0}
{'loss': 0.1428, 'learning_rate': 5e-06, 'epoch': 18.0}
{'eval_loss': 0.17686299979686737, 'eval_runtime': 41.4581, 'eval_samples_per_second': 9.721, 'eval_steps_per_second': 1.23, 'epoch': 18.0}
{'loss': 0.1356, 'learning_rate': 2.5e-06, 'epoch': 19.0}
{'eval_loss': 0.17180992662906647, 'eval_runtime': 41.4578, 'eval_samples_per_second': 9.721, 'eval_steps_per_second': 1.23, 'epoch': 19.0}
{'loss': 0.1301, 'learning_rate': 0.0, 'epoch': 20.0}
{'eval_loss': 0.16986818611621857, 'eval_runtime': 41.4657, 'eval_samples_per_second': 9.719, 'eval_steps_per_second': 1.23, 'epoch': 20.0}
{'train_runtime': 11767.2453, 'train_samples_per_second': 3.418, 'train_steps_per_second': 3.418, 'train_loss': 0.30853292274569943, 'epoch': 20.0}
100%|██████████████████████████████████████████████████████████████████████████| 40220/40220 [3:16:07<00:00,  3.42it/s]
Training done at: 2024-01-04 17:50:15.380360


{
  "best_metric": 0.16986818611621857,
  "best_model_checkpoint": "./Lora-Meta-Llama2-7b-hf-QandA_2g_v01-v03\\checkpoint-40220",
  "epoch": 20.0,
  "eval_steps": 500,
  "global_step": 40220,
  "is_hyper_param_search": false,
  "is_local_process_zero": true,
  "is_world_process_zero": true,
  "log_history": [
    {
      "epoch": 1.0,
      "learning_rate": 4.75e-05,
      "loss": 0.7906,
      "step": 2011
    },
    {
      "epoch": 1.0,
      "eval_loss": 0.7224752902984619,
      "eval_runtime": 41.5025,
      "eval_samples_per_second": 9.71,
      "eval_steps_per_second": 1.229,
      "step": 2011
    },
    {
      "epoch": 2.0,
      "learning_rate": 4.5e-05,
      "loss": 0.6145,
      "step": 4022
    },
    {
      "epoch": 2.0,
      "eval_loss": 0.6293410062789917,
      "eval_runtime": 41.4281,
      "eval_samples_per_second": 9.728,
      "eval_steps_per_second": 1.231,
      "step": 4022
    },
    {
      "epoch": 3.0,
      "learning_rate": 4.25e-05,
      "loss": 0.5393,
      "step": 6033
    },
    {
      "epoch": 3.0,
      "eval_loss": 0.5556005239486694,
      "eval_runtime": 41.4426,
      "eval_samples_per_second": 9.724,
      "eval_steps_per_second": 1.231,
      "step": 6033
    },
    {
      "epoch": 4.0,
      "learning_rate": 4e-05,
      "loss": 0.4774,
      "step": 8044
    },
    {
      "epoch": 4.0,
      "eval_loss": 0.5019810199737549,
      "eval_runtime": 41.4936,
      "eval_samples_per_second": 9.712,
      "eval_steps_per_second": 1.229,
      "step": 8044
    },
    {
      "epoch": 5.0,
      "learning_rate": 3.7500000000000003e-05,
      "loss": 0.4244,
      "step": 10055
    },
    {
      "epoch": 5.0,
      "eval_loss": 0.4469846785068512,
      "eval_runtime": 41.5516,
      "eval_samples_per_second": 9.699,
      "eval_steps_per_second": 1.227,
      "step": 10055
    },
    {
      "epoch": 6.0,
      "learning_rate": 3.5e-05,
      "loss": 0.3797,
      "step": 12066
    },
    {
      "epoch": 6.0,
      "eval_loss": 0.4054427742958069,
      "eval_runtime": 41.445,
      "eval_samples_per_second": 9.724,
      "eval_steps_per_second": 1.231,
      "step": 12066
    },
    {
      "epoch": 7.0,
      "learning_rate": 3.2500000000000004e-05,
      "loss": 0.3424,
      "step": 14077
    },
    {
      "epoch": 7.0,
      "eval_loss": 0.3668522238731384,
      "eval_runtime": 41.438,
      "eval_samples_per_second": 9.725,
      "eval_steps_per_second": 1.231,
      "step": 14077
    },
    {
      "epoch": 8.0,
      "learning_rate": 3e-05,
      "loss": 0.3111,
      "step": 16088
    },
    {
      "epoch": 8.0,
      "eval_loss": 0.34070536494255066,
      "eval_runtime": 41.4578,
      "eval_samples_per_second": 9.721,
      "eval_steps_per_second": 1.23,
      "step": 16088
    },
    {
      "epoch": 9.0,
      "learning_rate": 2.7500000000000004e-05,
      "loss": 0.2837,
      "step": 18099
    },
    {
      "epoch": 9.0,
      "eval_loss": 0.3126145303249359,
      "eval_runtime": 41.4312,
      "eval_samples_per_second": 9.727,
      "eval_steps_per_second": 1.231,
      "step": 18099
    },
    {
      "epoch": 10.0,
      "learning_rate": 2.5e-05,
      "loss": 0.2599,
      "step": 20110
    },
    {
      "epoch": 10.0,
      "eval_loss": 0.28896018862724304,
      "eval_runtime": 41.4486,
      "eval_samples_per_second": 9.723,
      "eval_steps_per_second": 1.23,
      "step": 20110
    },
    {
      "epoch": 11.0,
      "learning_rate": 2.25e-05,
      "loss": 0.2393,
      "step": 22121
    },
    {
      "epoch": 11.0,
      "eval_loss": 0.26615414023399353,
      "eval_runtime": 41.4467,
      "eval_samples_per_second": 9.723,
      "eval_steps_per_second": 1.23,
      "step": 22121
    },
    {
      "epoch": 12.0,
      "learning_rate": 2e-05,
      "loss": 0.2203,
      "step": 24132
    },
    {
      "epoch": 12.0,
      "eval_loss": 0.24929480254650116,
      "eval_runtime": 41.4549,
      "eval_samples_per_second": 9.721,
      "eval_steps_per_second": 1.23,
      "step": 24132
    },
    {
      "epoch": 13.0,
      "learning_rate": 1.75e-05,
      "loss": 0.2031,
      "step": 26143
    },
    {
      "epoch": 13.0,
      "eval_loss": 0.23146237432956696,
      "eval_runtime": 41.4637,
      "eval_samples_per_second": 9.719,
      "eval_steps_per_second": 1.23,
      "step": 26143
    },
    {
      "epoch": 14.0,
      "learning_rate": 1.5e-05,
      "loss": 0.1879,
      "step": 28154
    },
    {
      "epoch": 14.0,
      "eval_loss": 0.216956228017807,
      "eval_runtime": 41.4526,
      "eval_samples_per_second": 9.722,
      "eval_steps_per_second": 1.23,
      "step": 28154
    },
    {
      "epoch": 15.0,
      "learning_rate": 1.25e-05,
      "loss": 0.1742,
      "step": 30165
    },
    {
      "epoch": 15.0,
      "eval_loss": 0.20527032017707825,
      "eval_runtime": 41.4392,
      "eval_samples_per_second": 9.725,
      "eval_steps_per_second": 1.231,
      "step": 30165
    },
    {
      "epoch": 16.0,
      "learning_rate": 1e-05,
      "loss": 0.1624,
      "step": 32176
    },
    {
      "epoch": 16.0,
      "eval_loss": 0.19351345300674438,
      "eval_runtime": 41.402,
      "eval_samples_per_second": 9.734,
      "eval_steps_per_second": 1.232,
      "step": 32176
    },
    {
      "epoch": 17.0,
      "learning_rate": 7.5e-06,
      "loss": 0.1517,
      "step": 34187
    },
    {
      "epoch": 17.0,
      "eval_loss": 0.1841140240430832,
      "eval_runtime": 41.3718,
      "eval_samples_per_second": 9.741,
      "eval_steps_per_second": 1.233,
      "step": 34187
    },
    {
      "epoch": 18.0,
      "learning_rate": 5e-06,
      "loss": 0.1428,
      "step": 36198
    },
    {
      "epoch": 18.0,
      "eval_loss": 0.17686299979686737,
      "eval_runtime": 41.4581,
      "eval_samples_per_second": 9.721,
      "eval_steps_per_second": 1.23,
      "step": 36198
    },
    {
      "epoch": 19.0,
      "learning_rate": 2.5e-06,
      "loss": 0.1356,
      "step": 38209
    },
    {
      "epoch": 19.0,
      "eval_loss": 0.17180992662906647,
      "eval_runtime": 41.4578,
      "eval_samples_per_second": 9.721,
      "eval_steps_per_second": 1.23,
      "step": 38209
    },
    {
      "epoch": 20.0,
      "learning_rate": 0.0,
      "loss": 0.1301,
      "step": 40220
    },
    {
      "epoch": 20.0,
      "eval_loss": 0.16986818611621857,
      "eval_runtime": 41.4657,
      "eval_samples_per_second": 9.719,
      "eval_steps_per_second": 1.23,
      "step": 40220
    }
  ],
  "logging_steps": 500,
  "max_steps": 40220,
  "num_train_epochs": 20,
  "save_steps": 500,
  "total_flos": 2.344189529530368e+17,
  "trial_name": null,
  "trial_params": null
}

Model Details

Model Description

Developed by: [More Information Needed]
Funded by [optional]: [More Information Needed]
Shared by [optional]: [More Information Needed]
Model type: [More Information Needed]
Language(s) (NLP): [More Information Needed]
License: [More Information Needed]
Finetuned from model [optional]: [More Information Needed]

Model Sources [optional]

Repository: [More Information Needed]
Paper [optional]: [More Information Needed]
Demo [optional]: [More Information Needed]

Uses

Direct Use

[More Information Needed]

Downstream Use [optional]

[More Information Needed]

Out-of-Scope Use

[More Information Needed]

Bias, Risks, and Limitations

[More Information Needed]

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

Use the code below to get started with the model.

[More Information Needed]

Training Details

Training Data

[More Information Needed]

Training Procedure

Preprocessing [optional]

[More Information Needed]

Training Hyperparameters

Training regime: [More Information Needed]

Speeds, Sizes, Times [optional]

[More Information Needed]

Evaluation

Testing Data, Factors & Metrics

Testing Data

[More Information Needed]

Factors

[More Information Needed]

Metrics

[More Information Needed]

Results

[More Information Needed]

Summary

Model Examination [optional]

[More Information Needed]

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

Hardware Type: [More Information Needed]
Hours used: [More Information Needed]
Cloud Provider: [More Information Needed]
Compute Region: [More Information Needed]
Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

[More Information Needed]

Compute Infrastructure

[More Information Needed]

Hardware

[More Information Needed]

Software

[More Information Needed]

Citation [optional]

BibTeX:

[More Information Needed]

APA:

[More Information Needed]

Glossary [optional]

[More Information Needed]

More Information [optional]

[More Information Needed]

Model Card Authors [optional]

[More Information Needed]

Model Card Contact

[More Information Needed]

Training procedure

The following bitsandbytes quantization config was used during training:

quant_method: bitsandbytes
load_in_8bit: False
load_in_4bit: True
llm_int8_threshold: 6.0
llm_int8_skip_modules: None
llm_int8_enable_fp32_cpu_offload: False
llm_int8_has_fp16_weight: False
bnb_4bit_quant_type: nf4
bnb_4bit_use_double_quant: True
bnb_4bit_compute_dtype: float16

Framework versions

PEFT 0.6.2

Training procedure

The following bitsandbytes quantization config was used during training:

quant_method: bitsandbytes
load_in_8bit: False
load_in_4bit: True
llm_int8_threshold: 6.0
llm_int8_skip_modules: None
llm_int8_enable_fp32_cpu_offload: False
llm_int8_has_fp16_weight: False
bnb_4bit_quant_type: nf4
bnb_4bit_use_double_quant: True
bnb_4bit_compute_dtype: float16

Framework versions

PEFT 0.6.2

Training procedure

The following bitsandbytes quantization config was used during training:

quant_method: bitsandbytes
load_in_8bit: False
load_in_4bit: True
llm_int8_threshold: 6.0
llm_int8_skip_modules: None
llm_int8_enable_fp32_cpu_offload: False
llm_int8_has_fp16_weight: False
bnb_4bit_quant_type: nf4
bnb_4bit_use_double_quant: True
bnb_4bit_compute_dtype: float16

Framework versions

PEFT 0.6.2

daochf/Lora-Meta-Llama2-7b-hf-QandA_2g_v01-v03

Model Card for Model ID

Model Details

Model Description

Model Sources [optional]

Uses

Direct Use

Downstream Use [optional]

Out-of-Scope Use

Bias, Risks, and Limitations

Recommendations

How to Get Started with the Model

Training Details

Training Data

Training Procedure

Preprocessing [optional]

Training Hyperparameters

Speeds, Sizes, Times [optional]

Evaluation

Testing Data, Factors & Metrics

Testing Data

Factors

Metrics

Results

Summary

Model Examination [optional]

Environmental Impact

Technical Specifications [optional]

Model Architecture and Objective

Compute Infrastructure

Hardware

Software

Citation [optional]

Glossary [optional]

More Information [optional]

Model Card Authors [optional]

Model Card Contact

Training procedure

Framework versions

Training procedure

Framework versions

Training procedure

Framework versions

daochf/
Lora-Meta-Llama2-7b-hf-QandA_2g_v01-v03