Training

deepspeed --include=node-0:3 --master_port=12001 sft_prompt_tuning.py --deepspeed dp_zero0.json \
    --model_name_or_path="guanaco_Llama-2-7b-chat-hf_freeze_embed_tokens_q_v_proj" \
    --dataset_name="timdettmers/openassistant-guanaco" \
    --dataset_text_field="text" \
    --report_to="tensorboard" \
    --learning_rate=1e-5 \
    --per_device_train_batch_size=32 \
    --gradient_accumulation_steps=4 \
    --output_dir="guanaco_Llama-2-7b-chat-hf_freeze_embed_tokens_q_v_projs_prompttuning" \
    --logging_steps=1 \
    --num_train_epochs=15 \
    --max_steps=-1 \
    --save_steps=0.3 \
    --gradient_checkpointing \
    --fp16

Model Card for Model ID

Model Details

Model Description

• Developed by: [More Information Needed]
• Funded by [optional]: [More Information Needed]
• Shared by [optional]: [More Information Needed]
• Model type: [More Information Needed]
• Language(s) (NLP): [More Information Needed]
• License: [More Information Needed]
• Finetuned from model [optional]: [More Information Needed]

Model Sources [optional]

• Repository: [More Information Needed]
• Paper [optional]: [More Information Needed]
• Demo [optional]: [More Information Needed]

Uses

Direct Use

Downstream Use [optional]

Out-of-Scope Use

Bias, Risks, and Limitations

Recommendations

How to Get Started with the Model

Training Details

Training Data

Training Procedure

Preprocessing [optional]

Training Hyperparameters

• Training regime: [More Information Needed]

Speeds, Sizes, Times [optional]

Evaluation

Testing Data, Factors & Metrics

Testing Data

Factors

Metrics

Results

Summary

Model Examination [optional]

Environmental Impact

• Hardware Type: [More Information Needed]
• Hours used: [More Information Needed]
• Cloud Provider: [More Information Needed]
• Compute Region: [More Information Needed]
• Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

Compute Infrastructure

Hardware

Software

Citation [optional]

Glossary [optional]

More Information [optional]

Model Card Authors [optional]

Model Card Contact

Framework versions

• PEFT 0.10.1.dev0