peft_config = LoraConfig(
r=4, # TODO: play with this number
lora_alpha=8, # TODO: play with this number
target_modules=['q_proj', 'v_proj', 'k_proj'],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM" # TODO: you need to figure this out. HINT https://github.com/huggingface/peft/blob/3d2bf9a8b261ed2960f26e61246cf0aa624a6115/src/peft/utils/peft_types.py#L67
)
training_args = TrainingArguments(
per_device_train_batch_size=2,
gradient_accumulation_steps=2,
gradient_checkpointing =False,
max_grad_norm= 0.3,
num_train_epochs=2, # TODO: play with this number
save_steps= 100,
learning_rate=0.0005, # TODO: play with this number
bf16=True,
save_total_limit=3,
logging_steps=10,
output_dir='./sft_models',
optim="adamw_torch",
lr_scheduler_type="cosine",
warmup_ratio=0.05,
remove_unused_columns=False,
report_to="none",
)
generate_max_length: int = 64
Model Description
This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
Developed by: [More Information Needed]
Funded by [optional]: [More Information Needed]
Shared by [optional]: [More Information Needed]
Model type: [More Information Needed]
Language(s) (NLP): [More Information Needed]
License: [More Information Needed]
Finetuned from model [optional]: [More Information Needed]
Model Sources [optional]
Repository: [More Information Needed]
Paper [optional]: [More Information Needed]
Demo [optional]: [More Information Needed]
Uses
Direct Use
[More Information Needed]
Downstream Use [optional]
[More Information Needed]
Out-of-Scope Use
[More Information Needed]
Bias, Risks, and Limitations
[More Information Needed]
Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
