Model Card for Model ID

Model Details

Model Description

• Developed by: [More Information Needed]
• Funded by [optional]: [More Information Needed]
• Shared by [optional]: [More Information Needed]
• Model type: [More Information Needed]
• Language(s) (NLP): [More Information Needed]
• License: [More Information Needed]
• Finetuned from model [optional]: [More Information Needed]

Model Sources [optional]

• Repository: [More Information Needed]
• Paper [optional]: [More Information Needed]
• Demo [optional]: [More Information Needed]

Uses

Direct Use

Downstream Use [optional]

Out-of-Scope Use

Bias, Risks, and Limitations

Recommendations

How to Get Started with the Model

Training Details

Training Data

Training Procedure

Preprocessing [optional]

Training Hyperparameters

model_args = ModelArguments(
    model_name="google-t5/t5-small",
    use_4bit=True,
    use_nested_quant=True,
    bnb_4bit_compute_dtype="bfloat16",
    bnb_4bit_quant_dtype="nf4",
    lora_alpha=128,
    lora_dropout=0.1,
    lora_r=8
)

script_args = ScriptArguments(
    per_device_train_batch_size=64,
    per_device_eval_batch_size=64,
    # auto_find_batch_size=True,
    gradient_accumulation_steps=4,
    learning_rate=2e-5,
    weight_decay=0.01,
    max_seq_length=512,
    dataset_name="4DR1455/finance_questions",
    bf16=True,
    optim='adafactor',
    lr_scheduler_type='cosine',
    packing=True,
    num_train_epochs=3,
    save_steps=20,
    logging_steps=10,
    eval_steps=20,
    warmup_steps=500,
    eval_strategy='steps',
    run_name="finetuning-llm-3",
    report_to="wandb",
    save_safetensors=False,
    label_names=['labels'],
    load_best_model_at_end=True,
    dataloader_num_workers=10,
)

Speeds, Sizes, Times [optional]

Evaluation

Testing Data, Factors & Metrics

Testing Data

Factors

Metrics

Results

Summary

Model Examination [optional]

Environmental Impact

• Hardware Type: [More Information Needed]
• Hours used: [More Information Needed]
• Cloud Provider: [More Information Needed]
• Compute Region: [More Information Needed]
• Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

Compute Infrastructure

Hardware

Software

Citation [optional]

Glossary [optional]

More Information [optional]

Model Card Authors [optional]

Model Card Contact