• Model creator: https://huggingface.co/Na0s/
• Original model: https://huggingface.co/Na0s/Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT/

Original model description:

• Open-Orca/SlimOrca pipeline_tag: text-generation base_model: Na0s/Llama-3.1-8b-Pruned-4-Layers

Model Card for Na0s/Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT

Model Details

Model Description

• Finetuned from model:[Na0s/Llama-3.1-8b-Pruned-4-Layers]

Training Details

    LoRA BF16, 
    batch_size=2, 
    steps=10000, gradient_accumulation_steps = 4,
    warmup_steps = 5,
    max_steps = 10000
    learning_rate = 2e-4,
    fp16 = not is_bfloat16_supported(),
    bf16 = is_bfloat16_supported(),
    logging_steps = 1,
    optim = "adamw_8bit",
    weight_decay = 0.01,
    lr_scheduler_type = "linear",
    seed = 3407

Training Data

Evaluation

Evaluation Data

Environmental Impact