Uploaded model

• Developed by: Paul Chbat
• License: apache-2.0
• Finetuned from model : unsloth/llama-3.2-3b-instruct-unsloth-bnb-4bit

Explanation of Key Fine-Tuning Parameters

Initial Configuration & Model Loading

• max_seq_length:
  • Value: 2048
  • Description: Defines the maximum number of tokens the model can process in a single input sequence during training and inference. Unsloth automatically handles RoPE scaling if the chosen length exceeds the model's original training length.
• dtype:
  • Value: None (auto-detection)
  • Description: Specifies the data type for model computations (e.g., float16, bfloat16). None allows Unsloth to automatically select the best supported dtype based on the available hardware (BF16 for Ampere GPUs and newer, FP16 otherwise).
• load_in_4bit:
  • Value: True
  • Description: Enables 4-bit quantization using bitsandbytes. This significantly reduces the model's memory footprint (VRAM usage), allowing larger models to be run on hardware with limited memory, at the cost of a minor potential precision loss. Setting it to False loads the model in its default precision (FP16/BF16).
• model_name (within FastLanguageModel.from_pretrained):
  • Value: "unsloth/Llama-3.2-3B-Instruct"
  • Description: The identifier of the pre-trained model to be loaded from the Hugging Face Hub or a local path. This is the base model that will be fine-tuned. Unsloth provides optimized 4-bit versions of many popular models.

PEFT (LoRA) Configuration (FastLanguageModel.get_peft_model)

• r:
  • Value: 32
  • Description: The rank (dimension) of the low-rank adaptation (LoRA) matrices. A higher rank allows the LoRA adapter to capture more complex patterns but increases the number of trainable parameters and memory usage. Common values are 8, 16, 32, 64, 128.
• target_modules:
  • Value: ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
  • Description: A list of the specific modules/layers within the base model where the LoRA matrices (adapters) will be injected. These typically target the attention mechanism's projection layers and feed-forward network layers.
• lora_alpha:
  • Value: 32
  • Description: The scaling factor for the LoRA adaptations. It acts as a learning rate for the LoRA weights. A common practice is to set lora_alpha equal to r or double r.
• lora_dropout:
  • Value: 0
  • Description: The dropout probability applied to the LoRA layers. Dropout is a regularization technique to prevent overfitting. Unsloth optimizes training when dropout is set to 0.
• bias:
  • Value: "none"
  • Description: Specifies whether to train biases within the LoRA layers ("all", "lora_only") or not ("none"). "none" is often preferred for LoRA and is optimized by Unsloth.
• use_gradient_checkpointing:
  • Value: "unsloth"
  • Description: A technique to save VRAM during training by recomputing activations during the backward pass instead of storing them. Setting it to "unsloth" uses Unsloth's optimized version, which significantly reduces memory usage, especially for long sequences, allowing for larger batch sizes. True uses the standard Hugging Face implementation.

Training Configuration (SFTTrainer and TrainingArguments)

• train_dataset:
  • Value: dataset (variable assumed to hold the training data)
  • Description: The dataset object (e.g., from the datasets library) containing the data used for fine-tuning.
• dataset_text_field:
  • Value: "text"
  • Description: The name of the column/field within the train_dataset that contains the text data to be used for training.
• packing:
  • Value: False
  • Description: If set to True, multiple short sequences from the dataset are concatenated and packed into a single max_seq_length input. This can drastically speed up training (up to 5x) if the dataset consists mainly of short sequences, by improving GPU utilization. It's disabled here.
• per_device_train_batch_size (within TrainingArguments):
  • Value: 2
  • Description: The number of training examples processed by each GPU in a single forward/backward pass. Adjust based on available VRAM.
• gradient_accumulation_steps (within TrainingArguments):
  • Value: 4
  • Description: The number of steps over which gradients are accumulated before performing an optimizer step and clearing the gradients. This effectively simulates a larger batch size (per_device_train_batch_size * gradient_accumulation_steps * num_gpus) without increasing VRAM usage proportionally.
• warmup_steps (within TrainingArguments):
  • Value: 5
  • Description: The number of initial training steps during which the learning rate increases linearly from 0 to the specified learning_rate. This helps stabilize training early on.
• max_steps (within TrainingArguments):
  • Value: 100
  • Description: The total number of training optimization steps to perform. This determines the training duration. If num_train_epochs is set, max_steps overrides it.
• learning_rate (within TrainingArguments):
  • Value: 2e-4 (or 0.0002)
  • Description: The initial learning rate for the optimizer. Controls the step size during gradient descent. It's a crucial hyperparameter for successful convergence.
• fp16 (within TrainingArguments):
  • Value: not is_bfloat16_supported()
  • Description: Enables automatic mixed-precision training using float16. This speeds up training and reduces memory usage on GPUs that support it well (like Tesla T4, V100). Disabled if bf16 is enabled/supported.
• bf16 (within TrainingArguments):
  • Value: is_bfloat16_supported()
  • Description: Enables automatic mixed-precision training using bfloat16. Offers better numerical stability than fp16 for training deep learning models and is well-supported on newer GPUs (Ampere architecture and later). Enabled if supported by the hardware.
• logging_steps (within TrainingArguments):
  • Value: 1
  • Description: How often (in terms of training steps) to log training metrics like loss, learning rate, etc.
• optim (within TrainingArguments):
  • Value: "adamw_8bit"
  • Description: Specifies the optimizer to use. adamw_8bit is a version of the AdamW optimizer that uses 8-bit quantization for its states, further reducing memory usage during training.
• weight_decay (within TrainingArguments):
  • Value: 0.01
  • Description: Applies L2 regularization (weight decay) to the model's weights (excluding bias and LayerNorm parameters) to help prevent overfitting.
• lr_scheduler_type (within TrainingArguments):
  • Value: "linear"
  • Description: Defines how the learning rate changes over the course of training after the initial warmup phase. "linear" decays the learning rate linearly from its initial value down to 0 over the max_steps.
• seed (within TrainingArguments):
  • Value: 3407
  • Description: Sets the random seed for various components (like data shuffling, model initialization if applicable) to ensure reproducibility of the training run.
• output_dir (within TrainingArguments):
  • Value: "outputs"
  • Description: The directory where the training outputs (like model checkpoints, logs, and potentially the final adapter model) will be saved.

Training Result

• Loss at Final Step: The training loss observed at the final epoch (100) for a run using these parameters was 0.708600.