Views
No views yet
| Parameter | Value |
|---|---|
| Model | model |
| Tokenizer | tokenizer |
| Train Dataset | dataset |
| Dataset Text Field | text |
| Max Sequence Length | max_seq_length |
| Dataset Number of Processes | 2 |
| Packing | False (Can make training 5x faster for short sequences.) |
| Training Arguments | |
| - Per Device Train Batch Size | 2 |
| - Gradient Accumulation Steps | 4 |
| - Warmup Steps | 5 |
| - Number of Train Epochs | 1 (Set this for 1 full training run.) |
| - Max Steps | 60 |
| - Learning Rate | 2e-4 |
| - FP16 | not is_bfloat16_supported() |
| - BF16 | is_bfloat16_supported() |
| - Logging Steps | 1 |
| - Optimizer | adamw_8bit |
| - Weight Decay | 0.01 |
| - LR Scheduler Type | linear |
| - Seed | 3407 |
| - Output Directory | outputs |