Views
No views yet
nv3[v0] (1700) | nv4[v1-2k] (4000) | nv4[v1-210k] (b1b2: 4000)google/siglip2-large-patch16-512 instead of dino v2 for a model difference (turns out 1% better than google/siglip2-base-patch16-512)..wandb: Run summary:
wandb: eval/accuracy 0.77533
wandb: eval/loss 0.4809
wandb: eval/runtime 15.9025
wandb: eval/samples_per_second 111.114
wandb: eval/steps_per_second 0.692
wandb: total_flos 1.4915777670524436e+20
wandb: train/epoch 10.0
wandb: train/global_step 570
wandb: train/grad_norm 375217.9375
wandb: train/learning_rate 0.0
wandb: train/loss 0.286
wandb: train_loss 0.40591
wandb: train_runtime 1032.5423
wandb: train_samples_per_second 96.974
wandb: train_steps_per_second 0.5521#!/bin/bash
2
3# =================== BEGIN NOTES =======================
4
5# BS24 ooms; bs18 66943MiB / 81559MiB; try bs22
6# bs22 (try to match siglip2-base for large as much as possible): 77679MiB / 81559MiB
7
8# ORIGINAL AUGMENTATION:
9# - model trained on this with exact config had eval/accuracy 0.77533
10
11# train_transforms = Compose([
12# RandomResizedCrop(size),
13# RandomHorizontalFlip(),
14# ToTensor(),
15# normalize,
16# ])
17
18# MODIFIED AUGMENTATION:
19
20# from torchvision.transforms import Compose, RandomResizedCrop, RandomRotation, RandomHorizontalFlip, ColorJitter, RandomApply, GaussianBlur, ToTensor
21
22# train_transforms = Compose([
23# RandomResizedCrop(size=224, scale=(0.8, 1.0), ratio=(0.9, 1.1)),
24# RandomRotation(5),
25# RandomHorizontalFlip(p=0.2),
26# ColorJitter(brightness=0.1, contrast=0.1, saturation=0.1, hue=0.05),
27# RandomApply([GaussianBlur(kernel_size=3, sigma=(0.5, 1.5))], p=0.1),
28# ToTensor(),
29# normalize,
30# ])
31
32
33# =================== END NOTES ==========================
34
35# Define variables
36BASE_MODEL="google/siglip2-large-patch16-512"
37DATASET="distill-lab/COMBINE_nai-distill_00-01_eagle.library"
38TASK="classification"
39NUM_EPOCHS=10
40
41
42# Run training command
43python -m trainlib.hf_trainer.cli \
44 --model_name_or_path $BASE_MODEL \
45 --dataset_name $DATASET \
46 --output_dir distill-n4_00-01_combined_cls_v1b2_classification_$BASE_MODEL \
47 --remove_unused_columns False \
48 --label_column_name star \
49 --task $TASK \
50 --do_train \
51 --do_eval \
52 --eval_strategy steps \
53 --eval_steps 100 \
54 --learning_rate 5e-6 \
55 --num_train_epochs $NUM_EPOCHS \
56 --per_device_train_batch_size 22 \
57 --per_device_eval_batch_size 22 \
58 --logging_strategy steps \
59 --logging_steps 2 \
60 --save_total_limit 1 \
61 --seed 1337 \
62 --lr_scheduler_type cosine \
63 --dataloader_num_workers 16 \
64 --ignore_mismatched_sizes True \
65 --fp16 True # EXTRA ARGUMENT
66