dobrien/

ViT-B-32-DTD-dummy-TINet-1e-3-arithmetic-split

    ## Dataset Location: tanganke/dtd
    ## Dummy Dataset: TINet
    ## Dummy Dataset Location: zh-plus/tiny-imagenet
    ## Loss Term: 1e-3
    ## Merge Method: arithmetic
    ## Test-Set Accuracy: 0.804964542388916
    ## Test-Set Loss: 1.128464937210083
    ## Training Hyperparameters:
    - Seed: 599121577
    - Device: cuda
    - Machine: L40S
    - Number of Devices: 1
    - Precision: Float32
    - Gradient Norm Clip Value: 10.0
    - Accumulate Gradient Batches: 1
    - Batch Size: 32
    - Loss: Cross Entropy
    - Epochs: 76
    - Optimizer: AdamW
    - Learning Rate: 1e-5
    - Weight Decay: 0.1
    - Scheduler: CosineAnnealingLR
    - Train Split Percent: 90%
    - Validation Split Percent: 10%
    - Test Split Percent: 100%
    ## Method:
    1. Compute standard loss on target dataset using current model.
    2. Merge current model with fine-tuned dummy dataset model through arithmetic, with a coefficient of 0.5.
    3. Compute dummy loss on dummy dataset using merged encoder.
    4. Multiply the dummy loss by the (1e-3) and add it to the standard loss.
    5. Backpropagate based on new loss.