dobrien/

ViT-B-32-MNIST-dummy-RESISC45-1e-0-arithmetic-split

    ## Dataset Location: ylecun/mnist
    ## Dummy Dataset: RESISC45
    ## Dummy Dataset Location: tanganke/resisc45
    ## Loss Term: 1e-0
    ## Merge Method: arithmetic
    ## Test-Set Accuracy: 0.996666669845581
    ## Test-Set Loss: 0.01834259746952894
    ## Training Hyperparameters:
    - Seed: 599121577
    - Device: cuda
    - Machine: L40S
    - Number of Devices: 1
    - Precision: Float32
    - Gradient Norm Clip Value: 10.0
    - Accumulate Gradient Batches: 1
    - Batch Size: 32
    - Loss: Cross Entropy
    - Epochs: 5
    - Optimizer: AdamW
    - Learning Rate: 1e-5
    - Weight Decay: 0.1
    - Scheduler: CosineAnnealingLR
    - Train Split Percent: 90%
    - Validation Split Percent: 10%
    - Test Split Percent: 100%
    ## Method:
    1. Compute standard loss on target dataset using current model.
    2. Merge current model with fine-tuned dummy dataset model through arithmetic, with a coefficient of 0.5.
    3. Compute dummy loss on dummy dataset using merged encoder.
    4. Multiply the dummy loss by the (1e-0) and add it to the standard loss.
    5. Backpropagate based on new loss.