1. Started smaller. This model is 4x1b, with the positive and negative prompts from the original 8x1b doubled up. Layer 1 is the only layer underutilized in this setup. I did not try to give it a random mask because I am unsure if my method was functional.
2. Decreased learning rate. The loss on the previous CosMoE had sharp spikes, suggesting overshooting appropriate solutions. Here I started with a learning rate of 1/4 the rate used on the smaller model.
3. Increased batch size. Thought this also might be helpful in smoothing out learning.
4. Avoided loading in 8-bit since I am having issues with that currently.

• Training and validation loss is a bit higher than on the small model; not quite as fit.
• Train/loss is much less spiky than prior attempt. There's one modest spike to 2.225 at around 25% through the epoch, and otherwise it's not very spiky at all.
• GPU memory utilization was at about 30% of an 80GB GPU. Hardware was probably overkill.

lora-out-2

• Loss: 1.0984

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 1

Training results

Framework versions

• PEFT 0.9.0
• Transformers 4.39.0.dev0
• Pytorch 2.1.2+cu118
• Datasets 2.18.0
• Tokenizers 0.15.0