These results are retrieved from AzureML Studio's resource monitoring module. All experiments were ran on AzureML's low priority clusters.
*Compute cost is calculated from run duration and SKU's price per hour. Updated SKU pricing could be found here:
https://azure.microsoft.com/en-us/pricing/details/machine-learning/
*Peak memory usage is calculated from average peak across all utilized GPUs.
These results are obtained using
codecarbon. The carbon emission is estimated from training runtime only (excluding setup and evaluation runtime).
CodeCarbon:
https://github.com/mlco2/codecarbon1fp16: True
2per device batch size: 8
3effective batch size: 64
4epoch: 3.0
5learning rate: 1e-4
6weight decay: 0.1
7seed: 1
1 "zero_optimization": {
2 "stage": 2,
3 "allgather_partitions": true,
4 "allgather_bucket_size": 1300000000,
5 "overlap_comm": true,
6 "reduce_scatter": true,
7 "reduce_bucket_size": 1300000000,
8 "contiguous_gradients": true
9 }
1from transformers import pipeline
2summarizer = pipeline("summarization", model="henryu-lin/t5-large-samsum-deepspeed")
3
4conversation = '''Kevin: Hey man, are you excited to watch Finding Nemo tonight?
5 Henry: Yea, I can't wait to watch that same movie for the 89th time. Is Nate coming over to watch it with us tonight?
6 Kevin: Yep, he said he'll be arriving a bit later at around 7 since he gets off of work at 6. Have you taken out the garbage yet? It's starting to make the kitchen really smell.
7 Henry: Oh I forgot. I'll do that once I'm finished with my assignment for my math class. I didn't get to start on it until an hour ago, and it's due in 30 minutes.
8 Kevin: Okay dude, you should take it out as soon as possible. By the way, Nate is bringing his girlfriend and their cat too.
9 Henry: Nice, I'm really looking forward to seeing them again.
10'''
11summarizer(conversation)
