• Model creator: https://huggingface.co/pszemraj/
• Original model: https://huggingface.co/pszemraj/pythia-31m-goodwiki-deduped-2048-scratch/

Original model description:

• generated_from_trainer metrics:
• accuracy inference: parameters: max_new_tokens: 64 do_sample: true repetition_penalty: 1.1 no_repeat_ngram_size: 5 guidance_scale: 1.01 eta_cutoff: 0.001 widget:
• text: My name is El Microondas the Wise and example_title: El Microondas
• text: A meme is example_title: meme
• text: >- Barack Obama nominated Hilary Clinton as his secretary of state on Monday. He chose her because she had example_title: Coreference resolution
• text: >- On a shelf, there are five books: a gray book, a red book, a purple book, a blue book, and a black book example_title: Logic puzzles
• text: >- The two men running to become New York City's next mayor will face off in their first debate Wednesday night example_title: Reading comprehension pipeline_tag: text-generation license: apache-2.0 datasets:
• euirim/goodwiki language:
• en

pythia-31m-goodwiki-deduped-2048-scratch

• Loss: 4.5181
• Accuracy: 0.2680

Model description

Intended uses & limitations

Training and evaluation data

***** eval metrics *****                                              
  epoch                   =        3.0                   
  eval_accuracy           =     0.2694                                  eval_loss               =     4.4986                                
  eval_runtime            = 0:00:14.62                                
  eval_samples            =        500                                  eval_samples_per_second =     34.187                                  eval_steps_per_second   =     17.093                              
  perplexity              =    89.8934

Training procedure

Training hyperparameters

• learning_rate: 0.0005
• train_batch_size: 2
• eval_batch_size: 2
• seed: 80085
• gradient_accumulation_steps: 64
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.99) and epsilon=1e-07
• lr_scheduler_type: inverse_sqrt
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 3.0

Training results

Framework versions

• Transformers 4.33.1
• Pytorch 2.2.0.dev20230907+cu118
• Datasets 2.14.5
• Tokenizers 0.13.3

Open LLM Leaderboard Evaluation Results