Model Card: Uplimit Project 1 part 1
Model Description:
This is a model to test run publishing models. It has no real model assessment value.
This is a Large Language Model (LLM) trained on a dataset of DIBT/10k_prompts_ranked.
It was evaluated using using Eleuther Evaluation Harness
Hellaswag
Passed argument batch_size = auto:4.0. Detecting largest batch size
Determined largest batch size: 64
Passed argument batch_size = auto:4.0. Detecting largest batch size
Determined largest batch size: 64
hf (pretrained=EleutherAI/pythia-160m,revision=step100000,dtype=float), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto:4 (64,64,64,64,64)
| Tasks | Version | Filter | n-shot | Metric | | Value | | Stderr |
|---|
| hellaswag | 1 | none | 0 | acc | ↑ | 0.2872 | ± | 0.0045 |
| | none | 0 | acc_norm | ↑ | 0.3082 | ± | 0.0046 |
Interpretation (curtesy for Prepelexit.ai):
Accuracy Metrics:
Standard accuracy: 0.2872 (28.72%)
Normalized accuracy: 0.3082 (30.82%)
Context:
The HellaSwag task is a challenging commonsense reasoning benchmark that tests a model's ability to complete sentences or scenarios in a sensible way.
The task is considered difficult even for larger language models.
Interpretation
Baseline Performance: The model achieves an accuracy of 28.72% on the standard HellaSwag task, which is significantly above random guessing (25% for a 4-way multiple choice task)1.
Normalized Performance: The normalized accuracy of 30.82% is slightly higher than the standard accuracy, suggesting that the model performs marginally better when accounting for potential biases in the task1.
Model Size Consideration: Given that Pythia 160M is a relatively small language model (160 million parameters), these results are not unexpected2.
Comparative Analysis: While not directly comparable without benchmarks from other models, this performance is likely lower than what larger models (e.g., GPT-3, PaLM) would achieve on the same task2.
Learning Progress: As this is an intermediate checkpoint (step 100000), it's possible that the model's performance could improve with further training
