Mixtress 135M is a transformer model based upon the
Mixtral architecture. It is the culmination of approximately 20 weeks of
Kaggle free hours, and 67 twelve-hour training runs.
This model was trained for 2.15 billion tokens over 20,000 optimizer steps. It was trained as a masked autoregressive language model, using cross-entropy loss.
The final train loss was 1.941, validation loss was 2.206, and perplexity was 9.136.
Mixtress was pre-trained and fine-tuned simultaneously. Full reproduction code may be found
at this URL, or in the Jupyter notebook
in this repository.
The model is best at what it was pretrained for, which is generating conversational text and answering questions from a prompt.
You can use this model directly with a pipeline for text generation. This example generates a different sequence each time it's run:
1>>> from transformers import pipeline
2>>> generator = pipeline('text-generation', model='UNSAFE/Mixtress-135M')
3>>> generator("In a shocking finding, ", do_sample=True, temperature=0.7, min_length=50)
4
5[{'generated_text': 'In a shocking finding, 20 years ago, U.S. President Donald Trump'}]
All evaluations were done using the
Pythia evaluation harness.
If you would like to chat with us, please join the
Discord server!
