This model uses an 1.3 billion parameters model as base to be further fine-tuned on the same data as palmer. It works pretty good and even surpasses sota model on hellaswag.

This model was trained on less than 25% of the dataset yet achieves competitive performance to current sota on open llm leaderboard.

Training took ~5 P100 gpu hours. It was trained on 15,000 gpt-4 shuffled samples. no-prompt was fine-tuned using lower learning rates ensuring it keeps as much general knowledge as possible.

Hallucinations are frequent, just as any transformer model this size.