Quantization made by Richard Erkhov.

TinyDolphin-2.8.2-1.1b-laser - bnb 4bits

license: apache-2.0 datasets:

Join Our Discord! https://discord.gg/cognitivecomputations

This is an version 3 of a model trained on 3 3090's by Kearm on the new Dolphin 2.8 dataset by Eric Hartford https://erichartford.com/dolphin 🐬

This model uses our laser technique from https://github.com/cognitivecomputations/laserRMT to denoise the model!

For this version we increased the epochs as well as refined the datasets used.

TBD

Support my efforts! https://ko-fi.com/kearm

The TinyLlama project aims to pretrain a 1.1B Llama model on 3 trillion tokens. With some proper optimization, we can achieve this within a span of "just" 90 days using 16 A100-40G GPUs 🚀🚀. The training has started on 2023-09-01.

We adopted exactly the same architecture and tokenizer as Llama 2. This means TinyLlama can be plugged and played in many open-source projects built upon Llama. Besides, TinyLlama is compact with only 1.1B parameters. This compactness allows it to cater to a multitude of applications demanding a restricted computation and memory footprint.

This collection contains all checkpoints after the 1T fix. Branch name indicates the step and number of tokens seen.