An instruction-following large language model based on pythia-70m and trained on Databricks' 15k instruction
with capability domains from the InstructGPT paper, including brainstorming, classification, closed QA, generation, information extraction, open QA and summarization.
This model is an experiment in using small base model (pythia-70m) to build a model similar to Databricks' dolly model.
Usage
To use the model with the transformers library, first make sure you have the transformers and accelerate libraries installed :
1import torch
2from transformers import pipeline
34generate_text = pipeline(model="databricks/dolly-v2-12b", torch_dtype=torch.bfloat16, trust_remote_code=True, device_map="auto")56res = generate_text("What is the capital of France ?")78print(res[0]["generated_text"])9
Training
The model was trained using Databricks' 15k instruction on a recent Dell PC with 32G of RAM with a core i7 CPU.
The training took around 12 hours !
When prompted with What is the capital of France ?, the model answers with :
"The World". It is an artwork for "working time" called «The Middle East Today". It comes from Paris, Belgium, in local variation, including large cities as described in English language photographs which portray a crescent and sunrise of late note, Bangourt before Paris.
“Countries like Pakistan and throughout East Africa close to Australia have constructed a watered havock which can be felt ever longer. Bombardment and booby traps tend to occupy space by wind and water, as were effectively used for material and equipment which have a green signal leading in the images."
The accuracy between the base model used in this model (pythia-70m) and the base models used by Databricks (pythia-2.8b and pythia-12b) is huge ! And it makes all the difference in terms of accuracy.
The only thing worth mentioning here is the model's size, at around 160M it's orders of magnitude smaller than the Databricks ones.
