This model is gpjt/8xa100m40, a trained-from-scratch base model using
the GPT-2-style architecture from
Sebastian Raschka's book
"
Build a Large Language Model (from Scratch)".
Don't have high expectations for the model! It has only 163M parameters (the GPT-2 "small" size)
and was trained on roughly the Chinchilla-optimal number of tokens (~20x the number of parameters), which means that it doesn't know
many facts and is not terribly smart. If you want to do serious work, use a serious model (I like
Qwen's). But if you want to build on this and see what you can do with a 2020-vintage
LLM, please do feel free to play with it!
1from transformers import pipeline
2pipe = pipeline("text-generation", model="gpjt/8xa100m40", trust_remote_code=True)
3out = pipe(
4 "Every effort moves you",
5 max_new_tokens=20,
6 do_sample=True,
7 temperature=1.4,
8 top_k=25,
9)
10print(out[0]["generated_text"])
1>>> from transformers import AutoTokenizer, AutoModel, AutoModelForCausalLM
2>>> tokenizer = AutoTokenizer.from_pretrained("gpjt/8xa100m40")
3>>> model = AutoModel.from_pretrained("gpjt/8xa100m40", trust_remote_code=True)
4>>> llm_model = AutoModelForCausalLM.from_pretrained("gpjt/8xa100m40", trust_remote_code=True)
You can also fine-tune it;
this notebook has an example.
Again, don't expect too much from this model! It's a 163M-parameter GPT-2 one, trained on a limited
number of tokens. It's
both dumb and ignorant ;-)
