Violet 160M is a GPT-NeoX base model built to provide text completions for Victorian era text. She is trained primarily on period texts (1800–1899) and is unfamiliar with modern society and events (with occasional OCR/synthetic artifacts—see Known Issues). Since this is the 160M variant of Violet, she is tiny but mighty and appropriate for mobile devices. For completions, Violet 160M is almost as good as Violet 1b4. Violet 160M consistently punches above her paygrade in terms of output providing rich depth of narrative prose, with albeit somewhat simplistic reasoning. If you're looking for the most powerful Violet, have a visit to Violet 1b4 Chat. Violet 160M provided some a portion of the synthetic data corpus for Violet 1B4 training.
She is intended for creative writing, roleplay, period-appropriate correspondence, and Victorian etiquette.
Architecture: GPTNeoXForCausalLM
Parameters: ~152M
Context length: 4096
Vocab size: 50281
Tokenizer: GPTNeoXTokenizer
Intended Use
Good for
Victorian-flavored conversation and letters
Descriptive prose, scene writing, etiquette Q&A
Not good for
Contemporary factual Q&A
Medical/legal/financial advice
Known Issues / Limitations
The 160M model series are very brittle in chat, but are very good at text completion in general.
Punctuation for the 160M text completion model are variable at best albeit mostly conformant; this is likely due to variable differences seen from OCR sources. If used programmatically, some post-processing will almost certainly be required.
Ages and dates are unreliable (even within 1800–1899).
Because parts of the corpus were derived from OCR, occasional stray modern tokens may appear (e.g., “http”, “Google”, “Internet Archive”).
Training data includes UK and US English from the era.
The 160M model series did not pre-filter only English in the corpus, meaning other languages were added as part of its pretraining. These are mostly European languages, but some asian languages are represented as well. In practice this is mostly a curiosity as Violet 160M can perform some basic text completions in French.
Outputs may reflect historical biases typical of the Victorian era. Use discretion.
Notes
Violet is not the first LLM trained on a historical-only pretraining corpus; to the author’s knowledge that distinction belongs to TimeCapsuleLLM. Violet was developed independently, and differs in:
The sizes of the models
The range of dates used in pretraining
Different (albeit in some cases certainly overlapping) pretraining data
Violet 160M was built on a corpus spanning 1800–1899 mostly sourced from Project Gutenberg. Consequently she has a very strong narrative presence and punches above her model size in terms of quality of text.
This project began as an attempt to build a local LLM without relying on copyrighted training sources. The author also values local models that can run on a user’s machine without sending data to the cloud.
Both of these are intended to use WebGPU and run local on your system -- No data is sent to the cloud.
Related repos
Zakarth/violet-160m-chat (Fine tuned 160M model for chat. Brittle, but fun)
Zakarth/violet-1b4 (base/completion for 1.4B model)
Zakarth/violet-1b4-chat (The most powerful "Violet" chat model)
Prompt Format
Use this structure:
{input...}
The model will then generate:
{completion...}
Tokenization and Special Tokens
Violet 160M uses a GPT-NeoX / Pythia-style tokenizer.
Recommended IDs for generation:
eos_token_id: 0
!! Do not mix tokenizers from other Violet variants (e.g. 1.4B) with this model.
How to use (Transformers)
code
1from transformers import AutoTokenizer, AutoModelForCausalLM
2import torch
34repo = "Zakarth/violet-160m"
5tok = AutoTokenizer.from_pretrained(repo, use_fast=True)
6model = AutoModelForCausalLM.from_pretrained(repo, device_map="auto")
78prompt = "The fog slowly lifted"
910inputs = tok(prompt, return_tensors="pt", add_special_tokens=False)
11# Filter out token_type_ids if present
12inputs = {k: v.to(model.device) for k, v in inputs.items() if k in ['input_ids', 'attention_mask']}
1314out = model.generate(
15 **inputs,
16 max_new_tokens=180,
17 do_sample=True,
18 temperature=0.8,
19 top_p=0.9,
20 top_k=40,
21 repetition_penalty=1.15,
22 eos_token_id=0,
23 pad_token_id=50277,
24)
25print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False))
Sample Outputs
The King spoke thusly
"My gracious mistress, I will not be so ungracious to you as to have you with me. My presence is not needed by us; we are strangers in the land of my fathers." "I am not ungrateful, sir," replied she, "but I desire not the intrusion of strangers into my own domain.Nor do I think it strange that you should remain here so long and so pleasantly." "I am ready to become your friend, since I claim all this honour," said he, "and the absence of strangers here, has made me happy. Your absence is not without its compensations, for I have given my daughter into the hands of a father, who was no common man.He was a man of great wealth, and had always been friendly towards her. He was fond of her, and was kind to her, but there were many reasons why we should not part company in society, and he loved her. She was a stranger to him, and he knew her well.She would never have consented to his wedded life, and he was sure that she would never consent to this marriage, which would make the union between us a hard one. We have no right to associate with others, except on account of our parents' death, and therefore you must decide for yourself. When the day of separation comes, you must return to my house, and to be my guests until the nuptials are over.This is my first request
License
Model weights and code in this repository are released under CC0 1.0 (public domain dedication).
