A very small (5.6M parameter) GPT-style language model, trained from scratch on a narrow
corpus of English-language Vipassana meditation texts that use Pali terminology. It is a
hands-on research artifact from a "baby LLM" project, not a general-purpose assistant.
This is a personal, non-commercial machine-learning experiment. The author's question was
narrow and technical:
How much knowledge and structure can a small language model acquire if it is trained
only on Dhamma/Vipassana texts, with no other text of any kind in the corpus?
No general web text, no books outside the domain, no code, no instruction data. Everything the
model knows about English, about sentence structure, and about the subject matter comes from
one small single-domain corpus. That constraint is the entire point of the experiment, and it
explains most of the model's behaviour.
It is not:
a source of Dhamma teaching, instruction, or guidance
affiliated with, endorsed by, reviewed by, or connected to the Vipassana Research Institute
(VRI), Dhamma.org, any Vipassana centre, or any teacher or assistant teacher
a representation of what any teacher actually said
a product, a service, or anything anyone should rely on
Its output is machine-generated text that will contain hallucinations, invented
"quotations", doctrinal errors, and heavy repetition. See
What the output looks like for measured numbers. If you want
to learn about Vipassana, go to the original sources or to a course. Not here.
Read this first: it is small, and it emits tags
1. It really is tiny. 8 layers, hidden size 216, 3 attention heads, a 5000-token
vocabulary and a 360-token context, trained on roughly 1.85M tokens of a single-domain corpus.
For scale, that is about 5.6M parameters against the billions in models you may be used to.
Expect short-range fluency, limited coherence, repetition, and a strong tendency to reproduce
corpus phrasing. Judge it as a learning exercise.
2. It was trained on text marked up with structural tags, and it will generate them. The
corpus is annotated with an XML-like grammar that encodes style and layout, and those 20 tags
are real single tokens (ids 0-19) rather than text the tokenizer splits up. The model learned
them as part of the language, so tags will appear in the output by design -- that is not
corruption.
Tag
Meaning
<topic:essay>
essay-style section (the most common topic wrapper)
<topic:recollection>
personal recollection
<topic:qa>
question-and-answer section
<h>
heading; closed by </h>
<quote>
block quotation; closed by </quote>
<list>
list container; holds <item> ... </item> entries
<q>
a question, inside <topic:qa>; answered in <a> ... </a>
Every container tag has a matching closer (</topic:essay>, </list>, ...). <pad> and
<unk> are ordinary special tokens.
You can use these to steer style: open with <topic:essay> for essay-like prose,
<topic:recollection> for personal narrative, or <topic:qa> with <q>/<a> for a
question-and-answer shape.
In the GGUF these tags are exported as USER_DEFINED rather than CONTROL, so llama.cpp
renders them as visible text and matches them in prompts without needing --special. Marked
CONTROL, they would silently render as zero bytes and you would see structure with no
markup, which is confusing.
Usage
This is not a chat model. It is a pure text-continuation machine.
There was no instruction tuning, no chat fine-tuning, no RLHF and no alignment of any kind.
The model has never seen a conversation, a system prompt, a role marker or a chat template.
It does exactly one thing: given a piece of text, it predicts what text plausibly comes
next, and then keeps going.
Practical consequences:
Asking it a question does not get you an answer. It continues the text of your
question -- often with more questions, or with prose that drifts around the topic.
Instructions are not followed. "Summarize this", "answer briefly", "act as a teacher"
are just words to continue, not commands.
The <topic:qa> / <q> / <a> tags produce question-and-answer shaped text. That
is imitation of the corpus layout, not instruction following, and the answers are not
answers to your question in any reliable sense.
It never stops on its own. There is no EOS token; you must bound generation with a
token limit or a stop string.
In chat UIs (LM Studio, Ollama, etc.) use raw completion mode. Chat mode wraps your
input in template tokens the model has never seen, and output gets worse.
Write a prompt as the opening of a passage you want continued, not as a request.
In LM Studio, use raw completion mode, not chat. A chat template wraps the prompt in
tokens this model has never seen. An ollamaModelfile is included.
Prompting notes
Start plain prompts with a leading space. The tokenizer was trained with
add_prefix_space=True; llama.cpp does not apply it on the BPE path, and
encode(" " + s) == encode(s), so a literal leading space reproduces training-time
tokenization. Prompts that open with a tag are unaffected.
There is no EOS token. Nothing stops generation on its own -- bound it with
max_tokens / -n, or stop on a closing tag such as </topic:essay>.
Context is 360 tokens, shared between prompt and completion.
Model details
Architecture
decoder-only transformer, exported as GPTNeoXForCausalLM
Parameters
5,581,872 (6,661,872 exported, with an untied output head)
Trained from scratch on ~1.85M tokens for 14000 steps (batch 96, lr 0.0002, dropout 0.125),
with cross-entropy and label smoothing 0.1. Best validation loss 4.162 (perplexity
64.2) at step 14000. No instruction tuning, no RLHF, no alignment of any kind -- this is a
base completion model.
Training data
51 English-language documents, roughly 7.6M characters / 1.85M tokens, all within the
Dhamma/Vipassana domain. Sources:
Material from the VRI website's section for old students. The author is an old student
and accessed this material as such.
Books purchased by the author, including freely-distributed VRI PDF editions.
Publicly available articles and posts in which old students shared their impressions
after courses. These are the basis for the <topic:recollection> style.
One AI-generated research summary about the Vipassana tradition, written by a large
language model -- 22,373 bytes / 22,035 characters / 5,657 tokens, i.e. 0.31% of the
corpus -- included as general background prose. It is the only file in the corpus not
written by a human, and some of the model's "encyclopedic" register comes from it rather
than from tradition material.
What is deliberately NOT in the corpus
The model was not trained on course instructions or on evening discourses. No course
audio, no discourse transcripts, no guided-meditation instructions, and no material intended
to be received only within a course were used at any point.
The corpus is limited to material that was, in its original form, intended to be shared
among old students or with the general public -- newsletters, journals, published books,
letters printed in newsletters, biographical and explanatory articles, and old students'
publicly shared course impressions.
The corpus itself is not distributed with this model, and no part of it is reproduced in
this repository. Only trained weights are published here.
Preprocessing
The source texts were re-composed by hand: paragraph and line-break structure was rebuilt, and
structural tags (<h>, <list>, <item>, <quote>, <topic:*>) were added to mark headings,
lists, quotations and style. This changes layout and markup only -- the underlying wording is
unchanged, and the tags are the reason the model emits them.
What the output looks like
Measured on ~1.0M generated tokens across the four sampling modes, so these are numbers rather
than impressions.
Repetition and looping
This is the model's most visible failure mode, and it is worse than the memorization:
trigram repetition ratio 0.11-0.16; unique-token ratio 0.45-0.49
a single 12-token phrase repeated up to 47 times within one sample
a single 5-gram repeated up to 599 times within one sample
Expect loops. A repetition penalty and a stop condition on a closing tag help a lot.
Hallucination
The model has no knowledge base, no retrieval and no grounding. It fluently produces
statements that are wrong, including about Buddhist doctrine, history and practice, and it
will invent quotations and attribute them to teachers. It has no notion of who said what.
Nothing it outputs should be attributed to any teacher or treated as doctrinally correct.
Memorization
A model this small on a corpus this narrow can echo training text, so this was measured
explicitly rather than assumed. Method: exact n-gram overlap between generations and the
training corpus, with Pali encoding normalized on both sides so that Pali terms compare on
their content and not on their byte encoding.
Share of generated n-grams that appear verbatim in the corpus:
n-gram size
share found in corpus
5
24.2% - 32.5%
8
2.5% - 3.8%
10
0.55% - 0.99%
12
0.14% - 0.29%
Share of generated tokens lying inside any verbatim span of a given length (tags removed):
span length
share of tokens
>= 12 tokens
0.63% - 1.43%
>= 15 tokens
0.13% - 0.27%
>= 20 tokens
0.01% - 0.03%
Reading: more than 98.5% of generated tokens are not part of any 12-token verbatim copy,
and overlap decays geometrically as n grows -- the signature of recombination rather than
recitation. The longest verbatim match found was 28 tokens including tags, 23 tokens of
prose. Across ~1.0M generated tokens there were 128 spans of >= 15 tokens (87 distinct).
Inspecting those spans, most are formulaic material that repeats inside the corpus itself:
course-eligibility criteria, metta well-wishing formulas, standard doctrinal enumerations, and
canonical Pali phrases. A small number are single sentences traceable to one specific passage.
Nothing approaching a paragraph, a section or a document is reproduced.
This is a property of the model, not a feature: do not treat output as original text, and do
not use it to reproduce source material.
Provenance and validation
Exported from a PyTorch checkpoint via a HuggingFace GPTNeoXForCausalLM directory, with
equivalence checked at every stage before the GGUF was written:
exported model vs original checkpoint: worst absolute logit difference 8.23e-06,
identical argmax at every position, identical greedy continuations
tokenizer: 25 corpus lines re-encoded, token ids identical
GGUF: all 100 tensors bit-identical to the checkpoint recomputed and cast to f16
config.json intentionally carries bothrotary_pct and rope_parameters. Removing the
legacy rotary_pct key silently breaks the model on transformers >= 5:
partial_rotary_factor falls back to 0.25 and the rotary dimension becomes 18 instead of 72,
with no error raised.
Limitations
Repetition. The dominant failure mode; see the measured numbers above.
Not factual. No knowledge base; states things that are wrong, including about Buddhist
doctrine. Not a source of teaching or guidance.
Memorization. Can reproduce short training phrases close to verbatim.
Single narrow domain. Off-topic prompts produce meandering meditation prose.
Vocabulary gaps. Only bytes seen in the corpus have tokens and there is no byte
fallback, so unusual input (including a literal backslash, tab or CR) becomes <unk>.
Context. Exact behaviour is only defined up to 360 tokens.
No safety work of any kind. No alignment, no filtering, no refusal behaviour.
Numerical note. llama.cpp's GELU is the tanh approximation while the PyTorch and
transformers paths use exact erf, so GGUF output can diverge by the occasional token.
Intended use
Intended for research and education about small language models: studying what a model
learns from a tiny single-domain corpus, tokenizer and tagging behaviour, memorization
measurement, and training dynamics at small scale.
Not intended for any use as a source of information about Buddhism, Vipassana, or meditation
practice, nor for any commercial use.
License, rights and contact
license: other, deliberately.
The model weights, configuration, export tooling and this model card are released by the
author for non-commercial research and educational use.
The training corpus is third-party copyrighted material and is not distributed here. The
author claims no rights in those source texts and grants none. Publishing these weights is
not a license to any underlying work, and the author's hand-added markup and arrangement do
not affect the rights of the original authors and publishers.
No affiliation or endorsement. This work is independent of VRI, Dhamma.org, and any
Vipassana centre, teacher or assistant teacher. Nothing here is authorized by them.
If you hold rights in any of the source material and object to this model being published,
please open a discussion on this repository or contact the author, and it will be taken down.
