A Q/A-finetuned variant of KjeldGPT 1.1B, a 1.1B-parameter GPT trained from scratch. It answers questions in a fixed Question: ... / Answer: ... format instead of continuing prose, and it is designed to answer from a retrieved passage, not from parametric memory.

This repo holds checkpoint v7, the production checkpoint.

Source, RAG index builder, and evaluation harness: https://github.com/kristofferkjeldby/KjeldGPT-1.1B

The weights are a plain safetensors dump of a custom nn.Module defined in model.py. AutoModelForCausalLM.from_pretrained will not work. Load it with the repo's own architecture:

strict=False is required and expected — head.weight is deliberately absent, since it shares storage with tok_emb.weight under tied=True and safetensors will not serialize aliased tensors. The alias is restored by GPT.__init__ before loading.

Tokenizer (a tokenizers JSON, identical to the base model's — finetuning does not retrain it):

The model was finetuned on exactly two templates, and it degrades badly outside them.

Closed-book:

Question: {question}
Answer:

With a retrieved passage — this is the intended path:

Context: {context}
Question: {question}
Answer:

Generation stops at the end of the answer. A ready-made REPL that does retrieval, prompting and streaming for you is chat.py; a browser UI over the same pipeline is chat_gui.py.

These weights are half the system. The other half is a passage index built from Wikipedia's community-curated Vital Articles (Level 5, ~45k titles), fetched live via the MediaWiki API, embedded with all-MiniLM-L6-v2 and reranked with a ms-marco-MiniLM-L-6-v2 cross-encoder. Rebuild it from the GitHub repo with rag/fetch_vital_articles.py + rag/embed_passages.py — the ~45k-title input list is tracked there, so the index is reproducible. Used closed-book, this model is much weaker than the numbers below suggest.

Resumed from the base checkpoint (never chained on top of a previous finetune) over a synthetic Q/A corpus of ~51,200 pairs generated from real Wikipedia passages: grounded Q/A, deliberately non-factual no-context pairs, ~4,000 false-premise correction pairs, a grounding corpus targeting demonstrated context-discarding failures, and a value-discrimination corpus targeting picking the specific value a question asks for over the most salient one nearby. Batch size 8, peak LR 1e-5 (cosine to 1e-6), dropout 0.2. Training and validation are split by source passage, not by individual Q/A pair, so validation measures the model on passages it never trained on -- what it actually faces at inference. The promoted checkpoint is step 2,700 of a 3,789-step run, chosen by evaluation score rather than by early stopping (see finetune/FINETUNE_PARAMS.md's "Checkpoint selection").

Both the finetuning corpus and the RAG index are Wikipedia-only, even though the base model's pretraining corpus is Gutenberg + Wikipedia: Gutenberg passages — novels, first-person narrative, bibliography pages — made unreliable context and could outrank a better Wikipedia match on sheer corpus volume. Full rationale: finetune/FINETUNE_PARAMS.md.

426 questions — 346 factual, 50 false-premise, 30 non-factual — run through seven models and graded by the same judge. Blackbox: question in, answer out. KjeldChat's retrieval is invisible in the comparison and no external model was handed a passage.

It clears a model larger than itself (GPT-2 XL) and an instruction-tuned model of the same size (TinyLlama), and lands below davinci-002.

False-premise handling is its least reliable trained behavior: of 50 questions embedding a false premise, KjeldChat explicitly corrects 5 and accepts 45 (inventing a supporting fact rather than flagging the error) -- weaker than davinci-002's 8/42 and TinyLlama's 6/44 at the same task. The ~4,000 targeted training pairs (finetune/data/generate_false_premise_*.py) help against a fully naive baseline but don't close the gap to models with a stronger pretrained prior to draw on.

A hobby project and a learning exercise, published openly in that spirit. Not a product; no safety tuning.

MIT, for the code and these weights. Wikipedia-derived training content is CC BY-SA.