nekGPT
nekGPT is a small character-voice model derived from GPT-2 and tuned on authored X posts from @nekstoer.
This release intentionally uses the best validation checkpoint from training rather than the final checkpoint. Validation loss was lowest at epoch 2, then degraded with additional training:
- epoch 1:
0.7919
- epoch 2:
0.7898 <- selected checkpoint
- epoch 3:
0.8061
- epoch 4:
0.8148
- epoch 5:
0.8266
Description
The goal was not generic assistant quality. The goal was to force a small model to latch onto a narrow posting voice strongly enough to survive open-ended prompts better than a larger, more stubborn base model.
The result is more voice-dominant than the larger Qwen experiments, but also less stable and less coherent over long conversations. It works best as a compact style model and experimental chat toy, not as a reliable assistant.
Methodology
Corpus
- Source account:
@nekstoer
- Collection method: GetXAPI scrape of authored posts only
- Reposts: excluded
- Replies: included
- Quotes: included
- Raw authored posts collected:
1526
Dataset shaping
The original scrape was converted into higher-signal supervised examples:
- standalone posts:
write a tweet -> post
- replies:
reply to this post from @user: ... -> reply
- quotes:
quote tweet this post: ... -> quote tweet
To make the voice distribution more pronounced, a subset of standout standalone posts was repeated in the training split.
Effective dataset sizes for the GPT-2 run:
- train:
1786
- valid:
20
- test:
20
Training
- base model:
openai-community/gpt2
- hardware: Apple Silicon MPS
- max length:
256
- epochs:
5
- learning rate:
5e-5
- batch size:
4
- gradient accumulation:
1
Although the full run completed, this repo publishes the epoch-2 checkpoint because it had the best validation loss.
Known behavior
Strengths:
- strongly picks up local tone and phrasing
- responds less like a default assistant than the larger Qwen run
Weaknesses:
- drifts or collapses in longer multi-turn chats
- can become incoherent
- can overfit to fragments, usernames, and recurring motifs from the corpus
Usage
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3repo_id = "potteryrage/nekGPT"
4tokenizer = AutoTokenizer.from_pretrained(repo_id)
5model = AutoModelForCausalLM.from_pretrained(repo_id)
6
7prompt = "User: hi nek\nAssistant:"
8inputs = tokenizer(prompt, return_tensors="pt")
9outputs = model.generate(
10 **inputs,
11 max_new_tokens=48,
12 do_sample=True,
13 temperature=0.9,
14 top_p=0.95,
15 pad_token_id=tokenizer.eos_token_id,
16)
17print(tokenizer.decode(outputs[0], skip_special_tokens=False))
License
Apache-2.0
