NekGPT XL
NekGPT XL is a full fine-tune of Qwen/Qwen3.5-0.8B-Base on authored twitter posts from @nekstoer.
This release is the larger follow-up to the original nekGPT toy model. The target was simple: make it more cohesive, less funny, and less likely to collapse immediately in short conversations.
Validation loss was best after epoch 1 and worsened slightly at epoch 2:
- epoch 1:
0.4954
- epoch 2:
0.5053
Description
The goal was not alignment polish or generic assistant quality. The goal was a model that still understands basic questions, but answers with the voice pressure of the source corpus instead of defaulting back into a helpful assistant persona.
The result is more coherent than the original GPT-2 release and less sanitized than the base Qwen checkpoint, but it still drifts under sustained multi-turn chat. It works best as an experimental voice model and chat toy, not as a reliable factual assistant.
Methodology
Corpus
- Source account:
@nekstoer
- Collection method: GetXAPI scrape of authored posts only
- Reposts: excluded
- Replies: included
- Quotes: included
- Raw authored posts collected:
1526
Dataset shaping
The scrape was converted into higher-signal supervised examples:
- standalone posts:
write a tweet -> post
- replies:
reply to this post from @user: ... -> reply
- quotes:
quote tweet this post: ... -> quote tweet
To make the voice distribution more pronounced, a subset of standout standalone posts was repeated in the training split.
Effective dataset sizes for this run:
- train:
1786
- valid:
20
- test:
20
Training
- base model:
Qwen/Qwen3.5-0.8B-Base
- hardware: Apple Silicon MPS
- max length:
384
- epochs:
2
- learning rate:
2e-5
- batch size:
1
- gradient accumulation:
8
- gradient checkpointing: enabled
Known behavior
Strengths:
- answers simple questions more coherently than the original
nekGPT
- holds the local tone and phrasing much more aggressively than base Qwen
- stays chaotic without immediately collapsing into nonsense
Weaknesses:
- still drifts in longer multi-turn chats
- factual grounding is weak and can fail even on basic prompts
- overfits recurring motifs, formatting quirks, and usernames from the corpus
- the epoch-1 checkpoint had the best validation loss, while the final exported model is the epoch-2 artifact used for this release
Usage
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3repo_id = "potteryrage/NekGPT-XL"
4tokenizer = AutoTokenizer.from_pretrained(repo_id)
5model = AutoModelForCausalLM.from_pretrained(repo_id)
6
7prompt = "System: Output only the completion.\nUser: write a tweet\nAssistant:"
8inputs = tokenizer(prompt, return_tensors="pt")
9outputs = model.generate(
10 **inputs,
11 max_new_tokens=64,
12 do_sample=True,
13 temperature=1.05,
14 top_p=0.98,
15 pad_token_id=tokenizer.eos_token_id,
16)
17print(tokenizer.decode(outputs[0], skip_special_tokens=False))
License
Apache-2.0
