A compact Yuna model fine-tuned for prompted short-story generation.
Parameters
Context
Stage
SFT tokens
Status
Important: This is a creative-writing model. It is optimized to invent prose, not to provide accurate information, advice, or dependable instruction-following.
YunaGPT-124M V1 architecture and training infographic
Overview
YunaGPT-124M V1 Story is the creative-writing branch of YunaGPT-124M V1 Base. It was trained with response-only supervised fine-tuning on short-story and writing-prompt datasets. The uploaded weights exactly match the final checkpoint after creative-writing epoch 3.
Despite the name, Story is still a 124M-parameter experimental model. It can produce recognizable narrative structure and imaginative passages, but it often loses coherence, repeats ideas, misuses words, or ends abruptly.
Project background
Yuna began as a 30M-parameter educational language-model project inspired by Sebastian Raschka's Build a Large Language Model (From Scratch). It later moved to Hugging Face's native LLaMA implementation and became an experiment in training a small model on a home RTX 3090. The broader project also studies synthetic Final Fantasy X data and how specialized data changes a compact model's output.
Any apparent franchise knowledge is unreliable. This model may imitate names or settings while inventing unsupported lore.
Model summary
Item
Value
Parameters
124,445,376
Model class
LlamaForCausalLM
Training stage
Creative-writing SFT
Lineage
Base → Story
Creative-writing examples
5,580
Recorded SFT tokens
Approximately 20.65 million
Context length
2,048 tokens
Vocabulary
24,000 tokens
Tokenizer
Byte-level BPE
Hidden layers
25
Hidden size
576
Attention / KV heads
9 / 3
Weight format
safetensors, FP32
Primary language
English
Prompt format
Story uses the same instruction wrapper as the project's instruction SFT, not a plain continuation or chat template:
text
1Below is an instruction that describes a task. Write a response that appropriately completes the request.
23### Instruction:
4{writing_prompt}
56### Response:
An optional ### Input section may be inserted between the instruction and response headings, but most creative-writing examples used a self-contained instruction.
Run it yourself
pip install torch transformers
python
1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
34model_id ="YOUR_USERNAME/YunaGPT-124M-V1-Story"56tokenizer = AutoTokenizer.from_pretrained(model_id)7model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")8model.eval()91011defformat_story_prompt(instruction:str, input_text:str="")->str:12 prompt =(13"Below is an instruction that describes a task. "14"Write a response that appropriately completes the request.\n\n"15f"### Instruction:\n{instruction.strip()}"16)17if input_text.strip():18 prompt +=f"\n\n### Input:\n{input_text.strip()}"19return prompt +"\n\n### Response:\n"202122prompt = format_story_prompt(23"Write a short fantasy story about a knight who discovers that the "24"dragon is protecting the last surviving library."25)26inputs = tokenizer(prompt, return_tensors="pt")2728with torch.no_grad():29 output = model.generate(30**inputs,31 max_new_tokens=500,32 do_sample=True,33 temperature=0.85,34 top_p=0.95,35 repetition_penalty=1.08,36 pad_token_id=tokenizer.pad_token_id,37 eos_token_id=tokenizer.eos_token_id,38)3940new_tokens = output[0, inputs["input_ids"].shape[1]:]41print(tokenizer.decode(new_tokens, skip_special_tokens=True).strip())
Replace the placeholder repository name with the final model ID or a local folder. Lowering temperature generally makes output more predictable; raising it can increase variety and instability.
Architecture
Component
Configuration
Architecture
Decoder-only Transformer
Attention
Grouped-Query Attention (GQA)
Hidden size
576
Intermediate size
2,048
Layers
25
Attention heads
9
Key/value heads
3
Activation
SiLU / SwiGLU feed-forward blocks
Normalization
RMSNorm, epsilon 1e-6
Position encoding
RoPE, theta 10,000
Maximum positions
2,048
Input/output embeddings
Tied
Creative-writing training
The training mixture contains 5,580 filtered and deduplicated examples:
Source
Examples
DataMajin/Data-Majin_Short-Stories
354
nchapman/figaro-creative-writing
4,055
Gryphe/ChatGPT-4o-Writing-Prompts
1,171
The final checkpoint completed three configured epochs and recorded approximately 20.65 million processed tokens. The lowest internal validation loss was approximately 2.68; the final recorded value was approximately 2.76. These are internal next-token validation measurements, not standardized writing-quality benchmarks.
Only the response and EOS target contributed to the training loss. Prompts were retained as context but masked from loss. Source dataset licenses and terms remain applicable.
Intended uses
Short stories and fictional scene generation with human editing.
Creative-writing experiments and prompt studies.
Educational study of specialization in compact language models.
A starting point for further writing or role-play fine-tuning.
Limitations and safety
Expected limitations include:
inconsistent plots, characterization, tense, and point of view;
repetition, nonsensical wording, abrupt endings, and topic drift;
difficulty sustaining stories near the full context limit;
unreliable factual knowledge and instruction following;
generation of stereotypes, graphic material, sexual content, or other unsafe prose inherited from source data;
possible imitation of phrases, characters, or information present in training data.
Do not use Story for factual advice or high-impact decisions. Review generated prose for safety, privacy, originality, and suitability before publishing it.
Evaluation status
No standardized creative-writing, factuality, bias, toxicity, memorization, or safety benchmarks are included with this release. Training loss alone does not establish narrative quality or safe deployment.
Related variants
Base: raw next-token completion checkpoint.
Instruct: general single-turn instruction SFT branch.
Conversation: continued from this Story checkpoint with role-play conversation SFT and a different prompt format.
License and attribution
No model-weight license was declared in the project metadata when this card was prepared. Add an explicit license before public distribution. A model license does not override the source datasets' licenses, terms, or attribution requirements.
YunaGPT-124M V1 Story is an experimental creative-writing model. Review and edit its output before use.
