Aurelius-Llama-4.0M-v1.0
Part of The Aurelius TinyStories Collection, a specialized series of highly optimized, sub-10M parameter models trained exclusively on the TinyStories dataset. This collection explores the boundaries of grammatical coherence, narrative depth, and reasoning capacity at micro-scale.
Aurelius-Llama-4.0M-v1.0 is the official stable release of our 4.03M parameter architecture. Representing a significant leap in representational density, this model utilizes a standard LLaMA-style causal architecture compiled directly into native weights. It requires no custom code, special configurations, or remote execution flags to run.
Model Details
- Collection: The Aurelius TinyStories Collection
- Version: 1.0
- Total Parameters (with tied embeddings):
4,033,728 ($\approx 4.03\text{M}$)
- Active Parameters (excluding embeddings):
3,247,296 ($\approx 3.25\text{M}$)
- Layers ($L$): 8 (Upgraded from 6)
- Hidden Dimension ($d_{\text{model}}$): 192
- Intermediate Dimension ($d_{\text{ff}}$): 512 (SwiGLU)
- Attention Heads: 6 Query Heads / 3 Key-Value Heads (Grouped-Query Attention)
- Context Length ($L_{\text{seq}}$): 256 tokens (scalable to 512)
- Vocabulary Size ($V$): 4,096 (Custom Byte-Level BPE)
Key Observations & Improvements
Compared to previous versions in the collection, the v1.0 release exhibits several notable advancements in qualitative performance:
- Significantly increased narrative coherence: The model demonstrates robust control over long-range story arcs, maintaining thematic focus and preventing the structural drift common in ultra-small architectures.
- Dynamic character tracking: Enhanced capability to maintain stable, non-contradictory agent identities, objects, and relationships across multi-sentence generations.
- Structural and grammatical integrity: Highly refined orthography, spacing, and punctuation formatting, particularly within complex, turn-taking dialogue sequences.
- Emergent prompt responsiveness: Improved sensitivity to direct prompt framing, demonstrating basic contextual reasoning, descriptive instruction-following, and question-answering capabilities.
Evaluation & Performance
Trained on the synthesized prose of the TinyStoriesV2 dataset, the model achieves the following validation benchmarks:
- Validation Cross-Entropy Loss:
1.4031
- Validation Perplexity:
4.07
While the raw validation loss reflects the model's highly regularized and continuous semantic space, the qualitative generation capability represents a substantial advancement over previous iterations.
Important Disclaimer & Liability Limitation
This model is provided strictly "as is" and "with all faults," without warranty of any kind, express or implied.
- Experimental & Research Nature Only: This model is an experimental, micro-scale prototype developed strictly for educational, scientific, and academic benchmarking purposes. Any stated "improvements," "advancements," or "capabilities" are relative only to other micro-scale baselines and do not indicate suitability for production tasks, commercial applications, or consumer-facing products.
- No Safety Alignment: While the pre-training dataset (
TinyStoriesV2) is conceptually designed around simple, child-like narratives, this model has not undergone any safety tuning, RLHF, supervised fine-tuning (SFT) for alignment, or toxic content filtering. It can output unpredictable, nonsensical, or potentially inappropriate text. Consequently, under no circumstances should this model or its outputs be deemed safe, verified, or appropriate for children or general public interaction.
- User Assumption of Risk: Any output generated by this model is the result of statistical text completion and does not represent the views, opinions, or endorsements of the developers or hosting entities. The end-user assumes all liabilities and risks associated with running, testing, fine-tuning, or utilizing the model or any downstream text generated by it.
- Architectural and Trademark Clarification: The use of "Llama" in the model name refers solely to the underlying open-source mathematical architecture used to structure the neural network layers (such as RMSNorm, SwiGLU, and RoPE). This model is trained from scratch and is not affiliated with, endorsed by, or associated with Meta Platforms, Inc. or any of its affiliates.
Usage
Because this model has been compiled into standard native layers, you can load it using the standard Hugging Face transformers library with zero runtime dependencies.
To experience the highest-fidelity generation, we recommend using a lower temperature paired with Min-P sampling:
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model_name = "MultivexAI/Aurelius-Llama-4.0M-v1.0"
4
5tokenizer = AutoTokenizer.from_pretrained(model_name)
6model = AutoModelForCausalLM.from_pretrained(model_name)
7
8prompt = "Once upon"
9inputs = tokenizer(prompt, return_tensors="pt")
10
11# Recommended
12outputs = model.generate(
13 **inputs,
14 max_new_tokens=128,
15 temperature=0.1,
16 min_p=0.20,
17 repetition_penalty=1.0,
18 do_sample=True
19)
20print(tokenizer.decode(outputs[0], skip_special_tokens=True))
