AppleMind-1.0-Mini is a compact decoder-only causal language model trained from scratch by AppleMind on a 300M-token curriculum.
The model has 1,020,480 parameters, a 256-token context window, and a 50,263-token digit-aware byte-level BPE tokenizer (GPT-2 + special tokens).
Model Details
Field
Value
Parameters
1,020,480
Architecture
AppleMind 1.0 Mini decoder-only Transformer
Layers
4
Hidden size
128
Intermediate size
512
Attention heads
4
KV heads
4
Head dim
32
Attention style
Multi-head causal self-attention
MLP
GELU
Position embeddings
Learned positional embeddings
Normalization
LayerNorm
Vocabulary size
50,263
Context length
256
Embeddings
Tied input/output embeddings
Tokenizer
Digit-aware byte-level BPE (GPT-2 + special tokens)
Weight format
safetensors
HF architecture
GPT2LMHeadModel
HF model type
gpt2
Final training steps
2,288
Tokens seen
299,892,736
Tokens/parameter
293.87:1
Training data
FineWeb-Edu + FineWeb-HQ + SmolLM-Corpus
Training mixture
100M + 100M + 100M tokens
Precision
BF16
Final model
AppleMind 1.0 Mini
Credits to BananaMind for inspiring me to make AppleMind.
Tokenizer
AppleMind 1.0 Mini uses a 50,263-token digit-aware byte-level BPE tokenizer based on the GPT-2 tokenizer, with 3 additional special tokens. Digits are handled individually rather than being collapsed into large number tokens.
Special token
ID
<|pad|>
50,260
<|bos|>
50,261
<|eos|>
50,262
Training Data
Dataset
Target Tokens
Share
FineWeb-Edu
100M
33.33%
FineWeb-HQ
100M
33.33%
SmolLM-Corpus
100M
33.33%
Total
300M
100%
The training run used an equal mixture of FineWeb-Edu, FineWeb-HQ, and SmolLM-Corpus, with 100M tokens sampled from each dataset.
Training Setup
Field
Value
Sequence length
256
Micro batch
512 sequences
Gradient accumulation
1
Effective batch
512 sequences
Tokens per optimizer step
131,072
Final optimizer step
2,288
Configured optimizer steps
2,289
Optimizer
AdamW
Peak learning rate
0.0001
Learning rate at final step
3.114e-06
LR schedule
Cosine decay
Gradient clipping
1
Weight format
safetensors
Training tokens
299,892,736
Target tokens
300,000,000
Tokens/parameter
293.87:1
Evaluation
AppleMind 1.0 Mini has not been formally evaluated with lm_eval yet. No benchmark scores are currently reported.
Benchmark
Score
Metric
Average
N/A
mean
ARC Easy
N/A
acc_norm,none
PIQA
N/A
acc_norm,none
ARC Challenge
N/A
acc_norm,none
HellaSwag
N/A
acc_norm,none
The model's current generation quality has been checked with basic text-generation prompts, including:
Once upon a time
The little boy
In the forest
Prompt: Once upon a time
Once upon a time It the several times- nowThis, does to form provide from find another times work not atl The couldWhen on all be way H It lives among times always, worked
its G during work used after several There and at there
b known came be very that It thought It betweenIn course does. case other It 5?: or often I at's the: enough could in many
Prompt: The little boy
The little boy's form among. I and H be from� used. still all- G, lives take same always often its find amonged provide- number because: another now? then there among use course not thought case work usel result It between 5 It well atWhen new.: thatThis work on think 3 interestB then It does could do the among Ire use among now does to and many
Prompt: In the forest
In the forest often
form and times on during severall find, several The then number between: well work take there provide, times among anotherWhen same Ged but lives could the to times its- course same enough and same I used
to same other not thought There H think� same 2 I body nowe cameb 5- do among's and several? same after- still,This- interest but
Usage
AppleMind 1.0 Mini uses custom architecture code, so load it with trust_remote_code=True.
