FlashLM v4 "Bolt" — GreedyPhrase Training Results

Overview

Training Configuration

Dataset

• Source: TinyStories (roneneldan/TinyStories), 2.1 GB text
• Preprocessing: <|endoftext|> replaced with </s> (EOS token ID 3)
• Tokenized size: 248M tokens (496 MB binary uint16)
• Compression ratio: ~8.88 bytes/token (vs ~4.5 for GPT-2)
• Train/val split: 99.5% / 0.5%

Results

Loss Curve

Step        Train Loss    Val Loss
     0       11.13          —
   500        6.73         5.96
  1000        5.46         5.12
  2500        4.72         4.61
  5000        4.43         4.39
 10000        4.17         4.19
 20000        4.03         4.03
 30000        3.95         3.97
 40000        3.92         3.95
 50000        3.94         3.94
 Best          —           3.9352 (step 47500)

Metrics

Comparing Val Loss Across Tokenizers

1. Larger vocabulary = harder prediction task. Random-chance loss is ln(65280) = 11.09 vs ln(10000) = 9.21. The model must distribute probability over 6.5x more tokens.
2. Fewer tokens per story. GreedyPhrase compresses TinyStories at ~9 bytes/token vs ~4.5 bytes/token for GPT-2. Each token carries more information, so predicting the next token is inherently harder.
3. Bits-per-character is the fair comparison. At 0.64 BPC this model is competitive with the original's 0.88 BPC, suggesting the GreedyPhrase tokenizer's higher compression ratio pays off in information-theoretic efficiency.

Generation Samples (Step 49,500)

Once upon a time there was a little girl named Sarah. She was only three years old and loved exploring. One day Sarah went to the park with her mother. She saw a little boy playing with a ball.

Once upon a time there was a very deep lake. It was great! Every morning he would jump off the water and look for something wonderful.

Once upon a time there was a little girl named Mary. Mary loved animals, especially especially loved the ocean. Every day Mary would go out on a walk around the waves and swimming around on the beach.

Prompt: "The little dog"

The little dog wanted to protect his bone, so he held it up to the cat and tried to protect him. But the big cat was jealous. It wanted to take the bone from him, but it ran away.

The cat was sad and began to cry. Then, he saw a big hole in the ground and started to shake it. The cat growled and tried to run away. The dog was scared and ran back to the cat. The cat saw the fox and was scared. The cat took the kitten and ran away. The dog was sad. The fox did not get the mitten anymore. The cat was happy and played with Spot and the other friends.

Files

Observations

1. Convergence was smooth. Loss dropped from 11.13 to ~3.94 over 50K steps with no instability, despite ternary weight quantization via straight-through estimators.
2. The loss curve was still slowly declining at 50K steps. Extended training or a second cosine cycle could improve results further.
3. GreedyPhrase's long phrases help coherence. With ~9 bytes/token, the 256-token context window covers ~2,300 characters (~400 words), much more than the original's ~1,150 characters. This gives the model more context per sequence.
4. The larger embedding table dominates parameter count. 65K vocab x 192 dim = 12.5M parameters in the embedding alone (84% of total), vs 1.9M for the original's 10K vocab. The model body (blocks) is identical.
5. Throughput benefited from GPU + AMP. At 103K tokens/sec on an RTX 2080 Ti, this is 70x faster than the original's 1.5K tokens/sec on CPU, allowing 3.3 full epochs in roughly the same wall-clock time.