You can deploy this model and start earning money today!

shikhar-srivastava/
llama-130m-prenorm-train_c4-2B-tok_t5base

Name: llama-130m-prenorm-train_c4-2B-tok_t5base
Availability: InStock

Views

No views yet

Tokenizer Study | LLaMA 130M | SentencePiece (T5 Tokenizer: t5-base)

LLaMA 130M (Implementation: https://github.com/lmsdss/LayerNorm-Scaling)

Pre-Training: C4 [~2.054B tokens (BPE), ~2.00B tokens (SentencePiece)]

Tokenizer: SentencePiece (T5 Tokenizer, t5-base)

Evals:

Perplexity

Bits-per-byte

Checkpoints:

80K steps (Local steps 10K; 8 grad accumulations)
- Path: /model_10000
- Evals:
  - Perplexity: 30.366
  - Bits-per-byte: 0.4528