You can deploy this model and start earning money today!

The-CoLab/
llama3-7b-en1-en2-134k

Name: llama3-7b-en1-en2-134k
Availability: InStock

Views

No views yet

llama3-7b-en1-en2-134k

LLaMA-3 7B pretrained on English bilingual data (en1 + en2), 134k total steps. Uses a 2×vocabulary (65k paired tokenizer) with separate en1/en2 token spaces.

Validation Results

Final checkpoint: step 133,600

Validation set	Cross-entropy loss (nats)	Perplexity
English 1 (en1)	2.4598	11.70
English 2 (en2)	2.5351	12.62

Training Curves

Training and validation loss curves are in training_curves/:

train_en1_en2_134k.csv — step, train_loss
val_en1_en2_134k.csv — step, lang, val_loss

Evaluation Results

EEE-format evaluation results are stored under eval_results/eee/ in this repository. Tasks: Global MMLU (EN), PIQA, ECLeKTic, Fictive Entity (2-rate mix).

Citation

Part of the The-CoLab multilingual-transfer collection.

The-CoLab/llama3-7b-en1-en2-134k