You can deploy this model and start earning money today!

The-CoLab/
llama3-7b-en1-en2-codeswitching-134k

Name: llama3-7b-en1-en2-codeswitching-134k
Availability: InStock

Views

No views yet

llama3-7b-en1-en2-codeswitching-134k

LLaMA-3 7B pretrained on English bilingual data (en1 + en2) with code-switching, 134k total steps.

Training proceeded in two stages: stage 1 (110k steps, standard bilingual) + stage 2 (23.6k steps, 0.6 code-switching rate). Uses a 2×vocabulary (65k paired tokenizer) with separate en1/en2 token spaces.

Validation Results

Final checkpoint: step 133,600

Validation set	Cross-entropy loss (nats)	Perplexity
English 1 (en1)	2.2185	9.19
English 2 (en2)	2.2175	9.18

Training Curves

Training and validation loss curves are in training_curves/:

train_en1_en2_codeswitching_134k.csv — step, train_loss (11,560 entries, steps 18,010–133,600; earlier steps not available in logs)
val_en1_en2_codeswitching_134k.csv — step, lang, val_loss (87 checkpoints × 2 languages, steps 18,704–133,600)

Evaluation Results

EEE-format evaluation results are stored under eval_results/eee/ in this repository. Tasks: Global MMLU (EN), PIQA, ECLeKTic, Fictive Entity (2-rate mix).

Citation

Part of the The-CoLab multilingual-transfer collection.

The-CoLab/llama3-7b-en1-en2-codeswitching-134k