Toy LLaMA-39M

• This is a tiny LLaMA model pretrained on Recag/Rp_C4_55, a small subset of C4 with seq_len=512.
  • Model architecture
    json

    1{ 2 "hidden_size": 512, 3 "intermediate_size": 2048, 4 "max_position_embeddings": 2048, 5 "num_attention_heads": 8, 6 "num_hidden_layers": 2, 7 "num_key_value_heads": 8 8}
  • Load model and tokenizer:
    python

    1from transformers import AutoTokenizer, AutoModelForCausalLM 2model = AutoModelForCausalLM.from_pretrained("Cheng98/llama-39m") 3tokenizer = AutoTokenizer.from_pretrained("Cheng98/llama-39m")
  • Training script: huggingface/transformers/examples/pytorch/language-modeling/run_clm.py
    python

    1# "train" split is created from the last 95% samples of original "train" subset 2raw_datasets["validation"] = load_dataset("Recag/Rp_C4_55", split="train[5%:]")
• Evaluation (seq_len=512):

  Dataset	Eval loss	Perplexity	Accuracy	block_size
  Recag/Rp_C4_55	3.63	37.78	0.3561	512
  Wikitext2	4.58	97.48	0.2719	512

  • Evaluation command (Wikitext2):
    bash

    1# Evaluation command 2python run_clm.py --model_name_or_path Cheng98/llama-39m \ 3 --dataset_name wikitext \ 4 --dataset_config_name wikitext-2-raw-v1 \ 5 --block_size 512 \ 6 --do_eval \ 7 --output_dir ./results
  • Evaluation on Recag/Rp_C4_55 (seq_len=512):
    python

    1# "validation" split is created from the first 5% samples of original "train" subset 2raw_datasets["validation"] = load_dataset("Recag/Rp_C4_55", split="train[:5%]")
    Results
    json

    1{ 2 "eval_accuracy": 0.3561766818954313, 3 "eval_loss": 3.6318140029907227, 4 "eval_runtime": 190.8411, 5 "eval_samples": 19413, 6 "eval_samples_per_second": 101.723, 7 "eval_steps_per_second": 1.593, 8 "perplexity": 37.7812898658763 9}
  • Evaluation on Wikitext2 (seq_len=512):
    json

    1{ 2 "eval_accuracy": 0.2718795201225219, 3 "eval_loss": 4.579628944396973, 4 "eval_runtime": 3.939, 5 "eval_samples": 575, 6 "eval_samples_per_second": 145.976, 7 "eval_steps_per_second": 0.762, 8 "perplexity": 97.47821765687856 9}