• Model creator: https://huggingface.co/abhinand/
• Original model: https://huggingface.co/abhinand/gemma-2b-tamil/

Original model description:

• en
• ta license: other base_model: google/gemma-2b datasets:
• wikimedia/wikipedia license_name: gemma-terms-of-use license_link: https://ai.google.dev/gemma/terms model-index:
• name: gemma-2b-tamil results:
  • task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge (25-Shot) type: ai2_arc config: ARC-Challenge split: test args: num_few_shot: 25 metrics:
    • type: acc_norm value: 47.44 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=abhinand/gemma-2b-tamil name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: HellaSwag (10-Shot) type: hellaswag split: validation args: num_few_shot: 10 metrics:
    • type: acc_norm value: 71.3 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=abhinand/gemma-2b-tamil name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: MMLU (5-Shot) type: cais/mmlu config: all split: test args: num_few_shot: 5 metrics:
    • type: acc value: 38.21 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=abhinand/gemma-2b-tamil name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: TruthfulQA (0-shot) type: truthful_qa config: multiple_choice split: validation args: num_few_shot: 0 metrics:
    • type: mc2 value: 34.93 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=abhinand/gemma-2b-tamil name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: Winogrande (5-shot) type: winogrande config: winogrande_xl split: validation args: num_few_shot: 5 metrics:
    • type: acc value: 65.98 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=abhinand/gemma-2b-tamil name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: GSM8k (5-shot) type: gsm8k config: main split: test args: num_few_shot: 5 metrics:
    • type: acc value: 12.89 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=abhinand/gemma-2b-tamil name: Open LLM Leaderboard

Gemma 2B Tamil v0.1 Alpha - Base Model [Experimental Release]

Please Note: This model, labeled as a FOUNDATIONAL Language Model (LLM), is designed primarily for Causal Language Modeling (LM) purposes. In other words, if you are looking for an instruction following model in Tamil, you may find abhinand/gemma-2b-it-tamil-v0.1-alpha more suitable for your needs.

1. The Gemma base model was continually pretrained on all available Tamil Wikipedia data for 3 epochs.
2. The updated model was then finetuned on a mix of English and Tamil alpaca datasets for 5 epochs. Finetuned model can be found here.

Note: This project is currently under development (FOR TAMIL). The initial pretraining phase may not have been extensive enough, which suggests that the model's performance could improve by extending the pretraining on a larger dataset, such as CulturaX.

Model description

• Model type: A 2B parameter GPT-like model continually pretrained on all available Tamil data from Wikipedia dataset.
• Language(s): Bilingual. English and Tamil.
• License: Google Gemma Terms of Use
• Training Precision: bfloat16
• Training Hardware: 4x Nvidia RTX 3090 GPUs
• Training Cost: $20

Support my work

Usage Note

Meet the Developers

• Abhinand Balachandran

Open LLM Leaderboard Evaluation Results