Mikav

Goals

• Train a Malayalam-first language model.
• Keep data preparation, tokenizer training, model training, and evaluation reproducible.
• Publish model checkpoints, tokenizer files, and model cards on Hugging Face.

Repository Layout

• configs/ contains tokenizer, model, pretraining, and fine-tuning settings.
• data/ contains raw, cleaned, deduplicated, and tokenized datasets.
• tokenizer/ contains tokenizer assets and training scripts.
• mikav/ contains model and tokenizer Python package code.
• scripts/ contains data, training, evaluation, and Hugging Face upload scripts.
• training/ contains distributed training configs.
• evaluation/ contains benchmark and prompt evaluation assets.
• model_card/ contains Hugging Face model-card documentation.
• examples/ contains inference examples.
• tests/ contains basic checks.

Quick Start

License