Kirundi Tokenizer and LoRA Model

Model Description

1. A BPE tokenizer trained specifically for the Kirundi language (ISO code: run)
2. A LoRA adapter trained for Kirundi language processing

Tokenizer Details

• Type: BPE (Byte-Pair Encoding)
• Vocabulary Size: 30,000 tokens
• Special Tokens: [UNK], [CLS], [SEP], [PAD], [MASK]
• Pre-tokenization: Whitespace-based

LoRA Adapter Details

• Base Model: [To be filled with your chosen base model]
• Rank: 8
• Alpha: 32
• Target Modules: Query and Value attention matrices
• Dropout: 0.05

Intended Uses & Limitations

Intended Uses

• Text processing for Kirundi language
• Machine translation tasks involving Kirundi
• Natural language understanding tasks for Kirundi content
• Foundation for developing Kirundi language applications

Limitations

• The tokenizer is trained on a specific corpus and may not cover all Kirundi dialects
• Limited to the vocabulary observed in the training data
• Performance may vary on domain-specific text

Training Data

• Dataset: eligapris/kirundi-english
• Size: 21.4k sentence pairs
• Nature: Parallel corpus with Kirundi and English translations
• Domain: Mixed domain including religious, general, and conversational text

Training Procedure

Tokenizer Training

• Trained using Hugging Face's Tokenizers library
• BPE algorithm with a vocabulary size of 30k
• Includes special tokens for task-specific usage
• Trained on the Kirundi portion of the parallel corpus

LoRA Training

• Number of epochs:
• Batch size:
• Learning rate:
• Training hardware:
• Training time:

Evaluation Results

• Coverage statistics:
• Out-of-vocabulary rate:
• Task-specific metrics:

Environmental Impact

• Estimated CO2 emissions:
• Hardware used:
• Training duration:

Technical Specifications

Model Architecture

• Tokenizer: BPE-based with custom vocabulary
• LoRA Configuration:
  • r=8 (rank)
  • α=32 (scaling)
  • Trained on specific attention layers
  • Dropout rate: 0.05

Software Requirements

How to Use

Loading the Tokenizer

Loading the LoRA Model

Contact

Updates and Versions

• v1.0.0 (Initial Release)
  • Base tokenizer and LoRA model
  • Trained on Kirundi-English parallel corpus
  • Basic functionality and documentation

Acknowledgments

• Dataset provided by eligapris
• Hugging Face's Transformers and Tokenizers libraries
• PEFT library for LoRA implementation