import sentencepiece as spm
import torch

# Load tokenizer
sp = spm.SentencePieceProcessor(model_file='tokenizer.model')

# Load model
model = torch.load('pytorch_model.bin', map_location='cpu')

Tokenizing Text
   Copied # Encode
text = "Asalam"
pieces = sp.encode_as_pieces(text)
ids = sp.encode_as_ids(text)

# Decode
decoded = sp.decode_ids(ids) Files

pytorch_model.bin: Model weights
config.json: Model configuration
tokenizer.model: SentencePiece BPE tokenizer
tokenizer_config.json: Tokenizer configuration
special_tokens_map.json: Special tokens mapping

Languages
Supports Tamazight text in multiple scripts:

Tifinagh: Traditional Amazigh script
Latin: Latin transliteration
Arabic: Arabic script representation

Created on: 2026-02-07

Velkamez/tamazight-sp-bpe

Tamazight Language Model with SentencePiece BPE Tokenizer

Model Details

Architecture

Tokenizer

Usage

Loading the Model

Velkamez/
tamazight-sp-bpe