You can deploy this model and start earning money today!

UIC-AI-lab/
llama2-tokenizer

Name: llama2-tokenizer
Availability: InStock

Views

No views yet

Llama-2 Tokenizer (Mirror for KnowRL Project)

This is a mirror of the tokenizer files from meta-llama/Llama-2-7b-hf, provided as a public, gated-free alternative for users who cannot access the original gated repo.

Why this mirror exists

The KnowRL project's QuCo reward function uses an Infini-gram index built with the Llama-2 tokenizer. To query the index, the exact same tokenizer is required. Since meta-llama/Llama-2-7b-hf is gated, users without approved access cannot run QuCo.

This repo contains only the tokenizer files (no model weights):

tokenizer.json — fast tokenizer
tokenizer.model — SentencePiece model
tokenizer_config.json — tokenizer configuration
special_tokens_map.json — special token mappings

Usage

python

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("UIC-R2-lab/llama2-tokenizer")

License

Follows the original Llama 2 Community License Agreement from Meta.

UIC-AI-lab/llama2-tokenizer

Llama-2 Tokenizer (Mirror for KnowRL Project)

Why this mirror exists

Usage

License

UIC-AI-lab/
llama2-tokenizer