checkpoints/best.pt — best validation checkpoint; recommended for generation.
checkpoints/latest.pt — final early-stopped checkpoint.
checkpoints/ckpt_*.pt — retained periodic checkpoints.
artifacts/final_report.json — final training report.
artifacts/train.log — full training log.
artifacts/accuracy_val_best.json and accuracy_val_latest.json — held-out next-byte accuracy estimates.
artifacts/data_meta.json — cached byte-shard metadata.
source/ — full ks_byte_lm project source used for the run.

Key metrics

Validation next-byte accuracy using best.pt: 76.42% on 3,276,800 evaluated byte tokens.

Training summary:

Best validation BPB: 0.9593
Final validation BPB: 0.9862
Final CE: 0.6836
Final word PPL estimate: 849.0
Early stopped at step 4751 / 5000

Dataset/cache metadata:

Train byte tokens: 45,362,173
Validation byte tokens: 1,622,371
Test byte tokens: 3,074,698
Words: 5,074,066

Usage

This is not a standard Transformers-format model. Do not use AutoModel.from_pretrained. It is a custom PyTorch checkpoint release for the included ksbyte source code.

Recommended checkpoint:

checkpoints/best.pt

best.pt is preferred over latest.pt because it had the best validation BPB.

1. Download and install

If the repository is private, first login with an account that has access:

bash

pip install -U huggingface_hub
huggingface-cli login

Download the full snapshot:

bash

python - <<'PY'
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="Omarrran/ks-byte-lm-spacebyte-v1",
    repo_type="model",
    local_dir="ks-byte-lm-spacebyte-v1",
    local_dir_use_symlinks=False,
)
PY

Install dependencies:

bash

cd ks-byte-lm-spacebyte-v1
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

2. Generate text

Make the bundled source package importable and run generation:

bash

export PYTHONPATH="$PWD/source:$PYTHONPATH"

PYTHONUTF8=1 python -m ksbyte.generate \
  --ckpt checkpoints/best.pt \
  --prompt "کٲشِرۍ پَلو چھِ اکثر خٕطہٕ کِس تٲریٖخٕچ دٔلیٖل ونان، یُس مُختٔلِف ثقافتَن ہٕنٛدۍ اثرات ظٲہِر کران" \
  --n 200 \
  --temperature 0.8 \
  --top_k 50 \
  --device auto

--device auto uses CUDA when available and falls back to CPU otherwise.

Example verified output from the trained checkpoint:

 کٲشِرۍ پَلو چھِ اکثر خٕطہٕ کِس تٲریٖخٕچ دٔلیٖل ونان، یُس مُختٔلِف ثقافتَن ہٕنٛدۍ اثرات ظٲہِر کران

3. Minimal Python API

python

import torch
from ksbyte.generate import load_model, encode_prompt, decode_tokens
from ksbyte.config import EOS_ID

ckpt = "checkpoints/best.pt"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model, cfg = load_model(ckpt, device)
idx = torch.tensor([encode_prompt("کٲشِرۍ پَلو چھِ اکثر خٕطہٕ کِس تٲریٖخٕچ دٔلیٖل ونان، یُس مُختٔلِف ثقافتَن ہٕنٛدۍ اثرات ظٲہِر کران", cfg)], dtype=torch.long, device=device)

out = model.generate(
    idx,
    max_new_tokens=200,
    temperature=0.8,
    top_k=50,
    eos_id=EOS_ID,
)
print(decode_tokens(out[0].tolist()))

Troubleshooting:

If ModuleNotFoundError: No module named 'ksbyte', run export PYTHONPATH="$PWD/source:$PYTHONPATH" from the repo root.
If CUDA is unavailable, use --device cpu or keep --device auto.
Lower --temperature such as 0.7 for more conservative generation.

Checkpoint notes

Use checkpoints/best.pt for generation and evaluation. checkpoints/latest.pt is the final early-stopped checkpoint and is slightly weaker than best.pt.

Caveats

This is a small experimental byte-level LM. It produces Kashmiri-looking text and has learned script/byte structure, but generations can be incomplete, hallucinated, or semantically weak.

License

No explicit open-source license is declared for this experimental release. Contact the repository owner for reuse permissions.

Faizaniqbal/ks-byte-lm-spacebyte-v1

ks_byte_lm SpaceByte v1

Contents

Key metrics

Usage

1. Download and install

2. Generate text

3. Minimal Python API

Checkpoint notes

Caveats

License

Faizaniqbal/
ks-byte-lm-spacebyte-v1