RichardErkhov/

1bitLLM_-_bitnet_b1_58-large-gguf

Quantization made by Richard Erkhov.

bitnet_b1_58-large - GGUF

This is a reproduction of the BitNet b1.58 paper. The models are trained with RedPajama dataset for 100B tokens. The hypers, as well as two-stage LR and weight decay, are implemented as suggested in their following paper. All models are open-source in the repo. We will train larger models and/or more tokens when resource is available.

PPL and zero-shot accuracy:

The differences between the reported numbers and the reproduced results are possibly variances from the training data processing, seeds, or other random factors.

The evaluation pipelines are from the paper authors. Here is the commands to run the evaluation:

python eval_task.py --hf_path 1bitLLM/bitnet_b1_58-3B \
    --batch_size 1 \
    --tasks \
    --output_path result.json \
    --num_fewshot 0 \
    --ctx_size 2048