This is the model card of Falcon-E-1.2-10B-Exp, an experimental ternary (1.58bits) language model trained on SFT agentic, and STEM data using axolotl framework combined with onebitllm library.

The model has been trained starting from tiiuae/Falcon3-10B-1.58bits-Base-prequantized checkpoint using full-finetuning for 3 epochs. That model has been 'hacked' by simply injecting quantization scales inside ternary model weights, and saved in bf16 format to enable training, thus the performance of the model is very limited. Below are the hyper-parameters used for fine-tuning:

And we used a context parallel size of 8.

The model uses think mode by default, this can be disabled and switched to non-thiking mode. You can use the model with different frameworks such as HF transformers, llama.cpp or mlx-lm

You can further fine-tune this model, or the base model using their prequantized version. Refer to the axolotl config to get started on fine-tuning these models:

Falcon-E-3B-Chat-Exp models are built using Falcon LLM technology from the Technology Innovation Institute.