cc0-1.0
Small description
VDrontV1.5-Flash is a 60M-parameter GPT-like model. Train in 1.5b tokens.
The model was trained on FineWeb-Edu + FineWeb-Edu-Instruct.
Stored in Hugging Face format and DrontQ(.dq) quantized format.
VDrontV1.5-Flash/ - HF format 32 bit
VDrontV1.5-Flash_quant/ .dq format, the folder contains DQuse.py for using .dq.
Formats
| format | bit | Path | Use | Size | Generation quality |
|---|
| Hugging Face .safetensors | 32 bit | VDrontV1.5-Flash/ | use.py | 250mb | Full |
| .dq (dront quant) | 16 bit | VDrontV1.5-Flash_quant/VDrontV1.5-Flash_Q16.dq | VDrontV1.5_quant/DQuse.py | 175mb | slightly worse |
| .dq (dront quant) | 12 bit | VDrontV1.5-Flash_quant/VDrontV1.5-Flash_Q12.dq | VDrontV1.5_quant/DQuse.py | 130mb | between 8-bit and 16-bit |
| .dq (dront quant) | 8 bit | VDrontV1.5-Flash_quant/VDrontV1.5-Flash_Q8.dq | VDrontV1.5_quant/DQuse.py | 90mb | worse |
Details
Context length: 512
Tokenizer: gpt2
Base model: VdrontV1 (ours)
Fine-tuning on GPU using the Lion optimizer
Ratio of FW-Edu to FW-Edu-instruct training = 3:1
Use of Drontal Quant.
License : cc0-1.0.
Guide
To use the HF model, go to the files, download the VDrontV1.5-Flash/ folder and use.py, and place them together. use.py includes prompt formatting with <|user|> and <|assistant|> and is ready for immediate use.
To use Dront Quant models, go to the VDrontV1.5-Flash_quant/ folder. It contains 3 .dq models (8-bit, 12-bit, and 16-bit) as well as DQuse.py. Download the model you are interested in and place DQuse.py and the .dq model next to each other. At the end of the DQuse.py file, change the model name to the one you want to use. Note that the prompt formatting uses <|user|> and <|assistant|> tokens. (388 line: chat_with_dq_model("[path].dq", 256) )
Fine-tune
If you fine-tune the VDrontV1.5-Flash model, we will most likely add it to the VDrontV collection.
Python libraries:
DQuse.py : torch json os struct numpy collections transformers
use.py : torch transformers threading
Hugging Face : VDC-team/
