MicroBOB-python is a new, from scratch micro model based on RWKV x051a, which doesnt require a special kernel to train or inference. Developed and trained using a modified version of nanoRWKV.

MicroBOB-python is a new, from scratch micro model based on RWKV x051a, which doesnt require a special kernel to train or inference. Developed and trained using a modified version of nanoRWKV. Base trained on 10's of thousands of lines of opensource and internal python code, and finetuned in 5 rounds using kejian/codesearchnet-python-raw, AdapterOcean/python-code-instructions-18k-alpaca-standardized_cluster_1_alpaca and 3 others in the same series.

Developed for an in-house python code editor to act as a simple autocomplete, it's gotten smart enough for it's extreme small size (30 million parameters) I thought I should share. Model weights only licenced under MIT.

Simple autocompletion of python or python syntax like code.

https://github.com/BlinkDL/nanoRWKV sample.py is sufficient to inference, and the training script included does work with the model weights, though at a slight loss of performance as far as training speed and memory usage during training, but should produce functionally identical results.

Code replacement and re-formatting - It is possible with a small amount of finetuning and clever python code to use the model to replace words, functions, and variables in python code.

HRLF Training can be used to instruct train the model with limited success. HRLF training blunts the models intelligence due to the limited availible parameters. The HRLF training replaces info from the dataset in the model weights.

Model has no bias training or safety guardrails. It was trained on code from open web sources. The model may incidentally produce mallicious or insecure code. Use at your own risk! You are fully responsible for any generations produced using these model weights.

Clone https://github.com/BlinkDL/nanoRWKV. All the code contained therein is compatible with this model. While the code that generated the model is optimized and customized, the base nanoRWKV package can finetune the MicroBOB-python weights without issue at a reduced performance memory-wise.

https://huggingface.co/datasets/AdapterOcean/python-code-instructions-18k-alpaca-standardized_cluster_1_alpaca https://huggingface.co/datasets/AdapterOcean/python-code-instructions-18k-alpaca-standardized_cluster_2_alpaca https://huggingface.co/datasets/AdapterOcean/python-code-instructions-18k-alpaca-standardized_cluster_3_alpaca https://huggingface.co/datasets/AdapterOcean/python-code-instructions-18k-alpaca-standardized_cluster_4_alpaca https://huggingface.co/datasets/kejian/codesearchnet-python-raw My personal python code folder with 40+ projects and 30k lines of code

Standard nanoRWKV data prep, custom training loop.

Tokenized all datasets with gpt2 encoding for simplicity. Version of MicroBOB with custom BPE encoder in development.