A model made to continue off my previous work on
Magnum 4B, A small model made for creative writing / General assistant tasks, finetuned ontop of
IntervitensInc/Llama-3.1-Minitron-4B-Width-Base-chatml, this model is made to be more coherent and generally be better then the 4B at both writing and assistant tasks.
Model has been Instruct tuned with the ChatML formatting. A typical input would look like this:
1"""<|im_start|>system
2system prompt<|im_end|>
3<|im_start|>user
4Hi there!<|im_end|>
5<|im_start|>assistant
6Nice to meet you!<|im_end|>
7<|im_start|>user
8Can I ask a question?<|im_end|>
9<|im_start|>assistant
10"""
To run inference on this model, you'll need to use Aphrodite, vLLM or EXL 2/tabbyAPI, as llama.cpp hasn't yet merged the required pull request to fix the llama 3.1 rope_freqs issue with custom head dimensions.
However, you can work around this by quantizing the model yourself to create a functional GGUF file. Note that until
this PR is merged, the context will be limited to 8 k tokens.
These modifications should allow you to use the model with llama. Cpp, albeit with the mentioned context limitation.
The training was done for 2 epochs. We used 2 x
RTX 6000s GPUs graciously provided by
Kubernetes_Bad for the full-parameter fine-tuning of the model.
