Warning: This model, like it's predecessor, can be rather unpredictable and may output undesired content.
This model uses all of the same data as the original Dendrite but I took it over to runpod where I could give it a much deeper and higher quality LoRA session which allowed it to regain overall coherence without the need for being merged.
I highly recommend that you have EOS tokens unbanned when using this model. If it fails to trigger an EOS it will just start repeating itself.
To recap:
Dendrite is an almagamation of Llama-2-chat13B and Enterredaas33B (both fantastic models that you should check out in and of themselves)
The block-diagonal merge script was used to graft attention heads from Enterredaas33B onto Llama-2-chat13B upping its parameter count to 22B.
Upon testing I found the results surprisingly coherent although there were some gaps in its ability to even respond at all to lengthy context (it would simply spam \n once context got to a certain point)
I used a private dataset that I constructed for previous unreleased experiments to fill in the gaps that were caused by the merge.
The model is very good at philosophical debate.
Sometimes it needs to be "woken up" at the start of a conversation by asking for self reflection. E.g. "Tell me a joke only an AI language model would understand" and then after that it is ready for some very cerebral conversations about the nature of existence itself.
I personally use it with a modified llama-2-chat prompt format for SillyTavern/Simple-proxy but it's fairly adaptable with regards to your prompt format choices so I would definitely encourage experimentation.
