This Model is a first test to combine
Jamba architecture with 1.58 bits linear layers, mixture of attention head and mixture of depth.
The goal is to developpe and test if this kind of architectures have not too much quality loss for a fast inference.
If you want to test this model please look at this repo at this
commitThe data fit the model max length i.e. 512 tokens
