Biggerbrain AI 118 million parameter model is the full title of this model. It is based on the MoLaMart Architecture invented by me.
MoLaMart stands for Mixture of Layers(Dynamic FFN block switching), and Memory augmented(GRU and custom memory module), Recurrent Transformer.
(A transformer with at least 40% of its layers in an iteration loop.)
This is the space for testing the model's current capabilities:
https://huggingface.co/spaces/Skull18500/BIggerbrain-118-millionThe Github link for the project's complete files and features is:
https://github.com/skull18500/BiggerBrain-AIIm 13 y.o. trying to perfect this architecture on a 4060. Currently, my AI is very good at making 1 sentence or 2 sentence stories,
due to too much Tinystories in the training data. It used a mixture of random data shuffling(data split into 511 token sequences, then
shuffled) to make sure it didnt memorize the order. It has 6.1 Billion training tokens, but it has estimated only seen 1-3 billion. I used
A mixture of Tinystories, wikitext, Gutenberg, Etc, though some other sources were accidentally incorperated into the mix in small amounts.
