This is the medium version of the bilinear transformers trained on FineWeb-edu.
The primary purpose of this model is interpretability, most design choices were made with that in mind.
The code to run this custom model can be found
here, along with many utility functions for weight-based interpretability.
