gustavecortal/

fr-boris-8bit

This is a version of Cedille's GPT-J (fr-boris) with 6 billion parameters that is modified so you can generate and fine-tune the model in colab or equivalent desktop gpu (e.g. single 1080Ti). Inspired by GPT-J 8bit.

Here's how to run it:

This model can be easily loaded using the GPTJForCausalLM functionality:

Boris is a 6B parameter autoregressive language model based on the GPT-J architecture and trained using the mesh-transformer-jax codebase.

Boris was trained on around 78B tokens of French text from the C4 dataset.