LongLLaMA-Code 7B Instruct is LongLLaMA-Code 7B tuned on TIGER-Lab/MathInstruct, OpenOrca, and ShareGPT-Processed datasets. It can answer basic questions about research papers and code. It can also perform a simple code refactoring. You can try the quantized version of the model using a free GPU in Google Colab.

The model was tuned on a TPU v3-128 pod with 128 batch size.
For tuning, we have used the data preparation pipeline available in instruction_fine_tuning. However, we have replaced the Hugging Face Trainer with a modification of FoT continued pretraining code. This modification boils down to propagating the memory cache throughout the model (basically reproducing the Pytorch inference code functionality in JAX).

Here, we present the basic information about how the model was tuned. For more details, see the GitHub repo.

All inputs were truncated and randomly padded (left/right) to 3072 tokens.
The last context length was set to 1536.
The model was trained for 9k steps, started with a learning rate of 1.2e-5, 700 steps of warmup, and finished with a learning rate of 0.
The optimizer was adamw.

The question prompt (pre_question_text) was:

To trigger the model answer one can use:

The chat prompt was:

To denote the assistant one can write:

To denote the user one can write:

To improve the quality of the data, the datasets were filtered using regular expressions.

The instruction/chat-tuned models are for research purposes only. LongLLaMA-Code 7B Instruct is LongLLaMA-Code 7B tuned on TIGER-Lab/MathInstruct, OpenOrca, and ShareGPT-Processed datasets. Note that those datasets contain outputs from ChatGPT. See also the codellama/CodeLlama-7b-hf license.

We gratefully acknowledge the TPU Research Cloud program, which was instrumental to our research by providing significant computational resources.