We propose GrammarCoder, a grammar-based model built on a decoder-only architecture, which excels in auto-regressive tasks like code generation, completion, and translation. To enhance its ability to code generation, we apply continued pre-training and instruction tuning on existing code model weights (i.e., DeepSeek-Coder-1.3B-Base, Qwen2.5-1.5B-Base, and Qwen2.5-7B-Base), expanding its knowledge base.

Compared with the model with the same experiment setting, Grammar-coder gained a better preformance on the dataset. The following table presents the code generation accuracy compared with the baseline.

The model has been open-sourced, and the model and the corresponding tokenizer are stored in HuggingFace-GrammarCoder.

Please refer to Github (https://github.com/LIANGQINGYUAN/GrammarCoder) for details on the evaluation.

@article{liang2025grammar,
  title={Grammar-Based Code Representation: Is It a Worthy Pursuit for LLMs?},
  author={Liang, Qingyuan and Zhang, Zhao and Sun, Zeyu and Lin, Zheng and Luo, Qi and Xiao, Yueyi and Chen, Yizhou and Zhang, Yuqun and Zhang, Haotian and Zhang, Lu and Bin, Chen and Yingfei Xiong},
  journal={arXiv preprint arXiv:2503.05507},
  year={2025}
}