A domain-specific tokenizer trained from the GPT-2 tokenizer, fine-tuned on Python source code to better handle code syntax, identifiers, and structure compared to a general-purpose English tokenizer.
Standard NLP tokenizers like GPT-2's are trained on natural language and tend to over-fragment code — splitting common patterns like indentation, snake_case identifiers, and Python keywords into many small subword tokens. Training on a code-specific corpus lets the tokenizer learn more efficient, code-aware merges, reducing the number of tokens needed to represent typical Python source.
1from transformers import AutoTokenizer
2
3tokenizer = AutoTokenizer.from_pretrained("AlexStamp/code-search-net-tokenizer")
4tokens = tokenizer.tokenize("def hello_world():\n print('Hello!')")
This tokenizer was trained as part of working through the Hugging Face LLM course (Chapter 6), as a portfolio exercise in tokenizer training and domain adaptation.
