Views
No views yet
gpt2 tokenizer and further trained on the Python subset of the CodeSearchNet dataset. The tokenizer is designed to efficiently tokenize Python code, which can be useful for various downstream tasks like code generation, code completion, and code analysis.whole_func_string column of the train split from the claudios/code_search_net dataset, specifically focusing on Python code examples. The training corpus consisted of approximately 412,178 Python function strings.gpt2 tokenizer.train_new_from_iterator method from transformers.PreTrainedTokenizerFast was used to train a new vocabulary and merges from the CodeSearchNet Python code corpus. The new vocabulary size was set to 52,000 tokens.transformers library:1from transformers import AutoTokenizer
2
3# Load the tokenizer from the Hugging Face Hub
4tokenizer = AutoTokenizer.from_pretrained("rajaykumar12959/new_tokeniser")
5
6# Example usage
7example_code = """class LinearLayer():
8 def __init__(self, input_size, output_size):
9 self.weight = torch.randn(input_size, output_size)
10 self.bias = torch.zeros(output_size)
11
12 def __call__(self, x):
13 return x @ self.weights + self.bias
14 """
15
16tokens = tokenizer.tokenize(example_code)
17print(tokens)
18# Output will be similar to:
19# ['class', 'ĠLinear', 'Layer', '():', 'ĊĠĠĠ', 'Ġdef', 'Ġ__', 'init', '__(', 'self', ',', 'Ġinput', '_', 'size', ',', 'Ġoutput', '_', 'size', '):', 'ĊĠĠĠĠĠĠĠ', 'Ġself', '.', 'weight', 'Ġ=', 'Ġtorch', '.', 'randn', '(', 'input', '_', 'size', ',', 'Ġoutput', '_', 'size', ')', 'ĊĠĠĠĠĠĠĠ', 'Ġself', '.', 'bias', 'Ġ=', 'Ġtorch', '.', 'zeros', '(', 'output', '_', 'size', ')', 'ĊĊĠĠĠ', 'Ġdef', 'Ġ__', 'call', '__(', 'self', ',', 'Ġx', '):', 'ĊĠĠĠĠĠĠĠ', 'Ġreturn', 'Ġx', 'Ġ@', 'Ġself', '.', 'weights', 'Ġ+', 'Ġself', '.', 'bias', 'ĊĠĠĠĠ']
20
21encoded_input = tokenizer(example_code, return_tensors="pt")
22print(encoded_input)