Model Card — J-Raposo/code-search-net-tokenizer

Model name

Short description

Model details

• Author: J-Raposo (Hugging Face username: J-Raposo)
• Model type: Tokenizer (Byte-level BPE, GPT-2 style)
• Language(s): Python (source code), English (comments & docstrings)
• License: [To be defined by repo owner — e.g., mit, apache-2.0]
• Intended use: Tokenization for code modeling tasks (code search, code completion, summarization, classification, and fine-tuning code LLMs on Python).
• Not intended for: Producing runnable or secure code without downstream model fine-tuning; this tokenizer only affects tokenization behavior, not model logic or correctness of generated code.

Summary

Training data

• Dataset: CodeSearchNet — python subset (loaded via datasets.load_dataset("code_search_net", "python")).
• Preprocessing: Source files and docstrings were extracted. Common normalization steps applied (e.g., newline normalization). Comments and docstrings were retained to preserve natural language context alongside code.
• Notes: Tokenizer was trained only on the Python portion; tokenization quality for other languages (JavaScript, Java, C, etc.) may be lower.

Tokenizer details / configuration

• Tokenizer type: Byte-level BPE (GPT-2–style / tokenizers fast API).
• Vocabulary size: 50,257 (GPT-2 default)
• Special tokens: standard GPT-2 tokens (e.g., ``) or custom tokens if you added any. Ensure tokenizer_config.json in the repo lists them.
• Normalization: Byte-level normalization (works with arbitrary byte sequences / UTF-8).
• Files included: tokenizer.json (preferred tokenizers fast format) or vocab.json + merges.txt (legacy), and tokenizer_config.json.

Uses

Direct Use

• Tokenize Python code and docstrings for input into language models.
• Use as a drop-in tokenizer when fine-tuning GPT-2–style or encoder-decoder models for code tasks if they support the tokenizer format.

Downstream Use

• Fine-tuning code generation or code search LLMs.
• Preprocessing pipelines for supervised tasks on code (classification, summarization, code-to-text).
• As a tokenizer for dataset preparation for model pretraining/finetuning on code corpora.

Out-of-Scope Use

• This tokenizer alone does not produce correct or secure code — it only affects token representation. Use caution when deploying downstream models that generate or modify code; do not rely on tokenization to ensure correctness or security.

Bias, Risks, and Limitations

• Data bias: The tokenizer reflects distributional properties of public repositories in CodeSearchNet: common libraries, styles, and naming conventions are better represented than niche or private coding styles.
• Technical limitations: Training on the Python subset causes suboptimal tokenization for other languages. Extremely long or adversarial identifiers may still be split into many sub-tokens.
• Downstream risks: Tokenization decisions affect model training and generation; poor tokenization can amplify biases or lead to awkward/generated outputs in downstream models. Tokenizers do not mitigate issues like hallucinations, insecure code generation, or toxic outputs.

Recommendations

• Use this tokenizer for Python-focused models or mixed pipelines where Python is dominant.
• Evaluate tokenization quality on your downstream tasks (e.g., token length distributions, OOV handling).
• If you plan to use proprietary source code for training, do not upload proprietary content to public repos — consider training a private tokenizer or using private HF repos.

How to get started (load & use)