| --- |
| license: mit |
| tags: |
| - tokenizer |
| - code |
| - python |
| --- |
| |
| # Python Code Tokenizer (CodeSearchNet) |
|
|
| A domain-specific tokenizer trained from the GPT-2 tokenizer, fine-tuned on Python source code to better handle code syntax, identifiers, and structure compared to a general-purpose English tokenizer. |
|
|
| ## Details |
|
|
| - **Base tokenizer:** GPT-2 (`gpt2`) |
| - **Training corpus:** [CodeSearchNet](https://huggingface.co/datasets/code-search-net/code_search_net), Python split |
| - **Vocabulary size:** 52,000 |
| - **Training method:** `train_new_from_iterator` (Hugging Face `tokenizers` library) |
|
|
| ## Motivation |
|
|
| Standard NLP tokenizers like GPT-2's are trained on natural language and tend to over-fragment code — splitting common patterns like indentation, snake_case identifiers, and Python keywords into many small subword tokens. Training on a code-specific corpus lets the tokenizer learn more efficient, code-aware merges, reducing the number of tokens needed to represent typical Python source. |
| |
| ## Usage |
| |
| ```python |
| from transformers import AutoTokenizer |
| |
| tokenizer = AutoTokenizer.from_pretrained("AlexStamp/code-search-net-tokenizer") |
| tokens = tokenizer.tokenize("def hello_world():\n print('Hello!')") |
| ``` |
| |
| ## Notes |
| |
| This tokenizer was trained as part of working through the Hugging Face LLM course (Chapter 6), as a portfolio exercise in tokenizer training and domain adaptation. |
