File size: 555 Bytes
2f8dec7 a77660a 2f8dec7 a77660a 2f8dec7 a77660a 2f8dec7 a77660a 2f8dec7 a77660a 2f8dec7 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
---
library_name: transformers
tags: []
---
# Tokenizer Card
<!-- Provide a quick summary of what the model is/does. -->
```bash
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("azherali/python-code-search-tokenizer")
example = """class LinearLayer():
def __init__(self, input_size, output_size):
self.weight = torch.randn(input_size, output_size)
self.bias = torch.zeros(output_size)
def __call__(self, x):
return x @ self.weights + self.bias
"""
tokenizer.tokenize(example)
|