Commit
·
669ef59
1
Parent(s):
f7848c8
Update README.md
Browse files
README.md
CHANGED
|
@@ -9,9 +9,9 @@ The Code Search Net Tokenizer is a custom tokenizer specifically trained for tok
|
|
| 9 |
|
| 10 |
**Model Details:**
|
| 11 |
|
| 12 |
-
Name: Code Search Net Tokenizer
|
| 13 |
-
Model Type: Custom Tokenizer
|
| 14 |
-
Language: Python
|
| 15 |
|
| 16 |
**Training Data:**
|
| 17 |
|
|
@@ -19,9 +19,9 @@ The tokenizer was trained on a corpus of Python code snippets from the CodeSearc
|
|
| 19 |
|
| 20 |
**Tokenizer Features:**
|
| 21 |
|
| 22 |
-
|
| 23 |
|
| 24 |
-
|
| 25 |
|
| 26 |
**Usage:**
|
| 27 |
|
|
|
|
| 9 |
|
| 10 |
**Model Details:**
|
| 11 |
|
| 12 |
+
- Name: Code Search Net Tokenizer
|
| 13 |
+
- Model Type: Custom Tokenizer
|
| 14 |
+
- Language: Python
|
| 15 |
|
| 16 |
**Training Data:**
|
| 17 |
|
|
|
|
| 19 |
|
| 20 |
**Tokenizer Features:**
|
| 21 |
|
| 22 |
+
- The Code Search Net Tokenizer offers the following features:
|
| 23 |
|
| 24 |
+
- Tokenization of Python code: The tokenizer can effectively split Python code snippets into individual tokens, making it suitable for downstream tasks that involve code processing and understanding.
|
| 25 |
|
| 26 |
**Usage:**
|
| 27 |
|