Skier8402
/

code-search-net-tokenizer

Text Generation

Model card Files Files and versions

Skier8402 commited on Jan 10, 2024

Commit

23c6c52

·

1 Parent(s): 300b8e4

Create README.md

Files changed (1) hide show

README.md +7 -0

README.md ADDED Viewed

	@@ -0,0 +1,7 @@

+---
+license: cc0-1.0
+datasets:
+- code_search_net
+library_name: transformers
+---
+This is an adapted tokenizer from GPT2 that can recognize tokens to do with Python coding. It is part of the [huggingfaceNLP course exercise](https://huggingface.co/learn/nlp-course/chapter6/2). It uses the method `train_new_from_iterator()`