Mirror facebook/bart-large tokenizer files

README.md

@@ -0,0 +1,20 @@
+---
+tags:
+  - tokenizer
+  - bart
+library_name: transformers
+license: apache-2.0
+---
+
+# BART-large tokenizer (mirror)
+
+Files copied verbatim from [facebook/bart-large](https://huggingface.co/facebook/bart-large) for archival.
+
+## Usage
+
+```python
+from transformers import AutoTokenizer
+tk = AutoTokenizer.from_pretrained("arrow-hf/bart-large-tokenizer")
+```
+
+Used by [arrow-hf/xvla-robotwin-stack-bowls-two-40pct](https://huggingface.co/arrow-hf/xvla-robotwin-stack-bowls-two-40pct) (max_length=50). The X-VLA model uses this as its language tokenizer (Florence2 backbone consumes the BART vocabulary).

tokenizer_config.json

@@ -0,0 +1,16 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "errors": "replace",
+  "is_local": false,
+  "mask_token": "<mask>",
+  "model_max_length": 1024,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "tokenizer_class": "RobertaTokenizer",
+  "trim_offsets": true,
+  "unk_token": "<unk>"
+}