Upload the model

README.md

@@ -0,0 +1,170 @@
+# BertForTokenClassificationWithFourO
+
+[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/USERNAME/MODEL_NAME)
+
+A specialized token classification model built on BERT with a custom classifier for Persian text spacing and formatting tasks.
+
+## Model Description
+
+This model is built on a BERT architecture with a custom token classification head called FourOClassifier. It's specifically designed for processing Persian text to correct or add proper spacing characters.
+
+### Task
+
+The model performs token classification to detect where spacing characters should be inserted in Persian text. It can operate in two modes:
+- **Spacing Mode**: Uses pure model predictions to insert spaces
+- **Correction Mode**: Combines model predictions with existing spacing in the text
+
+### Model Architecture
+
+The model is based on the BERT architecture with a custom classifier head (FourOClassifier) that includes:
+- Dense layer with ReLU activation
+- Dropout for regularization
+- Batch normalization
+- Output projection layer
+
+## Usage
+
+### Installation
+
+```bash
+pip install transformers torch
+```
+
+### Basic Usage
+
+```python
+from transformers import AutoTokenizer
+from modeling_custom import BertForTokenClassificationWithFourO
+from labeler import Labeler
+import torch
+
+# Load model and tokenizer
+model_path = "USERNAME/MODEL_NAME"
+tokenizer = AutoTokenizer.from_pretrained(model_path)
+model = BertForTokenClassificationWithFourO.from_pretrained(model_path)
+model.eval()
+
+# Initialize labeler
+labeler = Labeler(tags=(1, 2),
+                 regexes=(r'[^\S\r\n\v\f]', r'\u200c'),
+                 chars=(" ", "‌"),
+                 class_count=2)
+
+# Process text
+def process_text(text, mode="space"):
+    # Create a pipeline for processing
+    from run import ModelPipeline
+    pipeline = ModelPipeline(model_path)
+    result = pipeline.process_text(text, mode)
+    return result
+
+# Example
+text = "این متن نمونه فارسی بدون فاصله گذاری مناسب است"
+result = process_text(text, mode="space")
+print(result)
+```
+
+### Command-line Usage
+
+You can also use the provided command-line interface:
+
+```bash
+python run.py --text "متن فارسی شما در اینجا" --mode space
+```
+
+Or process a file:
+
+```bash
+python run.py --file input.txt --output result.txt --mode correct
+```
+
+The repository includes a sample `input.txt` file that you can use to test the model.
+
+## Parameters
+
+- `mode`: 
+  - `space`: Uses model predictions to add spaces
+  - `correct`: Combines model predictions with original text spacing (recommended for texts with some correct spacing)
+
+## Evaluation
+
+The model achieves excellent performance in both operating modes:
+
+### Spacing Mode Evaluation
+
+```
+╒═════════╤═════════════╤══════════╤════════════╤════════════╕
+│ Label   │   Precision │   Recall │   Accuracy │   F1 Score │
+╞═════════╪═════════════╪══════════╪════════════╪════════════╡
+│ 0       │    0.994663 │ 0.997324 │   0.997324 │   0.995992 │
+├─────────┼─────────────┼──────────┼────────────┼────────────┤
+│ 1       │    0.989546 │ 0.987828 │   0.987828 │   0.988686 │
+├─────────┼─────────────┼──────────┼────────────┼────────────┤
+│ 2       │    0.913413 │ 0.932125 │   0.932125 │   0.922674 │
+├─────────┼─────────────┼──────────┼────────────┼────────────┤
+│ Average │    0.965874 │ 0.972426 │   0.972426 │   0.969117 │
+╘═════════╧═════════════╧══════════╧════════════╧════════════╛
+```
+
+### Correction Mode Evaluation
+
+```
+╒═════════╤═════════════╤══════════╤════════════╤════════════╕
+│ Label   │   Precision │   Recall │   Accuracy │   F1 Score │
+╞═════════╪═════════════╪══════════╪════════════╪════════════╡
+│ 0       │    0.995932 │ 0.998386 │   0.998386 │   0.997157 │
+├─────────┼─────────────┼──────────┼────────────┼────���───────┤
+│ 1       │    0.992917 │ 0.992227 │   0.992227 │   0.992572 │
+├─────────┼─────────────┼──────────┼────────────┼────────────┤
+│ 2       │    0.944612 │ 0.959428 │   0.959428 │   0.951962 │
+├─────────┼─────────────┼──────────┼────────────┼────────────┤
+│ Average │    0.97782  │ 0.983347 │   0.983347 │   0.980564 │
+╘═════════╧═════════════╧══════════╧════════════╧════════════╛
+```
+
+Note that the correction mode achieves slightly better results by combining model predictions with existing text spacing.
+
+### Label Meaning
+- Label 0: No spacing needed
+- Label 1: Regular space character needed
+- Label 2: ZWNJ character (‌) needed
+
+## Use Cases
+
+This model is particularly useful for:
+- Correcting Persian text with improper spacing
+- Normalizing text from different sources
+- Improving text readability for downstream NLP tasks
+- Preprocessing Persian text for search engines or text analysis
+
+## Training
+
+The model was trained on [DATASET_NAME] of Persian text with proper spacing annotations.
+
+Training hyperparameters:
+- Learning rate: [VALUE]
+- Batch size: [VALUE]
+- Training steps: [VALUE]
+- [OTHER PARAMETERS]
+
+## Limitations
+
+- The model is specifically designed for Persian text
+- Performance may vary on specialized domains or technical texts
+- Very long texts should be processed in chunks for optimal performance
+- Tuned for execution on devices with CUDA
+- [ANY OTHER LIMITATIONS]
+
+## Citation
+
+```
+[CITATION_INFO]
+```
+
+## License
+
+[LICENSE_INFO]
+
+## Contact
+
+For questions or feedback, please contact [CONTACT_INFO].

__init__.py

@@ -0,0 +1 @@
+from .modeling_custom import BertForTokenClassificationWithFourO, FourOClassifier

config.json

@@ -0,0 +1,41 @@
+{
+  "_name_or_path": "bert-base-multilingual-uncased",
+  "architectures": [
+    "BertForTokenClassificationWithFourO"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "directionality": "bidi",
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "LABEL_0",
+    "1": "LABEL_1",
+    "2": "LABEL_2"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "LABEL_0": 0,
+    "LABEL_1": 1,
+    "LABEL_2": 2
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "pooler_fc_size": 768,
+  "pooler_num_attention_heads": 12,
+  "pooler_num_fc_layers": 3,
+  "pooler_size_per_head": 128,
+  "pooler_type": "first_token_transform",
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.44.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 105879
+}

input.txt

@@ -0,0 +1 @@
+میدانید که ادبیات فارسی گنجینه‌ای از آثار پر مغز و زیباست. خواندن متون فاخر فارسی نه تنها موجب افزایش دانش‌ بلکه باعث غنی‌تر شدن روح انسان می‌شود.بسیاری از اشعار ونثرهای کهن، مانند آثارحافظ‌سعدی،و فردوسی،باید با دقت خوانده شوند تا معنای کامل آنهارا دریافت کنیم.درک معنای ژرف چنینآثاری بدون داشتن دانش کافی از زبان و فرهنگ فارسی‌ امکانپذیر نیست.بنابراین، آموزش درست به کودکان و نوجوانان نقشحیاتی در زنده نگه داشتن این میراث فرهنگی دارد.فضای مجازی امروز نقش عمده‌ای در انتشار آثار فارسی دارد،اما بایدبه شیوه‌ای صحیح ازآن بهره ببریم تا از تحریف یا اشتباه در انتقال متون جلوگیری شود.

labeler.py

@@ -0,0 +1,179 @@
+import re
+import numpy as np
+import sys
+
+# Set the print option for numpy arrays to display the whole array without truncation
+np.set_printoptions(threshold=sys.maxsize)
+
+
+class Labeler:
+    # Define corpus types as class constants
+    WHOLE_RAW = 'whole_raw'
+    SENTS_RAW = 'sents_raw'
+
+    def __init__(self, tags=(1, 2),
+                 regexes=(r'[^\S\r\n\v\f]', r'\u200c'),
+                 chars=(" ", "‌"),
+                 class_count=2):
+        self._tags = tags
+        self._regexes = regexes
+        self._class_chars = chars
+        self.class_count = class_count
+
+        self.data = None
+        self.labels = []
+        self.corpus_type = None
+
+    def _sent_labeler(self, sent: str):
+        """Label a single sentence and return characters and labels.
+
+        Args:
+            sent: The sentence to be labeled
+
+        Returns:
+            A tuple of (characters, labels)
+        """
+        # Initialize an empty list to store the labels
+        labels = [0] * len(sent)
+        # Convert the input sentence into a list of characters for the output
+        characters = list(sent)
+        # Initialize an empty list to store the indices of characters to be deleted
+        deletable = []
+
+        # Loop through the classes
+        for i in range(self.class_count):
+            # Find all the matches of the regular expression for the current class in the sentence
+            for match in re.finditer(self._regexes[i], sent):
+                # Get the index of the match
+                idx = match.start()
+                # Assign the corresponding tag to the label of the character before the match
+                labels[idx - 1] = self._tags[i]
+                # Add the index of the match to the list of deletable
+                deletable.append(idx)
+
+        # Sort the deletable in descending order to avoid index shifting
+        deletable = sorted(deletable, reverse=True)
+
+        # Remove the deletable characters and their labels
+        for idx in deletable:
+            characters.pop(idx)
+            labels.pop(idx)
+
+        return characters, labels
+
+    def _text_labeler(self):
+        """Label the whole text and return characters and labels."""
+        # Initialize labels with all zeros
+        labels = [0] * len(self.data)
+        # Convert characters to a list
+        characters = list(self.data)
+        # Track indices to delete
+        deletable = []
+
+        # Loop through the classes
+        for i in range(self.class_count):
+            # Find all matches for the current class
+            for match in re.finditer(self._regexes[i], self.data):
+                idx = match.start()
+                # Label the character before the match
+                labels[idx - 1] = self._tags[i]
+                # Mark this character for deletion
+                deletable.append(idx)
+
+        # Sort deletable indices in descending order
+        deletable.sort(reverse=True)
+
+        # Delete characters and labels at the specified indices
+        for idx in deletable:
+            del characters[idx]
+            del labels[idx]
+
+        return characters, labels
+
+    def _labeler(self):
+        """Label the data and return characters and labels."""
+        # Initialize empty lists for results
+        result_chars = []
+        result_labels = []
+
+        # Process based on corpus type
+        if self.corpus_type == self.SENTS_RAW:
+            for sent in self.data:
+                # Label each sentence individually
+                characters, labels = self._sent_labeler(sent)
+                result_chars.append(characters)
+                result_labels.append(labels)
+        elif self.corpus_type == self.WHOLE_RAW:
+            # Label the entire text at once
+            result_chars, result_labels = self._text_labeler()
+            # Wrap the results in lists to maintain consistent return structure
+            result_chars = [result_chars]
+            result_labels = [result_labels]
+
+        return result_chars, result_labels
+
+    def label_text(self, textinput, corpus_type):
+        """Label text and return characters and labels.
+
+        Args:
+            textinput: Either a string or a list of strings to label
+            corpus_type: Either Labeler.WHOLE_RAW or Labeler.SENTS_RAW
+
+        Returns:
+            A tuple of (characters, labels)
+        """
+
+        # Validate input types
+        if corpus_type == self.WHOLE_RAW and isinstance(textinput, str):
+            self.data = textinput
+            self.corpus_type = corpus_type
+        elif corpus_type == self.SENTS_RAW and isinstance(textinput, list):
+            self.data = textinput
+            self.corpus_type = corpus_type
+        else:
+            raise ValueError(f"Invalid input: expected {corpus_type} with compatible data type")
+
+        return self._labeler()
+
+    def _text_generator(self, chars, labels):
+        """Generate text with labels inserted.
+
+        Args:
+            chars: A list of characters
+            labels: A list of labels for those characters
+
+        Returns:
+            A string with class characters inserted according to the labels
+        """
+        result = []
+        for char, label in zip(chars, labels):
+            # Always add the character
+            result.append(char)
+
+            # Add class character if needed
+            if label != 0:
+                for i in range(self.class_count):
+                    if label == self._tags[i]:
+                        result.append(self._class_chars[i])
+                        break
+
+        return ''.join(result)
+
+    def text_generator(self, chars, labels, corpus_type):
+        """Generate text with labels inserted.
+
+        Args:
+            chars: Either a list of characters or a list of lists of characters
+            labels: Either a list of labels or a list of lists of labels
+            corpus_type: Either Labeler.WHOLE_RAW or Labeler.SENTS_RAW
+
+        Returns:
+            Either a string or a list of strings with class characters inserted
+        """
+        if corpus_type == self.SENTS_RAW:
+            # Process each sentence separately
+            return [self._text_generator(sent_chars, sent_labels)
+                    for sent_chars, sent_labels in zip(chars, labels)]
+        elif corpus_type == self.WHOLE_RAW:
+            # Process the whole text at once
+            return self._text_generator(chars, labels)

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7af3dfbb0383158d574242df30addc291152cf5dce6948c96118460d87362666
+size 669471252

modeling_custom.py

@@ -0,0 +1,59 @@
+import torch
+import torch.nn as nn
+from transformers import BertForTokenClassification
+
+
+class FourOClassifier(nn.Module):
+    def __init__(self, clf_hidden_size, num_labels):
+        super(FourOClassifier, self).__init__()
+        self.dense = nn.Linear(clf_hidden_size, clf_hidden_size)
+        self.activation = nn.ReLU()
+        self.dropout = nn.Dropout(p=0.1)
+        self.batch_norm = nn.BatchNorm1d(clf_hidden_size)
+        self.output_layer = nn.Linear(clf_hidden_size, num_labels)
+
+    def forward(self, clf_input):
+        x = self.dense(clf_input)
+        x = self.activation(x)
+        x = self.dropout(x)
+        x = self.batch_norm(x.permute(0, 2, 1)).permute(0, 2, 1)  # BatchNorm1d expects (N, C, L)
+        x = self.output_layer(x)
+        return x
+
+
+class BertForTokenClassificationWithFourO(BertForTokenClassification):
+    def __init__(self, config):
+        super().__init__(config)
+        self.num_labels = config.num_labels
+        self.classifier = FourOClassifier(config.hidden_size, config.num_labels)
+        self.init_weights()
+
+    @classmethod
+    def from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs):
+        model = super().from_pretrained(pretrained_model_name_or_path, *model_args, **kwargs)
+        model.check_classifier_initialization()
+        return model
+
+    def check_classifier_initialization(self):
+        # Check if classifier weights seem to be randomly initialized
+        def is_randomly_initialized(tensor):
+            return torch.abs(tensor.mean()) < 1e-3 < tensor.std() < 1e-1
+
+        classifier_weights = [
+            self.classifier.dense.weight,
+            self.classifier.dense.bias,
+            self.classifier.output_layer.weight,
+            self.classifier.output_layer.bias
+        ]
+
+    def freeze_bert(self):
+        """Freezes the BERT layers to prevent their parameters from being updated during training."""
+        for param in self.bert.parameters():
+            param.requires_grad = False
+        print("BERT layers frozen.")
+
+    def unfreeze_bert(self):
+        """Unfreezes the BERT layers to allow their parameters to be updated during training."""
+        for param in self.bert.parameters():
+            param.requires_grad = True
+        print("BERT layers unfrozen.")

result.txt

@@ -0,0 +1 @@
+می‌دانید که ادبیات فارسی گنجینه‌ای از آثار پرمغز و زیباست. خواندن متون فاخر فارسی نه‌تنها موجب افزایش دانش بلکه باعث غنی‌تر شدن روح انسان می‌شود. بسیاری از اشعار و نثرهای کهن، مانند آثار حافظ سعدی، و فردوسی، باید با دقت خوانده شوند تا معنای کامل آنها را دریافت کنیم. درک معنای ژرف چنین آثاری بدون داشتن دانش کافی از زبان و فرهنگ فارسی امکان‌پذیر نیست. بنابراین، آموزش درست به کودکان و نوجوانان نقش حیاتی در زنده نگه داشتن این میراث فرهنگی دارد. فضای مجازی امروز نقش عمده‌ای در انتشار آثار فارسی دارد، اما باید به شیوه‌ای صحیح از آن بهره ببریم تا از تحریف یا اشتباه در انتقال متون جلوگیری شود.

run.py

@@ -0,0 +1,214 @@
+import os
+import json
+import torch
+import argparse
+from transformers import AutoTokenizer
+from modeling_custom import BertForTokenClassificationWithFourO
+from labeler import Labeler
+
+
+class ModelPipeline:
+    """
+    Pipeline for text processing using the BertForTokenClassificationWithFourO model.
+    Handles preprocessing, inference, and postprocessing.
+    """
+
+    def __init__(self, model_path=None):
+        """
+        Initialize the pipeline with model and tokenizer.
+
+        Args:
+            model_path: Path to the model directory. Defaults to current directory.
+        """
+        # Use current directory if no path specified
+        if model_path is None:
+            model_path = os.path.dirname(os.path.abspath(__file__))
+
+        # Load tokenizer and model
+        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+        print(f"Using device: {self.device}")
+
+        self.tokenizer = AutoTokenizer.from_pretrained(model_path)
+        self.model = BertForTokenClassificationWithFourO.from_pretrained(model_path)
+        self.model.to(self.device)
+        self.model.eval()
+
+        # Load config for any custom settings
+        config_path = os.path.join(model_path, "config.json")
+        with open(config_path, 'r') as f:
+            self.config = json.load(f)
+
+        # Initialize labeler for postprocessing
+        self.labeler = Labeler(tags=(1, 2),
+                               regexes=(r'[^\S\r\n\v\f]', r'\u200c'),
+                               chars=(" ", "‌"),
+                               class_count=2)
+
+    def _tokenize(self, chars, chunk_size=512):
+        input_ids = []
+        attention_masks = []
+
+        ids_ = [101] + [self.tokenizer.encode(char)[1] for char in chars] + [102]
+        for i in range(0, len(ids_), chunk_size):
+            chunked_ids = ids_[i:i + chunk_size]
+            attention_mask = [1] * len(chunked_ids)
+
+            if len(chunked_ids) != chunk_size:
+                attention_mask += [0] * (chunk_size - len(chunked_ids))  # padding the attention mask accordingly
+                chunked_ids += [0] * (chunk_size - len(chunked_ids))  # padding the last chunk to chunk size
+
+            input_ids.append(chunked_ids)
+            attention_masks.append(attention_mask)
+
+        return input_ids, attention_masks
+
+    def preprocess(self, text):
+        chars, labels = self.labeler.label_text(text, corpus_type=Labeler.WHOLE_RAW)
+        input_ids, attention_mask = self._tokenize(chars[0], chunk_size=512)
+        input_ids = torch.tensor(input_ids)
+        attention_mask = torch.tensor(attention_mask)
+        labels = [0] + labels[0] + [0]
+        return {"input_ids": input_ids, "attention_mask": attention_mask}, chars, labels
+
+    def predict(self, encoded_inputs):
+        """
+        Run model inference on preprocessed inputs.
+
+        Args:
+            encoded_inputs: Tokenized inputs from preprocess method
+
+        Returns:
+            Model predictions
+        """
+        # Move input tensors to the same device as the model
+        input_ids = encoded_inputs["input_ids"].to(self.device)
+        attention_mask = encoded_inputs["attention_mask"].to(self.device)
+
+        with torch.no_grad():
+            outputs = self.model(
+                input_ids=input_ids,
+                attention_mask=attention_mask
+            )
+
+        # Get predicted class for each token
+        predictions = torch.argmax(outputs.logits, dim=-1)
+        return predictions, outputs.logits
+
+    def postprocess_spacing(self, predictions, chars):
+        # Move predictions back to CPU if needed for postprocessing
+        predictions_cpu = predictions.cpu()
+        predictions_flat = [label for sample in predictions_cpu.tolist() for label in sample]
+        return self.labeler.text_generator([' '] + chars[0] + [' '],
+                                           predictions_flat[:len(chars[0]) + 2],
+                                           corpus_type=Labeler.WHOLE_RAW).strip()
+
+    def postprocess_correcting(self, logits, chars, labels, alpha):
+        # Process the labels to match the sequence length
+
+        sequence_length = logits.size(1)
+        if len(labels) < sequence_length:
+            labels.extend([0] * (sequence_length - len(labels)))  # Padding with 0
+
+        else:
+            labels = labels[:sequence_length]  # Truncate if longer
+
+        # Convert labels to one-hot encoding
+        num_classes = logits.size(-1)
+        user_labels_tensor = torch.tensor([labels], device=self.device)
+        user_labels_one_hot = torch.nn.functional.one_hot(user_labels_tensor, num_classes=num_classes).float()
+
+        # Expand dimensions to match logits shape if needed
+        if logits.dim() > user_labels_one_hot.dim():
+            user_labels_one_hot = user_labels_one_hot.unsqueeze(0)
+
+        # Combine logits and user labels
+        combined_logits = logits * (1 - alpha) + user_labels_one_hot * alpha
+
+        # Apply softmax to get probabilities
+        combined_probs = torch.nn.functional.softmax(combined_logits, dim=-1)
+
+        # Get the final predictions
+        final_predictions = torch.argmax(combined_probs, dim=-1)
+
+        # Move to CPU for postprocessing
+        final_predictions_cpu = final_predictions.cpu()
+
+        # Flatten predictions
+        predictions_flat = [label for sample in final_predictions_cpu.tolist() for label in sample]
+
+        # Generate text with combined predictions
+        return self.labeler.text_generator([' '] + chars[0] + [' '],
+                                           predictions_flat[:len(chars[0]) + 2],
+                                           corpus_type=Labeler.WHOLE_RAW).strip()
+
+    def process_text(self, text, mode):
+        """
+        Process text through the entire pipeline.
+
+        Args:
+            text: Input text as string or list of strings
+            mode: Processing mode ('space' or 'correct')
+
+        Returns:
+            Processed text with labels applied
+        """
+        # Run the full pipeline
+        encoded_inputs, chars, labels = self.preprocess(text)
+
+        predictions, logits = self.predict(encoded_inputs)
+
+        if mode == 'space':
+            result = self.postprocess_spacing(predictions, chars)
+        elif mode == 'correct':
+            result = self.postprocess_correcting(logits, chars, labels, 0.5)
+        else:
+            raise ValueError(f"Unrecognized mode: {mode}")
+        return result
+
+
+def main():
+    """Command line interface for the model pipeline."""
+    parser = argparse.ArgumentParser(description="Process text using token classification model")
+    parser.add_argument("--text", type=str, help="Text to process")
+    parser.add_argument("--file", type=str, help="Path to file containing text to process")
+    parser.add_argument("--output", type=str, help="Path to output file")
+    parser.add_argument("--mode", type=str, choices=['space', 'correct'], default='space',
+                        help="Processing mode: 'space' uses model outputs only, 'correct' combines model results with "
+                             "original text spacing")
+    parser.add_argument("--model_path", type=str, default=None, help="Path to model directory")
+    args = parser.parse_args()
+
+    # Initialize pipeline
+    pipeline = ModelPipeline(args.model_path)
+
+    # Get input text
+    if args.text:
+        input_text = args.text
+    elif args.file:
+        with open(args.file, 'r', encoding='utf-8') as f:
+            input_text = f.read()
+    else:
+        print("Please provide either --text or --file")
+        return
+
+    # Process the text
+    result = pipeline.process_text(input_text, args.mode)
+
+    # Output results
+    if args.output:
+        with open(args.output, 'w', encoding='utf-8') as f:
+            if isinstance(result, list):
+                f.write('\n'.join(result))
+            else:
+                f.write(result)
+    else:
+        print("\nProcessed Text:")
+        if isinstance(result, list):
+            for item in result:
+                print(item)
+        else:
+            print(result)
+
+
+if __name__ == "__main__":
+    main()

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer_config.json

@@ -0,0 +1,55 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_lower_case": true,
+  "mask_token": "[MASK]",
+  "model_max_length": 512,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c82f1de4729fbd371473ac1e44782c7a63067513492c68ebacd386520759be1d
+size 5176