Upload folder using huggingface_hub

README.md

@@ -0,0 +1,168 @@
+---
+license: apache-2.0
+base_model: answerdotai/ModernBERT-base
+pipeline_tag: token-classification
+tags:
+- text-segmentation
+- semantic-chunking
+- rag
+- chunking
+datasets:
+- saeedabc/wiki727k
+language:
+- en
+---
+
+# Semantic chunker (ModernBERT-base)
+
+Predicts, for each sentence, whether a topic boundary follows it. Intended as a
+drop-in chunker for RAG pipelines.
+
+## What is different here
+
+- **2048-token training context.** The closest existing model trains at 1024.
+  Median Wikipedia article is ~980 tokens, so 2048 keeps ~82% of documents in a
+  single window instead of ~53%.
+- **Supervision at sentence-final tokens only.** Labelling every token puts the
+  positive rate near 1% and the model collapses toward never predicting a split.
+  Scoring only sentence-final positions raises it to ~12%.
+- **Trained on Wikipedia section structure**, not book paragraphing.
+
+## Results
+
+Both models are ModernBERT-base token taggers sharing a tokenizer, so they are
+scored at identical positions: P(boundary) read at each sentence's final token.
+Threshold swept 0.05–0.95; the best operating point is reported.
+
+### wiki727k test — in-distribution for this model
+
+| Model | F1 | Precision | Recall | Threshold |
+|---|---|---|---|---|
+| **this model** | 0.8156 | 0.8390 | 0.7934 | 0.70 |
+| `mirth/chonky_modernbert_base_1` | 0.4986 | 0.4543 | 0.5524 | 0.90 |
+| baseline:every-k (best k, oracle-tuned) (k=2) | 0.1984 | 0.1228 | 0.5159 | — |
+| baseline:base-rate random | 0.1177 | 0.1166 | 0.1187 | — |
+
+### PubMed-RCT — out-of-distribution for both models
+
+Non-Wikipedia scientific prose with human-assigned section labels. Boundary rate
+is 35.2% here versus ~11.7% on Wikipedia, so
+scores are **not** comparable across the two tables — only across models within
+one table.
+
+| Model | F1 | Precision | Recall | Threshold |
+|---|---|---|---|---|
+| **this model** | 0.6932 | 0.7293 | 0.6605 | 0.55 |
+| `mirth/chonky_modernbert_base_1` | 0.5024 | 0.5095 | 0.4954 | 0.85 |
+| baseline:every-k (best k, oracle-tuned) (k=2) | 0.4034 | 0.3500 | 0.4759 | — |
+| baseline:base-rate random | 0.3490 | 0.3521 | 0.3460 | — |
+
+### Retrieval — does better boundary detection actually help?
+
+Macro-average over 3 corpora (gov_report, qmsum, stackoverflow), embedder `BAAI/bge-small-en-v1.5`, identical 512-token cap on every strategy so chunk size cannot confound the comparison.
+
+| Strategy | nDCG@10 | R@1 | R@5 | R@10 | mean chunk tokens |
+|---|---|---|---|---|---|
+| `fixed-512-ovl64` | 0.8625 | 0.8254 | 0.9218 | 0.9500 | 388 |
+| `chonky-min256` | 0.8615 | 0.7995 | 0.9353 | 0.9635 | 274 |
+| **ours-min256** (this model) | 0.8583 | 0.7914 | 0.9328 | 0.9659 | 283 |
+| **ours** (this model) | 0.8575 | 0.7854 | 0.9321 | 0.9685 | 215 |
+| `sentence-8` | 0.8536 | 0.7956 | 0.9267 | 0.9536 | 182 |
+| `fixed-512` | 0.8531 | 0.8019 | 0.9279 | 0.9512 | 385 |
+| `recursive-512` | 0.8528 | 0.7903 | 0.9218 | 0.9598 | 308 |
+| `chonky` | 0.8498 | 0.7702 | 0.9321 | 0.9684 | 146 |
+
+Per corpus (nDCG@10):
+
+| Strategy | gov_report | qmsum | stackoverflow |
+|---|---|---|---|
+| `fixed-512-ovl64` | **0.9423** | 0.6988 | 0.9464 |
+| `chonky-min256` | 0.9285 | 0.7099 | 0.9461 |
+| **ours-min256** | 0.9225 | 0.7056 | 0.9467 |
+| **ours** | 0.9191 | **0.7254** | 0.9280 |
+| `sentence-8` | 0.9272 | 0.6865 | **0.9472** |
+| `fixed-512` | 0.9348 | 0.6782 | 0.9463 |
+| `recursive-512` | 0.9214 | 0.6910 | 0.9459 |
+| `chonky` | 0.9046 | 0.7118 | 0.9329 |
+
+### Reading these numbers honestly
+
+`baseline:every-k` splits every k sentences with k chosen to maximise its own
+score — a model-free floor with oracle tuning in its favour. The wiki727k
+comparison flatters this model: Wikipedia section boundaries are its training
+distribution and are out-of-distribution for chonky, which trained on BookCorpus
+paragraphs. **The PubMed table is the fair comparison.**
+
+**Better boundaries did not produce a uniform retrieval win, and the retrieval
+table above should be read before adopting this.** That result is consistent
+with the published critique of semantic chunking, and it is reported here rather
+than omitted. Boundary F1 and retrieval quality are different things; this model
+is much better at the first and situationally better at the second.
+
+## When this helps, and when it does not
+
+**Use it** for long documents whose topic genuinely shifts partway through —
+transcripts, reports, articles, manuals. That is where fixed-size splitting cuts
+through the middle of an idea and where boundary detection pays.
+
+**Do not use it** for corpora whose documents are already shorter than your
+embedder's window. Splitting a 150-token document into two 75-token fragments
+makes retrieval worse, not better, no matter how correct the boundary is. On
+such corpora fixed-size chunking is competitive, faster, and has no
+dependencies — use that instead.
+
+**`min_chunk_tokens` is a corpus-dependent knob, not a default.** Setting it to
+256 recovered most of the short-document deficit (+0.019 nDCG@10 on
+stackoverflow) but cost almost exactly as much on long transcripts (−0.020 on
+qmsum), by merging away the boundaries that made the model useful there. Set it
+high for short documents, leave it at 0 for long ones, and measure on your own
+corpus rather than trusting either default.
+
+**Cost.** Chunking with this model is roughly one to two orders of magnitude
+slower than a fixed-size splitter (per-strategy wall time is recorded in
+`retrieval_eval.json`). On corpora where it does not win, that cost buys
+nothing.
+
+## Usage
+
+`chunk.py` ships in this repo — grab it alongside the weights:
+
+```bash
+hf download 0xKitkat/semantic-chunker-modernbert-base chunk.py --local-dir .
+```
+
+```python
+from chunk import SemanticChunker
+
+chunker = SemanticChunker("0xKitkat/semantic-chunker-modernbert-base", threshold=0.5, max_chunk_tokens=512)
+for c in chunker.split(document_text):
+    print(c.n_tokens, c.text[:80])
+```
+
+`max_chunk_tokens` force-splits over-long chunks at the model's lowest-confidence
+interior boundary rather than at an arbitrary offset.
+
+## Training
+
+| | |
+|---|---|
+| Base | `answerdotai/ModernBERT-base` (149M) |
+| Data | wiki727k, 582,160 train docs / 30,580,099 sentences |
+| Boundary rate | 11.7% |
+| Max length | 2048 |
+| Effective batch | 32 |
+| LR / schedule | 3e-5, linear, 6% warmup |
+| Epochs | 1 |
+| Loss | class-weighted CE (positive weight 3.0) |
+| Hardware | 1x RTX 4070 SUPER (12GB) |
+
+Section titles are dropped from the training text: with headings present the
+task partly degenerates into "a heading follows", which does not transfer to the
+unformatted prose a chunker sees in production.
+
+## Limitations
+
+- English only.
+- Trained on encyclopedic prose; conversational transcripts, code, and tabular
+  documents are out of distribution.
+- Boundary F1 is a proxy. If your goal is retrieval quality, measure retrieval.

chunk.py

@@ -0,0 +1,360 @@
+"""Inference-side semantic chunker.
+
+The training task is "does a segment boundary follow this sentence"; this module
+turns that into the thing a RAG pipeline actually wants: a list of text chunks,
+subject to hard size limits an embedder can accept.
+
+Two things here are not in the training loop and matter in production:
+
+1. Windowed inference. Real documents run far past 2048 tokens. We slide a
+   window with sentence-level left context so boundary decisions near a window
+   edge still see what came before, instead of being scored cold.
+2. Hard size caps. A semantic boundary model will happily emit a 4000-token
+   chunk if the topic does not shift. Embedders truncate at their own limit, so
+   `max_chunk_tokens` force-splits at the lowest-confidence interior sentence
+   rather than at an arbitrary character offset.
+"""
+
+from __future__ import annotations
+
+import os
+import re
+from dataclasses import dataclass
+from typing import Callable, Iterable
+
+import numpy as np
+from transformers import AutoTokenizer
+
+# Sentence splitter: deliberately dependency-free. Handles the common
+# abbreviation cases that naive `.` splitting gets wrong. Pass your own via
+# `sentence_splitter=` if you already have spacy/nltk/pysbd in the pipeline.
+_ABBREV = (
+    r"(?<!\bMr)(?<!\bMrs)(?<!\bMs)(?<!\bDr)(?<!\bProf)(?<!\bSt)(?<!\bJr)(?<!\bSr)"
+    r"(?<!\bInc)(?<!\bLtd)(?<!\bCo)(?<!\bvs)(?<!\betc)(?<!\bi\.e)(?<!\be\.g)"
+    r"(?<!\bFig)(?<!\bNo)(?<!\bVol)(?<!\bApprox)(?<!\b[A-Z])"
+)
+_SENT_RE = re.compile(rf"{_ABBREV}(?<=[.!?])[\"')\]]*\s+(?=[A-Z0-9\"'(\[])")
+
+
+def default_sentence_splitter(text: str) -> list[str]:
+    out = []
+    for block in re.split(r"\n\s*\n", text):  # paragraph breaks are always splits
+        block = block.strip()
+        if not block:
+            continue
+        out.extend(s.strip() for s in _SENT_RE.split(block) if s.strip())
+    return out
+
+
+def _softmax_last(x: np.ndarray) -> np.ndarray:
+    x = x - x.max(axis=-1, keepdims=True)
+    e = np.exp(x)
+    return e / e.sum(axis=-1, keepdims=True)
+
+
+class TorchBackend:
+    """Default backend. torch is imported here so the ONNX path stays torch-free."""
+
+    def __init__(self, model_path: str, device: str | None = None):
+        import torch
+        from transformers import AutoModelForTokenClassification
+
+        self._torch = torch
+        self.device = device or ("cuda" if torch.cuda.is_available() else "cpu")
+        self.model = (
+            AutoModelForTokenClassification.from_pretrained(model_path)
+            .eval()
+            .to(self.device)
+        )
+        self.id2label = dict(self.model.config.id2label)
+
+    def __call__(self, input_ids: np.ndarray, attention_mask: np.ndarray) -> np.ndarray:
+        t = self._torch
+        with t.no_grad():
+            out = self.model(
+                input_ids=t.tensor(input_ids, dtype=t.long, device=self.device),
+                attention_mask=t.tensor(attention_mask, dtype=t.long, device=self.device),
+            ).logits
+        return out.float().cpu().numpy()
+
+
+class OnnxBackend:
+    """CPU inference with onnxruntime only -- no torch in the dependency tree."""
+
+    def __init__(self, model_dir: str, providers: list[str] | None = None):
+        import json
+        import onnxruntime as ort
+
+        path = os.path.join(model_dir, "model.onnx")
+        if not os.path.exists(path):
+            raise FileNotFoundError(f"no model.onnx under {model_dir}")
+        self.sess = ort.InferenceSession(
+            path, providers=providers or ["CPUExecutionProvider"]
+        )
+        with open(os.path.join(model_dir, "config.json"), encoding="utf-8") as f:
+            cfg = json.load(f)
+        self.id2label = cfg.get("id2label", {0: "O", 1: "semantic-shift"})
+
+    def __call__(self, input_ids: np.ndarray, attention_mask: np.ndarray) -> np.ndarray:
+        return self.sess.run(
+            None,
+            {
+                "input_ids": input_ids.astype(np.int64),
+                "attention_mask": attention_mask.astype(np.int64),
+            },
+        )[0]
+
+
+@dataclass
+class Chunk:
+    text: str
+    start_sentence: int
+    end_sentence: int
+    n_tokens: int
+    boundary_score: float  # confidence of the boundary that closed this chunk
+
+
+class SemanticChunker:
+    def __init__(
+        self,
+        model_path: str,
+        threshold: float = 0.5,
+        max_chunk_tokens: int = 512,
+        min_chunk_tokens: int = 0,
+        min_chunk_sentences: int = 1,
+        window_tokens: int = 2048,
+        context_sentences: int = 3,
+        device: str | None = None,
+        sentence_splitter: Callable[[str], list[str]] | None = None,
+        batch_size: int = 8,
+        backend: str = "torch",
+        tokenizer_path: str | None = None,
+    ):
+        self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_path or model_path)
+        if backend == "onnx":
+            self.backend = OnnxBackend(model_path)
+        elif backend == "torch":
+            self.backend = TorchBackend(model_path, device)
+        else:
+            raise ValueError(f"unknown backend {backend!r}, expected 'torch' or 'onnx'")
+        self.device = getattr(self.backend, "device", "cpu")
+        self.threshold = threshold
+        self.max_chunk_tokens = max_chunk_tokens
+        self.min_chunk_tokens = min_chunk_tokens
+        self.min_chunk_sentences = max(1, min_chunk_sentences)
+        self.window_tokens = window_tokens
+        self.context_sentences = context_sentences
+        self.split_sentences = sentence_splitter or default_sentence_splitter
+        self.batch_size = batch_size
+
+        id2label = dict(self.backend.id2label)
+        self.pos_idx = next(
+            (int(i) for i, n in id2label.items()
+             if str(n).lower() in {"semantic-shift", "separator", "shift"}),
+            1,
+        )
+
+    # ------------------------------------------------------------------
+    def boundary_scores(self, sentences: list[str]) -> list[float]:
+        """P(boundary follows) for each sentence. Last entry is always 1.0."""
+        if not sentences:
+            return []
+        if len(sentences) == 1:
+            return [1.0]
+
+        toks = self.tokenizer(
+            [s if i == 0 else " " + s for i, s in enumerate(sentences)],
+            add_special_tokens=False,
+        )["input_ids"]
+
+        budget = self.window_tokens - 2
+        windows = []  # (ids, [(sentence_idx, token_pos)])
+        start = 0
+        n = len(sentences)
+        while start < n:
+            ids, marks = [], []
+            ctx_start = max(0, start - self.context_sentences)
+            for i in range(ctx_start, start):
+                ids.extend(toks[i])
+            i = start
+            while i < n:
+                t = toks[i] or [self.tokenizer.unk_token_id]
+                if len(t) > budget:
+                    t = t[:budget]
+                if len(ids) + len(t) > budget:
+                    break
+                ids.extend(t)
+                marks.append((i, len(ids) - 1))
+                i += 1
+            if i == start:  # pathological single sentence
+                marks.append((start, max(len(ids) - 1, 0)))
+                i = start + 1
+            windows.append((ids, marks))
+            start = i
+
+        scores = [0.0] * n
+        for b in range(0, len(windows), self.batch_size):
+            batch = windows[b : b + self.batch_size]
+            maxlen = max(len(w[0]) for w in batch) + 2
+            input_ids = np.full(
+                (len(batch), maxlen), self.tokenizer.pad_token_id, dtype=np.int64
+            )
+            attn = np.zeros((len(batch), maxlen), dtype=np.int64)
+            for r, (ids, _) in enumerate(batch):
+                seq = [self.tokenizer.cls_token_id] + ids + [self.tokenizer.sep_token_id]
+                input_ids[r, : len(seq)] = seq
+                attn[r, : len(seq)] = 1
+            logits = self.backend(input_ids, attn)
+            probs = _softmax_last(logits.astype(np.float32))[..., self.pos_idx]
+            for r, (_, marks) in enumerate(batch):
+                for sent_idx, tok_pos in marks:
+                    scores[sent_idx] = float(probs[r, tok_pos + 1])  # +1 for CLS
+
+        scores[-1] = 1.0  # document end is always a boundary
+        return scores
+
+    # ------------------------------------------------------------------
+    def _token_lens(self, texts: list[str]) -> list[int]:
+        """One batched tokenizer call, not one per sentence."""
+        if not texts:
+            return []
+        enc = self.tokenizer(texts, add_special_tokens=False)["input_ids"]
+        return [len(x) for x in enc]
+
+    def split(self, text: str) -> list[Chunk]:
+        sentences = self.split_sentences(text)
+        if not sentences:
+            return []
+        scores = self.boundary_scores(sentences)
+        sent_tokens = self._token_lens(sentences)
+
+        chunks: list[Chunk] = []
+        buf_start = 0
+        buf_tokens = 0
+
+        def emit(end_idx: int, score: float):
+            nonlocal buf_start, buf_tokens
+            body = " ".join(sentences[buf_start : end_idx + 1]).strip()
+            if body:
+                chunks.append(
+                    Chunk(
+                        text=body,
+                        start_sentence=buf_start,
+                        end_sentence=end_idx,
+                        n_tokens=sum(sent_tokens[buf_start : end_idx + 1]),
+                        boundary_score=score,
+                    )
+                )
+            buf_start = end_idx + 1
+            buf_tokens = 0
+
+        for i, sent in enumerate(sentences):
+            buf_tokens += sent_tokens[i]
+            n_in_buf = i - buf_start + 1
+
+            # hard cap wins over semantics: split at the weakest interior
+            # boundary so we cut where the model is least confident, not
+            # wherever the token counter happened to run out.
+            if buf_tokens > self.max_chunk_tokens and n_in_buf > 1:
+                interior = range(buf_start, i)
+                cut = max(interior, key=lambda j: scores[j])
+                emit(cut, scores[cut])
+                buf_tokens = sum(sent_tokens[buf_start : i + 1])
+                n_in_buf = i - buf_start + 1
+
+            if scores[i] >= self.threshold and n_in_buf >= self.min_chunk_sentences:
+                emit(i, scores[i])
+
+        if buf_start < len(sentences):
+            emit(len(sentences) - 1, 1.0)
+
+        return self._merge_undersized(chunks, sentences, sent_tokens)
+
+    def _merge_undersized(self, chunks, sentences, sent_tokens):
+        """Merge chunks below min_chunk_tokens into their neighbour.
+
+        Measured need, not speculation: on short-document corpora the boundary
+        model happily splits a 150-token document in two, and two 75-token
+        fragments retrieve worse than one whole document. A topic boundary is
+        real there, but acting on it is counterproductive -- the embedder needs
+        enough text to place the vector well.
+
+        Merges forward by preference (keeps reading order), backward for a
+        trailing runt, and never produces a chunk exceeding max_chunk_tokens.
+        """
+        if self.min_chunk_tokens <= 0 or len(chunks) < 2:
+            return chunks
+
+        out = []
+        i = 0
+        while i < len(chunks):
+            cur = chunks[i]
+            while (
+                cur.n_tokens < self.min_chunk_tokens
+                and i + 1 < len(chunks)
+                and cur.n_tokens + chunks[i + 1].n_tokens <= self.max_chunk_tokens
+            ):
+                nxt = chunks[i + 1]
+                cur = Chunk(
+                    text=(cur.text + " " + nxt.text).strip(),
+                    start_sentence=cur.start_sentence,
+                    end_sentence=nxt.end_sentence,
+                    n_tokens=cur.n_tokens + nxt.n_tokens,
+                    boundary_score=nxt.boundary_score,
+                )
+                i += 1
+            out.append(cur)
+            i += 1
+
+        # a trailing runt has no forward neighbour left; fold it back
+        if (
+            len(out) > 1
+            and out[-1].n_tokens < self.min_chunk_tokens
+            and out[-2].n_tokens + out[-1].n_tokens <= self.max_chunk_tokens
+        ):
+            last, prev = out.pop(), out.pop()
+            out.append(
+                Chunk(
+                    text=(prev.text + " " + last.text).strip(),
+                    start_sentence=prev.start_sentence,
+                    end_sentence=last.end_sentence,
+                    n_tokens=prev.n_tokens + last.n_tokens,
+                    boundary_score=last.boundary_score,
+                )
+            )
+        return out
+
+    def split_text(self, text: str) -> list[str]:
+        """Convenience: just the strings."""
+        return [c.text for c in self.split(text)]
+
+    def batch_split(self, texts: Iterable[str]) -> list[list[str]]:
+        return [self.split_text(t) for t in texts]
+
+
+if __name__ == "__main__":
+    import argparse
+    import sys
+
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--model", default=r"D:\LocalAI\chunker\out\run1\final")
+    ap.add_argument("--threshold", type=float, default=0.5)
+    ap.add_argument("--max-chunk-tokens", type=int, default=512)
+    ap.add_argument("--file", help="read text from file; otherwise stdin")
+    ap.add_argument("--backend", default="torch", choices=["torch", "onnx"])
+    ap.add_argument("--tokenizer", default=None, help="only needed for a bare onnx dir")
+    args = ap.parse_args()
+
+    text = (
+        open(args.file, encoding="utf-8").read() if args.file else sys.stdin.read()
+    )
+    ch = SemanticChunker(
+        args.model,
+        threshold=args.threshold,
+        max_chunk_tokens=args.max_chunk_tokens,
+        backend=args.backend,
+        tokenizer_path=args.tokenizer,
+    )
+    for i, c in enumerate(ch.split(text)):
+        print(f"\n--- chunk {i}  ({c.n_tokens} tok, score {c.boundary_score:.3f}) ---")
+        print(c.text)

config.json

@@ -0,0 +1,86 @@
+{
+  "architectures": [
+    "ModernBertForTokenClassification"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 50281,
+  "classifier_activation": "gelu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "mean",
+  "cls_token_id": 50281,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "float32",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 50282,
+  "global_attn_every_n_layers": 3,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 768,
+  "id2label": {
+    "0": "semantic-continuity",
+    "1": "semantic-shift"
+  },
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 1152,
+  "label2id": {
+    "semantic-continuity": 0,
+    "semantic-shift": 1
+  },
+  "layer_norm_eps": 1e-05,
+  "layer_types": [
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "local_attention": 128,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 12,
+  "num_hidden_layers": 22,
+  "pad_token_id": 50283,
+  "position_embedding_type": "absolute",
+  "rope_parameters": {
+    "full_attention": {
+      "rope_theta": 160000.0,
+      "rope_type": "default"
+    },
+    "sliding_attention": {
+      "rope_theta": 10000.0,
+      "rope_type": "default"
+    }
+  },
+  "sep_token_id": 50282,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.14.1",
+  "use_cache": false,
+  "vocab_size": 50368
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:18ea0a1f198166cc336403c2e6ef0dc99bfe2cd19e31a5579dd1a33fb0b460af
+size 598439784

onnx/model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5772dddc662de3944429e0f6b05037fb17b6a20706f8d160e2f70af2826e8662
+size 599052097

tokenizer_config.json

@@ -0,0 +1,17 @@
+{
+  "backend": "tokenizers",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "is_local": true,
+  "local_files_only": false,
+  "mask_token": "[MASK]",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 8192,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "[UNK]"
+}