Add TwIL-LM2 weights (smollm2-1.7b-formal-logic-mgpo)

README.md

@@ -1,7 +1,7 @@
 ---
 language:
   - en
-library_name: peft
+library_name: transformers
 pipeline_tag: text-generation
 base_model: HuggingFaceTB/SmolLM2-1.7B-Instruct
 license: other
@@ -11,329 +11,178 @@ tags:
   - formal-logic
   - reasoning
   - lora
-  - peft
-  - gguf
-  - llama.cpp
+  - model-merging
+  - wise-ft
+  - reinforcement-learning
+  - grpo
   - smollm2
   - twil-lm
 ---
 
-# TwiL-LM(1.7B)
+# TwIL-LM2
 
-TwiL-LM(1.7B) is a parameter-efficient LoRA adapter for
-[SmolLM2-1.7B-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct).
-It is designed for formal-logic tasks including first-order logic (FOL)
-translation, entailment classification, multiple-choice reasoning, semantic
-parsing, Lean assistance, and procedural reasoning.
+A 1.7B reasoning model for **formal logic** tasks, built from
+[`HuggingFaceTB/SmolLM2-1.7B-Instruct`](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct)
+through LoRA supervised fine-tuning, checkpoint fusion, WiSE-FT weight interpolation, and
+entropy-weighted GRPO reinforcement learning.
 
-## Model details
+It raises in-domain formal-logic performance by **+49% relative** over its base model
+(macro gate 0.263 → 0.393) while holding held-out general capability roughly flat.
 
-- **Developed by:** webAI Intelligence Lab
-- **Model repository:** [webAI-Official/TwIL-LM](https://huggingface.co/webAI-Official/TwIL-LM)
-- **Model type:** Decoder-only Transformer with a PEFT LoRA adapter
-- **Base model:** SmolLM2-1.7B-Instruct
-- **Language:** English
-- **License:** webAI Non-Commercial License ver. 1.0
-- **LoRA configuration:** rank 64, alpha 128, dropout 0.0, no bias
-- **Context window:** 8,192 tokens, inherited from the base model
-- **Runtime parameters:** 1,783,728,128 total, consisting of approximately
-  1.711B backbone parameters and 72.35M LoRA parameters
-- **Release formats:** PEFT adapter and optional GGUF artifacts
+Its larger sibling, **TwIL-LM3** (3B, from SmolLM3), trades a smaller in-domain gain for
+strictly better held-out retention. If you care about not regressing on general benchmarks,
+prefer that one.
 
-This is an adapter, not an independently trained foundation model. Use it with
-the exact base checkpoint and tokenizer listed above.
+## Results
 
-## Model description
+### Track A — in-domain formal logic
 
-TwiL-LM adds a relatively small set of trainable weights to SmolLM2-1.7B-Instruct.
-The adapter was trained on a proprietary multi-task corpus engineered for formal
-logic, multi-step deduction, and mathematical reasoning. The corpus combines
-examples from 47 reasoning sources with filtered synthetic examples.
+The **macro gate** is the mean of five objective scores: entailment labelling, multiple-choice
+answering, procedural reasoning, Lean proof critique, and rule induction (scored by its
+continuous derivation score). MCQ and procedural are credited as `max(exact_match, loose_match)`.
+`n = 200` prompts per objective, greedy decoding, 2048 max new tokens.
 
-In the evaluation described below, TwiL-LM achieved a macro-primary score of
-**0.361**, compared with **0.185** for the unadapted SmolLM2-1.7B base. It led
-the sub-2B models included in this comparison. This result is specific to the
-reported formal-logic suite and should not be interpreted as a general measure
-of intelligence or performance on all reasoning tasks.
+| objective | SmolLM2-1.7B-Instruct | TwIL-LM2 | Δ |
+|---|---:|---:|---:|
+| entailment_label | 0.245 | **0.585** | +0.340 |
+| rule_induction | 0.135 | **0.514** | +0.379 |
+| lean_critic | 0.490 | **0.525** | +0.035 |
+| mcq_answer | 0.290 | 0.270 | −0.020 |
+| procedural | 0.155 | 0.070 | −0.085 |
+| **macro gate** | **0.2630** | **0.3927** | **+0.1297** |
 
-## Intended uses
+The gain is concentrated in entailment labelling and rule induction. MCQ answering and
+procedural reasoning **regressed**, and that is not hidden by the macro — it is averaged into
+the number above.
 
-### Direct use
+### Track B — held-out benchmarks
 
-- Translating English statements into first-order logic.
-- Classifying whether a conclusion follows from a set of premises.
-- Answering multiple-choice logic questions.
-- Parsing natural language into structured representations.
-- Drafting or critiquing Lean formalizations with external verification.
-- Research and experimentation on small formal-reasoning models.
+Nothing in this suite was trained on. Scores are re-derived from saved generations with
+delimiter-aware answer extractors rather than read from harness metrics.
 
-### Downstream use
+| | SmolLM2-1.7B-Instruct | TwIL-LM2 | Δ |
+|---|---:|---:|---:|
+| core average | 0.499 | **0.508** | +0.009 |
+| suite average (14 datasets) | 0.384 | 0.374 | −0.010 |
 
-The model can be integrated into systems that combine language-model generation
-with symbolic solvers, theorem provers, schema validators, or human review.
-Downstream developers should validate outputs for their target domain and retain
-the license and safety restrictions of both this adapter and its base model.
+Per-dataset, largest moves in each direction:
 
-### Out-of-scope uses
+| dataset | base | TwIL-LM2 | Δ |
+|---|---:|---:|---:|
+| GSM-Symbolic | 0.220 | 0.260 | +0.040 |
+| CommonsenseQA | 0.397 | 0.433 | +0.037 |
+| LogicBench BQA | 0.507 | 0.540 | +0.033 |
+| MATH-500 | 0.190 | 0.210 | +0.020 |
+| IFEval (strict) | 0.470 | 0.430 | −0.040 |
+| SVAMP | 0.487 | 0.383 | −0.103 |
+| MuSR | 0.422 | 0.313 | −0.109 |
 
-- Autonomous medical, legal, financial, or safety-critical decisions.
-- Treating generated FOL or Lean as verified without running an appropriate
-  checker or theorem prover.
-- Unsupervised deployment where logically incorrect or fabricated output can
-  cause material harm.
-- Uses prohibited by the webAI Non-Commercial License or the base-model license.
-- Impersonation, deceptive systems, autonomous weaponry, or surveillance that
-  targets protected classes.
+**This model does not pass a no-regression bar on held-out tasks.** MuSR and SVAMP lose about
+ten points each. The suite average is slightly negative. The honest summary is that in-domain
+logic improves substantially and general capability is approximately preserved *on average*,
+with real losses on multi-step narrative and word-problem reasoning.
 
-## Training details
-
-### Training data
-
-The training corpus is proprietary and is not currently published as a Hugging
-Face dataset. It was assembled from 47 reasoning sources and augmented with
-synthetic examples. Samples were normalized into a shared schema, filtered for
-structural quality, deduplicated, and checked using a two-stage LLM verification
-process. Synthetic data can still contain undetected errors, stylistic artifacts,
-or verifier preferences.
-
-Because the complete training corpus and all source-level mixture weights are not
-public, independent reproduction and contamination auditing are limited. Users
-should account for this limitation when comparing results or deploying the model.
-
-### Training procedure
-
-- **Method:** Supervised fine-tuning with PEFT LoRA
-- **Rank:** 64
-- **Alpha:** 128
-- **Dropout:** 0.0
-- **Bias:** none
-- **Target modules:** `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`,
-  `up_proj`, and `down_proj`
-- **Task type:** `CAUSAL_LM`
-
-Optimizer settings, learning-rate schedule, epoch count, random seed, training
-hardware, wall-clock time, and energy consumption are not documented in the
-current release artifacts. These should be added when verified records become
-available.
-
-
-
-## Evaluation
-
-### Formal-logic evaluation
-
-The in-domain evaluation used held-out samples with up to 200 rows per objective
-and greedy decoding. The reported macro-primary score is an equal-weight average
-over the eligible objectives. `mcq_answer` and `procedural` use
-`max(accuracy, loose_match)`. `lean_prove` and perplexity corpora are excluded
-from the macro score.
-
-
-The broader comparison includes checkpoints with different parameter counts,
-training procedures, and adaptation methods. It is useful for context but is not
-a controlled architecture or scaling study.
-
-TwiL-LM performs best on entailment labeling (**0.655**) and improves FOL
-translation (**0.235**, compared with approximately zero for the base). Lean
-formalization, Lean proving, and procedural reasoning remain weak.
-
-### Out-of-Distribution (OOD) evaluation
-
-OOD transfer was measured on GSM8K, ARC-Challenge chain-of-thought,
-ARC-Challenge 25-shot log-likelihood (`acc_norm`), and LogicBench BQA.
-
-Compared with SmolLM2-1.7B, TwiL-LM improves LogicBench BQA
-(**0.590 vs 0.563**) but is lower on GSM8K (**0.380 vs 0.413**), ARC-C
-chain-of-thought (**0.463 vs 0.587**), and ARC-C 25-shot log-likelihood
-(**0.460 vs 0.490**).
-
-### Throughput evaluation
-
-
-The formal-logic run generated 422,627 tokens in 1,152.34 generation seconds,
-or **366.8 aggregate tokens per second**. The configuration used a maximum of
-1,024 new tokens per example. Outputs averaged 264.1 new tokens; the median was
-92 tokens, the 90th percentile was 1,024 tokens, and 16.1% of outputs reached the
-generation limit.
-
-This is aggregate evaluation throughput, not single-request latency or a
-controlled serving benchmark. It depends on hardware, precision, backend,
-batching, prompt length, output length, and stopping behavior. The comparison
-runs did not consistently control all of these variables, so the figure should
-not be used to claim that LoRA intrinsically accelerates generation.
-
-### Evaluation limitations
-
-- The formal-logic suite includes custom tasks and metrics and is not a standard
-  Hugging Face benchmark dataset.
-- Some comparison checkpoints used different output limits and evaluation runs.
-- The evaluation primarily covers English.
-- The current records do not provide confidence intervals or repeated-seed
-  variance.
-- Exact-match metrics can penalize semantically equivalent formal expressions.
-- The training corpus is not public, limiting independent contamination checks.
-
-Structured Hub evaluation files are not included because the internal suite is
-not registered as a Hugging Face Benchmark and the available records do not
-contain verified task IDs for the OOD benchmarks. The scores above are therefore
-reported in the card rather than submitted as verified Hub leaderboard results.
-
-## How to use
-
-### Installation and authentication
-
-The model repository may require access approval. Install the dependencies and
-authenticate with the Hugging Face Hub:
-
-```bash
-pip install -U torch transformers peft accelerate huggingface_hub
-hf auth login
-```
-
-### Transformers with PEFT
+## Usage
 
 ```python
 import torch
-from peft import PeftModel
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
-base_id = "HuggingFaceTB/SmolLM2-1.7B-Instruct"
-adapter_id = "webAI-Official/TwIL-LM"
-
-tokenizer = AutoTokenizer.from_pretrained(base_id)
-base_model = AutoModelForCausalLM.from_pretrained(
-    base_id,
-    torch_dtype="auto",
-    device_map="auto",
+model_id = "webAI-Official/TwIL-LM"
+tok = AutoTokenizer.from_pretrained(model_id, revision="TwIL-LM2")
+model = AutoModelForCausalLM.from_pretrained(
+    model_id, revision="TwIL-LM2", torch_dtype=torch.bfloat16, device_map="auto"
 )
-model = PeftModel.from_pretrained(base_model, adapter_id)
-model.eval()
-
-messages = [
-    {
-        "role": "user",
-        "content": "Translate to first-order logic: All cats are mammals.",
-    }
-]
-input_ids = tokenizer.apply_chat_template(
-    messages,
-    tokenize=True,
-    add_generation_prompt=True,
-    return_tensors="pt",
-).to(model.device)
-
-with torch.inference_mode():
-    output_ids = model.generate(
-        input_ids,
-        max_new_tokens=256,
-        do_sample=False,
-    )
-
-new_tokens = output_ids[0, input_ids.shape[-1]:]
-print(tokenizer.decode(new_tokens, skip_special_tokens=True))
-```
-
-Use the base model's tokenizer and chat template. Loading the adapter over a
-different base checkpoint is unsupported and can cause incorrect output or tensor
-shape errors.
 
-### llama.cpp with GGUF
-
-Download the model repository:
-
-```bash
-hf download webAI-Official/TwIL-LM --local-dir TwIL-LM
-```
-
-If the release contains a GGUF LoRA adapter, apply it to a GGUF converted from
-the matching SmolLM2-1.7B-Instruct base:
-
-```bash
-llama-cli \
-  -m /path/to/smollm2-1.7b-instruct-f16.gguf \
-  --lora TwIL-LM/smollm2-lorav1-ps-lora-f16.gguf \
-  -p "Translate to FOL: All cats are mammals." \
-  -n 256
-```
-
-If the release contains a merged GGUF, use it without `--lora`:
-
-```bash
-llama-cli \
-  -m TwIL-LM/smollm2-lorav1-ps-merged-Q4_K_M.gguf \
-  -p "Translate to FOL: All cats are mammals." \
-  -n 256
-```
+messages = [{"role": "user", "content":
+             "Does 'All dogs are mammals. Rex is a dog.' entail 'Rex is a mammal'? "
+             "Answer entailment, contradiction, or neutral."}]
+inputs = tok.apply_chat_template(
+    messages, add_generation_prompt=True,
+    return_tensors="pt", return_dict=True,
+).to(model.device)
 
-GGUF filenames can vary between releases. Confirm the downloaded filenames
-before running these commands.
-
-## Files and formats
-
-- `adapter_model.safetensors`: PEFT LoRA weights, approximately 289 MB.
-- `adapter_config.json`: PEFT configuration that identifies the required base
-  model.
-- `smollm2-lorav1-ps-lora-f16.gguf`: optional GGUF LoRA adapter, approximately
-  145 MB.
-- Merged GGUF variants may be provided as FP16 (approximately 3.42 GB), Q8_0
-  (1.82 GB), Q5_K_M (1.23 GB), or Q4_K_M (1.06 GB).
-
-Availability and filenames can vary by release. Quantized GGUF variants may not
-match the evaluation quality reported for the original adapter; validate the
-selected artifact on the target workload.
-
-## Limitations, risks, and biases
-
-- **Not a verifier:** Plausible-looking FOL or Lean output can be syntactically or
-  semantically wrong. Use a symbolic solver, Lean/Mathlib, or expert review.
-- **Narrow specialization:** The adapter is designed for formal logic, not as a
-  replacement for a general-purpose assistant.
-- **Mixed OOD performance:** Improvements on LogicBench do not transfer
-  consistently to GSM8K or ARC-Challenge.
-- **Small-model capacity:** Long reasoning chains, deeply nested quantifiers,
-  complex rule induction, and long formal contexts can fail or hallucinate.
-- **Synthetic-data artifacts:** Synthetic examples and LLM verification can
-  introduce systematic style, content, or verifier bias.
-- **English-first:** Performance in other languages is not established.
-- **Base-model inheritance:** The adapter retains the base model's limitations,
-  biases, and potential for unsafe or inaccurate content.
-- **Context limit:** Inputs near 8,192 tokens leave less room for generation and
-  can be truncated by serving frameworks.
-- **Quantization effects:** Q4 and Q5 builds may change formal-token generation
-  and exact-match accuracy.
-- **No uncertainty calibration:** Scores do not establish that model confidence
-  corresponds to correctness.
-
-## Environmental impact
-
-Training-energy use, hardware type, training duration, datacenter region, and
-carbon emissions were not recorded in the available release artifacts. No
-emissions estimate is provided. Inference impact varies with hardware, precision,
-quantization, sequence length, and utilization.
-
-## License
-
-The model weights are distributed under the
-[webAI Non-Commercial License ver. 1.0](https://huggingface.co/webAI-Official/webAI-ColVec1-4b/blob/main/LICENSE.md).
-Review that license and the
-[SmolLM2 base-model terms](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct)
-before use. The license applied to source code in this repository does not
-override the model-weight license.
-
-## Citation
-
-```bibtex
-@misc{twil_lm_1_7b,
-  title  = {TwiL-LM(1.7B) Formal-Logic LoRA},
-  author = {webAI Intelligence Lab},
-  year   = {2026},
-  url    = {https://huggingface.co/webAI-Official/TwIL-LM}
-}
+out = model.generate(**inputs, max_new_tokens=2048, do_sample=False)
+print(tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
 ```
 
-## Contact
-
-For model questions, licensing requests, or reports of harmful behavior, use the
-Community tab of the
-[webAI-Official/TwIL-LM](https://huggingface.co/webAI-Official/TwIL-LM)
-repository.
+`return_dict=True` matters on transformers 5.x, where `apply_chat_template` returns a
+`BatchEncoding` rather than a bare tensor; the above works on both 4.x and 5.x.
+
+The model was trained and evaluated with **greedy decoding** (`do_sample=False`) and a
+**2048-token** generation budget. It usually opens a `<think>...</think>` reasoning block before
+answering, so give it room — the reported numbers assume at least 2048 new tokens, and a shorter
+budget truncates reasoning and scores far worse.
+
+## How it was built
+
+Four stages on top of the base model:
+
+1. **LoRA supervised fine-tuning** on a synthetic formal-logic corpus covering the Track A
+   objectives (first-order-logic translation, entailment labelling, semantic parsing, Lean
+   formalisation and critique, procedural reasoning, rule induction).
+2. **Checkpoint fusion** — parameter-space averaging of intermediate SFT checkpoints selected
+   by a diversity probe, rather than taking the final checkpoint.
+3. **WiSE-FT interpolation** toward the pretrained base, `W = (1 − λ)·W_base + λ·W_finetuned`
+   with **λ = 0.75**. λ was chosen by constrained optimisation: maximise in-domain score
+   subject to minimal degradation on held-out benchmarks.
+4. **MGPO** — entropy-weighted GRPO reinforcement learning against a programmatic verifier,
+   with partial credit for loose matches and token-F1 so that all-fail prompt groups still
+   produce gradient. Published checkpoint is **step 1680**.
+
+## Limitations and caveats
+
+**Truncation.** At a 2048-token budget, 6.9% of Track A generations hit the cap (down from
+11.7% for the base). Our protocol marks a comparison `rankable` only below 2% truncation, so
+both the base and this model are formally **not rankable** on Track A and the macro gate should
+be read as indicative rather than exact. A truncated response scores zero regardless of whether
+its reasoning was sound, so both numbers are pessimistic — the base more so.
+
+**Scope.** Tuned for formal logic. The Track B suite does not cover code generation or tool use
+(HumanEval, LiveCodeBench and BFCL were not run for this model or its base), so this release
+makes no claim about those.
+
+**Not a chat model.** It was optimised against automatic verifiers on logic tasks. It has had no
+safety tuning beyond whatever the base model carries, and no instruction-following alignment
+work — IFEval in fact regressed.
+
+**Failed consolidation stage.** A post-RL self-distillation round (SDFT) was attempted to recover
+held-out capability and made both tracks worse at every budget tried. It is not part of this
+model. See the accompanying `SDFT_RESULT.md` in the project repository.
+
+## Evaluation protocol
+
+- Track A: `n = 200` per objective, greedy (`temperature = 0`), `max_new_tokens = 2048`, one
+  retry at 4096 for truncated rows, `max_seq_len = 8192`, seed 42.
+- Track B: 300 examples per task, greedy, `max_gen_toks = 4096`, `max_model_len = 8192`,
+  `repetition_penalty = 1.0`, chat template applied, vLLM backend.
+- Both tracks use the same protocol for the model and its base, in a paired run over identical
+  sampled rows.
+
+`repetition_penalty = 1.0` is load-bearing. A 1.1 penalty produced apparent 20-point swings on
+Track B that were pure decoding artefact; the decoding kwargs are hashed into the protocol
+identity so a mismatched runner fails loudly instead of quietly producing a different number.
+
+## Relationship to the `main` branch
+
+The `main` branch of this repository holds the original TwIL-LM release: a PEFT **LoRA adapter**
+(plus GGUF builds) for the supervised fine-tuning stage only. This branch holds a **full merged
+model** from a later point in the pipeline — after fusion, WiSE-FT interpolation and MGPO
+reinforcement learning — so it is loaded directly with `AutoModelForCausalLM`, with no adapter
+and no base checkpoint required.
+
+The two are scored on different protocols and their headline numbers are not directly
+comparable: `main` reports a macro-*primary* average, while this card reports the five-component
+macro *gate* described above.
+
+## License and attribution
+
+Released under the **webAI Non-Commercial License ver. 1.0** — see `LICENSE.md` in this
+repository. This matches the licensing of the `main` branch.
+
+The base model,
+[`HuggingFaceTB/SmolLM2-1.7B-Instruct`](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct),
+is Apache 2.0; its licence text is retained as `apache-2.0-LICENSE.txt` and all credit for the
+base model goes to the HuggingFaceTB team. Apache 2.0 permits distributing derivative works
+under different terms provided attribution is preserved, which is what the pair of licence files
+in this repository does.

config.json

@@ -1,43 +1,45 @@
 {
-    "architectures": [
-        "LlamaForCausalLM"
-    ],
-    "attention_bias": false,
-    "attention_dropout": 0.0,
-    "bos_token_id": 1,
-    "torch_dtype": "bfloat16",
-    "eos_token_id": 2,
-    "head_dim": 64,
-    "hidden_act": "silu",
-    "hidden_size": 2048,
-    "initializer_range": 0.02,
-    "intermediate_size": 8192,
-    "max_position_embeddings": 8192,
-    "mlp_bias": false,
-    "model_type": "llama",
-    "num_attention_heads": 32,
-    "num_hidden_layers": 24,
-    "num_key_value_heads": 32,
-    "pad_token_id": 0,
-    "pretraining_tp": 1,
-    "rms_norm_eps": 1e-05,
-    "rope_parameters": {
-        "rope_theta": 130000,
-        "rope_type": "default"
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "dtype": "bfloat16",
+  "eos_token_id": 2,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "intermediate_size": 8192,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 24,
+  "num_key_value_heads": 32,
+  "pad_token_id": 2,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_parameters": {
+    "rope_theta": 130000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers.js_config": {
+    "dtype": "q4",
+    "kv_cache_dtype": {
+      "fp16": "float16",
+      "q4f16": "float16"
     },
-    "tie_word_embeddings": true,
-    "transformers.js_config": {
-        "torch_dtype": "q4",
-        "kv_cache_dtype": {
-            "fp16": "float16",
-            "q4f16": "float16"
-        },
-        "use_external_data_format": {
-            "model.onnx": true,
-            "model_fp16.onnx": true
-        }
-    },
-    "unsloth_version": "2026.7.4",
-    "use_cache": true,
-    "vocab_size": 49152
-}
+    "use_external_data_format": {
+      "model.onnx": true,
+      "model_fp16.onnx": true
+    }
+  },
+  "transformers_version": "5.5.0",
+  "use_cache": true,
+  "vocab_size": 49152,
+  "rope_theta": 130000,
+  "torch_dtype": "bfloat16"
+}

generation_config.json

@@ -1,10 +1,7 @@
 {
   "_from_model_config": true,
   "bos_token_id": 1,
-  "eos_token_id": [
-    2
-  ],
-  "max_length": 8192,
-  "pad_token_id": 0,
+  "eos_token_id": 2,
+  "pad_token_id": 2,
   "transformers_version": "5.5.0"
 }

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2f84bad0cbadc4b908fbfe6d350ff025df2d229a3e82e3caeeb526410ef8e26c
+size 3422777952

tokenizer_config.json

@@ -5,153 +5,10 @@
   "clean_up_tokenization_spaces": false,
   "eos_token": "<|im_end|>",
   "errors": "replace",
-  "extra_special_tokens": [
-    "<|im_start|>",
-    "<|im_end|>"
-  ],
-  "is_local": false,
+  "is_local": true,
   "model_max_length": 8192,
-  "pad_token": "<|endoftext|>",
-  "padding_side": "left",
+  "pad_token": "<|im_end|>",
   "tokenizer_class": "GPT2Tokenizer",
   "unk_token": "<|endoftext|>",
-  "vocab_size": 49152,
-  "added_tokens_decoder": {
-    "0": {
-      "content": "<|endoftext|>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    "1": {
-      "content": "<|im_start|>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    "2": {
-      "content": "<|im_end|>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    "3": {
-      "content": "<repo_name>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    "4": {
-      "content": "<reponame>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    "5": {
-      "content": "<file_sep>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    "6": {
-      "content": "<filename>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    "7": {
-      "content": "<gh_stars>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    "8": {
-      "content": "<issue_start>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    "9": {
-      "content": "<issue_comment>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    "10": {
-      "content": "<issue_closed>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    "11": {
-      "content": "<jupyter_start>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    "12": {
-      "content": "<jupyter_text>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    "13": {
-      "content": "<jupyter_code>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    "14": {
-      "content": "<jupyter_output>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    "15": {
-      "content": "<jupyter_script>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    },
-    "16": {
-      "content": "<empty_output>",
-      "single_word": false,
-      "lstrip": false,
-      "rstrip": false,
-      "normalized": false,
-      "special": true
-    }
-  }
+  "vocab_size": 49152
 }