Reorganize into internlm2-1.8b-cpu-int4-awq and internlm2-7b-cpu-int4-awq directories

README.md

@@ -1,260 +0,0 @@
-<!--
-Copyright (C)  [2026]  Advanced Micro Devices, Inc. All rights reserved. Portions of this file consist of AI generated content
--->
-
-# InternLM2 Model Export for ONNX Runtime GenAI
-
-This example demonstrates how to export InternLM2 models to ONNX format using ONNX Runtime GenAI.
-
-## Supported Models
-
-All InternLM2 model sizes are supported:
-
-- ✅ **InternLM2-1.8B** - Tested and verified
-- ✅ **InternLM2-7B** - Tested and verified
-- ✅ **InternLM2-20B** - Fully compatible
-- ✅ **InternLM2-Chat variants** - All sizes supported
-
-The implementation is architecture-based and automatically adapts to any InternLM2 model size.
-
-## Model Architecture
-
-InternLM2 uses a Llama-based architecture with the following key features:
-
-- **Attention**: Grouped Query Attention (GQA) with grouped/interleaved QKV layout
-- **Normalization**: RMSNorm (eps: 1e-05)
-- **Activation**: SiLU
-- **Positional Encoding**: RoPE with theta=1,000,000
-
-### Architecture Specifications
-
-| Parameter | 1.8B | 7B | 20B |
-|-----------|------|-----|-----|
-| **Hidden Size** | 2048 | 4096 | 6144 |
-| **Num Layers** | 24 | 32 | 48 |
-| **Q Heads** | 16 | 32 | 48 |
-| **KV Heads** | 8 | 8 | 8 |
-| **Head Dim** | 128 | 128 | 128 |
-| **Intermediate** | 8192 | 14336 | 16384 |
-| **GQA Ratio** | 2:1 | 4:1 | 6:1 |
-| **Context Length** | 32,768 | 32,768 | 32,768 |
-| **Vocab Size** | 92,544 | 92,544 | 92,544 |
-
-## Export Examples
-
-### InternLM2-1.8B
-
-**FP32 (Best quality baseline):**
-```bash
-python -m onnxruntime_genai.models.builder \
-    --input internlm/internlm2-1_8b \
-    --output ./internlm2-1.8b-cpu-fp32 \
-    --precision fp32 \
-    --execution_provider cpu
-```
-
-**INT4 RTN (Fast quantization):**
-```bash
-python -m onnxruntime_genai.models.builder \
-    --input internlm/internlm2-1_8b \
-    --output ./internlm2-1.8b-cpu-int4 \
-    --precision int4 \
-    --execution_provider cpu
-```
-
-**INT4 AWQ (Best quality, recommended):**
-```bash
-python -m onnxruntime_genai.models.builder \
-    --input internlm/internlm2-1_8b \
-    --output ./internlm2-1.8b-cpu-int4-awq \
-    --precision int4 \
-    --execution_provider cpu \
-    --extra_options int4_accuracy_level=4
-```
-
-### InternLM2-7B
-
-**INT4 AWQ CPU (Recommended for most users):**
-```bash
-python -m onnxruntime_genai.models.builder \
-    --input internlm/internlm2-7b \
-    --output ./internlm2-7b-cpu-int4-awq \
-    --precision int4 \
-    --execution_provider cpu \
-    --extra_options int4_accuracy_level=4
-```
-
-**INT4 AWQ CUDA (For GPU inference):**
-```bash
-python -m onnxruntime_genai.models.builder \
-    --input internlm/internlm2-7b \
-    --output ./internlm2-7b-cuda-int4-awq \
-    --precision int4 \
-    --execution_provider cuda \
-    --extra_options int4_accuracy_level=4
-```
-
-**FP16 CUDA (Highest quality on GPU):**
-```bash
-python -m onnxruntime_genai.models.builder \
-    --input internlm/internlm2-7b \
-    --output ./internlm2-7b-cuda-fp16 \
-    --precision fp16 \
-    --execution_provider cuda
-```
-
-### InternLM2-20B
-
-**INT4 AWQ CUDA (Recommended):**
-```bash
-python -m onnxruntime_genai.models.builder \
-    --input internlm/internlm2-20b \
-    --output ./internlm2-20b-cuda-int4-awq \
-    --precision int4 \
-    --execution_provider cuda \
-    --extra_options int4_accuracy_level=4
-```
-
-## Model Size & Performance
-
-| Model | Original Size | INT4 Quantized | FP16 | Recommended RAM |
-|-------|--------------|----------------|------|-----------------|
-| **InternLM2-1.8B** | ~3.6 GB | ~1.0 GB | ~3.6 GB | 4 GB |
-| **InternLM2-7B** | ~14 GB | ~3.8 GB | ~14 GB | 8 GB |
-| **InternLM2-20B** | ~40 GB | ~10.5 GB | ~40 GB | 24 GB |
-
-**CPU Inference (Approximate):**
-
-| Model | Min RAM | Recommended RAM | Typical Speed |
-|-------|---------|-----------------|---------------|
-| 1.8B INT4 | 4 GB | 8 GB | 8-12 tok/s |
-| 7B INT4 | 8 GB | 16 GB | 2-4 tok/s |
-| 20B INT4 | 16 GB | 32 GB | 0.5-1 tok/s |
-
-**GPU Inference (CUDA):**
-
-| Model | Min VRAM | Recommended VRAM | Typical Speed |
-|-------|----------|------------------|---------------|
-| 1.8B INT4 | 2 GB | 4 GB | 50-80 tok/s |
-| 7B INT4 | 6 GB | 8 GB | 30-50 tok/s |
-| 7B FP16 | 14 GB | 16 GB | 40-60 tok/s |
-| 20B INT4 | 12 GB | 16 GB | 20-30 tok/s |
-| 20B FP16 | 40 GB | 48 GB | 25-35 tok/s |
-
-## Inference Example
-
-```python
-import onnxruntime_genai as og
-
-# Works with any InternLM2 size!
-model = og.Model("./internlm2-7b-cpu-int4-awq")
-tokenizer = og.Tokenizer(model)
-tokenizer_stream = tokenizer.create_stream()
-
-# Set generation parameters
-prompt = "What is the meaning of life?"
-tokens = tokenizer.encode(prompt)
-
-params = og.GeneratorParams(model)
-params.set_search_options(
-    max_length=200,
-    temperature=0.7,
-    top_p=0.9,
-    top_k=40
-)
-
-# Generate text
-generator = og.Generator(model, params)
-generator.append_tokens(tokens)
-
-print(prompt, end="", flush=True)
-while not generator.is_done():
-    generator.generate_next_token()
-    new_token = generator.get_next_tokens()[0]
-    print(tokenizer_stream.decode(new_token), end="", flush=True)
-print()
-```
-
-## Why Multi-Size Support Works
-
-### Architecture-Based Implementation
-
-The implementation is **size-agnostic** because it:
-
-1. **Dynamically reads config parameters** from each model:
-   - `num_attention_heads`
-   - `num_key_value_heads`
-   - `hidden_size`
-   - `num_hidden_layers`
-   - `intermediate_size`
-
-2. **Uses config-driven weight splitting**:
-   ```python
-   # Reads from model config
-   num_q_heads = config.num_attention_heads  # 16 for 1.8B, 32 for 7B, 48 for 20B
-   num_kv_heads = config.num_key_value_heads  # Always 8 for InternLM2
-   head_dim = config.hidden_size // num_q_heads  # Always 128
-   
-   # Calculates group size dynamically
-   num_kv_groups = num_q_heads // num_kv_heads  # 2 for 1.8B, 4 for 7B, 6 for 20B
-   group_size = num_kv_groups + 2
-   ```
-
-3. **Handles grouped QKV layout** for any GQA ratio:
-   - Layout: `[Group0: Q0,Q1,...,K0,V0 | Group1: Q2,Q3,...,K1,V1 | ...]`
-   - Each KV group contains multiple Q heads followed by K and V
-   - Correctly extracts weights regardless of the Q/KV head ratio
-
-4. **No hardcoded sizes** anywhere in the code
-
-### Key Implementation Notes
-
-**Grouped QKV Layout:**
-- InternLM2 uses a grouped/interleaved QKV weight layout for efficient Grouped Query Attention
-- The implementation in `src/python/py/models/builders/internlm.py` correctly handles this layout during weight extraction
-
-**Model Configuration:**
-- The exported model uses `model_type: "llama"` for ONNX Runtime GenAI compatibility
-- Tokenizer uses `tokenizer_class: "LlamaTokenizer"` (SentencePiece-based)
-
-## Recommendations by Use Case
-
-### Development & Testing
-- **InternLM2-1.8B INT4 AWQ** (1 GB)
-- Fast iteration, quick testing
-- Good for prototyping
-
-### Production Applications
-- **InternLM2-7B INT4 AWQ** (3.8 GB)
-- Best balance of quality and performance
-- Suitable for most real-world applications
-
-### High-Quality Applications
-- **InternLM2-7B FP16 CUDA** (14 GB) or
-- **InternLM2-20B INT4 CUDA** (10.5 GB)
-- Maximum quality for critical applications
-
-## Troubleshooting
-
-### "Out of Memory" errors
-- Use INT4 quantization instead of FP16/FP32
-- Enable GPU inference for larger models
-- Use batch_size=1 for inference
-
-### Slow inference on CPU
-- This is expected for 7B+ models
-- Consider GPU inference
-- Use INT4 quantization (2-3x faster than FP16)
-
-### Model not loading
-- Ensure you have enough RAM/VRAM
-- Check that you're using `--execution_provider cuda` for GPU models
-- Verify ONNX Runtime GenAI installation
-
-## References
-
-- Model Hub (1.8B): https://huggingface.co/internlm/internlm2-1_8b
-- Model Hub (7B): https://huggingface.co/internlm/internlm2-7b
-- Model Hub (20B): https://huggingface.co/internlm/internlm2-20b
-- Paper: https://arxiv.org/abs/2403.17297
-- GitHub: https://github.com/InternLM/InternLM

internlm2-1.8b-cpu-int4-awq/model.onnx.data

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3dc0644a406bab41fc434b82d5c8d15052a192ac65c04b5c1390b3e6c55a1490
+size 1837563904

internlm2-1.8b-cpu-int4-awq/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53bf68bdb380527f8c67449108021556e11c9de61aee72f818778143a99ecb50
+size 10540375

internlm2-1.8b-cpu-int4-awq/tokenizer_config.json

@@ -0,0 +1,46 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "auto_map": {
+    "AutoTokenizer": [
+      "tokenization_internlm2.InternLM2Tokenizer",
+      "tokenization_internlm2_fast.InternLM2TokenizerFast"
+    ]
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "decode_with_prefix_space": false,
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "sp_model_kwargs": null,
+  "tokenizer_class": "InternLM2Tokenizer",
+  "unk_token": "<unk>"
+}

internlm2-7b-cpu-int4-awq/genai_config.json

@@ -0,0 +1,49 @@
+{
+    "model": {
+        "bos_token_id": 1,
+        "context_length": 32768,
+        "decoder": {
+            "session_options": {
+                "log_id": "onnxruntime-genai",
+                "provider_options": []
+            },
+            "filename": "model.onnx",
+            "head_size": 128,
+            "hidden_size": 4096,
+            "inputs": {
+                "input_ids": "input_ids",
+                "attention_mask": "attention_mask",
+                "past_key_names": "past_key_values.%d.key",
+                "past_value_names": "past_key_values.%d.value"
+            },
+            "outputs": {
+                "logits": "logits",
+                "present_key_names": "present.%d.key",
+                "present_value_names": "present.%d.value"
+            },
+            "num_attention_heads": 32,
+            "num_hidden_layers": 32,
+            "num_key_value_heads": 8
+        },
+        "eos_token_id": 2,
+        "pad_token_id": 2,
+        "type": "llama",
+        "vocab_size": 92544
+    },
+    "search": {
+        "diversity_penalty": 0.0,
+        "do_sample": false,
+        "early_stopping": true,
+        "length_penalty": 1.0,
+        "max_length": 32768,
+        "min_length": 0,
+        "no_repeat_ngram_size": 0,
+        "num_beams": 1,
+        "num_return_sequences": 1,
+        "past_present_share_buffer": true,
+        "repetition_penalty": 1.0,
+        "temperature": 1.0,
+        "top_k": 50,
+        "top_p": 1.0
+    }
+}

internlm2-7b-cpu-int4-awq/model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:862b2f22bc845237107303a06832042c9d4641fab30a740b7ef6dfed99b146c8
+size 239348

internlm2-7b-cpu-int4-awq/model.onnx.data

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8a41e20041f6eb7f1b24f6c539c9bb6681dd7ab5450f5351fa8966d0514f05f7
+size 6133121024

internlm2-7b-cpu-int4-awq/special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<s>",
+  "eos_token": "</s>",
+  "pad_token": "</s>",
+  "unk_token": "<unk>"
+}

internlm2-7b-cpu-int4-awq/tokenization_internlm2.py

@@ -0,0 +1,236 @@
+# coding=utf-8
+# Copyright (c) The InternLM team and The HuggingFace Inc. team. All rights reserved.
+#
+# This code is based on transformers/src/transformers/models/llama/tokenization_llama.py
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tokenization classes for InternLM."""
+import os
+from shutil import copyfile
+from typing import Any, Dict, List, Optional, Tuple
+
+import sentencepiece as spm
+from transformers.tokenization_utils import PreTrainedTokenizer
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+VOCAB_FILES_NAMES = {"vocab_file": "./tokenizer.model"}
+
+PRETRAINED_VOCAB_FILES_MAP = {}
+
+
+# Modified from transformers.model.llama.tokenization_llama.LlamaTokenizer
+class InternLM2Tokenizer(PreTrainedTokenizer):
+    """
+    Construct a InternLM2 tokenizer. Based on byte-level Byte-Pair-Encoding.
+
+    Args:
+        vocab_file (`str`):
+            Path to the vocabulary file.
+    """
+
+    vocab_files_names = VOCAB_FILES_NAMES
+    pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
+    model_input_names = ["input_ids", "attention_mask"]
+    _auto_class = "AutoTokenizer"
+
+    def __init__(
+        self,
+        vocab_file,
+        unk_token="<unk>",
+        bos_token="<s>",
+        eos_token="</s>",
+        pad_token="</s>",
+        sp_model_kwargs: Optional[Dict[str, Any]] = None,
+        add_bos_token=True,
+        add_eos_token=False,
+        decode_with_prefix_space=False,
+        clean_up_tokenization_spaces=False,
+        **kwargs,
+    ):
+        self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs
+        self.vocab_file = vocab_file
+        self.add_bos_token = add_bos_token
+        self.add_eos_token = add_eos_token
+        self.decode_with_prefix_space = decode_with_prefix_space
+        self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
+        self.sp_model.Load(vocab_file)
+        self._no_prefix_space_tokens = None
+        super().__init__(
+            bos_token=bos_token,
+            eos_token=eos_token,
+            unk_token=unk_token,
+            pad_token=pad_token,
+            clean_up_tokenization_spaces=clean_up_tokenization_spaces,
+            **kwargs,
+        )
+
+    @property
+    def no_prefix_space_tokens(self):
+        if self._no_prefix_space_tokens is None:
+            vocab = self.convert_ids_to_tokens(list(range(self.vocab_size)))
+            self._no_prefix_space_tokens = {i for i, tok in enumerate(vocab) if not tok.startswith("▁")}
+        return self._no_prefix_space_tokens
+
+    @property
+    def vocab_size(self):
+        """Returns vocab size"""
+        return self.sp_model.get_piece_size()
+
+    @property
+    def bos_token_id(self) -> Optional[int]:
+        return self.sp_model.bos_id()
+
+    @property
+    def eos_token_id(self) -> Optional[int]:
+        return self.sp_model.eos_id()
+
+    def get_vocab(self):
+        """Returns vocab as a dict"""
+        vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)}
+        vocab.update(self.added_tokens_encoder)
+        return vocab
+
+    def _tokenize(self, text):
+        """Returns a tokenized string."""
+        return self.sp_model.encode(text, out_type=str)
+
+    def _convert_token_to_id(self, token):
+        """Converts a token (str) in an id using the vocab."""
+        return self.sp_model.piece_to_id(token)
+
+    def _convert_id_to_token(self, index):
+        """Converts an index (integer) in a token (str) using the vocab."""
+        token = self.sp_model.IdToPiece(index)
+        return token
+
+    def _maybe_add_prefix_space(self, tokens, decoded):
+        if tokens and tokens[0] not in self.no_prefix_space_tokens:
+            return " " + decoded
+        else:
+            return decoded
+
+    def convert_tokens_to_string(self, tokens):
+        """Converts a sequence of tokens (string) in a single string."""
+        current_sub_tokens = []
+        out_string = ""
+        prev_is_special = False
+        for token in tokens:
+            # make sure that special tokens are not decoded using sentencepiece model
+            if token in self.all_special_tokens:
+                if not prev_is_special:
+                    out_string += " "
+                out_string += self.sp_model.decode(current_sub_tokens) + token
+                prev_is_special = True
+                current_sub_tokens = []
+            else:
+                current_sub_tokens.append(token)
+                prev_is_special = False
+        out_string += self.sp_model.decode(current_sub_tokens)
+        out_string = self.clean_up_tokenization(out_string)
+        out_string = self._maybe_add_prefix_space(tokens=tokens, decoded=out_string)
+        return out_string[1:]
+
+    def save_vocabulary(self, save_directory, filename_prefix: Optional[str] = None) -> Tuple[str]:
+        """
+        Save the vocabulary and special tokens file to a directory.
+
+        Args:
+            save_directory (`str`):
+                The directory in which to save the vocabulary.
+
+        Returns:
+            `Tuple(str)`: Paths to the files saved.
+        """
+        if not os.path.isdir(save_directory):
+            logger.error(f"Vocabulary path ({save_directory}) should be a directory")
+            return
+        out_vocab_file = os.path.join(
+            save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
+        )
+
+        if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file) and os.path.isfile(self.vocab_file):
+            copyfile(self.vocab_file, out_vocab_file)
+        elif not os.path.isfile(self.vocab_file):
+            with open(out_vocab_file, "wb") as fi:
+                content_spiece_model = self.sp_model.serialized_model_proto()
+                fi.write(content_spiece_model)
+
+        return (out_vocab_file,)
+
+    def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None):
+        if self.add_bos_token:
+            bos_token_ids = [self.bos_token_id]
+        else:
+            bos_token_ids = []
+
+        output = bos_token_ids + token_ids_0
+
+        if token_ids_1 is not None:
+            output = output + token_ids_1
+
+        if self.add_eos_token:
+            output = output + [self.eos_token_id]
+
+        return output
+
+    def get_special_tokens_mask(
+        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False
+    ) -> List[int]:
+        """
+        Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
+        special tokens using the tokenizer `prepare_for_model` method.
+
+        Args:
+            token_ids_0 (`List[int]`):
+                List of IDs.
+            token_ids_1 (`List[int]`, *optional*):
+                Optional second list of IDs for sequence pairs.
+            already_has_special_tokens (`bool`, *optional*, defaults to `False`):
+                Whether or not the token list is already formatted with special tokens for the model.
+
+        Returns:
+            `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
+        """
+        if already_has_special_tokens:
+            return super().get_special_tokens_mask(
+                token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True
+            )
+
+        if token_ids_1 is None:
+            return [1] + ([0] * len(token_ids_0)) + [1]
+        return [1] + ([0] * len(token_ids_0)) + [1, 1] + ([0] * len(token_ids_1)) + [1]
+
+    def create_token_type_ids_from_sequences(
+        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
+    ) -> List[int]:
+        """
+        Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make
+        use of token type ids, therefore a list of zeros is returned.
+
+        Args:
+            token_ids_0 (`List[int]`):
+                List of IDs.
+            token_ids_1 (`List[int]`, *optional*):
+                Optional second list of IDs for sequence pairs.
+
+        Returns:
+            `List[int]`: List of zeros.
+        """
+        eos = [self.eos_token_id]
+
+        if token_ids_1 is None:
+            return len(token_ids_0 + eos) * [0]
+        return len(token_ids_0 + eos + token_ids_1 + eos) * [0]

internlm2-7b-cpu-int4-awq/tokenization_internlm2_fast.py

@@ -0,0 +1,214 @@
+# coding=utf-8
+# Copyright (c) The InternLM team and The HuggingFace Inc. team. All rights reserved.
+#
+# This code is based on transformers/src/transformers/models/llama/tokenization_llama_fast.py
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tokenization Fast class for InternLM."""
+import os
+from shutil import copyfile
+from typing import Any, Dict, Optional, Tuple
+
+from tokenizers import processors, decoders, Tokenizer, normalizers
+from tokenizers.models import BPE
+
+from transformers.tokenization_utils_fast import PreTrainedTokenizerFast
+from transformers.utils import logging
+
+from transformers.convert_slow_tokenizer import (
+    SLOW_TO_FAST_CONVERTERS,
+    SpmConverter,
+    SentencePieceExtractor,
+)
+
+from .tokenization_internlm2 import InternLM2Tokenizer
+
+logger = logging.get_logger(__name__)
+
+VOCAB_FILES_NAMES = {"vocab_file": "./tokenizer.model"}
+
+# Modified from transformers.convert_slow_tokenizer.LlamaConverter
+class InternLM2Converter(SpmConverter):
+    handle_byte_fallback = True
+
+    def vocab(self, proto):
+        vocab = [
+            ("<unk>", 0.0),
+            ("<s>", 0.0),
+            ("</s>", 0.0),
+        ]
+        vocab += [(piece.piece, piece.score) for piece in proto.pieces[3:]]
+        return vocab
+
+    def unk_id(self, proto):
+        unk_id = 0
+        return unk_id
+
+    def decoder(self, replacement, add_prefix_space):
+        decoders_sequence = [
+            decoders.Replace("▁", " "),
+            decoders.ByteFallback(),
+            decoders.Fuse(),
+        ]
+        if self.proto.normalizer_spec.add_dummy_prefix:
+            decoders_sequence.append(decoders.Strip(content=" ", left=1))
+        return decoders.Sequence(decoders_sequence)
+
+    def tokenizer(self, proto):
+        model_type = proto.trainer_spec.model_type
+        vocab_scores = self.vocab(proto)
+        # special tokens
+        added_tokens = self.original_tokenizer.added_tokens_decoder
+        for i in range(len(vocab_scores)):
+            piece, score = vocab_scores[i]
+            if i in added_tokens:
+                vocab_scores[i] = (added_tokens[i].content, score)
+        if model_type == 1:
+            raise RuntimeError("InternLM2 is supposed to be a BPE model!")
+
+        elif model_type == 2:
+            _, merges = SentencePieceExtractor(self.original_tokenizer.vocab_file).extract(vocab_scores)
+            bpe_vocab = {word: i for i, (word, _score) in enumerate(vocab_scores)}
+            tokenizer = Tokenizer(
+                BPE(bpe_vocab, merges, unk_token=proto.trainer_spec.unk_piece, fuse_unk=True, byte_fallback=True)
+            )
+            tokenizer.add_special_tokens(
+                [ added_token for index, added_token in added_tokens.items()]
+            )
+        else:
+            raise Exception(
+                "You're trying to run a `Unigram` model but you're file was trained with a different algorithm"
+            )
+
+        return tokenizer
+
+    def normalizer(self, proto):
+        normalizers_list = []
+        if proto.normalizer_spec.add_dummy_prefix:
+            normalizers_list.append(normalizers.Prepend(prepend="▁"))
+        normalizers_list.append(normalizers.Replace(pattern=" ", content="▁"))
+        return normalizers.Sequence(normalizers_list)
+
+    def pre_tokenizer(self, replacement, add_prefix_space):
+        return None
+
+SLOW_TO_FAST_CONVERTERS["InternLM2Tokenizer"] = InternLM2Converter
+
+
+# Modified from transformers.model.llama.tokenization_llama_fast.LlamaTokenizerFast -> InternLM2TokenizerFast
+class InternLM2TokenizerFast(PreTrainedTokenizerFast):
+    vocab_files_names = VOCAB_FILES_NAMES
+    slow_tokenizer_class = InternLM2Tokenizer
+    padding_side = "left"
+    model_input_names = ["input_ids", "attention_mask"]
+    _auto_class = "AutoTokenizer"
+
+    def __init__(
+        self,
+        vocab_file,
+        unk_token="<unk>",
+        bos_token="<s>",
+        eos_token="</s>",
+        pad_token="</s>",
+        sp_model_kwargs: Optional[Dict[str, Any]] = None,
+        add_bos_token=True,
+        add_eos_token=False,
+        decode_with_prefix_space=False,
+        clean_up_tokenization_spaces=False,
+        **kwargs,
+    ):
+        super().__init__(
+            vocab_file=vocab_file,
+            unk_token=unk_token,
+            bos_token=bos_token,
+            eos_token=eos_token,
+            pad_token=pad_token,
+            sp_model_kwargs=sp_model_kwargs,
+            add_bos_token=add_bos_token,
+            add_eos_token=add_eos_token,
+            decode_with_prefix_space=decode_with_prefix_space,
+            clean_up_tokenization_spaces=clean_up_tokenization_spaces,
+            **kwargs,
+        )
+        self._add_bos_token = add_bos_token
+        self._add_eos_token = add_eos_token
+        self.update_post_processor()
+        self.vocab_file = vocab_file
+
+    @property
+    def can_save_slow_tokenizer(self) -> bool:
+        return os.path.isfile(self.vocab_file) if self.vocab_file else False
+
+    def update_post_processor(self):
+        """
+        Updates the underlying post processor with the current `bos_token` and `eos_token`.
+        """
+        bos = self.bos_token
+        bos_token_id = self.bos_token_id
+        if bos is None and self.add_bos_token:
+            raise ValueError("add_bos_token = True but bos_token = None")
+
+        eos = self.eos_token
+        eos_token_id = self.eos_token_id
+        if eos is None and self.add_eos_token:
+            raise ValueError("add_eos_token = True but eos_token = None")
+
+        single = f"{(bos+':0 ') if self.add_bos_token else ''}$A:0{(' '+eos+':0') if self.add_eos_token else ''}"
+        pair = f"{single}{(' '+bos+':1') if self.add_bos_token else ''} $B:1{(' '+eos+':1') if self.add_eos_token else ''}"
+
+        special_tokens = []
+        if self.add_bos_token:
+            special_tokens.append((bos, bos_token_id))
+        if self.add_eos_token:
+            special_tokens.append((eos, eos_token_id))
+        self._tokenizer.post_processor = processors.TemplateProcessing(
+            single=single, pair=pair, special_tokens=special_tokens
+        )
+
+    @property
+    def add_eos_token(self):
+        return self._add_eos_token
+
+    @property
+    def add_bos_token(self):
+        return self._add_bos_token
+
+    @add_eos_token.setter
+    def add_eos_token(self, value):
+        self._add_eos_token = value
+        self.update_post_processor()
+
+    @add_bos_token.setter
+    def add_bos_token(self, value):
+        self._add_bos_token = value
+        self.update_post_processor()
+
+    def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
+        if not self.can_save_slow_tokenizer:
+            raise ValueError(
+                "Your fast tokenizer does not have the necessary information to save the vocabulary for a slow "
+                "tokenizer."
+            )
+
+        if not os.path.isdir(save_directory):
+            logger.error(f"Vocabulary path ({save_directory}) should be a directory")
+            return
+        out_vocab_file = os.path.join(
+            save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
+        )
+
+        if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file):
+            copyfile(self.vocab_file, out_vocab_file)
+
+        return (out_vocab_file,)

internlm2-7b-cpu-int4-awq/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c6fe97617a059964f5afbfb575339f10960465c2f4ae16d8f533d7766092181a
+size 10540271

internlm2-7b-cpu-int4-awq/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f868398fc4e05ee1e8aeba95ddf18ddcc45b8bce55d5093bead5bbf80429b48b
+size 1477754