Upload 11 files

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+d84e1b4a-ef6a-11ef-be05-a8a159eaf1f4 filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,143 @@
----
-license: apache-2.0
----
+---
+tags:
+- question-answering
+- bert
+license: apache-2.0
+datasets:
+- squad
+language:
+- en
+model-index:
+- name: dynamic-tinybert
+  results: 
+  - task:
+      type: question-answering
+      name: question-answering
+    metrics:
+    - type: f1
+      value: 88.71
+
+---
+
+## Model Details: Dynamic-TinyBERT: Boost TinyBERT's Inference Efficiency by Dynamic Sequence Length
+
+Dynamic-TinyBERT has been fine-tuned for the NLP task of question answering, trained on the SQuAD 1.1 dataset. [Guskin et al. (2021)](https://neurips2021-nlp.github.io/papers/16/CameraReady/Dynamic_TinyBERT_NLSP2021_camera_ready.pdf) note:
+
+> Dynamic-TinyBERT is a TinyBERT model that utilizes sequence-length reduction and Hyperparameter Optimization for enhanced inference efficiency per any computational budget. Dynamic-TinyBERT is trained only once, performing on-par with BERT and achieving an accuracy-speedup trade-off superior to any other efficient approaches (up to 3.3x with <1% loss-drop).
+
+
+
+| Model Detail | Description |
+| ----------- | ----------- | 
+| Model Authors - Company | Intel | 
+| Model Card Authors | Intel in collaboration with Hugging Face | 
+| Date | November 22, 2021 | 
+| Version | 1 | 
+| Type | NLP - Question Answering | 
+| Architecture | "For our Dynamic-TinyBERT model we use the architecture of TinyBERT6L: a small BERT model with 6 layers, a hidden size of 768, a feed forward size of 3072 and 12 heads." [Guskin et al. (2021)](https://gyuwankim.github.io/publication/dynamic-tinybert/poster.pdf) |
+| Paper or Other Resources | [Paper](https://neurips2021-nlp.github.io/papers/16/CameraReady/Dynamic_TinyBERT_NLSP2021_camera_ready.pdf); [Poster](https://gyuwankim.github.io/publication/dynamic-tinybert/poster.pdf); [GitHub Repo](https://github.com/IntelLabs/Model-Compression-Research-Package) | 
+| License | Apache 2.0 |
+| Questions or Comments | [Community Tab](https://huggingface.co/Intel/dynamic_tinybert/discussions) and [Intel Developers Discord](https://discord.gg/rv2Gp55UJQ)|
+
+| Intended Use | Description |
+| ----------- | ----------- | 
+| Primary intended uses | You can use the model for the NLP task of question answering: given a corpus of text, you can ask it a question about that text, and it will find the answer in the text. | 
+| Primary intended users | Anyone doing question answering | 
+| Out-of-scope uses |  The model should not be used to intentionally create hostile or alienating environments for people.|
+
+### How to use
+
+Here is how to import this model in Python:
+
+ <details>
+<summary> Click to expand </summary>
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForQuestionAnswering
+
+tokenizer = AutoTokenizer.from_pretrained("Intel/dynamic_tinybert")
+model = AutoModelForQuestionAnswering.from_pretrained("Intel/dynamic_tinybert")
+
+context = "remember the number 123456, I'll ask you later."
+question = "What is the number I told you?"
+
+# Tokenize the context and question
+tokens = tokenizer.encode_plus(question, context, return_tensors="pt", truncation=True)
+
+# Get the input IDs and attention mask
+input_ids = tokens["input_ids"]
+attention_mask = tokens["attention_mask"]
+
+# Perform question answering
+outputs = model(input_ids, attention_mask=attention_mask)
+start_scores = outputs.start_logits
+end_scores = outputs.end_logits
+
+# Find the start and end positions of the answer
+answer_start = torch.argmax(start_scores)
+answer_end = torch.argmax(end_scores) + 1
+answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[0][answer_start:answer_end]))
+
+# Print the answer
+print("Answer:", answer)
+ ```
+</details>
+
+
+| Factors | Description | 
+| ----------- | ----------- | 
+| Groups | Many Wikipedia articles with question and answer labels are contained in the training data | 
+| Instrumentation | - |
+| Environment | Training was completed on a Titan GPU. |
+| Card Prompts | Model deployment on alternate hardware and software will change model performance |
+
+| Metrics | Description | 
+| ----------- | ----------- | 
+| Model performance measures | F1 |
+| Decision thresholds | - | 
+| Approaches to uncertainty and variability | - | 
+
+| Training and Evaluation Data | Description | 
+| ----------- | ----------- | 
+| Datasets | SQuAD1.1: "Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable." (https://huggingface.co/datasets/squad)|
+| Motivation | To build an efficient and accurate model for the question answering task. |
+| Preprocessing | "We start with a pre-trained general-TinyBERT student, which was trained to learn the general knowledge of BERT using the general-distillation method presented by TinyBERT. We perform transformer distillation from a fine- tuned BERT teacher to the student, following the same training steps used in the original TinyBERT: (1) intermediate-layer distillation (ID) — learning the knowledge residing in the hidden states and attentions matrices, and (2) prediction-layer distillation (PD) — fitting the predictions of the teacher." ([Guskin et al., 2021](https://neurips2021-nlp.github.io/papers/16/CameraReady/Dynamic_TinyBERT_NLSP2021_camera_ready.pdf))| 
+
+Model Performance Analysis:
+
+| Model            | Max F1 (full model) | Best Speedup within BERT-1% |
+|------------------|---------------------|-----------------------------|
+| Dynamic-TinyBERT | 88.71               | 3.3x                        |
+
+| Ethical Considerations | Description | 
+| ----------- | ----------- | 
+| Data | The training data come from Wikipedia articles |
+| Human life | The model is not intended to inform decisions central to human life or flourishing. It is an aggregated set of labelled Wikipedia articles. | 
+| Mitigations | No additional risk mitigation strategies were considered during model development. |
+| Risks and harms | Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al., 2021](https://aclanthology.org/2021.acl-long.330.pdf), and [Bender et al., 2021](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. Beyond this, the extent of the risks involved by using the model remain unknown.|
+| Use cases | - | 
+
+
+| Caveats and Recommendations |
+| ----------- | 
+| Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. There are no additional caveats or recommendations for this model. |
+
+
+### BibTeX entry and citation info
+```bibtex
+@misc{https://doi.org/10.48550/arxiv.2111.09645,
+  doi = {10.48550/ARXIV.2111.09645},
+  
+  url = {https://arxiv.org/abs/2111.09645},
+  
+  author = {Guskin, Shira and Wasserblat, Moshe and Ding, Ke and Kim, Gyuwan},
+  
+  keywords = {Computation and Language (cs.CL), Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences},
+  
+  title = {Dynamic-TinyBERT: Boost TinyBERT's Inference Efficiency by Dynamic Sequence Length},
+  
+  publisher = {arXiv},
+  
+  year = {2021},
+```

config.json

@@ -0,0 +1,28 @@
+{
+  "_name_or_path": "/store/nosnap/results/inter6_bert_24.8.13.50/checkpoint-last",
+  "architectures": [
+    "TinyBertForQuestionAnswering"
+  ],
+  "attention_head_size": 26,
+  "attention_probs_dropout_prob": 0.1,
+  "cell": {},
+  "gradient_checkpointing": false,
+  "hidden_act": "relu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 6,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "pre_trained": "",
+  "structure": [],
+  "transformers_version": "4.7.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

d84e1b4a-ef6a-11ef-be05-a8a159eaf1f4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b49586a9c1b0b622ab68f429e58309cb7c94a253ae952a32c097d24b9a021b48
+size 265463808

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:559bc6e27704c82f08642c84a235a152983d619e9ba7e63fc2d6325c914f6e43
+size 267855035

special_tokens_map.json

@@ -0,0 +1 @@
+{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}

sym_shape_infer_temp.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b807c8f575fd06776726d5c485c7907e94f6adf2333fb1c80865ca907604ff33
+size 170716

to_onnx.py

@@ -0,0 +1,188 @@
+import torch
+from transformers import AutoModelForQuestionAnswering
+from transformers import AutoTokenizer, BertConfig
+import onnx
+from onnxruntime.quantization import quantize_dynamic, QuantType
+from onnxruntime.quantization import shape_inference
+import os
+import logging
+from typing import Optional, Dict, Any
+import subprocess  # Import the subprocess module
+
+class ONNXModelConverter:
+    def __init__(self, model_name: str, output_dir: str):
+        self.model_name = model_name
+        self.output_dir = output_dir
+        self.setup_logging()
+
+        os.makedirs(output_dir, exist_ok=True)
+
+        self.logger.info(f"Loading tokenizer {model_name}...")
+        self.tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
+
+        self.logger.info(f"Loading model {model_name}...")
+        self.model = AutoModelForQuestionAnswering.from_pretrained(
+                    model_name,
+                    trust_remote_code=True,
+                    torch_dtype=torch.float32
+                )
+        self.model.eval()
+
+    def setup_logging(self):
+        self.logger = logging.getLogger(__name__)
+        self.logger.setLevel(logging.INFO)
+        handler = logging.StreamHandler()
+        formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
+        handler.setFormatter(formatter)
+        self.logger.addHandler(handler)
+
+    def prepare_dummy_inputs(self):
+        dummy_input = self.tokenizer(
+            "Hello, how are you?",
+            return_tensors="pt",
+            padding=True,
+            truncation=True,
+            max_length=128
+        )
+        return {
+            'input_ids': dummy_input['input_ids'],
+            'attention_mask': dummy_input['attention_mask'],
+            'token_type_ids': dummy_input['token_type_ids']
+        }
+
+    def export_to_onnx(self):
+        output_path = os.path.join(self.output_dir, "model.onnx")
+        inputs = self.prepare_dummy_inputs()
+
+        dynamic_axes = {
+            'input_ids': {0: 'batch_size', 1: 'sequence_length'},
+            'attention_mask': {0: 'batch_size', 1: 'sequence_length'},
+            'token_type_ids': {0: 'batch_size', 1: 'sequence_length'},
+            'start_logits': {0: 'batch_size', 1: 'sequence_length'},
+            'end_logits': {0: 'batch_size', 1: 'sequence_length'},
+        }
+
+        class ModelWrapper(torch.nn.Module):
+            def __init__(self, model):
+                super().__init__()
+                self.model = model
+
+            def forward(self, input_ids, attention_mask, token_type_ids):
+                outputs = self.model(input_ids=input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids)
+                return outputs.start_logits, outputs.end_logits
+
+        wrapped_model = ModelWrapper(self.model)
+
+        try:
+            torch.onnx.export(
+                wrapped_model,
+                (inputs['input_ids'], inputs['attention_mask'], inputs['token_type_ids']),
+                output_path,
+                export_params=True,
+                opset_version=14,  # Or a suitable version
+                do_constant_folding=True,
+                input_names=['input_ids', 'attention_mask', 'token_type_ids'],
+                output_names=['start_logits', 'end_logits'],
+                dynamic_axes=dynamic_axes,
+                verbose=False
+            )
+            self.logger.info(f"Model exported to {output_path}")
+            return output_path
+        except Exception as e:
+            self.logger.error(f"ONNX export failed: {str(e)}")
+            raise
+
+    def verify_model(self, model_path: str):
+        try:
+            onnx_model = onnx.load(model_path)
+            onnx.checker.check_model(onnx_model)
+            self.logger.info("ONNX model verification successful")
+            return True
+        except Exception as e:
+            self.logger.error(f"Model verification failed: {str(e)}")
+            return False
+
+    def preprocess_model(self, model_path: str) -> str:
+        preprocessed_path = os.path.join(self.output_dir, "model-infer.onnx")
+        try:
+            command = [
+                "python", "-m", "onnxruntime.quantization.preprocess",
+                "--input", model_path,
+                "--output", preprocessed_path
+            ]
+            result = subprocess.run(command, check=True, capture_output=True, text=True)
+            if result.returncode == 0:
+                self.logger.info(f"Model preprocessing successful. Output saved to {preprocessed_path}")
+                return preprocessed_path
+            else:
+                raise subprocess.CalledProcessError(result.returncode, command, result.stdout, result.stderr)
+        except subprocess.CalledProcessError as e:
+            self.logger.error(f"Preprocessing failed: {e.stderr}")
+            raise
+        except Exception as e:
+            self.logger.error(f"Preprocessing failed: {str(e)}")
+            raise
+
+    def quantize_model(self, model_path: str):
+        weight_types = {'int4':QuantType.QInt4, 'int8':QuantType.QInt8, 'uint4':QuantType.QUInt4, 'uint8':QuantType.QUInt8, 'uint16':QuantType.QUInt16, 'int16':QuantType.QInt16}
+        all_quantized_paths = []
+        for weight_type in weight_types.keys():
+            quantized_path = os.path.join(self.output_dir, "model_" + weight_type + ".onnx")
+
+            try:
+                quantize_dynamic(
+                    model_path,
+                    quantized_path,
+                    weight_type=weight_types[weight_type]
+                )
+                self.logger.info(f"Model quantized ({weight_type}) and saved to {quantized_path}")
+                all_quantized_paths.append(quantized_path)
+            except Exception as e:
+                self.logger.error(f"Quantization ({weight_type}) failed: {str(e)}")
+                raise
+
+        return all_quantized_paths
+
+
+    def convert(self):
+        try:
+            onnx_path = self.export_to_onnx()
+
+            if self.verify_model(onnx_path):
+                # Add preprocessing step before quantization
+                # preprocessed_path = self.preprocess_model(onnx_path)
+
+                # Use preprocessed model for quantization
+                quantized_paths = self.quantize_model(onnx_path)
+
+                tokenizer_path = os.path.join(self.output_dir, "tokenizer")
+                self.tokenizer.save_pretrained(tokenizer_path)
+                self.logger.info(f"Tokenizer saved to {tokenizer_path}")
+
+                return {
+                    'onnx_model': onnx_path,
+                    'quantized_models': quantized_paths,  # Return a list of quantized model paths
+                    'tokenizer': tokenizer_path
+                }
+            else:
+                raise Exception("Model verification failed")
+
+        except Exception as e:
+            self.logger.error(f"Conversion process failed: {str(e)}")
+            raise
+
+if __name__ == "__main__":
+    MODEL_NAME = "Intel/dynamic_tinybert"  # Or any other suitable model
+    OUTPUT_DIR = "onnx"
+
+    try:
+        converter = ONNXModelConverter(MODEL_NAME, OUTPUT_DIR)
+        results = converter.convert()
+
+        print("\nConversion completed successfully!")
+        print(f"ONNX model path: {results['onnx_model']}")
+        print(f"Quantized model paths: {results['quantized_models']}") # Print the list
+        print(f"Tokenizer path: {results['tokenizer']}")
+
+    except Exception as e:
+        print(f"Conversion failed: {str(e)}")

tokenizer_config.json

@@ -0,0 +1 @@
+{"do_lower_case": true, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "special_tokens_map_file": null, "name_or_path": "/store/nosnap/results/inter6_bert_24.8.13.50/checkpoint-last", "do_basic_tokenize": true, "never_split": null}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:45211a37428e561ecc29dc69804a75bca37187c651ccb38f8fa237eefa978c1e
+size 2203