Upload 12 files

codet5p/README.md

@@ -0,0 +1,116 @@
+---
+license: bsd-3-clause
+---
+
+# CodeT5+ 110M Embedding Models
+
+## Model description
+
+[CodeT5+](https://github.com/salesforce/CodeT5/tree/main/CodeT5+) is a new family of open code large language models
+with an encoder-decoder architecture that can flexibly operate in different modes (i.e. _encoder-only_, _decoder-only_,
+and _encoder-decoder_) to support a wide range of code understanding and generation tasks.
+It is introduced in the paper:
+
+[CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://arxiv.org/pdf/2305.07922.pdf)
+by [Yue Wang](https://yuewang-cuhk.github.io/)\*, [Hung Le](https://sites.google.com/view/henryle2018/home?pli=1)\*, [Akhilesh Deepak Gotmare](https://akhileshgotmare.github.io/), [Nghi D.Q. Bui](https://bdqnghi.github.io/), [Junnan Li](https://sites.google.com/site/junnanlics), [Steven C.H. Hoi](https://sites.google.com/view/stevenhoi/home) (*
+indicates equal contribution).
+
+Compared to the original CodeT5 family (base: `220M`, large: `770M`), CodeT5+ is pretrained with a diverse set of
+pretraining tasks including _span denoising_, _causal language modeling_, _contrastive learning_, and _text-code
+matching_ to learn rich representations from both unimodal code data and bimodal code-text data.
+Additionally, it employs a simple yet effective _compute-efficient pretraining_ method to initialize the model
+components with frozen off-the-shelf LLMs such as [CodeGen](https://github.com/salesforce/CodeGen) to efficiently scale
+up the model (i.e. `2B`, `6B`, `16B`), and adopts a "shallow encoder and deep decoder" architecture.
+Furthermore, it is instruction-tuned to align with natural language instructions (see our InstructCodeT5+ 16B)
+following [Code Alpaca](https://github.com/sahil280114/codealpaca).
+
+## How to use
+
+This checkpoint consists of an encoder of CodeT5+ 220M model (pretrained from 2 stages on both unimodal and bimodal) and a projection layer, which can be used to extract code
+embeddings of 256 dimension. It can be easily loaded using the `AutoModel` functionality and employs the
+same [CodeT5](https://github.com/salesforce/CodeT5) tokenizer.
+
+```python
+from transformers import AutoModel, AutoTokenizer
+
+checkpoint = "Salesforce/codet5p-110m-embedding"
+device = "cuda"  # for GPU usage or "cpu" for CPU usage
+
+tokenizer = AutoTokenizer.from_pretrained(checkpoint, trust_remote_code=True)
+model = AutoModel.from_pretrained(checkpoint, trust_remote_code=True).to(device)
+
+inputs = tokenizer.encode("def print_hello_world():\tprint('Hello World!')", return_tensors="pt").to(device)
+embedding = model(inputs)[0]
+print(f'Dimension of the embedding: {embedding.size()[0]}, with norm={embedding.norm().item()}')
+# Dimension of the embedding: 256, with norm=1.0
+print(embedding)
+# tensor([ 0.0185,  0.0229, -0.0315, -0.0307, -0.1421, -0.0575, -0.0275,  0.0501,
+#          0.0203,  0.0337, -0.0067, -0.0075, -0.0222, -0.0107, -0.0250, -0.0657,
+#          0.1571, -0.0994, -0.0370,  0.0164, -0.0948,  0.0490, -0.0352,  0.0907,
+#         -0.0198,  0.0130, -0.0921,  0.0209,  0.0651,  0.0319,  0.0299, -0.0173,
+#         -0.0693, -0.0798, -0.0066, -0.0417,  0.1076,  0.0597, -0.0316,  0.0940,
+#         -0.0313,  0.0993,  0.0931, -0.0427,  0.0256,  0.0297, -0.0561, -0.0155,
+#         -0.0496, -0.0697, -0.1011,  0.1178,  0.0283, -0.0571, -0.0635, -0.0222,
+#          0.0710, -0.0617,  0.0423, -0.0057,  0.0620, -0.0262,  0.0441,  0.0425,
+#         -0.0413, -0.0245,  0.0043,  0.0185,  0.0060, -0.1727, -0.1152,  0.0655,
+#         -0.0235, -0.1465, -0.1359,  0.0022,  0.0177, -0.0176, -0.0361, -0.0750,
+#         -0.0464, -0.0846, -0.0088,  0.0136, -0.0221,  0.0591,  0.0876, -0.0903,
+#          0.0271, -0.1165, -0.0169, -0.0566,  0.1173, -0.0801,  0.0430,  0.0236,
+#          0.0060, -0.0778, -0.0570,  0.0102, -0.0172, -0.0051, -0.0891, -0.0620,
+#         -0.0536,  0.0190, -0.0039, -0.0189, -0.0267, -0.0389, -0.0208,  0.0076,
+#         -0.0676,  0.0630, -0.0962,  0.0418, -0.0172, -0.0229, -0.0452,  0.0401,
+#          0.0270,  0.0677, -0.0111, -0.0089,  0.0175,  0.0703,  0.0714, -0.0068,
+#          0.1214, -0.0004,  0.0020,  0.0255,  0.0424, -0.0030,  0.0318,  0.1227,
+#          0.0676, -0.0723,  0.0970,  0.0637, -0.0140, -0.0283, -0.0120,  0.0343,
+#         -0.0890,  0.0680,  0.0514,  0.0513,  0.0627, -0.0284, -0.0479,  0.0068,
+#         -0.0794,  0.0202,  0.0208, -0.0113, -0.0747,  0.0045, -0.0854, -0.0609,
+#         -0.0078,  0.1168,  0.0618, -0.0223, -0.0755,  0.0182, -0.0128,  0.1116,
+#          0.0240,  0.0342,  0.0119, -0.0235, -0.0150, -0.0228, -0.0568, -0.1528,
+#          0.0164, -0.0268,  0.0727, -0.0569,  0.1306,  0.0643, -0.0158, -0.1070,
+#         -0.0107, -0.0139, -0.0363,  0.0366, -0.0986, -0.0628, -0.0277,  0.0316,
+#          0.0363,  0.0038, -0.1092, -0.0679, -0.1398, -0.0648,  0.1711, -0.0666,
+#          0.0563,  0.0581,  0.0226,  0.0347, -0.0672, -0.0229, -0.0565,  0.0623,
+#          0.1089, -0.0687, -0.0901, -0.0073,  0.0426,  0.0870, -0.0390, -0.0144,
+#         -0.0166,  0.0262, -0.0310,  0.0467, -0.0164, -0.0700, -0.0602, -0.0720,
+#         -0.0386,  0.0067, -0.0337, -0.0053,  0.0829,  0.1004,  0.0427,  0.0026,
+#         -0.0537,  0.0951,  0.0584, -0.0583, -0.0208,  0.0124,  0.0067,  0.0403,
+#          0.0091, -0.0044, -0.0036,  0.0524,  0.1103, -0.1511, -0.0479,  0.1709,
+#          0.0772,  0.0721, -0.0332,  0.0866,  0.0799, -0.0581,  0.0713,  0.0218],
+#        device='cuda:0', grad_fn=<SelectBackward0>)
+```
+
+## Pretraining data
+
+This checkpoint is trained on the stricter permissive subset of the deduplicated version of
+the [github-code dataset](https://huggingface.co/datasets/codeparrot/github-code).
+The data is preprocessed by reserving only permissively licensed code ("mit" “apache-2”, “bsd-3-clause”, “bsd-2-clause”,
+“cc0-1.0”, “unlicense”, “isc”).
+Supported languages (9 in total) are as follows:
+`c`, `c++`, `c-sharp`,  `go`, `java`, `javascript`,  `php`, `python`, `ruby.`
+
+## Training procedure
+
+This checkpoint is first trained on the unimodal code data at the first-stage pretraining and then on bimodal text-code
+pair data using the proposed mixture of pretraining tasks.
+Please refer to the paper for more details.
+
+## Evaluation results
+
+We show the zero-shot results of this checkpoint on 6 downstream code retrieval tasks from CodeXGLUE in the following table.
+| Ruby  | JavaScript | Go    | Python | Java  | PHP   | Overall |
+| ----- | ---------- | ----- | ------ | ----- | ----- | ------- |
+| 74.51 | 69.07      | 90.69 | 71.55  | 71.82 | 67.72 | 74.23   |
+
+## BibTeX entry and citation info
+
+```bibtex
+@article{wang2023codet5plus,
+  title={CodeT5+: Open Code Large Language Models for Code Understanding and Generation},
+  author={Wang, Yue and Le, Hung and Gotmare, Akhilesh Deepak and Bui, Nghi D.Q. and Li, Junnan and Hoi, Steven C. H.},
+  journal={arXiv preprint},
+  year={2023}
+}
+```
+
+## Ethical Considerations
+This release is for research purposes only in support of an academic paper. Our models, datasets, and code are not specifically designed or evaluated for all downstream purposes. We strongly recommend users evaluate and address potential concerns related to accuracy, safety, and fairness before deploying this model. We encourage users to consider the common limitations of AI, comply with applicable laws, and leverage best practices when selecting use cases, particularly for high-risk scenarios where errors or misuse could significantly impact people’s lives, rights, or safety. For further guidance on use cases, refer to our AUP and AI AUP.

codet5p/added_tokens.json

@@ -0,0 +1,5 @@
+{
+  "[CDEC]": 32102,
+  "[ENC]": 32100,
+  "[TDEC]": 32101
+}

codet5p/config.json

@@ -0,0 +1,44 @@
+{
+  "_name_or_path": "Salesforce/codet5p-110m-embedding",
+  "architectures": [
+    "CodeT5p_Embedding"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_codet5p_embedding.CodeT5pEmbeddingConfig",
+    "AutoModel": "modeling_codet5p_embedding.CodeT5pEmbeddingModel"
+  },
+  "bos_token_id": 1,
+  "d_ff": 3072,
+  "d_kv": 64,
+  "d_model": 768,
+  "embed_dim": 256,
+  "decoder_start_token_id": 0,
+  "dense_act_fn": "relu",
+  "dropout_rate": 0.1,
+  "eos_token_id": 2,
+  "feed_forward_proj": "relu",
+  "gradient_checkpointing": false,
+  "id2label": {
+    "0": "LABEL_0"
+  },
+  "initializer_factor": 1.0,
+  "is_encoder_decoder": true,
+  "is_gated_act": false,
+  "label2id": {
+    "LABEL_0": 0
+  },
+  "layer_norm_epsilon": 1e-06,
+  "model_type": "codet5p_embedding",
+  "n_positions": 512,
+  "num_heads": 12,
+  "num_layers": 12,
+  "output_past": true,
+  "pad_token_id": 0,
+  "relative_attention_max_distance": 128,
+  "relative_attention_num_buckets": 32,
+
+  "torch_dtype": "float32",
+  "transformers_version": "4.21.3",
+  "use_cache": true,
+  "vocab_size": 32103
+}

codet5p/configuration_codet5p_embedding.py

@@ -0,0 +1,72 @@
+# coding=utf-8
+# Copyright 2023 Salesforce authors, The EleutherAI, and HuggingFace Teams. All rights reserved.
+
+""" CodeT5+ embedding model configuration"""
+from transformers.configuration_utils import PretrainedConfig
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+
+class CodeT5pEmbeddingConfig(PretrainedConfig):
+    model_type = "codet5p_embedding"
+    keys_to_ignore_at_inference = ["past_key_values"]
+    attribute_map = {"hidden_size": "d_model", "num_attention_heads": "num_heads", "num_hidden_layers": "num_layers"}
+
+    def __init__(
+            self,
+            vocab_size=32103,
+            d_model=768,
+            embed_dim=256,
+            d_kv=64,
+            d_ff=3072,
+            num_layers=12,
+            num_heads=12,
+            relative_attention_num_buckets=32,
+            relative_attention_max_distance=128,
+            dropout_rate=0.1,
+            layer_norm_epsilon=1e-6,
+            initializer_factor=1.0,
+            feed_forward_proj="relu",
+            is_encoder_decoder=False,
+            use_cache=True,
+            pad_token_id=0,
+            eos_token_id=2,
+            **kwargs
+    ):
+        self.vocab_size = vocab_size
+        self.d_model = d_model
+        self.embed_dim = embed_dim
+        self.d_kv = d_kv
+        self.d_ff = d_ff
+        self.num_layers = num_layers
+        self.num_heads = num_heads
+        self.relative_attention_num_buckets = relative_attention_num_buckets
+        self.relative_attention_max_distance = relative_attention_max_distance
+        self.dropout_rate = dropout_rate
+        self.layer_norm_epsilon = layer_norm_epsilon
+        self.initializer_factor = initializer_factor
+        self.feed_forward_proj = feed_forward_proj
+        self.use_cache = use_cache
+
+        act_info = self.feed_forward_proj.split("-")
+        self.dense_act_fn = act_info[-1]
+        self.is_gated_act = act_info[0] == "gated"
+
+        if len(act_info) > 1 and act_info[0] != "gated" or len(act_info) > 2:
+            raise ValueError(
+                f"`feed_forward_proj`: {feed_forward_proj} is not a valid activation function of the dense layer."
+                "Please make sure `feed_forward_proj` is of the format `gated-{ACT_FN}` or `{ACT_FN}`, e.g. "
+                "'gated-gelu' or 'relu'"
+            )
+
+        # for backwards compatibility
+        if feed_forward_proj == "gated-gelu":
+            self.dense_act_fn = "gelu_new"
+
+        super().__init__(
+            pad_token_id=pad_token_id,
+            eos_token_id=eos_token_id,
+            is_encoder_decoder=is_encoder_decoder,
+            **kwargs,
+        )

codet5p/gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

codet5p/modeling_codet5p_embedding.py

@@ -0,0 +1,53 @@
+# coding=utf-8
+# Copyright 2023 Salesforce authors, The EleutherAI, and HuggingFace Teams. All rights reserved.
+""" PyTorch CodeT5+ mbedding models.
+The implementation is based on transformers.models.t5.modeling_t5 by adding a projection layer on T5EncoderModel
+"""
+
+from typing import Optional, Tuple, Union
+import torch
+from torch import nn
+import torch.nn.functional as F
+from transformers import T5EncoderModel
+from transformers.modeling_outputs import (
+    BaseModelOutput,
+)
+from .configuration_codet5p_embedding import CodeT5pEmbeddingConfig
+
+
+class CodeT5pEmbeddingModel(T5EncoderModel):
+    config_class = CodeT5pEmbeddingConfig
+
+    authorized_missing_keys = [
+        r"encoder.embed_tokens.weight",
+    ]
+
+    def __init__(self, config: CodeT5pEmbeddingConfig):
+        super().__init__(config)
+        self.proj = nn.Linear(config.d_model, config.embed_dim)
+
+    def forward(
+            self,
+            input_ids: Optional[torch.LongTensor] = None,
+            attention_mask: Optional[torch.FloatTensor] = None,
+            head_mask: Optional[torch.FloatTensor] = None,
+            inputs_embeds: Optional[torch.FloatTensor] = None,
+            output_attentions: Optional[bool] = None,
+            output_hidden_states: Optional[bool] = None,
+            return_dict: Optional[bool] = None,
+    ) -> Union[Tuple[torch.FloatTensor], BaseModelOutput]:
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        encoder_outputs = self.encoder(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            inputs_embeds=inputs_embeds,
+            head_mask=head_mask,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+
+        embedding = F.normalize(self.proj(encoder_outputs.last_hidden_state[:, 0, :]), dim=-1)
+
+        return embedding

codet5p/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:097d1bd8c5df11eb82aa5b750d208eee17d570babf94c77158279dc992c6829b
+size 439257889

codet5p/special_tokens_map.json

@@ -0,0 +1,56 @@
+{
+  "additional_special_tokens": [
+    "[ENC]",
+    "[TDEC]",
+    "[CDEC]"
+  ],
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

codet5p/tokenizer_config.json

@@ -0,0 +1,64 @@
+{
+  "add_prefix_space": false,
+  "bos_token": {
+    "__type": "AddedToken",
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": {
+    "__type": "AddedToken",
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "__type": "AddedToken",
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "errors": "replace",
+  "mask_token": {
+    "__type": "AddedToken",
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "model_max_length": 512,
+  "pad_token": {
+    "__type": "AddedToken",
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "__type": "AddedToken",
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "tokenizer_class": "RobertaTokenizer",
+  "trim_offsets": true,
+  "unk_token": {
+    "__type": "AddedToken",
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}