Upload 6 files

README.md

@@ -0,0 +1,31 @@
+# SOBertLarge
+
+## Model Description
+
+SOBertBase is a 762M parameter BERT models trained on 27 billion tokens of SO data StackOverflow answer and comment text using the Megatron Toolkit. 
+
+SOBert is pre-trained with 19 GB data presented as 15 million samples where each sample contains an entire post and all its corresponding comments. We also include
+all code in each answer so that our model is bimodal in nature. We use a SentencePiece tokenizer trained with BytePair Encoding, which has the benefit over WordPiece of never labeling tokens as “unknown". 
+Additionally, SOBert is trained with a a maximum sequence length of 2048 based on the empirical length distribution of StackOverflow posts and a relatively
+large batch size of 0.5M tokens. A smaller 109 million parameter model can also be found [here](https://huggingface.co/mmukh/SOBertBase) . More details can be found in the paper 
+[Stack Over-Flowing with Results: The Case for Domain-Specific Pre-Training Over One-Size-Fits-All Models](https://arxiv.org/pdf/2306.03268).
+
+#### How to use
+
+```python
+from transformers import AutoTokenizer,AutoModel
+model = AutoModel.from_pretrained(mmukh/SOBertLarge")
+tokenizer = AutoTokenizer.from_pretrained("mmukh/SOBertLarge")
+
+```
+
+### BibTeX entry and citation info
+
+```bibtex
+@article{mukherjee2023stack,
+  title={Stack Over-Flowing with Results: The Case for Domain-Specific Pre-Training Over One-Size-Fits-All Models},
+  author={Mukherjee, Manisha and Hellendoorn, Vincent J},
+  journal={arXiv preprint arXiv:2306.03268},
+  year={2023}
+}
+```

config.json

@@ -0,0 +1,20 @@
+{
+  "attention_probs_dropout_prob": 0.1,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 1536,
+  "initializer_range": 0.02,
+  "intermediate_size": 6144,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 2048,
+  "model_type": "megatron-bert",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "tokenizer_type": "SentencePieceTokenizer",
+  "transformers_version": "4.31.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 50048
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f9478715f1eb7f048c6d9f29a04eb4dafd543553b1102a3be1155aab4513cb52
+size 1524894129

special_tokens_map.json

@@ -0,0 +1 @@
+{}

tokenizer_config.json

@@ -0,0 +1,4 @@
+{
+  "model_max_length": 2048,
+  "tokenizer_class": "PreTrainedTokenizerFast"
+}