Upload 10 files

1_Pooling/config.json

@@ -0,0 +1,7 @@
+{
+  "word_embedding_dimension": 312,
+  "pooling_mode_cls_token": true,
+  "pooling_mode_mean_tokens": false,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false
+}

README.md

@@ -1,3 +1,133 @@
 ---
+language:
+- ru
+
+pipeline_tag: sentence-similarity
+
+tags:
+- russian
+- pretraining
+- embeddings
+- tiny
+- feature-extraction
+- sentence-similarity
+- sentence-transformers
+- transformers
+
+datasets:
+  - IlyaGusev/gazeta
+  - zloelias/lenta-ru
+
 license: mit
+base_model: cointegrated/rubert-tiny2
+
 ---
+
+## Базовый Bert для Semantic text similarity (STS) на CPU
+
+Базовая модель BERT для расчетов компактных эмбедингов предложений на русском языке. Модель основана на [cointegrated/rubert-tiny2](https://huggingface.co/cointegrated/rubert-tiny2) - имеет аналогичные размеры контекста (2048) и ембединга (312), количество слоев увеличено с 3 до 7.
+
+На STS и близких задачах (PI, NLI, SA, TI) для русского языка превосходит по качеству [sergeyzh/rubert-tiny-sts](https://huggingface.co/sergeyzh/rubert-tiny-sts). Для работы с контекстом свыше 512 токенов требует дообучения под целевой домен.
+
+## Лучшая модель для использования в составе RAG LLMs при инференсе на CPU:
+- отличный метрики на задачах STS, PI, NLI обеспечивают высокое качество при нечетких запросах;
+- средние показатели на задачах SA, TI снижают влияние авторского стиля и личного отношения автора на ембединг;
+- высокая скорость работы на CPU (> 500 предложений в секунду) позволяет легко расширять базу текстовых документов;
+- пониженная размерность эмбединга (312) ускоряет дальнейшую работу алгоритмов knn при поиске соответствий;
+- совместимость с [SentenceTransformer](https://github.com/UKPLab/sentence-transformers). 
+
+## Использование модели с библиотекой `transformers`:
+
+```python
+# pip install transformers sentencepiece
+import torch
+from transformers import AutoTokenizer, AutoModel
+tokenizer = AutoTokenizer.from_pretrained("sergeyzh/rubert-mini-sts")
+model = AutoModel.from_pretrained("sergeyzh/rubert-mini-sts")
+# model.cuda()  # uncomment it if you have a GPU
+
+def embed_bert_cls(text, model, tokenizer):
+    t = tokenizer(text, padding=True, truncation=True, return_tensors='pt')
+    with torch.no_grad():
+        model_output = model(**{k: v.to(model.device) for k, v in t.items()})
+    embeddings = model_output.last_hidden_state[:, 0, :]
+    embeddings = torch.nn.functional.normalize(embeddings)
+    return embeddings[0].cpu().numpy()
+
+print(embed_bert_cls('привет мир', model, tokenizer).shape)
+# (312,)
+```
+
+## Использование с `sentence_transformers`:
+```Python
+from sentence_transformers import SentenceTransformer, util
+
+model = SentenceTransformer('sergeyzh/rubert-mini-sts')
+
+sentences = ["привет мир", "hello world", "здравствуй вселенная"]
+embeddings = model.encode(sentences)
+print(util.dot_score(embeddings, embeddings))
+```
+
+## Метрики
+Оценки модели на бенчмарке [encodechka](https://github.com/avidale/encodechka):
+
+| Модель                           | STS       | PI        | NLI       | SA        | TI        |
+|:---------------------------------|:---------:|:---------:|:---------:|:---------:|:---------:|
+| [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large)   |   0.862   |   0.727   |   0.473   |   0.810   |   0.979   |
+| [sergeyzh/LaBSE-ru-sts](https://huggingface.co/sergeyzh/LaBSE-ru-sts)       |    0.845   |   0.737   |   0.481   |   0.805   |   0.957   |
+| **sergeyzh/rubert-mini-sts**     | **0.815** | **0.723** | **0.477** | **0.791** | **0.949** |
+| [sergeyzh/rubert-tiny-sts](https://huggingface.co/sergeyzh/rubert-tiny-sts)     |   0.797   |   0.702   |   0.453   |   0.778   |   0.946   |
+| [Tochka-AI/ruRoPEBert-e5-base-512](https://huggingface.co/Tochka-AI/ruRoPEBert-e5-base-512) |   0.793   |   0.704   |   0.457   |   0.803   |   0.970   |
+| [cointegrated/LaBSE-en-ru](https://huggingface.co/cointegrated/LaBSE-en-ru)         |   0.794   |   0.659   |   0.431   |   0.761   |   0.946   |
+| [cointegrated/rubert-tiny2](https://huggingface.co/cointegrated/rubert-tiny2)        |   0.750   |   0.651   |   0.417   |   0.737   |   0.937   |
+
+**Задачи:**
+
+- Semantic text similarity (**STS**);
+- Paraphrase identification (**PI**);
+- Natural language inference (**NLI**);
+- Sentiment analysis (**SA**);
+- Toxicity identification (**TI**).
+
+## Быстродействие и размеры
+
+На бенчмарке [encodechka](https://github.com/avidale/encodechka):
+
+| Модель                           | CPU       | GPU       | size      | dim       | n_ctx     | n_vocab   |
+|:---------------------------------|----------:|----------:|----------:|----------:|----------:|----------:|
+| [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large)   | 149.026   |  15.629   |   2136    |   1024    |    514    |  250002   |
+| [sergeyzh/LaBSE-ru-sts](https://huggingface.co/sergeyzh/LaBSE-ru-sts)      |  42.835   |   8.561   |    490    |    768    |    512    |   55083    |
+| **sergeyzh/rubert-mini-sts**     | **6.417** | **5.517** |  **123**  |  **312**  |  **2048** | **83828** |
+| [sergeyzh/rubert-tiny-sts](https://huggingface.co/sergeyzh/rubert-tiny-sts)     |   3.208   |   3.379   |    111    |    312    |    2048   |   83828   |
+| [Tochka-AI/ruRoPEBert-e5-base-512](https://huggingface.co/Tochka-AI/ruRoPEBert-e5-base-512) |  43.314   |   9.338   |    532    |    768    |    512    |   69382   |
+| [cointegrated/LaBSE-en-ru](https://huggingface.co/cointegrated/LaBSE-en-ru)         |  42.867   |   8.549   |    490    |    768    |    512    |   55083   |
+| [cointegrated/rubert-tiny2](https://huggingface.co/cointegrated/rubert-tiny2)        |   3.212   |   3.384   |    111    |    312    |    2048   |   83828   |
+
+
+
+При использовании батчей с `sentence_transformers`:
+
+```python
+from sentence_transformers import SentenceTransformer
+
+model_name = 'sergeyzh/rubert-mini-sts'
+model = SentenceTransformer(model_name, device='cpu')
+sentences = ["Тест быстродействия на CPU Ryzen 7 3800X: batch = 500"] * 500
+%timeit -n 5 -r 3 model.encode(sentences)
+
+# 927 ms ± 7.88 ms per loop (mean ± std. dev. of 3 runs, 5 loops each)
+# 500/0.927 = 539 snt/s
+
+model = SentenceTransformer(model_name, device='cuda')
+sentences = ["Тест быстродействия на GPU RTX 3060: batch = 5000"] * 5000
+%timeit -n 5 -r 3 model.encode(sentences)
+
+# 964 ms ± 26.8 ms per loop (mean ± std. dev. of 3 runs, 5 loops each)
+# 5000/0.964 = 5187 snt/s
+```
+
+
+## Связанные ресурсы
+Вопросы использования модели обсуждаются в [русскоязычном чате NLP](https://t.me/natural_language_processing).
+

config.json

@@ -0,0 +1,27 @@
+{
+  "_name_or_path": "sergeyzh/rubert-mini-sts",
+  "architectures": [
+    "BertForPreTraining"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "emb_size": 312,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 312,
+  "initializer_range": 0.02,
+  "intermediate_size": 600,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 2048,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 7,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.38.1",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 83828
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d03d7f78677afbc28c3958138b6a0757ed7e53d37d0aa3535b898cbcfd9da2d
+size 129795584

modules.json

@@ -0,0 +1,20 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_Normalize",
+    "type": "sentence_transformers.models.Normalize"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 2048,
+  "do_lower_case": false
+}

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,57 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": false,
+  "mask_token": "[MASK]",
+  "model_max_length": 6144,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}