Update README.md

README.md

@@ -1,10 +1,18 @@
 ---
-pipeline_tag: sentence-similarity
 tags:
 - sentence-transformers
 - feature-extraction
 - sentence-similarity
-
+license: apache-2.0
+datasets:
+- wikimedia/wikipedia
+- SiberiaSoft/SiberianPersonaChat-2
+language:
+- ru
+- en
+metrics:
+- mse
+library_name: transformers
 ---
 
 # FractalGPT/SbertSVDDistil
@@ -17,35 +25,105 @@ This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentence
 
 Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
 
+* [Run example in Collab](https://colab.research.google.com/drive/1m3fyh632htPs9UiEu4_AkQfrUtjDqIQq)
+
+
+
 ```
-pip install -U sentence-transformers
+pip install -U sentence-transformers -q
 ```
 
 Then you can use the model like this:
 
 ```python
+from transformers import BertModel
+import numpy as np
+import torch
+from torch import nn
+from sentence_transformers import SentenceTransformer, util
+```
+
+```python
+class SVDLinearLayer(nn.Module):
+    def __init__(self, in_features, out_features, h_dim):
+        super(SVDLinearLayer, self).__init__()
+        self.encoder = nn.Linear(in_features, h_dim, bias=False)
+        self.decoder = nn.Linear(h_dim, out_features, bias=True)
+
+
+    def forward(self, x):
+        x = self.encoder(x)
+        x = self.decoder(x)
+        return x
+
+
+class SVDBertModel(BertModel):
+    def __init__(self, config):
+        super(SVDBertModel, self).__init__(config)
+
+        for i, layer in enumerate(self.encoder.layer):
+            intermediate_size = layer.intermediate.dense.out_features
+            output_size = layer.output.dense.out_features
+
+            if i > 0:
+              layer.intermediate.dense = SVDLinearLayer(layer.intermediate.dense.in_features, intermediate_size, 5)
+              layer.output.dense = SVDLinearLayer(layer.output.dense.in_features, output_size, 5)
+            else:
+              layer.intermediate.dense = nn.Linear(layer.intermediate.dense.in_features, intermediate_size, True)
+              layer.output.dense = nn.Linear(layer.output.dense.in_features, output_size, True)
+
+
+def sim(texts_1, texts_2):
+  embedding_1 = model.encode(texts_1)
+  embedding_2 = model.encode(texts_2)
+  s = util.pytorch_cos_sim(embedding_1, embedding_2)
+  return s.detach().numpy()
+```
+
+```python
+path = 'FractalGPT/SbertSVDDistil'
+model = SentenceTransformer(path)
+model[0].auto_model = SVDBertModel.from_pretrained(path) # Правильная загрузка слоев с SVD
+```
 
+```python
+sim(["I'm happy",
+     "Transistor (English transistor, an acronym invented in 1947 - from the English transfer + English resistor [1] - for a device for passing current through a resistance), semiconductor triode - an electronic component made of semiconductor material, capable of controlling a significant current into the output with a small input signal circuits, which allows it to be used to amplify, generate, switch and convert electrical signals. Currently, the transistor is the basis of the circuit design of the vast majority of electronic devices and integrated circuits.",
+     "That is a happy dog",
+     "Today is a sunny day",
+     "An electric vacuum triode, or simply triode, is an electronic tube that allows an input signal to control the current in an electrical circuit. It has three electrodes: a thermionic cathode (direct or indirectly heated), an anode and one control grid."],
+
+      ["Я счастлив",
+       "Транзи́стор (англ. transistor, придуманный в 1947 году акроним — от англ. transfer + англ. resistor[1] — для устройства пропуска тока через сопротивление), полупроводнико́вый трио́д — электронный компонент из полупроводникового материала, способный небольшим входным сигналом управлять значительным током в выходной цепи, что позволяет использовать его для усиления, генерирования, коммутации и преобразования электрических сигналов. В настоящее время транзистор является основой схемотехники подавляющего большинства электронных устройств и интегральных микросхем.",
+       "Это счастливая собака",
+       "Сегодня солнечный день",
+       "Эле́ктрова́куумный трио́д, или просто трио́д, — электронная лампа, позволяющая входным сигналом управлять током в электрической цепи. Имеет три электрода: термоэлектронный катод (прямого или косвенного накала), анод и одну управляющую сетку."])
 ```
 
 
+```
+array([[ 0.92624545, -0.1081745 ,  0.5569258 ,  0.4006917 ,  0.0524814 ],
+       [-0.10137352,  0.9214004 , -0.0590867 , -0.05579955,  0.6043041 ],
+       [ 0.56128216, -0.08206842,  0.9496383 ,  0.23291808,  0.03726077],
+       [ 0.34002465, -0.05840789,  0.240945  ,  0.9276679 ,  0.09676868],
+       [-0.01571994,  0.60077745, -0.00638374, -0.02819303,  0.8434113 ]],
+      dtype=float32)
+```
 
-## Evaluation Results
 
-<!--- Describe how your model was evaluated -->
 
-For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name=FractalGPT/SberDistil)
+## Training
 
+* Base model [FractalGPT/SbertDistil](https://huggingface.co/FractalGPT/SbertDistil).
 
+* Log of additional training after decomposition.
+   <img src="https://github.com/FractalGPT/ModelEmbedderDistilation/blob/main/SbertSVDDistil/Train/en.JPG?raw=true" width=700 />
 
 ## Full Model Architecture
 ```
 SentenceTransformer(
-  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
+  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: SVDBertModel 
   (1): Pooling({'word_embedding_dimension': 312, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
   (2): Dense({'in_features': 312, 'out_features': 384, 'bias': True, 'activation_function': 'torch.nn.modules.linear.Identity'})
 )
 ```
-
-## Citing & Authors
-
-<!--- Describe where people can find more information -->