How to use thanhpham1/Finetuned-all-mpnet-base-v2-with-technical-docs with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("thanhpham1/Finetuned-all-mpnet-base-v2-with-technical-docs") sentences = [ "import subprocess\nzen_of_python = subprocess.check_output([\"python\", \"-c\", \"import this\"])\ncorpus = zen_of_python.split()\n\nnum_partitions = 3\nchunk = len(corpus) // num_partitions\npartitions = [\ncorpus[i * chunk: (i + 1) * chunk] for i in range(num_partitions)\n]\n\nMapping Data#\nTo determine the map phase, we require a map function to use on each document.\nThe output is the pair (word, 1) for every word found in a document.\nFor basic text documents we load as Python strings, the process is as follows:\n\ndef map_function(document):\nfor word in document.lower().split():\nyield word, 1\n\nWe use the apply_map function on a large collection of documents by marking it as a task in Ray using the @ray.remote decorator.\nWhen we call apply_map, we apply it to three sets of document data (num_partitions=3).\nThe apply_map function returns three lists, one for each partition so that Ray can rearrange the results of the map phase and distribute them to the appropriate nodes.\n\nimport ray", "What does the map_function yield for each word in a document?", "What does PBT do differently from traditional hyperparameter tuning methods?", "What is returned by task_with_static_multiple_returns_good in the Actor class?" ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [4, 4]
