sentence-transformers How to use lw2134/policy_gte_large with sentence-transformers:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("lw2134/policy_gte_large", trust_remote_code=True)
sentences = [
"1. What measures should be taken to avoid \"mission creep\" when identifying goals for data collection? \n2. Why is it important to assess new privacy risks before using collected data in a different context?",
"narrow identified goals, to avoid \"mission creep.\" Anticipated data collection should be determined to be \nstrictly necessary to the identified goals and should be minimized as much as possible. Data collected based on \nthese identified goals and for a specific context should not be used in a different context without assessing for \nnew privacy risks and implementing appropriate mitigation measures, which may include express consent.",
"Promoting the Use of Trustworthy Artificial Intelligence in the Federal Government (December 2020). \nThis white paper recognizes that national security (which includes certain law enforcement and \nhomeland security activities) and defense activities are of increased sensitivity and interest to our nation’s \nadversaries and are often subject to special requirements, such as those governing classified information and \nother protected data. Such activities require alternative, compatible safeguards through existing policies that",
"establish and maintain the capabilities that will allow individuals to use their own automated systems to help \nthem make consent, access, and control decisions in a complex data ecosystem. Capabilities include machine \nreadable data, standardized data formats, metadata or tags for expressing data processing permissions and \npreferences and data provenance and lineage, context of use and access-specific tags, and training models for \nassessing privacy risk."
]
embeddings = model.encode(sentences)
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [4, 4] 
