Feature Extraction
sentence-transformers
PyTorch
Safetensors
Transformers
English
bert
fill-mask
learned sparse
opensearch
retrieval
passage-retrieval
document-expansion
bag-of-words
sparse-encoder
sparse
asymmetric
inference-free
splade
text-embeddings-inference
Instructions to use opensearch-project/opensearch-neural-sparse-encoding-doc-v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use opensearch-project/opensearch-neural-sparse-encoding-doc-v1 with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("opensearch-project/opensearch-neural-sparse-encoding-doc-v1") sentences = [ "The weather is lovely today.", "It's so sunny outside!", "He drove to the stadium." ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] - Transformers
How to use opensearch-project/opensearch-neural-sparse-encoding-doc-v1 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="opensearch-project/opensearch-neural-sparse-encoding-doc-v1")# Load model directly from transformers import AutoTokenizer, AutoModelForMaskedLM tokenizer = AutoTokenizer.from_pretrained("opensearch-project/opensearch-neural-sparse-encoding-doc-v1") model = AutoModelForMaskedLM.from_pretrained("opensearch-project/opensearch-neural-sparse-encoding-doc-v1") - Notebooks
- Google Colab
- Kaggle
Adapt to the new naming of IDF in sentence_transformers
#6
by arthurbresnu - opened
Hello!
Apologies for the second PR, this one reflects the renaming of the IDF module to SparseStaticEmbedding.
You can test it just like the previous one. It should remain compatible with the current sparse_implementation branch.
Thanks!
cc @tomaarsen
Arthur BRESNU
arthurbresnu changed pull request status to open
zhichao-geng changed pull request status to merged