Will there be a 8192 token input variant of DenseOn?
#3
by jshree - opened
Wondering if there is a future plan to increase the input context window capacity?
Hey @jshree , DenseOn is based on mmBERT which supports inputs up to 8192 tokens, but we set the model context length to 512 during retrieval training. You may bump it to 8192 tokens if you prefer:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("lightonai/DenseOn")
model.max_seq_length = 8192
Nevertheless, I recommend you to use our multilingual dense model https://huggingface.co/lightonai/mDenseOn which was specifically trained for long context text. You may also consider our multilingual multi-vector model https://huggingface.co/lightonai/mLateOn if your use case allows for it, since long context performance (MLDR benchmark) is much stronger for this model.