Have a plan to optimize long context window?
Hi, guys. In my scenario， I need at least 4k context window and the model is just optimized for 512 token. Do you have a plan to optimize the model long context embedding capability?
Hello! Thank you for your interest in our model and for reaching out.
To answer your question: yes, expanding the context window is on our immediate roadmap. In our upcoming release, we are planning to support at least a 4k context window, and we are actively exploring the possibility of pushing it up to 8k, depending on our computational resources and support capabilities.
If you need to process long documents right now:
We highly recommend using a document chunking strategy with a 10-15% overlap to maintain semantic continuity.
To make this seamless, you can use our custom vector database, Hyperspace DB. It is specifically built to handle Hyperbolic vectors and has native, out-of-the-box support for our embedding models. It also handles the text chunking and overlapping process automatically, which should perfectly cover your 4k+ context requirements in the meantime.
Hope this helps with your scenario! Let us know if you need any help setting it up.