What is the mode's context length?
^
From what I read, 512 tokens.
I know that is the limit for the .query method, is there no way to use a larger context window? Cant really do much 512 tokens!
True, maybe you could find some better results using sliding window, but not sure what options we have in this model.
I find the benchmarks here a little biased.
Really cool idea to have a lightweight LLM focused on accuracy and params < 1x10^9 though, was experimenting with making a desktop app for RAG use with textbooks but the limited context window killed that idea.
Hey all,
Yes the context window is 512 with the ability to scale up to 1024 (same as flan-t5).
This is a small model designed to run on a CPU- 1k tokens of context runs incredibly slow on a CPU, and so we are investing heavily in setting up RAG systems around teapotllm for fast client inference.
You can check out our library docs here: https://teapotai.com/docs#2-chat-with-retrieval-augmented-generation-rag
Additionally if you want help with any projects, come visit our discord! We've already seen some cool examples of FAQ chatbots and RAG applications.
https://discord.gg/hPxGSn5dST