README.md

metadata

title: Semantic Cache for LLMs
emoji: ⚡
colorFrom: yellow
colorTo: red
sdk: docker
app_port: 7860
pinned: false
short_description: Embedding-keyed LLM response cache

Semantic Cache for LLM Responses

Caches LLM answers by prompt embedding: a semantically similar prompt returns the cached response instantly, skipping the LLM call (saving latency and token cost). Hybrid exact-hash + cosine-similarity lookup with a tunable threshold, LRU eviction, and live hit-rate / savings metrics. Runs on free OpenRouter models.

Source: https://github.com/saiteja007-mv/semantic-cache