Spaces:
Sleeping
Sleeping
File size: 5,170 Bytes
bbb4f78 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 | # `app/llm` – LLM Integration Layer
This module abstracts and implements communication with **local** and **cloud-based** large language models (LLMs) via interchangeable client wrappers.
It defines:
- A common interface for all LLM clients (`LLMClient`)
- A wrapper for Google Gemini API (`ImprovedGeminiClient`)
- A wrapper for Ollama local models (`ImprovedOllamaClient`)
- A sentence transformer embedding model (`embedding_client.py`)
---
## Abstract Base – `llm_client.py`
This file defines the **contract** that all LLM clients must follow.
### `class LLMClient (ABC)`
An abstract base class using Python’s `abc` module.
```python
@abstractmethod
async def generate(system_prompt: str, context: List[dict], temperature: float, max_tokens: int) -> str
```
Every model wrapper must implement this coroutine to generate a response given:
- A system prompt (persona instructions)
- A user/system message context (list of `{role, content}` dicts)
- A temperature (float 0.0–1.0, typically scaled from 0–10)
- A token limit (integer)
---
## Gemini Client – `improved_gemini_client.py`
### Overview
- Communicates with **Google’s Gemini API** via `httpx`
- Dynamically injects the `system_prompt` into the context using `context_manager`
- Uses environment variables for API key and model name (`GEMINI_API_KEY`, `GEMINI_MODEL`)
### Key Features
| Feature | Description |
|--------|-------------|
| Context Prep | Uses `context_manager.prepare_context_for_llm()` to optimize message length |
| Endpoint | `https://generativelanguage.googleapis.com/v1beta/models/{model_name}:generateContent` |
| Content Format | Gemini expects JSON-formatted `contents`, not string prompts |
| Safety Settings | Blocks harmful or explicit content categories |
| Fallback Logic | Returns user-friendly error messages on bad or empty responses |
| Token Limit | `maxOutputTokens` passed explicitly |
### SafetyConfig JSON Example
```json
"safetySettings": [
{"category": "HARM_CATEGORY_HARASSMENT", "threshold": "BLOCK_MEDIUM_AND_ABOVE"},
{"category": "HARM_CATEGORY_HATE_SPEECH", "threshold": "BLOCK_MEDIUM_AND_ABOVE"}
]
```
### Differences from Ollama
- Requires an API key and runs over HTTPS
- Parses deeply nested JSON structures (candidates → content → parts)
- Strict token and safety controls
- More structured response format
---
## Ollama Client – `improved_ollama_client.py`
### Overview
- Interfaces with a **local Ollama model server** (`http://localhost:11434`)
- Sends prompts as raw formatted strings (not JSON "messages")
- Uses `context_manager` to prepare prompt text
### Key Features
| Feature | Description |
|--------|-------------|
| Endpoint | `/api/generate` |
| Payload | Flat prompt string + generation config |
| Cleansing | Strips verbose, inconsistent prefixes or filler |
| Quality Filter | Removes overly verbose or vague responses |
| Robust | Recovers from connection and timeout failures |
### Prompt Payload Example
```json
{
"model": "llama3.2:1b",
"prompt": "System: You are a helpful advisor...\nUser: What is...",
"stream": false,
"options": {
"temperature": 0.4,
"top_p": 0.9,
"top_k": 40,
"num_predict": 300,
"repeat_penalty": 1.1,
"stop": ["Student:", "User:", "Question:"]
}
}
```
### Differences from Gemini
| Area | Gemini | Ollama |
|------|--------|--------|
| Hosting | Cloud API | Local server |
| Format | JSON "messages" | Raw string prompt |
| Safety Filters | Yes | No |
| Token Control | `maxOutputTokens` | `num_predict` |
| Output | Structured parts | Single `response` string |
| Response Cleaning | Minimal | Aggressively stripped of fluff |
| Performance | High-quality, slower | Fast & offline |
---
## Embedding Model – `embedding_client.py`
### Purpose
Provides embedding vectors (used for semantic similarity and document retrieval) using `sentence-transformers`.
### Uses:
- Model: `all-MiniLM-L6-v2` (lightweight + performant)
- Library: `sentence-transformers`
- Function: `get_embedding(text: str) -> List[float]`
```python
embedding = get_embedding("example sentence")
```
### Notes
- This module does **not** use Gemini embeddings (for cost and simplicity)
- Can be upgraded later to use Gemini’s `embedding` endpoint or Ollama-based models with vector support
---
## Environment Variables
| Variable | Description | Example |
|----------|-------------|---------|
| `GEMINI_API_KEY` | API key for Google Gemini | `AIzz123...` |
| `GEMINI_MODEL` | Default Gemini model name | `gemini-2.0-flash` |
| `OLLAMA_BASE_URL` | Local server base URL | `http://localhost:11434` |
---
## Context Management Integration
Both clients use:
```python
context_window = context_manager.prepare_context_for_llm(...)
```
This ensures that:
- Prompt fits within model limits
- Truncation metadata is logged/debugged
- Messages are pre-formatted or optimized per provider
---
## Error Handling
All clients log internal issues and fallback to graceful responses. Each client handles:
- Timeouts (`httpx.TimeoutException`)
- API errors (`httpx.HTTPStatusError`, bad payloads)
- Unexpected failures (fallback strings are returned)
--- |