Terrasyncra / SYSTEM_OVERVIEW.md
nexusbert's picture
push all
e03d777
# TerraSyncra AI – Product & System Overview
## 1. Product Introduction
**TerraSyncra** is a multilingual agricultural intelligence agent designed specifically for Nigerian (and African) farmers. It provides comprehensive agricultural support through AI-powered assistance.
**Key Capabilities:**
- **Agricultural Q&A**: Answers questions about crops, livestock, soil, weather, pests, and diseases in multiple languages
- **Soil Analysis**: Provides expert soil health assessments from lab reports and field data using Gemini 3 Flash
- **Disease Detection**: Identifies plant and animal diseases from images, text descriptions, or voice input using Gemini 2.5 Flash
- **Live Agricultural Updates**: Delivers real-time weather information and agricultural news through RAG (Retrieval-Augmented Generation)
- **Live Voice Interaction**: Supports real-time voice conversations via WebSocket in local languages (Igbo, Hausa, Yoruba, English)
**Developer**: Ifeanyi Amogu Shalom
**Target Users**: Farmers, agronomists, agricultural extension officers, and agricultural support workers in Nigeria and similar contexts
---
## 2. Problem Statement
Nigerian smallholder farmers face significant challenges:
### 2.1 Limited Access to Agricultural Experts
- **Scarcity of agronomists and veterinarians** relative to the large farming population
- **Geographic barriers** preventing farmers from accessing expert advice
- **High consultation costs** that many smallholder farmers cannot afford
- **Long waiting times** for professional consultations, especially during critical periods (disease outbreaks, planting seasons)
### 2.2 Language Barriers
- Most agricultural information and resources are in **English**, while many farmers primarily speak **Hausa, Igbo, or Yoruba**
- **Technical terminology** is not easily accessible in local languages
- **Translation services** are often unavailable or unreliable
### 2.3 Fragmented Information Sources
- Weather data, soil reports, disease information, and market prices are scattered across different platforms
- **No unified system** to integrate and interpret multiple data sources
- **Information overload** without proper context or prioritization
### 2.4 Time-Sensitive Decision Making
- **Disease outbreaks** require immediate identification and treatment
- **Weather changes** affect planting, harvesting, and irrigation decisions
- **Pest attacks** can devastate crops if not addressed quickly
- **Delayed responses** lead to significant economic losses
### 2.5 Solution Approach
TerraSyncra addresses these challenges by providing:
- **Fast, AI-powered responses** available 24/7
- **Multilingual support** (English, Igbo, Hausa, Yoruba)
- **Integrated intelligence** combining expert models, RAG, and live data
- **Accessible interface** via text, voice, and image inputs
- **Professional consultation reminders** to ensure farmers seek expert confirmation when needed
---
## 3. System Architecture & Request Flows
### 3.1 General Agricultural Q&A – `POST /ask`
**Step-by-Step Process:**
1. **Input Reception**
- User sends `query` (text) with optional `session_id` for conversation continuity
2. **Language Detection**
- FastText model (`facebook/fasttext-language-identification`) detects input language
- Supports: English, Igbo, Hausa, Yoruba
3. **Translation (if needed)**
- If language ≠ English, translates to English using NLLB (`drrobot9/nllb-ig-yo-ha-finetuned`)
- Preserves original language for back-translation
4. **Intent Detection**
- Classifies query into categories:
- **Weather question**: Requests weather information (with/without Nigerian state)
- **Live update**: Requests current agricultural news or updates
- **Normal question**: General agricultural Q&A
- **Low confidence**: Falls back to RAG when intent is unclear
5. **Context Building**
- **Weather intent**: Calls WeatherAPI for state-specific weather data, embeds summary into context
- **Live update intent**: Queries live FAISS vectorstore index for latest agricultural documents
- **Low confidence**: Falls back to static FAISS index for safer, more general responses
6. **Conversation Memory**
- Loads per-session history from `MemoryStore` (TTL cache, 1-hour expiration)
- Trims to `MAX_HISTORY_MESSAGES` (default: 30) to prevent context overflow
7. **Expert Model Generation**
- Uses **Qwen/Qwen1.5-1.8B** (finetuned for Nigerian agriculture)
- Loaded lazily via `model_manager` (CPU-optimized, first-use loading)
- Builds chat messages: system prompt + conversation history + current user message + context
- System prompt restricts responses to **agriculture/farming topics only**
- Generates bounded-length answer (reduced token limit: 400 tokens for general, 256 for weather)
- Cleans response to remove any "Human: / Assistant:" style example continuations
8. **Back-Translation**
- If original language ≠ English, translates answer back to user's language using NLLB
9. **Response**
- Returns JSON: `{ query, answer, session_id, detected_language }`
**Safety & Focus:**
- System prompt enforces agriculture-only topic handling
- Unrelated questions are redirected back to farming topics
- Response cleaning prevents off-topic example continuations
---
### 3.2 Soil Analysis – `POST /analyze-soil`
**Step-by-Step Process:**
1. **Input Reception**
- `report_data`: Text description of soil report or lab results (required)
- Optional fields: `location`, `crop_type`, `field_size`, `previous_crops`, `additional_notes`
2. **Agent Processing**
- `soil_agent.analyze_soil()` builds comprehensive prompt with:
- Soil report data
- Field information (location, crop type, size, history)
- Regional context (Nigerian states, climate patterns)
3. **Gemini API Call**
- Model: `GEMINI_SOIL_MODEL = "gemini-3-flash-preview"`
- Prompt style: Brief, direct, actionable
- Focuses on:
- Current soil condition (short summary)
- Key nutrient issues (deficiencies or excesses)
- 1–3 best crops for this soil type
- Clear fertilizer and amendment recommendations
- Simple soil improvement steps
4. **Output**
- JSON response: `{ success, analysis, model_used }`
**Important Note:**
> Soil analysis is **advisory only** – not a formal agronomy diagnosis. The UI should encourage farmers to confirm with a local agronomist or extension officer for critical decisions.
---
### 3.3 Disease Detection
#### 3.3.1 Image-Based Detection – `POST /detect-disease-image`
**Step-by-Step Process:**
1. **Input Reception**
- Image file (JPEG, PNG, etc.)
- Optional `query`: Text description or question
2. **Agent Processing**
- `disease_agent.classify_disease_from_image()` processes:
- Image bytes + MIME type
- User query (if provided)
- Builds structured prompt for Gemini
3. **Gemini API Call**
- Model: `GEMINI_DISEASE_MODEL = "gemini-2.5-flash"`
- Prompt instructs Gemini to provide:
- Disease name (scientific + common name) in 1 short line
- **Threat level: Low / Moderate / High / Uncertain** (MANDATORY)
- 2–3 key symptoms visible in image
- 2–3 clear treatment steps (bullets)
- 1–2 simple prevention tips
- Brief, direct language with short sentences
4. **Backend Safety Enforcement**
- Backend **always appends** disclaimer:
> "IMPORTANT: This threat level is an estimate based only on the image/description. For an accurate diagnosis and treatment plan, please consult a qualified agronomist, veterinary doctor, or local agricultural extension officer."
5. **Output**
- JSON response: `{ success, classification, model_used, input_type }`
#### 3.3.2 Text/Voice-Based Detection – `POST /detect-disease-text`
**Step-by-Step Process:**
1. **Input Reception**
- `description`: Text description of disease symptoms or condition
- `language`: Language code (en, ig, ha, yo)
2. **Agent Processing**
- `disease_agent.classify_disease_from_text()` processes:
- Text description
- Language context
- Builds structured prompt for Gemini
3. **Gemini API Call**
- Same model and prompt structure as image-based detection
- Threat level assessment based on described symptoms
4. **Backend Safety Enforcement**
- Same disclaimer appended as image-based detection
5. **Output**
- JSON response: `{ success, classification, model_used, input_type }`
**Threat Level Guidelines:**
- **Low**: Mild or early-stage issue, unlikely to cause major losses if addressed soon
- **Moderate**: Noticeable risk that can reduce yield/health if not treated
- **High**: Serious or fast-spreading issue that can cause major losses or death (use cautiously, only when clearly severe)
- **Uncertain**: Insufficient or ambiguous data; model cannot safely rate risk (encouraged when not confident)
---
### 3.4 Live Voice Interaction – `WS /live-voice` & `POST /live-voice-start`
**Step-by-Step Process:**
1. **WebSocket Connection**
- Client connects to `/live-voice` endpoint
- Optional: Send image as JSON (base64 encoded) at session start
- Audio chunks streamed as raw PCM bytes (16kHz, mono, 16-bit)
2. **Agent Processing**
- `live_voice_agent.handle_live_voice_websocket()` manages:
- WebSocket connection lifecycle
- Image context (if provided)
- Audio streaming to Gemini Live API
- Audio response streaming back to client
3. **Gemini Live API**
- Model: `gemini-2.5-flash` via Gemini Live API
- System prompt: Brief, clear, focused on "what to do next" (2–4 key steps)
- Supports: Disease detection, soil analysis, general farming, weather
- Prefers short sentences and bullet points
4. **Response Streaming**
- Audio responses streamed back as PCM bytes
- Optional JSON messages for status/transcripts
5. **Safety Expectations**
- Same professional advice principle applies
- Frontends should display clear "not a replacement for a professional" banner
---
## 4. Technologies Used
### 4.1 Backend Framework & Infrastructure
- **FastAPI**: Modern Python web framework for building REST APIs and WebSocket endpoints
- **Uvicorn**: ASGI server for running FastAPI applications
- **Python 3.10**: Programming language
- **Docker**: Containerization for deployment
- **Hugging Face Spaces**: Deployment platform (Docker runtime, CPU-only environment)
### 4.2 Core Language Models
#### 4.2.1 Expert Model: Qwen/Qwen1.5-1.8B
- **Model**: `Qwen/Qwen1.5-1.8B` (via Hugging Face Transformers)
- **Purpose**: Primary agricultural Q&A and conversation
- **Specialization**: **Finetuned/specialized** for Nigerian agricultural context through:
- Custom system prompts focused on Nigerian farming practices
- Domain-specific training data integration
- Response formatting optimized for agricultural advice
- **Optimization**:
- Lazy loading via `model_manager` (loads on first use)
- CPU-optimized inference (float32, device_map="cpu")
- Reduced token limits to prevent over-generation
#### 4.2.2 Gemini Models (Google AI)
- **google-genai**: Official Python client for Google's Gemini API
- **gemini-3-flash-preview**: Used for soil analysis
- **gemini-2.5-flash**: Used for disease detection and live voice interaction
- **API Version**: v1alpha for advanced features (disease detection, live voice)
### 4.3 Retrieval-Augmented Generation (RAG)
- **LangChain**: Framework for building LLM applications
- **LangChain Community**: Community integrations and tools
- **SentenceTransformers**:
- Model: `paraphrase-multilingual-MiniLM-L12-v2`
- Purpose: Text embeddings for semantic search
- **FAISS (Facebook AI Similarity Search)**:
- Vector database for efficient similarity search
- Two indices: Static (general knowledge) and Live (current updates)
- **APScheduler**: Background job scheduler for periodic RAG updates
### 4.4 Language Processing
- **FastText**:
- Model: `facebook/fasttext-language-identification`
- Purpose: Language detection (English, Igbo, Hausa, Yoruba)
- **NLLB (No Language Left Behind)**:
- Model: `drrobot9/nllb-ig-yo-ha-finetuned`
- Purpose: Translation between English and Nigerian languages (Hausa, Igbo, Yoruba)
- Bidirectional translation support
### 4.5 External APIs & Data Sources
- **WeatherAPI**:
- Provides state-level weather data for Nigerian states
- Real-time weather information integration
- **AgroNigeria / HarvestPlus**:
- Agricultural news feeds for RAG updates
- News scraping and processing
### 4.6 Additional Libraries
- **transformers**: Hugging Face library for loading and using transformer models
- **torch**: PyTorch (CPU-optimized version)
- **numpy**: Numerical computing
- **requests**: HTTP library for API calls
- **beautifulsoup4**: Web scraping for news aggregation
- **python-multipart**: File upload support for FastAPI
- **python-dotenv**: Environment variable management
---
## 5. Threat Level & Safety Policy
### 5.1 Domain Scope
- **Plant and animal diseases only****NOT human health**
- Focuses on agricultural and veterinary contexts
- Does not provide medical advice for humans
### 5.2 Threat Level Categories
#### Low
- **Definition**: Mild or early-stage issue, unlikely to cause major losses if addressed soon
- **Characteristics**:
- Localized symptoms
- Slow progression
- Easily manageable with standard treatments
- **Example**: Minor leaf spots, early nutrient deficiency
#### Moderate
- **Definition**: Noticeable risk that can reduce yield/health if not treated
- **Characteristics**:
- Moderate spread or impact
- Requires timely intervention
- Can cause economic losses if ignored
- **Example**: Moderate pest infestation, developing fungal infection
#### High
- **Definition**: Serious or fast-spreading issue that can cause major losses or death
- **Characteristics**:
- Rapid spread or severe symptoms
- High potential for significant economic impact
- May require immediate professional intervention
- **Example**: Severe bacterial blight, fast-spreading viral disease
- **Usage Caution**: Only assigned when signs are **clearly severe** or fast-spreading
#### Uncertain
- **Definition**: Insufficient or ambiguous data; model cannot safely rate risk
- **Characteristics**:
- Unclear symptoms
- Multiple possible diagnoses
- Poor image quality or vague description
- **Usage**: Encouraged when model is not confident – **better to be uncertain than wrong**
### 5.3 Accuracy & Caution Approach
**Threat Level Assessment:**
- Based **only** on image + description – **no lab tests or physical examination**
- Prompts instruct Gemini to be **conservative and cautious**
- Model encouraged to use `Uncertain` when not clearly sure
- Final responses always embed a strong "consult professionals" reminder
**Professional Consultation Reminder:**
- Backend **always appends** disclaimer to disease detection responses
- Frontends should visually emphasize: "This is not a medical/veterinary/agronomic diagnosis"
- System is a **decision-support tool**, not a definitive diagnostic engine
**Important Note:**
> **This system is a decision-support tool, not a definitive diagnosis engine.**
> All disease/threat outputs must be treated as preliminary guidance only.
> Farmers should always consult qualified professionals for critical decisions.
---
## 6. Limitations & Issues Faced
### 6.1 Diagnostic Limitations
#### Input Quality Dependencies
- **Image Quality**: Blurry, poorly lit, or low-resolution images reduce accuracy
- **Description Clarity**: Vague or incomplete symptom descriptions limit diagnostic precision
- **Context Missing**: Lack of field history, crop variety, or environmental conditions affects recommendations
#### Inherent Limitations
- **No Physical Examination**: Cannot inspect internal plant structures or perform lab tests
- **No Real-Time Monitoring**: Cannot track disease progression over time
- **Regional Variations**: Some regional diseases may be under-represented in training data
- **Seasonal Factors**: Disease presentation may vary by season, which may not always be captured
### 6.2 Language & Translation Challenges
#### Translation Accuracy
- **NLLB Limitations**: Can misread slang, mixed-language (e.g., Pidgin + Hausa), or regional dialects
- **Technical Terminology**: Agricultural terms may not have direct translations, leading to approximations
- **Context Loss**: Subtle meaning can be lost across translation steps (user language → English → user language)
#### Language Detection
- **FastText Edge Cases**: May misclassify mixed-language inputs or code-switching
- **Dialect Variations**: Regional variations within languages may not be fully captured
### 6.3 Model Behavior Issues
#### Hallucination Risk
- **Qwen/Gemini Limitations**: Can generate confident but incorrect answers
- **Mitigations Applied**:
- Stricter system prompts with domain restrictions
- Shorter output limits (400 tokens for general, 256 for weather)
- Response cleaning to remove example continuations
- Topic redirection for unrelated questions
- **Not Bulletproof**: Hallucination can still occur, especially for edge cases
#### Response Drift
- **Off-Topic Continuations**: Models may continue with example conversations or unrelated content
- **Mitigation**: Response cleaning logic removes "Human: / Assistant:" patterns and unrelated content
### 6.4 Latency & Compute Constraints
#### First-Request Latency
- **Model Loading**: First Qwen/NLLB call is slower due to model + weights loading on CPU
- **Cold Start**: ~5-10 seconds for first request after deployment
- **Subsequent Requests**: Faster due to cached models in memory
#### CPU-Only Environment
- **Inference Speed**: CPU inference is slower than GPU (acceptable for Hugging Face Spaces CPU tier)
- **Memory Constraints**: Limited RAM requires careful model management (lazy loading, model caching)
### 6.5 External Dependencies
#### WeatherAPI Issues
- **Outages**: WeatherAPI downtime affects weather-related responses
- **Rate Limits**: API quota limits may restrict frequent requests
- **Data Accuracy**: Weather data quality depends on third-party provider
#### News Source Reliability
- **Scraping Fragility**: News sources may change HTML structure, breaking scrapers
- **Update Frequency**: RAG updates are scheduled; failures can cause stale information
- **Content Quality**: News article quality and relevance vary
### 6.6 RAG & Data Freshness
#### Update Scheduling
- **Periodic Updates**: RAG indices updated on schedule (not real-time)
- **Job Failures**: If update job fails, index can lag behind real-world events
- **Index Rebuilding**: Full index rebuilds can be time-consuming
#### Vectorstore Limitations
- **Embedding Quality**: Semantic search quality depends on embedding model performance
- **Retrieval Accuracy**: Retrieved documents may not always be most relevant
- **Context Window**: Limited context window may truncate important information
### 6.7 Deployment & Infrastructure
#### Hugging Face Spaces Constraints
- **CPU-Only**: No GPU acceleration available
- **Memory Limits**: Limited RAM requires optimization (lazy loading, model size reduction)
- **Build Time**: Docker builds can be slow, especially with large dependencies
- **Cold Starts**: Spaces may spin down after inactivity, causing cold start delays
#### Docker Build Issues
- **Dependency Conflicts**: Some Python packages may conflict (e.g., pyaudio requiring system libraries)
- **Build Timeouts**: Long build times may cause deployment failures
- **Cache Management**: Docker layer caching can be inconsistent
---
## 7. Recommended UX & Safety Reminders
### 7.1 Visual Disclaimers
**Always display a clear banner near disease/soil results:**
> "⚠️ **This is AI-generated guidance. Always confirm with a local agronomist, veterinary doctor, or agricultural extension officer before taking major actions.**"
### 7.2 Threat Level Display
- **Visual Highlighting**: Display threat level prominently with color coding:
- 🟢 **Low**: Green
- 🟡 **Moderate**: Yellow
- 🔴 **High**: Red
- ⚪ **Uncertain**: Gray
- **Tooltips**: Provide explanations for each threat level
- **Always Pair with Disclaimer**: Never show threat level without the professional consultation reminder
### 7.3 Call-to-Action Buttons
Provide quick access to professional help:
- **"Contact an Extension Officer"** button/link
- **"Find a Vet/Agronomist Near You"** button/link
- **"Schedule a Consultation"** option (if available)
### 7.4 Response Quality Indicators
- Show **confidence indicators** when available (e.g., "High confidence" vs "Uncertain")
- Display **input quality warnings** (e.g., "Image quality may affect accuracy")
- Provide **feedback mechanisms** for users to report incorrect diagnoses
### 7.5 Language Support
- Clearly indicate **detected language** in responses
- Provide **language switcher** for users to change language preference
- Show **translation quality warnings** if translation may be approximate
---
## 8. System Summary
### 8.1 Problem Addressed
Nigerian smallholder farmers face critical challenges:
- **Limited access to agricultural experts** (agronomists, veterinarians)
- **Language barriers** (most resources in English, farmers speak Hausa/Igbo/Yoruba)
- **Fragmented information sources** (weather, soil, disease data scattered)
- **Time-sensitive decision making** (disease outbreaks, weather changes, pest attacks)
### 8.2 Solution Provided
TerraSyncra combines multiple AI technologies to provide:
- **Fast, 24/7 AI-powered responses** in multiple languages
- **Integrated intelligence**:
- **Finetuned Qwen 1.8B** expert model for agricultural Q&A
- **Gemini 3/2.5 Flash** for soil analysis and disease detection
- **RAG + Weather + News** for live, contextual information
- **CPU-optimized, multilingual backend** (FastAPI on Hugging Face Spaces)
- **Multiple input modalities**: Text, voice, and image support
### 8.3 Safety & Professional Consultation
**Every disease assessment includes:**
- Explicit **Threat level** (Low / Moderate / High / Uncertain)
- Clear **professional consultation reminder**
- Emphasis that threat levels are **estimates**, not definitive diagnoses
### 8.4 Key Technologies
- **Expert Model**: Qwen/Qwen1.5-1.8B (finetuned for Nigerian agriculture)
- **Gemini Models**: gemini-3-flash-preview (soil), gemini-2.5-flash (disease, voice)
- **RAG**: LangChain + FAISS + SentenceTransformers
- **Language Processing**: FastText (detection) + NLLB (translation)
- **Backend**: FastAPI + Uvicorn + Docker
- **Deployment**: Hugging Face Spaces (CPU-optimized)
### 8.5 Developer & Credits
**Developer**: Ifeanyi Amogu Shalom
**Intended Users**: Farmers, agronomists, agricultural extension officers, and agricultural support workers in Nigeria and similar contexts
---
## 9. Future Improvements & Roadmap
### 9.1 Potential Enhancements
- **Model Fine-tuning**: Further fine-tune Qwen on Nigerian agricultural datasets
- **Multi-modal RAG**: Integrate images into RAG for visual similarity search
- **Offline Mode**: Support for offline operation in areas with poor connectivity
- **Mobile App**: Native mobile applications for better user experience
- **Expert Network Integration**: Direct connection to network of agronomists/veterinarians
- **Historical Tracking**: Track disease progression and treatment outcomes over time
### 9.2 Technical Improvements
- **Response Caching**: Cache common queries to reduce latency
- **Model Quantization**: Further optimize models for CPU inference
- **Better Error Handling**: More robust error messages and fallback mechanisms
- **Monitoring & Analytics**: Track system performance and user feedback
---
**Last Updated**: 2026
**Version**: 1.0
**Status**: Production (Hugging Face Spaces)