Documentation/API_DOCUMENTATION.md

Government Schemes RAG API Documentation (Multilingual)

Overview

FastAPI-based REST API for querying Indian Government Schemes using Retrieval-Augmented Generation (RAG) with support for 13+ Indian languages.

Base URL

http://127.0.0.1:8000

Key Features

• ✅ Multilingual support (13+ Indian languages)
• ✅ Automatic translation (Input & Output)
• ✅ Text-to-Speech capability (optional)
• ✅ RAG-powered intelligent search
• ✅ 3400+ government schemes database

API Endpoints

1. Root Endpoint

GET /

Returns API information, version, and supported languages.

Response:

{
  "message": "Government Schemes RAG API with Multilingual Support",
  "version": "2.0.0",
  "supported_languages": {
    "en": "English",
    "hi": "Hindi",
    "te": "Telugu",
    "ta": "Tamil",
    "ml": "Malayalam",
    "kn": "Kannada",
    "bn": "Bengali",
    "mr": "Marathi",
    "gu": "Gujarati",
    "pa": "Punjabi",
    "ur": "Urdu",
    "or": "Odia",
    "as": "Assamese"
  },
  "endpoints": {
    "POST /query": "Query government schemes with translation support",
    "GET /states": "Get list of Indian states",
    "GET /languages": "Get list of supported languages",
    "GET /health": "Health check"
  }
}

---

2. Health Check

GET /health

Check if the API and RAG system are running properly.

Response:

{
  "status": "healthy",
  "rag_system": "initialized"
}

---

3. Get Supported Languages

GET /languages

Get list of all supported languages for translation.

Response:

{
  "languages": {
    "en": "English",
    "hi": "Hindi",
    "te": "Telugu",
    "ta": "Tamil",
    "ml": "Malayalam",
    "kn": "Kannada",
    "bn": "Bengali",
    "mr": "Marathi",
    "gu": "Gujarati",
    "pa": "Punjabi",
    "ur": "Urdu",
    "or": "Odia",
    "as": "Assamese"
  }
}

---

4. Get States

GET /states

Get list of all Indian states and union territories.

Response:

{
  "states": [
    "All States",
    "Andhra Pradesh",
    "Arunachal Pradesh",
    ...
  ]
}

---

5. Query Schemes (with Multilingual Support)

POST /query

Query government schemes in any supported language. The API automatically translates the input to English, processes it through the RAG system, and returns the answer in the requested language.

Request Body:

{
  "question": "స్కాలర్‌షిప్ల గురించి చెప్పండి",  // Question in any language
  "state": "Telangana",  // Optional
  "language": "te"  // Language code (default: "en")
}

Response:

{
  "answer": "తెలంగాణలో అందుబాటులో ఉన్న స్కాలర్‌షిప్‌ల గురించి...",
  "sources": [
    "Scheme Name: Pre-Matric Scholarship for Backward Class Students...",
    "Scheme Name: Post-Matric Scholarship Scheme...",
    "Scheme Name: Merit-cum-Means Scholarship..."
  ]
}

Note: Audio is NOT automatically generated. Use the /generate-audio endpoint when the user clicks the speaker button.

Translation Flow:

User Question (Telugu) → Translate to English → RAG Processing → 
English Answer → Translate to Telugu → Return to User

---

6. Generate Audio (On-Demand)

POST /generate-audio

Generate audio from text. This endpoint should be called ONLY when the user clicks the "Play Audio" or speaker button on the UI.

Request Body:

{
  "text": "తెలంగాణలో అందుబాటులో ఉన్న స్కాలర్‌షిప్‌ల గురించి...",
  "language": "te"  // Language code (default: "en")
}

Response:

{
  "audio": "base64_encoded_mp3_audio_data"
}

Usage Flow:

1. User submits question → Receive answer (fast, no audio)
2. User clicks speaker button → Call /generate-audio → Play audio

Error Response (400 - Empty Text):

{
  "detail": "Text cannot be empty"
}

Error Response (400 - Unsupported Language):

{
  "detail": "Unsupported language. Supported: ['en', 'hi', 'te', 'ta', ...]"
}

Error Response (400 - Empty Question):

{
  "detail": "Question cannot be empty"
}

Error Response (400 - Unsupported Language):

{
  "detail": "Unsupported language. Supported: ['en', 'hi', 'te', 'ta', ...]"
}

Error Response (500):

{
  "detail": "Error processing query: [error message]"
}

---

Interactive API Documentation

FastAPI automatically generates interactive API documentation:

• Swagger UI: http://127.0.0.1:8000/docs
• ReDoc: http://127.0.0.1:8000/redoc

These interfaces allow you to:

• View all endpoints
• See request/response schemas
• Test API calls directly from the browser
• Download OpenAPI specification

---

Usage Examples

Using cURL

# Health check
curl http://127.0.0.1:8000/health

# Get supported languages
curl http://127.0.0.1:8000/languages

# Get states
curl http://127.0.0.1:8000/states

# Query in English
curl -X POST http://127.0.0.1:8000/query \
  -H "Content-Type: application/json" \
  -d '{
    "question": "What scholarships are available for SC students?",
    "state": "Karnataka",
    "language": "en"
  }'

# Query in Hindi
curl -X POST http://127.0.0.1:8000/query \
  -H "Content-Type: application/json" \
  -d '{
    "question": "छात्रवृत्ति के बारे में बताएं",
    "language": "hi"
  }'

# Query in Telugu
curl -X POST http://127.0.0.1:8000/query \
  -H "Content-Type: application/json" \
  -d '{
    "question": "స్కాలర్‌షిప్ల గురించి చెప్పండి",
    "state": "Telangana",
    "language": "te"
  }'

# Generate audio (when user clicks speaker button)
curl -X POST http://127.0.0.1:8000/generate-audio \
  -H "Content-Type: application/json" \
  -d '{
    "text": "తెలంగాణలో అందుబాటులో ఉన్న స్కాలర్‌షిప్‌లు...",
    "language": "te"
  }'

Using Python requests

import requests

# Query in English
response = requests.post(
    "http://127.0.0.1:8000/query",
    json={
        "question": "My daughter is studying in 9th standard. What schemes are applicable?",
        "state": "Maharashtra",
        "language": "en"
    }
)

data = response.json()
print(data["answer"])

# Query in Hindi
response_hindi = requests.post(
    "http://127.0.0.1:8000/query",
    json={
        "question": "मुझे छात्रवृत्ति चाहिए",
        "language": "hi"
    }
)

hindi_data = response_hindi.json()
print(hindi_data["answer"])  # Answer will be in Hindi

# Generate audio on-demand (when user clicks speaker button)
audio_response = requests.post(
    "http://127.0.0.1:8000/generate-audio",
    json={
        "text": hindi_data["answer"],
        "language": "hi"
    }
)

audio_data = audio_response.json()
# audio_data["audio"] contains base64 encoded MP3

Using JavaScript fetch

// Query in English
const response = await fetch('http://127.0.0.1:8000/query', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    question: 'What schemes are available for girl child education?',
    state: 'All States',
    language: 'en'
  })
});

const data = await response.json();
console.log(data.answer);

// Query in Telugu
const responseTelugu = await fetch('http://127.0.0.1:8000/query', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    question: 'బాలికల విద్య కోసం ఏ పథకాలు ఉన్నాయి?',
    language: 'te'
  })
});

const teluguData = await responseTelugu.json();
console.log(teluguData.answer);  // Answer in Telugu

// Generate audio when user clicks speaker button
const playAudio = async (text, language) => {
  const audioResponse = await fetch('http://127.0.0.1:8000/generate-audio', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      text: text,
      language: language
    })
  });
  
  const audioData = await audioResponse.json();
  const audio = new Audio(`data:audio/mp3;base64,${audioData.audio}`);
  audio.play();
};

// Usage: Call when user clicks speaker button
// playAudio(teluguData.answer, 'te');

Using React (Frontend Integration)

import React, { useState } from 'react';

function SchemeQuery() {
  const [language, setLanguage] = useState('en');
  const [question, setQuestion] = useState('');
  const [answer, setAnswer] = useState('');
  const [audioLoading, setAudioLoading] = useState(false);

  const handleSubmit = async (e) => {
    e.preventDefault();
    
    const response = await fetch('http://127.0.0.1:8000/query', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({
        question: question,
        language: language
      })
    });
    
    const data = await response.json();
    setAnswer(data.answer);
  };

  // Called only when user clicks speaker button
  const playAudio = async () => {
    if (!answer) return;
    
    setAudioLoading(true);
    try {
      const response = await fetch('http://127.0.0.1:8000/generate-audio', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({
          text: answer,
          language: language
        })
      });
      
      const data = await response.json();
      const audio = new Audio(`data:audio/mp3;base64,${data.audio}`);
      audio.play();
    } catch (error) {
      console.error('Audio generation failed:', error);
    } finally {
      setAudioLoading(false);
    }
  };

  return (
    <div>
      <select value={language} onChange={e => setLanguage(e.target.value)}>
        <option value="en">English</option>
        <option value="hi">Hindi</option>
        <option value="te">Telugu</option>
        <option value="ta">Tamil</option>
      </select>
      
      <form onSubmit={handleSubmit}>
        <input 
          value={question}
          onChange={e => setQuestion(e.target.value)}
          placeholder="Ask your question..."
        />
        <button type="submit">Ask</button>
      </form>
      
      {answer && (
        <div>
          <p>{answer}</p>
          <button onClick={playAudio} disabled={audioLoading}>
            {audioLoading ? '⏳ Generating...' : '🔊 Play Audio'}
          </button>
        </div>
      )}
    </div>
  );
}

const response = await fetch('http://127.0.0.1:8000/query', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    question: question,
    language: language
  })
});

const data = await response.json();
setAnswer(data.answer);

};

return (

<select value={language} onChange={e => setLanguage(e.target.value)}> English Hindi Telugu Tamil

  <form onSubmit={handleSubmit}>
    <input 
      value={question}
      onChange={e => setQuestion(e.target.value)}
      placeholder="Ask your question..."
    />
    <button type="submit">Ask</button>
  </form>
  
  {answer && <div>{answer}</div>}
</div>

); }

### Using Postman

1. **Method**: POST
2. **URL**: `http://127.0.0.1:8000/query`
3. **Headers**: 
   - `Content-Type: application/json`
4. **Body** (raw JSON):

**English:**
```json
{
  "question": "What are the schemes for construction workers?",
  "state": "Karnataka",
  "language": "en"
}

Hindi:

{
  "question": "निर्माण श्रमिकों के लिए क्या योजनाएं हैं?",
  "language": "hi"
}

Telugu:

{
  "question": "నిర్మాణ కార్మికులకు ఏ పథకాలు ఉన్నాయి?",
  "state": "Telangana",
  "language": "te"
}

---

Running the API

Start the Server

# Activate virtual environment
.venv\Scripts\activate

# Run the API
python app.py

The API will start on http://0.0.0.0:8000

Testing the API

Run the test script:

python test_api.py

---

CORS Configuration

The API is configured to accept requests from any origin (allow_origins=["*"]).

⚠️ For production, update the CORS settings in app.py:

app.add_middleware(
    CORSMiddleware,
    allow_origins=["https://yourdomain.com"],  # Specify allowed origins
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

---

Data Source

The API uses updated_data.csv containing 3400+ government schemes across categories:

• Education & Learning
• Social Welfare & Empowerment
• Health & Wellness
• Business & Entrepreneurship
• Women and Child
• And more...

---

Technology Stack

• Framework: FastAPI 0.104.1
• LLM: Groq API (llama-3.3-70b-versatile)
• Embeddings: HuggingFace sentence-transformers/all-MiniLM-L6-v2
• Vector DB: ChromaDB
• RAG Framework: LangChain 0.1.0
• Translation: deep-translator 1.11.4 (Google Translate)
• Text-to-Speech: gTTS 2.5.0 (Google Text-to-Speech)
• Server: Uvicorn

---

Multilingual Features

Translation Process

1. Input Translation: User's question in any Indian language → English
2. RAG Processing: English query → Vector search → LLM inference → English answer
3. Output Translation: English answer → User's selected language

Supported Language Codes

Code	Language	Code	Language
en	English	ml	Malayalam
hi	Hindi	kn	Kannada
te	Telugu	bn	Bengali
ta	Tamil	mr	Marathi
gu	Gujarati	pa	Punjabi
ur	Urdu	or	Odia
as	Assamese

Text-to-Speech (Optional)

To enable audio responses, uncomment the following lines in app.py:

# Line ~280 in app.py
audio_base64 = TranslationService.text_to_speech(final_answer, request.language)

When enabled, the API will return base64-encoded MP3 audio in the audio field.

---

Testing the Multilingual API

Using the Test Script

# Make sure the server is running first
python app.py

# In another terminal
python test_translation.py

The test script will:

1. Verify language endpoint
2. Test queries in English, Hindi, Telugu, Tamil, and Malayalam
3. Display translated responses

Manual Testing Checklist

• Test each supported language
• Verify translations are accurate
• Check source citations are included
• Test with state filters
• Test error handling (empty questions, invalid languages)
• Verify CORS headers for frontend integration

---

Rate Limits

Currently, there are no rate limits implemented. The API uses Groq's free tier which has its own rate limits.

For production deployment, consider implementing:

• Request rate limiting
• Authentication/API keys
• Caching for common queries

---

Error Handling

• 400 Bad Request: Invalid or empty question
• 500 Internal Server Error: Processing error (check GROQ_API_KEY)

---

Performance Notes

• First query may take 3-5 seconds (vector search + LLM inference)
• Subsequent queries are faster (~1-2 seconds)
• ChromaDB is persisted to disk (./chroma_db/) for faster restarts
• 3400 schemes are chunked into ~12,000-15,000 text segments

---

Deployment

Local Development

uvicorn app:app --reload --host 0.0.0.0 --port 8000

Production (with Gunicorn)

pip install gunicorn
gunicorn app:app -w 4 -k uvicorn.workers.UvicornWorker --bind 0.0.0.0:8000

Docker (Optional)

Create a Dockerfile:

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

---

Support

For issues or questions:

1. Check API docs at /docs
2. Review logs in terminal
3. Verify .env file has valid GROQ_API_KEY
4. Ensure updated_data.csv is present

---

Example Queries (Multilingual)

English

• "My daughter is studying in 9th standard. What schemes are applicable to her?"
• "What scholarships are available for SC/ST students?"
• "What are the schemes for construction workers?"
• "Tell me about Beti Bachao Beti Padhao scheme"

Hindi (हिंदी)

• "मेरी बेटी 9वीं कक्षा में पढ़ती है। उसके लिए कौन सी योजनाएं हैं?"
• "SC/ST छात्रों के लिए क्या छात्रवृत्ति उपलब्ध है?"
• "निर्माण श्रमिकों के लिए क्या योजनाएं हैं?"
• "बेटी बचाओ बेटी पढ़ाओ योजना के बारे में बताएं"

Telugu (తెలుగు)

• "నా కూతురు 9వ తరగతి చదువుతోంది. ఆమెకు ఏ పథకాలు వర్తిస్తాయి?"
• "SC/ST విద్యార్థులకు ఏ స్కాలర్‌షిప్‌లు అందుబాటులో ఉన్నాయి?"
• "నిర్మాణ కార్మికులకు ఏ పథకాలు ఉన్నాయి?"

Tamil (தமிழ்)

• "என் மகள் 9வது வகுப்பு படிக்கிறாள். அவளுக்கு என்ன திட்டங்கள் பொருந்தும்?"
• "SC/ST மாணவர்களுக்கு என்ன உதவித்தொகை கிடைக்கும்?"

Malayalam (മലയാളം)

• "എന്റെ മകൾ 9-ാം ക്ലാസിൽ പഠിക്കുന്നു. അവൾക്ക് എന്തെല്ലാം പദ്ധതികൾ ബാധകമാണ്?"
• "SC/ST വിദ്യാർത്ഥികൾക്ക് എന്ത് സ്കോളർഷിപ്പുകൾ ലഭ്യമാണ്?"

---

Frontend Integration Guide

For detailed React integration instructions, see: MULTILINGUAL_INTEGRATION_GUIDE.md

Key points for frontend developers:

1. Always send language parameter with queries
2. Backend handles ALL translation - no frontend translation needed
3. Use Web Speech API for voice input (browser native)
4. Use Speech Synthesis API for voice output (browser native)
5. Display loading states during translation/query processing

---

Performance & Optimization

Response Times

• Translation: ~0.5-1 second per translation
• RAG Query: ~2-3 seconds
• Total: ~3-5 seconds for multilingual queries
• English-only: ~2-3 seconds (no translation overhead)

Optimization Tips

1. Cache translations for common queries
2. Lazy load audio - only generate when user clicks "Play"
3. Use connection pooling for API calls
4. Implement request debouncing in frontend
5. Add response caching for identical queries

Scaling Considerations

• Translation uses free Google Translate API (via deep-translator)
• No rate limits on translation service currently
• Groq API has free tier limits (check console.groq.com)
• Consider premium APIs for production (Azure Translator, Google Cloud Translation)