Spaces:
Configuration error
Configuration error
Upload 12 files
#1
by krishhx - opened
- .gitattributes +2 -0
- Dockerfile +15 -0
- README.md +52 -15
- Tamil.wav +3 -0
- app.py +10 -0
- docker-compose.yml +19 -0
- hindi.wav +3 -0
- main.py +36 -0
- mock_hindi.mp3 +0 -0
- mock_tamil.mp3 +0 -0
- requirements.txt +19 -0
- test_groq.py +43 -0
- test_multilingual.py +43 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
hindi.wav filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
Tamil.wav filter=lfs diff=lfs merge=lfs -text
|
Dockerfile
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
FROM python:3.10-slim
|
| 2 |
+
|
| 3 |
+
WORKDIR /app
|
| 4 |
+
|
| 5 |
+
# Install system dependencies for audio processing
|
| 6 |
+
RUN apt-get update && apt-get install -y ffmpeg && rm -rf /var/lib/apt/lists/*
|
| 7 |
+
|
| 8 |
+
COPY requirements.txt .
|
| 9 |
+
RUN pip install --no-cache-dir -r requirements.txt
|
| 10 |
+
|
| 11 |
+
COPY . .
|
| 12 |
+
|
| 13 |
+
EXPOSE 7860
|
| 14 |
+
|
| 15 |
+
CMD ["python", "main.py", "--mode", "ui"]
|
README.md
CHANGED
|
@@ -1,15 +1,52 @@
|
|
| 1 |
-
-
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 🏦 FinVoice AI - Production BFSI Voice Agent
|
| 2 |
+
|
| 3 |
+
An autonomous, multi-lingual AI voice agent built for the Banking, Financial Services, and Insurance (BFSI) sector. This system is designed as a Forward Deployed Engineering (FDE) prototype that routes real-time voice queries using a stateful agent graph, processes Retrieval-Augmented Generation (RAG) for banking policies, calls mock REST APIs for customer data, and gracefully escalates frustrated customers to human agents.
|
| 4 |
+
|
| 5 |
+
## 🌟 Key Capabilities
|
| 6 |
+
* **Stateful Reasoning (LangGraph):** Uses a deterministic Directed Acyclic Graph (DAG) to guarantee reliable API routing and avoid infinite AI loops.
|
| 7 |
+
* **Multilingual Voice-to-Voice:**
|
| 8 |
+
* **STT**: Leverages Groq's `whisper-large-v3-turbo` for instantaneous multilingual transcription.
|
| 9 |
+
* **TTS**: Uses Microsoft Azure Neural TTS (`edge-tts`) to generate hyper-realistic, native-sounding Indian regional accents (Hindi, Tamil, Marathi, Telugu, etc.)
|
| 10 |
+
* **RAG Vector Search:** Uses HuggingFace Embeddings (`all-MiniLM-L6-v2`) and FAISS to retrieve real banking policies locally.
|
| 11 |
+
* **Contextual API Tool-Calling:** Executes backend mock functions (e.g. `get_loan_status`, `check_kyc`) dynamically based on user intent.
|
| 12 |
+
* **Dynamic Slang Generation:** Employs a 120-Billion parameter LLM (`openai/gpt-oss-120b` via Groq) specifically prompted to generate colloquial conversational dialects rather than robotic formal text.
|
| 13 |
+
|
| 14 |
+
## 🚀 Tech Stack
|
| 15 |
+
* **Agent Framework:** LangGraph, LangChain
|
| 16 |
+
* **LLM:** `gpt-oss-120b` (Groq API)
|
| 17 |
+
* **Backend:** FastAPI, Python 3.10+
|
| 18 |
+
* **Frontend:** Gradio (Enterprise UI Theme)
|
| 19 |
+
* **RAG:** FAISS, HuggingFace Sentence Transformers
|
| 20 |
+
* **Audio:** Groq Whisper API, Edge-TTS
|
| 21 |
+
|
| 22 |
+
## ⚙️ Installation & Local Setup
|
| 23 |
+
1. Clone the repository:
|
| 24 |
+
```bash
|
| 25 |
+
git clone https://github.com/yourusername/FinVoice.git
|
| 26 |
+
cd FinVoice
|
| 27 |
+
```
|
| 28 |
+
2. Install dependencies:
|
| 29 |
+
```bash
|
| 30 |
+
pip install -r requirements.txt
|
| 31 |
+
pip install edge-tts
|
| 32 |
+
```
|
| 33 |
+
3. Setup environment variables (`.env`):
|
| 34 |
+
```env
|
| 35 |
+
GROQ_API_KEY=your_groq_key
|
| 36 |
+
```
|
| 37 |
+
4. Run the Web Application:
|
| 38 |
+
```bash
|
| 39 |
+
python main.py --mode ui
|
| 40 |
+
```
|
| 41 |
+
5. Run automated Multilingual tests:
|
| 42 |
+
```bash
|
| 43 |
+
python test_multilingual.py
|
| 44 |
+
```
|
| 45 |
+
|
| 46 |
+
## 🏗️ Architecture
|
| 47 |
+
The core agent logic runs inside `app/agent/nodes.py` which defines 5 specific nodes:
|
| 48 |
+
1. `stt_node`: Ingests audio, calls Whisper API, detects language.
|
| 49 |
+
2. `intent_node`: Uses LLM to classify if the query needs a Tool, Policy (RAG), or Human.
|
| 50 |
+
3. `tool_execution_node`: Routes dynamically to backend Python functions.
|
| 51 |
+
4. `generation_node`: Translates the result into the detected language using colloquial slang.
|
| 52 |
+
5. `tts_node`: Generates Microsoft Azure Neural Voice files.
|
Tamil.wav
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b6d4a361d94d370024c758b70522de00695763edfa5b1297d43ecc1017cb02c9
|
| 3 |
+
size 518658
|
app.py
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import os
|
| 2 |
+
from frontend.app import ui
|
| 3 |
+
from dotenv import load_dotenv
|
| 4 |
+
|
| 5 |
+
load_dotenv()
|
| 6 |
+
|
| 7 |
+
# HuggingFace Spaces requires an app.py file in the root directory
|
| 8 |
+
if __name__ == "__main__":
|
| 9 |
+
print("🚀 Starting FinVoice on HuggingFace Spaces...")
|
| 10 |
+
ui.launch(server_name="0.0.0.0", server_port=7860)
|
docker-compose.yml
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version: '3.8'
|
| 2 |
+
|
| 3 |
+
services:
|
| 4 |
+
voice-agent-api:
|
| 5 |
+
build: .
|
| 6 |
+
ports:
|
| 7 |
+
- "8000:8000"
|
| 8 |
+
environment:
|
| 9 |
+
- GROQ_API_KEY=${GROQ_API_KEY}
|
| 10 |
+
- LANGCHAIN_API_KEY=${LANGCHAIN_API_KEY}
|
| 11 |
+
- LANGCHAIN_TRACING_V2=true
|
| 12 |
+
- LANGCHAIN_PROJECT=BFSI_Voice_Agent
|
| 13 |
+
volumes:
|
| 14 |
+
- .:/app
|
| 15 |
+
healthcheck:
|
| 16 |
+
test: ["CMD", "curl", "-f", "http://localhost:8000/docs"]
|
| 17 |
+
interval: 30s
|
| 18 |
+
timeout: 10s
|
| 19 |
+
retries: 3
|
hindi.wav
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a7c26e81018763701ce21640bf7ffa529428fb8abe91c7ae7dbc4ae3f936043c
|
| 3 |
+
size 243474
|
main.py
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import argparse
|
| 2 |
+
import uvicorn
|
| 3 |
+
import os
|
| 4 |
+
from dotenv import load_dotenv
|
| 5 |
+
|
| 6 |
+
# Load Environment variables
|
| 7 |
+
load_dotenv()
|
| 8 |
+
|
| 9 |
+
def main():
|
| 10 |
+
parser = argparse.ArgumentParser(description="Run BFSI Voice Agent")
|
| 11 |
+
parser.add_argument("--mode", choices=["ui", "api"], default="ui",
|
| 12 |
+
help="Run 'ui' for the Web Application or 'api' for the FastAPI backend")
|
| 13 |
+
parser.add_argument("--port", type=int, default=8000,
|
| 14 |
+
help="Port to run on")
|
| 15 |
+
|
| 16 |
+
args = parser.parse_args()
|
| 17 |
+
|
| 18 |
+
if not os.getenv("GROQ_API_KEY"):
|
| 19 |
+
print("⚠️ WARNING: GROQ_API_KEY is not set in .env! LLM features will fail.")
|
| 20 |
+
|
| 21 |
+
if args.mode == "ui":
|
| 22 |
+
print(f"🚀 Starting FinVoice: Production BFSI Support Web Application on port {args.port}...")
|
| 23 |
+
from frontend.app import ui
|
| 24 |
+
|
| 25 |
+
# HuggingFace Spaces injects SPACE_ID. If present, we must bind to 0.0.0.0 on port 7860.
|
| 26 |
+
if os.getenv("SPACE_ID"):
|
| 27 |
+
ui.launch(server_name="0.0.0.0", server_port=7860)
|
| 28 |
+
else:
|
| 29 |
+
ui.launch(server_port=args.port)
|
| 30 |
+
else:
|
| 31 |
+
print(f"🚀 Starting BFSI Voice Agent Production API on http://127.0.0.1:{args.port}/docs ...")
|
| 32 |
+
from backend.main import app as fastapi_app
|
| 33 |
+
uvicorn.run(fastapi_app, host="0.0.0.0", port=args.port)
|
| 34 |
+
|
| 35 |
+
if __name__ == "__main__":
|
| 36 |
+
main()
|
mock_hindi.mp3
ADDED
|
Binary file (20.5 kB). View file
|
|
|
mock_tamil.mp3
ADDED
|
Binary file (28.4 kB). View file
|
|
|
requirements.txt
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
fastapi
|
| 2 |
+
uvicorn
|
| 3 |
+
python-dotenv
|
| 4 |
+
python-multipart
|
| 5 |
+
pydantic
|
| 6 |
+
langgraph
|
| 7 |
+
langchain
|
| 8 |
+
langchain-groq
|
| 9 |
+
langchain-huggingface
|
| 10 |
+
langchain-community
|
| 11 |
+
langsmith
|
| 12 |
+
faiss-cpu
|
| 13 |
+
sentence-transformers
|
| 14 |
+
openai-whisper
|
| 15 |
+
gtts
|
| 16 |
+
pydub
|
| 17 |
+
numpy
|
| 18 |
+
requests
|
| 19 |
+
pytest
|
test_groq.py
ADDED
|
@@ -0,0 +1,43 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import os
|
| 2 |
+
from dotenv import load_dotenv
|
| 3 |
+
from langchain_groq import ChatGroq
|
| 4 |
+
from langchain_huggingface import HuggingFaceEmbeddings
|
| 5 |
+
|
| 6 |
+
def test_setup():
|
| 7 |
+
print("Testing dependencies and API keys...")
|
| 8 |
+
|
| 9 |
+
# 1. Load env
|
| 10 |
+
load_dotenv()
|
| 11 |
+
groq_key = os.getenv("GROQ_API_KEY")
|
| 12 |
+
|
| 13 |
+
if not groq_key or groq_key == "your_groq_api_key_here":
|
| 14 |
+
print(" Error: GROQ_API_KEY is not set in your environment or .env file.")
|
| 15 |
+
print("Please create a .env file and add your Groq API key.")
|
| 16 |
+
return
|
| 17 |
+
else:
|
| 18 |
+
print("Found GROQ_API_KEY.")
|
| 19 |
+
|
| 20 |
+
# 2. Test LLM
|
| 21 |
+
try:
|
| 22 |
+
print("Testing ChatGroq connection...")
|
| 23 |
+
llm = ChatGroq(model="openai/gpt-oss-120b", groq_api_key=groq_key)
|
| 24 |
+
response = llm.invoke("Hello, say 'Groq is working!' if you can read this.")
|
| 25 |
+
print(f"PASS LLM Response: {response.content}")
|
| 26 |
+
except Exception as e:
|
| 27 |
+
print(f"FAIL LLM Connection Failed: {e}")
|
| 28 |
+
return
|
| 29 |
+
|
| 30 |
+
# 3. Test Embeddings
|
| 31 |
+
try:
|
| 32 |
+
print("Testing HuggingFace Embeddings (all-MiniLM-L6-v2)...")
|
| 33 |
+
embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
|
| 34 |
+
result = embeddings.embed_query("Hello world")
|
| 35 |
+
print(f"PASS Embeddings generated successfully! Dimension: {len(result)}")
|
| 36 |
+
except Exception as e:
|
| 37 |
+
print(f"FAIL Embeddings Generation Failed: {e}")
|
| 38 |
+
return
|
| 39 |
+
|
| 40 |
+
print("\nEverything is working correctly! You can now run the evaluation or start the API.")
|
| 41 |
+
|
| 42 |
+
if __name__ == "__main__":
|
| 43 |
+
test_setup()
|
test_multilingual.py
ADDED
|
@@ -0,0 +1,43 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import os
|
| 2 |
+
from gtts import gTTS
|
| 3 |
+
from app.agent.graph import bfsi_agent
|
| 4 |
+
from dotenv import load_dotenv
|
| 5 |
+
|
| 6 |
+
load_dotenv()
|
| 7 |
+
|
| 8 |
+
def create_mock_customer_audio(text, lang, filename):
|
| 9 |
+
print(f"Simulating customer speaking {lang}...")
|
| 10 |
+
tts = gTTS(text=text, lang=lang)
|
| 11 |
+
tts.save(filename)
|
| 12 |
+
return filename
|
| 13 |
+
|
| 14 |
+
def run_test(audio_path, test_name):
|
| 15 |
+
print(f"\nRunning BFSI Agent Pipeline: {test_name}")
|
| 16 |
+
initial_state = {
|
| 17 |
+
"audio_path": audio_path,
|
| 18 |
+
"customer_id": "C1024",
|
| 19 |
+
"messages": []
|
| 20 |
+
}
|
| 21 |
+
|
| 22 |
+
result = bfsi_agent.invoke(initial_state)
|
| 23 |
+
|
| 24 |
+
# Safely print without emojis to avoid Windows cp1252 codec errors
|
| 25 |
+
print(f"STT Transcript: {result.get('transcript')}")
|
| 26 |
+
print(f"Language Detected: {result.get('language')}")
|
| 27 |
+
print(f"Intent Detected: {result.get('intent')}")
|
| 28 |
+
print(f"Agent Response: {result.get('response_text')}")
|
| 29 |
+
print(f"Agent Audio File Generated: {result.get('response_audio_path')}")
|
| 30 |
+
print("-" * 50)
|
| 31 |
+
|
| 32 |
+
if __name__ == "__main__":
|
| 33 |
+
print("Starting Multilingual Agent Evaluation...\n")
|
| 34 |
+
|
| 35 |
+
# 1. Hindi Test (Tool Calling Intent)
|
| 36 |
+
hi_audio = create_mock_customer_audio("मेरे लोन का स्टेटस क्या है?", "hi", "mock_hindi.mp3")
|
| 37 |
+
run_test(hi_audio, "Hindi Loan Status Query")
|
| 38 |
+
|
| 39 |
+
# 2. Tamil Test (RAG Knowledge Intent)
|
| 40 |
+
ta_audio = create_mock_customer_audio("தனிநபர் கடனுக்கான வட்டி விகிதம் என்ன?", "ta", "mock_tamil.mp3")
|
| 41 |
+
run_test(ta_audio, "Tamil Interest Rate Query")
|
| 42 |
+
|
| 43 |
+
print("Multilingual tests completed successfully!")
|