Files changed (13) hide show
  1. .gitattributes +2 -0
  2. Dockerfile +15 -0
  3. README.md +52 -15
  4. Tamil.wav +3 -0
  5. app.py +10 -0
  6. docker-compose.yml +19 -0
  7. hindi.wav +3 -0
  8. main.py +36 -0
  9. mock_hindi.mp3 +0 -0
  10. mock_tamil.mp3 +0 -0
  11. requirements.txt +19 -0
  12. test_groq.py +43 -0
  13. test_multilingual.py +43 -0
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ hindi.wav filter=lfs diff=lfs merge=lfs -text
37
+ Tamil.wav filter=lfs diff=lfs merge=lfs -text
Dockerfile ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ FROM python:3.10-slim
2
+
3
+ WORKDIR /app
4
+
5
+ # Install system dependencies for audio processing
6
+ RUN apt-get update && apt-get install -y ffmpeg && rm -rf /var/lib/apt/lists/*
7
+
8
+ COPY requirements.txt .
9
+ RUN pip install --no-cache-dir -r requirements.txt
10
+
11
+ COPY . .
12
+
13
+ EXPOSE 7860
14
+
15
+ CMD ["python", "main.py", "--mode", "ui"]
README.md CHANGED
@@ -1,15 +1,52 @@
1
- ---
2
- title: FinVoice
3
- emoji: ⚡
4
- colorFrom: purple
5
- colorTo: yellow
6
- sdk: gradio
7
- sdk_version: 6.29.0
8
- python_version: '3.12'
9
- app_file: app.py
10
- pinned: false
11
- license: mit
12
- short_description: An autonomous, multi-lingual AI voice agent built for BFSI
13
- ---
14
-
15
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # 🏦 FinVoice AI - Production BFSI Voice Agent
2
+
3
+ An autonomous, multi-lingual AI voice agent built for the Banking, Financial Services, and Insurance (BFSI) sector. This system is designed as a Forward Deployed Engineering (FDE) prototype that routes real-time voice queries using a stateful agent graph, processes Retrieval-Augmented Generation (RAG) for banking policies, calls mock REST APIs for customer data, and gracefully escalates frustrated customers to human agents.
4
+
5
+ ## 🌟 Key Capabilities
6
+ * **Stateful Reasoning (LangGraph):** Uses a deterministic Directed Acyclic Graph (DAG) to guarantee reliable API routing and avoid infinite AI loops.
7
+ * **Multilingual Voice-to-Voice:**
8
+ * **STT**: Leverages Groq's `whisper-large-v3-turbo` for instantaneous multilingual transcription.
9
+ * **TTS**: Uses Microsoft Azure Neural TTS (`edge-tts`) to generate hyper-realistic, native-sounding Indian regional accents (Hindi, Tamil, Marathi, Telugu, etc.)
10
+ * **RAG Vector Search:** Uses HuggingFace Embeddings (`all-MiniLM-L6-v2`) and FAISS to retrieve real banking policies locally.
11
+ * **Contextual API Tool-Calling:** Executes backend mock functions (e.g. `get_loan_status`, `check_kyc`) dynamically based on user intent.
12
+ * **Dynamic Slang Generation:** Employs a 120-Billion parameter LLM (`openai/gpt-oss-120b` via Groq) specifically prompted to generate colloquial conversational dialects rather than robotic formal text.
13
+
14
+ ## 🚀 Tech Stack
15
+ * **Agent Framework:** LangGraph, LangChain
16
+ * **LLM:** `gpt-oss-120b` (Groq API)
17
+ * **Backend:** FastAPI, Python 3.10+
18
+ * **Frontend:** Gradio (Enterprise UI Theme)
19
+ * **RAG:** FAISS, HuggingFace Sentence Transformers
20
+ * **Audio:** Groq Whisper API, Edge-TTS
21
+
22
+ ## ⚙️ Installation & Local Setup
23
+ 1. Clone the repository:
24
+ ```bash
25
+ git clone https://github.com/yourusername/FinVoice.git
26
+ cd FinVoice
27
+ ```
28
+ 2. Install dependencies:
29
+ ```bash
30
+ pip install -r requirements.txt
31
+ pip install edge-tts
32
+ ```
33
+ 3. Setup environment variables (`.env`):
34
+ ```env
35
+ GROQ_API_KEY=your_groq_key
36
+ ```
37
+ 4. Run the Web Application:
38
+ ```bash
39
+ python main.py --mode ui
40
+ ```
41
+ 5. Run automated Multilingual tests:
42
+ ```bash
43
+ python test_multilingual.py
44
+ ```
45
+
46
+ ## 🏗️ Architecture
47
+ The core agent logic runs inside `app/agent/nodes.py` which defines 5 specific nodes:
48
+ 1. `stt_node`: Ingests audio, calls Whisper API, detects language.
49
+ 2. `intent_node`: Uses LLM to classify if the query needs a Tool, Policy (RAG), or Human.
50
+ 3. `tool_execution_node`: Routes dynamically to backend Python functions.
51
+ 4. `generation_node`: Translates the result into the detected language using colloquial slang.
52
+ 5. `tts_node`: Generates Microsoft Azure Neural Voice files.
Tamil.wav ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b6d4a361d94d370024c758b70522de00695763edfa5b1297d43ecc1017cb02c9
3
+ size 518658
app.py ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ import os
2
+ from frontend.app import ui
3
+ from dotenv import load_dotenv
4
+
5
+ load_dotenv()
6
+
7
+ # HuggingFace Spaces requires an app.py file in the root directory
8
+ if __name__ == "__main__":
9
+ print("🚀 Starting FinVoice on HuggingFace Spaces...")
10
+ ui.launch(server_name="0.0.0.0", server_port=7860)
docker-compose.yml ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ version: '3.8'
2
+
3
+ services:
4
+ voice-agent-api:
5
+ build: .
6
+ ports:
7
+ - "8000:8000"
8
+ environment:
9
+ - GROQ_API_KEY=${GROQ_API_KEY}
10
+ - LANGCHAIN_API_KEY=${LANGCHAIN_API_KEY}
11
+ - LANGCHAIN_TRACING_V2=true
12
+ - LANGCHAIN_PROJECT=BFSI_Voice_Agent
13
+ volumes:
14
+ - .:/app
15
+ healthcheck:
16
+ test: ["CMD", "curl", "-f", "http://localhost:8000/docs"]
17
+ interval: 30s
18
+ timeout: 10s
19
+ retries: 3
hindi.wav ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a7c26e81018763701ce21640bf7ffa529428fb8abe91c7ae7dbc4ae3f936043c
3
+ size 243474
main.py ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import argparse
2
+ import uvicorn
3
+ import os
4
+ from dotenv import load_dotenv
5
+
6
+ # Load Environment variables
7
+ load_dotenv()
8
+
9
+ def main():
10
+ parser = argparse.ArgumentParser(description="Run BFSI Voice Agent")
11
+ parser.add_argument("--mode", choices=["ui", "api"], default="ui",
12
+ help="Run 'ui' for the Web Application or 'api' for the FastAPI backend")
13
+ parser.add_argument("--port", type=int, default=8000,
14
+ help="Port to run on")
15
+
16
+ args = parser.parse_args()
17
+
18
+ if not os.getenv("GROQ_API_KEY"):
19
+ print("⚠️ WARNING: GROQ_API_KEY is not set in .env! LLM features will fail.")
20
+
21
+ if args.mode == "ui":
22
+ print(f"🚀 Starting FinVoice: Production BFSI Support Web Application on port {args.port}...")
23
+ from frontend.app import ui
24
+
25
+ # HuggingFace Spaces injects SPACE_ID. If present, we must bind to 0.0.0.0 on port 7860.
26
+ if os.getenv("SPACE_ID"):
27
+ ui.launch(server_name="0.0.0.0", server_port=7860)
28
+ else:
29
+ ui.launch(server_port=args.port)
30
+ else:
31
+ print(f"🚀 Starting BFSI Voice Agent Production API on http://127.0.0.1:{args.port}/docs ...")
32
+ from backend.main import app as fastapi_app
33
+ uvicorn.run(fastapi_app, host="0.0.0.0", port=args.port)
34
+
35
+ if __name__ == "__main__":
36
+ main()
mock_hindi.mp3 ADDED
Binary file (20.5 kB). View file
 
mock_tamil.mp3 ADDED
Binary file (28.4 kB). View file
 
requirements.txt ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ fastapi
2
+ uvicorn
3
+ python-dotenv
4
+ python-multipart
5
+ pydantic
6
+ langgraph
7
+ langchain
8
+ langchain-groq
9
+ langchain-huggingface
10
+ langchain-community
11
+ langsmith
12
+ faiss-cpu
13
+ sentence-transformers
14
+ openai-whisper
15
+ gtts
16
+ pydub
17
+ numpy
18
+ requests
19
+ pytest
test_groq.py ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import os
2
+ from dotenv import load_dotenv
3
+ from langchain_groq import ChatGroq
4
+ from langchain_huggingface import HuggingFaceEmbeddings
5
+
6
+ def test_setup():
7
+ print("Testing dependencies and API keys...")
8
+
9
+ # 1. Load env
10
+ load_dotenv()
11
+ groq_key = os.getenv("GROQ_API_KEY")
12
+
13
+ if not groq_key or groq_key == "your_groq_api_key_here":
14
+ print(" Error: GROQ_API_KEY is not set in your environment or .env file.")
15
+ print("Please create a .env file and add your Groq API key.")
16
+ return
17
+ else:
18
+ print("Found GROQ_API_KEY.")
19
+
20
+ # 2. Test LLM
21
+ try:
22
+ print("Testing ChatGroq connection...")
23
+ llm = ChatGroq(model="openai/gpt-oss-120b", groq_api_key=groq_key)
24
+ response = llm.invoke("Hello, say 'Groq is working!' if you can read this.")
25
+ print(f"PASS LLM Response: {response.content}")
26
+ except Exception as e:
27
+ print(f"FAIL LLM Connection Failed: {e}")
28
+ return
29
+
30
+ # 3. Test Embeddings
31
+ try:
32
+ print("Testing HuggingFace Embeddings (all-MiniLM-L6-v2)...")
33
+ embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
34
+ result = embeddings.embed_query("Hello world")
35
+ print(f"PASS Embeddings generated successfully! Dimension: {len(result)}")
36
+ except Exception as e:
37
+ print(f"FAIL Embeddings Generation Failed: {e}")
38
+ return
39
+
40
+ print("\nEverything is working correctly! You can now run the evaluation or start the API.")
41
+
42
+ if __name__ == "__main__":
43
+ test_setup()
test_multilingual.py ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import os
2
+ from gtts import gTTS
3
+ from app.agent.graph import bfsi_agent
4
+ from dotenv import load_dotenv
5
+
6
+ load_dotenv()
7
+
8
+ def create_mock_customer_audio(text, lang, filename):
9
+ print(f"Simulating customer speaking {lang}...")
10
+ tts = gTTS(text=text, lang=lang)
11
+ tts.save(filename)
12
+ return filename
13
+
14
+ def run_test(audio_path, test_name):
15
+ print(f"\nRunning BFSI Agent Pipeline: {test_name}")
16
+ initial_state = {
17
+ "audio_path": audio_path,
18
+ "customer_id": "C1024",
19
+ "messages": []
20
+ }
21
+
22
+ result = bfsi_agent.invoke(initial_state)
23
+
24
+ # Safely print without emojis to avoid Windows cp1252 codec errors
25
+ print(f"STT Transcript: {result.get('transcript')}")
26
+ print(f"Language Detected: {result.get('language')}")
27
+ print(f"Intent Detected: {result.get('intent')}")
28
+ print(f"Agent Response: {result.get('response_text')}")
29
+ print(f"Agent Audio File Generated: {result.get('response_audio_path')}")
30
+ print("-" * 50)
31
+
32
+ if __name__ == "__main__":
33
+ print("Starting Multilingual Agent Evaluation...\n")
34
+
35
+ # 1. Hindi Test (Tool Calling Intent)
36
+ hi_audio = create_mock_customer_audio("मेरे लोन का स्टेटस क्या है?", "hi", "mock_hindi.mp3")
37
+ run_test(hi_audio, "Hindi Loan Status Query")
38
+
39
+ # 2. Tamil Test (RAG Knowledge Intent)
40
+ ta_audio = create_mock_customer_audio("தனிநபர் கடனுக்கான வட்டி விகிதம் என்ன?", "ta", "mock_tamil.mp3")
41
+ run_test(ta_audio, "Tamil Interest Rate Query")
42
+
43
+ print("Multilingual tests completed successfully!")