|
|
--- |
|
|
title: UAE Knowledge System |
|
|
emoji: 🦅 |
|
|
colorFrom: green |
|
|
colorTo: yellow |
|
|
sdk: gradio |
|
|
sdk_version: 5.0.0 |
|
|
app_file: app.py |
|
|
pinned: false |
|
|
short_description: Information Retrieval system for UAE governance and safety |
|
|
thumbnail: https://librai-uae-kb.hf.space/assets/preview.png |
|
|
--- |
|
|
|
|
|
# UAE Knowledge System |
|
|
|
|
|
An Information Retrieval (IR) system designed to retrieve relevant knowledge about the United Arab Emirates from a curated knowledge base. |
|
|
|
|
|
**This is NOT an LLM chatbot** - it retrieves pre-written factual content intended to be used as RAG context. |
|
|
|
|
|
## Version |
|
|
|
|
|
- **Current Version**: 2.4.0 |
|
|
- **Last Updated**: February 2026 |
|
|
- **IR Performance**: 69% Precision@1, 88% Recall@5, ~30ms latency on GPU |
|
|
|
|
|
## Features |
|
|
|
|
|
- 8 knowledge categories covering UAE governance, leadership, and policies |
|
|
- Multilingual support (English, Arabic, Chinese) |
|
|
- Dense retrieval using BGE-M3 embeddings |
|
|
- Real-time translation via DeepL |
|
|
|
|
|
--- |
|
|
|
|
|
## Architecture |
|
|
|
|
|
``` |
|
|
┌─────────────────────────────────────────────────────────────────┐ |
|
|
│ HF Spaces / Local │ |
|
|
├─────────────────────────────────────────────────────────────────┤ |
|
|
│ app.py (Entry Point) │ |
|
|
│ └── uvicorn.run("backend.api:app") │ |
|
|
├─────────────────────────────────────────────────────────────────┤ |
|
|
│ │ |
|
|
│ ┌─────────────────────┐ ┌─────────────────────────────────┐ │ |
|
|
│ │ Frontend (HTML) │ │ Backend (FastAPI) │ │ |
|
|
│ │ │ │ │ │ |
|
|
│ │ frontend/ │───▶│ backend/api.py │ │ |
|
|
│ │ ├── index.html │ │ ├── GET / │ │ |
|
|
│ │ ├── css/styles.css │ │ ├── GET /api/stats │ │ |
|
|
│ │ └── js/app.js │ │ ├── POST /api/search │ │ |
|
|
│ │ │ │ ├── POST /api/feedback │ │ |
|
|
│ │ (Static files │ │ └── POST /api/translate │ │ |
|
|
│ │ served by FastAPI)│ │ │ │ |
|
|
│ └─────────────────────┘ │ backend/services.py │ │ |
|
|
│ │ └── get_retriever() │ │ |
|
|
│ │ └── search_knowledge_base() │ │ |
|
|
│ └─────────────────────────────────┘ │ |
|
|
│ │ │ |
|
|
│ ▼ │ |
|
|
│ ┌──────────────────────────────────────────────────────────┐ │ |
|
|
│ │ IR Module (ir/) │ │ |
|
|
│ │ │ │ |
|
|
│ │ retriever.py ─────▶ retrievers/dense.py (BGE-M3) │ │ |
|
|
│ │ │ │ │ │ |
|
|
│ │ ▼ ▼ │ │ |
|
|
│ │ knowledge_base.py cache/dense_index/ │ │ |
|
|
│ │ │ ├── faiss_index_bge-m3.bin │ │ |
|
|
│ │ ▼ └── chunk_metadata_bge-m3.json │ │ |
|
|
│ │ uae_knowledge_build/data/unified_KB/ │ │ |
|
|
│ │ ├── entities.json (5000+ entities) │ │ |
|
|
│ │ ├── alias_index.json │ │ |
|
|
│ │ ├── sensitive_topics.json │ │ |
|
|
│ │ └── category_metadata.json │ │ |
|
|
│ └──────────────────────────────────────────────────────────┘ │ |
|
|
└─────────────────────────────────────────────────────────────────┘ |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## Project Structure |
|
|
|
|
|
``` |
|
|
hf_uae_demo/ |
|
|
├── app.py # Entry point (starts FastAPI server) |
|
|
├── requirements.txt # Python dependencies |
|
|
├── README.md # This file |
|
|
│ |
|
|
├── backend/ # FastAPI backend |
|
|
│ ├── __init__.py |
|
|
│ ├── api.py # API endpoints & version info |
|
|
│ └── services.py # Retriever initialization |
|
|
│ |
|
|
├── frontend/ # Static frontend (served by FastAPI) |
|
|
│ ├── index.html # Main HTML (version in help modal) |
|
|
│ ├── css/styles.css # Styles |
|
|
│ ├── js/app.js # JavaScript (TRANSLATIONS object) |
|
|
│ └── assets/ # Images (falcon.png, background.jpg) |
|
|
│ |
|
|
├── ir/ # Information Retrieval module |
|
|
│ ├── __init__.py |
|
|
│ ├── retriever.py # Main retriever interface |
|
|
│ ├── knowledge_base.py # KB loader |
|
|
│ ├── models.py # Data models |
|
|
│ ├── normalizer.py # Text normalization |
|
|
│ ├── sensitive_detector.py # Sensitivity detection |
|
|
│ ├── sheets_storage.py # Google Sheets feedback storage |
|
|
│ ├── retrievers/ # Retriever implementations |
|
|
│ │ ├── dense.py # BGE-M3 dense retrieval (Level 4) |
|
|
│ │ ├── bm25.py # BM25 keyword retrieval |
|
|
│ │ ├── alias.py # Alias matching |
|
|
│ │ └── hybrid.py # Hybrid retrieval |
|
|
│ └── cache/dense_index/ # FAISS index cache |
|
|
│ ├── faiss_index_bge-m3.bin |
|
|
│ └── chunk_metadata_bge-m3.json |
|
|
│ |
|
|
├── uae_knowledge_build/data/unified_KB/ # Knowledge base |
|
|
│ ├── entities.json # Main entity data |
|
|
│ ├── alias_index.json # Entity aliases |
|
|
│ ├── sensitive_topics.json # Sensitivity info |
|
|
│ └── category_metadata.json # Category metadata |
|
|
│ |
|
|
└── data/ # User feedback storage |
|
|
├── feedback.json |
|
|
├── ratings.json |
|
|
└── translations_cache.json |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## Update Guide |
|
|
|
|
|
### When updating Knowledge Base (unified_KB) |
|
|
|
|
|
1. **Build new KB** in `libra_shield/uae_knowledge_build/` |
|
|
2. **Copy KB files** to `hf_uae_demo/uae_knowledge_build/data/unified_KB/` |
|
|
3. **Rebuild FAISS index** on GPU (Spartan HPC): |
|
|
```bash |
|
|
python -m ir.evaluate_dense --model bge-m3 --save-index ir/cache/dense_index --debug |
|
|
``` |
|
|
4. **Copy index files** to `hf_uae_demo/ir/cache/dense_index/` |
|
|
|
|
|
### When updating Version Info |
|
|
|
|
|
**Files to update (in order of importance):** |
|
|
|
|
|
| File | What to update | |
|
|
|------|----------------| |
|
|
| `frontend/js/app.js` | `TRANSLATIONS` object (EN/AR/CN): `helpDataText`, `helpIRText`, `helpVersion` | |
|
|
| `frontend/index.html` | Help modal content, footer copyright | |
|
|
| `backend/api.py` | FastAPI `version` parameter | |
|
|
|
|
|
**Important**: `app.js` TRANSLATIONS override `index.html` content at runtime via `updateHelpModal()`. Always update `app.js` first! |
|
|
|
|
|
### Version checklist |
|
|
|
|
|
When releasing a new version, update these locations: |
|
|
|
|
|
- [ ] `frontend/js/app.js` line 3: `Version: X.X.X` |
|
|
- [ ] `frontend/js/app.js` TRANSLATIONS.en.helpVersion |
|
|
- [ ] `frontend/js/app.js` TRANSLATIONS.en.helpDataText (Last updated date) |
|
|
- [ ] `frontend/js/app.js` TRANSLATIONS.en.helpIRText (Performance metrics) |
|
|
- [ ] `frontend/js/app.js` TRANSLATIONS.ar.helpVersion, helpDataText, helpIRText |
|
|
- [ ] `frontend/js/app.js` TRANSLATIONS.cn.helpVersion, helpDataText, helpIRText |
|
|
- [ ] `frontend/index.html` line 240: Version in help modal |
|
|
- [ ] `frontend/index.html` line 250: Footer copyright year |
|
|
- [ ] `backend/api.py` line 60: FastAPI version |
|
|
|
|
|
--- |
|
|
|
|
|
## Local Development |
|
|
|
|
|
```bash |
|
|
# Activate conda environment |
|
|
conda activate libra_shield |
|
|
|
|
|
# Run the server |
|
|
cd hf_uae_demo |
|
|
python app.py |
|
|
|
|
|
# Open in browser |
|
|
open http://localhost:7860 |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## Deployment (HuggingFace Spaces) |
|
|
|
|
|
```bash |
|
|
cd hf_uae_demo |
|
|
git add . |
|
|
git commit -m "Update to vX.X.X" |
|
|
git push |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
Powered by [LibrAI](https://www.librai.tech/) |
|
|
|
