Spaces:

ItsXyron
/

hyperlookup-api

Running

Aditya

feat: Integrate Telegram Master DB 65M+ records with fast range routing and country detection

0dd4566 verified 4 days ago

8.61 kB

---
title: HyperLookup SaaS
emoji: ⚡
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
pinned: false
---
# ⚡ HyperLookup API — Ultra-Fast 2.5 Billion Record SaaS Engine

An enterprise-grade, high-performance lookup API and search dashboard engineered to query across **227 GB of partitioned columnar Parquet data (~2.5+ Billion Records)** in milliseconds — **hosted 100% FREE** with zero infrastructure bills.

---

## 🎯 Architecture Overview

```
                      +---------------------------------------+
                      |     Client (Web UI / REST API)        |
                      +-------------------+-------------------+
                                          |
                                          v
                      +---------------------------------------+
                      |     FastAPI Edge Server (Docker)      |
                      |  - In-Memory LRU Cache (0.05ms)       |
                      |  - O(1) Sparse Partition Router       |
                      +-------------------+-------------------+
                                          |
                                          v
                      +---------------------------------------+
                      |   DuckDB Vectorized Query Engine      |
                      |  - HTTP Range Requests (Byte Streaming)|
                      |  - Parquet Predicate Pushdown         |
                      +-------------------+-------------------+
                                          | (Co-located <5ms)
                                          v
+---------------------------------------------------------------------------------+
| Hugging Face Storage Bucket: ItsSecretz/icrm-hitek-full-db-mixed-bucket (227 GB)|
| - idx_phone.0.parquet   to   idx_phone.6.parquet   (~2.5 Billion Phone Records) |
| - idx_aadhar.0.parquet  to   idx_aadhar.4.parquet  (~1.4 Billion Aadhar Records)|
+---------------------------------------------------------------------------------+
```

### Why It's Blazing Fast:
1. **O(1) Sparse Router (`partitions.json`)**:
   Instead of scanning 227 GB, the router uses an in-memory boundary index to identify the exact single file partition in **0.05 milliseconds**, skipping 99% of all database files.
2. **Parquet Predicate Pushdown**:
   DuckDB fetches only the Parquet metadata footer and streams only the single matching row group (~1MB) via HTTP Range requests.
3. **In-Memory Query Cache**:
   Popular or repeated queries return in **0.05ms** from the in-memory TTL cache.
4. **Warm Engine Speed**:
   Once Parquet footers are cached in DuckDB memory, queries execute in **180ms – 300ms**!

---

## 🚀 Benchmark Results (Live Tested)

| Query Type | Database Scope | Latency | Result |
| :--- | :--- | :--- | :--- |
| **Warm Point Lookup (Phone)** | 2.5 Billion Records (16.6 GB Partition) | **200 ms - 219 ms** | ✅ Instant Exact Records |
| **Warm Point Lookup (Aadhar)**| 1.4 Billion Records (15.7 GB Partition) | **220 ms - 350 ms** | ✅ Instant Exact Records |
| **In-Memory Cache Hit** | Cached Query | **0.05 ms** | ✅ Sub-millisecond |
| **Cold Lookup (First Request)**| Uncached remote fetch | **3.8s - 10s** | ✅ Footers cached for warm queries |

---

## 🌐 100% Free Hosting & Deployment (Professional SaaS Level)

### 🥇 Option 1: Hugging Face Spaces (RECOMMENDED — BEST PERFORMANCE)
**Why it's the best:**
- **Specs**: 2 vCPU, **16 GB RAM** on the 100% Free Tier!
- **Zero Latency**: Hugging Face Spaces run inside the **exact same AWS us-east-1 datacenter** as the Storage Bucket (`ItsSecretz/icrm-hitek-full-db-mixed-bucket`). Network latency between your API and the bucket is **< 5ms**!
- **Bandwidth**: Unlimited, 100% Free egress.

#### How to Deploy on Hugging Face Spaces:
1. Go to [Hugging Face Spaces](https://huggingface.co/spaces) and click **"Create new Space"**.
2. Set Space Name: `hyperlookup-api` (or your choice).
3. Select **Space SDK**: **Docker** -> **Blank**.
4. Set Space Hardware: **CPU Basic (2 vCPU, 16GB RAM) - Free**.
5. Set Visibility: **Public** (or Private).
6. Clone the Space repo or push this codebase:
   ```bash
   git init
   git remote add origin https://huggingface.co/spaces/YOUR_USERNAME/hyperlookup-api
   git add .
   git commit -m "Deploy HyperLookup SaaS"
   git push -u origin main
   ```
7. Hugging Face will automatically build the `Dockerfile` and launch your API on `https://YOUR_USERNAME-hyperlookup-api.hf.space`.

---

### 🥈 Option 2: Koyeb (Free Tier Alternative)
1. Sign up at [Koyeb.com](https://www.koyeb.com) (Free Tier includes 512MB RAM micro instances).
2. Connect your GitHub repository.
3. Koyeb will detect the `Dockerfile` and auto-deploy with a global edge URL: `https://your-app.koyeb.app`.

---

### 🥉 Option 3: Render (Free Web Service)
1. Go to [Render.com](https://render.com) and click **"New Web Service"**.
2. Connect your GitHub repository.
3. Select Environment: **Docker**.
4. Free plan: Click **Deploy**.

---

## 👑 Professional Custom Domain & Global CDN Setup (Cloudflare)

To give your project a professional SaaS brand (e.g., `api.yourbrand.com` or `search.yourbrand.com`):

1. **Add Domain to Cloudflare** (Free Plan).
2. In Cloudflare DNS settings:
   - Type: `CNAME`
   - Name: `api` (or `@`)
   - Target: `YOUR_USERNAME-hyperlookup-api.hf.space`
   - Proxy status: **Proxied (Orange Cloud)**.
3. In Cloudflare **SSL/TLS**:
   - Set encryption mode to **Full (strict)**.
4. In Cloudflare **Caching**:
   - Create a Cache Rule for `/api/lookup*` with Edge TTL 12 hours.
   - Now repeated queries across the globe are answered from Cloudflare's 300+ Edge Data Centers in **under 20 milliseconds**!

---

## 💻 Local Development & Testing

### 1. Install Requirements
```bash
pip install -r requirements.txt
```

### 2. Run the Server
```bash
uvicorn main:app --host 0.0.0.0 --port 8000 --reload
```

### 3. Open in Browser
- **SaaS Dashboard**: [http://localhost:8000](http://localhost:8000)
- **Interactive Swagger Docs**: [http://localhost:8000/docs](http://localhost:8000/docs)
- **System Health & Stats**: [http://localhost:8000/api/stats](http://localhost:8000/api/stats)

---

## 📡 API Reference

### 1. Auto-Detect Lookup
Automatically identifies whether input is a 10-digit Phone or 12-digit Aadhar:
```bash
GET /api/lookup?q=9313336070
```

#### Sample JSON Response:
```json
{
  "success": true,
  "query_type": "phone",
  "query": "9313336070",
  "formatted_query": "+91 9313336070",
  "count": 2,
  "records": [
    {
      "name": "Aakash Jagdishbhai Makhija",
      "fathersName": "Makhija Bhai",
      "phoneNumber": "9313336070",
      "aadharNumber": null,
      "otherNumber": "7567303287",
      "address": "705/5, Ambawadi Near Police Station, Ahmedabad, Gujarat, 382475",
      "district": "Ahmedabad",
      "pincode": "382475",
      "state": "Gujarat",
      "town": null,
      "source": "inddata"
    }
  ],
  "target_partitions": ["idx_phone.4.parquet"],
  "latency_ms": 218.4,
  "cached": false
}
```

### 2. Phone Lookup
```bash
GET /api/phone/9313336070
```

### 3. Aadhar Lookup
```bash
GET /api/aadhar/351104219878
```

### 4. Telegram User ID Lookup (Global Master DB - 65M+ Records)
Fast, direct partition-routed lookup for Telegram accounts by User ID:
```bash
GET /tg/lookup?user_id=74252719
# or direct path:
GET /tg/74252719
```

#### Sample Telegram JSON Response:
```json
{
  "status": "success",
  "user_id": 74252719,
  "country": "Iran",
  "num": "+989192125069",
  "phone": "+989192125069",
  "formatted_phone": "+98 919 212 5069",
  "country_code": "IR",
  "query_type": "telegram",
  "target_partition": "part_0000_1_74252719.parquet",
  "latency_ms": 14.8,
  "cached": false
}
```

### 5. System Statistics
```bash
GET /api/stats
```

---

## 📦 Client Code Examples

### Python:
```python
import requests
url = "https://your-domain.com/api/lookup"
response = requests.get(url, params={"q": "9313336070"})
data = response.json()
print(f"Matches: {data['count']} in {data['latency_ms']} ms")
for record in data['records']:
    print(record['name'], record['phoneNumber'], record['address'])
```

### JavaScript / Node.js:
```javascript
const res = await fetch("https://your-domain.com/api/lookup?q=9313336070");
const data = await res.json();
console.log(`Latency: ${data.latency_ms}ms`, data.records);
```

### cURL:
```bash
curl -X GET "https://your-domain.com/api/lookup?q=9313336070"
```

---

## 🛡️ License & Disclaimers
This repository is engineered for ultra-fast columnar analytical lookups using open-source tools (FastAPI, DuckDB, Hugging Face Hub).

	---
	title: HyperLookup SaaS
	emoji: ⚡
	colorFrom: blue
	colorTo: indigo
	sdk: docker
	app_port: 7860
	pinned: false
	---

	# ⚡ HyperLookup API — Ultra-Fast 2.5 Billion Record SaaS Engine

	An enterprise-grade, high-performance lookup API and search dashboard engineered to query across 227 GB of partitioned columnar Parquet data (~2.5+ Billion Records) in milliseconds — hosted 100% FREE with zero infrastructure bills.

	---

	## 🎯 Architecture Overview

	```
	+---------------------------------------+
	\| Client (Web UI / REST API) \|
	+-------------------+-------------------+
	\|
	v
	+---------------------------------------+
	\| FastAPI Edge Server (Docker) \|
	\| - In-Memory LRU Cache (0.05ms) \|
	\| - O(1) Sparse Partition Router \|
	+-------------------+-------------------+
	\|
	v
	+---------------------------------------+
	\| DuckDB Vectorized Query Engine \|
	\| - HTTP Range Requests (Byte Streaming)\|
	\| - Parquet Predicate Pushdown \|
	+-------------------+-------------------+
	\| (Co-located <5ms)
	v
	+---------------------------------------------------------------------------------+
	\| Hugging Face Storage Bucket: ItsSecretz/icrm-hitek-full-db-mixed-bucket (227 GB)\|
	\| - idx_phone.0.parquet to idx_phone.6.parquet (~2.5 Billion Phone Records) \|
	\| - idx_aadhar.0.parquet to idx_aadhar.4.parquet (~1.4 Billion Aadhar Records)\|
	+---------------------------------------------------------------------------------+
	```

	### Why It's Blazing Fast:
	1. O(1) Sparse Router (`partitions.json`):
	Instead of scanning 227 GB, the router uses an in-memory boundary index to identify the exact single file partition in 0.05 milliseconds, skipping 99% of all database files.
	2. Parquet Predicate Pushdown:
	DuckDB fetches only the Parquet metadata footer and streams only the single matching row group (~1MB) via HTTP Range requests.
	3. In-Memory Query Cache:
	Popular or repeated queries return in 0.05ms from the in-memory TTL cache.
	4. Warm Engine Speed:
	Once Parquet footers are cached in DuckDB memory, queries execute in 180ms – 300ms!

	---

	## 🚀 Benchmark Results (Live Tested)

	\| Query Type \| Database Scope \| Latency \| Result \|
	\| :--- \| :--- \| :--- \| :--- \|
	\| Warm Point Lookup (Phone) \| 2.5 Billion Records (16.6 GB Partition) \| 200 ms - 219 ms \| ✅ Instant Exact Records \|
	\| Warm Point Lookup (Aadhar)\| 1.4 Billion Records (15.7 GB Partition) \| 220 ms - 350 ms \| ✅ Instant Exact Records \|
	\| In-Memory Cache Hit \| Cached Query \| 0.05 ms \| ✅ Sub-millisecond \|
	\| Cold Lookup (First Request)\| Uncached remote fetch \| 3.8s - 10s \| ✅ Footers cached for warm queries \|

	---

	## 🌐 100% Free Hosting & Deployment (Professional SaaS Level)

	### 🥇 Option 1: Hugging Face Spaces (RECOMMENDED — BEST PERFORMANCE)
	Why it's the best:
	- Specs: 2 vCPU, 16 GB RAM on the 100% Free Tier!
	- Zero Latency: Hugging Face Spaces run inside the exact same AWS us-east-1 datacenter as the Storage Bucket (`ItsSecretz/icrm-hitek-full-db-mixed-bucket`). Network latency between your API and the bucket is < 5ms!
	- Bandwidth: Unlimited, 100% Free egress.

	#### How to Deploy on Hugging Face Spaces:
	1. Go to [Hugging Face Spaces](https://huggingface.co/spaces) and click "Create new Space".
	2. Set Space Name: `hyperlookup-api` (or your choice).
	3. Select Space SDK: Docker -> Blank.
	4. Set Space Hardware: CPU Basic (2 vCPU, 16GB RAM) - Free.
	5. Set Visibility: Public (or Private).
	6. Clone the Space repo or push this codebase:
	```bash
	git init
	git remote add origin https://huggingface.co/spaces/YOUR_USERNAME/hyperlookup-api
	git add .
	git commit -m "Deploy HyperLookup SaaS"
	git push -u origin main
	```
	7. Hugging Face will automatically build the `Dockerfile` and launch your API on `https://YOUR_USERNAME-hyperlookup-api.hf.space`.

	---

	### 🥈 Option 2: Koyeb (Free Tier Alternative)
	1. Sign up at [Koyeb.com](https://www.koyeb.com) (Free Tier includes 512MB RAM micro instances).
	2. Connect your GitHub repository.
	3. Koyeb will detect the `Dockerfile` and auto-deploy with a global edge URL: `https://your-app.koyeb.app`.

	---

	### 🥉 Option 3: Render (Free Web Service)
	1. Go to [Render.com](https://render.com) and click "New Web Service".
	2. Connect your GitHub repository.
	3. Select Environment: Docker.
	4. Free plan: Click Deploy.

	---

	## 👑 Professional Custom Domain & Global CDN Setup (Cloudflare)

	To give your project a professional SaaS brand (e.g., `api.yourbrand.com` or `search.yourbrand.com`):

	1. Add Domain to Cloudflare (Free Plan).
	2. In Cloudflare DNS settings:
	- Type: `CNAME`
	- Name: `api` (or `@`)
	- Target: `YOUR_USERNAME-hyperlookup-api.hf.space`
	- Proxy status: Proxied (Orange Cloud).
	3. In Cloudflare SSL/TLS:
	- Set encryption mode to Full (strict).
	4. In Cloudflare Caching:
	- Create a Cache Rule for `/api/lookup*` with Edge TTL 12 hours.
	- Now repeated queries across the globe are answered from Cloudflare's 300+ Edge Data Centers in under 20 milliseconds!

	---

	## 💻 Local Development & Testing

	### 1. Install Requirements
	```bash
	pip install -r requirements.txt
	```

	### 2. Run the Server
	```bash
	uvicorn main:app --host 0.0.0.0 --port 8000 --reload
	```

	### 3. Open in Browser
	- SaaS Dashboard: [http://localhost:8000](http://localhost:8000)
	- Interactive Swagger Docs: [http://localhost:8000/docs](http://localhost:8000/docs)
	- System Health & Stats: [http://localhost:8000/api/stats](http://localhost:8000/api/stats)

	---

	## 📡 API Reference

	### 1. Auto-Detect Lookup
	Automatically identifies whether input is a 10-digit Phone or 12-digit Aadhar:
	```bash
	GET /api/lookup?q=9313336070
	```

	#### Sample JSON Response:
	```json
	{
	"success": true,
	"query_type": "phone",
	"query": "9313336070",
	"formatted_query": "+91 9313336070",
	"count": 2,
	"records": [
	{
	"name": "Aakash Jagdishbhai Makhija",
	"fathersName": "Makhija Bhai",
	"phoneNumber": "9313336070",
	"aadharNumber": null,
	"otherNumber": "7567303287",
	"address": "705/5, Ambawadi Near Police Station, Ahmedabad, Gujarat, 382475",
	"district": "Ahmedabad",
	"pincode": "382475",
	"state": "Gujarat",
	"town": null,
	"source": "inddata"
	}
	],
	"target_partitions": ["idx_phone.4.parquet"],
	"latency_ms": 218.4,
	"cached": false
	}
	```

	### 2. Phone Lookup
	```bash
	GET /api/phone/9313336070
	```

	### 3. Aadhar Lookup
	```bash
	GET /api/aadhar/351104219878
	```

	### 4. Telegram User ID Lookup (Global Master DB - 65M+ Records)
	Fast, direct partition-routed lookup for Telegram accounts by User ID:
	```bash
	GET /tg/lookup?user_id=74252719
	# or direct path:
	GET /tg/74252719
	```

	#### Sample Telegram JSON Response:
	```json
	{
	"status": "success",
	"user_id": 74252719,
	"country": "Iran",
	"num": "+989192125069",
	"phone": "+989192125069",
	"formatted_phone": "+98 919 212 5069",
	"country_code": "IR",
	"query_type": "telegram",
	"target_partition": "part_0000_1_74252719.parquet",
	"latency_ms": 14.8,
	"cached": false
	}
	```

	### 5. System Statistics
	```bash
	GET /api/stats
	```

	---

	## 📦 Client Code Examples

	### Python:
	```python
	import requests

	url = "https://your-domain.com/api/lookup"
	response = requests.get(url, params={"q": "9313336070"})
	data = response.json()

	print(f"Matches: {data['count']} in {data['latency_ms']} ms")
	for record in data['records']:
	print(record['name'], record['phoneNumber'], record['address'])
	```

	### JavaScript / Node.js:
	```javascript
	const res = await fetch("https://your-domain.com/api/lookup?q=9313336070");
	const data = await res.json();
	console.log(`Latency: ${data.latency_ms}ms`, data.records);
	```

	### cURL:
	```bash
	curl -X GET "https://your-domain.com/api/lookup?q=9313336070"
	```

	---

	## 🛡️ License & Disclaimers
	This repository is engineered for ultra-fast columnar analytical lookups using open-source tools (FastAPI, DuckDB, Hugging Face Hub).