Spaces:

GodRad
/

rag_saas_system

Sleeping

App Files Files Community

rag_saas_system / OPERATOR_GUIDE.md

GodRad

Upload 51 files

0c8aadf verified 9 months ago

preview code

Raw History Blame Contribute Delete

7.86 kB

# RAG Platform Operator Guide

## Overview

This guide explains how to operate the hardened RAG SaaS platform after implementing the 7 critical fixes.

## For Tenants

### 1. Creating a Tenant Account

```bash
curl -X POST http://localhost:7860/v1/auth/tenants \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Acme Corporation",
    "admin_email": "admin@acme.com",
    "admin_password": "secure_password"
  }'
```

**Response:**
```json
{
  "tenant_id": "abc123",
  "tenant_name": "Acme Corporation",
  "message": "Tenant created successfully. Use tenant_id 'abc123' for all future logins."
}
```

**IMPORTANT:** Save your `tenant_id` - you'll need it for all future logins.

---

### 2. Logging In

```bash
curl -X POST http://localhost:7860/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{
    "tenant_id": "abc123",
    "username": "admin@acme.com",
    "password": "secure_password"
  }'
```

**Response:**
```json
{
  "access_token": "abc123:admin@acme.com:token...",
  "token_type": "bearer",
  "tenant_id": "abc123",
  "tenant_name": "Acme Corporation"
}
```

---

### 3. Adding Sources

#### Document Upload
```bash
curl -X POST http://localhost:7860/v1/sources/ \
  -H "X-Tenant-ID: abc123" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "doc1",
    "tenant_id": "abc123",
    "type": "file",
    "name": "Company Handbook",
    "uri": "/uploads/handbook.pdf",
    "metadata": {}
  }'
```

#### Web URL
```bash
curl -X POST http://localhost:7860/v1/sources/ \
  -H "X-Tenant-ID: abc123" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "url1",
    "tenant_id": "abc123",
    "type": "web",
    "name": "Company Blog",
    "uri": "https://acme.com/blog",
    "metadata": {}
  }'
```

#### YouTube Video (with transcript upload)
```bash
curl -X POST http://localhost:7860/v1/sources/youtube \
  -H "X-Tenant-ID: abc123" \
  -F "youtube_url=https://youtube.com/watch?v=abc123" \
  -F "name=Product Demo" \
  -F "owner_attestation=true" \
  -F "transcript_file=@transcript.vtt"
```

**Note:** If you don't have a transcript file, the system will attempt to use YouTube captions.

---

### 4. Managing Sources

#### List All Sources
```bash
curl -X GET http://localhost:7860/v1/sources/ \
  -H "X-Tenant-ID: abc123"
```

#### List Active Sources Only
```bash
curl -X GET http://localhost:7860/v1/sources/active \
  -H "X-Tenant-ID: abc123"
```

#### Enable/Disable a Source
```bash
# Disable a source
curl -X PUT "http://localhost:7860/v1/sources/doc1/toggle?enabled=false" \
  -H "X-Tenant-ID: abc123"

# Re-enable a source
curl -X PUT "http://localhost:7860/v1/sources/doc1/toggle?enabled=true" \
  -H "X-Tenant-ID: abc123"
```

**Important:** Toggling sources is **additive**, not exclusive. Disabling one source does NOT affect others.

---

### 5. Asking Questions

```bash
curl -X POST http://localhost:7860/v1/ask/ \
  -H "X-Tenant-ID: abc123" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "What is our return policy?",
    "top_k": 10
  }'
```

**Response:**
```json
{
  "answer": "According to our company handbook...",
  "citations": [
    {
      "source_type": "file",
      "document_name": "Company Handbook",
      "page": 15
    },
    {
      "source_type": "web",
      "url": "https://acme.com/blog/returns"
    }
  ],
  "confidence": 0.92,
  "sources_used": ["file", "web"],
  "retrieval_time_ms": 45.2,
  "generation_time_ms": 320.5
}
```

**Note:** The system automatically retrieves from ALL active sources (documents + URLs + YouTube).

---

### 6. Viewing Your Metrics

```bash
curl -X GET "http://localhost:7860/v1/tenants/abc123/metrics?hours=24" \
  -H "X-Tenant-ID: abc123"
```

**Response:**
```json
{
  "tenant_id": "abc123",
  "counters": {
    "retrieval_total": 150,
    "generation_total": 150,
    "ingestion_total": 25,
    "sources_created": 10
  },
  "time_series": {
    "retrieval_latency_ms": [
      {"timestamp": "2026-02-10T10:00:00", "value": 45.2, "labels": {}},
      ...
    ]
  }
}
```

**Security:** You can only view your own tenant's metrics.

---

## For Administrators

### 1. Viewing Global Metrics

```bash
curl -X GET http://localhost:7860/v1/admin/metrics
```

**Response:**
```json
{
  "global": {
    "counters": {
      "retrieval_total": 5000,
      "generation_total": 5000,
      "ingestion_total": 500
    },
    "total_tenants": 25
  },
  "by_tenant": [
    {
      "tenant_id": "abc123",
      "metrics": {
        "retrieval_total": 150,
        "generation_total": 150
      }
    },
    ...
  ]
}
```

---

### 2. Viewing Model Cache Statistics

```bash
curl -X GET http://localhost:7860/v1/metrics/cache-stats
```

**Response:**
```json
{
  "cache_size": 450,
  "cache_max_size": 1000,
  "cache_hits": 2500,
  "cache_misses": 800,
  "hit_rate": 0.757,
  "model_load_time_seconds": 12.5
}
```

---

## System Architecture

### Multi-Source Retrieval Flow

```
User Query
    ↓
Query Rewrite (optional)
    ↓
Parallel Retrieval:
    ├─ Documents Search
    ├─ URLs Search
    └─ YouTube Search
    ↓
Score Normalization
    ↓
RRF Fusion (Reciprocal Rank Fusion)
    ↓
Diversity Filter
    ↓
Top K Results
    ↓
Answer Generation with Citations
```

### Model Lifecycle

- **Startup:** Models load once during server startup
- **Runtime:** All requests reuse the same model instances
- **Caching:** Embeddings are cached with LRU eviction
- **Performance:** Typical latency reduction: 70-90%

---

## Troubleshooting

### "Tenant not found" error
- Verify you're using the correct `tenant_id` from account creation
- Check that you're including the `X-Tenant-ID` header

### No results from YouTube sources
- Verify the video has captions available, or upload a transcript file
- Check that the source is enabled: `GET /v1/sources/active`

### Slow first request
- This is normal - models load on first use
- Subsequent requests will be much faster due to caching

### Missing source types in results
- Verify all sources are enabled: `GET /v1/sources/active`
- Check that content has been ingested for each source

---

## Best Practices

1. **Source Management:**
   - Keep sources enabled unless you have a specific reason to disable them
   - Use descriptive names for sources to aid in citation tracking

2. **Query Optimization:**
   - Use `top_k=10-20` for best balance of quality and speed
   - More sources = slightly higher latency but better coverage

3. **Metrics Monitoring:**
   - Check metrics regularly to understand usage patterns
   - Monitor `retrieval_latency_ms` for performance issues

4. **YouTube Ingestion:**
   - Always provide transcript files when available for best quality
   - Set `owner_attestation=true` to confirm you have rights to the content

---

## API Endpoint Reference

| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/v1/auth/tenants` | Create tenant account |
| POST | `/v1/auth/login` | Login with tenant_id |
| POST | `/v1/sources/` | Create source |
| GET | `/v1/sources/` | List all sources |
| GET | `/v1/sources/active` | List active sources |
| PUT | `/v1/sources/{id}/toggle` | Enable/disable source |
| POST | `/v1/sources/youtube` | Create YouTube source |
| POST | `/v1/ask/` | Ask question |
| GET | `/v1/tenants/{id}/metrics` | Get tenant metrics |
| GET | `/v1/admin/metrics` | Get global metrics (admin) |
| GET | `/v1/metrics/cache-stats` | Get cache statistics |

---

## Support

For issues or questions, contact your system administrator.

	# RAG Platform Operator Guide

	## Overview

	This guide explains how to operate the hardened RAG SaaS platform after implementing the 7 critical fixes.

	## For Tenants

	### 1. Creating a Tenant Account

	```bash
	curl -X POST http://localhost:7860/v1/auth/tenants \
	-H "Content-Type: application/json" \
	-d '{
	"name": "Acme Corporation",
	"admin_email": "admin@acme.com",
	"admin_password": "secure_password"
	}'
	```

	Response:
	```json
	{
	"tenant_id": "abc123",
	"tenant_name": "Acme Corporation",
	"message": "Tenant created successfully. Use tenant_id 'abc123' for all future logins."
	}
	```

	IMPORTANT: Save your `tenant_id` - you'll need it for all future logins.

	---

	### 2. Logging In

	```bash
	curl -X POST http://localhost:7860/v1/auth/login \
	-H "Content-Type: application/json" \
	-d '{
	"tenant_id": "abc123",
	"username": "admin@acme.com",
	"password": "secure_password"
	}'
	```

	Response:
	```json
	{
	"access_token": "abc123:admin@acme.com:token...",
	"token_type": "bearer",
	"tenant_id": "abc123",
	"tenant_name": "Acme Corporation"
	}
	```

	---

	### 3. Adding Sources

	#### Document Upload
	```bash
	curl -X POST http://localhost:7860/v1/sources/ \
	-H "X-Tenant-ID: abc123" \
	-H "Content-Type: application/json" \
	-d '{
	"id": "doc1",
	"tenant_id": "abc123",
	"type": "file",
	"name": "Company Handbook",
	"uri": "/uploads/handbook.pdf",
	"metadata": {}
	}'
	```

	#### Web URL
	```bash
	curl -X POST http://localhost:7860/v1/sources/ \
	-H "X-Tenant-ID: abc123" \
	-H "Content-Type: application/json" \
	-d '{
	"id": "url1",
	"tenant_id": "abc123",
	"type": "web",
	"name": "Company Blog",
	"uri": "https://acme.com/blog",
	"metadata": {}
	}'
	```

	#### YouTube Video (with transcript upload)
	```bash
	curl -X POST http://localhost:7860/v1/sources/youtube \
	-H "X-Tenant-ID: abc123" \
	-F "youtube_url=https://youtube.com/watch?v=abc123" \
	-F "name=Product Demo" \
	-F "owner_attestation=true" \
	-F "transcript_file=@transcript.vtt"
	```

	Note: If you don't have a transcript file, the system will attempt to use YouTube captions.

	---

	### 4. Managing Sources

	#### List All Sources
	```bash
	curl -X GET http://localhost:7860/v1/sources/ \
	-H "X-Tenant-ID: abc123"
	```

	#### List Active Sources Only
	```bash
	curl -X GET http://localhost:7860/v1/sources/active \
	-H "X-Tenant-ID: abc123"
	```

	#### Enable/Disable a Source
	```bash
	# Disable a source
	curl -X PUT "http://localhost:7860/v1/sources/doc1/toggle?enabled=false" \
	-H "X-Tenant-ID: abc123"

	# Re-enable a source
	curl -X PUT "http://localhost:7860/v1/sources/doc1/toggle?enabled=true" \
	-H "X-Tenant-ID: abc123"
	```

	Important: Toggling sources is additive, not exclusive. Disabling one source does NOT affect others.

	---

	### 5. Asking Questions

	```bash
	curl -X POST http://localhost:7860/v1/ask/ \
	-H "X-Tenant-ID: abc123" \
	-H "Content-Type: application/json" \
	-d '{
	"query": "What is our return policy?",
	"top_k": 10
	}'
	```

	Response:
	```json
	{
	"answer": "According to our company handbook...",
	"citations": [
	{
	"source_type": "file",
	"document_name": "Company Handbook",
	"page": 15
	},
	{
	"source_type": "web",
	"url": "https://acme.com/blog/returns"
	}
	],
	"confidence": 0.92,
	"sources_used": ["file", "web"],
	"retrieval_time_ms": 45.2,
	"generation_time_ms": 320.5
	}
	```

	Note: The system automatically retrieves from ALL active sources (documents + URLs + YouTube).

	---

	### 6. Viewing Your Metrics

	```bash
	curl -X GET "http://localhost:7860/v1/tenants/abc123/metrics?hours=24" \
	-H "X-Tenant-ID: abc123"
	```

	Response:
	```json
	{
	"tenant_id": "abc123",
	"counters": {
	"retrieval_total": 150,
	"generation_total": 150,
	"ingestion_total": 25,
	"sources_created": 10
	},
	"time_series": {
	"retrieval_latency_ms": [
	{"timestamp": "2026-02-10T10:00:00", "value": 45.2, "labels": {}},
	...
	]
	}
	}
	```

	Security: You can only view your own tenant's metrics.

	---

	## For Administrators

	### 1. Viewing Global Metrics

	```bash
	curl -X GET http://localhost:7860/v1/admin/metrics
	```

	Response:
	```json
	{
	"global": {
	"counters": {
	"retrieval_total": 5000,
	"generation_total": 5000,
	"ingestion_total": 500
	},
	"total_tenants": 25
	},
	"by_tenant": [
	{
	"tenant_id": "abc123",
	"metrics": {
	"retrieval_total": 150,
	"generation_total": 150
	}
	},
	...
	]
	}
	```

	---

	### 2. Viewing Model Cache Statistics

	```bash
	curl -X GET http://localhost:7860/v1/metrics/cache-stats
	```

	Response:
	```json
	{
	"cache_size": 450,
	"cache_max_size": 1000,
	"cache_hits": 2500,
	"cache_misses": 800,
	"hit_rate": 0.757,
	"model_load_time_seconds": 12.5
	}
	```

	---

	## System Architecture

	### Multi-Source Retrieval Flow

	```
	User Query
	↓
	Query Rewrite (optional)
	↓
	Parallel Retrieval:
	├─ Documents Search
	├─ URLs Search
	└─ YouTube Search
	↓
	Score Normalization
	↓
	RRF Fusion (Reciprocal Rank Fusion)
	↓
	Diversity Filter
	↓
	Top K Results
	↓
	Answer Generation with Citations
	```

	### Model Lifecycle

	- Startup: Models load once during server startup
	- Runtime: All requests reuse the same model instances
	- Caching: Embeddings are cached with LRU eviction
	- Performance: Typical latency reduction: 70-90%

	---

	## Troubleshooting

	### "Tenant not found" error
	- Verify you're using the correct `tenant_id` from account creation
	- Check that you're including the `X-Tenant-ID` header

	### No results from YouTube sources
	- Verify the video has captions available, or upload a transcript file
	- Check that the source is enabled: `GET /v1/sources/active`

	### Slow first request
	- This is normal - models load on first use
	- Subsequent requests will be much faster due to caching

	### Missing source types in results
	- Verify all sources are enabled: `GET /v1/sources/active`
	- Check that content has been ingested for each source

	---

	## Best Practices

	1. Source Management:
	- Keep sources enabled unless you have a specific reason to disable them
	- Use descriptive names for sources to aid in citation tracking

	2. Query Optimization:
	- Use `top_k=10-20` for best balance of quality and speed
	- More sources = slightly higher latency but better coverage

	3. Metrics Monitoring:
	- Check metrics regularly to understand usage patterns
	- Monitor `retrieval_latency_ms` for performance issues

	4. YouTube Ingestion:
	- Always provide transcript files when available for best quality
	- Set `owner_attestation=true` to confirm you have rights to the content

	---

	## API Endpoint Reference

	\| Method \| Endpoint \| Description \|
	\|--------\|----------\|-------------\|
	\| POST \| `/v1/auth/tenants` \| Create tenant account \|
	\| POST \| `/v1/auth/login` \| Login with tenant_id \|
	\| POST \| `/v1/sources/` \| Create source \|
	\| GET \| `/v1/sources/` \| List all sources \|
	\| GET \| `/v1/sources/active` \| List active sources \|
	\| PUT \| `/v1/sources/{id}/toggle` \| Enable/disable source \|
	\| POST \| `/v1/sources/youtube` \| Create YouTube source \|
	\| POST \| `/v1/ask/` \| Ask question \|
	\| GET \| `/v1/tenants/{id}/metrics` \| Get tenant metrics \|
	\| GET \| `/v1/admin/metrics` \| Get global metrics (admin) \|
	\| GET \| `/v1/metrics/cache-stats` \| Get cache statistics \|

	---

	## Support

	For issues or questions, contact your system administrator.