.opencode/plans/implementation-plan.md

GraphRAG Knowledge AI — Implementation Plan

Project Overview

• Assignment #10: Neo4j graph DB, entity/relationship extraction, Cypher traversal, hybrid search, community detection, multi-hop reasoning, evaluation dashboard
• Stack: Django 4.2 + Neo4j + ChromaDB + LangChain LLM (Groq/Gemini)
• 118 unit tests — all PASS
• 68 manual endpoint tests — 62 pass, 6 failures explained
• All frontend files exist but are 0 bytes — frontend will be built separately with Next.js

Strategy — Phase-Based Approach

Phase 1: Backend Fixes (Current Phase)

Goal: Make backend bulletproof — handle all worst-case scenarios Approach: All 10 fixes at once, then full testing Database: SQLite with WAL mode (good enough for assignment) Background: Simple polling (raw threads + DB progress fields)

Phase 2: Deploy to HuggingFace

Goal: Get backend running on HuggingFace Spaces Changes: Switch to PostgreSQL, add proper CORS, configure for HF deployment

Phase 3: Frontend Build

Goal: Complete professional frontend with real-time processing status Framework: Next.js (as mentioned in assignment) Features: Real-time progress polling, graph visualization, multi-hop reasoning UI

Phase 4: Full Celery + Redis Migration

Goal: Production-grade background task handling When: Only after Phase 1-3 work perfectly end-to-end Changes: Add Redis, Celery worker, Celery beat, proper task queue

---

Phase 1: Backend Fixes — Implementation Plan

Fix List (All 10 — Implement Together)

#	Fix	File(s)	Time	Status
1	Startup recovery — reset stuck PROCESSING docs	graphrag/apps.py (create)	30 min	PENDING
2	Gunicorn timeout 120→300s	Dockerfile:45	1 min	PENDING
3	LLM extractors raise on failure (not return [])	entity_extractor.py, relationship_extractor.py	1 hour	PENDING
4	Check API key before starting ingestion thread	views.py (trigger_ingestion_background)	30 min	PENDING
5	Add tenacity retry with exponential backoff	entity_extractor.py, relationship_extractor.py, neo4j_client.py	2 hours	PENDING
6	UUID-based ChromaDB IDs	vector_retriever.py:61	30 min	PENDING
7	Entity resolver LLM disambiguation	entity_resolver.py	1 hour	PENDING
8	Login empty body returns 400 not 401	views.py:140-145	15 min	PENDING
9	Empty relationship type fallback to RELATED_TO	neo4j_client.py:114,119	5 min	PENDING
10	Community endpoints graceful degradation	community_detector.py:38	15 min	PENDING

Progress Tracking (Simple Polling)

Add to Document model:

• processing_progress (IntegerField, 0-100)
• processing_step (CharField, max 200 chars)

Update graph_builder.py to set progress at each step:

1. Parsing document → 5%
2. Indexing vectors → 20%
3. Extracting entities → 20-80% (per section)
4. Extracting relationships → 20-80% (per section)
5. Resolving duplicates → 85%
6. Building knowledge graph → 95%
7. Complete → 100%

Frontend polling: GET /api/documents/{id}/ every 3 seconds

Implementation Order

Batch 1 — Quick Wins (30 min total):

1. Fix #2 — Gunicorn timeout (1 min)
2. Fix #9 — Empty rel type fallback (5 min)
3. Fix #8 — Login 400 (15 min)
4. Fix #10 — Community graceful degradation (15 min)

Batch 2 — Medium Complexity (1.5 hours): 5. Fix #6 — UUID ChromaDB IDs (30 min) 6. Fix #1 — Startup recovery apps.py (30 min) 7. Fix #4 — API key check (30 min)

Batch 3 — Complex (4 hours): 8. Fix #3 — Extractors raise instead of swallow (1 hour) 9. Fix #5 — Tenacity retry (2 hours) 10. Fix #7 — Entity resolver LLM disambiguation (1 hour)

Batch 4 — Progress Tracking (1.5 hours): 11. Add progress fields to Document model (30 min) 12. Update graph_builder.py with progress steps (1 hour)

Batch 5 — Testing (2 hours): 13. Run all 118 existing tests (5 min) 14. Write new tests for all fixes (2 hours)

Total: ~9.5 hours

Testing Strategy

After each batch:

1. Run python manage.py test graphrag.tests graphrag.tests_comprehensive (all 118 tests)
2. Run manual endpoint tests (verify nothing regressed)
3. Test worst-case scenarios:
   • Kill server mid-ingestion → restart → verify doc reset to FAILED
   • No API keys → upload → verify immediate FAILED
   • Empty login body → verify 400 response

After all batches:

1. Full 68-test manual verification
2. New tests for each fix
3. Load test with concurrent uploads (if possible)

---

Worst-Case Scenarios Handled

Scenario	Current Behavior	After Fix	Fix #
Server restarts mid-ingestion	Document stuck PROCESSING forever	Auto-reset to FAILED on startup	#1
Gunicorn kills worker at 120s	Ingestion thread dies	Timeout increased to 300s	#2
LLM API returns 500	Empty entities, document "completed"	Exception raised, document FAILED	#3
LLM API returns 429 (rate limit)	Section skipped silently	Retry 3 times with backoff	#5
No API keys configured	Document stuck PENDING	Immediate FAILED with error	#4
Same filename uploaded twice	ChromaDB vectors overwritten	UUID-based IDs prevent overwrite	#6
Neo4j goes down mid-ingestion	Partial graph data	Cleanup on failure + FAILED status	#3
Neo4j down during query	500 error	Graceful empty response	#10
LLM returns garbage relationship type	Invalid Cypher	Fallback to RELATED_TO	#9
"J. Smith" vs "John Smith"	Not merged (borderline score)	LLM disambiguation	#7
Frontend refreshes during processing	Shows stale status	Polling every 3s gets new progress	New

---

File Change List

File	Change	Lines Changed
graphrag/apps.py	CREATE — Startup recovery	~25 lines
graphrag/models.py	Add processing_progress, processing_step fields	~5 lines
graphrag/views.py	Fix #4 (API key check), Fix #8 (login 400)	~15 lines
graphrag/services/entity_extractor.py	Fix #3 (raise), Fix #5 (tenacity retry)	~15 lines
graphrag/services/relationship_extractor.py	Fix #3 (raise), Fix #5 (tenacity retry)	~15 lines
graphrag/services/graph_builder.py	Add progress tracking at each step	~30 lines
graphrag/services/neo4j_client.py	Fix #9 (empty rel type), Fix #5 (tenacity)	~10 lines
graphrag/services/entity_resolver.py	Fix #7 (LLM disambiguation)	~30 lines
graphrag/services/community_detector.py	Fix #10 (graceful degradation)	~5 lines
graphrag/services/vector_retriever.py	Fix #6 (UUID-based IDs)	~3 lines
graphrag/serializers.py	Add new fields to DocumentSerializer	~3 lines
Dockerfile	Fix #2 (timeout 300s)	~1 line

Total: 12 files, ~160 lines of changes

---

Commands to Run

Run all existing tests:

cd /home/creator/Desktop/ExcellenceTechnology/07.graphrag-knowledge-ai/backend
python manage.py test graphrag.tests graphrag.tests_comprehensive

Run manual endpoint tests:

cd /home/creator/Desktop/ExcellenceTechnology/07.graphrag-knowledge-ai
python manual_test.py

Check Docker services:

docker compose ps
docker compose logs backend --tail=50

---

Notes

• tenacity is already in requirements.txt but never imported/used
• No Redis/Celery yet — will add in Phase 4
• Frontend is 0 bytes across all 14 pages — will build with Next.js in Phase 3
• Deploy to HuggingFace in Phase 2 (after Phase 1 fixes work perfectly)
• Always test fixes against real Neo4j and ChromaDB when possible
• Keep all 118 existing tests passing — never break them
• Document every fix with before/after code in commit messages