docs/github-issues-draft.md

GitHub Issue Drafts (Complete Set)

You can copy and paste the following markdown directly into new GitHub Issues to track the architectural migration, codebase optimization, and SaaS evolution of Talos.

---

Issue 1: Consolidate Database Repositories into talos_db package

Labels: refactor, architecture, tech-debt

Description

Currently, database access logic is fragmented. The services/backend has its own repositories/config_repository.py, while packages/db/src/talos_db already contains a Supabase client and a repositories/ directory. To maintain a clean monorepo architecture, services should not contain direct database access logic.

Acceptance Criteria

• Move services/backend/repositories/config_repository.py to packages/db/src/talos_db/repositories/.
• Update all imports in services/backend to import the repository from talos_db.
• Ensure services/backend no longer has a repositories/ directory.

---

Issue 2: Migrate Search Execution Logic to talos_search package

Labels: refactor, architecture

Description

The search domain is currently split. packages/search/src/talos_search handles low-level Typesense client/indexer initialization, but all the actual query execution logic (hybrid.py, keyword.py, embeddings.py, vector.py) is locked inside services/backend/search/.

To allow other services (like talos-manager) to execute searches without duplicating code, all search logic must be consolidated into the shared package.

Acceptance Criteria

• Move the entire services/backend/search/ directory into packages/search/src/talos_search/.
• Update all routing and dependency imports in services/backend/api/routers/ to point to the new package location.
• Ensure services/backend no longer contains a search/ domain directory.

---

Issue 3: Extract AI Engines into talos_ai package

Labels: refactor, ai

Description

The core AI business logic (intent_engine, chat_engine, recommendation_engine) is tightly coupled to services/backend/. The packages/ai/src/talos_ai package currently only holds the base llm.py wrapper.

To enable talos-manager or background workers to simulate chats, test intents, and evaluate recommendations, these engines must be abstracted into the shared AI package.

Acceptance Criteria

• Move intent_engine, chat_engine, and recommendation_engine from services/backend/ into packages/ai/src/talos_ai/engines/.
• Verify that all internal FastAPI routers correctly import the engines from talos_ai.

---

Issue 4: Unify Configuration & Environment Loading

Labels: refactor, tech-debt

Description

There is fragmentation in how environment variables and configuration files are loaded across the monorepo (e.g., services/backend/core/config.py vs services/talos-manager/core/settings.py). This risks staging/production environments using mismatched settings across different services.

Acceptance Criteria

• Create a unified packages/config module utilizing Pydantic BaseSettings.
• Move all environment variable parsing (Supabase URLs, Typesense keys, LLM keys) to this central package.
• Refactor both backend and talos-manager to import their base configurations from packages/config.

---

Issue 5: Implement Provider Pattern for Search Engine (Abstract Typesense)

Labels: architecture, feature, search

Description

The SearchService currently has hardcoded dependencies on Typesense, executing Typesense-specific syntax and relying on the Typesense client. To prepare for future SaaS scale and allow hot-swapping of the search engine (e.g., to PostgreSQL or a custom Rust engine), we must abstract this into a Provider Pattern.

Acceptance Criteria

• Define an abstract base class BaseSearchBackend defining required methods (search, search_candidates, known_facet_values).
• Create TypesenseSearchBackend that implements this interface.
• Update SearchService to accept BaseSearchBackend via dependency injection, completely decoupling it from Typesense.

---

Issue 6: Implement Redis for Semantic Caching & Sessions

Labels: performance, architecture

Description

Talos currently relies on in-memory singletons (like the Refinement Engine's SessionManager) and lacks caching for expensive LLM intent-parsing calls. This prevents horizontal scaling of the FastAPI backend.

Acceptance Criteria

• Deploy a Redis container in docker-compose.yaml.
• Refactor SessionManager to serialize and store active refinement sessions in Redis using session_id as the key.
• Implement a semantic caching layer for POST /ai/recommend so identical semantic queries pull the parsed OntologyEntry from Redis rather than hitting the LLM.

---

Issue 7: Dynamic Header-Based Multi-Tenancy

Labels: architecture, security

Description

Talos currently operates in a multi-instance paradigm where one deployment serves one client by loading a COMPANY_ID environment variable at startup. To function as a true SaaS, the API must dynamically serve multiple clients simultaneously.

Acceptance Criteria

• Add an X-Client-ID (or API Key) header dependency to all relevant FastAPI endpoints.
• Refactor the application to fetch configuration dynamically from Supabase at runtime based on the client header (using local LRU cache/Redis to prevent database spam).
• Implement scoped search so Typesense queries are strictly isolated to the requesting client's collection prefix.

---

Issue 8: Setup Strict Linting, Formatting, and Pre-commit Hooks

Labels: tooling, dx

Description

To ensure code quality and prevent styling debates during code reviews, we need to enforce strict formatting and linting rules across both the Python and TypeScript codebases automatically.

Acceptance Criteria

• Add ruff and black to the Python backend requirements.
• Ensure mypy is configured for strict static type checking in Python.
• Add prettier and integrate it with ESLint in the frontend.
• Configure pre-commit hooks (via .pre-commit-config.yaml) to run all formatters and linters automatically before a commit is allowed.

---

Issue 9: Setup CI/CD Pipeline with GitHub Actions

Labels: ci-cd, infrastructure

Description

We currently lack an automated continuous integration pipeline. We need to ensure that tests, type checks, and linting rules are enforced on every Pull Request before code can be merged into main.

Acceptance Criteria

• Create a .github/workflows/ci.yml file.
• Configure jobs to run pytest for the backend and vitest for the frontend.
• Configure jobs to run ESLint, Ruff, Black, and mypy on every PR.
• Block merging if any of the CI checks fail.

---

Issue 10: Implement Monorepo Build Tooling (Turborepo)

Labels: tooling, infrastructure

Description

As the number of shared packages (e.g., talos_db, talos_search) and apps grows, manually managing dependencies and build scripts becomes unmanageable. We need a proper monorepo build system to cache outputs and run tasks in parallel.

Acceptance Criteria

• Initialize turborepo (or nx) at the root of the project.
• Configure turbo.json with build, lint, and test pipelines.
• Ensure npm run dev or turbo run dev seamlessly starts all frontend apps and backend services with hot-reloading.

---

Issue 11: Optimize Frontend Rendering & Code Splitting

Labels: performance, frontend

Description

To ensure the frontend remains snappy as the application grows, we need to implement React performance best practices to avoid unnecessary re-renders and bloated bundle sizes.

Acceptance Criteria

• Wrap heavy or off-screen components in React.lazy() for code splitting.
• Audit the workflow builder and properties sidebar to ensure useMemo and useCallback are used appropriately to prevent unnecessary re-renders on drag-and-drop actions.
• Resolve any strict-mode dependency warnings.

---

Issue 12: Implement Learning-to-Rank Telemetry Pipeline

Labels: ai, search, feature

Description

Talos's search accuracy is currently deterministic based on static YAML weights. To achieve SaaS-level accuracy, we must implement an automated feedback loop (Learning-to-Rank) that analyzes user interactions to adjust product scores dynamically.

Acceptance Criteria

• Utilize the existing /telemetry/action endpoint to track Click-Through Rates (CTR) and conversion events per search query.
• Create a background pipeline (cron job or Celery worker) that aggregates telemetry data.
• Automatically penalize or boost scoring weights in the search index if a product consistently underperforms or overperforms for a specific semantic intent.

---

Issue 13: EPIC: Evaluate Core Search Engine Alternatives

Labels: epic, architecture, research

Description

(Long-term Strategic Goal) To truly compete with Algolia and Typesense as a core Search Infrastructure SaaS, we cannot rely on Python to execute our core inverted index and vector similarity math due to the Global Interpreter Lock (GIL). We must evaluate migrating the underlying search execution engine.

Acceptance Criteria

• Research Spike 1: Benchmark PostgreSQL utilizing pgvector and pg_search (Tantivy/BM25) against our current Typesense setup for latency at scale.
• Research Spike 2: Evaluate wrapping Apache Lucene or OpenSearch with our AI routing logic.
• Research Spike 3: Scope the engineering effort required to build a custom memory-mapped inverted index and HNSW graph in Rust.
• Draft an Architectural Decision Record (ADR) detailing the chosen path forward.