docs/test-scripts-troubleshooting.md

# Test scripts — Azure OpenAI and TruLens troubleshooting

Ad-hoc Python scripts used to validate **Azure OpenAI** (agent / smolagents path) and **TruLens** (evaluation path) inside the **Docker backend** container. They append `/app` to `sys.path` so imports like `from app.config import get_settings` resolve when the working directory is `/app`.

## Script locations

| Script                          | Paths in this repo                                                 |
| ------------------------------- | ------------------------------------------------------------------ |
| Azure OpenAI + smolagents model | `backend/test_azure_openai.py`                                     |
| TruLens provider factory        | `test_trulens_config.py`, `backend/test_trulens_config.py`         |
| TruLens `evaluate_async` flow   | `test_trulens_evaluation.py`, `backend/test_trulens_evaluation.py` |

**Note:** The same TruLens helpers exist at the **repository root** and under **`backend/`**. Prefer a **single canonical copy** (e.g. only under `backend/`) to avoid drift; duplicates were created during troubleshooting by copying into the container.

## Prerequisites

- **Docker Compose** stack with the **backend** service running (`docker compose up -d`).
- **`.env`** loaded by Compose (`env_file: .env` in `docker-compose.yml`).
- **Azure / proxy variables** used by the app:
  - `MODEL_PROVIDER=azure_openai` (for agent tests)
  - `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_DEPLOYMENT`, `AZURE_OPENAI_API_VERSION`
- **TruLens:** `TRULENS_PROVIDER`, `TRULENS_STRATEGY`, `TRULENS_SAMPLE_RATE`, and **`TRULENS_FEEDBACK_TIMEOUT`** (seconds to wait for RAG Triad completion in the eval thread). For **`TRULENS_PROVIDER=azure_openai`**, credentials match the agent: `AZURE_OPENAI_*` and `azure_openai_api_version` in `Settings` (pydantic / `.env` as `AZURE_OPENAI_API_VERSION`).

After changing `.env`, recreate or restart the backend container so new variables are injected.

## How to run

From the repository root (adjust container name if yours differs; often `agentic-rag-backend-1`):

```bash
# Optional: copy a script from the host into the container if it is not in the image/volume
docker cp backend/test_azure_openai.py agentic-rag-backend-1:/app/test_azure_openai.py

docker compose exec backend python test_azure_openai.py
docker compose exec backend python test_trulens_config.py
docker compose exec backend python test_trulens_evaluation.py
```

If a script lives only at the repo root, copy it to `/app` first or run with an explicit path under the mounted `./backend` volume (e.g. `python /app/../` is not typical—prefer placing the script under `backend/` so it appears at `/app/test_*.py` when `./backend` is mounted to `/app`).

**Verify environment inside the container:**

```bash
docker compose exec backend env | grep -E 'AZURE_OPENAI|OPENAI_API_VERSION|TRULENS_'
```

---

## `backend/test_azure_openai.py`

**Purpose:** Confirms `Settings` and `app.agent.agent._build_model()` build a working smolagents model (`OpenAIServerModel` with `AzureOpenAI` client when `MODEL_PROVIDER=azure_openai`) and performs one **`model.generate([ChatMessage(...)]])`** call.

**Output:** Provider name, masked API key prefix, endpoint, deployment, API version; model class name; short completion text.

**Success:** `Model created successfully` and a non-empty model response.

**Typical failures:** `openai.NotFoundError` (deployment name, API version, or endpoint/proxy route mismatch), authentication errors, or missing dependencies if not run inside the backend container.

---

## `test_trulens_config.py` (and `backend/test_trulens_config.py`)

**Purpose:** Validates **`app.evaluation.trulens_eval._build_provider(settings)`** for the configured `TRULENS_PROVIDER`.

**Output:**

- `trulens_provider` and Azure deployment from settings.
- Environment visibility: `AZURE_OPENAI_API_KEY` (prefix), `AZURE_OPENAI_ENDPOINT`, **`OPENAI_API_VERSION`**.
- Whether the provider instance exists and exposes RAG Triad methods:
  - `context_relevance_with_cot_reasons`
  - `groundedness_measure_with_cot_reasons_consider_answerability`
  - `relevance_with_cot_reasons`

**Success:** Provider is not `None` and required methods are reported.

**Typical failures:** Provider is `None` when `TRULENS_PROVIDER=none`; constructor errors when Azure env/SDK requirements are not met.

---

## `test_trulens_evaluation.py` (and `backend/test_trulens_evaluation.py`)

**Purpose:** Smoke test of **`evaluate_async()`** with fixed `query`, `context`, `answer`, and `query_log_id`.

**Behavior:** Schedules work on the thread pool (returns a `Future`), then **`asyncio.wait_for(..., timeout=30)`** for completion.

**Success (script exit):** Exit code `0` when the future completes or when a timeout is treated as acceptable (the script treats timeout as OK for `TRULENS_STRATEGY=async`).

**Scores:** Printed fields may be numeric or **`nan`** if TruLens feedback calls fail at the HTTP layer (e.g. proxy returns *Route is not found* or auth errors) while the pipeline still records a `trulens_record_id`.

**Logs:** Watch backend logs for `OpenAIEndpoint request failed` and TruLens `feedback_name=... had an error during computation`.

---

## Related application code

| Area                          | File                                                                                                            |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------- |
| Agent LLM factory             | `backend/app/agent/agent.py` — `_build_model()`                                                                 |
| TruLens provider + evaluation | `backend/app/evaluation/trulens_eval.py` — `_build_provider()`, `evaluate_async()`, `_run_trulens_evaluation()` |
| Settings / `.env`             | `backend/app/config.py` — `Settings`, `get_settings()`                                                          |

**TruLens Azure note:** `_build_provider()` uses the official **`openai.AzureOpenAI`** client (same as `app.agent.agent`) and TruLens **`OpenAI`** with `client=` + `model_engine=deployment`. Shared capability flags disable TruLens’s **Responses** API path so judges use **chat completions** only (required for most Azure gateways and proxies).

---

## Security and hygiene

- Scripts print a **short prefix** of API keys for debugging. Do not commit real secrets; avoid sharing full logs publicly.
- Consider removing or relocating these scripts for production images if you do not want operational helpers in the deployment artifact.

---

## Historical note: `test_config.py`

During one troubleshooting session, a short-lived **`test_config.py`** compared raw `os.getenv` output to `Settings`; it was removed afterward. The scripts above replace that check.