docs/test-scripts-troubleshooting.md

Test scripts — Azure OpenAI and TruLens troubleshooting

Ad-hoc Python scripts used to validate Azure OpenAI (agent / smolagents path) and TruLens (evaluation path) inside the Docker backend container. They append /app to sys.path so imports like from app.config import get_settings resolve when the working directory is /app.

Script locations

Script	Paths in this repo
Azure OpenAI + smolagents model	backend/test_azure_openai.py
TruLens provider factory	test_trulens_config.py, backend/test_trulens_config.py
TruLens evaluate_async flow	test_trulens_evaluation.py, backend/test_trulens_evaluation.py

Note: The same TruLens helpers exist at the repository root and under backend/. Prefer a single canonical copy (e.g. only under backend/) to avoid drift; duplicates were created during troubleshooting by copying into the container.

Prerequisites

• Docker Compose stack with the backend service running (docker compose up -d).
• .env loaded by Compose (env_file: .env in docker-compose.yml).
• Azure / proxy variables used by the app:
  • MODEL_PROVIDER=azure_openai (for agent tests)
  • AZURE_OPENAI_API_KEY, AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_DEPLOYMENT, AZURE_OPENAI_API_VERSION
• TruLens: TRULENS_PROVIDER, TRULENS_STRATEGY, TRULENS_SAMPLE_RATE, and TRULENS_FEEDBACK_TIMEOUT (seconds to wait for RAG Triad completion in the eval thread). For TRULENS_PROVIDER=azure_openai, credentials match the agent: AZURE_OPENAI_* and azure_openai_api_version in Settings (pydantic / .env as AZURE_OPENAI_API_VERSION).

After changing .env, recreate or restart the backend container so new variables are injected.

How to run

From the repository root (adjust container name if yours differs; often agentic-rag-backend-1):

# Optional: copy a script from the host into the container if it is not in the image/volume
docker cp backend/test_azure_openai.py agentic-rag-backend-1:/app/test_azure_openai.py

docker compose exec backend python test_azure_openai.py
docker compose exec backend python test_trulens_config.py
docker compose exec backend python test_trulens_evaluation.py

If a script lives only at the repo root, copy it to /app first or run with an explicit path under the mounted ./backend volume (e.g. python /app/../ is not typical—prefer placing the script under backend/ so it appears at /app/test_*.py when ./backend is mounted to /app).

Verify environment inside the container:

docker compose exec backend env | grep -E 'AZURE_OPENAI|OPENAI_API_VERSION|TRULENS_'

---

backend/test_azure_openai.py

Purpose: Confirms Settings and app.agent.agent._build_model() build a working smolagents model (OpenAIServerModel with AzureOpenAI client when MODEL_PROVIDER=azure_openai) and performs one model.generate([ChatMessage(...)]]) call.

Output: Provider name, masked API key prefix, endpoint, deployment, API version; model class name; short completion text.

Success: Model created successfully and a non-empty model response.

Typical failures: openai.NotFoundError (deployment name, API version, or endpoint/proxy route mismatch), authentication errors, or missing dependencies if not run inside the backend container.

---

test_trulens_config.py (and backend/test_trulens_config.py)

Purpose: Validates app.evaluation.trulens_eval._build_provider(settings) for the configured TRULENS_PROVIDER.

Output:

• trulens_provider and Azure deployment from settings.
• Environment visibility: AZURE_OPENAI_API_KEY (prefix), AZURE_OPENAI_ENDPOINT, OPENAI_API_VERSION.
• Whether the provider instance exists and exposes RAG Triad methods:
  • context_relevance_with_cot_reasons
  • groundedness_measure_with_cot_reasons_consider_answerability
  • relevance_with_cot_reasons

Success: Provider is not None and required methods are reported.

Typical failures: Provider is None when TRULENS_PROVIDER=none; constructor errors when Azure env/SDK requirements are not met.

---

test_trulens_evaluation.py (and backend/test_trulens_evaluation.py)

Purpose: Smoke test of evaluate_async() with fixed query, context, answer, and query_log_id.

Behavior: Schedules work on the thread pool (returns a Future), then asyncio.wait_for(..., timeout=30) for completion.

Success (script exit): Exit code 0 when the future completes or when a timeout is treated as acceptable (the script treats timeout as OK for TRULENS_STRATEGY=async).

Scores: Printed fields may be numeric or nan if TruLens feedback calls fail at the HTTP layer (e.g. proxy returns Route is not found or auth errors) while the pipeline still records a trulens_record_id.

Logs: Watch backend logs for OpenAIEndpoint request failed and TruLens feedback_name=... had an error during computation.

---

Related application code

Area	File
Agent LLM factory	backend/app/agent/agent.py — _build_model()
TruLens provider + evaluation	backend/app/evaluation/trulens_eval.py — _build_provider(), evaluate_async(), _run_trulens_evaluation()
Settings / .env	backend/app/config.py — Settings, get_settings()

TruLens Azure note: _build_provider() uses the official openai.AzureOpenAI client (same as app.agent.agent) and TruLens OpenAI with client= + model_engine=deployment. Shared capability flags disable TruLens’s Responses API path so judges use chat completions only (required for most Azure gateways and proxies).

---

Security and hygiene

• Scripts print a short prefix of API keys for debugging. Do not commit real secrets; avoid sharing full logs publicly.
• Consider removing or relocating these scripts for production images if you do not want operational helpers in the deployment artifact.

---

Historical note: test_config.py

During one troubleshooting session, a short-lived test_config.py compared raw os.getenv output to Settings; it was removed afterward. The scripts above replace that check.