docs/stack-and-dependencies.md

Stack and Dependencies

Full list of the technology stack, Python packages, version pins, and the recommended upgrade path.

Stack at a glance

Layer	Technology	Why
Language	Python 3.11	Pinned; required by some typing features used in the code
UI framework	Streamlit	Inherited from v1; suits the data-analyst audience
Agent orchestration	LangGraph + LangChain	LangGraph for the state machine, LangChain for the message/prompt abstractions and LLM/retriever clients
LLM provider	OpenAI (production), Mistral/Ollama/OpenRouter (configurable)	See ADR 004
Vector database	Qdrant Cloud	Established before this rebuild; the corpus + payload indexes live there
Embedding model	BAAI/bge-m3 (dense, multilingual, 8K context)	Open-source; runs on CPU
Reranker model	BAAI/bge-reranker-v2-m3 (cross-encoder)	Open-source; runs on CPU; conditionally skipped (ADR 003)
PDF parsing	Docling (in ingestion scripts, not in live system)	Layout-aware chunking
Container	Docker, python:3.11-slim base	Standard ML deployment pattern
Deployment	Hugging Face Spaces (AWS underneath)	See ADR 005
Test framework	pytest	Standard
Dep management	requirements.txt (HF Spaces reads this) + pyproject.toml (project metadata) + uv.lock (full reproducibility)	See repo root files

Direct dependencies (declared in requirements.txt)

Package	Version	Why
pydantic	>=2.0.0	Required by LangChain. Floor pin; transitively pinned by uv.lock.
torch	>=2.0.0	Required by sentence-transformers for model inference. CPU-only build sufficient.
numpy	>=1.24.0	Scientific computing baseline.
pandas	>=2.0.0	Used in app.py for UI charts (chunk statistics).
FlagEmbedding	==1.3.5	Pinned because of internal API stability concerns.
sentence-transformers	>=2.2.2	Loads BAAI/bge-m3 and the cross-encoder reranker.
transformers	>=4.35.0	HuggingFace base library; tokeniser, model loading, cache.
streamlit	>=1.28.0	UI framework.
langchain	==0.3.25	LangChain core. Pinned — see "Upgrade path" below for the matched-set rule.
langchain-community	==0.3.24	LangChain community integrations. Matched set with langchain.
langchain-core	==0.3.79	LangChain primitives. Matched set with langchain.
langchain-huggingface	==0.3.0	HuggingFaceEmbeddings integration.
langchain-mistralai	==0.2.10	Mistral LLM client.
langchain-ollama	==0.3.3	Ollama LLM client (dev only).
langchain-openai	==0.3.23	OpenAI LLM client. Primary in production.
langchain-qdrant	==0.2.0	Qdrant vector store integration.
langchain-text-splitters	==0.3.8	Document chunking utilities.
langgraph	==0.6.10	The multi-agent state-machine framework.
qdrant-client	>=1.7.0	Direct Qdrant client (used for count() queries and metadata scrolling).
python-dotenv	>=1.0.0	Load .env for local dev.
openai	>=1.0.0	Required by langchain-openai as a transitive but listed explicitly.
pyyaml	>=6.0	Reading settings.yaml.
tqdm	>=4.65.0	Progress bars (model loading).
snowflake-connector-python	>=4.0.0	Optional feedback export to Snowflake.
plotly	>=5.0.0	UI charts (was previously installed separately in the Dockerfile; consolidated into requirements.txt during the May 2026 cleanup).

Total direct deps: 25. Total resolved package count (including transitive deps): 184 (per uv.lock).

Lockfile reproducibility

uv.lock pins every transitive dependency to an exact version + hash. Reproduce the exact same environment with:

uv sync

Or, if not using uv:

pip install -r requirements.txt

The lockfile means "fresh install today" and "fresh install in 6 months" produce the same environment, avoiding the "pydantic 2.5 → 2.7 silently broke our structured output parsing" class of bug.

Upgrade path — matched-set rules

Some packages must be upgraded together. Bumping one without the others breaks compatibility.

LangChain matched set

langchain + langchain-core + langchain-community + all langchain-* integration packages must move together. They follow a single release cadence with breaking changes between minor versions.

# Recommended: bump them all at once with explicit versions
pip install --upgrade \
  langchain==0.4.0 \
  langchain-core==0.4.x \
  langchain-community==0.4.x \
  langchain-huggingface==0.4.x \
  langchain-mistralai==0.4.x \
  langchain-ollama==0.4.x \
  langchain-openai==0.4.x \
  langchain-qdrant==0.4.x \
  langchain-text-splitters==0.4.x \
  langgraph==0.7.x

Then re-run the live test (python _archive/live_filter_check.py) and the test suites (pytest tests/) to catch regressions. LangGraph in particular tends to introduce small API changes between minor versions.

PyTorch + Transformers + sentence-transformers matched set

torch + transformers + sentence-transformers + huggingface_hub should also be upgraded together. Cross-version compatibility is generally maintained but device handling and quantisation features change.

pip install --upgrade torch transformers sentence-transformers huggingface_hub

After upgrade, verify model loading still works:

python download_models.py   # should re-download and instantiate without errors

Qdrant client + Qdrant Cloud

qdrant-client should be on a version compatible with the Qdrant Cloud server version. Check https://qdrant.tech/documentation/release-notes/ before bumping. Major Qdrant Cloud upgrades have at times required client-side upgrades too.

What was removed in the May 2026 cleanup

Direct dependencies removed (were only needed for the now-archived visual stack):

• colpali-engine (~150 MB install)
• pdf2image
• pypdf
• matplotlib
• peft (now only a transitive dep of accelerate)

Total saved: 5 direct deps + ~9 fewer transitive packages, ~150 MB smaller Docker image (rough estimate; the ColPali model weights themselves were never baked into the image).

Recommended upgrade cadence

• Security patches: as needed, prioritise pydantic, openai, qdrant-client.
• LangChain matched set: quarterly. LangChain moves fast; staying current avoids painful big-bang upgrades later.
• PyTorch / Transformers matched set: bi-annually unless a specific feature is needed.
• Streamlit: bi-annually. Watch for breaking changes in st.session_state semantics.

Always test after upgrade:

1. python -c "from src.agents import get_multi_agent_chatbot; print('imports ok')"
2. python _archive/live_filter_check.py
3. python -m pytest tests/test_filter_integration.py tests/test_agent_intelligence.py

---

Related: docs/system-requirements.md covers prerequisites for running the system. docs/architecture/05-deployment-view.md shows what's baked into the Docker image at build time. DEFERRED.md items #6, #9 cover dependency-management improvements.