LiberationLabs/emet-occrp-demo

Emet

AI-Powered Investigative Intelligence for Anti-Corruption Journalism

Emet is an autonomous investigation agent that traces corporate ownership, screens sanctions lists, analyzes blockchain flows, and synthesizes findings into auditable, publication-safe reports. It is built on the FollowTheMoney (FtM) data model -- the same schema used by Aleph and OCCRP's investigative data platform.

The name "Emet" (Hebrew: truth) is the word inscribed on the forehead of the Golem -- a guardian animated to protect its community. The first letter, Aleph, is also the name of OCCRP's investigative data platform.

---

What It Does

Give Emet a goal in plain language. It runs a multi-step investigation autonomously:

emet investigate \
  "Trace beneficial ownership of Meridian Holdings through offshore jurisdictions" \
  --llm ollama --max-turns 8

An LLM reasons about what to do next. The agent searches entity databases, screens sanctions and PEP lists, traces corporate ownership chains, monitors global news, investigates blockchain transactions, and synthesizes everything into a structured report -- with PII scrubbed at publication boundaries and a full forensic audit trail.

Investigation Tools

Tool	Function
search_entities	Federated entity search across Aleph, OpenSanctions, OpenCorporates, ICIJ Offshore Leaks, GLEIF, UK Companies House, SEC EDGAR
screen_sanctions	OFAC, EU, UN consolidated sanctions and PEP screening with fuzzy matching
trace_ownership	Multi-hop beneficial ownership tracer with effective stake computation through offshore structures
osint_recon	Domain, email, IP, and social footprint reconnaissance (via SpiderFoot)
investigate_blockchain	Ethereum, Bitcoin, Tron, and Solana transaction flow analysis, wallet clustering, mixer detection
monitor_entity	GDELT-powered real-time entity monitoring across global news
analyze_graph	Community detection, broker identification, circular ownership detection, shell company scoring, PageRank
generate_report	LLM-synthesized investigation reports with evidence chains and confidence scoring
search_aleph	Direct Aleph API search across collections, documents, and leaked records

Additional Capabilities

• Federated search across 7+ sources with parallel async fan-out, Jaccard deduplication, token-bucket rate limiting, and response caching
• Evidence chain tracking: every factual claim is bound to source references with a 0-1 confidence score; unsupported claims are flagged before publication
• Congressional STOCK Act disclosure analysis (House Clerk feed + FEC campaign finance)
• CourtListener/RECAP federal court record integration
• SEC EDGAR real-time filing feed (Form 4, SC 13D, 8-K)
• On-chain intelligence: OFAC-designated mixer detection (Tornado Cash, Sinbad), DeFi protocol labeling, exchange identification, wallet clustering via co-spend heuristic
• Interactive graph visualization (self-contained Cytoscape.js HTML, no CDN dependency)
• JSON-LD export with schema.org vocabulary mapping
• FtM bundle export (JSONL/zip) for round-trip Aleph re-import
• Entity resolution via Splink probabilistic linkage with pure-Python fallback
• Temporal pattern detection: burst analysis, coincidence detection across entity timelines

---

Architecture

Goal (natural language)
  |
  v
+----------------------------------------------------------------+
|  Agent Loop (InvestigationAgent)                                |
|  LLM decision -> tool execution -> result processing           |
|  One tool call per turn, budget-capped, with full audit trail   |
|  Heuristic fallback when LLM is unavailable                    |
+------+------------------+-------------------+---------+--------+
       |                  |                   |         |
  +----v-----+    +-------v--------+   +------v-----+  |
  | Safety   |    | Session State  |   | Tool       |  |
  | Harness  |    |                |   | Executor   |  |
  |          |    | FtM entities   |   | (MCP)      |  |
  | PII scrub|    | Findings+conf  |   |            |  |
  | Audit log|    | Leads+priority |   | 9 tools    |  |
  | Cost cap |    | Reasoning trace|   |            |  |
  +----------+    +----------------+   +------------+  |
                                                       |
  +----------------------------------------------------v--------+
  |  Data Layer                                                  |
  |                                                              |
  |  FtM Data Spine       Graph Engine (NetworkX)                |
  |  Aleph Client         7 investigative algorithms             |
  |  Federated Search     Multi-format export                    |
  |  Blockchain (4 chains)                                       |
  |  Document Sources     Export (Markdown, PDF, FtM, Timeline)  |
  +--------------------------------------------------------------+

Three-Tier Design

1. Collection -- Federated data retrieval from public registries, sanctions lists, blockchain explorers, court records, and news feeds. Each source adapter produces FtM-schema entities with provenance metadata.

2. Analysis -- Graph algorithms (PageRank, community detection via Louvain/label propagation, shortest path, broker identification, circular ownership detection, shell company topology scoring, structural anomaly detection), evidence chain confidence scoring, temporal pattern analysis, and entity resolution.

3. Orchestration -- LLM-driven agent loop that decides which tools to call, tracks leads by priority, manages investigation budget, and synthesizes findings into reports. Falls back to heuristic routing when no LLM is available.

Human review is required at the publication boundary. The safety harness operates in two modes: investigate (audit-only -- log everything, block nothing) and publish (enforcing -- PII scrubbed, sensitive data redacted, all outputs sanitized).

LLM Backend

Emet is LLM-agnostic with a tiered routing system:

Tier	Task Type	Default Model (Docker)
Fast	Entity extraction, classification, NER	Qwen3 8B
Balanced	Document analysis, sanctions screening	Qwen3 14B / Qwen3.5 27B
Powerful	Multi-step reasoning, report synthesis	Qwen3 14B / Qwen3.5 27B / Qwen3 235B MoE

Three provider backends:

• Ollama (default) -- local models, zero API cost, data never leaves the machine
• Anthropic Claude -- cloud API fallback
• Stub -- canned responses for testing and demo mode

Cascading fallback: if the configured provider is unavailable, Emet degrades gracefully through the chain (Ollama -> Anthropic -> Stub -> heuristic routing). It never crashes due to LLM unavailability.

Memory

Within-investigation: Three-stage Continuum Memory Architecture (CMA):

• Stage 1: Sliding-window dialogue compression with entropy-based filtering and atomic fact extraction (based on SimpleMem, arXiv:2601.02553)
• Stage 2: Significance scoring, temporal tagging, spaced retrieval scheduling
• Stage 3: Hybrid retrieval -- dense vectors (pgvector), lexical (tsvector/BM25), symbolic (significance score) with adaptive fusion

Cross-investigation: HippoRAG knowledge graph (based on arXiv:2502.14802) with CatRAG query-adaptive edge weighting. Entities from different investigations are linked automatically through co-occurrence triples and Personalized PageRank. This is where "Entity A is a donor in Investigation 1" connects to "Entity A is a contractor in Investigation 2."

Graph Capabilities

Built on NetworkX with investigative interpretations layered on standard algorithms:

Algorithm	Investigative Use
PageRank	Influence scoring -- identify the most structurally important entities in a network
Community Detection	Network cluster identification via Louvain or label propagation -- find groups of tightly connected entities
Shortest Path	Connection tracing -- find the shortest chain between two entities
Broker Identification	Betweenness centrality -- find intermediaries connecting otherwise separate groups
Circular Ownership	Detect ownership loops (A owns B owns C owns A) -- a shell company red flag
Shell Topology Scoring	Score entities on structural indicators of shell company behavior
Structural Anomaly Detection	Fan-in analysis -- flag entities owned by an unusual number of parents (pooled SPV / layering hub signal)
Beneficial Ownership (UBO) Tracing	Walk incoming ownership edges recursively, multiply share percentages to compute each owner's effective stake in a target

All results include human-readable explanations suitable for journalists and suggested follow-up actions.

Export formats: GEXF (Gephi), GraphML, D3 JSON, Cytoscape JSON, CSV, interactive HTML (Cytoscape.js).

FtM / Aleph Interoperability

All entities use the FollowTheMoney data model throughout the pipeline. This means:

• Entities produced by Emet can be exported as FtM JSONL bundles and imported directly into Aleph collections
• Aleph collections can be searched and retrieved natively via the search_aleph tool
• Entity schemas (Person, Company, LegalEntity, Ownership, Directorship, Payment, etc.) are consistent with OCCRP's existing data infrastructure
• Federated search results from all sources are normalized to FtM before entering the investigation graph

Security

Layer	Function
PII Scrubbing	Regex-based detection of email, phone, SSN, credit card (with Luhn validation), IP address, date of birth. Applied at every publication boundary.
Target Policy	Code-enforced rule: investigate organizations and public figures only. Private individuals are denied unless a logged public-interest override is supplied. Classification uses FtM schema + public-dataset provenance (congressional disclosures, FEC, EDGAR, sanctions).
Intent Capsules	HMAC-SHA256 signed mandates that authorize a bounded set of agent actions. Every action traces back to a signed capsule with budget, tool, and egress constraints.
Safety Harness	Two-mode (investigate vs publish). Pre-execution checks, circuit breakers, cost caps, and a full forensic audit trail.
Audit Archive	Every tool call, LLM exchange, and reasoning step captured in gzip-compressed JSONL with SHA-256 integrity verification. Actor identity recorded on every event.
Security Shield	Egress allowlisting, budget enforcement, blocked content patterns.
Behavior Monitor	Post-execution anomaly detection.

Ethics Governance

Emet's behavior is governed by VALUES.json, a machine-readable ethics constitution based on the SPJ Code of Ethics, OCCRP editorial standards, and GIJN verification methodology:

Pillar	Weight	Core Constraint
Accuracy	0.25	Every claim traceable to source material. AI inference always flagged.
Source Protection	0.25	Source identity never exposed without consent. Document metadata scrubbed.
Public Interest	0.20	Investigation scope proportionate to significance. Private privacy respected.
Proportionality	0.15	Least intrusive method preferred. Data collection limited to necessity.
Transparency	0.15	Methodology documented and auditable. AI tool usage disclosed.

Consensus gates require human editorial approval for publication, entity modification, and sensitive operations.

---

Deployment

Docker (recommended)

Three deployment tiers, all fully local -- no data leaves the machine, no cloud APIs called:

# Field -- laptop / Mac Mini (24-32GB RAM)
# Models: Qwen3 14B + Qwen3 8B (~20GB model footprint)
docker compose -f docker-compose.yml -f docker-compose.field.yml up -d

# Server -- office server (64GB+ RAM)
# Models: Qwen3.5 27B + Qwen3 8B (~40GB model footprint)
docker compose -f docker-compose.yml -f docker-compose.server.yml up -d

# Enterprise -- newsroom infrastructure (256GB+ RAM)
# Models: Qwen3 235B MoE + Qwen3.5 27B + Qwen3 8B (~170GB model footprint)
docker compose -f docker-compose.yml -f docker-compose.enterprise.yml up -d

On first run, the ollama-setup container pulls models automatically. Monitor progress:

docker compose logs -f ollama-setup

Verify the stack is running:

docker compose ps
# engine, db, redis, mcp, spiderfoot should be running

curl http://localhost:8000/health
# {"status": "ok"}

Local Development (pip)

Requires Python 3.11+.

pip install -e ".[dev]"
cp .env.example .env
# Edit .env with API keys (all optional -- demo mode works without any)

# Demo mode -- bundled scenario, no keys needed
emet investigate "Meridian Holdings offshore network" --llm stub --demo

# With local Ollama
emet investigate "Trace ownership of Acme Holdings" --llm ollama

# With Anthropic Claude
export ANTHROPIC_API_KEY=sk-ant-...
emet investigate "Trace ownership of Acme Holdings" --llm anthropic

Interfaces

Interface	Command	Use Case
CLI	emet investigate "..."	Direct investigations, scripting, automation
HTTP API	emet serve --http --port 8000	Web dashboards, programmatic access
WebSocket	ws://host:8000/ws/investigations/{id}	Real-time streaming updates
MCP	emet serve --transport stdio	Claude Desktop, MCP-compatible clients

API docs are auto-generated at http://localhost:8000/docs when the HTTP server is running.

Demo Script

# After Docker stack is running:
./demo_occrp.sh

Runs three investigations demonstrating ownership tracing, sanctions screening, and financial flow analysis. Results saved to ./investigations/.

---

Configuration

All data source API keys are optional. Emet degrades gracefully when sources are unavailable, and demo mode works with zero configuration.

See .env.example for the full list of configurable environment variables, grouped by category:

• Data sources -- OpenSanctions, OpenCorporates, UK Companies House, ICIJ, GLEIF, SEC EDGAR, FEC, CourtListener
• Blockchain -- Etherscan (ETH), Solana (public RPC, no key)
• News/OSINT -- GDELT (no key), SpiderFoot (self-hosted)
• Documents -- Datashare (ICIJ), DocumentCloud
• LLM -- Anthropic Claude, Ollama (local)
• Infrastructure -- PostgreSQL, Redis, Aleph

---

Testing

# Full unit + integration suite (~1,650 tests, ~3 minutes)
python -m pytest tests/ -q --ignore=tests/live

# Live integration tests (requires API keys)
python -m pytest -m live tests/live/ -v

# Key test modules
python -m pytest tests/test_agent.py               # Agent loop + session
python -m pytest tests/test_mcp_server.py           # MCP server + tool dispatch
python -m pytest tests/test_graph.py                # Graph analytics
python -m pytest tests/test_federation.py           # Federated search
python -m pytest tests/test_e2e_pipeline.py         # End-to-end pipeline
python -m pytest tests/test_security_pii.py         # PII detection + scrubbing
python -m pytest tests/test_export.py               # Export pipeline
python -m pytest tests/test_mock_fidelity.py        # Tool + demo mode fidelity

No external services required for the unit suite. All tests use stubs, mocks, and synthetic datasets.

---

Project Structure

emet/
  agent/                    # Autonomous investigation engine
    loop.py                 # InvestigationAgent: LLM decisions, tool execution
    session.py              # Investigation state (entities, findings, leads)
    safety_harness.py       # Two-mode safety: audit-only vs enforcing
    audit.py                # Forensic audit archive (gzip JSONL, SHA-256)
    persistence.py          # Session save/load for investigation resume
  mcp/                      # Model Context Protocol server
    server.py               # MCP server (stdio / SSE / HTTP transport)
    tools.py                # 9 MCP tools + EmetToolExecutor
    resources.py            # MCP resource providers
  api/                      # HTTP API (FastAPI)
    app.py                  # App factory
    websocket.py            # WebSocket streaming
    routes/                 # REST endpoints
  cognition/                # LLM abstraction
    llm_base.py             # LLMClient ABC
    llm_anthropic.py        # Anthropic Claude
    llm_ollama.py           # Ollama (local)
    llm_stub.py             # Canned responses (testing)
    llm_factory.py          # Provider factory + cascading fallback
    model_router.py         # Tier-to-model mapping
  ftm/                      # FollowTheMoney integration
    data_spine.py           # FtM entity factory
    aleph_client.py         # Async Aleph REST client
    external/               # Federated data sources
      federation.py         # Parallel async fan-out (7+ sources)
      adapters.py           # OpenSanctions, OpenCorporates, ICIJ, GLEIF
      companies_house.py    # UK Companies House
      edgar.py              # SEC EDGAR + real-time Atom feed
      congress.py           # Congressional STOCK Act disclosures
      fec.py                # FEC campaign finance
      courtlistener.py      # CourtListener / RECAP
      blockchain.py         # ETH, BTC, Tron, Solana
      crypto_intel.py       # Mixer detection, DeFi labels, risk scoring
      converters.py         # Source -> FtM converters
      rate_limit.py         # Token bucket, monthly counter, cache
  graph/                    # Network analysis
    algorithms.py           # 8 investigative algorithms + UBO tracer
    engine.py               # GraphEngine orchestrator
    ftm_loader.py           # FtM -> NetworkX conversion
    exporters.py            # GEXF, GraphML, CSV, D3, Cytoscape, JSON-LD
    visualizer.py           # Interactive Cytoscape.js HTML
  export/                   # Investigation output
    markdown.py             # Markdown reports
    pdf.py                  # PDF reports (reportlab)
    ftm_bundle.py           # FtM JSONL/zip for Aleph re-import
    evidence.py             # Evidence chain + confidence scoring
    timeline.py             # Temporal analysis + interactive HTML timeline
  security/                 # Security layer
    pii.py                  # PII detection + redaction
    target_policy.py        # Public figures only enforcement
    intent_capsule.py       # HMAC-SHA256 signed action mandates
    shield.py               # Egress allowlist, budget enforcement
    monitor.py              # Post-execution anomaly detection
    sandbox.py              # Execution sandbox
  memory/                   # Continuum Memory Architecture
    cma_stage1.py           # Semantic compression
    cma_stage2.py           # Significance scoring + spaced retrieval
    cma_stage3.py           # Hybrid retrieval (dense + lexical + symbolic)
  governance/               # Ethics enforcement
    consensus.py            # Consensus gates for publication actions
  config/                   # Settings + VALUES.json loader
  monitoring/               # Change detection + sanctions alerts
  workflows/                # Predefined investigation templates
  cli.py                    # CLI entry point
mnemosyne/                  # Cross-investigation memory modules
  hipporag-catrag-kg/       # Knowledge graph (HippoRAG + CatRAG)
  kintsugi-cma/             # Reference CMA implementation
  sira-enrichment/          # Semantic Indexed Retrieval
  tgs-rag-bridge/           # Memory-to-graph bridge
  tgs-verification/         # Memory verification
skills/                     # Investigation methodology docs (SKILL.md)
tests/                      # ~1,650 unit tests + 44 live integration tests
VALUES.json                 # Machine-readable ethics constitution

---

Data Sources

Source	Data	Key Required	Notes
OpenSanctions / yente	325+ sanctions & PEP lists	Yes (free tier)	Fuzzy matching, bulk screening
OpenCorporates	200M+ companies, 145+ jurisdictions	Yes (free tier, 200 req/mo)	Officer search, jurisdiction filtering
ICIJ Offshore Leaks	810K+ offshore entities	No	Panama Papers, Pandora Papers reconciliation
GLEIF	Legal Entity Identifiers	No	LEI lookup, ownership relationships
UK Companies House	600M+ records	Yes (free)	Officers, PSC (beneficial ownership)
SEC EDGAR	US securities filings	No (User-Agent required)	10-K, SC 13D, Form 4 + real-time feed
FEC / OpenFEC	US campaign finance	Yes (DEMO_KEY works)	Candidates, committees, contributions
CourtListener / RECAP	Federal court records	Optional (raises rate limit)	Dockets, parties, opinions
Etherscan	Ethereum blockchain	Yes (free tier)	Transactions, balances, counterparties
Blockstream	Bitcoin blockchain	No	BTC transaction + address lookup
Tronscan	Tron blockchain	No	USDT-TRC20 transfer tracking
Solana JSON-RPC	Solana blockchain	No	Balance, transaction signatures
GDELT	Global news (250M+ articles)	No	Real-time entity monitoring
SpiderFoot	OSINT recon (200+ modules)	Self-hosted	Domain, email, IP, social footprint
Aleph	OCCRP investigative data	Yes	Collections, documents, entity search

---

License

Hippocratic License 3.0 with AI Welfare module. See LICENSE.md.

Enabled modules: CL (labor rights), ECO (environmental), MEDIA (disinformation), MIL (military), SV (surveillance), XUAR (human rights).

Usage rights by organization type:

• Investigative journalists, newsrooms, press freedom organizations, anti-corruption NGOs, academic journalism programs -- free, no restrictions beyond the license terms.
• Commercial use (compliance, KYC/AML, due diligence, corporate intelligence) -- commercial license required. See LICENSE-COMMERCIAL-ADDENDUM.md.
• Prohibited under all circumstances -- surveillance of journalists or sources, press suppression, targeting whistleblowers, mass surveillance, circumventing press freedom protections.

---

Provenance

Built by Liberation Labs.

Core infrastructure derived from Project Kintsugi (self-repairing agentic harness). The governance, security, memory, and plugin layers are domain-agnostic by design. The investigative layers (agent loop, MCP tools, FtM spine, federated data sources, graph algorithms) are Emet-specific.

Key dependencies: FollowTheMoney (MIT), Aleph (MIT), FastAPI (MIT), NetworkX (BSD), httpx (BSD-3), Pydantic (MIT), SQLAlchemy (MIT), reportlab (BSD).