Spaces:

Kraft102
/

widgettdc-api

Paused

App Files Files Community

widgettdc-api / source_intel /INTELLIGENCE_DATA_FLOW.md

Kraft102

fix: sql.js Docker/Alpine compatibility layer for PatternMemory and FailureMemory

5a81b95 3 months ago

preview code

raw

history blame contribute delete

26.2 kB

Intelligence Data Flow Map

Complete Cross-Reference System for CIA Platform Intelligence Operations

Version: 1.0
Date: 2025-11-22
Status: Active
Purpose: Central navigation hub connecting data sources, database views, analytical frameworks, and intelligence products

📋 Quick Navigation

I Want To...	Navigate To
See the complete data pipeline	Complete Data Flow Diagram
Find views for an analysis type	Analysis Framework → View Mapping
Find analysis types for a view	View → Analysis Framework Mapping
Understand risk rule data sources	Risk Rule → View Mapping
Learn about analytical frameworks	DATA_ANALYSIS_INTOP_OSINT.md
Browse all database views	DATABASE_VIEW_INTELLIGENCE_CATALOG.md
Review risk detection rules	RISK_RULES_INTOP_OSINT.md

Complete Data Flow Diagram

This diagram shows the complete intelligence pipeline from OSINT collection through database views to analytical frameworks and intelligence products.

%%{init: {'theme':'base', 'themeVariables': { 'fontSize':'12px'}}}%%
flowchart TB
    subgraph COLLECTION["🔍 OSINT Collection Layer"]
        API1[📡 Riksdagen API<br/>Parliamentary Data]
        API2[📊 Election Authority<br/>Electoral Data]
        API3[💰 World Bank<br/>Economic Data]
        API4[🏛️ Financial Authority<br/>Government Finance]
    end
    
    subgraph DATABASE["🗄️ Database Views - 82 Total"]
        direction LR
        V1["<b>Politician Views</b><br/>8 views<br/>Performance tracking"]
        V2["<b>Party Views</b><br/>12 views<br/>Party analytics"]
        V3["<b>Committee Views</b><br/>12 views<br/>Legislative work"]
        V4["<b>Ministry Views</b><br/>7 views<br/>Government oversight"]
        V5["<b>Vote Data Views</b><br/>20 views<br/>Voting patterns"]
        V6["<b>Document Views</b><br/>7 views<br/>Legislative output"]
        V7["<b>Intelligence Views</b><br/>6 views<br/>Risk & anomaly"]
        V8["<b>Application Views</b><br/>10 views<br/>Platform usage"]
    end
    
    subgraph ANALYSIS["📊 Analytical Frameworks - 5 Types"]
        F1["<b>Temporal Analysis</b><br/>Trend detection<br/>Time series"]
        F2["<b>Comparative Analysis</b><br/>Benchmarking<br/>Peer comparison"]
        F3["<b>Pattern Recognition</b><br/>Anomaly detection<br/>Clustering"]
        F4["<b>Predictive Intelligence</b><br/>Forecasting<br/>Risk modeling"]
        F5["<b>Network Analysis</b><br/>Influence mapping<br/>Coalition dynamics"]
    end
    
    subgraph RULES["⚙️ Risk Detection - 45 Rules"]
        R1["<b>Politician Rules</b><br/>24 rules<br/>Individual behavior"]
        R2["<b>Party Rules</b><br/>10 rules<br/>Organizational health"]
        R3["<b>Committee Rules</b><br/>4 rules<br/>Legislative effectiveness"]
        R4["<b>Ministry Rules</b><br/>4 rules<br/>Government performance"]
        R5["<b>Other Rules</b><br/>3 rules<br/>Application monitoring"]
    end
    
    subgraph PRODUCTS["📑 Intelligence Products"]
        P1["<b>Political Scorecards</b><br/>Performance rankings"]
        P2["<b>Risk Assessments</b><br/>Threat analysis"]
        P3["<b>Coalition Analysis</b><br/>Alliance dynamics"]
        P4["<b>Trend Reports</b><br/>Pattern insights"]
        P5["<b>Compliance Dashboards</b><br/>Accountability tracking"]
    end
    
    API1 & API2 & API3 & API4 --> DATABASE
    DATABASE --> ANALYSIS
    DATABASE --> RULES
    ANALYSIS --> PRODUCTS
    RULES --> PRODUCTS
    
    style COLLECTION fill:#e1f5ff,stroke:#333,stroke-width:2px
    style DATABASE fill:#d1c4e9,stroke:#333,stroke-width:2px
    style ANALYSIS fill:#ffeb99,stroke:#333,stroke-width:2px
    style RULES fill:#ffcccc,stroke:#333,stroke-width:2px
    style PRODUCTS fill:#ccffcc,stroke:#333,stroke-width:2px

Analysis Framework → View Mapping

This table maps each analytical framework to the database views it uses most frequently.

Temporal Analysis Framework

Purpose: Detect trends, patterns, and changes over time
Documentation: DATA_ANALYSIS_INTOP_OSINT.md - Temporal Analysis

View Name	Purpose	Temporal Granularity	Link
view_riksdagen_politician_summary	Politician performance over time	Daily/Annual	View Docs
view_riksdagen_vote_data_ballot_politician_summary_daily	Daily voting activity	Daily	View Docs
view_riksdagen_vote_data_ballot_politician_summary_weekly	Weekly voting trends	Weekly	View Docs
view_riksdagen_vote_data_ballot_politician_summary_monthly	Monthly engagement patterns	Monthly	View Docs
view_riksdagen_vote_data_ballot_politician_summary_annual	Annual performance assessment	Annual	View Docs
view_riksdagen_party_ballot_support_annual_summary	Party voting patterns by year	Annual	View Docs
view_riksdagen_committee_decision_summary	Committee productivity trends	Ongoing	View Docs
view_riksdagen_politician_document_daily_summary	Document production trends	Daily	View Docs

Related Risk Rules: #1 PoliticianLazy, #4 PoliticianDecliningEngagement

Comparative Analysis Framework

Purpose: Benchmark performance against peers and standards
Documentation: DATA_ANALYSIS_INTOP_OSINT.md - Comparative Analysis

View Name	Purpose	Comparison Type	Link
view_riksdagen_politician_summary	Individual vs. party benchmarks	Peer comparison	View Docs
view_riksdagen_party_summary	Inter-party comparison	Party comparison	View Docs
view_riksdagen_party_ballot_support_annual_summary	Party voting effectiveness	Historical comparison	View Docs
view_riksdagen_committee_ballot_decision_summary	Committee effectiveness comparison	Committee benchmarking	View Docs
view_riksdagen_politician_document_daily_summary	Document productivity comparison	Output benchmarking	View Docs
view_riksdagen_vote_data_ballot_politician_summary_annual	Annual performance ranking	Percentile analysis	View Docs

Pattern Recognition Framework

Purpose: Identify anomalies, clusters, and behavioral patterns
Documentation: DATA_ANALYSIS_INTOP_OSINT.md - Pattern Recognition

View Name	Purpose	Pattern Type	Link
view_risk_rule_violation	Historical risk patterns	Anomaly detection	View Docs
view_riksdagen_vote_data_ballot_politician_summary	Voting behavior patterns	Behavioral clustering	View Docs
view_riksdagen_politician_ballot_support_annual_summary	Support pattern analysis	Coalition patterns	View Docs
view_riksdagen_politician_summary	Multi-dimensional behavior	Feature clustering	View Docs
view_riksdagen_party_ballot_support_annual_summary	Party alignment patterns	Voting bloc detection	View Docs

Related Risk Rules: #5 PoliticianCombinedRisk, #6 PoliticianAbstentionPattern

Predictive Intelligence Framework

Purpose: Forecast future outcomes and trends
Documentation: DATA_ANALYSIS_INTOP_OSINT.md - Predictive Intelligence

View Name	Purpose	Prediction Target	Link
view_riksdagen_vote_data_ballot_politician_summary_annual	Historical voting patterns	Future voting behavior	View Docs
view_riksdagen_party_ballot_support_annual_summary	Party trend trajectories	Coalition stability	View Docs
view_riksdagen_politician_summary	Performance trajectories	Re-election likelihood	View Docs
view_risk_rule_violation	Historical risk events	Future risk probability	View Docs
view_riksdagen_committee_decision_summary	Committee productivity	Legislative output forecast	View Docs

Network Analysis Framework

Purpose: Map relationships, influence, and coalition structures
Documentation: DATA_ANALYSIS_INTOP_OSINT.md - Network Analysis

View Name	Purpose	Network Type	Link
view_riksdagen_committee_role_member	Committee memberships	Formal networks	View Docs
view_riksdagen_politician_document_daily_summary	Co-authorship patterns	Collaboration networks	View Docs
view_riksdagen_party_summary	Party relationships	Coalition networks	View Docs
view_riksdagen_vote_data_ballot_politician_summary	Voting alignment	Voting bloc networks	View Docs
view_riksdagen_party_ballot_support_annual_summary	Inter-party coordination	Alliance networks	View Docs

Related Risk Rules: #9 PoliticianIsolatedBehavior, Committee Rules

View → Analysis Framework Mapping

This reverse mapping shows which analytical frameworks can utilize each major view category.

Politician Views

View Name	Temporal Analysis	Comparative Analysis	Pattern Recognition	Predictive Intelligence	Network Analysis	Documentation
view_riksdagen_politician	✅	✅	✅	✅	✅	View Docs
view_riksdagen_politician_summary	✅	✅	✅	✅	⭕	View Docs
view_riksdagen_politician_experience_summary	✅	✅	✅	✅	⭕	View Docs
view_riksdagen_politician_ballot_support_annual_summary	✅	✅	✅	✅	✅	View Docs

Party Views

View Name	Temporal Analysis	Comparative Analysis	Pattern Recognition	Predictive Intelligence	Network Analysis	Documentation
view_riksdagen_party_summary	✅	✅	✅	✅	✅	View Docs
view_riksdagen_party_ballot_support_annual_summary	✅	✅	✅	✅	✅	View Docs
view_riksdagen_party_document_daily_summary	✅	✅	⭕	✅	⭕	View Docs

Vote Data Views (20 views)

View Category	Temporal Analysis	Comparative Analysis	Pattern Recognition	Predictive Intelligence	Network Analysis	Documentation
Daily Summaries	✅	✅	✅	⭕	✅	View Docs
Weekly Summaries	✅	✅	✅	⭕	⭕	View Docs
Monthly Summaries	✅	✅	✅	✅	⭕	View Docs
Annual Summaries	✅	✅	✅	✅	✅	View Docs
All-Time Summaries	⭕	✅	⭕	⭕	⭕	View Docs

Committee Views

View Name	Temporal Analysis	Comparative Analysis	Pattern Recognition	Predictive Intelligence	Network Analysis	Documentation
view_riksdagen_committee_decision_summary	✅	✅	⭕	✅	⭕	View Docs
view_riksdagen_committee_ballot_decision_summary	✅	✅	⭕	⭕	⭕	View Docs
view_riksdagen_committee_role_member	⭕	⭕	⭕	⭕	✅	View Docs

Intelligence Views

View Name	Temporal Analysis	Comparative Analysis	Pattern Recognition	Predictive Intelligence	Network Analysis	Documentation
view_risk_rule_violation	✅	✅	✅	✅	⭕	View Docs

Legend:
✅ = Primary use case
⭕ = Secondary/Limited use case

Risk Rule → View Mapping

This table shows which database views are used by each category of risk detection rules.

Politician Risk Rules (24 rules)

Risk Rule Category	Primary Views	Risk Level Detection	Documentation
Absenteeism Detection	`view_riksdagen_vote_data_ballot_politician_summary_daily` `view_riksdagen_vote_data_ballot_politician_summary_monthly` `view_riksdagen_vote_data_ballot_politician_summary_annual`	🟡 MINOR → 🔴 CRITICAL	Rule #1
Effectiveness Tracking	`view_riksdagen_vote_data_ballot_politician_summary_annual` `view_riksdagen_politician_summary`	🟡 MINOR → 🔴 CRITICAL	Rule #2
Party Discipline	`view_riksdagen_vote_data_ballot_politician_summary_annual` `view_riksdagen_politician_ballot_support_annual_summary`	🟡 MINOR → 🔴 CRITICAL	Rule #3
Trend Analysis	`view_riksdagen_vote_data_ballot_politician_summary_monthly` `view_riksdagen_vote_data_ballot_politician_summary_annual`	🟠 MAJOR → 🔴 CRITICAL	Rule #4
Multi-Factor Assessment	`view_riksdagen_politician_summary` `view_riksdagen_vote_data_ballot_politician_summary_annual`	🟠 MAJOR → 🔴 CRITICAL	Rule #5
Strategic Behavior	`view_riksdagen_vote_data_ballot_politician_summary_annual`	🟡 MINOR → 🔴 CRITICAL	Rule #6
Participation Monitoring	`view_riksdagen_vote_data_ballot_politician_summary_annual` `view_riksdagen_politician_document_daily_summary`	🟠 MAJOR → 🔴 CRITICAL	Rule #7
Legislative Productivity	`view_riksdagen_politician_document_daily_summary`	🟡 MINOR → 🔴 CRITICAL	Rule #8
Collaboration Analysis	`view_riksdagen_politician_document_daily_summary` `view_riksdagen_committee_role_member`	🟠 MAJOR → 🔴 CRITICAL	Rule #9
Voting Participation	`view_riksdagen_vote_data_ballot_politician_summary_annual`	🟠 MAJOR → 🔴 CRITICAL	Rule #10

Party Risk Rules (10 rules)

Risk Rule Category	Primary Views	Risk Level Detection	Documentation
Government Support	`view_riksdagen_party_ballot_support_annual_summary`	🟡 MINOR → 🔴 CRITICAL	Party Rules
Party Cohesion	`view_riksdagen_party_ballot_support_annual_summary` `view_riksdagen_party_summary`	🟡 MINOR → 🔴 CRITICAL	Party Rules
Document Productivity	`view_riksdagen_party_document_daily_summary`	🟡 MINOR → 🔴 CRITICAL	Party Rules

Committee Risk Rules (4 rules)

Risk Rule Category	Primary Views	Risk Level Detection	Documentation
Committee Productivity	`view_riksdagen_committee_decision_summary` `view_riksdagen_committee_ballot_decision_summary`	🟡 MINOR → 🔴 CRITICAL	Committee Rules

Ministry Risk Rules (4 rules)

Risk Rule Category	Primary Views	Risk Level Detection	Documentation
Government Performance	`view_riksdagen_government_member_summary` `view_riksdagen_ministry_member_summary`	🟡 MINOR → 🔴 CRITICAL	Ministry Rules

Intelligence Product Generation

This section maps intelligence products back to their data sources and analytical methods.

flowchart LR
    subgraph SOURCES["Data Sources"]
        S1[Politician Views]
        S2[Party Views]
        S3[Vote Views]
        S4[Committee Views]
    end
    
    subgraph METHODS["Analytical Methods"]
        M1[Temporal Analysis]
        M2[Comparative Analysis]
        M3[Pattern Recognition]
        M4[Risk Rules]
    end
    
    subgraph PRODUCTS["Intelligence Products"]
        P1[Political Scorecards]
        P2[Risk Assessments]
        P3[Coalition Analysis]
        P4[Trend Reports]
    end
    
    S1 & S2 & S3 --> M1 & M2 & M3 & M4
    M1 & M2 --> P1
    M3 & M4 --> P2
    M1 & M2 & M3 --> P3
    M1 & M3 --> P4
    
    style SOURCES fill:#d1c4e9,stroke:#333,stroke-width:2px
    style METHODS fill:#ffeb99,stroke:#333,stroke-width:2px
    style PRODUCTS fill:#ccffcc,stroke:#333,stroke-width:2px

Political Scorecards

Data Sources:

view_riksdagen_politician_summary - Overall performance metrics
view_riksdagen_vote_data_ballot_politician_summary_annual - Voting records
view_riksdagen_politician_document_daily_summary - Legislative productivity

Analytical Methods:

Comparative Analysis (peer benchmarking)
Temporal Analysis (trend assessment)

Documentation: DATA_ANALYSIS_INTOP_OSINT.md

Risk Assessments

Data Sources:

view_risk_rule_violation - Historical violations
All politician, party, committee, ministry views

Analytical Methods:

Pattern Recognition (anomaly detection)
Risk Rules (45 detection rules)
Predictive Intelligence (risk forecasting)

Documentation: RISK_RULES_INTOP_OSINT.md

Coalition Analysis

Data Sources:

view_riksdagen_party_ballot_support_annual_summary - Voting alignment
view_riksdagen_party_summary - Party metrics

Analytical Methods:

Network Analysis (alliance mapping)
Comparative Analysis (inter-party comparison)
Pattern Recognition (voting bloc detection)

Documentation: DATA_ANALYSIS_INTOP_OSINT.md

Trend Reports

Data Sources:

All temporal views (daily, weekly, monthly, annual summaries)

Analytical Methods:

Temporal Analysis (trend detection)
Pattern Recognition (emerging patterns)

Documentation: DATA_ANALYSIS_INTOP_OSINT.md

View Categories and Their Primary Analysis Applications

Summary Table

View Category	View Count	Primary Analysis Types	Intelligence Value	Documentation
Politician Views	8	Temporal, Comparative, Pattern, Predictive	⭐⭐⭐⭐⭐ VERY HIGH	View Docs
Party Views	12	Comparative, Network, Temporal	⭐⭐⭐⭐ HIGH	View Docs
Committee Views	12	Temporal, Comparative, Network	⭐⭐⭐ MEDIUM	View Docs
Ministry Views	7	Temporal, Comparative	⭐⭐⭐ MEDIUM	View Docs
Vote Data Views	20	Temporal, Pattern, Comparative	⭐⭐⭐⭐⭐ VERY HIGH	View Docs
Document Views	7	Temporal, Comparative	⭐⭐⭐⭐ HIGH	View Docs
Intelligence Views	6	Pattern, Predictive, All	⭐⭐⭐⭐⭐ VERY HIGH	View Docs
Application Views	10	Temporal, Pattern	⭐⭐ LOW	View Docs

Document	Purpose	Link
DATA_ANALYSIS_INTOP_OSINT.md	Detailed analytical framework documentation	View
DATABASE_VIEW_INTELLIGENCE_CATALOG.md	Complete view catalog with examples	View
RISK_RULES_INTOP_OSINT.md	Risk detection rules documentation	View
DATA_MODEL.md	Database schema and relationships	View
LIQUIBASE_CHANGELOG_INTELLIGENCE_ANALYSIS.md	Schema evolution analysis	View

Maintenance Notes

Last Updated: 2025-11-22
Maintained By: CIA Intelligence Operations Team

Update Frequency:

Review quarterly or when new views/rules added
Validate all links when documentation structure changes
Update mappings when analytical frameworks evolve

Validation Checklist:

All view names match current database schema
All internal links resolve correctly
All risk rule references are accurate
Analysis framework descriptions are current
Intelligence product mappings are complete

This document serves as the central navigation hub for understanding data flow through the Citizen Intelligence Agency platform. For detailed documentation on specific components, follow the links to the respective specialized documents.

Intelligence Data Flow Map

Complete Cross-Reference System for CIA Platform Intelligence Operations

📋 Quick Navigation

Complete Data Flow Diagram

Analysis Framework → View Mapping

Temporal Analysis Framework

Comparative Analysis Framework

Pattern Recognition Framework

Predictive Intelligence Framework

Network Analysis Framework

View → Analysis Framework Mapping

Politician Views

Party Views

Vote Data Views (20 views)

Committee Views

Intelligence Views

Risk Rule → View Mapping

Politician Risk Rules (24 rules)

Party Risk Rules (10 rules)

Committee Risk Rules (4 rules)

Ministry Risk Rules (4 rules)

Intelligence Product Generation

Political Scorecards

Risk Assessments

Coalition Analysis

Trend Reports

View Categories and Their Primary Analysis Applications

Summary Table

Related Documentation

Maintenance Notes