docs/ARCHITECTURE.md

# Ergo-Agentic: Architecture

## Tech Stack

- **Orchestration**: LangGraph (Python)
- **AI Models**: Multi-model (Claude, GPT-4V, Gemini — configurable)
- **Rules Engine**: Static JSON datasources (no database for rules)
- **Report Generation**: Deterministic code (no AI)

## High-Level Pipeline

```

Input (images + metadata)

  │

  ▼

inspect_workspace_image ──── per image, parallel

cv_extract ──────────────── per image, parallel

  │

  ▼

build_routing_manifest ───── deterministic join of semantic manifest + CV + metadata

  │

  ▼

choose_applicable_parameters ── deterministic

  │

  ▼

assess_body_and_chair / assess_desk_and_arms / assess_screens

  ──── per relevant image × per model, parallel

  │

  ▼

aggregate_findings_by_assessment ── deterministic, assessment-scoped

  │

  ▼

confidence_gate ─────────── deterministic accept / targeted-review split

  │

  ▼

targeted_review_evidence ── model review over weak/conflicting request batches

  │

  ▼

global_consistency_audit ── deterministic flags only, does not change outcomes

  │

  ▼

build_final_assessment_matrix ── deterministic

```

## Pipeline Nodes

### 1. inspect_workspace_image

**Type**: AI (vision LLM)
**Runs**: Once per uploaded image, in parallel
**Input**: Single image
**Output**: Workspace semantic manifest

```

WorkspaceSemanticManifest:

  image_id: str

  work_location: str | null        # homeOffice, couch, bed, diningTable, floorSitting

  person_visible: bool

  body_coverage: str               # none | partial | posture_coverage_sufficient

  visible_body_regions: list[str]  # head, torso, hips, legs, hands, feet, etc.

  posture_context_hint: str        # sitting | standing | mixed | unknown

  chair_visible: bool

  chair_type: str | null           # office chair, dining chair, stool, etc.

  desk_visible: bool

  feet_visible: bool

  screens: list                    # type, coarse_layout_position, display_visible_state

  keyboard_mouse_visible: bool

  detected_accessories: list       # ergonomic objects: footrest, laptop stand,

                                   # sit-stand converter, Wacom tablet,

                                   # document reader, wrist rest, etc.

  notes: str                       # anything unusual

```

**Purpose**: Understand what each image contains so downstream routing can avoid images that cannot answer certain questions. This node does not judge posture quality, screen arrangement, primary screen intent, or standing desk presence from a standing person. It only reports visible scene contents and coverage.

### 2. build_routing_manifest

**Type**: Deterministic code (no AI)
**Input**: `WorkspaceSemanticManifest[]` + `cv_results[]` + user metadata
**Output**: `RoutingManifest` + `scene_config`

`RoutingManifest` is the planner's truth source. It merges semantic coverage,
CV availability, pose/detection hints, and metadata into stable routing facts:

```

RoutingManifest:

  body_and_chair_coverage: FocusGroupCoverage

  desk_and_arms_coverage: FocusGroupCoverage

  screens_coverage: FocusGroupCoverage

  standing_work_possible: bool

  standing_desk_equipment_visible: bool

  posture_context: sitting | standing | mixed

  image_focus_coverage: dict[focus_group, list[image_id]]

  warnings: list[RoutingWarning]

  evidence_sources: list[EvidenceSourceRef]

```

Standing desk equipment is true only when sit/stand equipment is visible or
metadata says it exists. A standing person can set `standing_work_possible`,
but does not enable standing desk equipment parameters by itself.

### 3. choose_applicable_parameters

**Type**: Deterministic code (no AI)
**Input**: `RoutingManifest` + `common-assessment-parameters.json`
**Output**: Execution plan

```

ExecutionPlan:

  assessable_parameters: list[str]          # human-readable labels with image coverage

  assessable_parameter_ids: list[str]       # parameter IDs used as stable join keys

  assessable_parameter_details: list[dict]  # id, label, group, question for traces

  skipped_parameters: list[str]             # no image can answer these

  skipped_parameter_details: list[dict]     # id, label, group, question for traces

  focus_group_assignments:

    "body_and_chair": list[image_id]        # which images to send

    "desk_and_arms": list[image_id]

    "screens": list[image_id]

  scene_config:

    screen_count: int

    has_standing_desk: bool

    work_location: str

```

**Logic**:
- If routing has no `posture_coverage_sufficient`, skip body/chair parameters
- If no image has `feet_visible`, skip feet grounding
- If `screen_count == 1`, skip screen arrangement parameters
- If `standing_desk_equipment_visible == false`, skip standing desk parameters
- Screen type logic: single monitor → monitor params; single laptop → laptop params; mixed/multiple → include arrangement rubrics, but the screen pass still judges the outcome
- Filters `common-assessment-parameters.json` to only relevant parameters

### 4. assess_body_and_chair / assess_desk_and_arms / assess_screens



**Type**: AI (vision LLM)

**Runs**: Per image × per model × per focus group (all parallel)

**Input**: Image(s) + parameter rubric for the focus group



Three focus groups, each assessing 3-4 parameters:



#### Pass 1: Body & Chair

**Visual focus**: The person's body and chair

**Parameters**:

- Back posture (slouching / sitting back / perched / leaning forward)

- Sitting height (hips vs knees)

- Seat pan depth (gap behind knee)

- Feet grounding (feet flat, dangling, etc.)



#### Pass 2: Desk & Arms

**Visual focus**: Desk surface and arm positioning

**Parameters**:

- Desk height relative to elbow (sitting)

- Standing desk height relative to elbow (if applicable)

- Armrest height relative to elbow

- Keyboard & mouse placement (distance from desk edge)



#### Pass 3: Screens

**Visual focus**: Monitor/laptop area

**Parameters**:

- Screen/laptop height relative to eye level

- Screen/laptop distance from user

- Screen arrangement — only if multiple screens (is person centered on primary?)



**Screen logic**: Single monitor → assess monitor only. Single laptop → assess laptop only. Mixed setup → monitor is primary. Agents simply assess what they see; mapping to specific outcome key variants (twoMulti-*, twoScreens-*, etc.) happens in the aggregator.



**Output per parameter**:

```

Observation:

  parameter_id: str
  outcome: str | null              # the option key, or null if not determinable
  visibility: clear | partial | not_visible

  confidence: "high" | "medium" | "low"

  evidence_note: str               # what the model actually observed (specific visual cues)
```



**Prompt design**:

- Each pass receives ONLY its 3-4 parameters plus a **visual cue guide** describing what to look at

- The parameter rubric includes the reference images (`optionImage`) as visual anchors

- The model is asked: "Which of these options best matches what you see?"

- Evidence notes must reference specific observable features, not subjective impressions

- Explicitly told: "If you cannot clearly determine this parameter from the image, return null"



See [agent-orchestration.md](architecture/agent-orchestration.md) for the full visual cue guide per focus group.



### 5. Assessment-Scoped Worst-Case Aggregator



**Type**: Deterministic code (no AI)

**Input**: All observations from all vision passes

**Output**: Aggregated results + conflicts



This is still one deterministic reducer node in the graph, but it does not

blend unrelated tasks together. It first buckets observations by

`assessment_id` and then applies worst-case aggregation inside each bucket.



**Logic per assessment**:



1. **Across images**: If image 1 shows `chair-posture-sittingBack` but image 2 shows `chair-posture-slouching`, take the **worst** outcome (highest risk). This follows the "worst observed" rule.



2. **Across models**: Collect votes. If all 3 models agree, high confidence in the result. If they disagree, flag as a conflict.



3. **Risk ranking**: Uses `issue-based-outcomes.json` to rank outcomes by risk level (high > medium > low > null/good habit) and `postureScore` (lower score = worse).



```
AggregatedResult:
  parameter_id: str

  worst_outcome: str
  model_votes: dict[str, str]      # model_name -> outcome
  agreement: bool                  # all models agree?
  confidence: str                  # derived from vote agreement
  
Conflict:
  parameter_id: str

  disagreements: dict[str, str]    # model_name -> outcome
  images_analyzed: list[str]

```



### 6. Layered Review



**Type**: deterministic gate + targeted AI review + deterministic audit

**Runs**: Gate once, targeted review only for weak/conflicting findings, audit once

**Input**: Candidate findings, conflicts, relevant images, routing summary

**Output**: Final confirmed outcome list plus audit flags



**Purpose**:

- Accept strong, uncontested findings without another model call

- Resolve conflicts or weak evidence using only relevant images and rubrics

- Batch related weak findings into one review request when they share focus

  group, posture context, and image subset

- Catch any obvious errors (e.g., model said "slouching" but person is clearly standing)

- Check matrix consistency without directly overriding outcomes

- Produce the definitive list of outcome keys



**Output**:

```

ReviewResult:

  final_outcomes: list[str]        # confirmed outcome keys
  overrides: list[Override]        # where review agent changed the aggregated result
  skipped_parameters: list[str]    # insufficient evidence

  global_audit_flags: list[GlobalAuditFlag]

```



### 7. Outcome Matrix Builder



**Type**: Deterministic code (no AI)

**Input**: Final outcomes + evidence trail + scene config

**Output**: Outcome matrix (parameter x outcome x evidence)



The **primary output** of the agentic pipeline. A structured matrix of which parameters were assessed, which outcome(s) were selected, evidence summaries, and review decisions. This is the core deliverable — everything downstream is deterministic transformation.



### 8. Report Builder (optional, separate concern)



**Type**: Deterministic code (no AI)

**Input**: Outcome matrix + all datasource JSONs + metadata

**Output**: Complete report JSON (matching `report-sample.json` structure)



Can run inside or outside the LangGraph pipeline. The agentic work is done once the outcome matrix is built.



**Logic**:

1. For each outcome in the matrix:

   - Look up in `issue-based-outcomes.json` → get `isGoodHabit`, `riskLevel`, `currentHabit`, `recommendation`, `postureScore`

   - Look up in `body-based-outcomes.json` → get affected body parts, conditions, body-part-specific recommendations

2. Build `goodHabits` list from outcomes where `isGoodHabit == true`

3. Build `issueBasedReport` from outcomes where `isGoodHabit == false`

4. Build `potentialRiskPart` by grouping issues per body part, taking worst condition per body part

5. Build `actionPlans` from recommendations, ordered by priority (points)

6. Match `mergedProducts` from product datasource based on outcomes

7. Calculate `ergoPostureScore` from posture scores



## Anti-Hallucination Strategy



| Layer | Mechanism |

|---|---|

| **Image Analyzer** | Produces a visibility/scene manifest only; no ergonomic judgments |

| **Routing Manifest** | Deterministically maps semantic/CV/metadata facts to focus-group coverage |

| **Focused passes** | 3-4 params per call, not 12+; model stays grounded on one visual region |

| **Visual cue guides** | Each pass includes specific visual cues (what to look at, what angles to check) |

| **Reference images** | Option images from rubric serve as visual anchors ("does it look like A or B?") |

| **Observable evidence** | Models must cite specific visual features, not subjective impressions |

| **Null allowed** | Models can say "not determinable" — no pressure to fill every field |

| **Multi-model voting** | Single-model hallucination caught by disagreement with other models |

| **Layered review** | Strong findings are accepted deterministically; only weak/conflicting findings get targeted image review |



## Model Routing



Models are configurable per role:



```

model_config:
  image_analyzer: "claude-sonnet"

  vision_passes:
    - "claude-sonnet"
    - "gpt-4o"
    - "gemini-pro-vision"
  review_agent: "claude-opus"

```



This is designed for experimentation — swap models, compare accuracy, find the best combination.



## LangGraph State Schema



```python

class ErgoState(TypedDict):

    # Input

    images: list[ImageInput]

    metadata: dict

    

    # Image Analyzer output

    image_manifests: list[WorkspaceSemanticManifest]
    cv_results: list[ImageCVResult]

    routing_manifest: RoutingManifest

    work_location: str

    scene_config: SceneConfig

    

    # Planner output

    assessable_parameters: list[str]          # human-readable labels

    assessable_parameter_ids: list[str]       # stable IDs for internal joins

    assessable_parameter_details: list[dict]

    skipped_parameters: list[str]             # stable IDs

    skipped_parameter_details: list[dict]

    execution_plan: ExecutionPlan

    

    # Vision pass outputs

    observations: list[Observation]

    

    # Aggregator output

    aggregated_findings: dict[str, AggregatedFinding]

    candidate_findings: dict[str, CandidateFinding]

    conflicts: list[Conflict]

    

    # Review output

    review_requests: list[ReviewRequest]

    review_decisions: list[ReviewDecision]

    global_audit_flags: list[GlobalAuditFlag]

    final_outcomes: list[str]

    evidence_trail: dict[str, Evidence]

    

    # Primary output

    outcome_matrix: OutcomeMatrix    # parameter × outcome × evidence

    

    # Optional report

    report: dict | None

```


## Scalability Considerations

- Vision passes are embarrassingly parallel: `images × models × focus_groups` can all run concurrently
- For 3 images, 3 models, 3 focus groups = 27 parallel calls + 3 image analyzer calls + 1 review call = 31 total LLM calls
- Report building is sub-second (JSON lookups)
- New parameters/groups only require datasource updates; pipeline auto-adapts via Parameter Planner

## See Also

For detailed implementation specs, see `docs/architecture/`:

- [Domain Model](architecture/domain-model.md) — canonical entities, enums, relationships, open decisions
- [Datasource Contracts](architecture/datasource-contracts.md) — schemas, validation rules, migration notes
- [Agent Orchestration](architecture/agent-orchestration.md) — LangGraph state, nodes, routing, conflict resolution
- [Report Contract](architecture/report-contract.md) — exact output schema, scoring, rendering