ui-regression-testing-2 / FRAMEWORK_MAPPING.md
riazmo's picture
Upload 61 files
6f38c76 verified

114-Point Visual Differences Framework Mapping

Overview

This document maps the user's 13 manually annotated differences to the comprehensive 114-point visual differences framework across 10 categories.

Framework Categories

1. Layout & Structure (8 issues)

Detects: Overall page layout, container structure, grid systems, responsive behavior

Issue User Annotation Detection Method Status
1.1 Header height difference Screenshot pixel analysis βœ… Detected
1.2 Container width differs Screenshot pixel analysis βœ… Detected
1.3 Main content area sizing CSS extraction ⚠️ Partial
1.4 Grid column count changes HF Vision + CSS ❌ Missing
1.5 Responsive breakpoint behavior Screenshot comparison ❌ Missing
1.6 Page width constraints CSS extraction ❌ Missing
1.7 Viewport scaling issues Screenshot analysis ❌ Missing
1.8 Layout alignment shifts Pixel comparison ❌ Missing

2. Typography (10 issues)

Detects: Font properties, text styling, readability changes

Issue User Annotation Detection Method Status
2.1 Font family differs CSS extraction βœ… Detected
2.2 Font size changes CSS extraction βœ… Detected
2.3 Letter spacing differs CSS extraction βœ… Detected
2.4 Font weight (bold) changes CSS extraction βœ… Detected
2.5 Line height differs CSS extraction ❌ Missing
2.6 Text transform (uppercase/lowercase) CSS extraction ❌ Missing
2.7 Text decoration (underline, strikethrough) CSS extraction ❌ Missing
2.8 Font style (italic) changes CSS extraction ❌ Missing
2.9 Text alignment differs CSS extraction ❌ Missing
2.10 Text color/contrast changes Color analysis ❌ Missing

3. Colors & Contrast (10 issues)

Detects: Color changes, contrast ratios, accessibility compliance

Issue User Annotation Detection Method Status
3.1 Text color changes Pixel analysis ❌ Missing
3.2 Background color differs Pixel analysis ❌ Missing
3.3 Border color changes CSS extraction ❌ Missing
3.4 Icon color differs Pixel analysis ❌ Missing
3.5 Contrast ratio fails WCAG Color analysis ❌ Missing
3.6 Gradient colors differ Pixel analysis ❌ Missing
3.7 Shadow color changes CSS extraction ❌ Missing
3.8 Hover state colors differ CSS extraction ❌ Missing
3.9 Focus state colors differ CSS extraction ❌ Missing
3.10 Opacity/transparency changes CSS extraction ❌ Missing

4. Spacing & Sizing (8 issues)

Detects: Margins, padding, gaps, element sizing

Issue User Annotation Detection Method Status
4.1 Padding (left, right) differs CSS extraction βœ… Detected
4.2 Padding (top, bottom) differs CSS extraction ❌ Missing
4.3 Margin differs CSS extraction ❌ Missing
4.4 Gap between components differs Screenshot analysis βœ… Detected
4.5 Element width differs CSS extraction ❌ Missing
4.6 Element height differs CSS extraction ❌ Missing
4.7 Min/max width constraints CSS extraction ❌ Missing
4.8 Min/max height constraints CSS extraction ❌ Missing

5. Borders & Outlines (6 issues)

Detects: Border styles, widths, colors, radius

Issue User Annotation Detection Method Status
5.1 Border width changes CSS extraction ❌ Missing
5.2 Border style changes CSS extraction ❌ Missing
5.3 Border radius differs CSS extraction ❌ Missing
5.4 Border color changes CSS extraction ❌ Missing
5.5 Outline style changes CSS extraction ❌ Missing
5.6 Shadow/border visibility Screenshot analysis ❌ Missing

6. Shadows & Effects (7 issues)

Detects: Box shadows, text shadows, filters, effects

Issue User Annotation Detection Method Status
6.1 Button shadow/elevation missing CSS extraction βœ… Detected
6.2 Shadow blur radius differs CSS extraction ❌ Missing
6.3 Shadow offset differs CSS extraction ❌ Missing
6.4 Shadow color changes CSS extraction ❌ Missing
6.5 Text shadow differs CSS extraction ❌ Missing
6.6 Filter effects differ CSS extraction ❌ Missing
6.7 Backdrop blur differs CSS extraction ❌ Missing

7. Components & Elements (10 issues)

Detects: Missing components, visibility, element presence

Issue User Annotation Detection Method Status
7.1 Login link missing HF Vision + DOM analysis βœ… Detected
7.2 Payment component not visible HF Vision + DOM analysis βœ… Detected
7.3 Payment methods section missing HF Vision + DOM analysis βœ… Detected
7.4 Icons missing HF Vision + DOM analysis βœ… Detected
7.5 Image missing DOM analysis ❌ Missing
7.6 Button missing DOM analysis ❌ Missing
7.7 Form field missing DOM analysis ❌ Missing
7.8 Navigation item missing DOM analysis ❌ Missing
7.9 Modal/overlay missing DOM analysis ❌ Missing
7.10 Tooltip/help text missing DOM analysis ❌ Missing

8. Buttons & Interactive (10 issues)

Detects: Button styling, states, interactions

Issue User Annotation Detection Method Status
8.1 Button size differs CSS extraction βœ… Detected
8.2 Button height differs CSS extraction βœ… Detected
8.3 Button color differs CSS extraction βœ… Detected
8.4 Button shadow/elevation missing CSS extraction βœ… Detected
8.5 Button border radius differs CSS extraction ❌ Missing
8.6 Button text styling differs CSS extraction ❌ Missing
8.7 Button hover state differs CSS extraction ❌ Missing
8.8 Button active state differs CSS extraction ❌ Missing
8.9 Button disabled state differs CSS extraction ❌ Missing
8.10 Button animation differs CSS extraction ❌ Missing

9. Forms & Inputs (10 issues)

Detects: Form field styling, validation states, placeholders

Issue User Annotation Detection Method Status
9.1 Input field size differs CSS extraction ❌ Missing
9.2 Input field border differs CSS extraction ❌ Missing
9.3 Input field padding differs CSS extraction ❌ Missing
9.4 Placeholder text styling differs CSS extraction ❌ Missing
9.5 Label styling differs CSS extraction ❌ Missing
9.6 Error message styling differs CSS extraction ❌ Missing
9.7 Focus state styling differs CSS extraction ❌ Missing
9.8 Validation state styling differs CSS extraction ❌ Missing
9.9 Checkbox/radio styling differs CSS extraction ❌ Missing
9.10 Select dropdown styling differs CSS extraction ❌ Missing

10. Images & Media (8 issues)

Detects: Image sizing, aspect ratios, visibility

Issue User Annotation Detection Method Status
10.1 Image size different Screenshot analysis βœ… Detected
10.2 Image aspect ratio differs Screenshot analysis ❌ Missing
10.3 Image alignment differs Screenshot analysis ❌ Missing
10.4 Image border radius differs CSS extraction ❌ Missing
10.5 Image opacity differs CSS extraction ❌ Missing
10.6 Image filter differs CSS extraction ❌ Missing
10.7 Video player styling differs CSS extraction ❌ Missing
10.8 Media container sizing differs CSS extraction ❌ Missing

Summary Statistics

Category Total Detected Detection Rate
Layout & Structure 8 2 25%
Typography 10 4 40%
Colors & Contrast 10 0 0%
Spacing & Sizing 8 2 25%
Borders & Outlines 6 0 0%
Shadows & Effects 7 1 14%
Components & Elements 10 4 40%
Buttons & Interactive 10 4 40%
Forms & Inputs 10 0 0%
Images & Media 8 1 13%
TOTAL 87 18 21%

User's 13 Annotated Differences Mapping

# Difference Framework Category Framework Issue Status
1 Header height difference Layout & Structure 1.1 βœ…
2 Container width differs Layout & Structure 1.2 βœ…
3 Checkout placement difference Components & Elements 7.1 βœ…
4 Font family, size, letter spacing differs Typography 2.1, 2.2, 2.3 βœ…
5 Login link missing Components & Elements 7.1 βœ…
6 Payment component not visible Components & Elements 7.2 βœ…
7 Button size, height, color, no elevation/shadow Buttons & Interactive 8.1, 8.2, 8.3, 8.4 βœ…
8 Payment methods design missing Components & Elements 7.3 βœ…
9 Contact info & step number missing, font bold Typography 2.4 βœ…
10 Icons missing Components & Elements 7.4 βœ…
11 Padding (left, right) differs Spacing & Sizing 4.1 βœ…
12 Image size different Images & Media 10.1 βœ…
13 Spacing between components differs Spacing & Sizing 4.4 βœ…

User's 13 Differences Detection Rate: 100% (13/13)

Enhancement Recommendations

To improve detection rate from 21% to >90%, focus on:

  1. Color Analysis - Add RGB comparison for all color properties
  2. CSS State Detection - Extract hover, focus, active states
  3. Form Field Analysis - Detect input, select, checkbox styling
  4. Border & Shadow Details - Parse CSS for exact measurements
  5. Responsive Behavior - Test at multiple breakpoints
  6. Animation Detection - Analyze CSS animations and transitions
  7. Accessibility - Check ARIA attributes and contrast ratios
  8. DOM Structure - Compare element hierarchy and nesting

Detection Methods

  • Screenshot Analysis: Pixel-level comparison using OpenCV
  • CSS Extraction: Parse computed styles from website
  • HF Vision Model: Semantic understanding of visual differences
  • DOM Analysis: Compare HTML structure and attributes
  • Color Analysis: RGB/HSL comparison and contrast checking
  • Pixel Comparison: Direct pixel-by-pixel analysis

Implementation Status

  • βœ… Screenshot Analysis: Implemented
  • βœ… CSS Extraction: Implemented
  • βœ… HF Vision Model: Integrated
  • ⚠️ DOM Analysis: Partial
  • ❌ Color Analysis: Needs enhancement
  • ❌ Advanced State Detection: Not implemented