deploy at 2026-01-05 20:05:15.533029

CLAUDE.md

@@ -0,0 +1,87 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+This is an NLP-based readability feedback tool that helps writers improve their text by identifying sentence segments where related words are separated by long interruptions. The tool is based on the linguistic principle of Dependency Length Minimisation (DLM), which suggests that keeping syntactically related words close together reduces cognitive load and improves readability.
+
+The application is built with FastHTML and uses spaCy for dependency parsing, deployed as a Docker container suitable for Hugging Face Spaces.
+
+## Development Commands
+
+### Setup
+```bash
+# Install dependencies (including spaCy model)
+pip install -r requirements.txt
+```
+
+### Running the Application
+```bash
+# Run the web app locally (starts on port 5001 by default)
+python main.py
+```
+
+### Docker
+```bash
+# Build the Docker image
+docker build -t readability-feedback .
+
+# Run the container
+docker run -p 7860:7860 readability-feedback
+```
+
+## Architecture
+
+### Core NLP Pipeline (main.py:29-220)
+
+The application uses a custom dependency parsing approach that converts spaCy's output to Surface Syntactic Universal Dependencies (SUD) framework:
+
+1. **sudify(doc)** (main.py:29-157): Transforms spaCy dependency relations into SUD-style relations with three main categories:
+   - `subj`: subject relations (nsubj, nsubjpass, csubj, etc.)
+   - `comp`: complement relations (dobj, ccomp, xcomp, aux, etc.)
+   - `mod`: modifier relations (advmod, advcl, relcl, etc.)
+   - `udep`: underspecified dependencies (acl, amod, nmod, etc.)
+
+   The function performs several transformations:
+   - Reverses auxiliary/mark/case dependencies to create more linguistically accurate heads
+   - Handles semicolons/colons as clause separators (main.py:111-133)
+   - Classifies prepositional phrases as modifiers or complements based on position
+   - Identifies oblique clauses (ccomp that should be mod)
+
+2. **flyover(token)** (main.py:160-176): For each token with a `subj` or `comp` relation, calculates the "flyover" span (words between head and dependent) and its complexity (number of heads in the span). This implements the revised dependency length metric from Yadav et al. (2022).
+
+3. **get_fluff(doc)** (main.py:179-219): Identifies all maximal flyover spans that should be highlighted in the output. Uses a filtering algorithm to:
+   - Remove nested flyovers (keep only the outermost or most complex ones)
+   - Collect interstices (non-flyover spans) to reconstruct the full sentence
+   - Return a sorted list of (span, complexity) tuples for rendering
+
+### Web Application (main.py:222-410)
+
+Built with FastHTML, a modern Python web framework:
+
+- **index()** route (main.py:230-254): Main page with textarea input and output display area using HTMX for dynamic updates
+- **about()** route (main.py:257-365): Documentation page explaining the linguistic theory and technical implementation
+- **send(text)** route (main.py:368-406): POST endpoint that:
+  - Splits input into paragraphs
+  - Processes each paragraph through the NLP pipeline
+  - Returns HTML with highlighted spans using opacity proportional to flyover complexity
+
+### Key Implementation Details
+
+- The app uses spaCy's `en_core_web_sm` model (loaded at main.py:6)
+- Highlighting uses background color with opacity scaled by dependency distance: `rgba(237, 201, 241, {a[1]/5})` for light mode
+- The sudify function handles complex edge cases like:
+  - Multiple subjects (main.py:96-103): only the closest subject is kept as `subj`, others become `comp`
+  - Parenthetical expressions: semicolons/colons are ignored if wrapped in parentheses (main.py:117-128)
+  - Post-core conjunctions: conj dependents after the core clause become modifiers (main.py:134-150)
+
+## Notes on Modification
+
+When modifying the dependency transformation logic (sudify function), be aware that:
+- Order of transformations matters: some rules depend on earlier transformations
+- The function modifies the spaCy Doc object in place by reassigning `.head` and `.dep_` attributes
+- Edge case handling for punctuation and parentheses is critical for accurate parsing
+- The three-pass structure (main.py:29-55, 56-156, 92-156) progressively refines the dependency tree
+
+The highlighting algorithm (get_fluff) uses span comparison logic that can be subtle to debug. The key insight is that overlapping flyovers are resolved by keeping the longer or more complex one.

main.py

@@ -229,14 +229,23 @@ app, rt = fast_app(pico=True)
 @app.get
 def index():
     page = Div(
-        Form(hx_post=send, hx_target="#output", hx_swap="outerHTML")(
+        Form(
+            hx_post=send,
+            hx_target="#output",
+            hx_swap="outerHTML show:none",
+        )(
             Div(
                 Span(
                     Button("Check"),
                     A("How this works", href="/about"),
                     style="margin-bottom: 1rem; display: flex; gap: 1rem; align-items: center",
                 ),
-                Textarea(name="text", style="height: calc(100vh - 11rem)"),
+                Textarea(
+                    name="text",
+                    id="input-text",
+                    style="height: calc(100vh - 11rem);",
+                    onscroll="document.getElementById('output').scrollTop = this.scrollTop + 1; document.getElementById('output').scrollLeft = this.scrollLeft;",
+                ),
             )
         ),
         Div(
@@ -247,7 +256,7 @@ def index():
                 cls="overflow-auto",
                 style="height: 4rem; text-wrap: balance; padding: 0rem 1rem",
             ),
-            Div(id="output", style="padding: 1rem; padding-bottom: calc(1rem - 5px)"),
+            Div(id="output", style="padding: 1rem calc(1rem - 5px)"),
         ),
         cls="grid",
     )
@@ -370,7 +379,13 @@ def send(text: str):
     paragraphs = re.sub(r"[^\S\r\n]+", " ", text).split("\r\n\r\n")
     docs = [sudify(nlp(para)) for para in paragraphs]
     annot_paras = [get_fluff(doc) for doc in docs]
+
+    sync_script = Script(
+        "setTimeout(() => { const textarea = document.getElementById('input-text'); const output = document.getElementById('output'); if (textarea && output) { output.scrollTop = textarea.scrollTop + 1; output.scrollLeft = textarea.scrollLeft; } }, 100);"
+    )
+
     return Div(
+        sync_script,
         *[
             P(
                 *[
@@ -402,7 +417,8 @@ def send(text: str):
         ),
         id="output",
         cls="overflow-auto",
-        style="height: calc(100vh - 11rem); padding: 1rem; padding-bottom: calc(1rem - 5px)",
+        style="height: calc(100vh - 11rem); padding: 1rem; padding-bottom: calc(1rem - 5px);",
+        onscroll="document.getElementById('input-text').scrollTop = this.scrollTop - 1; document.getElementById('input-text').scrollLeft = this.scrollLeft;",
     )