app.py

"""
GLM-OCR Hugging Face Space app for PDF/image OCR with header inclusion
and table-structure stabilization for downstream bank-statement pipelines.
Hard-coded knobs (no environment variables required).
Primary goals for reconcile rate:
- preserve right-most columns (often "Balance") by higher DPI render + right padding
- keep tables as tables (convert markdown pipe tables -> HTML table)
- return ---page-separator--- between pages
- normalize HTML tables generically so downstream parsing/classification is stable:
  1) expand colspan/rowspan into a rectangular grid
  2) drop truly-empty columns (common in summary tables)
  3) merge "blank header" columns that contain text into the left column (common when DESCRIPTION is split)
  4) clean common header artifacts (e.g. "DESCRIPTIONBeginning Balance", "BALANCE$3,447.10")
  5) recover fused first data row generically (two-signal guard: text fusion + money fusion must both fire)
  6) promote misplaced header row: when real column headers land in a data row, restructure the grid
  7) fix fused key-value rows in summary sections (e.g. Interest Summary) appended to transaction tables
  8) reconstruct mid-table separator rows split across columns by OCR (e.g. 'Card account # XXXX 2 | | 889')
- footer extraction enabled on all pages with deduplication guard (no double-print if body OCR already captured it)
- Fix 4: cross-validate OCR table row counts against PDF text layer; inject missing duplicate rows
         (safe no-op for scanned PDFs and non-transaction pages); after injection, re-run normalize_html_tables
         + subset dedupe so UCB-style fused cells from the text layer get Fix 13/14
- Fix NF-1: extract Navy Federal "Summary of your deposit accounts" table from PDF text layer
            using spatial word positions (GLM-OCR fails on this wide-column layout)
- Fix 13: collapse Date|Description|Amount|Description|Amount tables to 3 columns; fix fused
            Beginning/Ending balance rows (UCB-style wide summary tables)
- Fix 14: UCB 3-col tables where Beginning Balance amount is fused into description and Amount is $0.00
- Fix 15: UCB (and similar) 3-col rows where Date+Description are fused in col1 and Amount sits in col2 with col3 empty
- Fix 16: remove duplicate rows within a single Date|Description|Amount table; multi-pass table dedupe + final doc-wide dedupe
- Fix 17: UCB page 3 — split one merged table under Deposits (continued) into Deposits + Electronic Credits +
            Electronic Debits (PDF text order); drop orphan empty EC/ED headers that followed the merged table
- Fix 17b: When markdown has Electronic Credits (+ optional Electronic Debits headings) but a single fused
            Date|Description|Amount table (common when Deposits (continued) is absent from OCR), split after a
            leading ITM DEPOSIT run when the next row matches generic electronic-debit patterns (POS purchase,
            ATM withdrawal, etc.). Safe no-op when there is no ITM prefix or no such debit row after it.
- Fix 17c: When Electronic Debits appears *before* the table but every data row is credit-side (no strong
            debit pattern) and at least one ITM DEPOSIT row exists, OCR stacked headings wrong — emit
            Electronic Credits, then the table, then Electronic Debits (section continues below). Safe no-op
            if the Debits heading was not before the table or any row looks like a card/ATM debit.
- Fix 17c-html: Same as 17c using raw HTML cues when TableGridParser/header normalization would skip the table.
- Fix 17d: Drop a second Electronic Debits block + DA3 table when its rows are a loose (date+amount)
            multiset subset of the immediately preceding DA3 table (OCR duplicate tail / wrong split).
- Fix 17f: Same as 17d when the duplicate table *starts with* ITM DEPOSIT rows (copied from Electronic Credits)
            then repeats debits from the table above — 17d alone misses those because ITM keys are not in the
            debits-only predecessor table. Strip ITM rows from the multiset check, then apply the 17d subset rule.
- Fix 17g: When 17f cannot drop the whole block (e.g. one orphan debit row only appears in the duplicate tail),
            drop leading ITM rows and each fragment row whose (date, amount) still has count in the previous
            debits table; remove the ED wrapper if the table becomes empty. If a small orphan tail remains
            (≤6 rows), append those rows to the preceding debits table and drop the extra ED heading.
            Skip the *first* Electronic Debits + table on the page when the previous table is ITM/credits-only
            and the fragment is clearly debits (so we do not treat the main debits section as a duplicate).
- Fix 17h: UCB sometimes OCRs Electronic Credits (+ optional Electronic Debits) then the *main debits* DA3
            table first and a *second* DA3 table whose rows start with ITM DEPOSIT (credits) plus duplicate
            debits. Reorder to Credits + ITM table + Electronic Debits + merged debits; drop dup keys from the
            second table against the first; merge remaining orphan rows into the debits table.
- Fix 17i: After Fix 17b splits ITM prefix from debits, OCR often leaves *credit* rows (POS Return, DDA
            Transfer IN, Apple Cash INST R / CUPERTINO) in the debits grid — move those rows to Electronic
            Credits (contiguous-tail case on g_ed after 17b, or any matching row via 17j).
- Fix 17j: When OCR emits a *second* Electronic Debits table (duplicate block), credit-only rows often appear
            only there — relocate them to the nearest preceding Electronic Credits table (same patterns as 17i).
- Fix 17e: Repeat EC/ED fused + 17d passes until stable (handles chained replacements). Centered section titles
            match both align=\"center\" and style=\"text-align:center\" on divs.
- Post-pass: dedupe 3-col Date|Description|Amount tables when one is a duplicate fragment
            (later subset of earlier, earlier subset of later, or identical row sets)
- Fix 18: adjacent DA3 tables — if the last K data rows of table N match the first K rows of table N+1
            (same normalized Date|Description|Amount keys), strip that suffix from table N. Fixes
            formats (e.g. Truist) where deposit rows are glued onto the withdrawals table and repeated
            under the proper deposits heading (K>=2; first table must keep >=1 data row).
- Fix 19: DA3 rows whose Amount cell is not strict currency (e.g. fused card digits + merchant tail)
            — take the rightmost plausible money token from Description+Amount text for the amount.
- Fix 20: DA3 in-table dedupe — cap consecutive identical normalized keys at 2 rows (keeps legitimate
            back-to-back identical charges; still collapses 3+ OCR stutters).
- Fix 21: Cross-table DA3 repair for MSBILL / MICROSOFT#G… PIN rows — when Amount is junk in one table
            but another table repeats the same authorization id with a strict amount, copy that amount
            (then subset dedupe can drop the stray duplicate block, e.g. under Deposits).
- Fix 22: Drop a single OCR "flat line" between pages: (continued) + DESCRIPTION + AMOUNT + many MM/DD
            tokens (noise before a proper <table>).
- Fix 23: Doc-wide DA3 table dedupe using multiset of (date, amount) keys — catches duplicate sections when
            OCR varies description text between copies (e.g. withdrawals repeated under Deposits).
- Fix 24: Run orphan flatline strip on merged PDF output and after text-layer patch so page-boundary
            noise is removed consistently.
- Fix 25: After a centered "deposits + credits/interest" heading, if two DA3 tables appear back-to-back
            and the first is mostly credit/deposit rows while the second is mostly debit/withdrawal-style
            rows (generic keywords), drop the second — fixes OCR pasting the withdrawals block again.
- Fix 23b: Loose multiset keys use normalized float amounts so comma/spacing variants still match.
- Fix 26: Drop small DA3 fragments (3–14 rows) that are a multiset subset of an earlier larger DA3
            table — matches PDF layouts that re-print a header and repeat a few rows.
- Fix 27: Strip a lone "…(continued)" account banner line (no DATE/DESCRIPTION/AMOUNT) between
            page separators — duplicate header noise without the long flatline.
- Fix 28: Strip leading credit/deposit rows that OCR prepends to a withdrawals/debits DA3 table
            when the preceding DA3 table was under a deposits/credits heading. Only strips the
            contiguous prefix of the debit table whose rows appear in the credit table; requires
            >=1 data row to remain; entire function wrapped in try/except (safe no-op on any error).
- Fix 29: Move a misplaced debit/withdrawal heading to before its orphan DA3 table when OCR
            places the heading AFTER the table instead of before it. Pattern detected:
            [credit heading+table] ... [orphan DA3 table] ... [debit heading] — the heading is
            relocated to just before the orphan table. Guards: no debit heading already before
            table; credit context must precede table; no other table between orphan and heading;
            try/except wrapper (safe no-op on any error).
- Fix 31: On check / receipt image pages, GLM sometimes emits bogus table rows (e.g. \"Witness LA…\",
            \"[hotline Control]\" after amounts, long trace ids). Strip those rows and bracket junk after
            currency amounts; drop a stray \"## UCB\" heading before such tables. Safe no-op when absent.
- Fix 20-override: _dedupe_duplicate_rows_in_da3_table now uses the PDF text-layer frequency
            map (_tl_freq_context) as its cap so legitimate repeated transactions (e.g. 6x
            Temporary Credit Reversal on the same date/amount) are preserved rather than
            being collapsed to 2. Keys absent from the text layer still cap at 2 (OCR stutter
            behaviour unchanged). Safe no-op for scanned PDFs with no text layer.
- Fix 30: Drop small DA3 fragments (1-5 rows) that are a loose (date+amount) subset of an
            earlier larger DA3 table — catches OCR-truncated description duplicates such as
            the BoA service-fee page where a VIDDYOZE -1.11 fragment table appears after the
            full service-fee table with a longer description string.
- Fix 32: Headless trailing <th> over currency cells — reuse the prior money header (Additions/
            Subtractions) when present, else \"Amount\". Fix 32b aligns Date|Transaction Description|
            Additions/Subtractions tables to the PDF: credits as Number|Date|Description|Additions
            (blank Number when unused); debits as Date|Description|Subtractions (no Number); merges
            OCR column splits. Adjacent duplicate HTML tables (same header + row multiset) are removed.
- Fix TD-1: TD Bank continuation register: (a) four OCR columns with DATE+DESCRIPTION fused +
            DEBIT/CREDIT/BALANCE — split col0 into Date|Description; (b) three OCR columns
            Date|Transaction Description|Amount where Amount is the running balance — expand to
            five columns using trailing amount in the description cell and TD-specific debit/credit
            routing. Fires on \"continued\" in-table text and/or TD body cues (TD ZELLE, LENDINGCLUB,
            etc.). Safe no-op elsewhere.
- Fix MD-1: Markdown fidelity — (a) collapse adjacent HTML tables that are the same register
            (identical multiset of date + debit + credit + balance per row, descriptions may differ);
            keep the table with richer description text so output matches the PDF once. (b) Common OCR
            typos (e.g. CINTHIAL→CINTHIA L) and double-escaped ampersands (&amp;amp;→&amp;).
            Gap between tables must not contain another <table> or page-separator.
- Fix MD-2: Skip blank-header column merge for YOUR CHECKS SEQUENCED–style tables (title + repeated
            Amount columns + blank th between triplets); merge was collapsing multi-column check rows.
- Fix MD-3: Check image pages — GLM often emits ![](bbox…) plus <div align="center"> fragments
            (e.g. 2293 $6,721.00 4/29/2025). Collapse detected gallery pages into one Check #|Amount|Date
            table when bbox/image markers + repeated check lines are present; skips chunks with YOUR CHECKS SEQUENCED.
"""

# Patch asyncio first (before Gradio imports it) to suppress Python 3.13 cleanup noise
import asyncio
try:
    _orig_close = asyncio.BaseEventLoop.close
    def _safe_close(self):
        try:
            _orig_close(self)
        except (ValueError, OSError):
            pass
    asyncio.BaseEventLoop.close = _safe_close
except Exception:
    pass

import logging
import os
import re
import html
import tempfile
from typing import List, Optional, Tuple
from collections import Counter, defaultdict
from html.parser import HTMLParser

import yaml
import gradio as gr
import glmocr

log = logging.getLogger("glmocr_app")
logging.basicConfig(level=logging.INFO)

GLMOCR_BASE = os.path.dirname(glmocr.__file__)
CONFIG_PATH = os.path.join(GLMOCR_BASE, "config.yaml")
FORMATTER_PATH = os.path.join(GLMOCR_BASE, "postprocess", "result_formatter.py")

# ============================================================
# HARD-CODED SETTINGS (edit these numbers to tune quality/speed)
# ============================================================

GLMOCR_API_KEY = "cee1d52dd91a4ab591b3f6e105f8ad89.LgbQTECuzX0zrito"

# Higher = better OCR for small/right-aligned digits; slower
RENDER_SCALE = 2.2  # try 2.5 if right-side numbers are missed

# Padding to protect columns near edges (Balance is often right-most)
PAD_LEFT_FRAC = 0.02
PAD_RIGHT_FRAC = 0.06  # try 0.10 if right-most balances are missing
PAD_TOP_FRAC = 0.01
PAD_BOTTOM_FRAC = 0.01

ENABLE_CONTRAST = True

DEFAULT_ZONE_FRAC = 0.12
PDF_HEADER_BAND_FRAC = 0.10

ENABLE_FOOTER_OCR = True        # enabled — 3-step fallback with dedup guard
PDF_FOOTER_BAND_FRAC = 0.88

MIN_CROP_HEIGHT = 112
MIN_CROP_PIXELS = 112 * 112

# ============================================================

_parser = None

def get_parser():
    global _parser
    if _parser is None:
        from glmocr import GlmOcr
        _parser = GlmOcr(api_key=GLMOCR_API_KEY, mode="maas")
    return _parser

# ---------------------------------------------------------------------------
# Best-effort config tweaks (safe to fail on read-only HF env)
# ---------------------------------------------------------------------------
try:
    with open(CONFIG_PATH, "r") as f:
        config = yaml.safe_load(f)
    config["pipeline"]["maas"]["enabled"] = True
    config["pipeline"]["maas"]["api_key"] = GLMOCR_API_KEY
    with open(CONFIG_PATH, "w") as f:
        yaml.dump(config, f, default_flow_style=False, sort_keys=False)
except Exception:
    pass

# Best-effort formatter tweak: avoid stripping header/footer labels
try:
    with open(FORMATTER_PATH, "r") as f:
        source = f.read()
    for label in (
        '"header"', "'header'",
        '"footer"', "'footer'",
        '"doc_header"', "'doc_header'",
        '"doc_footer"', "'doc_footer'",
    ):
        source = re.sub(r",\s*" + re.escape(label), "", source)
        source = re.sub(re.escape(label) + r"\s*,", "", source)
        source = re.sub(re.escape(label), "", source)
    with open(FORMATTER_PATH, "w") as f:
        f.write(source)
except Exception:
    pass

# --------------------------
# Header/footer helpers
# --------------------------
def get_header_footer_zones(regions, norm_height=1000):
    if not regions:
        return None, None
    y_tops, y_bottoms = [], []
    for r in regions:
        bbox = r.get("bbox_2d") if isinstance(r, dict) else getattr(r, "bbox_2d", None)
        if bbox and len(bbox) >= 4:
            y_tops.append(bbox[1])
            y_bottoms.append(bbox[3])
    if not y_tops:
        return None, None
    return min(y_tops) / norm_height, max(y_bottoms) / norm_height

def extract_zone_text_pdf(pdf_path, page_num, y_start_frac, y_end_frac):
    try:
        import pymupdf as fitz
        doc = fitz.open(pdf_path)
        page = doc[page_num]
        h, w = page.rect.height, page.rect.width
        rect = fitz.Rect(0, h * y_start_frac, w, h * y_end_frac)
        text = page.get_text(clip=rect).strip()
        doc.close()
        return text
    except Exception:
        return ""

def extract_pdf_text_in_band(pdf_path, page_num, y_start_frac, y_end_frac):
    try:
        import pymupdf as fitz
        doc = fitz.open(pdf_path)
        page = doc[page_num]
        h = page.rect.height
        y_lo = h * y_start_frac
        y_hi = h * y_end_frac
        words = page.get_text("words")
        doc.close()
        parts = []
        for w in words:
            if len(w) >= 5:
                y0, y1 = float(w[1]), float(w[3])
                if y0 < y_hi and y1 > y_lo:
                    parts.append(w[4])
        return " ".join(parts).strip()
    except Exception:
        return ""

def ocr_zone(image_path, y_start_frac, y_end_frac):
    zone_name = "header" if y_end_frac < 0.5 else "footer"
    try:
        from PIL import Image
        img = Image.open(image_path).convert("RGB")
        w, h = img.size
        y0 = max(0, int(h * y_start_frac))
        y1 = min(h, int(h * y_end_frac))
        if y1 <= y0:
            return ""

        crop = img.crop((0, y0, w, y1))
        cw, ch = crop.size

        if ch < MIN_CROP_HEIGHT or (cw * ch) < MIN_CROP_PIXELS:
            need_h = max(ch, MIN_CROP_HEIGHT)
            need_w = max(cw, 1)
            if (need_w * need_h) < MIN_CROP_PIXELS:
                need_w = max(need_w, (MIN_CROP_PIXELS + need_h - 1) // need_h)
            canvas = Image.new("RGB", (need_w, need_h), (255, 255, 255))
            if zone_name == "header":
                canvas.paste(crop, (0, 0))
            else:
                canvas.paste(crop, (0, need_h - ch))
            crop = canvas

        fd, path = tempfile.mkstemp(suffix=".jpg")
        os.close(fd)
        try:
            crop.save(path, "JPEG", quality=92)
            parser = get_parser()
            out = parser.parse(path)
            if not isinstance(out, list):
                out = [out]
            if out and getattr(out[0], "markdown_result", None):
                return (out[0].markdown_result or "").strip()
        finally:
            try:
                os.unlink(path)
            except Exception:
                pass
    except Exception as e:
        log.warning("[%s] ocr_zone failed: %s", zone_name, e, exc_info=True)
    return ""

def fix_account_number(hdr: str) -> str:
    if not hdr:
        return hdr
    if "Account Number:" in hdr and "Account Number: " not in hdr:
        m = re.search(r"[0-9]{5,}", hdr)
        if m:
            hdr = hdr.replace("Account Number:", "Account Number: " + m.group(0))
    acct_match = re.search(r"Account Number: ([0-9]{5,})", hdr)
    if acct_match:
        acct = acct_match.group(1)
        if hdr.startswith(acct):
            hdr = hdr[len(acct):].lstrip()
    return hdr

# --------------------------
# Table stabilization helpers
# --------------------------
def close_unclosed_html(md: str) -> str:
    if not md:
        return md
    open_tags = re.findall(r"<(table|tbody|thead|tr|td|th)\b", md, flags=re.IGNORECASE)
    close_tags = re.findall(r"</(table|tbody|thead|tr|td|th)>", md, flags=re.IGNORECASE)

    def count(tags, name):
        return sum(1 for t in tags if t.lower() == name)

    for tag in reversed(["td", "th", "tr", "thead", "tbody", "table"]):
        opened = count(open_tags, tag)
        closed = count(close_tags, tag)
        if opened > closed:
            md += ("</%s>" % tag) * (opened - closed)
    return md

def looks_like_markdown_table(block: str) -> bool:
    lines = [ln.rstrip() for ln in block.strip().splitlines() if ln.strip()]
    if len(lines) < 2:
        return False
    if "|" not in lines[0]:
        return False
    sep = lines[1].replace(" ", "")
    return ("---" in sep) and ("|" in sep)

def md_table_to_html(block: str) -> str:
    lines = [ln.strip() for ln in block.strip().splitlines() if ln.strip()]
    if len(lines) < 2:
        return block

    def split_row(row: str):
        row = row.strip()
        if row.startswith("|"):
            row = row[1:]
        if row.endswith("|"):
            row = row[:-1]
        return [p.strip() for p in row.split("|")]

    header = split_row(lines[0])
    body_lines = [ln for ln in lines[2:] if "|" in ln]

    html_rows = []
    html_rows.append("<tr>" + "".join(f"<th>{html.escape(c)}</th>" for c in header) + "</tr>")
    for ln in body_lines:
        cols = split_row(ln)
        if len(cols) < len(header):
            cols += [""] * (len(header) - len(cols))
        html_rows.append("<tr>" + "".join(f"<td>{html.escape(c)}</td>" for c in cols[: len(header)]) + "</tr>")
    return "<table>\n" + "\n".join(html_rows) + "\n</table>"

def normalize_money_glyphs(text: str) -> str:
    if not text:
        return text
    t = text.replace("−", "-").replace("–", "-").replace("—", "-")
    t = re.sub(
        r"\(\s*\$?\s*([0-9]{1,3}(?:,[0-9]{3})*|[0-9]+)(\.[0-9]{2})\s*\)",
        r"-\1\2",
        t,
    )

    def o_to_zero(m):
        token = m.group(0)
        return token.replace("O", "0").replace("o", "0")

    t = re.sub(r"\b[0-9Oo\$,.\-]{4,}\b", o_to_zero, t)
    return t

# ---- Generic HTML table normalizer ----
class TableGridParser(HTMLParser):
    """Parse a <table> into rows of (text, colspan, rowspan)."""

    def __init__(self):
        super().__init__()
        self.rows = []
        self._current_row = []
        self._cell_text = []
        self._colspan = 1
        self._rowspan = 1
        self._in_cell = False

    def handle_starttag(self, tag, attrs):
        if tag == "tr":
            self._current_row = []
        elif tag in ("td", "th"):
            attrs_d = dict(attrs)
            self._colspan = max(1, int(attrs_d.get("colspan", 1)))
            self._rowspan = max(1, int(attrs_d.get("rowspan", 1)))
            self._cell_text = []
            self._in_cell = True

    def handle_endtag(self, tag):
        if tag in ("td", "th"):
            text = "".join(self._cell_text).strip().replace("\n", " ")
            self._current_row.append((text, self._colspan, self._rowspan))
            self._in_cell = False
        elif tag == "tr":
            self.rows.append(self._current_row)

    def handle_data(self, data):
        if self._in_cell:
            self._cell_text.append(data)

def _build_grid(rows_data):
    if not rows_data:
        return []
    blocked = defaultdict(set)
    grid = []

    for r, row_cells in enumerate(rows_data):
        grid.append([])
        col = 0
        for content, C, R in row_cells:
            while col in blocked[r]:
                grid[r].append("")
                col += 1
            for k in range(C):
                grid[r].append(content if k == 0 else "")
            for k in range(1, R):
                blocked[r + k].add(col)
            col += C

    max_cols = max(len(row) for row in grid) if grid else 0
    for row in grid:
        while len(row) < max_cols:
            row.append("")
    return grid

def _grid_to_html(grid):
    if not grid:
        return ""
    lines = ["<table>"]
    for r, row in enumerate(grid):
        lines.append("<tr>")
        tag = "th" if r == 0 else "td"
        for cell in row:
            escaped = (
                (cell or "")
                .replace("&", "&amp;")
                .replace("<", "&lt;")
                .replace(">", "&gt;")
                .replace('"', "&quot;")
            )
            lines.append(f"<{tag}>{escaped}</{tag}>")
        lines.append("</tr>")
    lines.append("</table>")
    return "\n".join(lines)

def _is_amount_like(s: str) -> bool:
    if not s:
        return False
    return re.fullmatch(r"\$?-?\d{1,3}(?:,\d{3})*(?:\.\d{2})?", s.strip()) is not None

def _drop_truly_empty_columns(grid):
    if not grid or len(grid) < 1:
        return grid

    header = [str(c or "").strip() for c in grid[0]]
    ncols = len(header)
    if ncols <= 1:
        return grid

    body = grid[1:]
    keep = [True] * ncols

    for i in range(ncols):
        if header[i] != "":
            continue
        total = 0
        non_empty = 0
        for row in body:
            if i >= len(row):
                continue
            total += 1
            if str(row[i] or "").strip():
                non_empty += 1
        if total > 0 and (non_empty / total) <= 0.05:
            keep[i] = False

    if all(keep):
        return grid

    new_grid = []
    for row in grid:
        new_grid.append([cell for idx, cell in enumerate(row) if idx < len(keep) and keep[idx]])
    return new_grid


def _is_multigroup_check_list_header(header: List[str]) -> bool:
    """
    Wide check listings (e.g. YOUR CHECKS SEQUENCED) use one row with a title cell, repeated
    Amount headers, and blank <th> slots between DATE | CHECK # | AMOUNT triplets. Fix 3
    blank-header merge would otherwise fuse the next column into Amount or the section title,
    leaving only the first date column visibly populated.
    """
    if not header:
        return False
    joined = " ".join(str(c or "").strip() for c in header).upper()
    if "SEQUENCED" in joined and "CHECK" in joined:
        return True
    # Repeated Amount columns in one header row = multi-group layout; do not merge blanks.
    if sum(1 for c in header if str(c or "").strip().upper() == "AMOUNT") >= 2:
        return True
    return False


def _merge_blank_header_text_columns(grid):
    if not grid or len(grid) < 2:
        return grid

    header = [str(c or "").strip() for c in grid[0]]
    if _is_multigroup_check_list_header(header):
        return grid

    ncols = len(header)
    body = grid[1:]
    blank_cols = [i for i, h in enumerate(header) if h == ""]
    if not blank_cols:
        return grid

    keep = [True] * ncols

    for i in blank_cols:
        j = i - 1
        while j >= 0 and header[j] == "":
            j -= 1
        if j < 0:
            continue

        total = 0
        non_empty = 0
        texty = 0
        for row in body:
            if i >= len(row) or j >= len(row):
                continue
            v = str(row[i] or "").strip()
            total += 1
            if v:
                non_empty += 1
                if not _is_amount_like(v):
                    texty += 1

        if total == 0:
            continue
        non_empty_ratio = non_empty / total
        texty_ratio = (texty / non_empty) if non_empty else 0.0

        if non_empty_ratio >= 0.55 and texty_ratio >= 0.70:
            for r in range(1, len(grid)):
                row = grid[r]
                if i >= len(row) or j >= len(row):
                    continue
                left = str(row[j] or "").strip()
                right = str(row[i] or "").strip()
                if right:
                    row[j] = (left + " " + right).strip() if left else right
            keep[i] = False

    if all(keep):
        return grid

    new_grid = []
    for row in grid:
        new_grid.append([cell for idx, cell in enumerate(row) if idx < len(keep) and keep[idx]])
    return new_grid


def _fill_headless_trailing_amount_headers(grid):
    """
    Fix 32a: PDF layouts often omit a header above the rightmost amount column; OCR leaves a
    trailing empty <th>. Downstream parsers then mis-associate columns. Fill only when the
    column is predominantly strict currency (generic, bank-agnostic).
    """
    if not grid or len(grid) < 2:
        return grid

    header = [str(c or "").strip() for c in grid[0]]
    ncols = len(header)
    if ncols < 2:
        return grid

    i = ncols - 1
    while i >= 0 and header[i] == "":
        i -= 1
    empty_tail_indices = list(range(i + 1, ncols))
    if not empty_tail_indices:
        return grid

    body = grid[1:]
    new_header = list(header)

    for j in empty_tail_indices:
        total = 0
        amt_hits = 0
        for row in body:
            if j >= len(row):
                continue
            total += 1
            v = str(row[j] or "").strip()
            if v and _is_amount_like(v):
                amt_hits += 1
        if total == 0:
            continue
        if (amt_hits / total) < 0.45:
            continue

        left = j - 1
        while left >= 0 and not new_header[left]:
            left -= 1
        prev = (new_header[left] if left >= 0 else "").strip()
        pl = prev.lower()
        if any(
            k in pl
            for k in (
                "addition",
                "subtraction",
                "credit",
                "debit",
                "deposit",
                "withdrawal",
            )
        ):
            new_header[j] = prev
        else:
            new_header[j] = "Amount"

    grid[0] = new_header
    return grid


_MMDD_DATE_RE = re.compile(r"^\d{1,2}[-/]\d{2}(?:[-/]\d{2,4})?$")

# Leading transaction-type cell when OCR splits category vs merchant (East West / similar).
_STACKED_TXN_TYPE_RE = re.compile(
    r"^(?:Pre-Auth Credit|Pre-Auth Debit|Preauth Debit|"
    r"Withdrawal Trns|Withdrawal(?:\s|$)|Deposit(?:\s|$)|Deposit Transfe|"
    r"Outgoing Wire|Incoming Wire|"
    r"Service Charge|ACH Debit|ACH Credit|ACH Payment|ACH PMT|"
    r"Wire Transfer|POS Purchase|ATM Withdrawal|Transfer Trns)\b",
    re.IGNORECASE | re.DOTALL,
)


def _normalize_date_desc_money_table_alignment(grid):
    """
    Fix 32b: Align OCR tables with common bank PDF layouts.

    * Credits (Additions): Number | Date | Transaction Description | Additions — Number is often
      blank in the PDF but must exist as a column; OCR may omit it or split description across
      columns and park amounts under a bogus \"Amount\" header.
    * Debits (Subtractions): PDF is usually Date | Description | Subtractions (no Number) —
      when OCR uses the same 4-column split error, merge back to three columns.

    Handles (a) Date | short type | merchant tail | amount, (b) Date | full desc | empty | amount,
    (c) continuation rows with a blank date. Per-row merge (no single global mode) so mixed OCR
    errors (e.g. wrapped merchant lines) still normalize.
    """
    if not grid or len(grid) < 3:
        return grid

    header = [str(c or "").strip() for c in grid[0]]
    n = len(header)

    def _pad_row(cells, width):
        cells = [str(c or "").strip() for c in cells]
        while len(cells) < width:
            cells.append("")
        return cells[:width]

    h0n = re.sub(r"\s+", " ", header[0]).strip().lower()
    h1n = re.sub(r"\s+", " ", header[1]).strip().lower() if n > 1 else ""
    h2n = re.sub(r"\s+", " ", header[2]).strip().lower() if n > 2 else ""
    h3n = re.sub(r"\s+", " ", header[3]).strip().lower() if n > 3 else ""

    money_label = header[2].strip() if n > 2 else ""
    want_number_col = "addition" in h2n

    # --- 3-column Date | Transaction Description | Additions/Subtractions: prepend Number for credits only ---
    if n == 3:
        if h0n != "date" or "transaction" not in h1n or "description" not in h1n:
            return grid
        if not any(
            k in h2n
            for k in (
                "addition",
                "subtraction",
                "credit",
                "debit",
                "deposit",
                "withdrawal",
            )
        ):
            return grid
        if not want_number_col:
            return grid
        out = [["Number", header[0], header[1], money_label or "Additions"]]
        for row in grid[1:]:
            c = _pad_row(row, 3)
            if not any(c):
                out.append(["", "", "", ""])
                continue
            out.append(["", c[0], c[1], c[2]])
        return out

    if n != 4:
        return grid

    if h0n != "date":
        return grid
    if "transaction" not in h1n or "description" not in h1n:
        return grid
    if not any(
        k in h2n
        for k in (
            "addition",
            "subtraction",
            "credit",
            "debit",
            "deposit",
            "withdrawal",
        )
    ):
        return grid
    if h3n not in ("", "amount") and not any(
        k in h3n for k in ("addition", "subtraction", "credit", "debit")
    ):
        return grid

    data_rows = [r for r in grid[1:] if any(str(c or "").strip() for c in r)]
    if not data_rows:
        return grid

    has_date_led = False
    for row in data_rows:
        c = _pad_row(row, 4)
        if c[0] and _MMDD_DATE_RE.match(c[0]):
            has_date_led = True
            break
    if not has_date_led:
        return grid

    def _merge_date_led_row(c0, c1, c2, c3):
        split_ok = (
            bool(c1)
            and bool(_STACKED_TXN_TYPE_RE.match(c1))
            and bool(c2)
            and not _is_amount_like(c2)
            and _is_amount_like(c3)
        )
        if split_ok:
            return (c1 + " " + c2).strip(), c3
        shift_ok = (
            bool(c1)
            and not _is_amount_like(c1)
            and not (c2 or "").strip()
            and _is_amount_like(c3)
        )
        if shift_ok:
            return c1, c3
        dep_ok = (
            bool(c1)
            and bool(_STACKED_TXN_TYPE_RE.match(c1))
            and not (c2 or "").strip()
            and _is_amount_like(c3)
        )
        if dep_ok:
            return c1, c3
        desc = ((c1 + " " + c2).strip() or c1).strip()
        amt = c3 if _is_amount_like(c3) else ""
        return desc, amt

    ml = money_label or ("Additions" if want_number_col else "Subtractions")

    if want_number_col:
        out_h = ["Number", header[0], header[1], ml]
    else:
        out_h = [header[0], header[1], ml]

    out = [out_h]
    for row in grid[1:]:
        c = _pad_row(row, 4)
        c0, c1, c2, c3 = c[0], c[1], c[2], c[3]
        if not any(c):
            out.append([""] * len(out_h))
            continue
        if c0 and _MMDD_DATE_RE.match(c0):
            desc, amt = _merge_date_led_row(c0, c1, c2, c3)
            if want_number_col:
                out.append(["", c0, desc, amt])
            else:
                out.append([c0, desc, amt])
            continue
        if want_number_col:
            desc_t = (c1 + " " + c2).strip()
            amt_t = c3 if _is_amount_like(c3) else ""
            out.append(["", c0, desc_t, amt_t])
        else:
            desc_t = (c1 + " " + c2).strip()
            amt_t = c3 if _is_amount_like(c3) else ""
            out.append([c0, desc_t, amt_t])

    return out


# --------------------------
# Shared keyword definitions
# --------------------------

_MONEY_RE = re.compile(
    r"^(?:[\$£€¥]|Rs\.?|INR|PKR)?\s*-?\s?\d{1,3}(?:,\d{3})*(?:\.\d{1,4})?$",
    re.IGNORECASE,
)

_HEADER_KW_PATTERNS = [
    r"DATE",
    r"POSTING\s+DATE",
    r"VALUE\s+DATE",
    r"TXN\s+DATE",
    r"TRANSACTION\s+DATE",
    r"ENTRY\s+DATE",
    r"EFFECTIVE\s+DATE",
    r"TRANSACTION(?:\s+(?:ID|TYPE|NO|NUMBER))?",
    r"TXN(?:\s+(?:ID|NO|TYPE))?",
    r"REF(?:ERENCE)?(?:\s*(?:NO|NUM|NUMBER))?",
    r"CHEQUE(?:\s*(?:NO|NUMBER))?",
    r"CHQ(?:\s*(?:NO|NUMBER))?",
    r"VOUCHER(?:\s*(?:NO|NUMBER))?",
    r"SR\.?\s*NO\.?",
    r"SERIAL(?:\s*(?:NO|NUMBER))?",
    r"DESCRIPTION",
    r"DETAILS?",
    r"PARTICULARS?(?:\s+OF\s+TRANSACTION)?",
    r"NARRATION",
    r"REMARKS?",
    r"NOTES?",
    r"DEBIT",
    r"DEBITS?",
    r"DR\.?",
    r"WITHDRAWALS?",
    r"PAID\s+OUT",
    r"MONEY\s+OUT",
    r"CREDIT",
    r"CREDITS?",
    r"CR\.?",
    r"DEPOSITS?",
    r"PAID\s+IN",
    r"MONEY\s+IN",
    r"AMOUNT",
    r"BALANCE",
    r"BAL\.?",
    r"RUNNING\s+BALANCE",
    r"RUNNING\s+BAL\.?",
    r"AVAILABLE\s+BALANCE",
    r"AVAILABLE\s+BAL\.?",
    r"AVAIL\.?\s+BAL\.?",
    r"LEDGER\s+BALANCE",
    r"LEDGER\s+BAL\.?",
    r"CLOSING\s+BALANCE",
    r"CLOSING\s+BAL\.?",
    r"OPENING\s+BALANCE",
    r"OPENING\s+BAL\.?",
]

_HEADER_KW_RES = [
    re.compile(r"^(" + p + r")(.*)", re.IGNORECASE | re.DOTALL)
    for p in _HEADER_KW_PATTERNS
]

_HEADER_KW_EXACT_RE = re.compile(
    r"^(?:" + r"|".join(_HEADER_KW_PATTERNS) + r")$",
    re.IGNORECASE,
)

_HEADER_ROW_KEYWORD_THRESHOLD = 0.5

# --------------------------
# Fix 1: Fused-header recovery
# --------------------------

def _split_keyword_remainder(cell_text: str):
    s = cell_text.strip()
    for pattern in _HEADER_KW_RES:
        m = pattern.match(s)
        if m:
            keyword = m.group(1).strip()
            remainder = m.group(2).strip()
            return keyword, remainder
    return s, ""

def _extract_fused_header_artifacts(header_row):
    if not header_row:
        return list(header_row), None

    ncols = len(header_row)
    cleaned = list(header_row)
    recovered = [""] * ncols

    text_fused = []
    money_fused = []

    for idx, cell in enumerate(header_row):
        cell_s = str(cell or "").strip()
        if not cell_s:
            continue

        keyword, remainder = _split_keyword_remainder(cell_s)

        if not remainder:
            continue

        remainder_no_space = remainder.replace(" ", "")

        if _MONEY_RE.match(remainder_no_space):
            cleaned[idx] = keyword
            recovered[idx] = remainder
            money_fused.append(idx)

        elif not _HEADER_KW_EXACT_RE.match(remainder.split()[0] if remainder.split() else ""):
            cleaned[idx] = keyword
            recovered[idx] = remainder
            text_fused.append(idx)

    if text_fused and money_fused:
        return cleaned, recovered

    return list(header_row), None

def _clean_header_artifacts(grid):
    if not grid or not grid[0]:
        return grid, None

    cleaned_header, recovered_row = _extract_fused_header_artifacts(grid[0])
    grid[0] = cleaned_header
    return grid, recovered_row

# --------------------------
# Fix 2: Misplaced header row promotion
# --------------------------

def _row_keyword_score(row):
    non_empty = [str(c or "").strip() for c in row if str(c or "").strip()]
    if not non_empty:
        return 0.0

    keyword_hits = 0
    for cell in non_empty:
        if _HEADER_KW_EXACT_RE.match(cell):
            keyword_hits += 1
            continue
        parts = cell.split()
        if all(_HEADER_KW_EXACT_RE.match(p) for p in parts) and len(parts) > 1:
            keyword_hits += 1
            continue

    return keyword_hits / len(non_empty)

_BALANCE_INFO_RE = re.compile(
    r"(beginning|ending|opening|closing)\s+balance"
    r"|account\s*#?\s*\d{5,}"
    r"|\b\d{7,}\b"
    r"|\$\d{1,3}(?:,\d{3})*\.\d{2}",
    re.IGNORECASE,
)

def _header_contains_balance_info(header_row):
    for cell in header_row:
        if _BALANCE_INFO_RE.search(str(cell or "")):
            return True
    return False

def _promote_misplaced_header_row(grid):
    if not grid or len(grid) < 2:
        return grid

    current_header_score = _row_keyword_score(grid[0])
    if current_header_score >= _HEADER_ROW_KEYWORD_THRESHOLD:
        return grid

    candidate_idx = None
    candidate_score = 0.0
    for i in range(1, min(len(grid), 6)):
        score = _row_keyword_score(grid[i])
        if score >= _HEADER_ROW_KEYWORD_THRESHOLD and score > candidate_score:
            candidate_score = score
            candidate_idx = i

    if candidate_idx is None:
        return grid

    candidate_row = grid[candidate_idx]

    expanded_header = []
    for cell in candidate_row:
        cell_s = str(cell or "").strip()
        parts = cell_s.split()
        if len(parts) > 1 and all(_HEADER_KW_EXACT_RE.match(p) for p in parts):
            expanded_header.extend(parts)
        else:
            expanded_header.append(cell_s)

    new_ncols = len(expanded_header)
    col_expansion = new_ncols - len(candidate_row)

    metadata_row = None
    if _header_contains_balance_info(grid[0]):
        meta_cells = [str(c or "").strip() for c in grid[0]]
        meta_text = " ".join(c for c in meta_cells if c).strip()
        if meta_text:
            metadata_row = [meta_text] + [""] * (new_ncols - 1)

    new_data_rows = []
    for row in grid[candidate_idx + 1:]:
        if col_expansion > 0 and row:
            first_cell = str(row[0] or "").strip()
            date_match = re.match(r"^(\d{1,2}/\d{1,2})\s+(.*)", first_cell, re.DOTALL)
            if date_match and col_expansion == 1:
                new_row = [date_match.group(1), date_match.group(2).strip()] + list(row[1:])
            else:
                new_row = list(row) + [""] * col_expansion
            new_data_rows.append(new_row)
        else:
            new_data_rows.append(list(row))

    def pad(row, n):
        r = list(row)
        while len(r) < n:
            r.append("")
        return r[:n]

    new_grid = [pad(expanded_header, new_ncols)]
    if metadata_row is not None:
        new_grid.append(pad(metadata_row, new_ncols))
    for row in new_data_rows:
        new_grid.append(pad(row, new_ncols))

    return new_grid

# --------------------------
# Fix 3: Fused key-value rows in summary sections
# --------------------------

_FUSED_KV_RE = re.compile(
    r"^(.+?)(-?\d{1,3}(?:,\d{3})*(?:\.\d+)?%?)$",
    re.DOTALL,
)

def _fix_fused_keyvalue_rows(grid):
    if not grid or len(grid) < 2:
        return grid

    _DOLLAR_AMOUNT_RE = re.compile(r"\$\s*\d{1,3}(?:,\d{3})*\.\d{2}")
    _LONG_NUMBER_RE = re.compile(r"\d{7,}")

    ncols = len(grid[0])
    new_grid = [grid[0]]

    for row in grid[1:]:
        cells = [str(c or "").strip() for c in row]
        if not cells:
            new_grid.append(row)
            continue

        first = cells[0]
        rest_empty = all(c == "" for c in cells[1:])

        if not rest_empty or not first:
            new_grid.append(row)
            continue

        if _DOLLAR_AMOUNT_RE.search(first):
            new_grid.append(row)
            continue

        if _LONG_NUMBER_RE.search(first):
            new_grid.append(row)
            continue

        if re.search(r"#\s*[\dX]", first):
            new_grid.append(row)
            continue

        m = _FUSED_KV_RE.match(first)
        if not m:
            new_grid.append(row)
            continue

        label_raw = m.group(1)
        value = m.group(2).strip()

        if label_raw != label_raw.rstrip():
            new_grid.append(row)
            continue

        label = label_raw.strip()

        if not label or not value:
            new_grid.append(row)
            continue

        new_row = [""] * ncols
        new_row[0] = label
        new_row[ncols - 1] = value
        new_grid.append(new_row)

    return new_grid

# --------------------------
# Fix 4: PDF text-layer row-count patch
# --------------------------

def _extract_textlayer_rows(pdf_path: str, page_num: int):
    try:
        import pymupdf as fitz

        doc = fitz.open(pdf_path)
        page = doc[page_num]
        words = page.get_text("words")
        doc.close()

        if not words:
            return []

        lines_by_y = {}
        for w in words:
            x0, y0, word = float(w[0]), float(w[1]), w[4]
            bucket = None
            for existing_y in lines_by_y:
                if abs(existing_y - y0) <= 5:
                    bucket = existing_y
                    break
            if bucket is None:
                bucket = y0
            lines_by_y.setdefault(bucket, []).append((x0, word))

        sorted_lines = []
        for y in sorted(lines_by_y):
            line_words = sorted(lines_by_y[y], key=lambda t: t[0])
            sorted_lines.append([w for _, w in line_words])

        _MONTHS = r"(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)"
        date_re = re.compile(
            r"^\d{1,2}/\d{2}(?:/\d{2,4})?$"
            r"|^\d{1,2}-\d{2}(?:-\d{2,4})?$"
            r"|^\d{4}-\d{2}-\d{2}$"
            r"|^" + _MONTHS + r"$",
            re.IGNORECASE,
        )
        month_re = re.compile(r"^" + _MONTHS + r"$", re.IGNORECASE)
        day_re   = re.compile(r"^\d{1,2}$")

        amount_re = re.compile(
            r"^-?\$?\d{1,3}(?:,\d{3})*(?:\.\d{1,4})?$"
            r"|^\$-?\d{1,3}(?:,\d{3})*(?:\.\d{1,4})?$"
            r"|^-\s+\$\d{1,3}(?:,\d{3})*(?:\.\d{1,4})?$"
        )

        rows = []
        for line in sorted_lines:
            if len(line) < 2:
                continue

            date_str = None
            desc_start = None

            if date_re.match(line[0]):
                if month_re.match(line[0]) and len(line) > 1 and day_re.match(line[1]):
                    date_str = line[0] + " " + line[1]
                    desc_start = 2
                else:
                    date_str = line[0]
                    desc_start = 1
            else:
                continue

            if desc_start is None or desc_start >= len(line):
                continue

            remaining = line[desc_start:]
            post_date_str = ""
            if remaining and month_re.match(remaining[0]):
                if len(remaining) > 1 and day_re.match(remaining[1]):
                    post_date_str = remaining[0] + " " + remaining[1]
                    remaining = remaining[2:]
                elif len(remaining) > 0:
                    remaining = remaining[1:]
            elif remaining and date_re.match(remaining[0]) and not month_re.match(remaining[0]):
                post_date_str = remaining[0]
                remaining = remaining[1:]

            if len(remaining) < 2:
                continue

            if (len(remaining) >= 2
                    and remaining[-2] == "-"
                    and remaining[-1].startswith("$")):
                amount_candidate = remaining[-2] + " " + remaining[-1]
                desc_tokens = remaining[:-2]
            else:
                amount_candidate = remaining[-1]
                desc_tokens = remaining[:-1]

            if not amount_re.match(amount_candidate):
                continue

            desc = " ".join(desc_tokens).strip()
            if not desc:
                continue

            if not re.search(r"[A-Za-z]", desc):
                continue

            rows.append({"date": date_str, "post_date": post_date_str, "desc": desc, "amount": amount_candidate})

        return rows if len(rows) >= 2 else []

    except Exception as e:
        log.debug("_extract_textlayer_rows failed (page %d): %s", page_num, e)
        return []

def _jb_extract_checks_fallback(text_layer: str) -> List[Tuple[str, str, str]]:
    if not text_layer:
        return []
    m = re.search(
        r"Checks\s*\([^\n]*\n(.*?)(?=^Daily\s+Account\s+Balance\s*$)",
        text_layer,
        re.DOTALL | re.MULTILINE | re.IGNORECASE,
    )
    if not m:
        return []
    chunk = m.group(1)
    chunk = re.sub(
        r"^\s*Date\s+Number\s+Amount\s*$",
        "",
        chunk,
        flags=re.MULTILINE | re.IGNORECASE,
    )
    _triplet = re.compile(
        r"(\d{2}-\d{2})\s+(\*?\d+)\s+(\d{1,3}(?:,\d{3})*\.\d{2})"
    )
    out: List[Tuple[str, str, str]] = []
    seen = set()
    for t in _triplet.findall(chunk):
        if t not in seen:
            seen.add(t)
            out.append(t)
    return out

def _extract_jb_summary_sections(text_layer: str, needs_checks: bool = True, needs_dab: bool = True) -> str:
    if not text_layer:
        return ""

    _DATE_RE  = re.compile(r"^\d{2}-\d{2}$")
    _MONEY_RE = re.compile(r"^\d{1,3}(?:,\d{3})*\.\d{2}$")
    _CHECK_RE = re.compile(r"^\*?\d+$")

    lines = [l.strip() for l in text_layer.splitlines()]
    output = []

    if needs_checks:
        checks = []
        i = 0
        while i < len(lines):
            if lines[i].lower().startswith("checks"):
                i += 1
                while i < len(lines):
                    if "daily account balance" in lines[i].lower():
                        break
                    if lines[i] == "Date":
                        i += 1
                        dates = []
                        while i < len(lines) and _DATE_RE.match(lines[i]):
                            dates.append(lines[i]); i += 1
                        while i < len(lines) and lines[i] != "Number":
                            i += 1
                        i += 1
                        numbers = []
                        while i < len(lines):
                            if _CHECK_RE.match(lines[i]):
                                numbers.append(lines[i]); i += 1
                            elif lines[i] == "":
                                i += 1
                            else:
                                break
                        while i < len(lines) and lines[i] != "Amount":
                            i += 1
                        i += 1
                        amounts = []
                        while i < len(lines):
                            if _MONEY_RE.match(lines[i]):
                                amounts.append(lines[i]); i += 1
                            elif lines[i] == "":
                                i += 1
                            else:
                                break
                        for d, n, a in zip(dates, numbers, amounts):
                            checks.append((d, n, a))
                    else:
                        i += 1
                break
            i += 1

        if not checks and re.search(r"\bChecks\s*\(", text_layer, re.I):
            checks = _jb_extract_checks_fallback(text_layer)

        if checks:
            rows = "\n".join(
                "<tr><td>" + d + "</td><td>" + n + "</td><td>" + a + "</td></tr>"
                for d, n, a in checks
            )
            output.append("Checks")
            output.append(
                '<table class="jb-summary-checks">\n<tr><th>Date</th><th>Number</th><th>Amount</th></tr>\n'
                + rows + "\n</table>"
            )

    if needs_dab:
        balances = []
        i = 0
        while i < len(lines):
            if "daily account balance" in lines[i].lower():
                i += 1
                while i < len(lines):
                    if lines[i] == "Date":
                        i += 1
                        dates = []
                        while i < len(lines) and _DATE_RE.match(lines[i]):
                            dates.append(lines[i]); i += 1
                        while i < len(lines) and lines[i] != "Balance":
                            i += 1
                        i += 1
                        bals = []
                        while i < len(lines):
                            if _MONEY_RE.match(lines[i]):
                                bals.append(lines[i]); i += 1
                            elif lines[i] == "":
                                i += 1
                            else:
                                break
                        for d, b in zip(dates, bals):
                            balances.append((d, b))
                    else:
                        i += 1
                break
            i += 1

        if balances:
            rows = "\n".join(
                "<tr><td>" + d + "</td><td>" + b + "</td></tr>"
                for d, b in balances
            )
            output.append("Daily Account Balance")
            output.append(
                "<table>\n<tr><th>Date</th><th>Balance</th></tr>\n"
                + rows + "\n</table>"
            )

    return "\n\n".join(output)

def _patch_ocr_with_textlayer(page_md: str, pdf_path: str, page_num: int) -> str:
    if not page_md or "<table" not in page_md.lower():
        return page_md

    try:
        tl_rows = _extract_textlayer_rows(pdf_path, page_num)
        if not tl_rows:
            return page_md

        def _norm(s):
            return re.sub(r"\s+", " ", str(s or "").strip().lower())

        tl_freq = {}
        for r in tl_rows:
            key = (_norm(r["date"]), _norm(r["desc"]), _norm(r["amount"]))
            tl_freq[key] = tl_freq.get(key, 0) + 1

        # Share with _dedupe_duplicate_rows_in_da3_table (Fix-20 override).
        # Keys MUST match _transaction_row_dedupe_key (date/desc/amount, no spaces in
        # date/desc; amount stripped of $/,/space). The injection map tl_freq uses
        # _norm() with spaces — different tuple shape — so we keep a parallel count.
        global _tl_freq_context
        tl_dedupe = {}
        for r in tl_rows:
            dk = _transaction_row_dedupe_key([r["date"], r["desc"], r["amount"]])
            tl_dedupe[dk] = tl_dedupe.get(dk, 0) + 1
        _tl_freq_context = tl_dedupe

        table_pattern = re.compile(r"<table[^>]*>.*?</table>", re.DOTALL | re.IGNORECASE)
        result = page_md

        for tbl_match in table_pattern.finditer(page_md):
            table_html = tbl_match.group(0)

            p = TableGridParser()
            p.feed(table_html)
            grid = _build_grid(p.rows)
            if len(grid) < 2:
                continue

            header = [str(c or "").strip().upper() for c in grid[0]]
            date_col = desc_col = amt_col = None
            for ci, h in enumerate(header):
                if re.search(r"\bDATE\b", h) and date_col is None:
                    date_col = ci
                elif re.search(r"\b(DESCRIPTION|DETAILS?|NARRATION|PARTICULARS?)\b", h) and desc_col is None:
                    desc_col = ci
                elif re.search(r"\b(AMOUNT|DEBIT|CREDIT|DR|CR)\b", h) and amt_col is None:
                    amt_col = ci

            if date_col is None or desc_col is None or amt_col is None:
                continue

            ocr_freq = {}
            ocr_rows_indexed = []
            for ri, row in enumerate(grid[1:], start=1):
                cells = [str(c or "").strip() for c in row]
                if len(cells) <= max(date_col, desc_col, amt_col):
                    continue
                d = _norm(cells[date_col])
                desc = _norm(cells[desc_col])
                amt = _norm(cells[amt_col])
                if not d or not desc:
                    continue
                key = (d, desc, amt)
                ocr_freq[key] = ocr_freq.get(key, 0) + 1
                ocr_rows_indexed.append((ri, key))

            overlap = set(ocr_freq.keys()) & set(tl_freq.keys())

            _ocr_dates = [
                _norm(str(row[date_col] or ""))
                for row in grid[1:]
                if len(row) > date_col and str(row[date_col] or "").strip()
            ]
            _tl_dates = [_norm(r["date"]) for r in tl_rows]
            _slash_re  = re.compile(r"^\d{1,2}/\d{2}")
            _hyphen_re = re.compile(r"^\d{1,2}-\d{2}")
            def _date_fmt(dates):
                if any(_slash_re.match(d) for d in dates):  return "slash"
                if any(_hyphen_re.match(d) for d in dates): return "hyphen"
                return "other"
            ocr_fmt = _date_fmt(_ocr_dates)
            tl_fmt  = _date_fmt(_tl_dates)
            date_fmt_match = (ocr_fmt == tl_fmt) or "other" in (ocr_fmt, tl_fmt)

            to_inject = {}
            to_inject_new = {}

            for key in tl_freq:
                ocr_count = ocr_freq.get(key, 0)
                tl_count = tl_freq[key]
                if tl_count > ocr_count:
                    missing = tl_count - ocr_count
                    if ocr_count > 0:
                        if overlap:
                            to_inject[key] = missing
                    else:
                        if date_fmt_match:
                            tl_row_data = next(
                                (r for r in tl_rows
                                 if (_norm(r["date"]), _norm(r["desc"]), _norm(r["amount"])) == key),
                                None
                            )
                            if tl_row_data:
                                to_inject_new[key] = (missing, tl_row_data)

            if not to_inject and not to_inject_new:
                continue

            new_grid = [grid[0]]
            for ri, row in enumerate(grid[1:], start=1):
                new_grid.append(row)
                cells = [str(c or "").strip() for c in row]
                if len(cells) <= max(date_col, desc_col, amt_col):
                    continue
                d = _norm(cells[date_col])
                desc = _norm(cells[desc_col])
                amt = _norm(cells[amt_col])
                key = (d, desc, amt)
                if key in to_inject and to_inject[key] > 0:
                    last_occ = max(idx for idx, k in ocr_rows_indexed if k == key)
                    if ri == last_occ:
                        for _ in range(to_inject[key]):
                            new_grid.append(list(row))
                        to_inject[key] = 0

            new_table_html = _grid_to_html(new_grid)
            result = result.replace(table_html, new_table_html, 1)

            if to_inject_new:
                ncols = len(grid[0])
                header_row = grid[0]

                post_date_col = None
                for ci, h in enumerate(header_row):
                    hh = str(h or "").strip().upper()
                    if "POST" in hh and "DATE" in hh:
                        post_date_col = ci
                        break

                new_rows = []
                for key, (count, tl_row_data) in to_inject_new.items():
                    for _ in range(count):
                        new_row = [""] * ncols
                        new_row[date_col] = tl_row_data["date"]
                        new_row[desc_col] = tl_row_data["desc"]
                        new_row[amt_col]  = tl_row_data["amount"]
                        if post_date_col is not None:
                            new_row[post_date_col] = tl_row_data.get("post_date", "")
                        new_rows.append(new_row)
                        log.info(
                            "page %d: new table row from text layer: %s %s",
                            page_num, tl_row_data["date"], tl_row_data["desc"][:40]
                        )

                if new_rows:
                    extra_grid = [header_row] + new_rows
                    extra_html = "\n\n" + _grid_to_html(extra_grid)
                    insert_pos = result.find(new_table_html) + len(new_table_html)
                    result = result[:insert_pos] + extra_html + result[insert_pos:]

        if tl_rows:
            _has_txn_table = False
            for _tbl in table_pattern.finditer(result):
                _p2 = TableGridParser()
                _p2.feed(_tbl.group(0))
                _g2 = _build_grid(_p2.rows)
                if len(_g2) < 2:
                    continue
                _h2 = [str(c or "").strip().upper() for c in _g2[0]]
                if (any(re.search(r"\bDATE\b", x) for x in _h2) and
                        any(re.search(r"\b(DESCRIPTION|DETAILS?|NARRATION|PARTICULARS?)\b", x) for x in _h2)):
                    _has_txn_table = True
                    break

            if not _has_txn_table:
                _has_post = any(r.get("post_date") for r in tl_rows)
                if _has_post:
                    _chdr = ["Date", "Post Date", "Transaction Description", "Amount"]
                    _di2, _pi2, _xi2, _ai2 = 0, 1, 2, 3
                else:
                    _chdr = ["Date", "Transaction Description", "Amount"]
                    _di2, _xi2, _ai2 = 0, 1, 2
                _cnew_rows = []
                for r in tl_rows:
                    _crow = [""] * len(_chdr)
                    _crow[_di2] = r["date"]
                    _crow[_xi2] = r["desc"]
                    _crow[_ai2] = r["amount"]
                    if _has_post:
                        _crow[_pi2] = r.get("post_date", "")
                    _cnew_rows.append(_crow)
                result = _grid_to_html([_chdr] + _cnew_rows) + "\n\n" + result
                log.info("page %d: Case C — new table (%d rows) from text layer",
                         page_num, len(_cnew_rows))

        return result

    except Exception as e:
        log.warning("_patch_ocr_with_textlayer failed (page %d): %s", page_num, e)
        _tl_freq_context.clear()
        return page_md

# --------------------------
# Fix 5: Mid-table separator row reconstruction
# --------------------------

_DATE_RE = re.compile(r"^\d{1,2}/\d{2}(?:/\d{2,4})?$")
_SPLIT_AMOUNT_RE = re.compile(
    r"^-?\$?\d{1,3}(?:,\d{3})*(?:\.\d{1,4})?$|^\$-?\d{1,3}(?:,\d{3})*(?:\.\d{1,4})?$"
)

def _reconstruct_separator_rows(grid):
    if not grid or len(grid) < 2:
        return grid

    ncols = len(grid[0])
    if ncols < 2:
        return grid

    new_grid = [grid[0]]

    for row in grid[1:]:
        cells = [str(c or "").strip() for c in row]
        while len(cells) < ncols:
            cells.append("")

        non_empty = [(i, c) for i, c in enumerate(cells) if c]

        if len(non_empty) == 0:
            new_grid.append(row)
            continue

        first_idx, first_val = non_empty[0]

        if _DATE_RE.match(first_val):
            new_grid.append(row)
            continue

        if not re.search(r"[A-Za-z]", first_val):
            new_grid.append(row)
            continue

        if len(non_empty) == 1:
            new_row = [""] * ncols
            new_row[0] = first_val
            new_grid.append(new_row)
            continue

        trailing = [val for _, val in non_empty[1:]]
        all_trailing_are_digit_fragments = all(
            re.fullmatch(r"\d{1,4}", v) for v in trailing
        )

        if not all_trailing_are_digit_fragments:
            new_grid.append(row)
            continue

        reconstructed = "".join(val for _, val in non_empty).strip()
        new_row = [""] * ncols
        new_row[0] = reconstructed
        new_grid.append(new_row)

    return new_grid

# --------------------------
# Normalizer entry point
# --------------------------

def _merge_split_rows(grid):
    if not grid or len(grid) < 3:
        return grid

    ncols = len(grid[0])
    if ncols < 3:
        return grid

    header = [str(c or "").strip().upper() for c in grid[0]]
    date_col = desc_col = bal_col = None
    debit_cols = []
    credit_cols = []

    for ci, h in enumerate(header):
        if re.search(r"\bDATE\b", h) and date_col is None:
            date_col = ci
        if re.search(r"\b(DESCRIPTION|DETAILS?|NARRATION|PARTICULARS?|TRANSACTION)\b", h) and desc_col is None:
            desc_col = ci
        if re.search(r"\bBALANCE\b", h) and bal_col is None:
            bal_col = ci
        if re.search(r"\b(DEBIT|DEBITS|DR|WITHDRAWAL|SUBTRACTIONS?)\b", h):
            debit_cols.append(ci)
        if re.search(r"\b(CREDIT|CREDITS|CR|DEPOSIT|ADDITIONS?)\b", h):
            credit_cols.append(ci)

    if date_col is None or desc_col is None:
        return grid
    if not debit_cols and not credit_cols:
        return grid

    money_cols = debit_cols + credit_cols
    _MONEY_RE2 = re.compile(r"^-?\$?\d{1,3}(?:,\d{3})*\.\d{1,4}$")
    _DATE_RE2  = re.compile(r"^\d{1,2}/\d{2}(?:/\d{2,4})?$|^\d{1,2}-\d{2}(?:-\d{2,4})?$")
    _FRAGMENT_RE = re.compile(r"^[^\s\.]{1,6}$")

    new_grid = [grid[0]]
    i = 1
    while i < len(grid):
        row = [str(c or "").strip() for c in grid[i]]
        while len(row) < ncols:
            row.append("")

        has_desc    = bool(row[desc_col])
        no_date     = not row[date_col]
        no_money    = all(not row[c] for c in money_cols)
        bal_is_frag = (bal_col is not None and
                       (_FRAGMENT_RE.match(row[bal_col]) or not row[bal_col]))

        if has_desc and no_date and no_money and bal_is_frag and (i + 1) < len(grid):
            next_row = [str(c or "").strip() for c in grid[i + 1]]
            while len(next_row) < ncols:
                next_row.append("")

            next_has_date  = bool(next_row[date_col]) and _DATE_RE2.match(next_row[date_col])
            next_has_money = any(bool(next_row[c]) for c in money_cols)

            if next_has_date and next_has_money:
                merged = list(next_row)
                if next_row[desc_col]:
                    merged[desc_col] = row[desc_col] + " " + next_row[desc_col]
                else:
                    merged[desc_col] = row[desc_col]
                new_grid.append(merged)
                i += 2
                continue

        new_grid.append(row)
        i += 1

    return new_grid

def _extract_fused_desc_amount(grid):
    if not grid or len(grid) < 2:
        return grid

    ncols = len(grid[0])
    header = [str(c or "").strip().upper() for c in grid[0]]

    date_col = desc_col = None
    debit_cols = []
    credit_cols = []
    for ci, h in enumerate(header):
        if re.search(r"\bDATE\b", h) and date_col is None:
            date_col = ci
        if re.search(r"\b(DESCRIPTION|DETAILS?|NARRATION|PARTICULARS?|TRANSACTION)\b", h) and desc_col is None:
            desc_col = ci
        if re.search(r"\b(DEBIT|DEBITS|DR|WITHDRAWAL|SUBTRACTIONS?)\b", h):
            debit_cols.append(ci)
        if re.search(r"\b(CREDIT|CREDITS|CR|DEPOSIT|ADDITIONS?)\b", h):
            credit_cols.append(ci)

    if date_col is None or desc_col is None or not debit_cols:
        return grid

    money_cols = debit_cols + credit_cols
    _TRAILING_AMT = re.compile(r"^(.+?)\s+(-?\d{1,3}(?:,\d{3})*\.\d{2})$")
    debit_col = debit_cols[0]

    new_grid = [grid[0]]
    for row in grid[1:]:
        cells = [str(c or "").strip() for c in row]
        while len(cells) < ncols:
            cells.append("")

        desc = cells[desc_col]
        no_money = all(not cells[c] for c in money_cols)
        has_date = bool(cells[date_col])

        if desc and no_money and has_date:
            m = _TRAILING_AMT.match(desc)
            if m:
                cells = list(cells)
                cells[desc_col] = m.group(1).strip()
                cells[debit_col] = m.group(2)
                new_grid.append(cells)
                continue

        new_grid.append(cells)

    return new_grid

def _is_junk_table(grid):
    if not grid or len(grid) < 2:
        return False
    ncols = len(grid[0])
    if ncols != 1:
        return False
    header_text = str(grid[0][0] or "").strip().upper()
    if "CUSTOMER INFORMATION" in header_text:
        data_rows = grid[1:]
        if all(len(str(r[0] or "").strip()) <= 10 for r in data_rows):
            return True
    return False

def _split_fused_multicolumn_header(grid):
    if not grid or len(grid) < 2:
        return grid

    ncols = len(grid[0])
    if ncols < 2:
        return grid

    first_header = str(grid[0][0] or "").strip()
    parts = first_header.split()
    kw_hits = [p for p in parts if _HEADER_KW_EXACT_RE.match(p)]
    if len(kw_hits) < 2:
        return grid

    rest_headers = [str(c or "").strip() for c in grid[0][1:]]
    if not all(_HEADER_KW_EXACT_RE.match(h) for h in rest_headers if h):
        return grid

    new_header_cols = []
    non_kw_parts = []
    for p in parts:
        if _HEADER_KW_EXACT_RE.match(p):
            if non_kw_parts:
                new_header_cols.append(" ".join(non_kw_parts))
                non_kw_parts = []
            new_header_cols.append(p)
        else:
            non_kw_parts.append(p)
    if non_kw_parts:
        new_header_cols.append(" ".join(non_kw_parts))

    new_header = new_header_cols + rest_headers
    new_ncols = len(new_header)
    extra_cols = new_ncols - ncols

    _DATE_COL_RE   = re.compile(r"\bDATE\b", re.IGNORECASE)
    _CARD_COL_RE   = re.compile(r"\bCARD\b", re.IGNORECASE)
    _DESC_COL_RE   = re.compile(r"\b(DESCRIPTION|DETAILS?|NARRATION|PARTICULARS?)\b", re.IGNORECASE)
    _MONEY_COL_RE  = re.compile(r"\b(DEPOSIT|WITHDRAWAL|DEBIT|CREDIT|AMOUNT|ADDITION|SUBTRACTION)S?\b", re.IGNORECASE)

    new_date_col = next((i for i,h in enumerate(new_header) if _DATE_COL_RE.search(h)), None)
    new_desc_col = next((i for i,h in enumerate(new_header) if _DESC_COL_RE.search(h)), None)
    new_card_col = next((i for i,h in enumerate(new_header) if _CARD_COL_RE.search(h)), None)

    if new_date_col is None or new_desc_col is None:
        return grid

    _DATE_PREFIX = re.compile(r"^(\d{1,2}/\d{2}(?:/\d{2,4})?)\s+(.*)", re.DOTALL)
    _CARD_SUFFIX = re.compile(r"^(.*?)\s+(\d{4})$", re.DOTALL)

    def pad(row, n):
        r = list(row)
        while len(r) < n: r.append("")
        return r[:n]

    new_grid = [pad(new_header, new_ncols)]
    for row in grid[1:]:
        cells = [str(c or "").strip() for c in row]
        while len(cells) < ncols: cells.append("")

        fused_cell = cells[0]
        rest_cells = cells[1:]

        date_val = ""
        desc_val = fused_cell
        card_val = ""

        dm = _DATE_PREFIX.match(fused_cell)
        if dm:
            date_val = dm.group(1)
            desc_val = dm.group(2).strip()

        if new_card_col is not None:
            cm = _CARD_SUFFIX.match(desc_val)
            if cm:
                desc_val = cm.group(1).strip()
                card_val = cm.group(2)

        new_row = [""] * new_ncols
        new_row[new_date_col] = date_val
        new_row[new_desc_col] = desc_val
        if new_card_col is not None:
            new_row[new_card_col] = card_val
        money_col_indices = [i for i,h in enumerate(new_header) if _MONEY_COL_RE.search(h)]
        for mi, ri in enumerate(range(len(rest_cells))):
            if mi < len(money_col_indices):
                new_row[money_col_indices[mi]] = rest_cells[ri]

        new_grid.append(new_row)

    return new_grid

def _normalize_daily_balance_table(grid):
    if not grid or len(grid) < 2:
        return grid

    header = [str(c or "").strip().upper() for c in grid[0]]
    ncols = len(header)

    if ncols < 4 or ncols % 2 != 0:
        return grid

    half = ncols // 2
    date_half  = header[:half]
    bal_half   = header[half:]

    if not all(h == "DATE" for h in date_half):
        return grid
    if not all(h == "BALANCE" for h in bal_half):
        return grid

    new_header = []
    for i in range(half):
        new_header.append("DATE")
        new_header.append("BALANCE")

    new_rows = []
    _DATE_RE3 = re.compile(r"^\d{1,2}/\d{2}(?:/\d{2,4})?$")

    for row in grid[1:]:
        cells = [str(c or "").strip() for c in row]
        while len(cells) < ncols:
            cells.append("")

        date_cells = [cells[0]]
        bal_cells  = cells[1:]

        first = date_cells[0]
        tokens = re.split(r"[\n\r\s]+", first)
        date_tokens = [t.strip() for t in tokens if t.strip() and _DATE_RE3.match(t.strip())]
        if len(date_tokens) >= 2:
            stacked = date_tokens
        else:
            stacked = date_cells

        new_row = [""] * ncols
        for i, date_val in enumerate(stacked):
            if not _DATE_RE3.match(date_val):
                continue
            bal_val = bal_cells[i] if i < len(bal_cells) else ""
            new_row[i * 2]     = date_val
            new_row[i * 2 + 1] = bal_val
        new_rows.append(new_row)

    if not new_rows:
        return grid

    return [new_header] + new_rows

def _normalize_checks_paid_table(grid):
    if not grid or len(grid) < 2:
        return grid

    ncols = len(grid[0])
    if ncols < 2:
        return grid

    first_h = str(grid[0][0] or "").strip()
    _CHECKS_HDR = re.compile(r"^(DATE\s+CHECK\s+#\s*)+$", re.IGNORECASE)
    if not _CHECKS_HDR.match(first_h):
        return grid

    rest_h = [str(c or "").strip().upper() for c in grid[0][1:]]
    if not all(h in ("AMOUNT", "") for h in rest_h if h):
        return grid

    n_groups = len(re.findall(r"\bDATE\b", first_h, re.IGNORECASE))
    if n_groups < 1:
        return grid

    new_header = []
    for _ in range(n_groups):
        new_header.extend(["DATE", "CHECK #", "AMOUNT"])

    _DATE_RE = re.compile(r"^\d{1,2}/\d{2}$")

    new_rows = [new_header]
    for row in grid[1:]:
        cells = [str(c or "").strip() for c in row]
        while len(cells) < ncols:
            cells.append("")

        fused_data = cells[0]
        amounts    = cells[1:]

        tokens = fused_data.split()
        groups = []
        current = []
        for tok in tokens:
            if _DATE_RE.match(tok) and current:
                groups.append(current)
                current = [tok]
            else:
                current.append(tok)
        if current and current[0]:
            groups.append(current)

        new_row = [""] * len(new_header)
        for i, grp in enumerate(groups):
            if i >= n_groups:
                break
            new_row[i * 3]     = grp[0]
            new_row[i * 3 + 1] = " ".join(grp[1:]).strip()
            new_row[i * 3 + 2] = amounts[i] if i < len(amounts) else ""

        new_rows.append(new_row)

    return new_rows

_JB_SECTION_HDR = re.compile(
    r"^(Deposits?|Withdrawals?|Checks?(?:\s+Paid)?|Daily\s+Account\s+Balance)"
    r"\s*(?:\(cont\.?\))?\s*$",
    re.IGNORECASE,
)
_JB_COL_HDR = re.compile(
    r"^Date\s+(Description|Number)\s+Amount\s*$"
    r"|^Date\s+(?:Number\s+)?(?:Balance|Amount)\s*$",
    re.IGNORECASE,
)
_JB_DATE_DD  = re.compile(r"^\d{2}-\d{2}\s+")
_JB_TXN_ROW  = re.compile(r"^(\d{2}-\d{2})\s+(.+?)\s+(-?\d{1,3}(?:,\d{3})*\.\d{2})\s*$")
_JB_BAL_ROW  = re.compile(r"^(\d{2}-\d{2})\s+(\d{1,3}(?:,\d{3})*\.\d{2})\s*$")

def _jb_rows_to_html(header_cols, rows):
    th    = "".join("<th>" + h + "</th>" for h in header_cols)
    parts = ["<table>", "<tr>" + th + "</tr>"]
    for row in rows:
        td = "".join("<td>" + str(row.get(h, "")) + "</td>" for h in header_cols)
        parts.append("<tr>" + td + "</tr>")
    parts.append("</table>")
    return "\n".join(parts)

def _jb_section_has_table(lines, start, end):
    return any("<table" in l.lower() for l in lines[start:end])

def _find_first_data_row(lines, start):
    for i in range(start, min(start + 20, len(lines))):
        if lines[i].strip():
            return i
    return None

def convert_plaintext_bank_sections(text: str) -> str:
    lines = text.splitlines()
    out   = []
    i     = 0

    while i < len(lines):
        line = lines[i].strip()

        if not _JB_SECTION_HDR.match(line):
            out.append(lines[i])
            i += 1
            continue

        section_title = line
        j = i + 1

        while j < len(lines) and not lines[j].strip():
            j += 1

        if j >= len(lines) or not _JB_COL_HDR.match(lines[j].strip()):
            out.append(lines[i])
            i += 1
            continue

        col_hdr_line = lines[j].strip()
        data_start   = j + 1

        first_data_idx = _find_first_data_row(lines, data_start)

        if first_data_idx is None or not _JB_DATE_DD.match(lines[first_data_idx].strip()):
            out.append(lines[i])
            i += 1
            continue

        section_end = len(lines)
        for k in range(data_start, len(lines)):
            l = lines[k].strip()
            if (l != section_title and _JB_SECTION_HDR.match(l)
                    or l.startswith("---page-separator")
                    or l.startswith("Page:")):
                section_end = k
                break

        if _jb_section_has_table(lines, i, section_end):
            out.append(lines[i])
            i += 1
            continue

        parts_upper = col_hdr_line.upper().split()
        if len(parts_upper) == 2 and parts_upper[1] == "BALANCE":
            header_cols = ["Date", "Balance"]
        elif "NUMBER" in parts_upper:
            header_cols = ["Date", "Number", "Amount"]
        else:
            header_cols = ["Date", "Description", "Amount"]

        out.append(section_title)
        rows    = []
        current = None
        k       = data_start

        while k < section_end:
            raw      = lines[k]
            row_line = raw.strip()

            if not row_line:
                k += 1
                continue

            if (_JB_SECTION_HDR.match(row_line)
                    or row_line.startswith("---page-separator")
                    or row_line.startswith("Page:")):
                break

            mb = _JB_BAL_ROW.match(row_line)
            if mb and header_cols == ["Date", "Balance"]:
                if current:
                    rows.append(current)
                current = {"Date": mb.group(1), "Balance": mb.group(2)}
                k += 1
                continue

            mt = _JB_TXN_ROW.match(row_line)
            if mt:
                if current:
                    rows.append(current)
                desc_key = "Description" if "Description" in header_cols else "Number"
                current  = {
                    "Date":    mt.group(1),
                    desc_key:  mt.group(2).strip(),
                    "Amount":  mt.group(3),
                }
                k += 1
                continue

            if current and row_line:
                desc_key = "Description" if "Description" in header_cols else "Number"
                if desc_key in current:
                    current[desc_key] += " " + row_line
                k += 1
                continue

            k += 1

        if current:
            rows.append(current)

        if rows:
            out.append(_jb_rows_to_html(header_cols, rows))

        i = section_end
        continue

    return "\n".join(out)

# --------------------------
# Navy Federal full-page text-layer parser (Fix NF-1)
# --------------------------

def _is_nfcu_document(pdf_path: str) -> bool:
    try:
        import pymupdf as fitz
        doc = fitz.open(pdf_path)
        pages_to_check = min(2, len(doc))
        text = ""
        for i in range(pages_to_check):
            text += "\n" + (doc[i].get_text() or "")
        doc.close()
        t = text.lower()
        return (
            ("statement of account" in t)
            and ("access no." in t)
            and ("routing number" in t)
            and ("navy federal" in t or "navyfederal.org" in t)
        )
    except Exception:
        return False


def _parse_nfcu_page(pdf_path: str, page_num: int) -> str:
    try:
        import pymupdf as fitz

        doc = fitz.open(pdf_path)
        if page_num >= len(doc):
            doc.close()
            return ""
        page = doc[page_num]
        page_text = page.get_text() or ""
        words = page.get_text("words") or []
        doc.close()

        if not page_text or not words:
            return ""

        t = page_text.lower()
        is_nfcu = (
            ("statement of account" in t)
            and (("access no." in t) or ("routing number" in t) or ("navy federal" in t) or ("navyfederal.org" in t))
        )
        if not is_nfcu:
            return ""

        buckets = []
        for w in words:
            if len(w) < 5:
                continue
            x0 = float(w[0])
            y0 = float(w[1])
            token = w[4].strip()
            if not token:
                continue
            bi = None
            for i, (yb, _) in enumerate(buckets):
                if abs(yb - y0) <= 2.0:
                    bi = i
                    break
            if bi is None:
                buckets.append((y0, [(x0, token)]))
            else:
                buckets[bi][1].append((x0, token))

        lines = []
        for y, arr in sorted(buckets, key=lambda x: x[0]):
            toks = [tok for _, tok in sorted(arr, key=lambda x: x[0])]
            xs = [x for x, _ in sorted(arr, key=lambda x: x[0])]
            lines.append((y, xs, toks, " ".join(toks)))

        date_re = re.compile(r"^\d{2}-\d{2}$")
        money_re = re.compile(r"^[\d,]+\.\d{2}-?$")
        acct_re = re.compile(r"^(Business Checking|Mbr Business Savings)\s*-\s*(\d+)")
        stop_markers = (
            "Items Paid",
            "Average Daily Balance",
            "2024 Year to Date",
            "Disclosure",
            "What to Do",
            "Errors",
            "Payments",
            "SAVINGS DIVIDENDS",
            "REMITTANCE RECEIVED",
            "DEPOSIT VOUCHER",
            "ACCOUNT NUMBER",
            "MARK \"X\"",
            "ADDRESS/ORDER",
            "ITEMS ON REVERSE",
            "CHANGE OF ADDRESS",
            "PLEASE PRINT",
            "RANK/RATE",
            "PO BOX",
            "MERRIFIELD",
            "Questions about this Statement",
        )

        def parse_account_rows(start_idx):
            rows = []
            i = start_idx
            pending = None
            while i < len(lines):
                _, _, toks, text = lines[i]
                if not text:
                    i += 1
                    continue

                if acct_re.match(text) and i != start_idx:
                    break
                if any(text.startswith(s) for s in stop_markers):
                    break
                upper_text = text.upper()
                if (
                    "REMITTANCE RECEIVED" in upper_text
                    or "DEPOSIT VOUCHER" in upper_text
                    or "ACCOUNT NUMBER" in upper_text
                    or "ADDRESS/ORDER" in upper_text
                    or "ITEMS ON REVERSE" in upper_text
                    or "CHANGE OF ADDRESS" in upper_text
                    or "PLEASE PRINT" in upper_text
                ):
                    break
                if text in ("Checking", "Savings"):
                    break
                if text.startswith("Date Transaction Detail"):
                    i += 1
                    continue

                def flush_pending():
                    nonlocal pending
                    if pending is not None:
                        rows.append(pending)
                        pending = None

                if toks and date_re.match(toks[0]):
                    flush_pending()
                    date = toks[0]
                    rest = toks[1:]
                    mvals = [tok for tok in rest if money_re.match(tok)]
                    desc_tokens = [tok for tok in rest if not money_re.match(tok)]

                    amount = ""
                    balance = ""
                    if len(mvals) >= 2:
                        amount = mvals[-2]
                        balance = mvals[-1]
                    elif len(mvals) == 1:
                        balance = mvals[-1]

                    pending = {
                        "date": date,
                        "desc": " ".join(desc_tokens).strip(),
                        "amount": amount,
                        "balance": balance,
                    }
                else:
                    if pending is not None:
                        mvals = [tok for tok in toks if money_re.match(tok)]
                        desc_tokens = [tok for tok in toks if not money_re.match(tok)]

                        upper = text.upper()
                        if pending["balance"] and (
                            "REMITTANCE" in upper
                            or "DEPOSIT VOUCHER" in upper
                            or "ACCOUNT NUMBER" in upper
                            or "ADDRESS/ORDER" in upper
                            or "ITEMS ON REVERSE" in upper
                            or "PO BOX" in upper
                        ):
                            break

                        if (
                            ("BEGINNING BALANCE" in pending["desc"].upper() or "ENDING BALANCE" in pending["desc"].upper())
                            and not mvals
                            and not (toks and date_re.match(toks[0]))
                        ):
                            break
                        if desc_tokens:
                            pending["desc"] = (pending["desc"] + " " + " ".join(desc_tokens)).strip()
                        if len(mvals) >= 2 and (not pending["amount"] or not pending["balance"]):
                            if not pending["amount"]:
                                pending["amount"] = mvals[-2]
                            if not pending["balance"]:
                                pending["balance"] = mvals[-1]
                        elif len(mvals) == 1 and not pending["balance"]:
                            pending["balance"] = mvals[-1]
                i += 1

            if pending is not None:
                rows.append(pending)
            cleaned = [r for r in rows if r["date"] and r["desc"] and (r["amount"] or r["balance"])]
            return cleaned, i

        out = []
        i = 0
        while i < len(lines):
            _, _, _, text = lines[i]
            if not text:
                i += 1
                continue

            m = acct_re.match(text)
            if m:
                acct_name = m.group(1)
                acct_no = m.group(2)
                title = f"{acct_name} - {acct_no}"
                if i + 1 < len(lines) and "Continued" in lines[i + 1][3]:
                    title += " (Continued from previous page)"
                out.append(f"<b>{title}</b>")
                out.append("<table>")
                out.append("<tr><th>Date</th><th>Transaction Detail</th><th>Amount($)</th><th>Balance($)</th></tr>")
                rows, i2 = parse_account_rows(i + 1)
                for r in rows:
                    out.append(
                        f"<tr><td>{r['date']}</td><td>{r['desc']}</td><td>{r['amount']}</td><td>{r['balance']}</td></tr>"
                    )
                out.append("</table>")
                i = i2
                continue

            if text in ("Checking", "Savings"):
                out.append(f"<h3>{text}</h3>")
            elif text.startswith("Items Paid"):
                out.append("<b>Items Paid</b>")
            i += 1

        return "\n".join(out).strip()

    except Exception as e:
        log.warning("_parse_nfcu_page failed (page %d): %s", page_num, e)
        return ""

def _extract_nfcu_summary_table(pdf_path: str, page_num: int) -> str:
    try:
        import pymupdf as fitz

        doc = fitz.open(pdf_path)
        if page_num >= len(doc):
            doc.close()
            return ""
        page = doc[page_num]
        words = page.get_text("words") or []
        doc.close()
        if not words:
            return ""

        buckets = []
        for w in words:
            if len(w) < 5:
                continue
            x0 = float(w[0])
            y0 = float(w[1])
            tok = str(w[4] or "").strip()
            if not tok:
                continue
            bi = None
            for i, (yb, _) in enumerate(buckets):
                if abs(yb - y0) <= 2.0:
                    bi = i
                    break
            if bi is None:
                buckets.append((y0, [(x0, tok)]))
            else:
                buckets[bi][1].append((x0, tok))

        lines = []
        for y, arr in sorted(buckets, key=lambda x: x[0]):
            arr2 = sorted(arr, key=lambda x: x[0])
            toks = [t for _, t in arr2]
            text = " ".join(toks).strip()
            lines.append((y, toks, text))

        anchor_idx = None
        for i, (_, _, text) in enumerate(lines):
            tl = text.lower()
            if "summary of your deposit accounts" in tl or ("summary" in tl and "deposit accounts" in tl):
                anchor_idx = i
                break
        if anchor_idx is None:
            return ""

        money_re = re.compile(r"^\$?[\d,]+\.\d{2}-?$")
        acct_no_re = re.compile(r"^\d{8,12}$")
        stop_re = re.compile(
            r"(business checking\s*-|mbr business savings\s*-|date\s+transaction\s+detail|items paid)",
            re.IGNORECASE,
        )

        rows = []
        pending_account = ""
        account_name_re = re.compile(
            r"^(business checking|mbr business savings|checking|savings|money market|certificate).*$",
            re.IGNORECASE,
        )

        for _, toks, text in lines[anchor_idx + 1:]:
            if not text:
                continue
            if stop_re.search(text):
                break
            if not toks:
                continue
            t0 = toks[0].lower()
            tl = text.lower()

            if (
                "previous" in tl
                or "deposits" in tl
                or "withdrawals" in tl
                or "ending" in tl
                or "dividends" in tl
            ) and not any(money_re.match(t) for t in toks):
                continue

            if account_name_re.match(text.strip()) and not any(money_re.match(t) for t in toks):
                pending_account = text.strip()
                continue

            mvals = [t.replace("$", "") for t in toks if money_re.match(t)]
            acct_tokens = [t for t in toks if acct_no_re.match(t)]

            if t0 == "totals" and len(mvals) >= 5:
                rows.append(["Totals", mvals[0], mvals[1], mvals[2], mvals[3], mvals[4]])
                continue

            if acct_tokens and len(mvals) >= 5:
                acct = acct_tokens[0]
                acct_label = (pending_account + " - " + acct).strip(" -") if pending_account else acct
                rows.append([acct_label, mvals[0], mvals[1], mvals[2], mvals[3], mvals[4]])
                pending_account = ""
                continue

        if not rows:
            return ""

        hdr = [
            "Account",
            "Previous Balance",
            "Deposits/Credits",
            "Withdrawals/Debits",
            "Ending Balance",
            "YTD Dividends",
        ]
        return _grid_to_html([hdr] + rows)
    except Exception:
        return ""

def _replace_nfcu_summary_table(page_md: str, summary_table_html: str) -> str:
    if not page_md or not summary_table_html:
        return page_md
    table_pattern = re.compile(r"<table[^>]*>.*?</table>", re.DOTALL | re.IGNORECASE)
    lower = page_md.lower()
    anchor = lower.find("summary of your deposit accounts")

    if anchor >= 0:
        for m in table_pattern.finditer(page_md):
            if m.start() > anchor:
                old_table = m.group(0)
                return page_md.replace(old_table, summary_table_html, 1)

    for m in table_pattern.finditer(page_md):
        tbl = m.group(0)
        p = TableGridParser()
        p.feed(tbl)
        g = _build_grid(p.rows)
        if not g:
            continue
        h = " ".join(str(c or "").strip().lower() for c in g[0])
        if (
            "previous" in h
            and "deposits" in h
            and "withdrawals" in h
            and "ending" in h
            and "dividends" in h
        ):
            return page_md.replace(tbl, summary_table_html, 1)

    return page_md

def _split_fused_date_transaction_header(grid):
    if not grid or len(grid) < 2:
        return grid
    ncols = max(len(r) for r in grid if isinstance(r, list))
    if ncols < 3:
        return grid

    header_row_idx = None
    fused_col_idx = None
    for ri in range(min(4, len(grid))):
        r = [str(c or "").strip() for c in (grid[ri] + [""] * (ncols - len(grid[ri])))]
        for ci, c in enumerate(r):
            cc = re.sub(r"\s+", " ", c).strip().lower()
            if cc == "date transaction detail":
                row_text = " ".join(re.sub(r"\s+", " ", x).strip().lower() for x in r)
                if "amount" in row_text and "balance" in row_text:
                    header_row_idx = ri
                    fused_col_idx = ci
                    break
        if header_row_idx is not None:
            break
    if header_row_idx is None or fused_col_idx is None:
        return grid

    date_tok = re.compile(r"^\d{1,2}(?:[-/])\d{2}(?:[-/]\d{2,4})?$")

    new_grid = []
    for ri, row in enumerate(grid):
        cells = [str(c or "").strip() for c in (row + [""] * (ncols - len(row)))]
        first = cells[fused_col_idx]
        dval = ""
        tval = first
        if first:
            parts = first.split()
            if parts and date_tok.match(parts[0]):
                dval = parts[0]
                tval = " ".join(parts[1:]).strip()
        if ri == header_row_idx:
            split_row = (
                cells[:fused_col_idx]
                + ["Date", "Transaction Detail"]
                + cells[fused_col_idx + 1:]
            )
        elif ri < header_row_idx:
            split_row = cells[:fused_col_idx] + [cells[fused_col_idx], ""] + cells[fused_col_idx + 1:]
        else:
            split_row = cells[:fused_col_idx] + [dval, tval] + cells[fused_col_idx + 1:]
        new_grid.append(split_row)
    return new_grid

def _split_combined_summary_daily_balance_grid(grid):
    if not grid or len(grid) < 4:
        return None

    def _norm(s):
        return re.sub(r"\s+", " ", str(s or "").strip().lower())

    daily_idx = None
    for i, row in enumerate(grid):
        row_text = " ".join(_norm(c) for c in row if str(c or "").strip())
        if "daily balance" in row_text:
            daily_idx = i
            break
    if daily_idx is None or daily_idx <= 1 or daily_idx >= len(grid) - 2:
        return None

    date_re = re.compile(r"^\d{1,2}/\d{2}(?:/\d{2,4})?$")
    money_re = re.compile(r"^-?\$?\d{1,3}(?:,\d{3})*(?:\.\d{2})-?$")

    daily_rows = []
    for row in grid[daily_idx + 1:]:
        tokens = []
        for cell in row:
            ct = str(cell or "").strip()
            if not ct:
                continue
            tokens.extend(re.split(r"\s+", ct))
        dates = [t for t in tokens if date_re.match(t)]
        amts = [t for t in tokens if money_re.match(t)]
        if not dates or not amts:
            continue
        for d, a in zip(dates, amts):
            daily_rows.append([d, a])

    if len(daily_rows) < 2:
        return None

    summary = [r for r in grid[:daily_idx] if any(str(c or "").strip() for c in r)]
    if len(summary) < 2:
        return None

    daily = [["Date", "Ledger Balance"]] + daily_rows
    return [summary, daily]

def _normalize_double_desc_amount_header_table(grid):
    if not grid or len(grid) < 2:
        return grid

    def _hdr_tight(s):
        return re.sub(r"\s+", "", str(s or "").strip().lower())

    expected5_t = ["date", "description", "amount", "description", "amount"]
    expected4_t = ["date", "description", "amount", "description"]

    def _row_is_ucb_header(cells):
        if len(cells) == 5:
            return [_hdr_tight(c) for c in cells] == expected5_t
        if len(cells) == 4:
            return [_hdr_tight(c) for c in cells] == expected4_t
        return False

    hdr_idx = None
    ncols = 0
    for i in range(min(5, len(grid))):
        raw_try = [str(c or "").strip() for c in grid[i]]
        if _row_is_ucb_header(raw_try):
            hdr_idx = i
            ncols = len(raw_try)
            break
    if hdr_idx is None:
        return grid

    date_only_re = re.compile(r"^\d{1,2}/\d{1,2}/\d{2,4}$")
    date_prefix_re = re.compile(r"^(\d{1,2}/\d{1,2}/\d{2,4})\s*(.*)$")
    money_cell_re = re.compile(r"^-?\$?\d{1,3}(?:,\d{3})*(?:\.\d{2})-?$")

    def _is_money(s):
        if not s or not str(s).strip():
            return False
        return bool(money_cell_re.match(str(s).strip()))

    def _extract_money(s):
        if not s:
            return ""
        m = re.search(r"(-?\$?\d{1,3}(?:,\d{3})*(?:\.\d{2})-?)", str(s))
        return m.group(1).strip() if m else ""

    out = [["Date", "Description", "Amount"]]

    for row in grid[hdr_idx + 1 :]:
        cells = [str(c or "").strip() for c in row]
        while len(cells) < ncols:
            cells.append("")
        c0, c1, c2 = cells[0], cells[1] if len(cells) > 1 else "", cells[2] if len(cells) > 2 else ""
        c3 = cells[3] if len(cells) > 3 else ""
        c4 = cells[4] if len(cells) > 4 else ""

        if not any(cells[:ncols]):
            continue

        mpre = date_prefix_re.match(c0)
        if mpre and _is_money(c1):
            date_s = mpre.group(1).strip()
            low = (c0 + " " + (mpre.group(2) or "")).lower()
            if "beginning balance" in low:
                desc = "Beginning Balance"
            elif "ending balance" in low:
                desc = "Ending Balance"
            else:
                desc = (mpre.group(2) or "").strip() or "Transaction"
            out.append([date_s, desc, c1])
            continue

        if date_only_re.match(c0.strip()):
            date_s = c0.strip()
            desc = c1
            amt = c2
            if desc and "beginning balance" in desc.lower():
                mx = _extract_money(desc)
                if mx and (not amt or amt in ("$0.00", "0.00", "$0", ".00", "$.00")):
                    amt = mx
                    desc = re.sub(
                        r"(?i)beginning\s+balance\s*\$?[\d,]+\.?\d*\s*",
                        "Beginning Balance ",
                        desc,
                    )
                    desc = re.sub(r"(?i)\s*average\s+ledger\s+balance\s*", " ", desc).strip()
                    if not desc or desc == "Beginning Balance":
                        desc = "Beginning Balance"
            out.append([date_s, desc, amt])
            continue

        if not date_prefix_re.match(c0) and not date_only_re.match(c0.strip()) and _is_money(c1):
            out.append(["", c0, c1])
            continue

        out.append([c0, c1, c2])

    return out if len(out) >= 2 else grid


def _normalize_ucb_three_col_beginning_balance_fusion(grid):
    if not grid or len(grid) < 2:
        return grid

    def _tight(s):
        return re.sub(r"\s+", "", str(s or "").strip().lower())

    h0 = [str(c or "").strip() for c in grid[0]]
    if len(h0) < 3:
        return grid
    if _tight(h0[0]) != "date" or _tight(h0[1]) != "description" or _tight(h0[2]) != "amount":
        return grid

    date_only_re = re.compile(r"^\d{1,2}/\d{1,2}/\d{2,4}$")

    def _extract_money(s):
        if not s:
            return ""
        m = re.search(r"(-?\$?\d{1,3}(?:,\d{3})*(?:\.\d{2})-?)", str(s))
        return m.group(1).strip() if m else ""

    zeroish = {"", "$0.00", "0.00", "$0", ".00", "$.00", "0"}

    out = [list(grid[0])]
    changed = False
    for row in grid[1:]:
        cells = [str(c or "").strip() for c in row]
        while len(cells) < 3:
            cells.append("")
        c0, c1, c2 = cells[0], cells[1], cells[2]
        if (
            date_only_re.match(c0.strip())
            and c1
            and "beginning balance" in c1.lower()
        ):
            mx = _extract_money(c1)
            c2n = c2.strip() if c2 else ""
            if mx and (not c2n or c2n in zeroish):
                out.append([c0.strip(), "Beginning Balance", mx])
                changed = True
                continue
        out.append([c0, c1, c2])

    return out if changed else grid


def _fix_three_col_date_desc_amount_left_shift(grid):
    if not grid or len(grid) < 2:
        return grid

    def _tight(s):
        return re.sub(r"\s+", "", str(s or "").strip().lower())

    h0 = [str(c or "").strip() for c in grid[0]]
    if len(h0) < 3:
        return grid
    if _tight(h0[0]) != "date" or _tight(h0[1]) != "description" or _tight(h0[2]) != "amount":
        return grid

    date_then_desc = re.compile(
        r"^(\d{1,2}/\d{1,2}/\d{2,4})\s+(.+)$"
    )
    money_re = re.compile(r"^-?\$?\d{1,3}(?:,\d{3})*(?:\.\d{2})-?$")

    out = [list(grid[0])]
    changed = False
    for row in grid[1:]:
        cells = [str(c or "").strip() for c in row]
        while len(cells) < 3:
            cells.append("")
        c0, c1, c2 = cells[0], cells[1], cells[2]
        if not c0:
            out.append([c0, c1, c2])
            continue
        m = date_then_desc.match(c0)
        if not m:
            out.append([c0, c1, c2])
            continue
        desc_part = (m.group(2) or "").strip()
        if not desc_part:
            out.append([c0, c1, c2])
            continue
        if not (c1 and money_re.match(c1.strip())):
            out.append([c0, c1, c2])
            continue
        if c2 and str(c2).strip():
            out.append([c0, c1, c2])
            continue
        out.append([m.group(1).strip(), desc_part, c1.strip()])
        changed = True

    return out if changed else grid


def _normalize_fused_date_posted_amount_table(grid):
    if not grid or len(grid) < 2:
        return grid

    ncols = max(len(r) for r in grid if isinstance(r, list))
    if ncols < 2:
        return grid

    def _row_cells(r):
        return [str(c or "").strip() for c in (r + [""] * (ncols - len(r)))]

    hdr_idx = None
    for ri in range(min(6, len(grid))):
        cells = _row_cells(grid[ri])
        row_text = " ".join(cells).lower()
        if (
            "date posted transaction description" in row_text
            and "amount" in row_text
        ):
            hdr_idx = ri
            break
    if hdr_idx is None:
        return grid

    date_re = re.compile(r"^\d{1,2}[/-]\d{1,2}(?:[/-]\d{2,4})?$")
    out = []

    for ri, row in enumerate(grid):
        cells = _row_cells(row)
        if ri == hdr_idx:
            out.append(["Date", "Transaction Description", "Amount"])
            continue
        if ri < hdr_idx:
            joined = " ".join(c for c in cells if c).strip()
            out.append(["", joined, ""])
            continue

        c1 = cells[0].strip()
        c2 = cells[1].strip() if len(cells) > 1 else ""

        lower_c1 = c1.lower()
        if (
            lower_c1.startswith("date posted transaction description")
            or lower_c1 in {"ach additions", "ach deductions", "other additions", "other deductions", "service charges and fees"}
        ):
            continue

        date = ""
        desc = c1
        if c1:
            parts = c1.split()
            if parts and date_re.match(parts[0]):
                date = parts[0]
                desc = " ".join(parts[1:]).strip()

        amount = c2
        if not amount:
            money_cells = []
            for x in cells[1:]:
                t = str(x or "").strip()
                if re.match(r"^-?\$?\d{1,3}(?:,\d{3})*(?:\.\d{2})-?$", t):
                    money_cells.append(t)
            if money_cells:
                amount = money_cells[-1]

        if not amount and desc:
            m = re.search(r"(-?\$?\d{1,3}(?:,\d{3})*(?:\.\d{2})-?)\s*$", desc)
            if m:
                amount = m.group(1)
                desc = desc[:m.start()].strip()

        if not (date or desc or amount):
            continue
        out.append([date, desc, amount])

    if len(out) < 2:
        return grid
    return out


_TD_CONTINUED_BLOB_RE = re.compile(
    r"transactions\s+by\s+date.*continued|continued.*transactions\s+by\s+date",
    re.IGNORECASE | re.DOTALL,
)

# GLM often omits the continued banner inside a standalone <table>; match TD register body text.
_TD_REGISTER_BODY_SIGNAL_RE = re.compile(
    r"TD\s+ZELLE|\bZELLE\b.*\b(?:RECEIVED|SENT)\b|LENDINGCLUB|MM\s*&\s*(?:amp;)?\s*CB|"
    r"MAINTENANCE\s+FEE|OD\s+GRACE\s+FEE\s+REFUND|OVERDRAFT\s+PD",
    re.IGNORECASE,
)


def _is_td_four_col_ddcb_header(cells: List[str]) -> bool:
    """TD continuation: one fused DATE DESCRIPTION cell + DEBIT + CREDIT + BALANCE."""
    if len(cells) < 4:
        return False
    a = re.sub(r"\s+", " ", str(cells[0] or "").strip().lower())
    b = re.sub(r"\s+", " ", str(cells[1] or "").strip().lower())
    c = re.sub(r"\s+", " ", str(cells[2] or "").strip().lower())
    d = re.sub(r"\s+", " ", str(cells[3] or "").strip().lower())
    fused = "date" in a and "description" in a
    deb = b == "debit" or b.startswith("debit")
    cre = "credit" in c and "debit" not in c
    bal = "balance" in d
    return fused and deb and cre and bal


def _split_td_continuation_date_description_grid(grid):
    """
    Fix TD-1: TD Convenience Checking continuation pages use four physical columns; split col0
    into Date + Description so json_data_extractor's five-way column map aligns with row cells.
    """
    if not grid or len(grid) < 3:
        return grid

    blob = " ".join(
        " ".join(str(c or "") for c in row) for row in grid[: min(30, len(grid))]
    )
    if not _TD_CONTINUED_BLOB_RE.search(blob):
        return grid

    hdr_idx = None
    for ri in range(min(20, len(grid))):
        raw = [str(c or "").strip() for c in grid[ri]]
        while len(raw) < 4:
            raw.append("")
        if len(raw) >= 4 and _is_td_four_col_ddcb_header(raw[:4]):
            hdr_idx = ri
            break
    if hdr_idx is None:
        return grid

    date_led_re = re.compile(r"^(\d{1,2}/\d{1,2}(?:/\d{2,4})?)\s+(.*)$", re.DOTALL)
    new_grid: List[List[str]] = []
    for ri, row in enumerate(grid):
        cells = [str(c or "").strip() for c in row]
        if ri < hdr_idx:
            new_grid.append(cells)
            continue
        if ri == hdr_idx:
            c = cells[:4] + [""] * max(0, 4 - len(cells))
            new_grid.append(
                [
                    "DATE",
                    "DESCRIPTION",
                    c[1] or "DEBIT",
                    c[2] or "CREDIT",
                    c[3] or "BALANCE",
                ]
            )
            continue
        while len(cells) < 4:
            cells.append("")
        if len(cells) == 4:
            m = date_led_re.match(cells[0].strip())
            if m:
                new_grid.append(
                    [
                        m.group(1).strip(),
                        (m.group(2) or "").strip(),
                        cells[1],
                        cells[2],
                        cells[3],
                    ]
                )
            else:
                new_grid.append([cells[0], "", cells[1], cells[2], cells[3]])
        elif len(cells) >= 5:
            new_grid.append(cells)
        else:
            while len(cells) < 5:
                cells.append("")
            new_grid.append(cells[:5])

    return new_grid


def _is_td_three_col_register_header(cells: List[str]) -> bool:
    """OCR variant: Date | Transaction Description | Amount (Amount column is running balance)."""
    if len(cells) < 3:
        return False
    a = re.sub(r"\s+", " ", str(cells[0] or "").strip().lower())
    b = re.sub(r"\s+", " ", str(cells[1] or "").strip().lower())
    c = re.sub(r"\s+", " ", str(cells[2] or "").strip().lower())
    if a != "date":
        return False
    if "transaction" not in b or "description" not in b:
        return False
    if c not in ("amount", "balance") and "amount" not in c and "balance" not in c:
        return False
    return True


def _td_trailing_amount_from_description(desc: str) -> Tuple[str, str]:
    """Return (description_without_trailing_amount, amount_str or '')."""
    if not desc or not str(desc).strip():
        return "", ""
    s = str(desc).strip()
    m = re.search(r"(?:,\s*|\s)(-?\d{1,3}(?:,\d{3})*\.\d{2})\s*$", s)
    if not m:
        return s, ""
    amt = m.group(1).strip()
    left = s[: m.start()].strip().rstrip(",")
    return left, amt


def _td_route_debit_credit(desc_raw: str, amt_str: str) -> Tuple[str, str]:
    """Return (debit_cell, credit_cell) with positive amount string in one side."""
    if not amt_str:
        return "", ""
    u = str(desc_raw or "").upper()
    if (
        "ACH DEBIT" in u
        or "OVERDRAFT" in u
        or "ZELLE SENT" in u
        or "MAINTENANCE FEE" in u
        or "ATM DB" in u
        or "ATM DEBIT" in u
        or "ELECTRONIC PMT" in u
        or "PMT-WEB" in u
    ):
        return amt_str, ""
    if (
        "ZELLE RECEIVED" in u
        or "OD GRACE FEE REFUND" in u
        or u.startswith("CREDIT,")
        or u.startswith("DEPOSIT")
    ):
        return "", amt_str
    if "DEBIT" in u and "CREDIT," not in u[:12]:
        return amt_str, ""
    return "", amt_str


def _expand_td_continuation_three_col_register_grid(grid):
    """
    Fix TD-1b: GLM emits Date|Transaction Description|Amount where col3 is balance; expand to
    DATE|DESCRIPTION|DEBIT|CREDIT|BALANCE for json_data_extractor.
    """
    if not grid or len(grid) < 2:
        return grid

    blob = " ".join(
        " ".join(str(c or "") for c in row) for row in grid[: min(40, len(grid))]
    )
    if not (
        _TD_CONTINUED_BLOB_RE.search(blob)
        or _TD_REGISTER_BODY_SIGNAL_RE.search(blob)
    ):
        return grid

    hdr_idx = None
    for ri in range(min(15, len(grid))):
        raw = [str(c or "").strip() for c in grid[ri]]
        while len(raw) < 3:
            raw.append("")
        if _is_td_three_col_register_header(raw[:3]):
            hdr_idx = ri
            break
    if hdr_idx is None:
        return grid

    date_re = re.compile(r"^\d{1,2}/\d{1,2}(?:/\d{2,4})?$")
    new_grid: List[List[str]] = []
    for ri, row in enumerate(grid):
        cells = [str(c or "").strip() for c in row]
        if ri < hdr_idx:
            new_grid.append(cells)
            continue
        if ri == hdr_idx:
            new_grid.append(["DATE", "DESCRIPTION", "DEBIT", "CREDIT", "BALANCE"])
            continue
        while len(cells) < 3:
            cells.append("")
        if len(cells) != 3:
            new_grid.append(cells)
            continue
        d0, d1, bal = cells[0], cells[1], cells[2]
        if not date_re.match(d0):
            new_grid.append([d0, d1, "", "", bal])
            continue
        desc_only, amt_s = _td_trailing_amount_from_description(d1)
        db, cr = _td_route_debit_credit(d1, amt_s)
        new_grid.append([d0, desc_only, db, cr, bal])

    return new_grid


def normalize_html_tables(text: str) -> str:
    if not text or "<table" not in text.lower():
        return text

    pattern = re.compile(r"<table[^>]*>.*?</table>", re.DOTALL | re.IGNORECASE)
    out = []
    last = 0

    for m in pattern.finditer(text):
        out.append(text[last : m.start()])
        table_html = m.group(0)
        try:
            p = TableGridParser()
            p.feed(table_html)
            grid = _build_grid(p.rows)

            if _is_junk_table(grid):
                out.append("")
                last = m.end()
                continue

            grid = _normalize_double_desc_amount_header_table(grid)
            grid = _normalize_ucb_three_col_beginning_balance_fusion(grid)
            grid = _fix_three_col_date_desc_amount_left_shift(grid)

            grid = _drop_truly_empty_columns(grid)
            grid = _merge_blank_header_text_columns(grid)

            grid = _fill_headless_trailing_amount_headers(grid)
            grid = _normalize_date_desc_money_table_alignment(grid)

            grid = _reconstruct_separator_rows(grid)

            grid, recovered_row = _clean_header_artifacts(grid)
            if recovered_row is not None:
                grid.insert(1, recovered_row)

            grid = _promote_misplaced_header_row(grid)

            grid = _split_fused_date_transaction_header(grid)

            grid = _split_td_continuation_date_description_grid(grid)
            grid = _expand_td_continuation_three_col_register_grid(grid)

            grid = _normalize_fused_date_posted_amount_table(grid)

            grid = _split_fused_multicolumn_header(grid)

            grid = _normalize_daily_balance_table(grid)

            grid = _normalize_checks_paid_table(grid)

            grid = _merge_split_rows(grid)

            grid = _extract_fused_desc_amount(grid)

            grid = _repair_da3_garbled_amount_cells(grid)

            grid = _fix_fused_keyvalue_rows(grid)
            grid = _dedupe_duplicate_rows_in_da3_table(grid)
            split_tables = _split_combined_summary_daily_balance_grid(grid)
            if split_tables:
                out.append("\n\n".join(_grid_to_html(g) for g in split_tables if g))
            else:
                out.append(_grid_to_html(grid) if grid else table_html)
        except Exception:
            out.append(table_html)
        last = m.end()

    out.append(text[last:])
    return "".join(out)


def _transaction_row_dedupe_key(cells):
    parts = [str(c or "").strip() for c in cells[:3]]
    while len(parts) < 3:
        parts.append("")
    date_s, desc_s, amt_s = parts[0], parts[1], parts[2]
    date_s = re.sub(r"\s+", "", date_s)
    amt_s = re.sub(r"[\s$,]", "", amt_s).lower().replace("−", "-")
    desc_s = html.unescape(desc_s)
    desc_s = desc_s.replace("`", "'").replace("'", "'")
    desc_s = re.sub(r"\s+", "", desc_s.lower())
    return (date_s, desc_s, amt_s)


def _da3_loose_key(cells):
    parts = [str(c or "").strip() for c in cells[:3]]
    while len(parts) < 3:
        parts.append("")
    date_s, desc_s, amt_s = parts[0], parts[1], parts[2]
    if not _DA3_LEADING_DATE.match(date_s):
        return None
    comb = f"{date_s} {desc_s}".lower()
    if re.search(r"\btotal\b", comb) and len(desc_s) > 25:
        return None
    if not _DA3_STRICT_AMOUNT.match(amt_s.strip()):
        return None
    t_amt = re.sub(r"[\s$,]", "", amt_s).replace("−", "-").replace("$", "")
    try:
        af = round(float(t_amt), 2)
    except ValueError:
        return None
    d_norm = re.sub(r"\s+", "", date_s)
    return (d_norm, af)


def _row_counter_loose_da3(g):
    c = Counter()
    if not g or len(g) < 2:
        return c
    for r in g[1:]:
        cells = [str(x or "").strip() for x in r]
        if not any(cells):
            continue
        k = _da3_loose_key(cells)
        if k:
            c[k] += 1
    return c


def _overlap_ratio_counter(c_num, c_den):
    tot = sum(c_num.values())
    if tot == 0:
        return 0.0
    hit = sum(min(c_num[k], c_den.get(k, 0)) for k in c_num)
    return hit / tot


def _multiset_leq(c_small, c_big):
    return all(c_small[k] <= c_big.get(k, 0) for k in c_small)


def _is_da3_header(grid):
    if not grid or len(grid[0]) < 3:
        return False
    h = [re.sub(r"\s+", " ", str(c or "").strip().lower()) for c in grid[0]]
    if len(h) != 3:
        return False
    return h[0] == "date" and "description" in h[1] and (
        h[2] == "amount" or h[2].startswith("amount")
    )


_DA3_STRICT_AMOUNT = re.compile(r"^-?\$?\d{1,3}(?:,\d{3})*\.\d{2}-?$")
_DA3_MONEY_TOKEN = re.compile(
    r"(?<![\d,])(-?\$?\d{1,3}(?:,\d{3})*\.\d{2}|-?\$?\d{2,6}\.\d{2})(?!\d)"
)
_MSFT_PIN_G_REF = re.compile(r"MICROSOFT#(G\d{6,16})", re.IGNORECASE)

# Module-level text-layer frequency context (Fix-20 override).
# Set by _patch_ocr_with_textlayer; consumed by normalize_html_tables in run_ocr's
# post-patch pass. Cleared at the start of stabilize_tables_and_text (new page)
# and on patch failure so keys never bleed across pages.
# Keys match _transaction_row_dedupe_key (not the _norm()-based tl_freq map).
_tl_freq_context: dict = {}

_DA3_LEADING_DATE = re.compile(r"^\d{1,2}/\d{1,2}(?:/\d{2,4})?$")


def _repair_da3_garbled_amount_cells(grid):
    if not grid or len(grid) < 2 or not _is_da3_header(grid):
        return grid

    date_re = re.compile(r"^\d{1,2}/\d{1,2}(?:/\d{2,4})?$")
    out = [list(grid[0])]
    changed = False

    for row in grid[1:]:
        cells = [str(c or "").strip() for c in row]
        while len(cells) < 3:
            cells.append("")
        c0, c1, c2 = cells[0], cells[1], cells[2]
        if not date_re.match(c0):
            out.append(cells)
            continue
        c2s = c2.strip()
        if _DA3_STRICT_AMOUNT.match(c2s):
            out.append(cells)
            continue
        blob = f"{c1} {c2}".strip()
        if not blob:
            out.append(cells)
            continue
        found = _DA3_MONEY_TOKEN.findall(blob)
        if not found:
            out.append(cells)
            continue
        pick_raw = None
        for m in reversed(found):
            t = m.replace("$", "").replace(",", "")
            try:
                v = float(t)
            except ValueError:
                continue
            if 0.01 <= v <= 9_999_999.99:
                pick_raw = m.replace("$", "").strip()
                break
        if not pick_raw:
            out.append(cells)
            continue
        new_desc = c1
        for suf in (
            pick_raw,
            pick_raw.replace(",", ""),
            f"${pick_raw}",
            f"${pick_raw.replace(',', '')}",
        ):
            if suf and new_desc.rstrip().endswith(suf):
                new_desc = new_desc[: -len(suf)].rstrip()
                break
        out.append([c0, new_desc, pick_raw])
        changed = True

    return out if changed else grid


def _dedupe_duplicate_rows_in_da3_table(grid):
    if not grid or len(grid) < 2:
        return grid
    if not _is_da3_header(grid):
        return grid
    data_rows = []
    for row in grid[1:]:
        cells = [str(c or "").strip() for c in row]
        if not any(cells):
            continue
        data_rows.append(row)

    if not data_rows:
        return grid

    out = [grid[0]]
    i = 0
    while i < len(data_rows):
        row = data_rows[i]
        cells = [str(c or "").strip() for c in row]
        k = _transaction_row_dedupe_key(cells)
        j = i + 1
        while j < len(data_rows):
            ncells = [str(c or "").strip() for c in data_rows[j]]
            if _transaction_row_dedupe_key(ncells) != k:
                break
            j += 1
        run_len = j - i
        # Cap at 2 by default (Fix 20: collapse OCR stutters) but honour
        # the text-layer-confirmed count when available, so legitimate
        # repeated bank transactions (e.g. 6x Temporary Credit Reversal
        # on the same date/amount) are not incorrectly collapsed to 2.
        # Keys absent from _tl_freq_context still cap at 2 — safe no-op
        # for scanned PDFs or pages where no text layer was extracted.
        tl_confirmed = _tl_freq_context.get(k, 0)
        cap = max(tl_confirmed, 2) if tl_confirmed > 0 else 2
        take = min(run_len, cap)
        for t in range(take):
            out.append(data_rows[i + t])
        i = j

    return out


def _row_text_full_ucb(grid, r: int) -> str:
    if not grid or r < 0 or r >= len(grid):
        return ""
    parts = [str(c or "").strip().lower() for c in grid[r]]
    return " ".join(parts)


def _is_itm_deposit_row_ucb(grid, r: int) -> bool:
    if not grid or r >= len(grid) or len(grid[r]) < 2:
        return False
    desc = str(grid[r][1] or "").strip().lower()
    return "itm deposit" in desc


def _ucb_verify_electronic_credits_rows(grid, ec_start: int) -> bool:
    if ec_start + 2 >= len(grid):
        return False
    t0 = _row_text_full_ucb(grid, ec_start)
    t1 = _row_text_full_ucb(grid, ec_start + 1)
    t2 = _row_text_full_ucb(grid, ec_start + 2)
    ok0 = "pos return" in t0 or ("pos" in t0 and "return" in t0)
    ok1 = "acct fund" in t1 or ("apple" in t1 and "inst" in t1)
    ok2 = "dda transfer" in t2 and "in" in t2
    return bool(ok0 and ok1 and ok2)


def _ucb_split_grid_deposits_ec_ed(grid):
    if not grid or len(grid) < 2:
        return None
    if not _is_da3_header(grid):
        return None
    data_start = 1
    i = data_start
    while i < len(grid) and _is_itm_deposit_row_ucb(grid, i):
        i += 1
    if i == data_start:
        return None
    if i + 3 > len(grid):
        return None
    if not _ucb_verify_electronic_credits_rows(grid, i):
        return None
    ec_end = i + 3
    if ec_end >= len(grid):
        return None
    hdr = [list(grid[0])]
    g_dep = hdr + grid[data_start:i]
    g_ec = hdr + grid[i:ec_end]
    g_ed = hdr + grid[ec_end:]
    if len(g_ed) <= 1:
        return None
    return g_dep, g_ec, g_ed


def _is_strong_ucb_electronic_debit_row(grid, r: int) -> bool:
    """True for typical card/ACH debit lines (generic; avoids splitting on POS return / transfer-in credits)."""
    t = _row_text_full_ucb(grid, r)
    if not t or not t.strip():
        return False
    if "pos return" in t:
        return False
    if "dda transfer" in t and "in" in t:
        return False
    if "pos purchase" in t:
        return True
    if "atm withdrawal" in t:
        return True
    if "atm" in t and "withdrawal" in t:
        return True
    if "debit card" in t:
        return True
    if "point of sale" in t:
        return True
    if "recur payment" in t:
        return True
    return False


def _ucb_grid_has_any_strong_debit_row(grid) -> bool:
    if not grid:
        return False
    for r in range(1, len(grid)):
        if not any(str(c or "").strip() for c in grid[r]):
            continue
        if _is_strong_ucb_electronic_debit_row(grid, r):
            return True
    return False


def _ucb_has_itm_deposit_data_row(grid) -> bool:
    for r in range(1, len(grid)):
        if _is_itm_deposit_row_ucb(grid, r):
            return True
    return False


def _ucb_table_credits_only_no_strong_debits(grid) -> bool:
    """True if DA3 table has real data rows but none match strong electronic-debit patterns."""
    if not grid or len(grid) < 2 or not _is_da3_header(grid):
        return False
    has_data = False
    for r in range(1, len(grid)):
        row_txt = _row_text_full_ucb(grid, r).strip()
        if not row_txt:
            continue
        has_data = True
        if _is_strong_ucb_electronic_debit_row(grid, r):
            return False
    return bool(has_data and _ucb_has_itm_deposit_data_row(grid))


def _ucb_html_table_body_looks_itm_credits_only(table_html: str) -> bool:
    """
    Detect ITM-only credit block without relying on parsed grid headers (Fix 17c-html).
    Conservative: require ITM deposit text and absence of common electronic-debit verbs in the HTML.
    """
    if not table_html or "<table" not in table_html.lower():
        return False
    low = table_html.lower()
    if "itm deposit" not in low and "itm dep" not in low:
        return False
    if "pos purchase" in low or "atm withdrawal" in low:
        return False
    if "recur payment" in low or "debit card" in low:
        return False
    return True


def _is_ucb_electronic_credit_row_after_split(grid, r: int) -> bool:
    """
    Rows that belong under Electronic Credits but often land at the end of the debits table after OCR
    (Fix 17i). Must not match outgoing debits (POS purchase, Acct Fund SENT, etc.).
    """
    if not grid or r < 1 or r >= len(grid):
        return False
    if _is_strong_ucb_electronic_debit_row(grid, r):
        return False
    t = _row_text_full_ucb(grid, r)
    if "pos return" in t:
        return True
    if "dda transfer" in t and "in" in t:
        return True
    if "acct fund" in t and ("inst r" in t or "cupertino" in t):
        return True
    return False


def _ucb_move_trailing_credit_rows_from_ed_to_ec(g_ec, g_ed):
    """Fix 17i: credit-like rows that landed in g_ed (POS Return, DDA Transfer IN, Apple INST R) -> g_ec.

    OCR often orders them after outgoing debits on the same date block, so a contiguous tail scan
    misses them; we relocate every matching data row while preserving relative order.
    """
    if not g_ec or not g_ed or len(g_ed) < 2 or not _is_da3_header(g_ec) or not _is_da3_header(g_ed):
        return g_ec, g_ed
    credit_indices = [r for r in range(1, len(g_ed)) if _is_ucb_electronic_credit_row_after_split(g_ed, r)]
    if not credit_indices:
        return g_ec, g_ed
    credit_rows = [list(g_ed[r]) for r in credit_indices]
    drop = set(credit_indices)
    g_ed_new = [list(g_ed[0])] + [list(g_ed[r]) for r in range(1, len(g_ed)) if r not in drop]
    g_ec_new = [list(g_ec[0])] + [list(r) for r in g_ec[1:]] + credit_rows
    max_c = max(len(r) for r in g_ec_new + g_ed_new)
    for row in g_ec_new + g_ed_new:
        while len(row) < max_c:
            row.append("")
    return g_ec_new, g_ed_new


def _ucb_split_grid_electronic_credits_debits_after_itm_prefix(grid):
    """
    Fix 17b: one fused DA3 table under Electronic Credits (+ optional empty EC/ED headings) where OCR
    merged ITM deposits with POS/ATM debits. Split after the leading ITM DEPOSIT run when the next
    data row is a strong electronic-debit pattern.
    """
    if not grid or len(grid) < 3:
        return None
    if not _is_da3_header(grid):
        return None
    data_start = 1
    i = data_start
    while i < len(grid) and _is_itm_deposit_row_ucb(grid, i):
        i += 1
    if i == data_start or i >= len(grid):
        return None
    if not _is_strong_ucb_electronic_debit_row(grid, i):
        return None
    hdr = [list(grid[0])]
    g_ec = hdr + grid[data_start:i]
    g_ed = hdr + grid[i:]
    if len(g_ec) <= 1 or len(g_ed) <= 1:
        return None
    return g_ec, g_ed


def _try_split_ucb_ec_ed_from_fused_table_html(
    lead_before_ec: str, table_html: str, had_ed_heading_immediately_before_table: bool
):
    try:
        p = TableGridParser()
        p.feed(table_html)
        grid = _build_grid(p.rows)
    except Exception:
        return None
    pair = _ucb_split_grid_electronic_credits_debits_after_itm_prefix(grid)
    if pair:
        g_ec, g_ed = pair
        g_ec, g_ed = _ucb_move_trailing_credit_rows_from_ed_to_ec(g_ec, g_ed)
        sect = '<div align="center">\n\n{title}\n\n</div>\n\n'
        parts = []
        if lead_before_ec and lead_before_ec.strip():
            parts.append(lead_before_ec.rstrip())
        parts.append(sect.format(title="Electronic Credits"))
        parts.append(_grid_to_html(g_ec))
        if len(g_ed) > 1:
            parts.append(sect.format(title="Electronic Debits"))
            parts.append(_grid_to_html(g_ed))
        return "\n\n".join(parts)

    # Fix 17c: headings stacked as EC / ED / table but rows are all credits (e.g. ITM-only) —
    # put the table under Electronic Credits, then leave Electronic Debits for continuation.
    if had_ed_heading_immediately_before_table and _ucb_table_credits_only_no_strong_debits(grid):
        sect = '<div align="center">\n\n{title}\n\n</div>\n\n'
        parts = []
        if lead_before_ec and lead_before_ec.strip():
            parts.append(lead_before_ec.rstrip())
        parts.extend(
            [
                sect.format(title="Electronic Credits"),
                _grid_to_html(grid),
                sect.format(title="Electronic Debits"),
            ]
        )
        return "\n\n".join(parts)

    # Fix 17c-html: same reorder when grid header checks fail but raw HTML is clearly ITM-only credits.
    if had_ed_heading_immediately_before_table and _ucb_html_table_body_looks_itm_credits_only(table_html):
        sect = '<div align="center">\n\n{title}\n\n</div>\n\n'
        parts = []
        if lead_before_ec and lead_before_ec.strip():
            parts.append(lead_before_ec.rstrip())
        parts.extend(
            [
                sect.format(title="Electronic Credits"),
                table_html.strip(),
                sect.format(title="Electronic Debits"),
            ]
        )
        return "\n\n".join(parts)

    return None


# Centered block titles: align= center OR inline style (Gradio / browsers vary).
_UCB_CENTER_DIV = (
    r'(?:'
    r'<div[^>]*align\s*=\s*["\']center["\'][^>]*>'
    r'|'
    r'<div[^>]*style\s*=\s*["\'][^"\']*text-align\s*:\s*center[^"\']*["\'][^>]*>'
    r')'
)

# Optional empty centered banner only; EC/ED title divs are re-emitted after a successful split
# so we do not duplicate headings.
_UCB_EC_ED_SINGLE_FUSED_TABLE = re.compile(
    r'(?P<lead>(?:' + _UCB_CENTER_DIV + r'\s*</div>\s*)?)'
    r'(?P<ecdiv>' + _UCB_CENTER_DIV + r'\s*Electronic\s+Credits\s*</div>\s*)'
    r'(?P<ed_before>(?:' + _UCB_CENTER_DIV + r'\s*Electronic\s+Debits\s*</div>\s*)?)'
    r'(?P<table><table[^>]*>.*?</table>)',
    re.DOTALL | re.IGNORECASE,
)

# Fix 17h: debits DA3 first, then DA3 with leading ITM + duplicate tail (no ED between tables).
_UCB_EC_TWO_DA3_DEBITS_BEFORE_ITM_TAIL = re.compile(
    r'(?P<lead>(?:' + _UCB_CENTER_DIV + r'\s*</div>\s*)?)'
    r'(?P<ecdiv>' + _UCB_CENTER_DIV + r'\s*Electronic\s+Credits\s*</div>\s*)'
    r'(?P<ed_maybe>(?:' + _UCB_CENTER_DIV + r'\s*Electronic\s+Debits\s*</div>\s*)?)'
    r'(?P<t1><table[^>]*>.*?</table>)'
    r'\s*'
    r'(?P<t2><table[^>]*>.*?</table>)',
    re.DOTALL | re.IGNORECASE,
)


def _ucb_reorder_ec_ed_debits_first_itm_tail_html(text: str) -> str:
    """
    Fix 17h: Electronic Credits (+ optional Electronic Debits) then two DA3 tables where the first is
    clearly debits and the second begins with a run of ITM DEPOSIT rows (mis-ordered vs credits section).
    """
    if not text or "electronic credits" not in text.lower():
        return text
    low = text.lower()
    if "itm deposit" not in low and "itm dep" not in low:
        return text
    pat = _UCB_EC_TWO_DA3_DEBITS_BEFORE_ITM_TAIL
    _dn = re.compile(r"^\d{1,2}[/\-]\d{2}")
    _amt_ok = re.compile(r"^-?\d+(?:\.\d{1,4})?-?$")

    def _key_cells(cells):
        parts = [str(c or "").strip() for c in cells[:3]]
        while len(parts) < 3:
            parts.append("")
        date_s = re.sub(r"\s+", "", parts[0])
        amt_s = re.sub(r"[\s$,]", "", parts[2]).lower()
        amt_s = amt_s.replace("\u2212", "-").replace("\u2013", "-")
        return date_s, amt_s

    def _row_loose_key(grid, r):
        cells = [str(c or "").strip() for c in grid[r]]
        if not any(cells):
            return None
        k = _key_cells(cells)
        if not (_dn.match(k[0]) and _amt_ok.match(k[1]) and "." in k[1]):
            return None
        return k

    m = pat.search(text)
    if not m:
        return text
    g1 = _parse_da3_grid_from_table_html(m.group("t1"))
    g2 = _parse_da3_grid_from_table_html(m.group("t2"))
    if not g1 or not g2 or not _is_da3_header(g1) or not _is_da3_header(g2):
        return text
    if not _ucb_grid_has_any_strong_debit_row(g1):
        return text
    itm_end = 1
    while itm_end < len(g2):
        if not any(str(c or "").strip() for c in g2[itm_end]):
            itm_end += 1
            continue
        if not _is_itm_deposit_row_ucb(g2, itm_end):
            break
        itm_end += 1
    if itm_end <= 1:
        return text
    cb = Counter()
    for r in range(1, len(g1)):
        kk = _row_loose_key(g1, r)
        if kk:
            cb[kk] += 1
    new_tail = []
    for r in range(itm_end, len(g2)):
        if not any(str(c or "").strip() for c in g2[r]):
            continue
        if _is_itm_deposit_row_ucb(g2, r):
            continue
        kk = _row_loose_key(g2, r)
        if kk and cb.get(kk, 0) > 0:
            cb[kk] -= 1
            continue
        new_tail.append(list(g2[r]))
    g_itm = [list(g2[0])] + [list(g2[r]) for r in range(1, itm_end)]
    merged = [list(g1[0])] + [list(r) for r in g1[1:]] + new_tail
    max_c = max(len(r) for r in merged)
    for row in merged:
        while len(row) < max_c:
            row.append("")
    max_ci = max(len(r) for r in g_itm)
    for row in g_itm:
        while len(row) < max_ci:
            row.append("")
    sect = '<div align="center">\n\n{title}\n\n</div>\n\n'
    parts = []
    lead = m.group("lead") or ""
    if lead.strip():
        parts.append(lead.rstrip())
    parts.extend(
        [
            sect.format(title="Electronic Credits"),
            _grid_to_html(g_itm),
            sect.format(title="Electronic Debits"),
            _grid_to_html(merged),
        ]
    )
    new_blob = "\n\n".join(parts)
    return text[: m.start()] + new_blob + text[m.end() :]


def _ucb_strip_redundant_ed_subset_fragment(text: str) -> str:
    """
    Fix 17d / 17f: Remove a duplicate <div>Electronic Debits</div> + <table> when that table's rows are a
    loose (date, amount) multiset subset of the DA3 table that appears immediately before this block.
    Fix 17f: If the fragment also contains ITM DEPOSIT rows (mis-copied under Debits), ignore those rows
    for the subset check so the tail still compares against the preceding debits table.
    """
    if not text or "electronic debits" not in text.lower():
        return text
    try:
        ed_block = re.compile(
            _UCB_CENTER_DIV + r'\s*Electronic\s+Debits\s*</div>\s*'
            r'(?P<tbl><table[^>]*>.*?</table>)',
            re.DOTALL | re.IGNORECASE,
        )
        tbl_pat = re.compile(r'<table[^>]*>.*?</table>', re.DOTALL | re.IGNORECASE)
        _DATE_NONEMPTY = re.compile(r"^\d{1,2}[/\-]\d{2}")
        # Stripped amounts may be "2194.19" (no comma) — must not use comma-group regex.
        _AMT_STRIPPED_OK = re.compile(r"^-?\d+(?:\.\d{1,4})?-?$")

        def _loose_da_key_cells(cells):
            parts = [str(c or "").strip() for c in cells[:3]]
            while len(parts) < 3:
                parts.append("")
            date_s = re.sub(r"\s+", "", parts[0])
            amt_s = re.sub(r"[\s$,]", "", parts[2]).lower()
            amt_s = amt_s.replace("\u2212", "-").replace("\u2013", "-")
            return (date_s, amt_s)

        def _loose_keys_from_grid(g):
            if not g or not _is_da3_header(g):
                return None
            keys = []
            for row in g[1:]:
                cells = [str(c or "").strip() for c in row]
                if not any(cells):
                    continue
                k = _loose_da_key_cells(cells)
                if (
                    _DATE_NONEMPTY.match(k[0])
                    and _AMT_STRIPPED_OK.match(k[1])
                    and "." in k[1]
                ):
                    keys.append(k)
            return keys

        def _loose_keys_from_grid_skip_itm(g):
            if not g or not _is_da3_header(g):
                return None
            keys = []
            for r in range(1, len(g)):
                if _is_itm_deposit_row_ucb(g, r):
                    continue
                cells = [str(c or "").strip() for c in g[r]]
                if not any(cells):
                    continue
                k = _loose_da_key_cells(cells)
                if (
                    _DATE_NONEMPTY.match(k[0])
                    and _AMT_STRIPPED_OK.match(k[1])
                    and "." in k[1]
                ):
                    keys.append(k)
            return keys

        def _frag_has_itm_data_row(g):
            if not g:
                return False
            for r in range(1, len(g)):
                if not any(str(c or "").strip() for c in g[r]):
                    continue
                if _is_itm_deposit_row_ucb(g, r):
                    return True
            return False

        for _ in range(12):
            changed = False
            for m in ed_block.finditer(text):
                before = text[: m.start()]
                prev_tbl_m = None
                for tm in tbl_pat.finditer(before):
                    prev_tbl_m = tm
                if prev_tbl_m is None:
                    continue
                g_big = _parse_da3_grid_from_table_html(prev_tbl_m.group(0))
                g_frag = _parse_da3_grid_from_table_html(m.group("tbl"))
                kb = _loose_keys_from_grid(g_big)
                kf = _loose_keys_from_grid(g_frag)
                strip = False
                if kb and kf and len(kf) <= 24 and len(kf) < len(kb):
                    cb, cf = Counter(kb), Counter(kf)
                    if all(cf[k] <= cb.get(k, 0) for k in cf):
                        strip = True
                # Fix 17f: ITM rows duplicated from Credits; debit tail ⊆ previous debits table
                if (
                    not strip
                    and kb
                    and g_frag
                    and _frag_has_itm_data_row(g_frag)
                ):
                    kf2 = _loose_keys_from_grid_skip_itm(g_frag)
                    if (
                        kf2
                        and len(kf2) <= 24
                        and len(kf2) < len(kb)
                    ):
                        cb, cf2 = Counter(kb), Counter(kf2)
                        if all(cf2[k] <= cb.get(k, 0) for k in cf2):
                            strip = True
                if strip:
                    text = text[: m.start()] + text[m.end() :]
                    changed = True
                    break
            if not changed:
                break
        return text
    except Exception as e:
        log.warning("_ucb_strip_redundant_ed_subset_fragment failed: %s", e)
        return text


def _ucb_prune_ed_fragment_dupes_vs_prev_table(text: str) -> str:
    """
    Fix 17g: Under an Electronic Debits + DA3 table, drop leading ITM rows and rows that duplicate
    the immediately preceding DA3 table by (date, amount). Skips the case where the previous table is
    credits/ITM-only and this table is the main debits block. If ≤6 orphan rows remain, merge into prev
    and remove this ED block.
    """
    if not text or "electronic debits" not in text.lower():
        return text
    try:
        ed_block = re.compile(
            _UCB_CENTER_DIV + r"\s*Electronic\s+Debits\s*</div>\s*"
            r"(?P<tbl><table[^>]*>.*?</table>)",
            re.DOTALL | re.IGNORECASE,
        )
        tbl_pat = re.compile(r"<table[^>]*>.*?</table>", re.DOTALL | re.IGNORECASE)
        _dn = re.compile(r"^\d{1,2}[/\-]\d{2}")
        _amt_ok = re.compile(r"^-?\d+(?:\.\d{1,4})?-?$")

        def _key_cells(cells):
            parts = [str(c or "").strip() for c in cells[:3]]
            while len(parts) < 3:
                parts.append("")
            date_s = re.sub(r"\s+", "", parts[0])
            amt_s = re.sub(r"[\s$,]", "", parts[2]).lower()
            amt_s = amt_s.replace("\u2212", "-").replace("\u2013", "-")
            return date_s, amt_s

        def _row_loose_key(grid, r):
            cells = [str(c or "").strip() for c in grid[r]]
            if not any(cells):
                return None
            k = _key_cells(cells)
            if not (_dn.match(k[0]) and _amt_ok.match(k[1]) and "." in k[1]):
                return None
            return k

        for _ in range(12):
            changed = False
            for m in ed_block.finditer(text):
                before = text[: m.start()]
                prev_tbl_m = None
                for tm in tbl_pat.finditer(before):
                    prev_tbl_m = tm
                if prev_tbl_m is None:
                    continue
                g_big = _parse_da3_grid_from_table_html(prev_tbl_m.group(0))
                g_frag = _parse_da3_grid_from_table_html(m.group("tbl"))
                if (
                    not g_big
                    or not g_frag
                    or not _is_da3_header(g_big)
                    or not _is_da3_header(g_frag)
                ):
                    continue
                # Legitimate layout: ITM/credits-only table, then Electronic Debits + main debits — do not prune.
                if not _ucb_grid_has_any_strong_debit_row(g_big) and _ucb_grid_has_any_strong_debit_row(g_frag):
                    continue
                cb = Counter()
                for r in range(1, len(g_big)):
                    kk = _row_loose_key(g_big, r)
                    if kk:
                        cb[kk] += 1
                new_rows = []
                seen_non_itm = False
                for r in range(1, len(g_frag)):
                    if not any(str(c or "").strip() for c in g_frag[r]):
                        continue
                    if _is_itm_deposit_row_ucb(g_frag, r):
                        if not seen_non_itm:
                            continue
                        continue
                    seen_non_itm = True
                    kk = _row_loose_key(g_frag, r)
                    if kk and cb.get(kk, 0) > 0:
                        cb[kk] -= 1
                        continue
                    new_rows.append(list(g_frag[r]))
                if not new_rows:
                    text = text[: m.start()] + text[m.end() :]
                    changed = True
                    break
                if len(new_rows) <= 6:
                    merged = [list(g_big[0])] + [list(r) for r in g_big[1:]] + new_rows
                    max_c = max(len(r) for r in merged)
                    for row in merged:
                        while len(row) < max_c:
                            row.append("")
                    merged_html = _grid_to_html(merged)
                    ps, pe = prev_tbl_m.start(), prev_tbl_m.end()
                    text = text[:ps] + merged_html + text[pe : m.start()] + text[m.end() :]
                    changed = True
                    break
                hdr = [list(g_frag[0])]
                max_c = max(len(hdr[0]), max((len(x) for x in new_rows), default=0))
                for row in hdr + new_rows:
                    while len(row) < max_c:
                        row.append("")
                new_tbl = _grid_to_html(hdr + new_rows)
                full = m.group(0)
                tbl = m.group("tbl")
                prefix = full[: full.index(tbl)]
                text = text[: m.start()] + prefix + new_tbl + text[m.end() :]
                changed = True
                break
            if not changed:
                break
        return text
    except Exception as e:
        log.warning("_ucb_prune_ed_fragment_dupes_vs_prev_table failed: %s", e)
        return text


_UCB_ED_DIV_AND_TABLE = re.compile(
    _UCB_CENTER_DIV + r"\s*Electronic\s+Debits\s*</div>\s*(?P<tbl><table[^>]*>.*?</table>)",
    re.DOTALL | re.IGNORECASE,
)
_UCB_EC_DIV_AND_TABLE = re.compile(
    _UCB_CENTER_DIV + r"\s*Electronic\s+Credits\s*</div>\s*(?P<tbl><table[^>]*>.*?</table>)",
    re.DOTALL | re.IGNORECASE,
)


def _ucb_relocate_credit_rows_from_ed_tables_to_ec_html(text: str) -> str:
    """
    Fix 17j: Credit-side rows (POS Return, DDA Transfer IN, Apple Cash INST R) sometimes appear only under a
    duplicate Electronic Debits table, not in the fused EC table — move them to the preceding EC table.
    """
    if not text or "electronic debits" not in text.lower():
        return text
    try:
        while True:
            replaced = False
            for ed_m in _UCB_ED_DIV_AND_TABLE.finditer(text):
                g_ed = _parse_da3_grid_from_table_html(ed_m.group("tbl"))
                if not g_ed or len(g_ed) < 2 or not _is_da3_header(g_ed):
                    continue
                credit_ix = [
                    r
                    for r in range(1, len(g_ed))
                    if _is_ucb_electronic_credit_row_after_split(g_ed, r)
                ]
                if not credit_ix:
                    continue
                before = text[: ed_m.start()]
                ec_matches = list(_UCB_EC_DIV_AND_TABLE.finditer(before))
                if not ec_matches:
                    continue
                ec_m = ec_matches[-1]
                g_ec = _parse_da3_grid_from_table_html(ec_m.group("tbl"))
                if not g_ec or len(g_ec) < 2 or not _is_da3_header(g_ec):
                    continue
                drop = set(credit_ix)
                credit_rows = [list(g_ed[r]) for r in credit_ix]
                g_ed_new = [list(g_ed[0])] + [list(g_ed[r]) for r in range(1, len(g_ed)) if r not in drop]
                g_ec_new = [list(g_ec[0])] + [list(r) for r in g_ec[1:]] + credit_rows
                max_c = max(len(r) for r in g_ec_new + g_ed_new)
                for row in g_ec_new + g_ed_new:
                    while len(row) < max_c:
                        row.append("")
                new_ec = _grid_to_html(g_ec_new)
                if len(g_ed_new) <= 1:
                    text = (
                        text[: ec_m.start("tbl")]
                        + new_ec
                        + text[ec_m.end("tbl") : ed_m.start()]
                        + text[ed_m.end() :]
                    )
                else:
                    new_ed = _grid_to_html(g_ed_new)
                    text = (
                        text[: ec_m.start("tbl")]
                        + new_ec
                        + text[ec_m.end("tbl") : ed_m.start()]
                        + text[ed_m.start() : ed_m.start("tbl")]
                        + new_ed
                        + text[ed_m.end() :]
                    )
                log.info("Fix 17j: relocated %d credit row(s) from Electronic Debits to Electronic Credits", len(credit_rows))
                replaced = True
                break
            if not replaced:
                break
        return text
    except Exception as e:
        log.warning("_ucb_relocate_credit_rows_from_ed_tables_to_ec_html failed: %s", e)
        return text


def _split_ucb_fused_ec_ed_headings_single_table_html(text: str) -> str:
    if not text or "electronic credits" not in text.lower():
        return text
    if "<table" not in text.lower():
        return text

    def _repl(m):
        lead = m.group("lead")
        ed_before = m.group("ed_before") or ""
        table_html = m.group("table")
        had_ed = bool(ed_before.strip())
        new = _try_split_ucb_ec_ed_from_fused_table_html(lead, table_html, had_ed)
        return new if new is not None else m.group(0)

    return _UCB_EC_ED_SINGLE_FUSED_TABLE.sub(_repl, text)


def _try_split_ucb_deposits_table_to_three(prefix: str, table_html: str):
    try:
        p = TableGridParser()
        p.feed(table_html)
        grid = _build_grid(p.rows)
    except Exception:
        return None
    triple = _ucb_split_grid_deposits_ec_ed(grid)
    if not triple:
        return None
    g_dep, g_ec, g_ed = triple
    sect = '<div align="center">\n\n{title}\n\n</div>\n\n'
    parts = [prefix.rstrip(), _grid_to_html(g_dep), sect.format(title="Electronic Credits"), _grid_to_html(g_ec)]
    parts.append(sect.format(title="Electronic Debits"))
    parts.append(_grid_to_html(g_ed))
    return "\n\n".join(parts)


_UCB_DEP_CONT_BLOCK = re.compile(
    r'(' + _UCB_CENTER_DIV + r'\s*Deposits\s*\(continued\)\s*</div>\s*)'
    r'(<table[^>]*>.*?</table>)'
    r'(?:\s*' + _UCB_CENTER_DIV + r'\s*Electronic\s+Credits\s*</div>\s*'
    + _UCB_CENTER_DIV + r'\s*Electronic\s+Debits\s*</div>)?',
    re.DOTALL | re.IGNORECASE,
)


def _split_ucb_deposits_electronic_sections_html(text: str) -> str:
    if not text:
        return text
    if "deposits (continued)" in text.lower():

        def _repl(m):
            prefix = m.group(1)
            table_html = m.group(2)
            new = _try_split_ucb_deposits_table_to_three(prefix, table_html)
            if new is None:
                return m.group(0)
            return new

        text = _UCB_DEP_CONT_BLOCK.sub(_repl, text)
    # Fix 17e: repeat 17h reorder + fused EC/ED fix + duplicate ED strip + 17g prune until stable.
    for _ in range(16):
        nxt = _ucb_reorder_ec_ed_debits_first_itm_tail_html(text)
        nxt = _split_ucb_fused_ec_ed_headings_single_table_html(nxt)
        nxt = _ucb_strip_redundant_ed_subset_fragment(nxt)
        nxt = _ucb_prune_ed_fragment_dupes_vs_prev_table(nxt)
        nxt = _ucb_relocate_credit_rows_from_ed_tables_to_ec_html(nxt)
        if nxt == text:
            break
        text = nxt
    return text


def _parse_da3_grid_from_table_html(table_html: str):
    try:
        p = TableGridParser()
        p.feed(table_html)
        grid = _build_grid(p.rows)
        if grid and _is_da3_header(grid):
            return grid
    except Exception:
        pass
    return None


def _strip_orphan_continued_da3_flatline(text: str) -> str:
    if not text or "(continued)" not in text.lower():
        return text
    lines = text.splitlines(keepends=True)
    out = []
    for line in lines:
        core = line.rstrip("\r\n")
        date_hits = len(re.findall(r"\d{2}/\d{2}", core)) + len(
            re.findall(r"\b\d{2}-\d{2}-\d{2}\b", core)
        )
        if (
            len(core) > 80
            and "(continued)" in core.lower()
            and "description" in core.lower()
            and "amount" in core.lower()
            and date_hits >= 3
        ):
            continue
        out.append(line)
    return "".join(out)


def _strip_standalone_continued_banner_line(text: str) -> str:
    if not text or "(continued)" not in text.lower():
        return text
    lines = text.splitlines(keepends=True)
    out = []
    for line in lines:
        core = line.rstrip("\r\n")
        if not core.strip():
            out.append(line)
            continue
        c = core.lower()
        if "$" in core or "=" in core:
            out.append(line)
            continue
        if (
            "(continued)" in c
            and "date" not in c
            and "description" not in c
            and "amount" not in c
            and len(re.findall(r"\d{2}/\d{2}", core)) == 0
            and len(core) < 240
        ):
            continue
        out.append(line)
    return "".join(out)


def _patch_da3_msft_g_ref_amounts_across_tables(text: str) -> str:
    if not text or "<table" not in text.lower():
        return text
    if "microsoft#" not in text.lower():
        return text

    tbl_pat = re.compile(r"<table[^>]*>.*?</table>", re.DOTALL | re.IGNORECASE)

    def _collect():
        g_map = {}
        ambiguous = set()
        for m in tbl_pat.finditer(text):
            g = _parse_da3_grid_from_table_html(m.group(0))
            if not g:
                continue
            for row in g[1:]:
                cells = [str(c or "").strip() for c in row]
                while len(cells) < 3:
                    cells.append("")
                c0, c1, c2 = cells[0], cells[1], cells[2]
                if not _DA3_LEADING_DATE.match(c0):
                    continue
                if not _DA3_STRICT_AMOUNT.match(c2):
                    continue
                blob = f"{c0} {c1} {c2}"
                for gid in _MSFT_PIN_G_REF.findall(blob):
                    k = gid.upper()
                    if k in ambiguous:
                        continue
                    trip = (c0, c1, c2)
                    prev = g_map.get(k)
                    if prev is not None and prev != trip:
                        ambiguous.add(k)
                        g_map.pop(k, None)
                    elif prev is None:
                        g_map[k] = trip
        return g_map, ambiguous

    g_map, ambiguous = _collect()
    if not g_map:
        return text

    out_chunks = []
    last = 0
    changed_any = False
    for m in tbl_pat.finditer(text):
        out_chunks.append(text[last : m.start()])
        table_html = m.group(0)
        grid = _parse_da3_grid_from_table_html(table_html)
        if not grid:
            out_chunks.append(table_html)
            last = m.end()
            continue
        new_grid = [list(grid[0])]
        changed = False
        for row in grid[1:]:
            cells = [str(c or "").strip() for c in row]
            while len(cells) < 3:
                cells.append("")
            c0, c1, c2 = cells[0], cells[1], cells[2]
            if not _DA3_LEADING_DATE.match(c0):
                new_grid.append(row)
                continue
            if _DA3_STRICT_AMOUNT.match(c2):
                new_grid.append(row)
                continue
            blob = f"{c0} {c1} {c2}"
            ids = _MSFT_PIN_G_REF.findall(blob)
            if len(ids) != 1:
                new_grid.append(row)
                continue
            k = ids[0].upper()
            if k in ambiguous or k not in g_map:
                new_grid.append(row)
                continue
            d0, d1, d2 = g_map[k]
            new_grid.append([d0, d1, d2])
            changed = True
        if changed:
            out_chunks.append(_grid_to_html(new_grid))
            changed_any = True
        else:
            out_chunks.append(table_html)
        last = m.end()
    out_chunks.append(text[last:])
    return "".join(out_chunks) if changed_any else text


def _trim_adjacent_da3_suffix_duplicating_next_table_prefix(text: str) -> str:
    if not text or "<table" not in text.lower():
        return text

    pattern = re.compile(r"<table[^>]*>.*?</table>", re.DOTALL | re.IGNORECASE)

    for _ in range(48):
        matches = list(pattern.finditer(text))
        if len(matches) < 2:
            break
        changed = False
        for i in range(len(matches) - 1):
            g1 = _parse_da3_grid_from_table_html(matches[i].group(0))
            g2 = _parse_da3_grid_from_table_html(matches[i + 1].group(0))
            if not g1 or not g2 or len(g1) < 3 or len(g2) < 2:
                continue
            d1 = g1[1:]
            d2 = g2[1:]
            if len(d1) < 2 or len(d2) < 2:
                continue
            max_k = min(len(d1), len(d2))
            best_k = 0
            for k in range(max_k, 1, -1):
                ok = True
                for j in range(k):
                    c1 = [str(x or "").strip() for x in d1[-k + j][:3]]
                    c2 = [str(x or "").strip() for x in d2[j][:3]]
                    while len(c1) < 3:
                        c1.append("")
                    while len(c2) < 3:
                        c2.append("")
                    if _transaction_row_dedupe_key(c1) != _transaction_row_dedupe_key(c2):
                        ok = False
                        break
                if ok:
                    best_k = k
                    break
            if best_k < 2:
                continue
            if len(d1) - best_k < 1:
                continue
            new_grid = [g1[0]] + d1[:-best_k]
            new_html = _grid_to_html(new_grid)
            m = matches[i]
            text = text[: m.start()] + new_html + text[m.end() :]
            changed = True
            break
        if not changed:
            break
    return text


def _dedupe_da3_by_date_amount_loose(text: str) -> str:
    """
    Fix 30: Drop small DA3 table fragments (1-5 data rows) that are a loose
    subset of an earlier larger DA3 table, matched by (date, amount) key alone.

    This catches OCR-truncated duplicate rows where description text differs
    between the fragment and the anchor table, preventing the exact-key dedup
    in _dedupe_subset_transaction_tables_html from firing.

    Example: BoA service-fee page — VIDDYOZE -1.11 appears in a 1-row
    fragment table after a full 2-row service-fee table that has the same
    row with a much longer description string.

    Guards:
      - Fragment must have 1-5 data rows.
      - Anchor table must have more data rows than the fragment.
      - Every (date, amount) pair in the fragment must be covered by the anchor.
      - Date must be non-empty; amount must look like currency.
      - Wrapped in try/except (safe no-op on any error).
    """
    if not text or "<table" not in text.lower():
        return text
    try:
        pattern = re.compile(r"<table[^>]*>.*?</table>", re.DOTALL | re.IGNORECASE)
        matches = list(pattern.finditer(text))
        if len(matches) < 2:
            return text

        _DATE_NONEMPTY = re.compile(r"^\d{1,2}[/\-]\d{2}")
        _AMT_NONEMPTY  = re.compile(r"^-?\$?\d{1,3}(?:,\d{3})*(?:\.\d{2})-?$")

        def _da_key(cells):
            """Loose (date_norm, amount_norm) key — description ignored."""
            parts = [str(c or "").strip() for c in cells[:3]]
            while len(parts) < 3:
                parts.append("")
            date_s = re.sub(r"\s+", "", parts[0])
            amt_s  = re.sub(r"[\s$,]", "", parts[2]).lower()
            amt_s  = amt_s.replace("\u2212", "-").replace("\u2013", "-")
            return (date_s, amt_s)

        # Parse all DA3 tables and collect their (date, amount) key lists.
        da3_info = []
        for m in matches:
            g = _parse_da3_grid_from_table_html(m.group(0))
            if not g or not _is_da3_header(g) or len(g) < 2:
                continue
            keys = []
            for row in g[1:]:
                cells = [str(c or "").strip() for c in row]
                if not any(cells):
                    continue
                k = _da_key(cells)
                if _DATE_NONEMPTY.match(k[0]) and _AMT_NONEMPTY.match(k[1]):
                    keys.append(k)
            da3_info.append({"match": m, "keys": keys, "n": len(keys)})

        if len(da3_info) < 2:
            return text

        drop_set = set()
        for i in range(1, len(da3_info)):
            ti = da3_info[i]
            if ti["n"] == 0 or ti["n"] > 5:
                continue
            if ti["match"].start() in drop_set:
                continue
            frag_keys = Counter(ti["keys"])
            for j in range(i - 1, -1, -1):
                tj = da3_info[j]
                if tj["match"].start() in drop_set:
                    continue
                if tj["n"] <= ti["n"]:
                    continue
                big_keys = Counter(tj["keys"])
                # Every (date, amount) pair in the fragment must be covered
                # by the anchor table (multiset subset check).
                if all(frag_keys[k] <= big_keys.get(k, 0) for k in frag_keys):
                    drop_set.add(ti["match"].start())
                    log.info(
                        "Fix 30: dropped DA3 fragment (%d rows) as loose "
                        "(date+amount) subset of larger table (%d rows)",
                        ti["n"], tj["n"],
                    )
                    break

        if not drop_set:
            return text

        out = []
        last = 0
        for m in matches:
            out.append(text[last: m.start()])
            if m.start() not in drop_set:
                out.append(m.group(0))
            last = m.end()
        out.append(text[last:])
        return "".join(out)

    except Exception as e:
        log.warning("_dedupe_da3_by_date_amount_loose failed: %s", e)
        return text


def _dedupe_subset_transaction_tables_html(text: str) -> str:
    if not text or "<table" not in text.lower():
        return text

    pattern = re.compile(r"<table[^>]*>.*?</table>", re.DOTALL | re.IGNORECASE)

    def _parse_grid(html: str):
        try:
            p = TableGridParser()
            p.feed(html)
            return _build_grid(p.rows)
        except Exception:
            return None

    def _row_set(g):
        s = set()
        for r in g[1:]:
            if not any(str(c or "").strip() for c in r):
                continue
            cells = [str(c or "").strip() for c in r]
            if not any(cells):
                continue
            s.add(_transaction_row_dedupe_key(cells))
        return s

    def _one_pass(t: str) -> str:
        matches = list(pattern.finditer(t))
        if len(matches) < 2:
            return t

        grids = []
        for m in matches:
            grids.append((m, _parse_grid(m.group(0))))

        drop_idx = set()
        for i in range(1, len(grids)):
            if i in drop_idx:
                continue
            mi, gi = grids[i]
            if not _is_da3_header(gi):
                continue
            si = _row_set(gi)
            if not si:
                continue

            for j in range(i):
                if j in drop_idx:
                    continue
                mj, gj = grids[j]
                if not _is_da3_header(gj):
                    continue
                sj = _row_set(gj)
                if not sj:
                    continue

                if sj <= si and len(sj) < len(si):
                    drop_idx.add(j)
                    continue

                if si <= sj and len(si) < len(sj):
                    drop_idx.add(i)
                    break

                if si == sj:
                    drop_idx.add(i)
                    break

                inter = si & sj
                ri = len(inter) / len(si) if si else 0.0
                rj = len(inter) / len(sj) if sj else 0.0
                if ri >= 0.88 and len(si) <= len(sj):
                    drop_idx.add(i)
                    break
                if rj >= 0.88 and len(sj) < len(si):
                    drop_idx.add(j)
                    continue

                ci = _row_counter_loose_da3(gi)
                cj = _row_counter_loose_da3(gj)
                ni, nj = sum(ci.values()), sum(cj.values())
                if ni >= 1 and nj >= 1:
                    oi_j = _overlap_ratio_counter(ci, cj)
                    oj_i = _overlap_ratio_counter(cj, ci)
                    if (
                        nj >= 8
                        and 3 <= ni <= 14
                        and ni < nj
                        and _multiset_leq(ci, cj)
                        and oi_j >= 0.9
                    ):
                        drop_idx.add(i)
                        break
                    if ni >= 6 and nj >= 6:
                        if oi_j >= 0.88 and oj_i >= 0.88:
                            drop_idx.add(i)
                            break
                        if _multiset_leq(ci, cj) and ni < nj and oi_j >= 0.9:
                            drop_idx.add(i)
                            break
                        if _multiset_leq(cj, ci) and nj < ni and oj_i >= 0.9:
                            drop_idx.add(j)
                            continue

        if not drop_idx:
            return t

        out = []
        last = 0
        for idx, m in enumerate(matches):
            out.append(t[last : m.start()])
            if idx not in drop_idx:
                out.append(m.group(0))
            last = m.end()
        out.append(t[last:])
        return "".join(out)

    out = text
    for _ in range(12):
        nxt = _one_pass(out)
        if nxt == out:
            break
        out = nxt
    return out


_RE_CENTER_DIV_TWO_DA3_TABLES = re.compile(
    r'(<div\s+align=["\']center["\']\s*>\s*[^<]*?</div>\s*)'
    r'(<table[^>]*>.*?</table>)'
    r'(\s*<table[^>]*>.*?</table>)',
    re.IGNORECASE | re.DOTALL,
)


def _da3_row_credit_vs_debit_category(desc: str) -> str:
    d = (desc or "").lower()
    if "debit card return" in d:
        return "credit"
    if "debit card" in d and "return" in d:
        return "credit"
    if "deposit" in d and "debit card" not in d[:60]:
        return "credit"
    if "incoming wire" in d or "wire ref" in d:
        return "credit"
    if "zelle business reversal" in d or ("reversal" in d and "zelle" in d):
        return "credit"
    if "lrm claims" in d or ("claims" in d and "adj" in d):
        return "credit"
    if "adp tax" in d and "customer" in d:
        return "credit"
    if "debit card" in d:
        return "debit"
    if "ach corp debit" in d:
        return "debit"
    if "bus online ach settlement" in d:
        return "debit"
    if "zelle business payment to" in d:
        return "debit"
    if "visa money transfer debit" in d:
        return "debit"
    if "telephone payment" in d or "service charges" in d:
        return "debit"
    if "merch fee" in d:
        return "debit"
    if "deposit" in d:
        return "credit"
    return "unknown"


def _da3_table_credit_debit_fractions(grid):
    deb = cre = unk = 0
    if not grid or len(grid) < 2:
        return 0.0, 0.0, 0.0, 0
    for row in grid[1:]:
        cells = [str(c or "").strip() for c in row]
        while len(cells) < 3:
            cells.append("")
        if not _DA3_LEADING_DATE.match(cells[0]):
            continue
        if not _DA3_STRICT_AMOUNT.match((cells[2] or "").strip()):
            continue
        cat = _da3_row_credit_vs_debit_category(cells[1])
        if cat == "debit":
            deb += 1
        elif cat == "credit":
            cre += 1
        else:
            unk += 1
    tot = deb + cre + unk
    if tot == 0:
        return 0.0, 0.0, 0.0, 0
    return deb / tot, cre / tot, unk / tot, tot


def _drop_da3_misplaced_debit_after_deposit_heading_tables(text: str) -> str:
    if not text or "<table" not in text.lower():
        return text
    if "deposit" not in text.lower():
        return text

    for _ in range(12):
        replaced = False
        pos = 0
        while True:
            m = _RE_CENTER_DIV_TWO_DA3_TABLES.search(text, pos)
            if not m:
                break
            div = m.group(1)
            t1 = m.group(2)
            t2 = m.group(3)
            sl = div.lower()
            if "deposit" not in sl or not any(k in sl for k in ("credit", "interest", "credits")):
                pos = m.start() + 1
                continue

            g1 = _parse_da3_grid_from_table_html(t1)
            g2 = _parse_da3_grid_from_table_html(t2)
            if not g1 or not g2 or not _is_da3_header(g1) or not _is_da3_header(g2):
                pos = m.start() + 1
                continue

            db1, cr1, unk1, n1 = _da3_table_credit_debit_fractions(g1)
            db2, cr2, _unk2, n2 = _da3_table_credit_debit_fractions(g2)
            if n1 < 5 or n2 < 10:
                pos = m.start() + 1
                continue
            credit_like_first = cr1 + 0.42 * unk1
            if credit_like_first < 0.30 and credit_like_first <= db1 + 0.05:
                pos = m.start() + 1
                continue
            if db2 < 0.74 or db2 <= cr2 + 0.07:
                pos = m.start() + 1
                continue

            text = text[: m.start()] + div + t1 + text[m.end() :]
            replaced = True
            break
        if not replaced:
            break

    return text


# --------------------------
# Fix 28: Strip credit-row prefix from debit DA3 table
# --------------------------

_DEBIT_HEADING_KEYWORDS = re.compile(
    r"\b(withdrawal|withdrawals|debit|debits|charges?|subtraction|subtractions|"
    r"payments?\s+made|money\s+out|paid\s+out|electronic\s+debit)\b",
    re.IGNORECASE,
)
_CREDIT_HEADING_KEYWORDS = re.compile(
    r"\b(deposit|deposits|credit|credits|addition|additions|interest\s+paid|"
    r"money\s+in|paid\s+in|incoming|received)\b",
    re.IGNORECASE,
)
_HEADING_SCAN_WINDOW = 600


def _get_heading_before_table(text: str, table_start: int) -> str:
    window = text[max(0, table_start - _HEADING_SCAN_WINDOW): table_start]
    clean = re.sub(r"<[^>]+>", " ", window)
    lines = [ln.strip() for ln in clean.splitlines() if ln.strip()]
    for line in reversed(lines):
        if len(line) > 3 and "---" not in line:
            return line
    return ""


def _strip_credits_prefix_from_debit_da3_table(text: str) -> str:
    """
    Fix 28: When OCR prepends the rows of a preceding credits/deposits DA3 table
    onto the top of a debits/withdrawals DA3 table, strip that prefix.
    Safety guards (ALL must pass):
      1. Both tables must be valid DA3 (Date|Description|Amount header).
      2. Target (later) table must follow a debit/withdrawal heading.
      3. Source (earlier) table must follow a credit/deposit heading.
      4. Only the contiguous leading prefix of matching rows is removed.
      5. At least 1 data row must remain in the target after stripping.
      6. Strip count must be >= 1.
      7. Entire function wrapped in try/except — returns input unchanged on error.
    """
    if not text or "<table" not in text.lower():
        return text
    try:
        pattern = re.compile(r"<table[^>]*>.*?</table>", re.DOTALL | re.IGNORECASE)
        matches = list(pattern.finditer(text))
        if len(matches) < 2:
            return text

        da3_tables = []
        for m in matches:
            g = _parse_da3_grid_from_table_html(m.group(0))
            if not g or not _is_da3_header(g):
                continue
            heading = _get_heading_before_table(text, m.start())
            da3_tables.append({"match": m, "grid": g, "heading": heading})

        if len(da3_tables) < 2:
            return text

        def _row_keys(grid):
            keys = set()
            for row in grid[1:]:
                cells = [str(c or "").strip() for c in row]
                if any(cells):
                    keys.add(_transaction_row_dedupe_key(cells))
            return keys

        drop_map = {}
        for i in range(1, len(da3_tables)):
            ti = da3_tables[i]
            hi = ti["heading"].lower()
            if not _DEBIT_HEADING_KEYWORDS.search(hi):
                continue
            gi = ti["grid"]
            data_i = gi[1:]
            if len(data_i) < 2:
                continue
            for j in range(i - 1, -1, -1):
                tj = da3_tables[j]
                hj = tj["heading"].lower()
                if not _CREDIT_HEADING_KEYWORDS.search(hj):
                    continue
                gj = tj["grid"]
                keys_j = _row_keys(gj)
                if not keys_j:
                    continue
                strip_count = 0
                for row in data_i:
                    cells = [str(c or "").strip() for c in row]
                    k = _transaction_row_dedupe_key(cells)
                    if k in keys_j:
                        strip_count += 1
                    else:
                        break
                if strip_count < 1:
                    continue
                if len(data_i) - strip_count < 1:
                    continue
                drop_map[ti["match"].start()] = strip_count
                log.info(
                    "Fix 28: stripped %d leading credit row(s) from debit table at offset %d",
                    strip_count, ti["match"].start(),
                )
                break

        if not drop_map:
            return text

        out_chunks = []
        last = 0
        for m in matches:
            out_chunks.append(text[last: m.start()])
            if m.start() in drop_map:
                strip_n = drop_map[m.start()]
                g = _parse_da3_grid_from_table_html(m.group(0))
                if g and _is_da3_header(g) and len(g) - 1 - strip_n >= 1:
                    new_grid = [g[0]] + g[1 + strip_n:]
                    out_chunks.append(_grid_to_html(new_grid))
                else:
                    out_chunks.append(m.group(0))
            else:
                out_chunks.append(m.group(0))
            last = m.end()
        out_chunks.append(text[last:])
        return "".join(out_chunks)

    except Exception as e:
        log.warning("_strip_credits_prefix_from_debit_da3_table failed: %s", e)
        return text


# --------------------------
# Fix 29: Move misplaced debit heading to before its orphan DA3 table
# --------------------------

_CENTER_DIV_HDR_RE = re.compile(
    r'<div\s+align=["\']center["\']\s*>\s*(.*?)\s*</div>',
    re.IGNORECASE | re.DOTALL,
)
_HEADING_SCAN_FORWARD = 800


def _fix29_move_debit_heading_before_orphan_table(text: str) -> str:
    """
    Fix 29: When OCR places a debit/withdrawal section heading AFTER the withdrawals table
    instead of before it, move the heading to its correct position (just before the table).
    """
    if not text or "<table" not in text.lower():
        return text
    try:
        tbl_pat = re.compile(r"<table[^>]*>.*?</table>", re.DOTALL | re.IGNORECASE)
        matches = list(tbl_pat.finditer(text))
        if len(matches) < 2:
            return text

        da3_matches = [m for m in matches if _parse_da3_grid_from_table_html(m.group(0))]
        if len(da3_matches) < 2:
            return text

        moves = []

        for i in range(1, len(da3_matches)):
            orphan_m = da3_matches[i]
            orphan_start = orphan_m.start()
            orphan_end = orphan_m.end()
            orphan_tbl_html = orphan_m.group(0)

            # Do not treat UCB Electronic Credits ITM-only tables as "orphan debits" needing a heading
            # moved ahead of them — Fix 17b leaves ED after the ITM block; moving ED here stacks both
            # headings above ITM and leaves debits under the wrong banner.
            try:
                og = _parse_da3_grid_from_table_html(orphan_tbl_html)
                if og and (
                    _ucb_table_credits_only_no_strong_debits(og)
                    or _ucb_html_table_body_looks_itm_credits_only(orphan_tbl_html)
                ):
                    continue
            except Exception:
                pass

            before_window = text[max(0, orphan_start - _HEADING_SCAN_WINDOW): orphan_start]
            before_clean = re.sub(r"<[^>]+>", " ", before_window)
            lines_before = [l.strip() for l in before_clean.splitlines() if l.strip()]
            last_meaningful = ""
            for l in reversed(lines_before):
                if len(l) > 3 and "---" not in l:
                    last_meaningful = l
                    break
            if _DEBIT_HEADING_KEYWORDS.search(last_meaningful):
                continue

            big_before = text[max(0, orphan_start - 1200): orphan_start]
            if not _CREDIT_HEADING_KEYWORDS.search(re.sub(r"<[^>]+>", " ", big_before)):
                continue

            next_tbl_m = tbl_pat.search(text, orphan_end)

            dh_start = dh_end = None
            for div_m in _CENTER_DIV_HDR_RE.finditer(text, orphan_end, orphan_end + _HEADING_SCAN_FORWARD):
                div_clean = re.sub(r"<[^>]+>", "", div_m.group(1)).strip()
                if _DEBIT_HEADING_KEYWORDS.search(div_clean) and not _CREDIT_HEADING_KEYWORDS.search(div_clean):
                    if next_tbl_m and next_tbl_m.start() < div_m.start():
                        break
                    dh_start = div_m.start()
                    dh_end = div_m.end()
                    break

            if dh_start is None:
                continue

            moves.append((orphan_start, dh_start, dh_end))

        if not moves:
            return text

        for orphan_start, dh_start, dh_end in reversed(moves):
            orphan_m2 = next((m for m in tbl_pat.finditer(text) if m.start() == orphan_start), None)
            if not orphan_m2:
                continue
            orphan_end = orphan_m2.end()

            heading_html = text[dh_start: dh_end]
            gap = text[orphan_end: dh_start]
            orphan_html = text[orphan_start: orphan_end]
            part_before = text[:orphan_start]
            part_after = text[dh_end:]

            text = part_before + heading_html + "\n\n" + orphan_html + gap.rstrip() + part_after
            log.info(
                "Fix 29: moved debit heading (was at offset %d) to before orphan DA3 table (offset %d)",
                dh_start, orphan_start,
            )

        return text

    except Exception as e:
        log.warning("_fix29_move_debit_heading_before_orphan_table failed: %s", e)
        return text


_CHECK_IMAGE_GALLERY_LINE_RE = re.compile(
    r"(?<![\d])(\d{3,5})\s*\$\s*([\d,]+\.\d{2})\s*(\d{1,2}/\d{1,2}/\d{4})",
    re.IGNORECASE,
)


def _normalize_check_image_gallery_pages(text: str) -> str:
    """
    Fix MD-3: Replace GLM check-scan page noise (markdown images with bbox + scattered div/line text)
    with a single HTML table when the page looks like a check image gallery, not a structured table.
    """
    if not text or ("bbox=" not in text and "![](" not in text):
        return text
    try:
        sep = "---page-separator---"
        parts = text.split(sep)
        out: List[str] = []
        for chunk in parts:
            out.append(_maybe_convert_check_image_gallery_chunk(chunk))
        return sep.join(out)
    except Exception as e:
        log.warning("_normalize_check_image_gallery_pages failed: %s", e)
        return text


def _maybe_convert_check_image_gallery_chunk(chunk: str) -> str:
    if not chunk or not chunk.strip():
        return chunk
    if "YOUR CHECKS SEQUENCED" in chunk.upper():
        return chunk

    matches = list(_CHECK_IMAGE_GALLERY_LINE_RE.finditer(chunk))
    if len(matches) < 2:
        return chunk

    bbox_hits = chunk.count("bbox=")
    img_hits = len(re.findall(r"!\[\]\s*\(", chunk))
    if bbox_hits < 2 and img_hits < 2:
        return chunk

    rows: List[Tuple[str, str, str]] = []
    seen = set()
    for m in matches:
        key = (m.group(1), m.group(2), m.group(3))
        if key in seen:
            continue
        seen.add(key)
        rows.append(key)

    pos_img = chunk.find("![](")
    pos_first = matches[0].start()
    if pos_img == -1:
        start = pos_first
    else:
        start = min(pos_img, pos_first)

    prefix = chunk[:start].rstrip()
    lines = [
        "<table>",
        "<tr><th>Check #</th><th>Amount</th><th>Date</th></tr>",
    ]
    for num, amt, dt in rows:
        lines.append(f"<tr><td>{num}</td><td>{amt}</td><td>{dt}</td></tr>")
    lines.append("</table>")
    table_html = "\n".join(lines)
    return (prefix + "\n\n" + table_html) if prefix else table_html


def _strip_check_image_ocr_junk_html(text: str) -> str:
    """
    Fix 31: Remove hallucinated rows on scanned check/receipt regions and normalize amount cells.
    """
    if not text or "<table" not in text.lower():
        return text
    try:
        text = re.sub(
            r"(\$[\d,]+\.\d{2})\s*\[[^\]]+\]",
            r"\1",
            text,
        )
        tbl_pat = re.compile(r"(<table[^>]*>)([\s\S]*?)(</table>)", re.IGNORECASE)

        def _tr_effectively_empty(tr: str) -> bool:
            core = re.sub(r"<[^>]+>", " ", tr)
            return not core.strip()

        def repl(m) -> str:
            open_tag, body, close_tag = m.group(1), m.group(2), m.group(3)
            low = body.lower()
            if not any(
                k in low
                for k in (
                    "hotline",
                    "witness la",
                    "itm checking",
                    "care none",
                    "processed ",
                    "course dut",
                    "n00590409",
                )
            ):
                return m.group(0)
            kept: List[str] = []
            for trm in re.finditer(r"<tr\b[^>]*>[\s\S]*?</tr>", body, re.IGNORECASE):
                tr = trm.group(0)
                tl = tr.lower()
                if _tr_effectively_empty(tr):
                    continue
                if "[hotline" in tl or "hotline control" in tl:
                    continue
                if "witness la" in tl and ("course" in tl or "dut" in tl):
                    continue
                if "itm checking witness" in tl or (
                    "itm checking" in tl and "witness" in tl
                ):
                    continue
                if "care none" in tl and "account" in tl:
                    continue
                if "processed " in tl and " @ " in tl and "items" in tl:
                    continue
                if re.search(r"\bN\d{15,}\b", tr):
                    continue
                kept.append(tr)
            if not kept:
                return m.group(0)
            return open_tag + "".join(kept) + close_tag

        text = tbl_pat.sub(repl, text)
        text = re.sub(r"\n## UCB\s*\n+(?=<table)", "\n", text, flags=re.IGNORECASE)
        return text
    except Exception as e:
        log.warning("_strip_check_image_ocr_junk_html failed: %s", e)
        return text


def _fix_markdown_fidelity_typos_and_escapes(text: str) -> str:
    """Fix MD-1: common OCR name glitches and double HTML-escaped ampersands in OCR markdown."""
    if not text:
        return text
    text = re.sub(
        r"\bCINTHIAL\s+BARTON\b",
        "CINTHIA L BARTON",
        text,
        flags=re.IGNORECASE,
    )
    text = text.replace("&amp;amp;", "&amp;")
    return text


def _gap_ok_loose_register_pair(ma, mb, text: str) -> bool:
    gap = text[ma.end() : mb.start()]
    if re.search(r"<table", gap, re.IGNORECASE):
        return False
    if re.search(r"page-separator", gap, re.IGNORECASE):
        return False
    return True


def _norm_register_amount_key(s: str) -> str:
    t = str(s or "").strip().replace("$", "").replace(",", "").replace("−", "-")
    if not t or t in ("---", "—"):
        return ""
    try:
        return f"{float(t):.2f}"
    except ValueError:
        return t


def _register_row_loose_key_cells(cells: List[str]) -> Optional[Tuple[str, str, str, str]]:
    if len(cells) < 5:
        return None
    d = str(cells[0]).strip()
    if not re.match(r"^\d{1,2}/\d{1,2}(?:/\d{2,4})?$", d):
        return None
    deb = _norm_register_amount_key(cells[2])
    cre = _norm_register_amount_key(cells[3])
    bal = _norm_register_amount_key(cells[4])
    return (d, deb, cre, bal)


def _register_multiset_and_score(grid) -> Tuple[Optional[Counter], int]:
    if not grid or len(grid) < 2:
        return None, 0
    ncols = max(len(r) for r in grid if isinstance(r, list))
    if ncols < 5:
        return None, 0
    c = Counter()
    desc_score = 0
    for row in grid[1:]:
        cells = [str(x or "").strip() for x in row]
        while len(cells) < ncols:
            cells.append("")
        cells = cells[: max(5, ncols)]
        if len(cells) < 5:
            continue
        k = _register_row_loose_key_cells(cells[:5])
        if not k:
            continue
        c[k] += 1
        desc_score += len(cells[1])
    if sum(c.values()) < 3:
        return None, 0
    return c, desc_score


def _dedupe_adjacent_loose_register_duplicate_tables(text: str) -> str:
    """
    Fix MD-1: Drop adjacent duplicate bank register <table> blocks when date+debit+credit+balance
    multiset matches but description text differs (OCR double-emission). Keeps the richer table.
    """
    if not text or "<table" not in text.lower():
        return text
    try:
        tbl_pat = re.compile(r"<table[^>]*>.*?</table>", re.DOTALL | re.IGNORECASE)
        matches = list(tbl_pat.finditer(text))
        if len(matches) < 2:
            return text

        def _grid(m):
            p = TableGridParser()
            p.feed(m.group(0))
            return _build_grid(p.rows)

        keep_indices: List[int] = [0]
        for j in range(1, len(matches)):
            prev_i = keep_indices[-1]
            prev_m, cur_m = matches[prev_i], matches[j]
            if not _gap_ok_loose_register_pair(prev_m, cur_m, text):
                keep_indices.append(j)
                continue
            ga, gb = _grid(prev_m), _grid(cur_m)
            ma, sa = _register_multiset_and_score(ga)
            mb, sb = _register_multiset_and_score(gb)
            if ma is None or mb is None or ma != mb:
                keep_indices.append(j)
                continue
            if sb >= sa:
                keep_indices[-1] = j
                log.info(
                    "Fix MD-1: loose-dup register — kept table at offset %d (desc score %d vs %d)",
                    cur_m.start(),
                    sb,
                    sa,
                )
            else:
                log.info(
                    "Fix MD-1: dropped loose-dup register table at offset %d",
                    cur_m.start(),
                )

        drop = set(range(len(matches))) - set(keep_indices)
        if not drop:
            return text

        out: List[str] = []
        pos = 0
        for i, m in enumerate(matches):
            out.append(text[pos : m.start()])
            if i not in drop:
                out.append(m.group(0))
            pos = m.end()
        out.append(text[pos:])
        return "".join(out)
    except Exception as e:
        log.warning("_dedupe_adjacent_loose_register_duplicate_tables failed: %s", e)
        return text


def _dedupe_adjacent_duplicate_html_tables(text: str) -> str:
    """
    Drop consecutive <table> blocks that are identical after grid parse (same header + row multiset),
    with only whitespace between them. Catches OCR echo where the same credits/debits grid is emitted twice.
    """
    if not text or "<table" not in text.lower():
        return text
    tbl_pat = re.compile(r"<table[^>]*>.*?</table>", re.DOTALL | re.IGNORECASE)
    matches = list(tbl_pat.finditer(text))
    if len(matches) < 2:
        return text

    def _fingerprint(html: str):
        try:
            p = TableGridParser()
            p.feed(html)
            g = _build_grid(p.rows)
            if not g or len(g) < 3:
                return None
            hdr = tuple(str(c or "").strip() for c in g[0])
            body = []
            for r in g[1:]:
                if not any(str(c or "").strip() for c in r):
                    continue
                body.append(tuple(str(c or "").strip() for c in r))
            if len(body) < 2:
                return None
            return (hdr, tuple(sorted(Counter(body).items())))
        except Exception:
            return None

    def _gap_ok(ma, mb) -> bool:
        gap = text[ma.end() : mb.start()]
        return not re.search(r"<table", gap, re.IGNORECASE)

    drop = set()
    ref = 0
    for j in range(1, len(matches)):
        if j in drop:
            continue
        fp_r = _fingerprint(matches[ref].group(0))
        fp_j = _fingerprint(matches[j].group(0))
        if fp_r is not None and fp_j is not None and fp_r == fp_j and _gap_ok(
            matches[ref], matches[j]
        ):
            drop.add(j)
        else:
            ref = j

    if not drop:
        return text

    out: List[str] = []
    pos = 0
    for i, m in enumerate(matches):
        out.append(text[pos : m.start()])
        if i not in drop:
            out.append(m.group(0))
        pos = m.end()
    out.append(text[pos:])
    return "".join(out)


def stabilize_tables_and_text(page_md: str) -> str:
    if not page_md:
        return page_md

    # Fresh page: drop any text-layer dedupe caps so the first normalize pass
    # (before _patch_ocr_with_textlayer) does not see stale keys from a prior page.
    _tl_freq_context.clear()

    page_md = normalize_money_glyphs(page_md)
    page_md = _fix_markdown_fidelity_typos_and_escapes(page_md)

    blocks = re.split(r"\n\s*\n", page_md.strip())
    out_blocks = []
    for b in blocks:
        if looks_like_markdown_table(b):
            out_blocks.append(md_table_to_html(b))
        else:
            out_blocks.append(b)

    stabilized = "\n\n".join(out_blocks)
    stabilized = convert_plaintext_bank_sections(stabilized)
    stabilized = _strip_orphan_continued_da3_flatline(stabilized)
    stabilized = _strip_standalone_continued_banner_line(stabilized)
    stabilized = normalize_html_tables(stabilized)
    stabilized = _dedupe_adjacent_duplicate_html_tables(stabilized)
    stabilized = _dedupe_adjacent_loose_register_duplicate_tables(stabilized)
    stabilized = _fix_markdown_fidelity_typos_and_escapes(stabilized)
    stabilized = _trim_adjacent_da3_suffix_duplicating_next_table_prefix(stabilized)
    stabilized = _split_ucb_deposits_electronic_sections_html(stabilized)
    stabilized = _patch_da3_msft_g_ref_amounts_across_tables(stabilized)
    stabilized = _dedupe_subset_transaction_tables_html(stabilized)
    stabilized = _dedupe_da3_by_date_amount_loose(stabilized)  # Fix 30
    stabilized = _drop_da3_misplaced_debit_after_deposit_heading_tables(stabilized)
    stabilized = _strip_credits_prefix_from_debit_da3_table(stabilized)  # Fix 28
    stabilized = _fix29_move_debit_heading_before_orphan_table(stabilized)  # Fix 29
    stabilized = _strip_orphan_continued_da3_flatline(stabilized)
    stabilized = _strip_standalone_continued_banner_line(stabilized)
    stabilized = _strip_check_image_ocr_junk_html(stabilized)  # Fix 31
    stabilized = _normalize_check_image_gallery_pages(stabilized)  # Fix MD-3
    stabilized = _fix_markdown_fidelity_typos_and_escapes(stabilized)
    return close_unclosed_html(stabilized)

# --------------------------
# PDF rendering with padding
# --------------------------
def render_pdf_pages_to_images(pdf_path: str) -> Tuple[List[str], List[int]]:
    import pymupdf as fitz
    from PIL import Image, ImageEnhance

    doc = fitz.open(pdf_path)
    page_images: List[str] = []
    page_heights: List[int] = []

    for i in range(len(doc)):
        page = doc[i]
        pix = page.get_pixmap(matrix=fitz.Matrix(RENDER_SCALE, RENDER_SCALE), alpha=False)

        img = Image.frombytes("RGB", [pix.width, pix.height], pix.samples)

        if ENABLE_CONTRAST:
            img = ImageEnhance.Contrast(img).enhance(1.12)

        w, h = img.size
        pad_l = int(w * PAD_LEFT_FRAC)
        pad_r = int(w * PAD_RIGHT_FRAC)
        pad_t = int(h * PAD_TOP_FRAC)
        pad_b = int(h * PAD_BOTTOM_FRAC)

        if any(p > 0 for p in (pad_l, pad_r, pad_t, pad_b)):
            canvas = Image.new("RGB", (w + pad_l + pad_r, h + pad_t + pad_b), (255, 255, 255))
            canvas.paste(img, (pad_l, pad_t))
            img = canvas

        img_path = os.path.join(tempfile.gettempdir(), f"glmocr_page_{os.getpid()}_{i}.png")
        img.save(img_path, "PNG", compress_level=6)
        page_images.append(img_path)
        page_heights.append(img.height)

    doc.close()
    return page_images, page_heights

# --------------------------
# GLM-OCR result extraction
# --------------------------
def get_page_md_and_regions(page_result):
    md = ""
    if hasattr(page_result, "markdown_result") and page_result.markdown_result:
        md = (page_result.markdown_result or "").strip()
    regions = []
    if hasattr(page_result, "json_result"):
        jr = page_result.json_result
        if isinstance(jr, dict) and "regions" in jr:
            regions = jr.get("regions") or []
        elif isinstance(jr, list) and len(jr) > 0:
            r = jr[0] if isinstance(jr[0], list) else jr
            if isinstance(r, list):
                regions = r
            elif isinstance(r, dict) and "regions" in r:
                regions = r.get("regions") or []
    return md, regions

# --------------------------
# Main entry
# --------------------------
def run_ocr(uploaded_file):
    if uploaded_file is None:
        return "Please upload a file."

    page_images = []
    try:
        path = uploaded_file.name if hasattr(uploaded_file, "name") else str(uploaded_file)
        is_pdf = path.lower().endswith(".pdf")
        is_nfcu_doc = _is_nfcu_document(path) if is_pdf else False
        parser = get_parser()

        page_heights = []

        if is_pdf:
            page_images, page_heights = render_pdf_pages_to_images(path)
            results = parser.parse(page_images)
        else:
            page_images = [path]
            page_heights = [1000]
            results = parser.parse(path)

        if not isinstance(results, list):
            results = [results]

        all_pages = []
        for page_num, page_result in enumerate(results):
            # Header normalize runs before body patch; do not reuse prior page caps.
            _tl_freq_context.clear()
            page_md, regions = get_page_md_and_regions(page_result)
            img_h = page_heights[page_num] if page_num < len(page_heights) else 1000
            header_end_frac, footer_start_frac = get_header_footer_zones(regions, img_h)

            he = header_end_frac if header_end_frac is not None else DEFAULT_ZONE_FRAC
            fs = footer_start_frac if footer_start_frac is not None else (1.0 - DEFAULT_ZONE_FRAC)

            he = max(0.02, min(0.25, he))
            fs = max(0.75, min(0.98, fs))

            parts = []

            hdr = ""
            if is_pdf:
                hdr = extract_zone_text_pdf(path, page_num, 0, he)
                if not (hdr and hdr.strip()):
                    hdr = extract_pdf_text_in_band(path, page_num, 0, PDF_HEADER_BAND_FRAC)
            if not (hdr and hdr.strip()) and page_num < len(page_images):
                hdr = ocr_zone(page_images[page_num], 0, he)
            if hdr and hdr.strip():
                parts.append(normalize_html_tables(fix_account_number(normalize_money_glyphs(hdr.strip()))))

            if page_md and page_md.strip():
                stabilized = stabilize_tables_and_text(page_md.strip())
                if is_pdf and not is_nfcu_doc:
                    stabilized = _patch_ocr_with_textlayer(stabilized, path, page_num)
                    stabilized = _strip_orphan_continued_da3_flatline(stabilized)
                    stabilized = _strip_standalone_continued_banner_line(stabilized)
                    stabilized = normalize_html_tables(stabilized)
                    stabilized = _dedupe_adjacent_duplicate_html_tables(stabilized)
                    stabilized = _dedupe_adjacent_loose_register_duplicate_tables(stabilized)
                    stabilized = _fix_markdown_fidelity_typos_and_escapes(stabilized)
                    stabilized = _trim_adjacent_da3_suffix_duplicating_next_table_prefix(stabilized)
                    stabilized = _split_ucb_deposits_electronic_sections_html(stabilized)
                    stabilized = _patch_da3_msft_g_ref_amounts_across_tables(stabilized)
                    stabilized = _dedupe_subset_transaction_tables_html(stabilized)
                    stabilized = _dedupe_da3_by_date_amount_loose(stabilized)  # Fix 30
                    stabilized = _drop_da3_misplaced_debit_after_deposit_heading_tables(stabilized)
                    stabilized = _strip_credits_prefix_from_debit_da3_table(stabilized)  # Fix 28
                    stabilized = _fix29_move_debit_heading_before_orphan_table(stabilized)  # Fix 29
                    stabilized = _strip_standalone_continued_banner_line(stabilized)
                    stabilized = _strip_check_image_ocr_junk_html(stabilized)  # Fix 31
                    stabilized = _normalize_check_image_gallery_pages(stabilized)  # Fix MD-3
                elif is_pdf and is_nfcu_doc:
                    nfcu_summary = _extract_nfcu_summary_table(path, page_num)
                    if nfcu_summary:
                        stabilized = _replace_nfcu_summary_table(stabilized, nfcu_summary)

                if is_pdf:
                    try:
                        needs_checks = 'class="jb-summary-checks"' not in stabilized
                        needs_dab    = "daily account balance" not in stabilized.lower()
                        if needs_checks or needs_dab:
                            _txt4d = ""
                            try:
                                from pdfminer.high_level import extract_text as _pm_extract
                                _txt4d = _pm_extract(path, page_numbers=[page_num])
                            except Exception:
                                pass
                            if not _txt4d:
                                try:
                                    import pymupdf as _fitz4d
                                    _d4 = _fitz4d.open(path)
                                    _txt4d = _d4[page_num].get_text()
                                    _d4.close()
                                except Exception:
                                    pass
                            if _txt4d:
                                jb_extra = _extract_jb_summary_sections(_txt4d, needs_checks, needs_dab)
                                if jb_extra:
                                    stabilized = stabilized.rstrip() + "\n\n" + jb_extra
                    except Exception:
                        pass

                parts.append(stabilized)

            if ENABLE_FOOTER_OCR and page_num < len(page_images):
                ftr = ""
                if is_pdf:
                    ftr = extract_zone_text_pdf(path, page_num, fs, 1.0)
                    if not (ftr and ftr.strip()):
                        ftr = extract_pdf_text_in_band(path, page_num, PDF_FOOTER_BAND_FRAC, 1.0)
                if not (ftr and ftr.strip()):
                    ftr = ocr_zone(page_images[page_num], fs, 1.0)
                if ftr and ftr.strip():
                    ftr_clean = normalize_money_glyphs(ftr.strip())

                    ftr_first_line = next(
                        (ln.strip().lower() for ln in ftr_clean.splitlines() if ln.strip()),
                        ""
                    )
                    already_present = ftr_first_line and any(
                        ftr_first_line in part.lower() for part in parts
                    )

                    _footer_date_re = re.compile(r"\b\d{1,2}[-/]\d{2}\b")
                    _footer_amt_re  = re.compile(r"\b\d{1,3}(?:,\d{3})*\.\d{2}\b")
                    _date_hits = len(_footer_date_re.findall(ftr_clean))
                    _amt_hits  = len(_footer_amt_re.findall(ftr_clean))
                    is_txn_dump = _date_hits >= 3 and _amt_hits >= 3

                    if not already_present and not is_txn_dump:
                        parts.append(ftr_clean)

            if parts:
                all_pages.append("\n\n".join(parts))

        merged = "\n\n---page-separator---\n\n".join(all_pages) if all_pages else "(No content)"
        if all_pages and merged and not merged.startswith("Error") and "<table" in merged.lower():
            merged = _strip_orphan_continued_da3_flatline(merged)
            merged = _strip_standalone_continued_banner_line(merged)
            merged = _dedupe_adjacent_duplicate_html_tables(merged)
            merged = _dedupe_adjacent_loose_register_duplicate_tables(merged)
            merged = _fix_markdown_fidelity_typos_and_escapes(merged)
            merged = _trim_adjacent_da3_suffix_duplicating_next_table_prefix(merged)
            merged = _split_ucb_deposits_electronic_sections_html(merged)
            merged = _patch_da3_msft_g_ref_amounts_across_tables(merged)
            merged = _dedupe_subset_transaction_tables_html(merged)
            merged = _dedupe_da3_by_date_amount_loose(merged)  # Fix 30
            merged = _drop_da3_misplaced_debit_after_deposit_heading_tables(merged)
            merged = _strip_credits_prefix_from_debit_da3_table(merged)  # Fix 28
            merged = _fix29_move_debit_heading_before_orphan_table(merged)  # Fix 29
            merged = _strip_orphan_continued_da3_flatline(merged)
            merged = _strip_standalone_continued_banner_line(merged)
            merged = _strip_check_image_ocr_junk_html(merged)  # Fix 31
            merged = _normalize_check_image_gallery_pages(merged)  # Fix MD-3
        return merged

    except Exception as e:
        import traceback
        log.exception("run_ocr failed: %s", e)
        return f"Error: {e}\n\n{traceback.format_exc()}"

    finally:
        for p in page_images:
            try:
                if isinstance(p, str) and p.endswith(".png") and "glmocr_page_" in os.path.basename(p):
                    os.unlink(p)
            except Exception:
                pass

with gr.Blocks(title="GLM-OCR") as demo:
    gr.Markdown("# GLM-OCR\nUpload a PDF or image. Headers included; tables stabilized.")
    file_in = gr.File(label="Upload PDF or image", file_types=[".pdf", ".png", ".jpg", ".jpeg", ".tiff", ".bmp"])
    run_btn = gr.Button("Run OCR", variant="primary")
    out = gr.Textbox(lines=40, label="Output (markdown)")
    run_btn.click(fn=run_ocr, inputs=file_in, outputs=out)

if __name__ == "__main__":
    demo.launch()