""" GLM-OCR Hugging Face Space app for PDF/image OCR with header inclusion and table-structure stabilization for downstream bank-statement pipelines. Hard-coded knobs (no environment variables required). Primary goals for reconcile rate: - preserve right-most columns (often "Balance") by higher DPI render + right padding - keep tables as tables (convert markdown pipe tables -> HTML table) - return ---page-separator--- between pages - normalize HTML tables generically so downstream parsing/classification is stable: 1) expand colspan/rowspan into a rectangular grid 2) drop truly-empty columns (common in summary tables) 3) merge "blank header" columns that contain text into the left column (common when DESCRIPTION is split) 4) clean common header artifacts (e.g. "DESCRIPTIONBeginning Balance", "BALANCE$3,447.10") 5) recover fused first data row generically (two-signal guard: text fusion + money fusion must both fire) 6) promote misplaced header row: when real column headers land in a data row, restructure the grid 7) fix fused key-value rows in summary sections (e.g. Interest Summary) appended to transaction tables 8) reconstruct mid-table separator rows split across columns by OCR (e.g. 'Card account # XXXX 2 | | 889') - footer extraction enabled on all pages with deduplication guard (no double-print if body OCR already captured it) - Fix 4: cross-validate OCR table row counts against PDF text layer; inject missing duplicate rows (safe no-op for scanned PDFs and non-transaction pages); after injection, re-run normalize_html_tables + subset dedupe so UCB-style fused cells from the text layer get Fix 13/14 - Fix NF-1: extract Navy Federal "Summary of your deposit accounts" table from PDF text layer using spatial word positions (GLM-OCR fails on this wide-column layout) - Fix 13: collapse Date|Description|Amount|Description|Amount tables to 3 columns; fix fused Beginning/Ending balance rows (UCB-style wide summary tables) - Fix 14: UCB 3-col tables where Beginning Balance amount is fused into description and Amount is $0.00 - Fix 15: UCB (and similar) 3-col rows where Date+Description are fused in col1 and Amount sits in col2 with col3 empty - Fix 16: remove duplicate rows within a single Date|Description|Amount table; multi-pass table dedupe + final doc-wide dedupe - Fix 17: UCB page 3 — split one merged table under Deposits (continued) into Deposits + Electronic Credits + Electronic Debits (PDF text order); drop orphan empty EC/ED headers that followed the merged table - Fix 17b: When markdown has Electronic Credits (+ optional Electronic Debits headings) but a single fused Date|Description|Amount table (common when Deposits (continued) is absent from OCR), split after a leading ITM DEPOSIT run when the next row matches generic electronic-debit patterns (POS purchase, ATM withdrawal, etc.). Safe no-op when there is no ITM prefix or no such debit row after it. - Fix 17c: When Electronic Debits appears *before* the table but every data row is credit-side (no strong debit pattern) and at least one ITM DEPOSIT row exists, OCR stacked headings wrong — emit Electronic Credits, then the table, then Electronic Debits (section continues below). Safe no-op if the Debits heading was not before the table or any row looks like a card/ATM debit. - Fix 17c-html: Same as 17c using raw HTML cues when TableGridParser/header normalization would skip the table. - Fix 17d: Drop a second Electronic Debits block + DA3 table when its rows are a loose (date+amount) multiset subset of the immediately preceding DA3 table (OCR duplicate tail / wrong split). - Fix 17f: Same as 17d when the duplicate table *starts with* ITM DEPOSIT rows (copied from Electronic Credits) then repeats debits from the table above — 17d alone misses those because ITM keys are not in the debits-only predecessor table. Strip ITM rows from the multiset check, then apply the 17d subset rule. - Fix 17g: When 17f cannot drop the whole block (e.g. one orphan debit row only appears in the duplicate tail), drop leading ITM rows and each fragment row whose (date, amount) still has count in the previous debits table; remove the ED wrapper if the table becomes empty. If a small orphan tail remains (≤6 rows), append those rows to the preceding debits table and drop the extra ED heading. - Fix 17e: Repeat EC/ED fused + 17d passes until stable (handles chained replacements). Centered section titles match both align=\"center\" and style=\"text-align:center\" on divs. - Post-pass: dedupe 3-col Date|Description|Amount tables when one is a duplicate fragment (later subset of earlier, earlier subset of later, or identical row sets) - Fix 18: adjacent DA3 tables — if the last K data rows of table N match the first K rows of table N+1 (same normalized Date|Description|Amount keys), strip that suffix from table N. Fixes formats (e.g. Truist) where deposit rows are glued onto the withdrawals table and repeated under the proper deposits heading (K>=2; first table must keep >=1 data row). - Fix 19: DA3 rows whose Amount cell is not strict currency (e.g. fused card digits + merchant tail) — take the rightmost plausible money token from Description+Amount text for the amount. - Fix 20: DA3 in-table dedupe — cap consecutive identical normalized keys at 2 rows (keeps legitimate back-to-back identical charges; still collapses 3+ OCR stutters). - Fix 21: Cross-table DA3 repair for MSBILL / MICROSOFT#G… PIN rows — when Amount is junk in one table but another table repeats the same authorization id with a strict amount, copy that amount (then subset dedupe can drop the stray duplicate block, e.g. under Deposits). - Fix 22: Drop a single OCR "flat line" between pages: (continued) + DESCRIPTION + AMOUNT + many MM/DD tokens (noise before a proper ). - Fix 23: Doc-wide DA3 table dedupe using multiset of (date, amount) keys — catches duplicate sections when OCR varies description text between copies (e.g. withdrawals repeated under Deposits). - Fix 24: Run orphan flatline strip on merged PDF output and after text-layer patch so page-boundary noise is removed consistently. - Fix 25: After a centered "deposits + credits/interest" heading, if two DA3 tables appear back-to-back and the first is mostly credit/deposit rows while the second is mostly debit/withdrawal-style rows (generic keywords), drop the second — fixes OCR pasting the withdrawals block again. - Fix 23b: Loose multiset keys use normalized float amounts so comma/spacing variants still match. - Fix 26: Drop small DA3 fragments (3–14 rows) that are a multiset subset of an earlier larger DA3 table — matches PDF layouts that re-print a header and repeat a few rows. - Fix 27: Strip a lone "…(continued)" account banner line (no DATE/DESCRIPTION/AMOUNT) between page separators — duplicate header noise without the long flatline. - Fix 28: Strip leading credit/deposit rows that OCR prepends to a withdrawals/debits DA3 table when the preceding DA3 table was under a deposits/credits heading. Only strips the contiguous prefix of the debit table whose rows appear in the credit table; requires >=1 data row to remain; entire function wrapped in try/except (safe no-op on any error). - Fix 29: Move a misplaced debit/withdrawal heading to before its orphan DA3 table when OCR places the heading AFTER the table instead of before it. Pattern detected: [credit heading+table] ... [orphan DA3 table] ... [debit heading] — the heading is relocated to just before the orphan table. Guards: no debit heading already before table; credit context must precede table; no other table between orphan and heading; try/except wrapper (safe no-op on any error). - Fix 20-override: _dedupe_duplicate_rows_in_da3_table now uses the PDF text-layer frequency map (_tl_freq_context) as its cap so legitimate repeated transactions (e.g. 6x Temporary Credit Reversal on the same date/amount) are preserved rather than being collapsed to 2. Keys absent from the text layer still cap at 2 (OCR stutter behaviour unchanged). Safe no-op for scanned PDFs with no text layer. - Fix 30: Drop small DA3 fragments (1-5 rows) that are a loose (date+amount) subset of an earlier larger DA3 table — catches OCR-truncated description duplicates such as the BoA service-fee page where a VIDDYOZE -1.11 fragment table appears after the full service-fee table with a longer description string. """ # Patch asyncio first (before Gradio imports it) to suppress Python 3.13 cleanup noise import asyncio try: _orig_close = asyncio.BaseEventLoop.close def _safe_close(self): try: _orig_close(self) except (ValueError, OSError): pass asyncio.BaseEventLoop.close = _safe_close except Exception: pass import logging import os import re import html import tempfile from typing import List, Tuple from collections import Counter, defaultdict from html.parser import HTMLParser import yaml import gradio as gr import glmocr log = logging.getLogger("glmocr_app") logging.basicConfig(level=logging.INFO) GLMOCR_BASE = os.path.dirname(glmocr.__file__) CONFIG_PATH = os.path.join(GLMOCR_BASE, "config.yaml") FORMATTER_PATH = os.path.join(GLMOCR_BASE, "postprocess", "result_formatter.py") # ============================================================ # HARD-CODED SETTINGS (edit these numbers to tune quality/speed) # ============================================================ GLMOCR_API_KEY = "cee1d52dd91a4ab591b3f6e105f8ad89.LgbQTECuzX0zrito" # Higher = better OCR for small/right-aligned digits; slower RENDER_SCALE = 2.2 # try 2.5 if right-side numbers are missed # Padding to protect columns near edges (Balance is often right-most) PAD_LEFT_FRAC = 0.02 PAD_RIGHT_FRAC = 0.06 # try 0.10 if right-most balances are missing PAD_TOP_FRAC = 0.01 PAD_BOTTOM_FRAC = 0.01 ENABLE_CONTRAST = True DEFAULT_ZONE_FRAC = 0.12 PDF_HEADER_BAND_FRAC = 0.10 ENABLE_FOOTER_OCR = True # enabled — 3-step fallback with dedup guard PDF_FOOTER_BAND_FRAC = 0.88 MIN_CROP_HEIGHT = 112 MIN_CROP_PIXELS = 112 * 112 # ============================================================ _parser = None def get_parser(): global _parser if _parser is None: from glmocr import GlmOcr _parser = GlmOcr(api_key=GLMOCR_API_KEY, mode="maas") return _parser # --------------------------------------------------------------------------- # Best-effort config tweaks (safe to fail on read-only HF env) # --------------------------------------------------------------------------- try: with open(CONFIG_PATH, "r") as f: config = yaml.safe_load(f) config["pipeline"]["maas"]["enabled"] = True config["pipeline"]["maas"]["api_key"] = GLMOCR_API_KEY with open(CONFIG_PATH, "w") as f: yaml.dump(config, f, default_flow_style=False, sort_keys=False) except Exception: pass # Best-effort formatter tweak: avoid stripping header/footer labels try: with open(FORMATTER_PATH, "r") as f: source = f.read() for label in ( '"header"', "'header'", '"footer"', "'footer'", '"doc_header"', "'doc_header'", '"doc_footer"', "'doc_footer'", ): source = re.sub(r",\s*" + re.escape(label), "", source) source = re.sub(re.escape(label) + r"\s*,", "", source) source = re.sub(re.escape(label), "", source) with open(FORMATTER_PATH, "w") as f: f.write(source) except Exception: pass # -------------------------- # Header/footer helpers # -------------------------- def get_header_footer_zones(regions, norm_height=1000): if not regions: return None, None y_tops, y_bottoms = [], [] for r in regions: bbox = r.get("bbox_2d") if isinstance(r, dict) else getattr(r, "bbox_2d", None) if bbox and len(bbox) >= 4: y_tops.append(bbox[1]) y_bottoms.append(bbox[3]) if not y_tops: return None, None return min(y_tops) / norm_height, max(y_bottoms) / norm_height def extract_zone_text_pdf(pdf_path, page_num, y_start_frac, y_end_frac): try: import pymupdf as fitz doc = fitz.open(pdf_path) page = doc[page_num] h, w = page.rect.height, page.rect.width rect = fitz.Rect(0, h * y_start_frac, w, h * y_end_frac) text = page.get_text(clip=rect).strip() doc.close() return text except Exception: return "" def extract_pdf_text_in_band(pdf_path, page_num, y_start_frac, y_end_frac): try: import pymupdf as fitz doc = fitz.open(pdf_path) page = doc[page_num] h = page.rect.height y_lo = h * y_start_frac y_hi = h * y_end_frac words = page.get_text("words") doc.close() parts = [] for w in words: if len(w) >= 5: y0, y1 = float(w[1]), float(w[3]) if y0 < y_hi and y1 > y_lo: parts.append(w[4]) return " ".join(parts).strip() except Exception: return "" def ocr_zone(image_path, y_start_frac, y_end_frac): zone_name = "header" if y_end_frac < 0.5 else "footer" try: from PIL import Image img = Image.open(image_path).convert("RGB") w, h = img.size y0 = max(0, int(h * y_start_frac)) y1 = min(h, int(h * y_end_frac)) if y1 <= y0: return "" crop = img.crop((0, y0, w, y1)) cw, ch = crop.size if ch < MIN_CROP_HEIGHT or (cw * ch) < MIN_CROP_PIXELS: need_h = max(ch, MIN_CROP_HEIGHT) need_w = max(cw, 1) if (need_w * need_h) < MIN_CROP_PIXELS: need_w = max(need_w, (MIN_CROP_PIXELS + need_h - 1) // need_h) canvas = Image.new("RGB", (need_w, need_h), (255, 255, 255)) if zone_name == "header": canvas.paste(crop, (0, 0)) else: canvas.paste(crop, (0, need_h - ch)) crop = canvas fd, path = tempfile.mkstemp(suffix=".jpg") os.close(fd) try: crop.save(path, "JPEG", quality=92) parser = get_parser() out = parser.parse(path) if not isinstance(out, list): out = [out] if out and getattr(out[0], "markdown_result", None): return (out[0].markdown_result or "").strip() finally: try: os.unlink(path) except Exception: pass except Exception as e: log.warning("[%s] ocr_zone failed: %s", zone_name, e, exc_info=True) return "" def fix_account_number(hdr: str) -> str: if not hdr: return hdr if "Account Number:" in hdr and "Account Number: " not in hdr: m = re.search(r"[0-9]{5,}", hdr) if m: hdr = hdr.replace("Account Number:", "Account Number: " + m.group(0)) acct_match = re.search(r"Account Number: ([0-9]{5,})", hdr) if acct_match: acct = acct_match.group(1) if hdr.startswith(acct): hdr = hdr[len(acct):].lstrip() return hdr # -------------------------- # Table stabilization helpers # -------------------------- def close_unclosed_html(md: str) -> str: if not md: return md open_tags = re.findall(r"<(table|tbody|thead|tr|td|th)\b", md, flags=re.IGNORECASE) close_tags = re.findall(r"", md, flags=re.IGNORECASE) def count(tags, name): return sum(1 for t in tags if t.lower() == name) for tag in reversed(["td", "th", "tr", "thead", "tbody", "table"]): opened = count(open_tags, tag) closed = count(close_tags, tag) if opened > closed: md += ("" % tag) * (opened - closed) return md def looks_like_markdown_table(block: str) -> bool: lines = [ln.rstrip() for ln in block.strip().splitlines() if ln.strip()] if len(lines) < 2: return False if "|" not in lines[0]: return False sep = lines[1].replace(" ", "") return ("---" in sep) and ("|" in sep) def md_table_to_html(block: str) -> str: lines = [ln.strip() for ln in block.strip().splitlines() if ln.strip()] if len(lines) < 2: return block def split_row(row: str): row = row.strip() if row.startswith("|"): row = row[1:] if row.endswith("|"): row = row[:-1] return [p.strip() for p in row.split("|")] header = split_row(lines[0]) body_lines = [ln for ln in lines[2:] if "|" in ln] html_rows = [] html_rows.append("" + "".join(f"" for c in header) + "") for ln in body_lines: cols = split_row(ln) if len(cols) < len(header): cols += [""] * (len(header) - len(cols)) html_rows.append("" + "".join(f"" for c in cols[: len(header)]) + "") return "
{html.escape(c)}
{html.escape(c)}
\n" + "\n".join(html_rows) + "\n
" def normalize_money_glyphs(text: str) -> str: if not text: return text t = text.replace("−", "-").replace("–", "-").replace("—", "-") t = re.sub( r"\(\s*\$?\s*([0-9]{1,3}(?:,[0-9]{3})*|[0-9]+)(\.[0-9]{2})\s*\)", r"-\1\2", t, ) def o_to_zero(m): token = m.group(0) return token.replace("O", "0").replace("o", "0") t = re.sub(r"\b[0-9Oo\$,.\-]{4,}\b", o_to_zero, t) return t # ---- Generic HTML table normalizer ---- class TableGridParser(HTMLParser): """Parse a into rows of (text, colspan, rowspan).""" def __init__(self): super().__init__() self.rows = [] self._current_row = [] self._cell_text = [] self._colspan = 1 self._rowspan = 1 self._in_cell = False def handle_starttag(self, tag, attrs): if tag == "tr": self._current_row = [] elif tag in ("td", "th"): attrs_d = dict(attrs) self._colspan = max(1, int(attrs_d.get("colspan", 1))) self._rowspan = max(1, int(attrs_d.get("rowspan", 1))) self._cell_text = [] self._in_cell = True def handle_endtag(self, tag): if tag in ("td", "th"): text = "".join(self._cell_text).strip().replace("\n", " ") self._current_row.append((text, self._colspan, self._rowspan)) self._in_cell = False elif tag == "tr": self.rows.append(self._current_row) def handle_data(self, data): if self._in_cell: self._cell_text.append(data) def _build_grid(rows_data): if not rows_data: return [] blocked = defaultdict(set) grid = [] for r, row_cells in enumerate(rows_data): grid.append([]) col = 0 for content, C, R in row_cells: while col in blocked[r]: grid[r].append("") col += 1 for k in range(C): grid[r].append(content if k == 0 else "") for k in range(1, R): blocked[r + k].add(col) col += C max_cols = max(len(row) for row in grid) if grid else 0 for row in grid: while len(row) < max_cols: row.append("") return grid def _grid_to_html(grid): if not grid: return "" lines = ["
"] for r, row in enumerate(grid): lines.append("") tag = "th" if r == 0 else "td" for cell in row: escaped = ( (cell or "") .replace("&", "&") .replace("<", "<") .replace(">", ">") .replace('"', """) ) lines.append(f"<{tag}>{escaped}") lines.append("") lines.append("
") return "\n".join(lines) def _is_amount_like(s: str) -> bool: if not s: return False return re.fullmatch(r"\$?-?\d{1,3}(?:,\d{3})*(?:\.\d{2})?", s.strip()) is not None def _drop_truly_empty_columns(grid): if not grid or len(grid) < 1: return grid header = [str(c or "").strip() for c in grid[0]] ncols = len(header) if ncols <= 1: return grid body = grid[1:] keep = [True] * ncols for i in range(ncols): if header[i] != "": continue total = 0 non_empty = 0 for row in body: if i >= len(row): continue total += 1 if str(row[i] or "").strip(): non_empty += 1 if total > 0 and (non_empty / total) <= 0.05: keep[i] = False if all(keep): return grid new_grid = [] for row in grid: new_grid.append([cell for idx, cell in enumerate(row) if idx < len(keep) and keep[idx]]) return new_grid def _merge_blank_header_text_columns(grid): if not grid or len(grid) < 2: return grid header = [str(c or "").strip() for c in grid[0]] ncols = len(header) body = grid[1:] blank_cols = [i for i, h in enumerate(header) if h == ""] if not blank_cols: return grid keep = [True] * ncols for i in blank_cols: j = i - 1 while j >= 0 and header[j] == "": j -= 1 if j < 0: continue total = 0 non_empty = 0 texty = 0 for row in body: if i >= len(row) or j >= len(row): continue v = str(row[i] or "").strip() total += 1 if v: non_empty += 1 if not _is_amount_like(v): texty += 1 if total == 0: continue non_empty_ratio = non_empty / total texty_ratio = (texty / non_empty) if non_empty else 0.0 if non_empty_ratio >= 0.55 and texty_ratio >= 0.70: for r in range(1, len(grid)): row = grid[r] if i >= len(row) or j >= len(row): continue left = str(row[j] or "").strip() right = str(row[i] or "").strip() if right: row[j] = (left + " " + right).strip() if left else right keep[i] = False if all(keep): return grid new_grid = [] for row in grid: new_grid.append([cell for idx, cell in enumerate(row) if idx < len(keep) and keep[idx]]) return new_grid # -------------------------- # Shared keyword definitions # -------------------------- _MONEY_RE = re.compile( r"^(?:[\$£€¥]|Rs\.?|INR|PKR)?\s*-?\s?\d{1,3}(?:,\d{3})*(?:\.\d{1,4})?$", re.IGNORECASE, ) _HEADER_KW_PATTERNS = [ r"DATE", r"POSTING\s+DATE", r"VALUE\s+DATE", r"TXN\s+DATE", r"TRANSACTION\s+DATE", r"ENTRY\s+DATE", r"EFFECTIVE\s+DATE", r"TRANSACTION(?:\s+(?:ID|TYPE|NO|NUMBER))?", r"TXN(?:\s+(?:ID|NO|TYPE))?", r"REF(?:ERENCE)?(?:\s*(?:NO|NUM|NUMBER))?", r"CHEQUE(?:\s*(?:NO|NUMBER))?", r"CHQ(?:\s*(?:NO|NUMBER))?", r"VOUCHER(?:\s*(?:NO|NUMBER))?", r"SR\.?\s*NO\.?", r"SERIAL(?:\s*(?:NO|NUMBER))?", r"DESCRIPTION", r"DETAILS?", r"PARTICULARS?(?:\s+OF\s+TRANSACTION)?", r"NARRATION", r"REMARKS?", r"NOTES?", r"DEBIT", r"DEBITS?", r"DR\.?", r"WITHDRAWALS?", r"PAID\s+OUT", r"MONEY\s+OUT", r"CREDIT", r"CREDITS?", r"CR\.?", r"DEPOSITS?", r"PAID\s+IN", r"MONEY\s+IN", r"AMOUNT", r"BALANCE", r"BAL\.?", r"RUNNING\s+BALANCE", r"RUNNING\s+BAL\.?", r"AVAILABLE\s+BALANCE", r"AVAILABLE\s+BAL\.?", r"AVAIL\.?\s+BAL\.?", r"LEDGER\s+BALANCE", r"LEDGER\s+BAL\.?", r"CLOSING\s+BALANCE", r"CLOSING\s+BAL\.?", r"OPENING\s+BALANCE", r"OPENING\s+BAL\.?", ] _HEADER_KW_RES = [ re.compile(r"^(" + p + r")(.*)", re.IGNORECASE | re.DOTALL) for p in _HEADER_KW_PATTERNS ] _HEADER_KW_EXACT_RE = re.compile( r"^(?:" + r"|".join(_HEADER_KW_PATTERNS) + r")$", re.IGNORECASE, ) _HEADER_ROW_KEYWORD_THRESHOLD = 0.5 # -------------------------- # Fix 1: Fused-header recovery # -------------------------- def _split_keyword_remainder(cell_text: str): s = cell_text.strip() for pattern in _HEADER_KW_RES: m = pattern.match(s) if m: keyword = m.group(1).strip() remainder = m.group(2).strip() return keyword, remainder return s, "" def _extract_fused_header_artifacts(header_row): if not header_row: return list(header_row), None ncols = len(header_row) cleaned = list(header_row) recovered = [""] * ncols text_fused = [] money_fused = [] for idx, cell in enumerate(header_row): cell_s = str(cell or "").strip() if not cell_s: continue keyword, remainder = _split_keyword_remainder(cell_s) if not remainder: continue remainder_no_space = remainder.replace(" ", "") if _MONEY_RE.match(remainder_no_space): cleaned[idx] = keyword recovered[idx] = remainder money_fused.append(idx) elif not _HEADER_KW_EXACT_RE.match(remainder.split()[0] if remainder.split() else ""): cleaned[idx] = keyword recovered[idx] = remainder text_fused.append(idx) if text_fused and money_fused: return cleaned, recovered return list(header_row), None def _clean_header_artifacts(grid): if not grid or not grid[0]: return grid, None cleaned_header, recovered_row = _extract_fused_header_artifacts(grid[0]) grid[0] = cleaned_header return grid, recovered_row # -------------------------- # Fix 2: Misplaced header row promotion # -------------------------- def _row_keyword_score(row): non_empty = [str(c or "").strip() for c in row if str(c or "").strip()] if not non_empty: return 0.0 keyword_hits = 0 for cell in non_empty: if _HEADER_KW_EXACT_RE.match(cell): keyword_hits += 1 continue parts = cell.split() if all(_HEADER_KW_EXACT_RE.match(p) for p in parts) and len(parts) > 1: keyword_hits += 1 continue return keyword_hits / len(non_empty) _BALANCE_INFO_RE = re.compile( r"(beginning|ending|opening|closing)\s+balance" r"|account\s*#?\s*\d{5,}" r"|\b\d{7,}\b" r"|\$\d{1,3}(?:,\d{3})*\.\d{2}", re.IGNORECASE, ) def _header_contains_balance_info(header_row): for cell in header_row: if _BALANCE_INFO_RE.search(str(cell or "")): return True return False def _promote_misplaced_header_row(grid): if not grid or len(grid) < 2: return grid current_header_score = _row_keyword_score(grid[0]) if current_header_score >= _HEADER_ROW_KEYWORD_THRESHOLD: return grid candidate_idx = None candidate_score = 0.0 for i in range(1, min(len(grid), 6)): score = _row_keyword_score(grid[i]) if score >= _HEADER_ROW_KEYWORD_THRESHOLD and score > candidate_score: candidate_score = score candidate_idx = i if candidate_idx is None: return grid candidate_row = grid[candidate_idx] expanded_header = [] for cell in candidate_row: cell_s = str(cell or "").strip() parts = cell_s.split() if len(parts) > 1 and all(_HEADER_KW_EXACT_RE.match(p) for p in parts): expanded_header.extend(parts) else: expanded_header.append(cell_s) new_ncols = len(expanded_header) col_expansion = new_ncols - len(candidate_row) metadata_row = None if _header_contains_balance_info(grid[0]): meta_cells = [str(c or "").strip() for c in grid[0]] meta_text = " ".join(c for c in meta_cells if c).strip() if meta_text: metadata_row = [meta_text] + [""] * (new_ncols - 1) new_data_rows = [] for row in grid[candidate_idx + 1:]: if col_expansion > 0 and row: first_cell = str(row[0] or "").strip() date_match = re.match(r"^(\d{1,2}/\d{1,2})\s+(.*)", first_cell, re.DOTALL) if date_match and col_expansion == 1: new_row = [date_match.group(1), date_match.group(2).strip()] + list(row[1:]) else: new_row = list(row) + [""] * col_expansion new_data_rows.append(new_row) else: new_data_rows.append(list(row)) def pad(row, n): r = list(row) while len(r) < n: r.append("") return r[:n] new_grid = [pad(expanded_header, new_ncols)] if metadata_row is not None: new_grid.append(pad(metadata_row, new_ncols)) for row in new_data_rows: new_grid.append(pad(row, new_ncols)) return new_grid # -------------------------- # Fix 3: Fused key-value rows in summary sections # -------------------------- _FUSED_KV_RE = re.compile( r"^(.+?)(-?\d{1,3}(?:,\d{3})*(?:\.\d+)?%?)$", re.DOTALL, ) def _fix_fused_keyvalue_rows(grid): if not grid or len(grid) < 2: return grid _DOLLAR_AMOUNT_RE = re.compile(r"\$\s*\d{1,3}(?:,\d{3})*\.\d{2}") _LONG_NUMBER_RE = re.compile(r"\d{7,}") ncols = len(grid[0]) new_grid = [grid[0]] for row in grid[1:]: cells = [str(c or "").strip() for c in row] if not cells: new_grid.append(row) continue first = cells[0] rest_empty = all(c == "" for c in cells[1:]) if not rest_empty or not first: new_grid.append(row) continue if _DOLLAR_AMOUNT_RE.search(first): new_grid.append(row) continue if _LONG_NUMBER_RE.search(first): new_grid.append(row) continue if re.search(r"#\s*[\dX]", first): new_grid.append(row) continue m = _FUSED_KV_RE.match(first) if not m: new_grid.append(row) continue label_raw = m.group(1) value = m.group(2).strip() if label_raw != label_raw.rstrip(): new_grid.append(row) continue label = label_raw.strip() if not label or not value: new_grid.append(row) continue new_row = [""] * ncols new_row[0] = label new_row[ncols - 1] = value new_grid.append(new_row) return new_grid # -------------------------- # Fix 4: PDF text-layer row-count patch # -------------------------- def _extract_textlayer_rows(pdf_path: str, page_num: int): try: import pymupdf as fitz doc = fitz.open(pdf_path) page = doc[page_num] words = page.get_text("words") doc.close() if not words: return [] lines_by_y = {} for w in words: x0, y0, word = float(w[0]), float(w[1]), w[4] bucket = None for existing_y in lines_by_y: if abs(existing_y - y0) <= 5: bucket = existing_y break if bucket is None: bucket = y0 lines_by_y.setdefault(bucket, []).append((x0, word)) sorted_lines = [] for y in sorted(lines_by_y): line_words = sorted(lines_by_y[y], key=lambda t: t[0]) sorted_lines.append([w for _, w in line_words]) _MONTHS = r"(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)" date_re = re.compile( r"^\d{1,2}/\d{2}(?:/\d{2,4})?$" r"|^\d{1,2}-\d{2}(?:-\d{2,4})?$" r"|^\d{4}-\d{2}-\d{2}$" r"|^" + _MONTHS + r"$", re.IGNORECASE, ) month_re = re.compile(r"^" + _MONTHS + r"$", re.IGNORECASE) day_re = re.compile(r"^\d{1,2}$") amount_re = re.compile( r"^-?\$?\d{1,3}(?:,\d{3})*(?:\.\d{1,4})?$" r"|^\$-?\d{1,3}(?:,\d{3})*(?:\.\d{1,4})?$" r"|^-\s+\$\d{1,3}(?:,\d{3})*(?:\.\d{1,4})?$" ) rows = [] for line in sorted_lines: if len(line) < 2: continue date_str = None desc_start = None if date_re.match(line[0]): if month_re.match(line[0]) and len(line) > 1 and day_re.match(line[1]): date_str = line[0] + " " + line[1] desc_start = 2 else: date_str = line[0] desc_start = 1 else: continue if desc_start is None or desc_start >= len(line): continue remaining = line[desc_start:] post_date_str = "" if remaining and month_re.match(remaining[0]): if len(remaining) > 1 and day_re.match(remaining[1]): post_date_str = remaining[0] + " " + remaining[1] remaining = remaining[2:] elif len(remaining) > 0: remaining = remaining[1:] elif remaining and date_re.match(remaining[0]) and not month_re.match(remaining[0]): post_date_str = remaining[0] remaining = remaining[1:] if len(remaining) < 2: continue if (len(remaining) >= 2 and remaining[-2] == "-" and remaining[-1].startswith("$")): amount_candidate = remaining[-2] + " " + remaining[-1] desc_tokens = remaining[:-2] else: amount_candidate = remaining[-1] desc_tokens = remaining[:-1] if not amount_re.match(amount_candidate): continue desc = " ".join(desc_tokens).strip() if not desc: continue if not re.search(r"[A-Za-z]", desc): continue rows.append({"date": date_str, "post_date": post_date_str, "desc": desc, "amount": amount_candidate}) return rows if len(rows) >= 2 else [] except Exception as e: log.debug("_extract_textlayer_rows failed (page %d): %s", page_num, e) return [] def _jb_extract_checks_fallback(text_layer: str) -> List[Tuple[str, str, str]]: if not text_layer: return [] m = re.search( r"Checks\s*\([^\n]*\n(.*?)(?=^Daily\s+Account\s+Balance\s*$)", text_layer, re.DOTALL | re.MULTILINE | re.IGNORECASE, ) if not m: return [] chunk = m.group(1) chunk = re.sub( r"^\s*Date\s+Number\s+Amount\s*$", "", chunk, flags=re.MULTILINE | re.IGNORECASE, ) _triplet = re.compile( r"(\d{2}-\d{2})\s+(\*?\d+)\s+(\d{1,3}(?:,\d{3})*\.\d{2})" ) out: List[Tuple[str, str, str]] = [] seen = set() for t in _triplet.findall(chunk): if t not in seen: seen.add(t) out.append(t) return out def _extract_jb_summary_sections(text_layer: str, needs_checks: bool = True, needs_dab: bool = True) -> str: if not text_layer: return "" _DATE_RE = re.compile(r"^\d{2}-\d{2}$") _MONEY_RE = re.compile(r"^\d{1,3}(?:,\d{3})*\.\d{2}$") _CHECK_RE = re.compile(r"^\*?\d+$") lines = [l.strip() for l in text_layer.splitlines()] output = [] if needs_checks: checks = [] i = 0 while i < len(lines): if lines[i].lower().startswith("checks"): i += 1 while i < len(lines): if "daily account balance" in lines[i].lower(): break if lines[i] == "Date": i += 1 dates = [] while i < len(lines) and _DATE_RE.match(lines[i]): dates.append(lines[i]); i += 1 while i < len(lines) and lines[i] != "Number": i += 1 i += 1 numbers = [] while i < len(lines): if _CHECK_RE.match(lines[i]): numbers.append(lines[i]); i += 1 elif lines[i] == "": i += 1 else: break while i < len(lines) and lines[i] != "Amount": i += 1 i += 1 amounts = [] while i < len(lines): if _MONEY_RE.match(lines[i]): amounts.append(lines[i]); i += 1 elif lines[i] == "": i += 1 else: break for d, n, a in zip(dates, numbers, amounts): checks.append((d, n, a)) else: i += 1 break i += 1 if not checks and re.search(r"\bChecks\s*\(", text_layer, re.I): checks = _jb_extract_checks_fallback(text_layer) if checks: rows = "\n".join( "" + d + "" + n + "" + a + "" for d, n, a in checks ) output.append("Checks") output.append( '\n\n' + rows + "\n
DateNumberAmount
" ) if needs_dab: balances = [] i = 0 while i < len(lines): if "daily account balance" in lines[i].lower(): i += 1 while i < len(lines): if lines[i] == "Date": i += 1 dates = [] while i < len(lines) and _DATE_RE.match(lines[i]): dates.append(lines[i]); i += 1 while i < len(lines) and lines[i] != "Balance": i += 1 i += 1 bals = [] while i < len(lines): if _MONEY_RE.match(lines[i]): bals.append(lines[i]); i += 1 elif lines[i] == "": i += 1 else: break for d, b in zip(dates, bals): balances.append((d, b)) else: i += 1 break i += 1 if balances: rows = "\n".join( "" + d + "" + b + "" for d, b in balances ) output.append("Daily Account Balance") output.append( "\n\n" + rows + "\n
DateBalance
" ) return "\n\n".join(output) def _patch_ocr_with_textlayer(page_md: str, pdf_path: str, page_num: int) -> str: if not page_md or "]*>.*?", re.DOTALL | re.IGNORECASE) result = page_md for tbl_match in table_pattern.finditer(page_md): table_html = tbl_match.group(0) p = TableGridParser() p.feed(table_html) grid = _build_grid(p.rows) if len(grid) < 2: continue header = [str(c or "").strip().upper() for c in grid[0]] date_col = desc_col = amt_col = None for ci, h in enumerate(header): if re.search(r"\bDATE\b", h) and date_col is None: date_col = ci elif re.search(r"\b(DESCRIPTION|DETAILS?|NARRATION|PARTICULARS?)\b", h) and desc_col is None: desc_col = ci elif re.search(r"\b(AMOUNT|DEBIT|CREDIT|DR|CR)\b", h) and amt_col is None: amt_col = ci if date_col is None or desc_col is None or amt_col is None: continue ocr_freq = {} ocr_rows_indexed = [] for ri, row in enumerate(grid[1:], start=1): cells = [str(c or "").strip() for c in row] if len(cells) <= max(date_col, desc_col, amt_col): continue d = _norm(cells[date_col]) desc = _norm(cells[desc_col]) amt = _norm(cells[amt_col]) if not d or not desc: continue key = (d, desc, amt) ocr_freq[key] = ocr_freq.get(key, 0) + 1 ocr_rows_indexed.append((ri, key)) overlap = set(ocr_freq.keys()) & set(tl_freq.keys()) _ocr_dates = [ _norm(str(row[date_col] or "")) for row in grid[1:] if len(row) > date_col and str(row[date_col] or "").strip() ] _tl_dates = [_norm(r["date"]) for r in tl_rows] _slash_re = re.compile(r"^\d{1,2}/\d{2}") _hyphen_re = re.compile(r"^\d{1,2}-\d{2}") def _date_fmt(dates): if any(_slash_re.match(d) for d in dates): return "slash" if any(_hyphen_re.match(d) for d in dates): return "hyphen" return "other" ocr_fmt = _date_fmt(_ocr_dates) tl_fmt = _date_fmt(_tl_dates) date_fmt_match = (ocr_fmt == tl_fmt) or "other" in (ocr_fmt, tl_fmt) to_inject = {} to_inject_new = {} for key in tl_freq: ocr_count = ocr_freq.get(key, 0) tl_count = tl_freq[key] if tl_count > ocr_count: missing = tl_count - ocr_count if ocr_count > 0: if overlap: to_inject[key] = missing else: if date_fmt_match: tl_row_data = next( (r for r in tl_rows if (_norm(r["date"]), _norm(r["desc"]), _norm(r["amount"])) == key), None ) if tl_row_data: to_inject_new[key] = (missing, tl_row_data) if not to_inject and not to_inject_new: continue new_grid = [grid[0]] for ri, row in enumerate(grid[1:], start=1): new_grid.append(row) cells = [str(c or "").strip() for c in row] if len(cells) <= max(date_col, desc_col, amt_col): continue d = _norm(cells[date_col]) desc = _norm(cells[desc_col]) amt = _norm(cells[amt_col]) key = (d, desc, amt) if key in to_inject and to_inject[key] > 0: last_occ = max(idx for idx, k in ocr_rows_indexed if k == key) if ri == last_occ: for _ in range(to_inject[key]): new_grid.append(list(row)) to_inject[key] = 0 new_table_html = _grid_to_html(new_grid) result = result.replace(table_html, new_table_html, 1) if to_inject_new: ncols = len(grid[0]) header_row = grid[0] post_date_col = None for ci, h in enumerate(header_row): hh = str(h or "").strip().upper() if "POST" in hh and "DATE" in hh: post_date_col = ci break new_rows = [] for key, (count, tl_row_data) in to_inject_new.items(): for _ in range(count): new_row = [""] * ncols new_row[date_col] = tl_row_data["date"] new_row[desc_col] = tl_row_data["desc"] new_row[amt_col] = tl_row_data["amount"] if post_date_col is not None: new_row[post_date_col] = tl_row_data.get("post_date", "") new_rows.append(new_row) log.info( "page %d: new table row from text layer: %s %s", page_num, tl_row_data["date"], tl_row_data["desc"][:40] ) if new_rows: extra_grid = [header_row] + new_rows extra_html = "\n\n" + _grid_to_html(extra_grid) insert_pos = result.find(new_table_html) + len(new_table_html) result = result[:insert_pos] + extra_html + result[insert_pos:] if tl_rows: _has_txn_table = False for _tbl in table_pattern.finditer(result): _p2 = TableGridParser() _p2.feed(_tbl.group(0)) _g2 = _build_grid(_p2.rows) if len(_g2) < 2: continue _h2 = [str(c or "").strip().upper() for c in _g2[0]] if (any(re.search(r"\bDATE\b", x) for x in _h2) and any(re.search(r"\b(DESCRIPTION|DETAILS?|NARRATION|PARTICULARS?)\b", x) for x in _h2)): _has_txn_table = True break if not _has_txn_table: _has_post = any(r.get("post_date") for r in tl_rows) if _has_post: _chdr = ["Date", "Post Date", "Transaction Description", "Amount"] _di2, _pi2, _xi2, _ai2 = 0, 1, 2, 3 else: _chdr = ["Date", "Transaction Description", "Amount"] _di2, _xi2, _ai2 = 0, 1, 2 _cnew_rows = [] for r in tl_rows: _crow = [""] * len(_chdr) _crow[_di2] = r["date"] _crow[_xi2] = r["desc"] _crow[_ai2] = r["amount"] if _has_post: _crow[_pi2] = r.get("post_date", "") _cnew_rows.append(_crow) result = _grid_to_html([_chdr] + _cnew_rows) + "\n\n" + result log.info("page %d: Case C — new table (%d rows) from text layer", page_num, len(_cnew_rows)) return result except Exception as e: log.warning("_patch_ocr_with_textlayer failed (page %d): %s", page_num, e) _tl_freq_context.clear() return page_md # -------------------------- # Fix 5: Mid-table separator row reconstruction # -------------------------- _DATE_RE = re.compile(r"^\d{1,2}/\d{2}(?:/\d{2,4})?$") _SPLIT_AMOUNT_RE = re.compile( r"^-?\$?\d{1,3}(?:,\d{3})*(?:\.\d{1,4})?$|^\$-?\d{1,3}(?:,\d{3})*(?:\.\d{1,4})?$" ) def _reconstruct_separator_rows(grid): if not grid or len(grid) < 2: return grid ncols = len(grid[0]) if ncols < 2: return grid new_grid = [grid[0]] for row in grid[1:]: cells = [str(c or "").strip() for c in row] while len(cells) < ncols: cells.append("") non_empty = [(i, c) for i, c in enumerate(cells) if c] if len(non_empty) == 0: new_grid.append(row) continue first_idx, first_val = non_empty[0] if _DATE_RE.match(first_val): new_grid.append(row) continue if not re.search(r"[A-Za-z]", first_val): new_grid.append(row) continue if len(non_empty) == 1: new_row = [""] * ncols new_row[0] = first_val new_grid.append(new_row) continue trailing = [val for _, val in non_empty[1:]] all_trailing_are_digit_fragments = all( re.fullmatch(r"\d{1,4}", v) for v in trailing ) if not all_trailing_are_digit_fragments: new_grid.append(row) continue reconstructed = "".join(val for _, val in non_empty).strip() new_row = [""] * ncols new_row[0] = reconstructed new_grid.append(new_row) return new_grid # -------------------------- # Normalizer entry point # -------------------------- def _merge_split_rows(grid): if not grid or len(grid) < 3: return grid ncols = len(grid[0]) if ncols < 3: return grid header = [str(c or "").strip().upper() for c in grid[0]] date_col = desc_col = bal_col = None debit_cols = [] credit_cols = [] for ci, h in enumerate(header): if re.search(r"\bDATE\b", h) and date_col is None: date_col = ci if re.search(r"\b(DESCRIPTION|DETAILS?|NARRATION|PARTICULARS?|TRANSACTION)\b", h) and desc_col is None: desc_col = ci if re.search(r"\bBALANCE\b", h) and bal_col is None: bal_col = ci if re.search(r"\b(DEBIT|DEBITS|DR|WITHDRAWAL|SUBTRACTIONS?)\b", h): debit_cols.append(ci) if re.search(r"\b(CREDIT|CREDITS|CR|DEPOSIT|ADDITIONS?)\b", h): credit_cols.append(ci) if date_col is None or desc_col is None: return grid if not debit_cols and not credit_cols: return grid money_cols = debit_cols + credit_cols _MONEY_RE2 = re.compile(r"^-?\$?\d{1,3}(?:,\d{3})*\.\d{1,4}$") _DATE_RE2 = re.compile(r"^\d{1,2}/\d{2}(?:/\d{2,4})?$|^\d{1,2}-\d{2}(?:-\d{2,4})?$") _FRAGMENT_RE = re.compile(r"^[^\s\.]{1,6}$") new_grid = [grid[0]] i = 1 while i < len(grid): row = [str(c or "").strip() for c in grid[i]] while len(row) < ncols: row.append("") has_desc = bool(row[desc_col]) no_date = not row[date_col] no_money = all(not row[c] for c in money_cols) bal_is_frag = (bal_col is not None and (_FRAGMENT_RE.match(row[bal_col]) or not row[bal_col])) if has_desc and no_date and no_money and bal_is_frag and (i + 1) < len(grid): next_row = [str(c or "").strip() for c in grid[i + 1]] while len(next_row) < ncols: next_row.append("") next_has_date = bool(next_row[date_col]) and _DATE_RE2.match(next_row[date_col]) next_has_money = any(bool(next_row[c]) for c in money_cols) if next_has_date and next_has_money: merged = list(next_row) if next_row[desc_col]: merged[desc_col] = row[desc_col] + " " + next_row[desc_col] else: merged[desc_col] = row[desc_col] new_grid.append(merged) i += 2 continue new_grid.append(row) i += 1 return new_grid def _extract_fused_desc_amount(grid): if not grid or len(grid) < 2: return grid ncols = len(grid[0]) header = [str(c or "").strip().upper() for c in grid[0]] date_col = desc_col = None debit_cols = [] credit_cols = [] for ci, h in enumerate(header): if re.search(r"\bDATE\b", h) and date_col is None: date_col = ci if re.search(r"\b(DESCRIPTION|DETAILS?|NARRATION|PARTICULARS?|TRANSACTION)\b", h) and desc_col is None: desc_col = ci if re.search(r"\b(DEBIT|DEBITS|DR|WITHDRAWAL|SUBTRACTIONS?)\b", h): debit_cols.append(ci) if re.search(r"\b(CREDIT|CREDITS|CR|DEPOSIT|ADDITIONS?)\b", h): credit_cols.append(ci) if date_col is None or desc_col is None or not debit_cols: return grid money_cols = debit_cols + credit_cols _TRAILING_AMT = re.compile(r"^(.+?)\s+(-?\d{1,3}(?:,\d{3})*\.\d{2})$") debit_col = debit_cols[0] new_grid = [grid[0]] for row in grid[1:]: cells = [str(c or "").strip() for c in row] while len(cells) < ncols: cells.append("") desc = cells[desc_col] no_money = all(not cells[c] for c in money_cols) has_date = bool(cells[date_col]) if desc and no_money and has_date: m = _TRAILING_AMT.match(desc) if m: cells = list(cells) cells[desc_col] = m.group(1).strip() cells[debit_col] = m.group(2) new_grid.append(cells) continue new_grid.append(cells) return new_grid def _is_junk_table(grid): if not grid or len(grid) < 2: return False ncols = len(grid[0]) if ncols != 1: return False header_text = str(grid[0][0] or "").strip().upper() if "CUSTOMER INFORMATION" in header_text: data_rows = grid[1:] if all(len(str(r[0] or "").strip()) <= 10 for r in data_rows): return True return False def _split_fused_multicolumn_header(grid): if not grid or len(grid) < 2: return grid ncols = len(grid[0]) if ncols < 2: return grid first_header = str(grid[0][0] or "").strip() parts = first_header.split() kw_hits = [p for p in parts if _HEADER_KW_EXACT_RE.match(p)] if len(kw_hits) < 2: return grid rest_headers = [str(c or "").strip() for c in grid[0][1:]] if not all(_HEADER_KW_EXACT_RE.match(h) for h in rest_headers if h): return grid new_header_cols = [] non_kw_parts = [] for p in parts: if _HEADER_KW_EXACT_RE.match(p): if non_kw_parts: new_header_cols.append(" ".join(non_kw_parts)) non_kw_parts = [] new_header_cols.append(p) else: non_kw_parts.append(p) if non_kw_parts: new_header_cols.append(" ".join(non_kw_parts)) new_header = new_header_cols + rest_headers new_ncols = len(new_header) extra_cols = new_ncols - ncols _DATE_COL_RE = re.compile(r"\bDATE\b", re.IGNORECASE) _CARD_COL_RE = re.compile(r"\bCARD\b", re.IGNORECASE) _DESC_COL_RE = re.compile(r"\b(DESCRIPTION|DETAILS?|NARRATION|PARTICULARS?)\b", re.IGNORECASE) _MONEY_COL_RE = re.compile(r"\b(DEPOSIT|WITHDRAWAL|DEBIT|CREDIT|AMOUNT|ADDITION|SUBTRACTION)S?\b", re.IGNORECASE) new_date_col = next((i for i,h in enumerate(new_header) if _DATE_COL_RE.search(h)), None) new_desc_col = next((i for i,h in enumerate(new_header) if _DESC_COL_RE.search(h)), None) new_card_col = next((i for i,h in enumerate(new_header) if _CARD_COL_RE.search(h)), None) if new_date_col is None or new_desc_col is None: return grid _DATE_PREFIX = re.compile(r"^(\d{1,2}/\d{2}(?:/\d{2,4})?)\s+(.*)", re.DOTALL) _CARD_SUFFIX = re.compile(r"^(.*?)\s+(\d{4})$", re.DOTALL) def pad(row, n): r = list(row) while len(r) < n: r.append("") return r[:n] new_grid = [pad(new_header, new_ncols)] for row in grid[1:]: cells = [str(c or "").strip() for c in row] while len(cells) < ncols: cells.append("") fused_cell = cells[0] rest_cells = cells[1:] date_val = "" desc_val = fused_cell card_val = "" dm = _DATE_PREFIX.match(fused_cell) if dm: date_val = dm.group(1) desc_val = dm.group(2).strip() if new_card_col is not None: cm = _CARD_SUFFIX.match(desc_val) if cm: desc_val = cm.group(1).strip() card_val = cm.group(2) new_row = [""] * new_ncols new_row[new_date_col] = date_val new_row[new_desc_col] = desc_val if new_card_col is not None: new_row[new_card_col] = card_val money_col_indices = [i for i,h in enumerate(new_header) if _MONEY_COL_RE.search(h)] for mi, ri in enumerate(range(len(rest_cells))): if mi < len(money_col_indices): new_row[money_col_indices[mi]] = rest_cells[ri] new_grid.append(new_row) return new_grid def _normalize_daily_balance_table(grid): if not grid or len(grid) < 2: return grid header = [str(c or "").strip().upper() for c in grid[0]] ncols = len(header) if ncols < 4 or ncols % 2 != 0: return grid half = ncols // 2 date_half = header[:half] bal_half = header[half:] if not all(h == "DATE" for h in date_half): return grid if not all(h == "BALANCE" for h in bal_half): return grid new_header = [] for i in range(half): new_header.append("DATE") new_header.append("BALANCE") new_rows = [] _DATE_RE3 = re.compile(r"^\d{1,2}/\d{2}(?:/\d{2,4})?$") for row in grid[1:]: cells = [str(c or "").strip() for c in row] while len(cells) < ncols: cells.append("") date_cells = [cells[0]] bal_cells = cells[1:] first = date_cells[0] tokens = re.split(r"[\n\r\s]+", first) date_tokens = [t.strip() for t in tokens if t.strip() and _DATE_RE3.match(t.strip())] if len(date_tokens) >= 2: stacked = date_tokens else: stacked = date_cells new_row = [""] * ncols for i, date_val in enumerate(stacked): if not _DATE_RE3.match(date_val): continue bal_val = bal_cells[i] if i < len(bal_cells) else "" new_row[i * 2] = date_val new_row[i * 2 + 1] = bal_val new_rows.append(new_row) if not new_rows: return grid return [new_header] + new_rows def _normalize_checks_paid_table(grid): if not grid or len(grid) < 2: return grid ncols = len(grid[0]) if ncols < 2: return grid first_h = str(grid[0][0] or "").strip() _CHECKS_HDR = re.compile(r"^(DATE\s+CHECK\s+#\s*)+$", re.IGNORECASE) if not _CHECKS_HDR.match(first_h): return grid rest_h = [str(c or "").strip().upper() for c in grid[0][1:]] if not all(h in ("AMOUNT", "") for h in rest_h if h): return grid n_groups = len(re.findall(r"\bDATE\b", first_h, re.IGNORECASE)) if n_groups < 1: return grid new_header = [] for _ in range(n_groups): new_header.extend(["DATE", "CHECK #", "AMOUNT"]) _DATE_RE = re.compile(r"^\d{1,2}/\d{2}$") new_rows = [new_header] for row in grid[1:]: cells = [str(c or "").strip() for c in row] while len(cells) < ncols: cells.append("") fused_data = cells[0] amounts = cells[1:] tokens = fused_data.split() groups = [] current = [] for tok in tokens: if _DATE_RE.match(tok) and current: groups.append(current) current = [tok] else: current.append(tok) if current and current[0]: groups.append(current) new_row = [""] * len(new_header) for i, grp in enumerate(groups): if i >= n_groups: break new_row[i * 3] = grp[0] new_row[i * 3 + 1] = " ".join(grp[1:]).strip() new_row[i * 3 + 2] = amounts[i] if i < len(amounts) else "" new_rows.append(new_row) return new_rows _JB_SECTION_HDR = re.compile( r"^(Deposits?|Withdrawals?|Checks?(?:\s+Paid)?|Daily\s+Account\s+Balance)" r"\s*(?:\(cont\.?\))?\s*$", re.IGNORECASE, ) _JB_COL_HDR = re.compile( r"^Date\s+(Description|Number)\s+Amount\s*$" r"|^Date\s+(?:Number\s+)?(?:Balance|Amount)\s*$", re.IGNORECASE, ) _JB_DATE_DD = re.compile(r"^\d{2}-\d{2}\s+") _JB_TXN_ROW = re.compile(r"^(\d{2}-\d{2})\s+(.+?)\s+(-?\d{1,3}(?:,\d{3})*\.\d{2})\s*$") _JB_BAL_ROW = re.compile(r"^(\d{2}-\d{2})\s+(\d{1,3}(?:,\d{3})*\.\d{2})\s*$") def _jb_rows_to_html(header_cols, rows): th = "".join("" + h + "" for h in header_cols) parts = ["", "" + th + ""] for row in rows: td = "".join("" for h in header_cols) parts.append("" + td + "") parts.append("
" + str(row.get(h, "")) + "
") return "\n".join(parts) def _jb_section_has_table(lines, start, end): return any(" str: lines = text.splitlines() out = [] i = 0 while i < len(lines): line = lines[i].strip() if not _JB_SECTION_HDR.match(line): out.append(lines[i]) i += 1 continue section_title = line j = i + 1 while j < len(lines) and not lines[j].strip(): j += 1 if j >= len(lines) or not _JB_COL_HDR.match(lines[j].strip()): out.append(lines[i]) i += 1 continue col_hdr_line = lines[j].strip() data_start = j + 1 first_data_idx = _find_first_data_row(lines, data_start) if first_data_idx is None or not _JB_DATE_DD.match(lines[first_data_idx].strip()): out.append(lines[i]) i += 1 continue section_end = len(lines) for k in range(data_start, len(lines)): l = lines[k].strip() if (l != section_title and _JB_SECTION_HDR.match(l) or l.startswith("---page-separator") or l.startswith("Page:")): section_end = k break if _jb_section_has_table(lines, i, section_end): out.append(lines[i]) i += 1 continue parts_upper = col_hdr_line.upper().split() if len(parts_upper) == 2 and parts_upper[1] == "BALANCE": header_cols = ["Date", "Balance"] elif "NUMBER" in parts_upper: header_cols = ["Date", "Number", "Amount"] else: header_cols = ["Date", "Description", "Amount"] out.append(section_title) rows = [] current = None k = data_start while k < section_end: raw = lines[k] row_line = raw.strip() if not row_line: k += 1 continue if (_JB_SECTION_HDR.match(row_line) or row_line.startswith("---page-separator") or row_line.startswith("Page:")): break mb = _JB_BAL_ROW.match(row_line) if mb and header_cols == ["Date", "Balance"]: if current: rows.append(current) current = {"Date": mb.group(1), "Balance": mb.group(2)} k += 1 continue mt = _JB_TXN_ROW.match(row_line) if mt: if current: rows.append(current) desc_key = "Description" if "Description" in header_cols else "Number" current = { "Date": mt.group(1), desc_key: mt.group(2).strip(), "Amount": mt.group(3), } k += 1 continue if current and row_line: desc_key = "Description" if "Description" in header_cols else "Number" if desc_key in current: current[desc_key] += " " + row_line k += 1 continue k += 1 if current: rows.append(current) if rows: out.append(_jb_rows_to_html(header_cols, rows)) i = section_end continue return "\n".join(out) # -------------------------- # Navy Federal full-page text-layer parser (Fix NF-1) # -------------------------- def _is_nfcu_document(pdf_path: str) -> bool: try: import pymupdf as fitz doc = fitz.open(pdf_path) pages_to_check = min(2, len(doc)) text = "" for i in range(pages_to_check): text += "\n" + (doc[i].get_text() or "") doc.close() t = text.lower() return ( ("statement of account" in t) and ("access no." in t) and ("routing number" in t) and ("navy federal" in t or "navyfederal.org" in t) ) except Exception: return False def _parse_nfcu_page(pdf_path: str, page_num: int) -> str: try: import pymupdf as fitz doc = fitz.open(pdf_path) if page_num >= len(doc): doc.close() return "" page = doc[page_num] page_text = page.get_text() or "" words = page.get_text("words") or [] doc.close() if not page_text or not words: return "" t = page_text.lower() is_nfcu = ( ("statement of account" in t) and (("access no." in t) or ("routing number" in t) or ("navy federal" in t) or ("navyfederal.org" in t)) ) if not is_nfcu: return "" buckets = [] for w in words: if len(w) < 5: continue x0 = float(w[0]) y0 = float(w[1]) token = w[4].strip() if not token: continue bi = None for i, (yb, _) in enumerate(buckets): if abs(yb - y0) <= 2.0: bi = i break if bi is None: buckets.append((y0, [(x0, token)])) else: buckets[bi][1].append((x0, token)) lines = [] for y, arr in sorted(buckets, key=lambda x: x[0]): toks = [tok for _, tok in sorted(arr, key=lambda x: x[0])] xs = [x for x, _ in sorted(arr, key=lambda x: x[0])] lines.append((y, xs, toks, " ".join(toks))) date_re = re.compile(r"^\d{2}-\d{2}$") money_re = re.compile(r"^[\d,]+\.\d{2}-?$") acct_re = re.compile(r"^(Business Checking|Mbr Business Savings)\s*-\s*(\d+)") stop_markers = ( "Items Paid", "Average Daily Balance", "2024 Year to Date", "Disclosure", "What to Do", "Errors", "Payments", "SAVINGS DIVIDENDS", "REMITTANCE RECEIVED", "DEPOSIT VOUCHER", "ACCOUNT NUMBER", "MARK \"X\"", "ADDRESS/ORDER", "ITEMS ON REVERSE", "CHANGE OF ADDRESS", "PLEASE PRINT", "RANK/RATE", "PO BOX", "MERRIFIELD", "Questions about this Statement", ) def parse_account_rows(start_idx): rows = [] i = start_idx pending = None while i < len(lines): _, _, toks, text = lines[i] if not text: i += 1 continue if acct_re.match(text) and i != start_idx: break if any(text.startswith(s) for s in stop_markers): break upper_text = text.upper() if ( "REMITTANCE RECEIVED" in upper_text or "DEPOSIT VOUCHER" in upper_text or "ACCOUNT NUMBER" in upper_text or "ADDRESS/ORDER" in upper_text or "ITEMS ON REVERSE" in upper_text or "CHANGE OF ADDRESS" in upper_text or "PLEASE PRINT" in upper_text ): break if text in ("Checking", "Savings"): break if text.startswith("Date Transaction Detail"): i += 1 continue def flush_pending(): nonlocal pending if pending is not None: rows.append(pending) pending = None if toks and date_re.match(toks[0]): flush_pending() date = toks[0] rest = toks[1:] mvals = [tok for tok in rest if money_re.match(tok)] desc_tokens = [tok for tok in rest if not money_re.match(tok)] amount = "" balance = "" if len(mvals) >= 2: amount = mvals[-2] balance = mvals[-1] elif len(mvals) == 1: balance = mvals[-1] pending = { "date": date, "desc": " ".join(desc_tokens).strip(), "amount": amount, "balance": balance, } else: if pending is not None: mvals = [tok for tok in toks if money_re.match(tok)] desc_tokens = [tok for tok in toks if not money_re.match(tok)] upper = text.upper() if pending["balance"] and ( "REMITTANCE" in upper or "DEPOSIT VOUCHER" in upper or "ACCOUNT NUMBER" in upper or "ADDRESS/ORDER" in upper or "ITEMS ON REVERSE" in upper or "PO BOX" in upper ): break if ( ("BEGINNING BALANCE" in pending["desc"].upper() or "ENDING BALANCE" in pending["desc"].upper()) and not mvals and not (toks and date_re.match(toks[0])) ): break if desc_tokens: pending["desc"] = (pending["desc"] + " " + " ".join(desc_tokens)).strip() if len(mvals) >= 2 and (not pending["amount"] or not pending["balance"]): if not pending["amount"]: pending["amount"] = mvals[-2] if not pending["balance"]: pending["balance"] = mvals[-1] elif len(mvals) == 1 and not pending["balance"]: pending["balance"] = mvals[-1] i += 1 if pending is not None: rows.append(pending) cleaned = [r for r in rows if r["date"] and r["desc"] and (r["amount"] or r["balance"])] return cleaned, i out = [] i = 0 while i < len(lines): _, _, _, text = lines[i] if not text: i += 1 continue m = acct_re.match(text) if m: acct_name = m.group(1) acct_no = m.group(2) title = f"{acct_name} - {acct_no}" if i + 1 < len(lines) and "Continued" in lines[i + 1][3]: title += " (Continued from previous page)" out.append(f"{title}") out.append("") out.append("") rows, i2 = parse_account_rows(i + 1) for r in rows: out.append( f"" ) out.append("
DateTransaction DetailAmount($)Balance($)
{r['date']}{r['desc']}{r['amount']}{r['balance']}
") i = i2 continue if text in ("Checking", "Savings"): out.append(f"

{text}

") elif text.startswith("Items Paid"): out.append("Items Paid") i += 1 return "\n".join(out).strip() except Exception as e: log.warning("_parse_nfcu_page failed (page %d): %s", page_num, e) return "" def _extract_nfcu_summary_table(pdf_path: str, page_num: int) -> str: try: import pymupdf as fitz doc = fitz.open(pdf_path) if page_num >= len(doc): doc.close() return "" page = doc[page_num] words = page.get_text("words") or [] doc.close() if not words: return "" buckets = [] for w in words: if len(w) < 5: continue x0 = float(w[0]) y0 = float(w[1]) tok = str(w[4] or "").strip() if not tok: continue bi = None for i, (yb, _) in enumerate(buckets): if abs(yb - y0) <= 2.0: bi = i break if bi is None: buckets.append((y0, [(x0, tok)])) else: buckets[bi][1].append((x0, tok)) lines = [] for y, arr in sorted(buckets, key=lambda x: x[0]): arr2 = sorted(arr, key=lambda x: x[0]) toks = [t for _, t in arr2] text = " ".join(toks).strip() lines.append((y, toks, text)) anchor_idx = None for i, (_, _, text) in enumerate(lines): tl = text.lower() if "summary of your deposit accounts" in tl or ("summary" in tl and "deposit accounts" in tl): anchor_idx = i break if anchor_idx is None: return "" money_re = re.compile(r"^\$?[\d,]+\.\d{2}-?$") acct_no_re = re.compile(r"^\d{8,12}$") stop_re = re.compile( r"(business checking\s*-|mbr business savings\s*-|date\s+transaction\s+detail|items paid)", re.IGNORECASE, ) rows = [] pending_account = "" account_name_re = re.compile( r"^(business checking|mbr business savings|checking|savings|money market|certificate).*$", re.IGNORECASE, ) for _, toks, text in lines[anchor_idx + 1:]: if not text: continue if stop_re.search(text): break if not toks: continue t0 = toks[0].lower() tl = text.lower() if ( "previous" in tl or "deposits" in tl or "withdrawals" in tl or "ending" in tl or "dividends" in tl ) and not any(money_re.match(t) for t in toks): continue if account_name_re.match(text.strip()) and not any(money_re.match(t) for t in toks): pending_account = text.strip() continue mvals = [t.replace("$", "") for t in toks if money_re.match(t)] acct_tokens = [t for t in toks if acct_no_re.match(t)] if t0 == "totals" and len(mvals) >= 5: rows.append(["Totals", mvals[0], mvals[1], mvals[2], mvals[3], mvals[4]]) continue if acct_tokens and len(mvals) >= 5: acct = acct_tokens[0] acct_label = (pending_account + " - " + acct).strip(" -") if pending_account else acct rows.append([acct_label, mvals[0], mvals[1], mvals[2], mvals[3], mvals[4]]) pending_account = "" continue if not rows: return "" hdr = [ "Account", "Previous Balance", "Deposits/Credits", "Withdrawals/Debits", "Ending Balance", "YTD Dividends", ] return _grid_to_html([hdr] + rows) except Exception: return "" def _replace_nfcu_summary_table(page_md: str, summary_table_html: str) -> str: if not page_md or not summary_table_html: return page_md table_pattern = re.compile(r"]*>.*?", re.DOTALL | re.IGNORECASE) lower = page_md.lower() anchor = lower.find("summary of your deposit accounts") if anchor >= 0: for m in table_pattern.finditer(page_md): if m.start() > anchor: old_table = m.group(0) return page_md.replace(old_table, summary_table_html, 1) for m in table_pattern.finditer(page_md): tbl = m.group(0) p = TableGridParser() p.feed(tbl) g = _build_grid(p.rows) if not g: continue h = " ".join(str(c or "").strip().lower() for c in g[0]) if ( "previous" in h and "deposits" in h and "withdrawals" in h and "ending" in h and "dividends" in h ): return page_md.replace(tbl, summary_table_html, 1) return page_md def _split_fused_date_transaction_header(grid): if not grid or len(grid) < 2: return grid ncols = max(len(r) for r in grid if isinstance(r, list)) if ncols < 3: return grid header_row_idx = None fused_col_idx = None for ri in range(min(4, len(grid))): r = [str(c or "").strip() for c in (grid[ri] + [""] * (ncols - len(grid[ri])))] for ci, c in enumerate(r): cc = re.sub(r"\s+", " ", c).strip().lower() if cc == "date transaction detail": row_text = " ".join(re.sub(r"\s+", " ", x).strip().lower() for x in r) if "amount" in row_text and "balance" in row_text: header_row_idx = ri fused_col_idx = ci break if header_row_idx is not None: break if header_row_idx is None or fused_col_idx is None: return grid date_tok = re.compile(r"^\d{1,2}(?:[-/])\d{2}(?:[-/]\d{2,4})?$") new_grid = [] for ri, row in enumerate(grid): cells = [str(c or "").strip() for c in (row + [""] * (ncols - len(row)))] first = cells[fused_col_idx] dval = "" tval = first if first: parts = first.split() if parts and date_tok.match(parts[0]): dval = parts[0] tval = " ".join(parts[1:]).strip() if ri == header_row_idx: split_row = ( cells[:fused_col_idx] + ["Date", "Transaction Detail"] + cells[fused_col_idx + 1:] ) elif ri < header_row_idx: split_row = cells[:fused_col_idx] + [cells[fused_col_idx], ""] + cells[fused_col_idx + 1:] else: split_row = cells[:fused_col_idx] + [dval, tval] + cells[fused_col_idx + 1:] new_grid.append(split_row) return new_grid def _split_combined_summary_daily_balance_grid(grid): if not grid or len(grid) < 4: return None def _norm(s): return re.sub(r"\s+", " ", str(s or "").strip().lower()) daily_idx = None for i, row in enumerate(grid): row_text = " ".join(_norm(c) for c in row if str(c or "").strip()) if "daily balance" in row_text: daily_idx = i break if daily_idx is None or daily_idx <= 1 or daily_idx >= len(grid) - 2: return None date_re = re.compile(r"^\d{1,2}/\d{2}(?:/\d{2,4})?$") money_re = re.compile(r"^-?\$?\d{1,3}(?:,\d{3})*(?:\.\d{2})-?$") daily_rows = [] for row in grid[daily_idx + 1:]: tokens = [] for cell in row: ct = str(cell or "").strip() if not ct: continue tokens.extend(re.split(r"\s+", ct)) dates = [t for t in tokens if date_re.match(t)] amts = [t for t in tokens if money_re.match(t)] if not dates or not amts: continue for d, a in zip(dates, amts): daily_rows.append([d, a]) if len(daily_rows) < 2: return None summary = [r for r in grid[:daily_idx] if any(str(c or "").strip() for c in r)] if len(summary) < 2: return None daily = [["Date", "Ledger Balance"]] + daily_rows return [summary, daily] def _normalize_double_desc_amount_header_table(grid): if not grid or len(grid) < 2: return grid def _hdr_tight(s): return re.sub(r"\s+", "", str(s or "").strip().lower()) expected5_t = ["date", "description", "amount", "description", "amount"] expected4_t = ["date", "description", "amount", "description"] def _row_is_ucb_header(cells): if len(cells) == 5: return [_hdr_tight(c) for c in cells] == expected5_t if len(cells) == 4: return [_hdr_tight(c) for c in cells] == expected4_t return False hdr_idx = None ncols = 0 for i in range(min(5, len(grid))): raw_try = [str(c or "").strip() for c in grid[i]] if _row_is_ucb_header(raw_try): hdr_idx = i ncols = len(raw_try) break if hdr_idx is None: return grid date_only_re = re.compile(r"^\d{1,2}/\d{1,2}/\d{2,4}$") date_prefix_re = re.compile(r"^(\d{1,2}/\d{1,2}/\d{2,4})\s*(.*)$") money_cell_re = re.compile(r"^-?\$?\d{1,3}(?:,\d{3})*(?:\.\d{2})-?$") def _is_money(s): if not s or not str(s).strip(): return False return bool(money_cell_re.match(str(s).strip())) def _extract_money(s): if not s: return "" m = re.search(r"(-?\$?\d{1,3}(?:,\d{3})*(?:\.\d{2})-?)", str(s)) return m.group(1).strip() if m else "" out = [["Date", "Description", "Amount"]] for row in grid[hdr_idx + 1 :]: cells = [str(c or "").strip() for c in row] while len(cells) < ncols: cells.append("") c0, c1, c2 = cells[0], cells[1] if len(cells) > 1 else "", cells[2] if len(cells) > 2 else "" c3 = cells[3] if len(cells) > 3 else "" c4 = cells[4] if len(cells) > 4 else "" if not any(cells[:ncols]): continue mpre = date_prefix_re.match(c0) if mpre and _is_money(c1): date_s = mpre.group(1).strip() low = (c0 + " " + (mpre.group(2) or "")).lower() if "beginning balance" in low: desc = "Beginning Balance" elif "ending balance" in low: desc = "Ending Balance" else: desc = (mpre.group(2) or "").strip() or "Transaction" out.append([date_s, desc, c1]) continue if date_only_re.match(c0.strip()): date_s = c0.strip() desc = c1 amt = c2 if desc and "beginning balance" in desc.lower(): mx = _extract_money(desc) if mx and (not amt or amt in ("$0.00", "0.00", "$0", ".00", "$.00")): amt = mx desc = re.sub( r"(?i)beginning\s+balance\s*\$?[\d,]+\.?\d*\s*", "Beginning Balance ", desc, ) desc = re.sub(r"(?i)\s*average\s+ledger\s+balance\s*", " ", desc).strip() if not desc or desc == "Beginning Balance": desc = "Beginning Balance" out.append([date_s, desc, amt]) continue if not date_prefix_re.match(c0) and not date_only_re.match(c0.strip()) and _is_money(c1): out.append(["", c0, c1]) continue out.append([c0, c1, c2]) return out if len(out) >= 2 else grid def _normalize_ucb_three_col_beginning_balance_fusion(grid): if not grid or len(grid) < 2: return grid def _tight(s): return re.sub(r"\s+", "", str(s or "").strip().lower()) h0 = [str(c or "").strip() for c in grid[0]] if len(h0) < 3: return grid if _tight(h0[0]) != "date" or _tight(h0[1]) != "description" or _tight(h0[2]) != "amount": return grid date_only_re = re.compile(r"^\d{1,2}/\d{1,2}/\d{2,4}$") def _extract_money(s): if not s: return "" m = re.search(r"(-?\$?\d{1,3}(?:,\d{3})*(?:\.\d{2})-?)", str(s)) return m.group(1).strip() if m else "" zeroish = {"", "$0.00", "0.00", "$0", ".00", "$.00", "0"} out = [list(grid[0])] changed = False for row in grid[1:]: cells = [str(c or "").strip() for c in row] while len(cells) < 3: cells.append("") c0, c1, c2 = cells[0], cells[1], cells[2] if ( date_only_re.match(c0.strip()) and c1 and "beginning balance" in c1.lower() ): mx = _extract_money(c1) c2n = c2.strip() if c2 else "" if mx and (not c2n or c2n in zeroish): out.append([c0.strip(), "Beginning Balance", mx]) changed = True continue out.append([c0, c1, c2]) return out if changed else grid def _fix_three_col_date_desc_amount_left_shift(grid): if not grid or len(grid) < 2: return grid def _tight(s): return re.sub(r"\s+", "", str(s or "").strip().lower()) h0 = [str(c or "").strip() for c in grid[0]] if len(h0) < 3: return grid if _tight(h0[0]) != "date" or _tight(h0[1]) != "description" or _tight(h0[2]) != "amount": return grid date_then_desc = re.compile( r"^(\d{1,2}/\d{1,2}/\d{2,4})\s+(.+)$" ) money_re = re.compile(r"^-?\$?\d{1,3}(?:,\d{3})*(?:\.\d{2})-?$") out = [list(grid[0])] changed = False for row in grid[1:]: cells = [str(c or "").strip() for c in row] while len(cells) < 3: cells.append("") c0, c1, c2 = cells[0], cells[1], cells[2] if not c0: out.append([c0, c1, c2]) continue m = date_then_desc.match(c0) if not m: out.append([c0, c1, c2]) continue desc_part = (m.group(2) or "").strip() if not desc_part: out.append([c0, c1, c2]) continue if not (c1 and money_re.match(c1.strip())): out.append([c0, c1, c2]) continue if c2 and str(c2).strip(): out.append([c0, c1, c2]) continue out.append([m.group(1).strip(), desc_part, c1.strip()]) changed = True return out if changed else grid def _normalize_fused_date_posted_amount_table(grid): if not grid or len(grid) < 2: return grid ncols = max(len(r) for r in grid if isinstance(r, list)) if ncols < 2: return grid def _row_cells(r): return [str(c or "").strip() for c in (r + [""] * (ncols - len(r)))] hdr_idx = None for ri in range(min(6, len(grid))): cells = _row_cells(grid[ri]) row_text = " ".join(cells).lower() if ( "date posted transaction description" in row_text and "amount" in row_text ): hdr_idx = ri break if hdr_idx is None: return grid date_re = re.compile(r"^\d{1,2}[/-]\d{1,2}(?:[/-]\d{2,4})?$") out = [] for ri, row in enumerate(grid): cells = _row_cells(row) if ri == hdr_idx: out.append(["Date", "Transaction Description", "Amount"]) continue if ri < hdr_idx: joined = " ".join(c for c in cells if c).strip() out.append(["", joined, ""]) continue c1 = cells[0].strip() c2 = cells[1].strip() if len(cells) > 1 else "" lower_c1 = c1.lower() if ( lower_c1.startswith("date posted transaction description") or lower_c1 in {"ach additions", "ach deductions", "other additions", "other deductions", "service charges and fees"} ): continue date = "" desc = c1 if c1: parts = c1.split() if parts and date_re.match(parts[0]): date = parts[0] desc = " ".join(parts[1:]).strip() amount = c2 if not amount: money_cells = [] for x in cells[1:]: t = str(x or "").strip() if re.match(r"^-?\$?\d{1,3}(?:,\d{3})*(?:\.\d{2})-?$", t): money_cells.append(t) if money_cells: amount = money_cells[-1] if not amount and desc: m = re.search(r"(-?\$?\d{1,3}(?:,\d{3})*(?:\.\d{2})-?)\s*$", desc) if m: amount = m.group(1) desc = desc[:m.start()].strip() if not (date or desc or amount): continue out.append([date, desc, amount]) if len(out) < 2: return grid return out def normalize_html_tables(text: str) -> str: if not text or "]*>.*?", re.DOTALL | re.IGNORECASE) out = [] last = 0 for m in pattern.finditer(text): out.append(text[last : m.start()]) table_html = m.group(0) try: p = TableGridParser() p.feed(table_html) grid = _build_grid(p.rows) if _is_junk_table(grid): out.append("") last = m.end() continue grid = _normalize_double_desc_amount_header_table(grid) grid = _normalize_ucb_three_col_beginning_balance_fusion(grid) grid = _fix_three_col_date_desc_amount_left_shift(grid) grid = _drop_truly_empty_columns(grid) grid = _merge_blank_header_text_columns(grid) grid = _reconstruct_separator_rows(grid) grid, recovered_row = _clean_header_artifacts(grid) if recovered_row is not None: grid.insert(1, recovered_row) grid = _promote_misplaced_header_row(grid) grid = _split_fused_date_transaction_header(grid) grid = _normalize_fused_date_posted_amount_table(grid) grid = _split_fused_multicolumn_header(grid) grid = _normalize_daily_balance_table(grid) grid = _normalize_checks_paid_table(grid) grid = _merge_split_rows(grid) grid = _extract_fused_desc_amount(grid) grid = _repair_da3_garbled_amount_cells(grid) grid = _fix_fused_keyvalue_rows(grid) grid = _dedupe_duplicate_rows_in_da3_table(grid) split_tables = _split_combined_summary_daily_balance_grid(grid) if split_tables: out.append("\n\n".join(_grid_to_html(g) for g in split_tables if g)) else: out.append(_grid_to_html(grid) if grid else table_html) except Exception: out.append(table_html) last = m.end() out.append(text[last:]) return "".join(out) def _transaction_row_dedupe_key(cells): parts = [str(c or "").strip() for c in cells[:3]] while len(parts) < 3: parts.append("") date_s, desc_s, amt_s = parts[0], parts[1], parts[2] date_s = re.sub(r"\s+", "", date_s) amt_s = re.sub(r"[\s$,]", "", amt_s).lower().replace("−", "-") desc_s = html.unescape(desc_s) desc_s = desc_s.replace("`", "'").replace("'", "'") desc_s = re.sub(r"\s+", "", desc_s.lower()) return (date_s, desc_s, amt_s) def _da3_loose_key(cells): parts = [str(c or "").strip() for c in cells[:3]] while len(parts) < 3: parts.append("") date_s, desc_s, amt_s = parts[0], parts[1], parts[2] if not _DA3_LEADING_DATE.match(date_s): return None comb = f"{date_s} {desc_s}".lower() if re.search(r"\btotal\b", comb) and len(desc_s) > 25: return None if not _DA3_STRICT_AMOUNT.match(amt_s.strip()): return None t_amt = re.sub(r"[\s$,]", "", amt_s).replace("−", "-").replace("$", "") try: af = round(float(t_amt), 2) except ValueError: return None d_norm = re.sub(r"\s+", "", date_s) return (d_norm, af) def _row_counter_loose_da3(g): c = Counter() if not g or len(g) < 2: return c for r in g[1:]: cells = [str(x or "").strip() for x in r] if not any(cells): continue k = _da3_loose_key(cells) if k: c[k] += 1 return c def _overlap_ratio_counter(c_num, c_den): tot = sum(c_num.values()) if tot == 0: return 0.0 hit = sum(min(c_num[k], c_den.get(k, 0)) for k in c_num) return hit / tot def _multiset_leq(c_small, c_big): return all(c_small[k] <= c_big.get(k, 0) for k in c_small) def _is_da3_header(grid): if not grid or len(grid[0]) < 3: return False h = [re.sub(r"\s+", " ", str(c or "").strip().lower()) for c in grid[0]] if len(h) != 3: return False return h[0] == "date" and "description" in h[1] and ( h[2] == "amount" or h[2].startswith("amount") ) _DA3_STRICT_AMOUNT = re.compile(r"^-?\$?\d{1,3}(?:,\d{3})*\.\d{2}-?$") _DA3_MONEY_TOKEN = re.compile( r"(? 0 else 2 take = min(run_len, cap) for t in range(take): out.append(data_rows[i + t]) i = j return out def _row_text_full_ucb(grid, r: int) -> str: if not grid or r < 0 or r >= len(grid): return "" parts = [str(c or "").strip().lower() for c in grid[r]] return " ".join(parts) def _is_itm_deposit_row_ucb(grid, r: int) -> bool: if not grid or r >= len(grid) or len(grid[r]) < 2: return False desc = str(grid[r][1] or "").strip().lower() return "itm deposit" in desc def _ucb_verify_electronic_credits_rows(grid, ec_start: int) -> bool: if ec_start + 2 >= len(grid): return False t0 = _row_text_full_ucb(grid, ec_start) t1 = _row_text_full_ucb(grid, ec_start + 1) t2 = _row_text_full_ucb(grid, ec_start + 2) ok0 = "pos return" in t0 or ("pos" in t0 and "return" in t0) ok1 = "acct fund" in t1 or ("apple" in t1 and "inst" in t1) ok2 = "dda transfer" in t2 and "in" in t2 return bool(ok0 and ok1 and ok2) def _ucb_split_grid_deposits_ec_ed(grid): if not grid or len(grid) < 2: return None if not _is_da3_header(grid): return None data_start = 1 i = data_start while i < len(grid) and _is_itm_deposit_row_ucb(grid, i): i += 1 if i == data_start: return None if i + 3 > len(grid): return None if not _ucb_verify_electronic_credits_rows(grid, i): return None ec_end = i + 3 if ec_end >= len(grid): return None hdr = [list(grid[0])] g_dep = hdr + grid[data_start:i] g_ec = hdr + grid[i:ec_end] g_ed = hdr + grid[ec_end:] if len(g_ed) <= 1: return None return g_dep, g_ec, g_ed def _is_strong_ucb_electronic_debit_row(grid, r: int) -> bool: """True for typical card/ACH debit lines (generic; avoids splitting on POS return / transfer-in credits).""" t = _row_text_full_ucb(grid, r) if not t or not t.strip(): return False if "pos return" in t: return False if "dda transfer" in t and "in" in t: return False if "pos purchase" in t: return True if "atm withdrawal" in t: return True if "atm" in t and "withdrawal" in t: return True if "debit card" in t: return True if "point of sale" in t: return True if "recur payment" in t: return True return False def _ucb_has_itm_deposit_data_row(grid) -> bool: for r in range(1, len(grid)): if _is_itm_deposit_row_ucb(grid, r): return True return False def _ucb_table_credits_only_no_strong_debits(grid) -> bool: """True if DA3 table has real data rows but none match strong electronic-debit patterns.""" if not grid or len(grid) < 2 or not _is_da3_header(grid): return False has_data = False for r in range(1, len(grid)): row_txt = _row_text_full_ucb(grid, r).strip() if not row_txt: continue has_data = True if _is_strong_ucb_electronic_debit_row(grid, r): return False return bool(has_data and _ucb_has_itm_deposit_data_row(grid)) def _ucb_html_table_body_looks_itm_credits_only(table_html: str) -> bool: """ Detect ITM-only credit block without relying on parsed grid headers (Fix 17c-html). Conservative: require ITM deposit text and absence of common electronic-debit verbs in the HTML. """ if not table_html or "= len(grid): return None if not _is_strong_ucb_electronic_debit_row(grid, i): return None hdr = [list(grid[0])] g_ec = hdr + grid[data_start:i] g_ed = hdr + grid[i:] if len(g_ec) <= 1 or len(g_ed) <= 1: return None return g_ec, g_ed def _try_split_ucb_ec_ed_from_fused_table_html( lead_before_ec: str, table_html: str, had_ed_heading_immediately_before_table: bool ): try: p = TableGridParser() p.feed(table_html) grid = _build_grid(p.rows) except Exception: return None pair = _ucb_split_grid_electronic_credits_debits_after_itm_prefix(grid) if pair: g_ec, g_ed = pair sect = '
\n\n{title}\n\n
\n\n' parts = [] if lead_before_ec and lead_before_ec.strip(): parts.append(lead_before_ec.rstrip()) parts.extend( [ sect.format(title="Electronic Credits"), _grid_to_html(g_ec), sect.format(title="Electronic Debits"), _grid_to_html(g_ed), ] ) return "\n\n".join(parts) # Fix 17c: headings stacked as EC / ED / table but rows are all credits (e.g. ITM-only) — # put the table under Electronic Credits, then leave Electronic Debits for continuation. if had_ed_heading_immediately_before_table and _ucb_table_credits_only_no_strong_debits(grid): sect = '
\n\n{title}\n\n
\n\n' parts = [] if lead_before_ec and lead_before_ec.strip(): parts.append(lead_before_ec.rstrip()) parts.extend( [ sect.format(title="Electronic Credits"), _grid_to_html(grid), sect.format(title="Electronic Debits"), ] ) return "\n\n".join(parts) # Fix 17c-html: same reorder when grid header checks fail but raw HTML is clearly ITM-only credits. if had_ed_heading_immediately_before_table and _ucb_html_table_body_looks_itm_credits_only(table_html): sect = '
\n\n{title}\n\n
\n\n' parts = [] if lead_before_ec and lead_before_ec.strip(): parts.append(lead_before_ec.rstrip()) parts.extend( [ sect.format(title="Electronic Credits"), table_html.strip(), sect.format(title="Electronic Debits"), ] ) return "\n\n".join(parts) return None # Centered block titles: align= center OR inline style (Gradio / browsers vary). _UCB_CENTER_DIV = ( r'(?:' r']*align\s*=\s*["\']center["\'][^>]*>' r'|' r']*style\s*=\s*["\'][^"\']*text-align\s*:\s*center[^"\']*["\'][^>]*>' r')' ) # Optional empty centered banner only; EC/ED title divs are re-emitted after a successful split # so we do not duplicate headings. _UCB_EC_ED_SINGLE_FUSED_TABLE = re.compile( r'(?P(?:' + _UCB_CENTER_DIV + r'\s*\s*)?)' r'(?P' + _UCB_CENTER_DIV + r'\s*Electronic\s+Credits\s*\s*)' r'(?P(?:' + _UCB_CENTER_DIV + r'\s*Electronic\s+Debits\s*\s*)?)' r'(?P]*>.*?
)', re.DOTALL | re.IGNORECASE, ) def _ucb_strip_redundant_ed_subset_fragment(text: str) -> str: """ Fix 17d / 17f: Remove a duplicate
Electronic Debits
+ when that table's rows are a loose (date, amount) multiset subset of the DA3 table that appears immediately before this block. Fix 17f: If the fragment also contains ITM DEPOSIT rows (mis-copied under Debits), ignore those rows for the subset check so the tail still compares against the preceding debits table. """ if not text or "electronic debits" not in text.lower(): return text try: ed_block = re.compile( _UCB_CENTER_DIV + r'\s*Electronic\s+Debits\s*\s*' r'(?P]*>.*?
)', re.DOTALL | re.IGNORECASE, ) tbl_pat = re.compile(r']*>.*?', re.DOTALL | re.IGNORECASE) _DATE_NONEMPTY = re.compile(r"^\d{1,2}[/\-]\d{2}") # Stripped amounts may be "2194.19" (no comma) — must not use comma-group regex. _AMT_STRIPPED_OK = re.compile(r"^-?\d+(?:\.\d{1,4})?-?$") def _loose_da_key_cells(cells): parts = [str(c or "").strip() for c in cells[:3]] while len(parts) < 3: parts.append("") date_s = re.sub(r"\s+", "", parts[0]) amt_s = re.sub(r"[\s$,]", "", parts[2]).lower() amt_s = amt_s.replace("\u2212", "-").replace("\u2013", "-") return (date_s, amt_s) def _loose_keys_from_grid(g): if not g or not _is_da3_header(g): return None keys = [] for row in g[1:]: cells = [str(c or "").strip() for c in row] if not any(cells): continue k = _loose_da_key_cells(cells) if ( _DATE_NONEMPTY.match(k[0]) and _AMT_STRIPPED_OK.match(k[1]) and "." in k[1] ): keys.append(k) return keys def _loose_keys_from_grid_skip_itm(g): if not g or not _is_da3_header(g): return None keys = [] for r in range(1, len(g)): if _is_itm_deposit_row_ucb(g, r): continue cells = [str(c or "").strip() for c in g[r]] if not any(cells): continue k = _loose_da_key_cells(cells) if ( _DATE_NONEMPTY.match(k[0]) and _AMT_STRIPPED_OK.match(k[1]) and "." in k[1] ): keys.append(k) return keys def _frag_has_itm_data_row(g): if not g: return False for r in range(1, len(g)): if not any(str(c or "").strip() for c in g[r]): continue if _is_itm_deposit_row_ucb(g, r): return True return False for _ in range(12): changed = False for m in ed_block.finditer(text): before = text[: m.start()] prev_tbl_m = None for tm in tbl_pat.finditer(before): prev_tbl_m = tm if prev_tbl_m is None: continue g_big = _parse_da3_grid_from_table_html(prev_tbl_m.group(0)) g_frag = _parse_da3_grid_from_table_html(m.group("tbl")) kb = _loose_keys_from_grid(g_big) kf = _loose_keys_from_grid(g_frag) strip = False if kb and kf and len(kf) <= 24 and len(kf) < len(kb): cb, cf = Counter(kb), Counter(kf) if all(cf[k] <= cb.get(k, 0) for k in cf): strip = True # Fix 17f: ITM rows duplicated from Credits; debit tail ⊆ previous debits table if ( not strip and kb and g_frag and _frag_has_itm_data_row(g_frag) ): kf2 = _loose_keys_from_grid_skip_itm(g_frag) if ( kf2 and len(kf2) <= 24 and len(kf2) < len(kb) ): cb, cf2 = Counter(kb), Counter(kf2) if all(cf2[k] <= cb.get(k, 0) for k in cf2): strip = True if strip: text = text[: m.start()] + text[m.end() :] changed = True break if not changed: break return text except Exception as e: log.warning("_ucb_strip_redundant_ed_subset_fragment failed: %s", e) return text def _ucb_prune_ed_fragment_dupes_vs_prev_table(text: str) -> str: """ Fix 17g: Under a second Electronic Debits + DA3 table, drop leading ITM rows and rows that duplicate the immediately preceding debits table by (date, amount). Keeps orphan rows (same key absent from prev). Removes the whole block if only the header remains. If ≤6 orphan rows remain, merge them into the preceding table and remove this ED block. """ if not text or "electronic debits" not in text.lower(): return text try: ed_block = re.compile( _UCB_CENTER_DIV + r"\s*Electronic\s+Debits\s*\s*" r"(?P]*>.*?)", re.DOTALL | re.IGNORECASE, ) tbl_pat = re.compile(r"]*>.*?", re.DOTALL | re.IGNORECASE) _dn = re.compile(r"^\d{1,2}[/\-]\d{2}") _amt_ok = re.compile(r"^-?\d+(?:\.\d{1,4})?-?$") def _key_cells(cells): parts = [str(c or "").strip() for c in cells[:3]] while len(parts) < 3: parts.append("") date_s = re.sub(r"\s+", "", parts[0]) amt_s = re.sub(r"[\s$,]", "", parts[2]).lower() amt_s = amt_s.replace("\u2212", "-").replace("\u2013", "-") return date_s, amt_s def _row_loose_key(grid, r): cells = [str(c or "").strip() for c in grid[r]] if not any(cells): return None k = _key_cells(cells) if not (_dn.match(k[0]) and _amt_ok.match(k[1]) and "." in k[1]): return None return k def _has_itm_data(grid): for r in range(1, len(grid)): if not any(str(c or "").strip() for c in grid[r]): continue if _is_itm_deposit_row_ucb(grid, r): return True return False for _ in range(12): changed = False for m in ed_block.finditer(text): before = text[: m.start()] prev_tbl_m = None for tm in tbl_pat.finditer(before): prev_tbl_m = tm if prev_tbl_m is None: continue g_big = _parse_da3_grid_from_table_html(prev_tbl_m.group(0)) g_frag = _parse_da3_grid_from_table_html(m.group("tbl")) if ( not g_big or not g_frag or not _is_da3_header(g_big) or not _is_da3_header(g_frag) ): continue if not _has_itm_data(g_frag): continue cb = Counter() for r in range(1, len(g_big)): kk = _row_loose_key(g_big, r) if kk: cb[kk] += 1 new_rows = [] seen_non_itm = False for r in range(1, len(g_frag)): if not any(str(c or "").strip() for c in g_frag[r]): continue if _is_itm_deposit_row_ucb(g_frag, r): if not seen_non_itm: continue continue seen_non_itm = True kk = _row_loose_key(g_frag, r) if kk and cb.get(kk, 0) > 0: cb[kk] -= 1 continue new_rows.append(list(g_frag[r])) if not new_rows: text = text[: m.start()] + text[m.end() :] changed = True break if len(new_rows) <= 6: merged = [list(g_big[0])] + [list(r) for r in g_big[1:]] + new_rows max_c = max(len(r) for r in merged) for row in merged: while len(row) < max_c: row.append("") merged_html = _grid_to_html(merged) ps, pe = prev_tbl_m.start(), prev_tbl_m.end() text = text[:ps] + merged_html + text[pe : m.start()] + text[m.end() :] changed = True break hdr = [list(g_frag[0])] max_c = max(len(hdr[0]), max((len(x) for x in new_rows), default=0)) for row in hdr + new_rows: while len(row) < max_c: row.append("") new_tbl = _grid_to_html(hdr + new_rows) full = m.group(0) tbl = m.group("tbl") prefix = full[: full.index(tbl)] text = text[: m.start()] + prefix + new_tbl + text[m.end() :] changed = True break if not changed: break return text except Exception as e: log.warning("_ucb_prune_ed_fragment_dupes_vs_prev_table failed: %s", e) return text def _split_ucb_fused_ec_ed_headings_single_table_html(text: str) -> str: if not text or "electronic credits" not in text.lower(): return text if "\s*)' r'(]*>.*?)' r'(?:\s*' + _UCB_CENTER_DIV + r'\s*Electronic\s+Credits\s*\s*' + _UCB_CENTER_DIV + r'\s*Electronic\s+Debits\s*)?', re.DOTALL | re.IGNORECASE, ) def _split_ucb_deposits_electronic_sections_html(text: str) -> str: if not text: return text if "deposits (continued)" in text.lower(): def _repl(m): prefix = m.group(1) table_html = m.group(2) new = _try_split_ucb_deposits_table_to_three(prefix, table_html) if new is None: return m.group(0) return new text = _UCB_DEP_CONT_BLOCK.sub(_repl, text) # Fix 17e: repeat fused EC/ED fix + duplicate ED strip + 17g prune until stable. for _ in range(16): nxt = _split_ucb_fused_ec_ed_headings_single_table_html(text) nxt = _ucb_strip_redundant_ed_subset_fragment(nxt) nxt = _ucb_prune_ed_fragment_dupes_vs_prev_table(nxt) if nxt == text: break text = nxt return text def _parse_da3_grid_from_table_html(table_html: str): try: p = TableGridParser() p.feed(table_html) grid = _build_grid(p.rows) if grid and _is_da3_header(grid): return grid except Exception: pass return None def _strip_orphan_continued_da3_flatline(text: str) -> str: if not text or "(continued)" not in text.lower(): return text lines = text.splitlines(keepends=True) out = [] for line in lines: core = line.rstrip("\r\n") date_hits = len(re.findall(r"\d{2}/\d{2}", core)) + len( re.findall(r"\b\d{2}-\d{2}-\d{2}\b", core) ) if ( len(core) > 80 and "(continued)" in core.lower() and "description" in core.lower() and "amount" in core.lower() and date_hits >= 3 ): continue out.append(line) return "".join(out) def _strip_standalone_continued_banner_line(text: str) -> str: if not text or "(continued)" not in text.lower(): return text lines = text.splitlines(keepends=True) out = [] for line in lines: core = line.rstrip("\r\n") if not core.strip(): out.append(line) continue c = core.lower() if "$" in core or "=" in core: out.append(line) continue if ( "(continued)" in c and "date" not in c and "description" not in c and "amount" not in c and len(re.findall(r"\d{2}/\d{2}", core)) == 0 and len(core) < 240 ): continue out.append(line) return "".join(out) def _patch_da3_msft_g_ref_amounts_across_tables(text: str) -> str: if not text or "]*>.*?", re.DOTALL | re.IGNORECASE) def _collect(): g_map = {} ambiguous = set() for m in tbl_pat.finditer(text): g = _parse_da3_grid_from_table_html(m.group(0)) if not g: continue for row in g[1:]: cells = [str(c or "").strip() for c in row] while len(cells) < 3: cells.append("") c0, c1, c2 = cells[0], cells[1], cells[2] if not _DA3_LEADING_DATE.match(c0): continue if not _DA3_STRICT_AMOUNT.match(c2): continue blob = f"{c0} {c1} {c2}" for gid in _MSFT_PIN_G_REF.findall(blob): k = gid.upper() if k in ambiguous: continue trip = (c0, c1, c2) prev = g_map.get(k) if prev is not None and prev != trip: ambiguous.add(k) g_map.pop(k, None) elif prev is None: g_map[k] = trip return g_map, ambiguous g_map, ambiguous = _collect() if not g_map: return text out_chunks = [] last = 0 changed_any = False for m in tbl_pat.finditer(text): out_chunks.append(text[last : m.start()]) table_html = m.group(0) grid = _parse_da3_grid_from_table_html(table_html) if not grid: out_chunks.append(table_html) last = m.end() continue new_grid = [list(grid[0])] changed = False for row in grid[1:]: cells = [str(c or "").strip() for c in row] while len(cells) < 3: cells.append("") c0, c1, c2 = cells[0], cells[1], cells[2] if not _DA3_LEADING_DATE.match(c0): new_grid.append(row) continue if _DA3_STRICT_AMOUNT.match(c2): new_grid.append(row) continue blob = f"{c0} {c1} {c2}" ids = _MSFT_PIN_G_REF.findall(blob) if len(ids) != 1: new_grid.append(row) continue k = ids[0].upper() if k in ambiguous or k not in g_map: new_grid.append(row) continue d0, d1, d2 = g_map[k] new_grid.append([d0, d1, d2]) changed = True if changed: out_chunks.append(_grid_to_html(new_grid)) changed_any = True else: out_chunks.append(table_html) last = m.end() out_chunks.append(text[last:]) return "".join(out_chunks) if changed_any else text def _trim_adjacent_da3_suffix_duplicating_next_table_prefix(text: str) -> str: if not text or "]*>.*?", re.DOTALL | re.IGNORECASE) for _ in range(48): matches = list(pattern.finditer(text)) if len(matches) < 2: break changed = False for i in range(len(matches) - 1): g1 = _parse_da3_grid_from_table_html(matches[i].group(0)) g2 = _parse_da3_grid_from_table_html(matches[i + 1].group(0)) if not g1 or not g2 or len(g1) < 3 or len(g2) < 2: continue d1 = g1[1:] d2 = g2[1:] if len(d1) < 2 or len(d2) < 2: continue max_k = min(len(d1), len(d2)) best_k = 0 for k in range(max_k, 1, -1): ok = True for j in range(k): c1 = [str(x or "").strip() for x in d1[-k + j][:3]] c2 = [str(x or "").strip() for x in d2[j][:3]] while len(c1) < 3: c1.append("") while len(c2) < 3: c2.append("") if _transaction_row_dedupe_key(c1) != _transaction_row_dedupe_key(c2): ok = False break if ok: best_k = k break if best_k < 2: continue if len(d1) - best_k < 1: continue new_grid = [g1[0]] + d1[:-best_k] new_html = _grid_to_html(new_grid) m = matches[i] text = text[: m.start()] + new_html + text[m.end() :] changed = True break if not changed: break return text def _dedupe_da3_by_date_amount_loose(text: str) -> str: """ Fix 30: Drop small DA3 table fragments (1-5 data rows) that are a loose subset of an earlier larger DA3 table, matched by (date, amount) key alone. This catches OCR-truncated duplicate rows where description text differs between the fragment and the anchor table, preventing the exact-key dedup in _dedupe_subset_transaction_tables_html from firing. Example: BoA service-fee page — VIDDYOZE -1.11 appears in a 1-row fragment table after a full 2-row service-fee table that has the same row with a much longer description string. Guards: - Fragment must have 1-5 data rows. - Anchor table must have more data rows than the fragment. - Every (date, amount) pair in the fragment must be covered by the anchor. - Date must be non-empty; amount must look like currency. - Wrapped in try/except (safe no-op on any error). """ if not text or "]*>.*?", re.DOTALL | re.IGNORECASE) matches = list(pattern.finditer(text)) if len(matches) < 2: return text _DATE_NONEMPTY = re.compile(r"^\d{1,2}[/\-]\d{2}") _AMT_NONEMPTY = re.compile(r"^-?\$?\d{1,3}(?:,\d{3})*(?:\.\d{2})-?$") def _da_key(cells): """Loose (date_norm, amount_norm) key — description ignored.""" parts = [str(c or "").strip() for c in cells[:3]] while len(parts) < 3: parts.append("") date_s = re.sub(r"\s+", "", parts[0]) amt_s = re.sub(r"[\s$,]", "", parts[2]).lower() amt_s = amt_s.replace("\u2212", "-").replace("\u2013", "-") return (date_s, amt_s) # Parse all DA3 tables and collect their (date, amount) key lists. da3_info = [] for m in matches: g = _parse_da3_grid_from_table_html(m.group(0)) if not g or not _is_da3_header(g) or len(g) < 2: continue keys = [] for row in g[1:]: cells = [str(c or "").strip() for c in row] if not any(cells): continue k = _da_key(cells) if _DATE_NONEMPTY.match(k[0]) and _AMT_NONEMPTY.match(k[1]): keys.append(k) da3_info.append({"match": m, "keys": keys, "n": len(keys)}) if len(da3_info) < 2: return text drop_set = set() for i in range(1, len(da3_info)): ti = da3_info[i] if ti["n"] == 0 or ti["n"] > 5: continue if ti["match"].start() in drop_set: continue frag_keys = Counter(ti["keys"]) for j in range(i - 1, -1, -1): tj = da3_info[j] if tj["match"].start() in drop_set: continue if tj["n"] <= ti["n"]: continue big_keys = Counter(tj["keys"]) # Every (date, amount) pair in the fragment must be covered # by the anchor table (multiset subset check). if all(frag_keys[k] <= big_keys.get(k, 0) for k in frag_keys): drop_set.add(ti["match"].start()) log.info( "Fix 30: dropped DA3 fragment (%d rows) as loose " "(date+amount) subset of larger table (%d rows)", ti["n"], tj["n"], ) break if not drop_set: return text out = [] last = 0 for m in matches: out.append(text[last: m.start()]) if m.start() not in drop_set: out.append(m.group(0)) last = m.end() out.append(text[last:]) return "".join(out) except Exception as e: log.warning("_dedupe_da3_by_date_amount_loose failed: %s", e) return text def _dedupe_subset_transaction_tables_html(text: str) -> str: if not text or "]*>.*?", re.DOTALL | re.IGNORECASE) def _parse_grid(html: str): try: p = TableGridParser() p.feed(html) return _build_grid(p.rows) except Exception: return None def _row_set(g): s = set() for r in g[1:]: if not any(str(c or "").strip() for c in r): continue cells = [str(c or "").strip() for c in r] if not any(cells): continue s.add(_transaction_row_dedupe_key(cells)) return s def _one_pass(t: str) -> str: matches = list(pattern.finditer(t)) if len(matches) < 2: return t grids = [] for m in matches: grids.append((m, _parse_grid(m.group(0)))) drop_idx = set() for i in range(1, len(grids)): if i in drop_idx: continue mi, gi = grids[i] if not _is_da3_header(gi): continue si = _row_set(gi) if not si: continue for j in range(i): if j in drop_idx: continue mj, gj = grids[j] if not _is_da3_header(gj): continue sj = _row_set(gj) if not sj: continue if sj <= si and len(sj) < len(si): drop_idx.add(j) continue if si <= sj and len(si) < len(sj): drop_idx.add(i) break if si == sj: drop_idx.add(i) break inter = si & sj ri = len(inter) / len(si) if si else 0.0 rj = len(inter) / len(sj) if sj else 0.0 if ri >= 0.88 and len(si) <= len(sj): drop_idx.add(i) break if rj >= 0.88 and len(sj) < len(si): drop_idx.add(j) continue ci = _row_counter_loose_da3(gi) cj = _row_counter_loose_da3(gj) ni, nj = sum(ci.values()), sum(cj.values()) if ni >= 1 and nj >= 1: oi_j = _overlap_ratio_counter(ci, cj) oj_i = _overlap_ratio_counter(cj, ci) if ( nj >= 8 and 3 <= ni <= 14 and ni < nj and _multiset_leq(ci, cj) and oi_j >= 0.9 ): drop_idx.add(i) break if ni >= 6 and nj >= 6: if oi_j >= 0.88 and oj_i >= 0.88: drop_idx.add(i) break if _multiset_leq(ci, cj) and ni < nj and oi_j >= 0.9: drop_idx.add(i) break if _multiset_leq(cj, ci) and nj < ni and oj_i >= 0.9: drop_idx.add(j) continue if not drop_idx: return t out = [] last = 0 for idx, m in enumerate(matches): out.append(t[last : m.start()]) if idx not in drop_idx: out.append(m.group(0)) last = m.end() out.append(t[last:]) return "".join(out) out = text for _ in range(12): nxt = _one_pass(out) if nxt == out: break out = nxt return out _RE_CENTER_DIV_TWO_DA3_TABLES = re.compile( r'(\s*[^<]*?\s*)' r'(]*>.*?)' r'(\s*]*>.*?)', re.IGNORECASE | re.DOTALL, ) def _da3_row_credit_vs_debit_category(desc: str) -> str: d = (desc or "").lower() if "debit card return" in d: return "credit" if "debit card" in d and "return" in d: return "credit" if "deposit" in d and "debit card" not in d[:60]: return "credit" if "incoming wire" in d or "wire ref" in d: return "credit" if "zelle business reversal" in d or ("reversal" in d and "zelle" in d): return "credit" if "lrm claims" in d or ("claims" in d and "adj" in d): return "credit" if "adp tax" in d and "customer" in d: return "credit" if "debit card" in d: return "debit" if "ach corp debit" in d: return "debit" if "bus online ach settlement" in d: return "debit" if "zelle business payment to" in d: return "debit" if "visa money transfer debit" in d: return "debit" if "telephone payment" in d or "service charges" in d: return "debit" if "merch fee" in d: return "debit" if "deposit" in d: return "credit" return "unknown" def _da3_table_credit_debit_fractions(grid): deb = cre = unk = 0 if not grid or len(grid) < 2: return 0.0, 0.0, 0.0, 0 for row in grid[1:]: cells = [str(c or "").strip() for c in row] while len(cells) < 3: cells.append("") if not _DA3_LEADING_DATE.match(cells[0]): continue if not _DA3_STRICT_AMOUNT.match((cells[2] or "").strip()): continue cat = _da3_row_credit_vs_debit_category(cells[1]) if cat == "debit": deb += 1 elif cat == "credit": cre += 1 else: unk += 1 tot = deb + cre + unk if tot == 0: return 0.0, 0.0, 0.0, 0 return deb / tot, cre / tot, unk / tot, tot def _drop_da3_misplaced_debit_after_deposit_heading_tables(text: str) -> str: if not text or " str: window = text[max(0, table_start - _HEADING_SCAN_WINDOW): table_start] clean = re.sub(r"<[^>]+>", " ", window) lines = [ln.strip() for ln in clean.splitlines() if ln.strip()] for line in reversed(lines): if len(line) > 3 and "---" not in line: return line return "" def _strip_credits_prefix_from_debit_da3_table(text: str) -> str: """ Fix 28: When OCR prepends the rows of a preceding credits/deposits DA3 table onto the top of a debits/withdrawals DA3 table, strip that prefix. Safety guards (ALL must pass): 1. Both tables must be valid DA3 (Date|Description|Amount header). 2. Target (later) table must follow a debit/withdrawal heading. 3. Source (earlier) table must follow a credit/deposit heading. 4. Only the contiguous leading prefix of matching rows is removed. 5. At least 1 data row must remain in the target after stripping. 6. Strip count must be >= 1. 7. Entire function wrapped in try/except — returns input unchanged on error. """ if not text or "]*>.*?", re.DOTALL | re.IGNORECASE) matches = list(pattern.finditer(text)) if len(matches) < 2: return text da3_tables = [] for m in matches: g = _parse_da3_grid_from_table_html(m.group(0)) if not g or not _is_da3_header(g): continue heading = _get_heading_before_table(text, m.start()) da3_tables.append({"match": m, "grid": g, "heading": heading}) if len(da3_tables) < 2: return text def _row_keys(grid): keys = set() for row in grid[1:]: cells = [str(c or "").strip() for c in row] if any(cells): keys.add(_transaction_row_dedupe_key(cells)) return keys drop_map = {} for i in range(1, len(da3_tables)): ti = da3_tables[i] hi = ti["heading"].lower() if not _DEBIT_HEADING_KEYWORDS.search(hi): continue gi = ti["grid"] data_i = gi[1:] if len(data_i) < 2: continue for j in range(i - 1, -1, -1): tj = da3_tables[j] hj = tj["heading"].lower() if not _CREDIT_HEADING_KEYWORDS.search(hj): continue gj = tj["grid"] keys_j = _row_keys(gj) if not keys_j: continue strip_count = 0 for row in data_i: cells = [str(c or "").strip() for c in row] k = _transaction_row_dedupe_key(cells) if k in keys_j: strip_count += 1 else: break if strip_count < 1: continue if len(data_i) - strip_count < 1: continue drop_map[ti["match"].start()] = strip_count log.info( "Fix 28: stripped %d leading credit row(s) from debit table at offset %d", strip_count, ti["match"].start(), ) break if not drop_map: return text out_chunks = [] last = 0 for m in matches: out_chunks.append(text[last: m.start()]) if m.start() in drop_map: strip_n = drop_map[m.start()] g = _parse_da3_grid_from_table_html(m.group(0)) if g and _is_da3_header(g) and len(g) - 1 - strip_n >= 1: new_grid = [g[0]] + g[1 + strip_n:] out_chunks.append(_grid_to_html(new_grid)) else: out_chunks.append(m.group(0)) else: out_chunks.append(m.group(0)) last = m.end() out_chunks.append(text[last:]) return "".join(out_chunks) except Exception as e: log.warning("_strip_credits_prefix_from_debit_da3_table failed: %s", e) return text # -------------------------- # Fix 29: Move misplaced debit heading to before its orphan DA3 table # -------------------------- _CENTER_DIV_HDR_RE = re.compile( r'\s*(.*?)\s*', re.IGNORECASE | re.DOTALL, ) _HEADING_SCAN_FORWARD = 800 def _fix29_move_debit_heading_before_orphan_table(text: str) -> str: """ Fix 29: When OCR places a debit/withdrawal section heading AFTER the withdrawals table instead of before it, move the heading to its correct position (just before the table). """ if not text or "]*>.*?", re.DOTALL | re.IGNORECASE) matches = list(tbl_pat.finditer(text)) if len(matches) < 2: return text da3_matches = [m for m in matches if _parse_da3_grid_from_table_html(m.group(0))] if len(da3_matches) < 2: return text moves = [] for i in range(1, len(da3_matches)): orphan_m = da3_matches[i] orphan_start = orphan_m.start() orphan_end = orphan_m.end() before_window = text[max(0, orphan_start - _HEADING_SCAN_WINDOW): orphan_start] before_clean = re.sub(r"<[^>]+>", " ", before_window) lines_before = [l.strip() for l in before_clean.splitlines() if l.strip()] last_meaningful = "" for l in reversed(lines_before): if len(l) > 3 and "---" not in l: last_meaningful = l break if _DEBIT_HEADING_KEYWORDS.search(last_meaningful): continue big_before = text[max(0, orphan_start - 1200): orphan_start] if not _CREDIT_HEADING_KEYWORDS.search(re.sub(r"<[^>]+>", " ", big_before)): continue next_tbl_m = tbl_pat.search(text, orphan_end) dh_start = dh_end = None for div_m in _CENTER_DIV_HDR_RE.finditer(text, orphan_end, orphan_end + _HEADING_SCAN_FORWARD): div_clean = re.sub(r"<[^>]+>", "", div_m.group(1)).strip() if _DEBIT_HEADING_KEYWORDS.search(div_clean) and not _CREDIT_HEADING_KEYWORDS.search(div_clean): if next_tbl_m and next_tbl_m.start() < div_m.start(): break dh_start = div_m.start() dh_end = div_m.end() break if dh_start is None: continue moves.append((orphan_start, dh_start, dh_end)) if not moves: return text for orphan_start, dh_start, dh_end in reversed(moves): orphan_m2 = next((m for m in tbl_pat.finditer(text) if m.start() == orphan_start), None) if not orphan_m2: continue orphan_end = orphan_m2.end() heading_html = text[dh_start: dh_end] gap = text[orphan_end: dh_start] orphan_html = text[orphan_start: orphan_end] part_before = text[:orphan_start] part_after = text[dh_end:] text = part_before + heading_html + "\n\n" + orphan_html + gap.rstrip() + part_after log.info( "Fix 29: moved debit heading (was at offset %d) to before orphan DA3 table (offset %d)", dh_start, orphan_start, ) return text except Exception as e: log.warning("_fix29_move_debit_heading_before_orphan_table failed: %s", e) return text def stabilize_tables_and_text(page_md: str) -> str: if not page_md: return page_md # Fresh page: drop any text-layer dedupe caps so the first normalize pass # (before _patch_ocr_with_textlayer) does not see stale keys from a prior page. _tl_freq_context.clear() page_md = normalize_money_glyphs(page_md) blocks = re.split(r"\n\s*\n", page_md.strip()) out_blocks = [] for b in blocks: if looks_like_markdown_table(b): out_blocks.append(md_table_to_html(b)) else: out_blocks.append(b) stabilized = "\n\n".join(out_blocks) stabilized = convert_plaintext_bank_sections(stabilized) stabilized = _strip_orphan_continued_da3_flatline(stabilized) stabilized = _strip_standalone_continued_banner_line(stabilized) stabilized = normalize_html_tables(stabilized) stabilized = _trim_adjacent_da3_suffix_duplicating_next_table_prefix(stabilized) stabilized = _split_ucb_deposits_electronic_sections_html(stabilized) stabilized = _patch_da3_msft_g_ref_amounts_across_tables(stabilized) stabilized = _dedupe_subset_transaction_tables_html(stabilized) stabilized = _dedupe_da3_by_date_amount_loose(stabilized) # Fix 30 stabilized = _drop_da3_misplaced_debit_after_deposit_heading_tables(stabilized) stabilized = _strip_credits_prefix_from_debit_da3_table(stabilized) # Fix 28 stabilized = _fix29_move_debit_heading_before_orphan_table(stabilized) # Fix 29 stabilized = _strip_orphan_continued_da3_flatline(stabilized) stabilized = _strip_standalone_continued_banner_line(stabilized) return close_unclosed_html(stabilized) # -------------------------- # PDF rendering with padding # -------------------------- def render_pdf_pages_to_images(pdf_path: str) -> Tuple[List[str], List[int]]: import pymupdf as fitz from PIL import Image, ImageEnhance doc = fitz.open(pdf_path) page_images: List[str] = [] page_heights: List[int] = [] for i in range(len(doc)): page = doc[i] pix = page.get_pixmap(matrix=fitz.Matrix(RENDER_SCALE, RENDER_SCALE), alpha=False) img = Image.frombytes("RGB", [pix.width, pix.height], pix.samples) if ENABLE_CONTRAST: img = ImageEnhance.Contrast(img).enhance(1.12) w, h = img.size pad_l = int(w * PAD_LEFT_FRAC) pad_r = int(w * PAD_RIGHT_FRAC) pad_t = int(h * PAD_TOP_FRAC) pad_b = int(h * PAD_BOTTOM_FRAC) if any(p > 0 for p in (pad_l, pad_r, pad_t, pad_b)): canvas = Image.new("RGB", (w + pad_l + pad_r, h + pad_t + pad_b), (255, 255, 255)) canvas.paste(img, (pad_l, pad_t)) img = canvas img_path = os.path.join(tempfile.gettempdir(), f"glmocr_page_{os.getpid()}_{i}.png") img.save(img_path, "PNG", compress_level=6) page_images.append(img_path) page_heights.append(img.height) doc.close() return page_images, page_heights # -------------------------- # GLM-OCR result extraction # -------------------------- def get_page_md_and_regions(page_result): md = "" if hasattr(page_result, "markdown_result") and page_result.markdown_result: md = (page_result.markdown_result or "").strip() regions = [] if hasattr(page_result, "json_result"): jr = page_result.json_result if isinstance(jr, dict) and "regions" in jr: regions = jr.get("regions") or [] elif isinstance(jr, list) and len(jr) > 0: r = jr[0] if isinstance(jr[0], list) else jr if isinstance(r, list): regions = r elif isinstance(r, dict) and "regions" in r: regions = r.get("regions") or [] return md, regions # -------------------------- # Main entry # -------------------------- def run_ocr(uploaded_file): if uploaded_file is None: return "Please upload a file." page_images = [] try: path = uploaded_file.name if hasattr(uploaded_file, "name") else str(uploaded_file) is_pdf = path.lower().endswith(".pdf") is_nfcu_doc = _is_nfcu_document(path) if is_pdf else False parser = get_parser() page_heights = [] if is_pdf: page_images, page_heights = render_pdf_pages_to_images(path) results = parser.parse(page_images) else: page_images = [path] page_heights = [1000] results = parser.parse(path) if not isinstance(results, list): results = [results] all_pages = [] for page_num, page_result in enumerate(results): # Header normalize runs before body patch; do not reuse prior page caps. _tl_freq_context.clear() page_md, regions = get_page_md_and_regions(page_result) img_h = page_heights[page_num] if page_num < len(page_heights) else 1000 header_end_frac, footer_start_frac = get_header_footer_zones(regions, img_h) he = header_end_frac if header_end_frac is not None else DEFAULT_ZONE_FRAC fs = footer_start_frac if footer_start_frac is not None else (1.0 - DEFAULT_ZONE_FRAC) he = max(0.02, min(0.25, he)) fs = max(0.75, min(0.98, fs)) parts = [] hdr = "" if is_pdf: hdr = extract_zone_text_pdf(path, page_num, 0, he) if not (hdr and hdr.strip()): hdr = extract_pdf_text_in_band(path, page_num, 0, PDF_HEADER_BAND_FRAC) if not (hdr and hdr.strip()) and page_num < len(page_images): hdr = ocr_zone(page_images[page_num], 0, he) if hdr and hdr.strip(): parts.append(normalize_html_tables(fix_account_number(normalize_money_glyphs(hdr.strip())))) if page_md and page_md.strip(): stabilized = stabilize_tables_and_text(page_md.strip()) if is_pdf and not is_nfcu_doc: stabilized = _patch_ocr_with_textlayer(stabilized, path, page_num) stabilized = _strip_orphan_continued_da3_flatline(stabilized) stabilized = _strip_standalone_continued_banner_line(stabilized) stabilized = normalize_html_tables(stabilized) stabilized = _trim_adjacent_da3_suffix_duplicating_next_table_prefix(stabilized) stabilized = _split_ucb_deposits_electronic_sections_html(stabilized) stabilized = _patch_da3_msft_g_ref_amounts_across_tables(stabilized) stabilized = _dedupe_subset_transaction_tables_html(stabilized) stabilized = _dedupe_da3_by_date_amount_loose(stabilized) # Fix 30 stabilized = _drop_da3_misplaced_debit_after_deposit_heading_tables(stabilized) stabilized = _strip_credits_prefix_from_debit_da3_table(stabilized) # Fix 28 stabilized = _fix29_move_debit_heading_before_orphan_table(stabilized) # Fix 29 stabilized = _strip_standalone_continued_banner_line(stabilized) elif is_pdf and is_nfcu_doc: nfcu_summary = _extract_nfcu_summary_table(path, page_num) if nfcu_summary: stabilized = _replace_nfcu_summary_table(stabilized, nfcu_summary) if is_pdf: try: needs_checks = 'class="jb-summary-checks"' not in stabilized needs_dab = "daily account balance" not in stabilized.lower() if needs_checks or needs_dab: _txt4d = "" try: from pdfminer.high_level import extract_text as _pm_extract _txt4d = _pm_extract(path, page_numbers=[page_num]) except Exception: pass if not _txt4d: try: import pymupdf as _fitz4d _d4 = _fitz4d.open(path) _txt4d = _d4[page_num].get_text() _d4.close() except Exception: pass if _txt4d: jb_extra = _extract_jb_summary_sections(_txt4d, needs_checks, needs_dab) if jb_extra: stabilized = stabilized.rstrip() + "\n\n" + jb_extra except Exception: pass parts.append(stabilized) if ENABLE_FOOTER_OCR and page_num < len(page_images): ftr = "" if is_pdf: ftr = extract_zone_text_pdf(path, page_num, fs, 1.0) if not (ftr and ftr.strip()): ftr = extract_pdf_text_in_band(path, page_num, PDF_FOOTER_BAND_FRAC, 1.0) if not (ftr and ftr.strip()): ftr = ocr_zone(page_images[page_num], fs, 1.0) if ftr and ftr.strip(): ftr_clean = normalize_money_glyphs(ftr.strip()) ftr_first_line = next( (ln.strip().lower() for ln in ftr_clean.splitlines() if ln.strip()), "" ) already_present = ftr_first_line and any( ftr_first_line in part.lower() for part in parts ) _footer_date_re = re.compile(r"\b\d{1,2}[-/]\d{2}\b") _footer_amt_re = re.compile(r"\b\d{1,3}(?:,\d{3})*\.\d{2}\b") _date_hits = len(_footer_date_re.findall(ftr_clean)) _amt_hits = len(_footer_amt_re.findall(ftr_clean)) is_txn_dump = _date_hits >= 3 and _amt_hits >= 3 if not already_present and not is_txn_dump: parts.append(ftr_clean) if parts: all_pages.append("\n\n".join(parts)) merged = "\n\n---page-separator---\n\n".join(all_pages) if all_pages else "(No content)" if all_pages and merged and not merged.startswith("Error") and "