{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "Av5f24AacsF3" }, "source": [ "# Text2Receipt - 01. Synthetic Data Generation\n", "\n", "Messy free-text **Hebrew** income notes -> valid, complete Israeli fiscal documents (receipts & tax invoices).\n", "\n", "Hybrid pipeline: a rule-based generator writes perfect ground-truth labels, and a Hugging Face instruct model\n", "paraphrases the *input note only* for surface diversity. The deterministic `complete()` computes VAT, totals,\n", "serials and the allocation flag - so a model only ever predicts `parse`.\n", "\n", "Pools **A** (train/validation/iid_test) and **B** (ood_test) use fully disjoint Hebrew entities, giving a\n", "genuine out-of-distribution test. A fifth split, **`human_test`**, holds 25 hand-written Hebrew notes with a\n", "third, unseen vocabulary (pool **H**) - a real human-authored probe that the synthetic pipeline never saw.\n", "Runs on a free Colab **T4**.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "kwtRnNjActvE" }, "source": [ "# 0. Setup" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "_L61GqiyVWp8", "outputId": "4661e010-427f-43bd-b1e2-e26200fc1cd2" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "mode: FULL | target total: 10000\n" ] } ], "source": [ "!pip -q install -U \"huggingface_hub>=0.23\" transformers accelerate\n", "\n", "import os, json, random, collections, datetime as dt\n", "\n", "HF_DATASET_REPO = \"yonilev/Text2Receipt\" # dataset repo (not the Space)\n", "SEED = 7\n", "SAMPLE_MODE = False # True: fast check / False: full 10k build\n", "N_POOL_A = 600 if SAMPLE_MODE else 9000 # train/val/iid_test (80/10/10)\n", "N_POOL_B = 60 if SAMPLE_MODE else 1000 # ood_test (disjoint) -> 10,000 total\n", "PARAPHRASE_ENABLED = False\n", "PARAPHRASE_FRACTION = 0.5 # share of notes attempted\n", "PARA_ATTEMPTS = 2 # retries per note (raises kept-rate)\n", "PRIMARY_MODEL = \"Qwen/Qwen2.5-3B-Instruct\" # ungated, Apache-2.0, better Hebrew, T4-fp16\n", "FALLBACK_MODEL = \"Qwen/Qwen2.5-1.5B-Instruct\" # ungated, smaller\n", "PUSH_TO_HUB = True # uploads the dataset to the Hub\n", "\n", "DATA_DIR = \"data\"\n", "os.makedirs(DATA_DIR, exist_ok=True)\n", "random.seed(SEED)\n", "print(\"mode:\", \"SAMPLE\" if SAMPLE_MODE else \"FULL\", \"| target total:\", N_POOL_A + N_POOL_B)" ] }, { "cell_type": "markdown", "metadata": { "id": "DoP95xu-czaV" }, "source": [ "## 1. Core generator (`t2r_core.py`)\n", "\n", "Encodes the Israeli fiscal rules - VAT calendar, the date-stepped allocation threshold (25k in 2024 -> 5k from Jun 2026), valid ID/company check digits - and the disjoint A/B Hebrew entity pools. Pure-Python, CPU-only; the model paraphrase pass lives below, not here." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "hKHgFOh2cy3H", "outputId": "d571a21e-88dc-4656-a016-9175e12efe88" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Overwriting t2r_core.py\n" ] } ], "source": [ "%%writefile t2r_core.py\n", "# -*- coding: utf-8 -*-\n", "\"\"\"\n", "t2r_core.py - Text2Receipt core data-generation module.\n", "\n", "ARCHITECTURE (locked in project spec)\n", "-------------------------------------\n", " * Rule-based code produces PERFECT ground-truth labels.\n", " * A model (later, in 03/app) only has to predict `parse`.\n", " * Deterministic `complete()` computes everything calculable\n", " (VAT, subtotal, totals, serial numbers, allocation-number flag),\n", " so the model never predicts anything arithmetic.\n", "\n", "RECORD SCHEMA\n", "-------------\n", " raw_text : messy free-text Hebrew income note (model INPUT)\n", " parse : structured extraction of what the note STATES (model TARGET - the only thing predicted)\n", " completed : full, valid fiscal document computed from parse + issuer profile (deterministic)\n", " meta : pool (A/B), issuer/doc type, vat rate, paraphrase flag, ...\n", "\n", "ENTITIES ARE IN HEBREW (client names, business names, service descriptions),\n", "matching real Israeli income notes. Universal tech acronyms (API/ERP/SEO/UI)\n", "are kept as-is, as a Hebrew-speaking freelancer would write them.\n", "\n", "ISRAELI FISCAL RULES ENCODED\n", "----------------------------\n", " * VAT 18% since 2025-01-01; legacy 17% for documents dated in 2024.\n", " * Exempt dealer (osek patur) / institution: 0% VAT, RECEIPT only.\n", " * Allocation number (mispar haktzaa) required when ALL hold:\n", " doc_type is a tax invoice,\n", " issuer is an authorized dealer / company,\n", " client is a registered business,\n", " subtotal (before VAT) >= threshold for the document's date\n", " (timeline: 25,000 in 2024 -> ... -> 5,000 from 2026-06-01).\n", " * Israeli ID / company numbers carry a valid check digit.\n", "\n", "This module is pure-Python and runs on CPU. The optional paraphrasing\n", "pass (HF instruct model) lives in the notebook, not here.\n", "\"\"\"\n", "\n", "from __future__ import annotations\n", "import random\n", "import datetime as _dt\n", "from dataclasses import dataclass, field, asdict\n", "from typing import Optional\n", "\n", "\n", "# =====================================================================\n", "# 1. ISRAELI FISCAL CONSTANTS & CHECK DIGITS\n", "# =====================================================================\n", "\n", "VAT_NEW = 0.18 # since 2025-01-01\n", "VAT_LEGACY = 0.17 # documents dated in 2024\n", "VAT_NEW_FROM = _dt.date(2025, 1, 1)\n", "\n", "# Allocation-number threshold timeline (ILS, before VAT). Effective-from -> threshold.\n", "ALLOCATION_THRESHOLDS = [\n", " (_dt.date(2024, 5, 5), 25_000),\n", " (_dt.date(2025, 1, 1), 20_000),\n", " (_dt.date(2026, 1, 1), 10_000),\n", " (_dt.date(2026, 6, 1), 5_000),\n", "]\n", "\n", "\n", "def vat_rate_for_date(d: _dt.date) -> float:\n", " \"\"\"18% from 2025-01-01, otherwise 17% (legacy 2024 documents).\"\"\"\n", " return VAT_NEW if d >= VAT_NEW_FROM else VAT_LEGACY\n", "\n", "\n", "def allocation_threshold_for_date(d: _dt.date) -> int:\n", " \"\"\"Lowest applicable allocation-number threshold for a document date.\"\"\"\n", " thr = ALLOCATION_THRESHOLDS[0][1]\n", " for eff_from, value in ALLOCATION_THRESHOLDS:\n", " if d >= eff_from:\n", " thr = value\n", " return thr\n", "\n", "\n", "def israeli_check_digit(eight_digits: str) -> int:\n", " \"\"\"Compute the 9th check digit for an Israeli ID / company number.\n", "\n", " Weights alternate 1,2,1,2,...; products > 9 are reduced by summing digits.\n", " \"\"\"\n", " assert len(eight_digits) == 8 and eight_digits.isdigit()\n", " total = 0\n", " for i, ch in enumerate(eight_digits):\n", " p = int(ch) * (1 if i % 2 == 0 else 2)\n", " total += p if p < 10 else p - 9\n", " return (10 - (total % 10)) % 10\n", "\n", "\n", "def make_israeli_id(rng: random.Random) -> str:\n", " \"\"\"A 9-digit identifier with a valid Israeli check digit.\"\"\"\n", " base = \"\".join(str(rng.randint(0, 9)) for _ in range(8))\n", " return base + str(israeli_check_digit(base))\n", "\n", "\n", "def make_company_number(rng: random.Random) -> str:\n", " \"\"\"Israeli company number (chet-pe): 9 digits, starts with 5, valid check digit.\"\"\"\n", " base = \"5\" + \"\".join(str(rng.randint(0, 9)) for _ in range(7))\n", " return base + str(israeli_check_digit(base))\n", "\n", "\n", "def is_valid_israeli_id(num: str) -> bool:\n", " return len(num) == 9 and num.isdigit() and israeli_check_digit(num[:8]) == int(num[8])\n", "\n", "\n", "# =====================================================================\n", "# 2. ENTITY POOLS (A and B are DISJOINT -> B powers the OOD test)\n", "# =====================================================================\n", "# Each pool supplies: issuer business names, client first/last names,\n", "# and service catalogs - all in HEBREW. Nothing overlaps between A and B,\n", "# so a model trained on A and evaluated on B is tested on genuinely unseen\n", "# entities. Issuer names are neutral freelancer/studio names (the issuer is\n", "# never written in raw_text - it is the self-employed person themselves).\n", "\n", "POOLS = {\n", " \"A\": {\n", " \"issuer_biz\": [\n", " \"סטודיו רון\", \"מאיר כהן\", \"תכלת סטודיו\", \"גלים\", \"שחף\",\n", " \"נטף שירותים\", \"אלון ושות'\", \"מרום\", \"ליאת לוי\", \"צוף יצירה\",\n", " ],\n", " \"client_first\": [\"משה\", \"דנה\", \"יוסי\", \"נועה\", \"אבי\", \"תמר\", \"איתן\", \"שירה\"],\n", " \"client_last\": [\"כהן\", \"לוי\", \"מזרחי\", \"פרץ\", \"ביטון\", \"אזולאי\"],\n", " \"client_biz\": [\"מעגל בע\\\"מ\", \"אור-טק\", \"שקד אחזקות\", \"ניר לוגיסטיקה\"],\n", " \"services\": {\n", " \"consulting\": [(\"ייעוץ עסקי\", 350, 1200),\n", " (\"סדנת אסטרטגיה\", 800, 3500)],\n", " \"design\": [(\"עיצוב לוגו\", 600, 2500),\n", " (\"חבילת פוסטים לרשתות\", 250, 900)],\n", " \"tutoring\": [(\"שיעור מתמטיקה פרטי\", 120, 260),\n", " (\"שיעור אנגלית\", 130, 240)],\n", " \"trades\": [(\"החלפת ברז\", 200, 650),\n", " (\"פתיחת סתימה\", 250, 800)],\n", " \"catering\": [(\"מגש אירוח לאירוע\", 180, 540),\n", " (\"מגש קינוחים\", 150, 420)],\n", " \"software\": [(\"עמוד נחיתה\", 1500, 6000),\n", " (\"ריטיינר תיקוני באגים\", 400, 1600)],\n", " \"photography\": [(\"צילום אירוע\", 400, 1800),\n", " (\"סט צילומי מוצר\", 300, 1200)],\n", " \"marketing\": [(\"ניהול רשתות חודשי\", 900, 3200),\n", " (\"הקמת קמפיין פרסום\", 500, 2200)],\n", " },\n", " \"highvalue\": {\n", " \"software\": [(\"פרויקט אתר מלא\", 9000, 38000),\n", " (\"פרויקט אינטגרציית ERP\", 12000, 45000)],\n", " \"consulting\": [(\"ריטיינר ייעוץ רבעוני\", 6000, 18000),\n", " (\"תוכנית טרנספורמציה\", 9000, 30000)],\n", " \"design\": [(\"חבילת זהות מותגית\", 7000, 22000),\n", " (\"בניית מערכת עיצוב\", 8000, 26000)],\n", " \"marketing\": [(\"ריטיינר שיווק שנתי\", 6500, 20000),\n", " (\"קמפיין רב-ערוצי\", 7000, 24000)],\n", " },\n", " },\n", " \"B\": { # disjoint Hebrew vocabulary - never seen in training\n", " \"issuer_biz\": [\n", " \"סטודיו נגב\", \"אופק\", \"מעיין יצירה\", \"אורלי\", \"להב\",\n", " \"תבור\", \"נועם פריים\", \"גשר\", \"רקיע\", \"שלי ברק\",\n", " ],\n", " \"client_first\": [\"רוני\", \"גל\", \"עומר\", \"הילה\", \"נדב\", \"מאיה\", \"עידו\", \"ליאור\"],\n", " \"client_last\": [\"פרידמן\", \"סגל\", \"אוחיון\", \"דהאן\", \"כץ\", \"ברק\"],\n", " \"client_biz\": [\"כנרת גרופ\", \"וולט מערכות\", \"אשל סחר\", \"רקיע מדיה\"],\n", " \"services\": {\n", " \"consulting\": [(\"שעת ייעוץ פיננסי\", 400, 1300),\n", " (\"אודיט צמיחה\", 900, 3800)],\n", " \"design\": [(\"מדריך מותג\", 700, 2700),\n", " (\"מוקאפ אריזה\", 300, 1000)],\n", " \"tutoring\": [(\"שיעור פיזיקה\", 140, 280),\n", " (\"שיעור גיטרה\", 110, 230)],\n", " \"trades\": [(\"התקנת גוף תאורה\", 220, 700),\n", " (\"תיקון שקע\", 180, 520)],\n", " \"catering\": [(\"הקמת מזנון\", 200, 600),\n", " (\"קופסת מאפים\", 160, 460)],\n", " \"software\": [(\"בניית מסך מובייל\", 1600, 6500),\n", " (\"אינטגרציית API\", 500, 2000)],\n", " \"photography\": [(\"צילומי פורטרט\", 380, 1700),\n", " (\"סט צילומי נדל\\\"ן\", 320, 1300)],\n", " \"marketing\": [(\"ריטיינר תוכן\", 850, 3100),\n", " (\"הקמת SEO\", 550, 2300)],\n", " },\n", " \"highvalue\": {\n", " \"software\": [(\"בניית פלטפורמה רב-עמודית\", 9500, 40000),\n", " (\"פרויקט צינור נתונים\", 13000, 46000)],\n", " \"consulting\": [(\"ריטיינר ייעוץ שנתי\", 6500, 19000),\n", " (\"שדרוג תפעולי\", 9500, 31000)],\n", " \"design\": [(\"תוכנית מיתוג מחדש\", 7500, 24000),\n", " (\"פרויקט ערכת UI\", 8500, 27000)],\n", " \"marketing\": [(\"ריטיינר צמיחה שנתי\", 7000, 21000),\n", " (\"תוכנית קמפיין השקה\", 7500, 25000)],\n", " },\n", " },\n", "}\n", "\n", "PAYMENT_METHODS = [\"cash\", \"bank_transfer\", \"credit_card\", \"check\", \"bit\"]\n", "PAYMENT_HE = {\"cash\": \"מזומן\", \"bank_transfer\": \"העברה בנקאית\",\n", " \"credit_card\": \"אשראי\", \"check\": \"צ'ק\", \"bit\": \"ביט\"}\n", "\n", "DOC_TYPES = [\"receipt\", \"tax_invoice\", \"tax_invoice_receipt\"]\n", "DOC_TYPE_HE = {\"receipt\": \"קבלה\", \"tax_invoice\": \"חשבונית מס\",\n", " \"tax_invoice_receipt\": \"חשבונית מס/קבלה\"}\n", "\n", "# issuer fiscal status\n", "ISSUER_AUTHORIZED = \"authorized_dealer\" # osek murshe -> VAT, may issue tax invoice\n", "ISSUER_EXEMPT = \"exempt_dealer\" # osek patur -> 0% VAT, receipt only\n", "\n", "\n", "# =====================================================================\n", "# 3. ISSUER PROFILE\n", "# =====================================================================\n", "\n", "@dataclass\n", "class Issuer:\n", " name: str\n", " status: str # ISSUER_AUTHORIZED / ISSUER_EXEMPT\n", " tax_id: str # ID or company number\n", " is_company: bool\n", "\n", "\n", "def make_issuer(rng: random.Random, pool: str) -> Issuer:\n", " p = POOLS[pool]\n", " name = rng.choice(p[\"issuer_biz\"])\n", " # ~25% exempt dealers, rest authorized; ~40% of authorized are companies\n", " status = ISSUER_EXEMPT if rng.random() < 0.25 else ISSUER_AUTHORIZED\n", " is_company = status == ISSUER_AUTHORIZED and rng.random() < 0.40\n", " tax_id = make_company_number(rng) if is_company else make_israeli_id(rng)\n", " return Issuer(name=name, status=status, tax_id=tax_id, is_company=is_company)\n", "\n", "\n", "# =====================================================================\n", "# 4. RAW-TEXT TEMPLATES (messy, human-like Hebrew income notes)\n", "# =====================================================================\n", "# Diversity comes from: formality, abbreviations, typos, missing fields,\n", "# amount phrasing (gross vs net), and ordering. The note states only what\n", "# a human would jot down; complete() infers the rest.\n", "\n", "def _amount_phrase(rng: random.Random, amount: int) -> str:\n", " forms = [f\"{amount} ש\\\"ח\", f\"{amount} שקל\", f\"{amount}₪\", f\"{amount} שקלים\", f\"סכום {amount}\"]\n", " return rng.choice(forms)\n", "\n", "\n", "RAW_TEMPLATES = [\n", " \"קיבלתי {amount} מ{client} על {service}\",\n", " \"{client} שילם {amount} עבור {service}\",\n", " \"תקבול {amount} - {service} - {client}\",\n", " \"הכנסה: {service} ל{client}, {amount}, {pay}\",\n", " \"{service} {client} {amount} {pay}\",\n", " \"גבית {amount} מ{client} ({service})\",\n", " \"{client} העביר {amount} ב{pay} על {service}\",\n", " \"{amount} בעבור {service}. לקוח: {client}\",\n", " \"היום {client} - {service} - {amount}\",\n", " \"{service} - {amount} - שולם ב{pay}\",\n", "]\n", "\n", "# light typo injection to mimic real notes\n", "TYPO_MAP = {\"שילם\": \"שילמ\", \"קיבלתי\": \"קבלתי\", \"עבור\": \"עבר\", \"העביר\": \"העבירה\"}\n", "\n", "\n", "def _maybe_typo(rng: random.Random, text: str) -> str:\n", " if rng.random() < 0.15:\n", " for k, v in TYPO_MAP.items():\n", " if k in text and rng.random() < 0.5:\n", " text = text.replace(k, v, 1)\n", " break\n", " return text\n", "\n", "\n", "# =====================================================================\n", "# 5. SAMPLE GENERATION (parse target + raw text)\n", "# =====================================================================\n", "\n", "@dataclass\n", "class Meta:\n", " pool: str\n", " issuer_status: str\n", " issuer_is_company: bool\n", " doc_type: str\n", " vat_rate: float\n", " category: str = \"\"\n", " client_is_business: bool = False\n", " payment_method: str = \"\"\n", " year: int = 0\n", " paraphrased: bool = False\n", "\n", "\n", "def _pick_date(rng: random.Random) -> _dt.date:\n", " \"\"\"~10% in 2024 (legacy 17% VAT), the rest across 2025-2026.\"\"\"\n", " if rng.random() < 0.10:\n", " start = _dt.date(2024, 1, 1); span = 365\n", " else:\n", " start = _dt.date(2025, 1, 1); span = 530\n", " return start + _dt.timedelta(days=rng.randint(0, span - 1))\n", "\n", "\n", "def generate_sample(rng: random.Random, pool: str) -> dict:\n", " \"\"\"Produce one record's ground truth: issuer, parse target, raw_text, meta.\"\"\"\n", " p = POOLS[pool]\n", " issuer = make_issuer(rng, pool)\n", "\n", " # ---- client (business client more likely to trigger allocation rule) ----\n", " client_is_business = rng.random() < 0.45\n", " if client_is_business:\n", " client_name = rng.choice(p[\"client_biz\"])\n", " client_tax_id = make_company_number(rng) if rng.random() < 0.7 else make_israeli_id(rng)\n", " else:\n", " client_name = f\"{rng.choice(p['client_first'])} {rng.choice(p['client_last'])}\"\n", " client_tax_id = make_israeli_id(rng) if rng.random() < 0.25 else None\n", "\n", " # ---- items (1-3 line items from one service category) ----\n", " # Large B2B jobs cross the allocation threshold; ~55% of (non-exempt) business\n", " # clients get one so the allocation slice is meaningful. Exempt dealers (osek\n", " # patur, turnover-capped) only issue small single-item invoices.\n", " exempt = issuer.status == ISSUER_EXEMPT\n", " large = client_is_business and not exempt and rng.random() < 0.55\n", " if large:\n", " category = rng.choice(list(p[\"highvalue\"].keys()))\n", " catalog = p[\"highvalue\"][category]\n", " n_items = rng.choices([1, 2, 3], weights=[0.55, 0.30, 0.15])[0]\n", " qty_pool, qty_w = [1, 2, 3], [0.6, 0.28, 0.12]\n", " elif exempt:\n", " category = rng.choice([\"tutoring\", \"trades\", \"catering\", \"photography\", \"design\", \"marketing\"])\n", " catalog = p[\"services\"][category]\n", " n_items = 1\n", " qty_pool, qty_w = [1, 2], [0.85, 0.15]\n", " else:\n", " category = rng.choice(list(p[\"services\"].keys()))\n", " catalog = p[\"services\"][category]\n", " n_items = rng.choices([1, 2, 3], weights=[0.7, 0.22, 0.08])[0]\n", " qty_pool, qty_w = [1, 2, 3, 4], [0.75, 0.15, 0.06, 0.04]\n", " items = []\n", " for _ in range(n_items):\n", " desc, lo, hi = rng.choice(catalog)\n", " unit = rng.randint(lo, hi)\n", " qty = rng.choices(qty_pool, weights=qty_w)[0]\n", " items.append({\"description\": desc, \"unit_price\": float(unit), \"quantity\": qty})\n", "\n", " date = _pick_date(rng)\n", " payment = rng.choice(PAYMENT_METHODS)\n", "\n", " # ---- doc type is constrained by issuer status ----\n", " if issuer.status == ISSUER_EXEMPT:\n", " doc_type = \"receipt\" # exempt dealers cannot issue tax invoices\n", " else:\n", " doc_type = rng.choices(DOC_TYPES, weights=[0.30, 0.30, 0.40])[0]\n", "\n", " # amount stated in the note: gross (VAT-inclusive) or net\n", " amount_basis = rng.choices([\"gross\", \"net\"], weights=[0.6, 0.4])[0]\n", "\n", " parse = {\n", " \"client_name\": client_name,\n", " \"client_tax_id\": client_tax_id,\n", " \"client_is_business\": client_is_business,\n", " \"items\": items,\n", " \"amount_basis\": amount_basis,\n", " \"doc_type\": doc_type,\n", " \"payment_method\": payment,\n", " \"date\": date.isoformat(),\n", " \"currency\": \"ILS\",\n", " }\n", "\n", " # ---- raw_text: render a messy note from which `parse` is fully recoverable ----\n", " # States each item's quantity and UNIT price (never only a lump sum), so the\n", " # model extracts unit_price/quantity directly; complete() still does all the\n", " # arithmetic (line totals, VAT, subtotal, total).\n", " def _qty_desc(it):\n", " d, q = it[\"description\"], it[\"quantity\"]\n", " if q == 1:\n", " return d\n", " return rng.choice([f\"{q} {d}\", f\"{d} x{q}\", f\"{d} ({q} יח')\"])\n", "\n", " def _unit_phrase(it):\n", " u = int(it[\"unit_price\"])\n", " return rng.choice([f\"{u} כל אחד\", f\"{u} ליחידה\", f\"{u} ליח'\", f\"@{u}\"])\n", "\n", " if n_items == 1:\n", " it = items[0]\n", " if it[\"quantity\"] == 1:\n", " service_str = it[\"description\"]\n", " else:\n", " service_str = f\"{_qty_desc(it)}, {_unit_phrase(it)}\"\n", " headline_amount = int(it[\"unit_price\"] * it[\"quantity\"])\n", " else:\n", " parts = [f\"{_qty_desc(it)} {_unit_phrase(it)}\" for it in items]\n", " service_str = rng.choice([\", \", \" + \", \" ו\"]).join(parts)\n", " headline_amount = int(sum(it[\"unit_price\"] * it[\"quantity\"] for it in items))\n", "\n", " tmpl = rng.choice(RAW_TEMPLATES)\n", " raw = tmpl.format(\n", " amount=_amount_phrase(rng, headline_amount),\n", " client=client_name,\n", " service=service_str,\n", " pay=PAYMENT_HE[payment],\n", " )\n", " raw = _maybe_typo(rng, raw)\n", "\n", " meta = Meta(pool=pool, issuer_status=issuer.status,\n", " issuer_is_company=issuer.is_company, doc_type=doc_type,\n", " vat_rate=(0.0 if exempt else vat_rate_for_date(date)), category=category,\n", " client_is_business=client_is_business, payment_method=payment,\n", " year=date.year)\n", "\n", " return {\"issuer\": asdict(issuer), \"parse\": parse, \"raw_text\": raw, \"meta\": asdict(meta)}\n", "\n", "\n", "# =====================================================================\n", "# 6. complete() - DETERMINISTIC DOCUMENT COMPLETION\n", "# =====================================================================\n", "# Everything calculable is computed here. A model only ever predicts `parse`.\n", "\n", "_SERIAL_COUNTER = {\"value\": 1000}\n", "\n", "\n", "def _next_serial(rng: random.Random) -> int:\n", " _SERIAL_COUNTER[\"value\"] += rng.randint(1, 4)\n", " return _SERIAL_COUNTER[\"value\"]\n", "\n", "\n", "def complete(issuer: dict, parse: dict, rng: random.Random) -> dict:\n", " \"\"\"Compute a full, valid fiscal document from the issuer profile + parse.\n", "\n", " Returns the `completed` document. No randomness affects money; rng only\n", " advances serial numbers and the allocation token.\n", " \"\"\"\n", " d = _dt.date.fromisoformat(parse[\"date\"])\n", " exempt = issuer[\"status\"] == ISSUER_EXEMPT\n", " vat_rate = 0.0 if exempt else vat_rate_for_date(d)\n", "\n", " # line totals\n", " lines, gross_sum = [], 0.0\n", " for it in parse[\"items\"]:\n", " line_total = round(it[\"unit_price\"] * it[\"quantity\"], 2)\n", " gross_sum += line_total\n", " lines.append({**it, \"line_total\": line_total})\n", "\n", " # interpret stated amount basis -> subtotal (pre-VAT) and vat\n", " if exempt:\n", " subtotal = round(gross_sum, 2)\n", " vat_amount = 0.0\n", " elif parse[\"amount_basis\"] == \"gross\":\n", " subtotal = round(gross_sum / (1 + vat_rate), 2)\n", " vat_amount = round(gross_sum - subtotal, 2)\n", " else: # net\n", " subtotal = round(gross_sum, 2)\n", " vat_amount = round(subtotal * vat_rate, 2)\n", " total = round(subtotal + vat_amount, 2)\n", "\n", " # allocation number rule\n", " needs_alloc = (\n", " (not exempt)\n", " and parse[\"doc_type\"] in (\"tax_invoice\", \"tax_invoice_receipt\")\n", " and bool(parse.get(\"client_is_business\"))\n", " and subtotal >= allocation_threshold_for_date(d)\n", " )\n", " allocation_number = None\n", " if needs_alloc:\n", " allocation_number = f\"{rng.randint(10**8, 10**9 - 1)}\" # 9-digit token\n", "\n", " completed = {\n", " \"doc_type\": parse[\"doc_type\"],\n", " \"doc_type_he\": DOC_TYPE_HE[parse[\"doc_type\"]],\n", " \"serial_number\": _next_serial(rng),\n", " \"issue_date\": parse[\"date\"],\n", " \"issuer\": {\n", " \"name\": issuer[\"name\"],\n", " \"tax_id\": issuer[\"tax_id\"],\n", " \"status\": issuer[\"status\"],\n", " },\n", " \"client\": {\n", " \"name\": parse[\"client_name\"],\n", " \"tax_id\": parse[\"client_tax_id\"],\n", " \"is_business\": parse.get(\"client_is_business\", False),\n", " },\n", " \"lines\": lines,\n", " \"currency\": \"ILS\",\n", " \"subtotal\": subtotal,\n", " \"vat_rate\": vat_rate,\n", " \"vat_amount\": vat_amount,\n", " \"total\": total,\n", " \"payment_method\": parse[\"payment_method\"],\n", " \"allocation_required\": needs_alloc,\n", " \"allocation_number\": allocation_number,\n", " }\n", " return completed\n", "\n", "\n", "# =====================================================================\n", "# 7. INTEGRITY VALIDATION (zero-violation gate before publishing)\n", "# =====================================================================\n", "\n", "def validate_record(rec: dict) -> list[str]:\n", " \"\"\"Return a list of integrity violations for one record (empty == valid).\"\"\"\n", " v = []\n", " issuer, parse, comp = rec[\"issuer\"], rec[\"parse\"], rec[\"completed\"]\n", "\n", " # check digits\n", " if not is_valid_israeli_id(issuer[\"tax_id\"]):\n", " v.append(\"issuer_tax_id_checkdigit\")\n", " if parse.get(\"client_tax_id\") and not is_valid_israeli_id(parse[\"client_tax_id\"]):\n", " v.append(\"client_tax_id_checkdigit\")\n", "\n", " # exempt dealer constraints\n", " if issuer[\"status\"] == ISSUER_EXEMPT:\n", " if comp[\"vat_rate\"] != 0.0 or comp[\"vat_amount\"] != 0.0:\n", " v.append(\"exempt_dealer_has_vat\")\n", " if comp[\"doc_type\"] != \"receipt\":\n", " v.append(\"exempt_dealer_non_receipt\")\n", "\n", " # VAT rate matches the document date\n", " d = _dt.date.fromisoformat(comp[\"issue_date\"])\n", " if issuer[\"status\"] != ISSUER_EXEMPT and comp[\"vat_rate\"] != vat_rate_for_date(d):\n", " v.append(\"vat_rate_date_mismatch\")\n", "\n", " # money identity: subtotal + vat == total (to the agora)\n", " if round(comp[\"subtotal\"] + comp[\"vat_amount\"], 2) != comp[\"total\"]:\n", " v.append(\"money_identity\")\n", " if comp[\"subtotal\"] < 0 or comp[\"total\"] < 0:\n", " v.append(\"negative_money\")\n", "\n", " # allocation-number correctness (both directions)\n", " need = (\n", " issuer[\"status\"] != ISSUER_EXEMPT\n", " and comp[\"doc_type\"] in (\"tax_invoice\", \"tax_invoice_receipt\")\n", " and comp[\"client\"][\"is_business\"]\n", " and comp[\"subtotal\"] >= allocation_threshold_for_date(d)\n", " )\n", " if need and not comp[\"allocation_number\"]:\n", " v.append(\"allocation_missing\")\n", " if (not need) and comp[\"allocation_number\"]:\n", " v.append(\"allocation_unexpected\")\n", "\n", " return v\n", "\n", "# =====================================================================\n", "# 8. DATASET BUILD (A/B pools -> disjoint splits)\n", "# =====================================================================\n", "\n", "def _load_human_test(path: str = \"data/human_test.jsonl\") -> list:\n", " \"\"\"Load hand-written human_test notes from disk if present (else empty).\n", "\n", " These are real, hand-authored Hebrew notes with NEW issuers/clients\n", " (disjoint from pools A and B). Ground truth (`completed`) is still produced\n", " by `complete()`, so integrity holds. Returns records in the internal shape\n", " (with an `issuer` key) so validate_record() and the writers work unchanged.\n", " \"\"\"\n", " import os, json as _json\n", " if not os.path.exists(path):\n", " return []\n", " out = []\n", " with open(path, encoding=\"utf-8\") as f:\n", " for line in f:\n", " line = line.strip()\n", " if not line:\n", " continue\n", " r = _json.loads(line)\n", " # reconstruct internal issuer key from completed.issuer (writers strip it)\n", " if \"issuer\" not in r:\n", " r[\"issuer\"] = r[\"completed\"][\"issuer\"]\n", " out.append(r)\n", " return out\n", "\n", "def build_dataset(n_pool_a: int, n_pool_b: int, seed: int = 7) -> dict:\n", " \"\"\"Generate completed+validated records, split into train/val/iid/ood.\n", "\n", " Pool A -> train / validation / iid_test (80/10/10)\n", " Pool B -> ood_test (disjoint entities & templates)\n", " \"\"\"\n", " rng = random.Random(seed)\n", " _SERIAL_COUNTER[\"value\"] = 1000\n", "\n", " def gen(n, pool):\n", " out = []\n", " for _ in range(n):\n", " rec = generate_sample(rng, pool)\n", " rec[\"completed\"] = complete(rec[\"issuer\"], rec[\"parse\"], rng)\n", " out.append(rec)\n", " return out\n", "\n", " pool_a = gen(n_pool_a, \"A\")\n", " pool_b = gen(n_pool_b, \"B\")\n", "\n", " rng.shuffle(pool_a)\n", " n_tr = int(n_pool_a * 0.8)\n", " n_va = int(n_pool_a * 0.1)\n", " splits = {\n", " \"train\": pool_a[:n_tr],\n", " \"validation\": pool_a[n_tr:n_tr + n_va],\n", " \"iid_test\": pool_a[n_tr + n_va:],\n", " \"ood_test\": pool_b,\n", " \"human_test\": _load_human_test() # hand-written notes from data/human_test.jsonl\n", " }\n", " return splits\n", "\n", "\n", "def integrity_report(splits: dict) -> dict:\n", " \"\"\"Aggregate violations across all splits.\"\"\"\n", " counts, total, bad = {}, 0, 0\n", " for name, recs in splits.items():\n", " for r in recs:\n", " total += 1\n", " vs = validate_record(r)\n", " if vs:\n", " bad += 1\n", " for x in vs:\n", " counts[x] = counts.get(x, 0) + 1\n", " return {\"total\": total, \"violations\": bad, \"by_type\": counts}\n", "\n", "\n", "# =====================================================================\n", "# 9. SELF-TEST (dry run -> proves the module end-to-end)\n", "# =====================================================================\n", "\n", "if __name__ == \"__main__\":\n", " # check-digit sanity\n", " sample_id = make_israeli_id(random.Random(1))\n", " assert is_valid_israeli_id(sample_id), \"check-digit generator broken\"\n", "\n", " splits = build_dataset(n_pool_a=900, n_pool_b=100, seed=7)\n", " rep = integrity_report(splits)\n", " sizes = {k: len(v) for k, v in splits.items()}\n", " print(\"split sizes:\", sizes)\n", " print(\"integrity:\", rep)\n", " assert rep[\"violations\"] == 0, f\"INTEGRITY FAILURES: {rep}\"\n", "\n", " for ex in splits[\"train\"][:4]:\n", " print(\"\\nraw_text :\", ex[\"raw_text\"])\n", " print(\"client :\", ex[\"parse\"][\"client_name\"], \"| items:\", [it[\"description\"] for it in ex[\"parse\"][\"items\"]])\n", " print(\"doc :\", ex[\"completed\"][\"doc_type_he\"],\n", " \"| subtotal:\", ex[\"completed\"][\"subtotal\"],\n", " \"| vat:\", ex[\"completed\"][\"vat_amount\"],\n", " \"| total:\", ex[\"completed\"][\"total\"],\n", " \"| alloc:\", ex[\"completed\"][\"allocation_number\"])\n", " print(\"\\nOK - zero integrity violations.\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "R32HkXUgc29G" }, "outputs": [], "source": [ "import importlib, t2r_core\n", "importlib.reload(t2r_core)\n", "from t2r_core import build_dataset, integrity_report" ] }, { "cell_type": "markdown", "metadata": { "id": "ei_Q3lFYc4-D" }, "source": [ "## 2. Generate (disjoint A/B pools)\n", "\n", "Pool A -> `train` / `validation` / `iid_test` (80/10/10). Pool B -> `ood_test` (entities never seen in A)." ] }, { "cell_type": "markdown", "source": [ "### Upload the hand-written `human_test.jsonl`\n", "\n", "Before building, upload the 25 hand-authored notes so `build_dataset()` can fold them in as the\n", "`human_test` split. They use issuers/clients/services **disjoint from pools A and B** and their\n", "ground truth is produced by the same `complete()`, so they pass the identical integrity gate.\n", "*(If you skip this, the build simply omits the split.)*" ], "metadata": { "id": "NKomD8cXru9h" } }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 125 }, "id": "yvR7P5oBc4ZW", "outputId": "a100035c-bb39-45cb-c0e7-a9edc43701f2" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Pick human_test.jsonl:\n" ] }, { "output_type": "display_data", "data": { "text/plain": [ "" ], "text/html": [ "\n", " \n", " \n", " Upload widget is only available when the cell has been executed in the\n", " current browser session. Please rerun this cell to enable.\n", " \n", " " ] }, "metadata": {} }, { "output_type": "stream", "name": "stdout", "text": [ "Saving human_test.jsonl to human_test (1).jsonl\n", "saved -> data/human_test.jsonl\n", "human_test present: True | lines: 25\n" ] } ], "source": [ "# Upload data/human_test.jsonl (the 25 hand-written notes) into ./data/ before building.\n", "import os, shutil\n", "os.makedirs(DATA_DIR, exist_ok=True)\n", "try:\n", " from google.colab import files\n", " print(\"Pick human_test.jsonl:\")\n", " up = files.upload()\n", " for name, content in up.items():\n", " if name.endswith(\".jsonl\"):\n", " with open(os.path.join(DATA_DIR, \"human_test.jsonl\"), \"wb\") as f:\n", " f.write(content)\n", " print(\"saved -> data/human_test.jsonl\")\n", "except Exception as e:\n", " print(\"No Colab uploader (or skipped). If the file is already in data/, that's fine.\", e)\n", "\n", "_ht = os.path.join(DATA_DIR, \"human_test.jsonl\")\n", "print(\"human_test present:\", os.path.exists(_ht),\n", " \"| lines:\", sum(1 for _ in open(_ht, encoding=\"utf-8\")) if os.path.exists(_ht) else 0)" ] }, { "cell_type": "code", "source": [ "splits = build_dataset(n_pool_a=N_POOL_A, n_pool_b=N_POOL_B, seed=SEED)\n", "print(\"split sizes:\", {k: len(v) for k, v in splits.items()})\n", "if splits.get(\"human_test\"):\n", " print(\"human_test folded in:\", len(splits[\"human_test\"]), \"hand-written notes\")\n", "ex = splits[\"train\"][0]\n", "print(\"sample raw_text:\", ex[\"raw_text\"])\n", "print(\"sample completed:\", ex[\"completed\"][\"doc_type_he\"], \"| total\", ex[\"completed\"][\"total\"])" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "f2Tx0oUFryMz", "outputId": "7a826bb9-0431-4188-cb7c-612b76ec7bbf" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "split sizes: {'train': 7200, 'validation': 900, 'iid_test': 900, 'ood_test': 1000, 'human_test': 25}\n", "human_test folded in: 25 hand-written notes\n", "sample raw_text: קיבלתי 307₪ משקד אחזקות על החלפת ברז\n", "sample completed: קבלה | total 307.0\n" ] } ] }, { "cell_type": "markdown", "metadata": { "id": "8oCS2LuFc-lS" }, "source": [ "## 3. Paraphrase input notes (Hebrew-only)\n", "\n", "An instruct model rewrites a subset of notes into messier Hebrew. A strict guard keeps a rewrite only if it stays **clean Hebrew** (no Cyrillic / Arabic / CJK / accented-Latin drift) and preserves the amount and client name verbatim - otherwise the original template is kept." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Ur78Njjwc9vf" }, "outputs": [], "source": [ "def load_paraphraser():\n", " from transformers import AutoTokenizer, AutoModelForCausalLM\n", " import torch\n", " device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", " dtype = torch.float16 if device == \"cuda\" else torch.float32\n", " for model_id in (PRIMARY_MODEL, FALLBACK_MODEL):\n", " try:\n", " tok = AutoTokenizer.from_pretrained(model_id)\n", " tok.padding_side = \"left\"\n", " if tok.pad_token is None: tok.pad_token = tok.eos_token\n", " model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=dtype).to(device).eval()\n", " print(\"paraphraser:\", model_id, \"on\", device)\n", " SYS = (\"שכתב את הערת ההכנסה העסקית הקצרה הבאה לעברית טבעית ומעט לא מסודרת, \"\n", " \"כאילו עצמאי רשם אותה לעצמו. כללים נוקשים: \"\n", " \"(1) כתוב אך ורק בעברית; ללא אנגלית, רוסית, ערבית או כל שפה אחרת \"\n", " \"(ראשי תיבות טכניים כמו API מותר להשאיר). \"\n", " \"(2) שמור כל מספר, את שם הלקוח ואת שמות השירותים בדיוק כפי שהם. \"\n", " \"(3) אל תוסיף, תסיר או תשנה אף סכום. \"\n", " \"החזר אך ורק את ההערה המשוכתבת, בלי שום דבר נוסף.\")\n", " @torch.no_grad()\n", " def rewrite_batch(texts):\n", " prompts = [tok.apply_chat_template([{\"role\": \"user\", \"content\": f\"{SYS}\\n\\nהערה: {t}\"}],\n", " tokenize=False, add_generation_prompt=True) for t in texts]\n", " enc = tok(prompts, return_tensors=\"pt\", padding=True).to(device)\n", " out = model.generate(**enc, max_new_tokens=64, do_sample=True, temperature=0.9,\n", " top_p=0.95, pad_token_id=tok.pad_token_id)\n", " dec = tok.batch_decode(out[:, enc[\"input_ids\"].shape[1]:], skip_special_tokens=True)\n", " res = []\n", " for orig, g in zip(texts, dec):\n", " g = g.strip().splitlines()[0].strip().strip('\"').strip() if g.strip() else \"\"\n", " for _lbl in (\"הערה:\", \"הערה :\"): # drop echoed prompt label \"הערה:\"\n", " if g.startswith(_lbl): g = g[len(_lbl):].strip()\n", " res.append(g if 4 <= len(g) <= 160 else orig)\n", " return res\n", " return rewrite_batch\n", " except Exception as e:\n", " print(\" skip\", model_id, \":\", type(e).__name__)\n", " print(\"no paraphraser -> keeping templates\"); return None\n", "\n", "paraphraser = load_paraphraser() if PARAPHRASE_ENABLED else None" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "POCfRqkHdBUW", "outputId": "d9215268-2472-46a2-b50c-8710de87b119" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "paraphrased (kept): 0 / 4588 eligible (0.0%)\n" ] } ], "source": [ "# batched apply + Hebrew-only guard (amount, client name AND clean Hebrew must survive)\n", "import re\n", "_HEB = re.compile(r\"[֐-׿]\")\n", "_ALLOWED_EXTRA = set(\"₪–—\") # shekel sign + en/em dash (besides Hebrew + ASCII)\n", "\n", "def _is_clean_hebrew(t):\n", " # Reject any character outside {Hebrew block, ASCII, a few symbols} -> this blocks\n", " # Cyrillic, Arabic, CJK, accented Latin and emoji (the real paraphrase drift). Plain\n", " # ASCII acronyms (API/SEO) remain allowed, but the note must stay predominantly Hebrew.\n", " for ch in t:\n", " o = ord(ch)\n", " if not (0x0590 <= o <= 0x05FF or o < 128 or ch in _ALLOWED_EXTRA):\n", " return False\n", " heb = len(_HEB.findall(t))\n", " lat = sum(1 for ch in t if ch.isascii() and ch.isalpha())\n", " return heb >= 2 and lat <= heb\n", "\n", "def _amt(r): return str(int(sum(it[\"unit_price\"] * it[\"quantity\"] for it in r[\"parse\"][\"items\"])))\n", "def _passes(r, new):\n", " flat = new.replace(\",\", \"\").replace(\" \", \"\")\n", " return bool(new) and new != r[\"raw_text\"] and _is_clean_hebrew(new) \\\n", " and r[\"parse\"][\"client_name\"] in new and _amt(r) in flat\n", "\n", "PARA_BATCH = 16\n", "rng_p = random.Random(SEED)\n", "targets = [r for recs in splits.values() for r in recs if rng_p.random() < PARAPHRASE_FRACTION]\n", "# only attempt notes that already contain the client name + amount, so a rewrite can preserve them\n", "targets = [r for r in targets if r[\"parse\"][\"client_name\"] in r[\"raw_text\"]\n", " and _amt(r) in r[\"raw_text\"].replace(\",\", \"\").replace(\" \", \"\")]\n", "kept = 0\n", "if paraphraser is not None and targets:\n", " pending = list(targets)\n", " for attempt in range(PARA_ATTEMPTS):\n", " if not pending: break\n", " still = []\n", " for i in range(0, len(pending), PARA_BATCH):\n", " chunk = pending[i:i + PARA_BATCH]\n", " for r, new in zip(chunk, paraphraser([r[\"raw_text\"] for r in chunk])):\n", " if _passes(r, new):\n", " r[\"raw_text\"] = new; r[\"meta\"][\"paraphrased\"] = True; kept += 1\n", " else:\n", " still.append(r)\n", " pending = still\n", " print(f\" attempt {attempt + 1}/{PARA_ATTEMPTS}: kept {kept}, retry {len(pending)}\")\n", "print(f\"paraphrased (kept): {kept} / {len(targets)} eligible ({kept / max(len(targets), 1):.1%})\")" ] }, { "cell_type": "markdown", "metadata": { "id": "7nFUqI8-dE2V" }, "source": [ "## 4. Integrity gate\n", "\n", "Re-validates every record (check digits, VAT-by-date, money identity, allocation rule, exempt-dealer constraints). Publishing is **blocked** unless `violations == 0`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "en5Ywn4edCvT", "outputId": "fb5bcf1c-6b80-4e23-b220-825b3d88d7a0" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "integrity: {'total': 10025, 'violations': 0, 'by_type': {}}\n" ] } ], "source": [ "report = integrity_report(splits)\n", "print(\"integrity:\", report)\n", "assert report[\"violations\"] == 0, f\"NOT publishing: {report}\"" ] }, { "cell_type": "markdown", "metadata": { "id": "DBmhu5tYdNfL" }, "source": [ "## 5. Write splits + stats" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "wdqm-T4idM1a", "outputId": "64804944-cdba-48bf-a280-9feb4aee386b" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "{\n", " \"total\": 10025,\n", " \"splits\": {\n", " \"train\": 7200,\n", " \"validation\": 900,\n", " \"iid_test\": 900,\n", " \"ood_test\": 1000,\n", " \"human_test\": 25\n", " },\n", " \"vat\": {\n", " \"0%\": 2477,\n", " \"18%\": 6816,\n", " \"17%\": 732\n", " },\n", " \"doc_types\": {\n", " \"receipt\": 4702,\n", " \"tax_invoice_receipt\": 2985,\n", " \"tax_invoice\": 2338\n", " },\n", " \"categories\": {\n", " \"design\": 1613,\n", " \"marketing\": 1562,\n", " \"consulting\": 1164,\n", " \"software\": 1157,\n", " \"photography\": 1146,\n", " \"tutoring\": 1143,\n", " \"catering\": 1131,\n", " \"trades\": 1084,\n", " \"human\": 25\n", " },\n", " \"allocation_required\": 1029,\n", " \"paraphrased\": 0,\n", " \"integrity_violations\": 0,\n", " \"generated_at\": \"2026-06-21T10:54:08.906406+00:00\"\n", "}\n" ] } ], "source": [ "# clear any stale jsonl from a previous run, then write splits + a small stats file\n", "for fn in os.listdir(DATA_DIR):\n", " if fn.endswith(\".jsonl\"):\n", " os.remove(os.path.join(DATA_DIR, fn))\n", "\n", "def rec(r): return {\"raw_text\": r[\"raw_text\"], \"parse\": r[\"parse\"], \"completed\": r[\"completed\"], \"meta\": r[\"meta\"]}\n", "for name, recs in splits.items():\n", " if not recs: # skip empty splits (human_test) so no empty file is published\n", " continue\n", " with open(os.path.join(DATA_DIR, f\"{name}.jsonl\"), \"w\", encoding=\"utf-8\") as f:\n", " for r in recs: f.write(json.dumps(rec(r), ensure_ascii=False) + \"\\n\")\n", "\n", "allrec = [r for recs in splits.values() for r in recs]\n", "cc = collections.Counter(r[\"meta\"][\"category\"] for r in allrec)\n", "stats = {\"total\": len(allrec), \"splits\": {k: len(v) for k, v in splits.items() if v},\n", " \"vat\": dict(collections.Counter(f\"{r['completed']['vat_rate']:.0%}\" for r in allrec)),\n", " \"doc_types\": dict(collections.Counter(r[\"completed\"][\"doc_type\"] for r in allrec)),\n", " \"categories\": dict(cc.most_common()),\n", " \"allocation_required\": int(sum(r[\"completed\"][\"allocation_required\"] for r in allrec)),\n", " \"paraphrased\": int(sum(bool(r[\"meta\"].get(\"paraphrased\")) for r in allrec)),\n", " \"integrity_violations\": report[\"violations\"],\n", " \"generated_at\": dt.datetime.now(dt.timezone.utc).isoformat()}\n", "with open(os.path.join(DATA_DIR, \"dataset_stats.json\"), \"w\", encoding=\"utf-8\") as f:\n", " json.dump(stats, f, ensure_ascii=False, indent=2)\n", "print(json.dumps(stats, ensure_ascii=False, indent=2))" ] }, { "cell_type": "markdown", "metadata": { "id": "X_zoT-h-dQ6D" }, "source": [ "## 6. Minimal dataset card\n", "\n", "Only the YAML front-matter the HF viewer needs (so splits render), plus the title. The full, maximalist analysis lives in `02_eda` - not on the dataset page." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "lsRAcUOadPTv", "outputId": "0121cf7f-0b97-49d8-a43d-daf74080a1bb" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "README.md written (minimal):\n", "---\n", "language: [he]\n", "license: mit\n", "task_categories: [text-generation]\n", "tags: [hebrew, receipts, invoices, israeli-tax, synthetic, structured-output]\n", "pretty_name: Text2Receipt\n", "size_categories: [1K Secrets). The dataset publishes itself end-to-end." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "k9FI2KLHdSz7", "outputId": "f7983b5b-f6da-468d-b6ae-b46038b94931" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "published: https://huggingface.co/datasets/yonilev/Text2Receipt\n", "layout -> README.md | data/ (5 splits + stats) | src/t2r_core.py | notebooks/ (next cell)\n" ] } ], "source": [ "if PUSH_TO_HUB:\n", " from huggingface_hub import HfApi, login\n", " try:\n", " from google.colab import userdata\n", " login(token=userdata.get(\"HF_TOKEN\")) # Colab Secret\n", " except Exception:\n", " login() # interactive fallback\n", "\n", " api = HfApi()\n", " api.create_repo(HF_DATASET_REPO, repo_type=\"dataset\", exist_ok=True)\n", " # clean only the legacy flat-layout file; keep data/ (incl. human_test) intact\n", " for stale in (\"t2r_core.py\",):\n", " try: api.delete_file(stale, repo_id=HF_DATASET_REPO, repo_type=\"dataset\")\n", " except Exception: pass\n", "\n", " api.upload_file(path_or_fileobj=\"README.md\", path_in_repo=\"README.md\", repo_id=HF_DATASET_REPO, repo_type=\"dataset\")\n", " api.upload_file(path_or_fileobj=\"t2r_core.py\", path_in_repo=\"src/t2r_core.py\", repo_id=HF_DATASET_REPO, repo_type=\"dataset\")\n", " api.upload_folder(folder_path=DATA_DIR, path_in_repo=\"data\", repo_id=HF_DATASET_REPO, repo_type=\"dataset\")\n", " print(\"published:\", f\"https://huggingface.co/datasets/{HF_DATASET_REPO}\")\n", " n_splits = 5 if os.path.exists(os.path.join(DATA_DIR, \"human_test.jsonl\")) else 4\n", " print(f\"layout -> README.md | data/ ({n_splits} splits + stats) | src/t2r_core.py | notebooks/ (next cell)\")\n", "else:\n", " print(\"PUSH_TO_HUB is False - set it True to publish.\")\n" ] }, { "cell_type": "code", "source": [ "# Archive all THREE project notebooks into notebooks/ in the dataset repo.\n", "# Pick 01_data_generation.ipynb, 02_eda.ipynb, 03_models.ipynb together (Ctrl/Cmd-click).\n", "if PUSH_TO_HUB:\n", " from huggingface_hub import HfApi\n", " api = HfApi()\n", " # map any uploaded filename to its canonical repo path by keyword\n", " def _dest(name):\n", " low = name.lower()\n", " if \"01\" in low or \"data_generation\" in low or \"dataset_creation\" in low:\n", " return \"notebooks/01_data_generation.ipynb\"\n", " if \"02\" in low or \"eda\" in low:\n", " return \"notebooks/02_eda.ipynb\"\n", " if \"03\" in low or \"model\" in low:\n", " return \"notebooks/03_models.ipynb\"\n", " return f\"notebooks/{name}\"\n", " try:\n", " from google.colab import files\n", " print(\"Pick the .ipynb files to archive (Ctrl/Cmd-click for all three):\")\n", " up = files.upload()\n", " for name, content in up.items():\n", " if name.endswith(\".ipynb\"):\n", " dest = _dest(name)\n", " api.upload_file(path_or_fileobj=content, path_in_repo=dest,\n", " repo_id=HF_DATASET_REPO, repo_type=\"dataset\")\n", " print(f\"uploaded {name} -> {dest}\")\n", " except Exception as e:\n", " print(\"Use the Colab file picker here, or drag the .ipynb into notebooks/ via the HF web UI.\", e)\n" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 108 }, "id": "EUGAUHsK4djj", "outputId": "e911f47a-a1fb-423f-b71c-46f6773e25aa" }, "execution_count": null, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Pick the .ipynb files to archive (Ctrl/Cmd-click for all three):\n" ] }, { "output_type": "display_data", "data": { "text/plain": [ "" ], "text/html": [ "\n", " \n", " \n", " Upload widget is only available when the cell has been executed in the\n", " current browser session. Please rerun this cell to enable.\n", " \n", " " ] }, "metadata": {} }, { "output_type": "stream", "name": "stdout", "text": [ "Saving Dataset_Creation (6).ipynb to Dataset_Creation (6).ipynb\n", "uploaded Dataset_Creation (6).ipynb -> notebooks/01_data_generation.ipynb\n" ] } ] }, { "cell_type": "markdown", "metadata": { "id": "cOtFcO3udcI8" }, "source": [ "✔ Built with **zero integrity violations**. Five splits published: `train` / `validation` / `iid_test` (pool A), `ood_test` (pool B, disjoint), and `human_test` (pool H, 25 hand-written notes). Next: **02_eda** reads this dataset and writes the full card." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "e7MaXwoldeEB" }, "outputs": [], "source": [] } ], "metadata": { "colab": { "provenance": [] }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 0 }