title

LICENSE

@@ -1,19 +0,0 @@
-Local AI - Offline Multi-Modal Document Intelligence System
-Copyright (C) 2026 the project authors
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-The full text of the GNU General Public License v3.0 is available at:
-https://www.gnu.org/licenses/gpl-3.0.txt
-
-You should have received a copy of the GNU General Public License
-along with this program. If you did not, obtain one from the URL above
-before distributing this software.

README.md

@@ -1,248 +1,19 @@
-<<<<<<< HEAD
-# Local AI — Offline Multi-Modal Document Intelligence System
-
-Turn images, PDFs, plain text, audio, and video into structured JSON — **100% offline, CPU-only.**
-No cloud APIs, no API keys, no telemetry, no internet dependency after setup.
-
-Built with **Streamlit** (not React/FastAPI — swapped per project requirements for a single-process,
-easy-to-run app), Tesseract OCR, and a lightweight rule-based local extraction engine, with an
-optional upgrade path to a real local LLM via `llama-cpp-python`.
-
 ---
-
-## ✅ What's fully working out of the box (zero extra downloads)
-
-- **Images → OCR → JSON** (Tesseract, native)
-- **PDFs → text/OCR → JSON** (`pdfplumber`, with automatic OCR fallback for scanned pages)
-- **Plain text → JSON**
-- Automatic **document type classification** (invoice, resume, receipt, meeting notes, contract,
-  medical report, letter, research paper, certificate, general)
-- Rule-based **entity extraction**: people, emails, phones, dates, amounts, organizations,
-  invoice numbers, keywords, tags, summary, confidence score
-- **SQLite storage**, full **History** view, **Search**, **Dashboard** with charts
-- **Export** to JSON / CSV / Excel
-- **Duplicate detection** via SHA-256 file hashing
-- **Demo Mode** (one click, no upload needed)
-
-This mode (`rules` in Settings) is what the app uses by default and what was tested end-to-end
-while building this — see "What was tested" below.
-
-## 🔌 Optional upgrades (require one-time internet access on *your* machine)
-
-| Feature | Install | Notes |
-|---|---|---|
-| Real local LLM extraction | `pip install llama-cpp-python` + a `.gguf` model in `models/` | Switch to "llm" mode in Settings. See `backend/llm_local.py` for model recommendations (TinyLlama, Phi-3-mini, Qwen2.5-3B). |
-| Audio transcription | `pip install openai-whisper` + `ffmpeg` on PATH | Powers `.mp3/.wav/.m4a` uploads |
-| Video transcription | same as above | Extracts audio via ffmpeg, then transcribes |
-
-Once installed, these still run **entirely locally / CPU-only** — the "offline" install step is a
-one-time download of the model file, not a runtime API call.
-
----
-
-## Quickstart
-
-```bash
-git clone <this-repo>
-cd local-ai
-python -m venv venv && source venv/bin/activate   # optional but recommended
-pip install -r requirements.txt
-
-# Tesseract binary (required for OCR):
-#   Ubuntu/Debian:  sudo apt install tesseract-ocr
-#   macOS:          brew install tesseract
-#   Windows:        https://github.com/UB-Mannheim/tesseract/wiki
-
-streamlit run app.py
-```
-
-Open the URL Streamlit prints (usually http://localhost:8501). Try **Home → Run Demo Mode**, or
-upload a file from `sample_docs/`.
-
+title: Offlinenotesanalyzer
+emoji: 🚀
+colorFrom: red
+colorTo: red
+sdk: docker
+app_port: 8501
+tags:
+- streamlit
+pinned: false
+short_description: Streamlit template space
 ---
 
-## Architecture
-
-```
-Input (image/pdf/text/audio/video)
-        │
-        ▼
-   OCR / Transcription      (Tesseract / pdfplumber / Whisper)
-        │
-        ▼
-   Text Cleaning
-        │
-        ▼
-   Local AI Extraction      (rule-based engine, or local LLM via llama.cpp)
-        │
-        ▼
-   Structured JSON
-        │
-        ▼
-   SQLite Storage  ──►  Dashboard / History / Search / Export
-```
-
-## Folder structure
-
-```
-local-ai/
-├── app.py                # Streamlit UI (Home, Upload, Dashboard, History, Search, Settings)
-├── backend/
-│   ├── ocr.py             # Tesseract image OCR
-│   ├── pdf_extract.py     # PDF text extraction + OCR fallback for scans
-│   ├── extractor.py       # Document classification + rule-based JSON extraction
-│   ├── llm_local.py       # Optional llama-cpp-python local LLM extraction
-│   ├── audio.py           # Optional Whisper transcription
-│   ├── video.py           # Optional ffmpeg audio extraction + Whisper
-│   ├── pipeline.py         # Orchestrates the full flow + status updates
-│   ├── db.py               # SQLite storage (stdlib sqlite3, no ORM dependency)
-│   └── exporters.py        # JSON / CSV / Excel export
-├── prompts/
-│   └── templates.py        # Prompt templates per document type (used in LLM mode)
-├── sample_docs/            # Sample files for quick testing / Demo Mode
-├── models/                 # Drop a .gguf model here to enable LLM mode
-├── data/                   # SQLite database lives here (created at runtime)
-├── requirements.txt
-├── LICENSE                 # GPLv3
-└── README.md
-```
-
-## API surface (internal, called by the Streamlit UI)
-
-These are plain Python functions, not HTTP endpoints (no separate backend server is
-needed since Streamlit handles both UI and processing in one process):
-
-- `pipeline.process_file(filename, bytes, mode, model_path, ocr_lang, status_callback)`
-- `db.list_documents()`, `db.search_documents(query)`, `db.get_document(id)`, `db.delete_document(id)`, `db.stats()`
-- `exporters.to_json_bytes(rows)`, `to_csv_bytes(rows)`, `to_excel_bytes(rows)`
-
-If you specifically need a separate REST API (e.g. to call from another app), the same
-`backend/` modules can be wrapped in FastAPI with a thin `main.py` — they have no
-Streamlit dependency themselves.
-
-## What was tested while building this (in a sandboxed, no-internet environment)
-
-- Image OCR on a generated test image ✅
-- PDF native-text extraction on a generated test PDF ✅
-- Document classification across invoice/resume/meeting-notes samples ✅
-- Full entity extraction (emails, phones, dates, amounts, orgs, names) ✅
-- SQLite insert/list/search/stats/delete ✅
-- Duplicate detection via hashing ✅
-- JSON/CSV/Excel export ✅
-
-Not testable in that environment (no internet to install/download): the Streamlit UI render
-itself, `llama-cpp-python` LLM mode, and Whisper audio/video transcription. The code for all
-three is complete and follows each library's standard, stable API — install the optional
-dependencies above to light them up.
-
-## Security & privacy
-
-No cloud calls, no telemetry, no analytics, no external API keys anywhere in this codebase.
-All processing and storage stays on the machine running `streamlit run app.py`.
-
-## License
-
-GPLv3 — see [LICENSE](LICENSE).
-
-## Future improvements
-
-- Folder-watch mode for auto-processing new files
-- Custom JSON schema builder in Settings
-- Batch processing queue with pause/resume/retry
-- Side-by-side original-document + JSON viewer with click-to-highlight
-- Multi-user auth if deployed beyond a single local user
-=======
-# offline notes analyser
-
-
-
-## Getting started
-
-To make it easy for you to get started with GitLab, here's a list of recommended next steps.
-
-Already a pro? Just edit this README.md and make it your own. Want to make it easy? [Use the template at the bottom](#editing-this-readme)!
-
-## Add your files
-
-* [Create](https://docs.gitlab.com/user/project/repository/web_editor/#create-a-file) or [upload](https://docs.gitlab.com/user/project/repository/web_editor/#upload-a-file) files
-* [Add files using the command line](https://docs.gitlab.com/topics/git/add_files/#add-files-to-a-git-repository) or push an existing Git repository with the following command:
-
-```
-cd existing_repo
-git remote add origin https://code.swecha.org/sriharshini2901/offline-notes-analyser.git
-git branch -M main
-git push -uf origin main
-```
-
-## Integrate with your tools
-
-* [Set up project integrations](https://code.swecha.org/sriharshini2901/offline-notes-analyser/-/settings/integrations)
-
-## Collaborate with your team
-
-* [Invite team members and collaborators](https://docs.gitlab.com/user/project/members/)
-* [Create a new merge request](https://docs.gitlab.com/user/project/merge_requests/creating_merge_requests/)
-* [Automatically close issues from merge requests](https://docs.gitlab.com/user/project/issues/managing_issues/#closing-issues-automatically)
-* [Enable merge request approvals](https://docs.gitlab.com/user/project/merge_requests/approvals/)
-* [Set auto-merge](https://docs.gitlab.com/user/project/merge_requests/auto_merge/)
-
-## Test and Deploy
-
-Use the built-in continuous integration in GitLab.
-
-* [Get started with GitLab CI/CD](https://docs.gitlab.com/ci/quick_start/)
-* [Analyze your code for known vulnerabilities with Static Application Security Testing (SAST)](https://docs.gitlab.com/user/application_security/sast/)
-* [Deploy to Kubernetes, Amazon EC2, or Amazon ECS using Auto Deploy](https://docs.gitlab.com/topics/autodevops/requirements/)
-* [Use pull-based deployments for improved Kubernetes management](https://docs.gitlab.com/user/clusters/agent/)
-* [Set up protected environments](https://docs.gitlab.com/ci/environments/protected_environments/)
-
-***
-
-# Editing this README
-
-When you're ready to make this README your own, just edit this file and use the handy template below (or feel free to structure it however you want - this is just a starting point!). Thanks to [makeareadme.com](https://www.makeareadme.com/) for this template.
-
-## Suggestions for a good README
-
-Every project is different, so consider which of these sections apply to yours. The sections used in the template are suggestions for most open source projects. Also keep in mind that while a README can be too long and detailed, too long is better than too short. If you think your README is too long, consider utilizing another form of documentation rather than cutting out information.
-
-## Name
-Choose a self-explaining name for your project.
-
-## Description
-Let people know what your project can do specifically. Provide context and add a link to any reference visitors might be unfamiliar with. A list of Features or a Background subsection can also be added here. If there are alternatives to your project, this is a good place to list differentiating factors.
-
-## Badges
-On some READMEs, you may see small images that convey metadata, such as whether or not all the tests are passing for the project. You can use Shields to add some to your README. Many services also have instructions for adding a badge.
-
-## Visuals
-Depending on what you are making, it can be a good idea to include screenshots or even a video (you'll frequently see GIFs rather than actual videos). Tools like ttygif can help, but check out Asciinema for a more sophisticated method.
-
-## Installation
-Within a particular ecosystem, there may be a common way of installing things, such as using Yarn, NuGet, or Homebrew. However, consider the possibility that whoever is reading your README is a novice and would like more guidance. Listing specific steps helps remove ambiguity and gets people to using your project as quickly as possible. If it only runs in a specific context like a particular programming language version or operating system or has dependencies that have to be installed manually, also add a Requirements subsection.
-
-## Usage
-Use examples liberally, and show the expected output if you can. It's helpful to have inline the smallest example of usage that you can demonstrate, while providing links to more sophisticated examples if they are too long to reasonably include in the README.
-
-## Support
-Tell people where they can go to for help. It can be any combination of an issue tracker, a chat room, an email address, etc.
-
-## Roadmap
-If you have ideas for releases in the future, it is a good idea to list them in the README.
-
-## Contributing
-State if you are open to contributions and what your requirements are for accepting them.
-
-For people who want to make changes to your project, it's helpful to have some documentation on how to get started. Perhaps there is a script that they should run or some environment variables that they need to set. Make these steps explicit. These instructions could also be useful to your future self.
-
-You can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser.
-
-## Authors and acknowledgment
-Show your appreciation to those who have contributed to the project.
+# Welcome to Streamlit!
 
-## License
-For open source projects, say how it is licensed.
+Edit `/src/streamlit_app.py` to customize this app to your heart's desire. :heart:
 
-## Project status
-If you have run out of energy or time for your project, put a note at the top of the README saying that development has slowed down or stopped completely. Someone may choose to fork your project or volunteer to step in as a maintainer or owner, allowing your project to keep going. You can also make an explicit request for maintainers.
->>>>>>> 5198c728ba5184d464aecc5842b792739ae0ed27
+If you have any questions, checkout our [documentation](https://docs.streamlit.io) and [community
+forums](https://discuss.streamlit.io).

app.py

@@ -1,182 +0,0 @@
-"""
-app.py - Offline Notes Analyser
-Streamlit front-end. Run with:  streamlit run app.py
-"""
-import sys
-import os
-import json
-import time
-import logging
-
-logging.getLogger("pdfminer").setLevel(logging.ERROR)
-
-sys.path.insert(0, os.path.dirname(__file__))
-
-import streamlit as st
-import pandas as pd
-
-from backend import db, pipeline, exporters
-
-st.set_page_config(
-    page_title="Offline Notes Analyser",
-    page_icon="🧠",
-    layout="wide",
-    initial_sidebar_state="expanded",
-)
-
-db.init_db()
-
-# Fixed settings - kept simple, no settings page.
-PIPELINE_MODE = "rules"          # switch to "llm" in code if you set up a local model
-MODEL_PATH = "models/model.gguf"
-OCR_LANG = "eng"
-
-if "page" not in st.session_state:
-    st.session_state.page = "Analyze"
-
-# ------------------------------------------------------------------ style --
-st.markdown("""
-<style>
-.metric-card {
-    border:1px solid #2a2a2a22; border-radius:12px; padding:16px;
-    background:linear-gradient(135deg, #f8fafc 0%, #eef2ff 100%);
-}
-</style>
-""", unsafe_allow_html=True)
-
-# ---------------------------------------------------------------- sidebar --
-with st.sidebar:
-    st.markdown("## 🧠 Offline Notes Analyser")
-    st.caption("Upload a file, get structured notes, search them later.")
-    st.session_state.page = st.radio(
-        "Navigate",
-        ["Analyze", "History"],
-        index=["Analyze", "History"].index(st.session_state.page),
-        label_visibility="collapsed",
-    )
-    st.divider()
-    s = db.stats()
-    st.metric("Notes analyzed", s["total_documents"])
-    st.success("🟢 100% Offline")
-
-
-# ===================================================== ANALYZE (UPLOAD) ===
-def render_analyze():
-    st.title("📤 Analyze")
-    st.caption("Drop a file below. Everything happens locally on this machine.")
-
-    uploaded = st.file_uploader(
-        "Choose a file",
-        type=["jpg", "jpeg", "png", "pdf", "txt", "mp3", "wav", "m4a", "mp4", "mov", "mkv"],
-        accept_multiple_files=True,
-    )
-
-    if not uploaded:
-        st.info("Upload an image, PDF, text file, audio, or video to begin.")
-        return
-
-    for f in uploaded:
-        filename, file_bytes = f.name, f.read()
-        with st.container(border=True):
-            st.markdown(f"**{filename}**")
-            status_box = st.empty()
-            progress = st.progress(0)
-            steps_seen = []
-
-            def on_step(step_name, _seen=steps_seen, _box=status_box, _prog=progress):
-                _seen.append(step_name)
-                _box.write(" → ".join(_seen))
-                _prog.progress(min(0.95, len(_seen) * 0.15))
-
-            try:
-                result = pipeline.process_file(
-                    filename, file_bytes,
-                    mode=PIPELINE_MODE, model_path=MODEL_PATH, ocr_lang=OCR_LANG,
-                    status_callback=on_step,
-                )
-                progress.progress(1.0)
-            except Exception as e:
-                st.error(f"Processing failed: {e}")
-                continue
-
-            if result.get("duplicate"):
-                st.warning(f"⚠️ Already analyzed before — see document #{result['existing_id']} in History.")
-                continue
-
-            c1, c2, c3 = st.columns(3)
-            c1.metric("Type", result["doc_type"])
-            c2.metric("Confidence", f"{result['confidence']*100:.0f}%")
-            c3.metric("Time", f"{result['processing_time']}s")
-
-            tab1, tab2 = st.tabs(["📋 Structured result", "📝 Raw text"])
-            with tab1:
-                st.json(result["extracted"])
-                st.download_button(
-                    "⬇️ Download JSON",
-                    data=json.dumps(result["extracted"], indent=2),
-                    file_name=f"{filename}.json",
-                    mime="application/json",
-                    key=f"dl_{filename}_{result['id']}",
-                )
-                st.caption("Want CSV or Excel instead? Open this document in the **History** tab to download it in any format.")
-            with tab2:
-                st.text_area("Raw extracted text", result["raw_text"], height=200, key=f"raw_{result['id']}")
-
-    st.success("Done — find this anytime in History.")
-
-
-# ================================================ HISTORY (+ SEARCH) ======
-def render_history():
-    st.title("🗂️ History")
-
-    query = st.text_input("🔍 Search by name, email, company, invoice number, keyword…")
-    rows = db.search_documents(query) if query else db.list_documents(limit=500)
-
-    if not rows:
-        st.info("Nothing here yet — analyze a file first." if not query else "No matches.")
-        return
-
-    col1, col2, col3 = st.columns(3)
-    with col1:
-        try:
-            st.download_button("⬇️ Export all (JSON)", exporters.to_json_bytes(rows), "notes.json", "application/json")
-        except Exception as e:
-            st.error(f"JSON export failed: {e}")
-    with col2:
-        try:
-            st.download_button("⬇️ Export all (CSV)", exporters.to_csv_bytes(rows), "notes.csv", "text/csv")
-        except Exception as e:
-            st.error(f"CSV export failed: {e}")
-    with col3:
-        try:
-            st.download_button("⬇️ Export all (Excel)", exporters.to_excel_bytes(rows), "notes.xlsx",
-                                "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet")
-        except Exception as e:
-            st.error(f"Excel export failed: {e}")
-
-    st.divider()
-    for r in rows:
-        with st.expander(f"#{r['id']} · {r['filename']} · {r['doc_type']} · {r['confidence']*100:.0f}% confidence"):
-            st.caption(time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(r["created_at"])))
-            extracted = json.loads(r["extracted_json"])
-            st.json(extracted)
-            c_json, c_csv, c_xlsx, c_del = st.columns(4)
-            with c_json:
-                st.download_button("⬇️ JSON", json.dumps(extracted, indent=2),
-                                    f"{r['filename']}.json", key=f"hist_json_{r['id']}")
-            with c_csv:
-                st.download_button("⬇️ CSV", exporters.to_csv_bytes([r]),
-                                    f"{r['filename']}.csv", "text/csv", key=f"hist_csv_{r['id']}")
-            with c_xlsx:
-                st.download_button("⬇️ Excel", exporters.to_excel_bytes([r]),
-                                    f"{r['filename']}.xlsx",
-                                    "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
-                                    key=f"hist_xlsx_{r['id']}")
-            with c_del:
-                if st.button("🗑️ Delete", key=f"hist_del_{r['id']}"):
-                    db.delete_document(r["id"])
-                    st.rerun()
-
-
-PAGES = {"Analyze": render_analyze, "History": render_history}
-PAGES[st.session_state.page]()

audio.py

@@ -1,53 +0,0 @@
-"""
-audio.py - Offline speech-to-text via OpenAI's open-source Whisper model,
-run locally on CPU (NOT the OpenAI cloud API - this is the open-weights
-model running entirely on your machine).
-
-One-time setup on your own machine (needs internet just to install):
-    pip install openai-whisper
-    (also requires ffmpeg installed on the system PATH)
-
-After that, transcription runs fully offline / CPU-only.
-"""
-import tempfile
-import os
-
-_MODEL_CACHE = {}
-
-
-def _get_model(size: str = "base"):
-    if size in _MODEL_CACHE:
-        return _MODEL_CACHE[size]
-    try:
-        import whisper  # lazy import - only needed if audio/video features are used
-    except ImportError:
-        raise RuntimeError(
-            "Audio/video transcription needs the 'openai-whisper' package, "
-            "which isn't installed.\nRun:  pip install openai-whisper\n"
-            "(also requires ffmpeg installed on your system PATH)"
-        )
-    model = whisper.load_model(size)  # downloads once, then cached locally & used offline
-    _MODEL_CACHE[size] = model
-    return model
-
-
-def transcribe_audio(audio_bytes: bytes, filename_hint: str = "audio.wav", model_size: str = "base") -> dict:
-    suffix = os.path.splitext(filename_hint)[1] or ".wav"
-    with tempfile.NamedTemporaryFile(suffix=suffix, delete=False) as tmp:
-        tmp.write(audio_bytes)
-        tmp_path = tmp.name
-
-    try:
-        model = _get_model(model_size)
-        result = model.transcribe(tmp_path)
-        text = result.get("text", "").strip()
-        # Whisper doesn't give a single confidence number; approximate from
-        # average per-segment no-speech probability (inverted)
-        segments = result.get("segments", [])
-        if segments:
-            avg_conf = 1 - (sum(s.get("no_speech_prob", 0.2) for s in segments) / len(segments))
-        else:
-            avg_conf = 0.7
-        return {"text": text, "confidence": round(max(0.0, min(1.0, avg_conf)), 3)}
-    finally:
-        os.unlink(tmp_path)

db.py

@@ -1,122 +0,0 @@
-"""
-db.py - SQLite storage layer for Local AI.
-
-Uses Python's built-in sqlite3 module (zero extra dependencies, fully offline).
-"""
-import sqlite3
-import json
-import os
-import time
-import hashlib
-from contextlib import contextmanager
-from typing import Optional
-
-DB_PATH = os.path.join(os.path.dirname(os.path.dirname(__file__)), "data", "local_ai.db")
-
-
-SCHEMA = """
-CREATE TABLE IF NOT EXISTS documents (
-    id INTEGER PRIMARY KEY AUTOINCREMENT,
-    filename TEXT NOT NULL,
-    file_hash TEXT,
-    doc_type TEXT,
-    raw_text TEXT,
-    extracted_json TEXT,
-    confidence REAL,
-    processing_time REAL,
-    created_at REAL,
-    pipeline_mode TEXT
-);
-CREATE INDEX IF NOT EXISTS idx_doc_type ON documents(doc_type);
-CREATE INDEX IF NOT EXISTS idx_file_hash ON documents(file_hash);
-"""
-
-
-@contextmanager
-def get_conn():
-    os.makedirs(os.path.dirname(DB_PATH), exist_ok=True)
-    conn = sqlite3.connect(DB_PATH)
-    conn.row_factory = sqlite3.Row
-    try:
-        yield conn
-        conn.commit()
-    finally:
-        conn.close()
-
-
-def init_db():
-    with get_conn() as conn:
-        conn.executescript(SCHEMA)
-
-
-def file_hash_bytes(data: bytes) -> str:
-    return hashlib.sha256(data).hexdigest()
-
-
-def find_by_hash(file_hash: str) -> Optional[sqlite3.Row]:
-    with get_conn() as conn:
-        cur = conn.execute("SELECT * FROM documents WHERE file_hash = ? LIMIT 1", (file_hash,))
-        return cur.fetchone()
-
-
-def insert_document(filename, file_hash, doc_type, raw_text, extracted: dict,
-                     confidence: float, processing_time: float, pipeline_mode: str) -> int:
-    with get_conn() as conn:
-        cur = conn.execute(
-            """INSERT INTO documents
-               (filename, file_hash, doc_type, raw_text, extracted_json,
-                confidence, processing_time, created_at, pipeline_mode)
-               VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)""",
-            (filename, file_hash, doc_type, raw_text, json.dumps(extracted),
-             confidence, processing_time, time.time(), pipeline_mode),
-        )
-        return cur.lastrowid
-
-
-def list_documents(limit: int = 200):
-    with get_conn() as conn:
-        cur = conn.execute(
-            "SELECT * FROM documents ORDER BY created_at DESC LIMIT ?", (limit,)
-        )
-        return cur.fetchall()
-
-
-def get_document(doc_id: int):
-    with get_conn() as conn:
-        cur = conn.execute("SELECT * FROM documents WHERE id = ?", (doc_id,))
-        return cur.fetchone()
-
-
-def delete_document(doc_id: int):
-    with get_conn() as conn:
-        conn.execute("DELETE FROM documents WHERE id = ?", (doc_id,))
-
-
-def search_documents(query: str):
-    q = f"%{query.lower()}%"
-    with get_conn() as conn:
-        cur = conn.execute(
-            """SELECT * FROM documents
-               WHERE lower(filename) LIKE ?
-                  OR lower(raw_text) LIKE ?
-                  OR lower(extracted_json) LIKE ?
-               ORDER BY created_at DESC""",
-            (q, q, q),
-        )
-        return cur.fetchall()
-
-
-def stats():
-    with get_conn() as conn:
-        total = conn.execute("SELECT COUNT(*) c FROM documents").fetchone()["c"]
-        avg_conf = conn.execute("SELECT AVG(confidence) c FROM documents").fetchone()["c"] or 0
-        avg_time = conn.execute("SELECT AVG(processing_time) c FROM documents").fetchone()["c"] or 0
-        by_type = conn.execute(
-            "SELECT doc_type, COUNT(*) c FROM documents GROUP BY doc_type"
-        ).fetchall()
-        return {
-            "total_documents": total,
-            "avg_confidence": round(avg_conf, 3),
-            "avg_processing_time": round(avg_time, 3),
-            "by_type": {r["doc_type"]: r["c"] for r in by_type},
-        }

exporters.py

@@ -1,44 +0,0 @@
-"""
-exporters.py - Export stored documents as JSON, CSV, or Excel.
-"""
-import json
-import io
-import pandas as pd
-
-
-def to_json_bytes(rows) -> bytes:
-    docs = []
-    for r in rows:
-        docs.append({
-            "id": r["id"], "filename": r["filename"], "doc_type": r["doc_type"],
-            "confidence": r["confidence"], "processing_time": r["processing_time"],
-            "extracted": json.loads(r["extracted_json"]),
-        })
-    return json.dumps(docs, indent=2).encode("utf-8")
-
-
-def _flatten_rows(rows):
-    flat = []
-    for r in rows:
-        extracted = json.loads(r["extracted_json"])
-        row = {
-            "id": r["id"], "filename": r["filename"], "doc_type": r["doc_type"],
-            "confidence": r["confidence"], "processing_time": r["processing_time"],
-        }
-        for k, v in extracted.items():
-            row[k] = json.dumps(v) if isinstance(v, (list, dict)) else v
-        flat.append(row)
-    return flat
-
-
-def to_csv_bytes(rows) -> bytes:
-    df = pd.DataFrame(_flatten_rows(rows))
-    return df.to_csv(index=False).encode("utf-8")
-
-
-def to_excel_bytes(rows) -> bytes:
-    df = pd.DataFrame(_flatten_rows(rows))
-    buf = io.BytesIO()
-    with pd.ExcelWriter(buf, engine="openpyxl") as writer:
-        df.to_excel(writer, index=False, sheet_name="Documents")
-    return buf.getvalue()

extractor.py

@@ -1,147 +0,0 @@
-"""
-extractor.py - Turns cleaned text into structured JSON, 100% offline.
-
-Two modes (selectable in Settings):
-  - "rules"  : fast regex/heuristic extraction. No model download needed,
-               works immediately, zero extra dependencies. This is the
-               default and is what powers the app out of the box.
-  - "llm"    : routes through backend/llm_local.py, which calls a local
-               quantized GGUF model via llama-cpp-python (CPU only).
-               Requires the user to `pip install llama-cpp-python` and
-               place a .gguf model in models/ (see README) - both need
-               internet/disk access on the user's own machine.
-
-Both modes return the same JSON shape so the rest of the app doesn't care
-which one produced it.
-"""
-import re
-from collections import Counter
-
-STOPWORDS = set("""
-a an the and or but if of in on at to for with from by is are was were be
-been being this that these those it its as not no yes you your we our
-they their he she his her i me my mine us them than then so such can will
-would should could may might must do does did have has had
-""".split())
-
-EMAIL_RE = re.compile(r"[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+")
-PHONE_RE = re.compile(r"(?<!\w)(\+?\d{1,3}[\s.-])?\(?\d{3}\)?[\s.-]\d{3}[\s.-]\d{4}(?!\w)")
-DATE_RE = re.compile(
-    r"\b(\d{1,2}[/-]\d{1,2}[/-]\d{2,4}|\d{4}-\d{2}-\d{2}|"
-    r"(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[a-z]*\.?\s+\d{1,2},?\s+\d{4})\b",
-    re.IGNORECASE,
-)
-AMOUNT_RE = re.compile(r"(?:[$₹€£]\s?\d[\d,]*\.?\d{0,2}|\b\d[\d,]*\.\d{2}\b)")
-INVOICE_NO_RE = re.compile(r"\b(?:invoice|inv|receipt|order)[\s#:no.]*([A-Z0-9-]{4,})\b", re.IGNORECASE)
-NAME_LINE_RE = re.compile(r"\b([A-Z][a-z]+(?:[ \t]+[A-Z][a-z]+){1,2})\b")
-NAME_BLOCKLIST = {
-    "bill to", "amount due", "invoice no", "ship to", "pay to", "total due",
-    "due date", "order no", "purchase order", "account number", "sold to",
-    "action items", "meeting notes", "product roadmap",
-}
-ORG_HINT_RE = re.compile(
-    r"\b([A-Z][\w&]*(?:\s+[A-Z][\w&]*)*\s+(?:Inc\.?|LLC|Ltd\.?|Corp\.?|Company|Co\.|Technologies|Solutions|Group))\b"
-)
-
-DOC_TYPE_KEYWORDS = {
-    "invoice": ["invoice", "bill to", "subtotal", "tax", "amount due", "po number"],
-    "receipt": ["receipt", "total", "cash", "change due", "thank you for your purchase"],
-    "resume": ["experience", "education", "skills", "resume", "curriculum vitae", "objective"],
-    "medical_report": ["diagnosis", "patient", "physician", "prescribed", "symptoms", "treatment"],
-    "meeting_notes": ["agenda", "attendees", "action items", "minutes of meeting", "next steps"],
-    "contract": ["agreement", "party", "parties", "hereby", "terms and conditions", "termination"],
-    "letter": ["dear", "sincerely", "regards", "yours truly"],
-    "research_paper": ["abstract", "references", "introduction", "methodology", "conclusion"],
-    "certificate": ["certificate", "certify", "awarded", "completion"],
-}
-
-
-def classify_document(text: str) -> str:
-    lower = text.lower()
-    scores = {}
-    for doc_type, keywords in DOC_TYPE_KEYWORDS.items():
-        scores[doc_type] = sum(lower.count(k) for k in keywords)
-    best_type, best_score = max(scores.items(), key=lambda kv: kv[1])
-    return best_type if best_score > 0 else "general"
-
-
-def _top_keywords(text: str, n: int = 10) -> list:
-    words = re.findall(r"[A-Za-z]{4,}", text.lower())
-    words = [w for w in words if w not in STOPWORDS]
-    return [w for w, _ in Counter(words).most_common(n)]
-
-
-def _summary(text: str, max_sentences: int = 2) -> str:
-    sentences = re.split(r"(?<=[.!?])\s+", text.strip())
-    return " ".join(sentences[:max_sentences])[:400]
-
-
-def extract_entities(text: str) -> dict:
-    """Common entities every document type shares."""
-    emails = sorted(set(EMAIL_RE.findall(text)))
-    phones = sorted(set(m.group(0).strip() for m in PHONE_RE.finditer(text)))
-    dates = sorted(set(DATE_RE.findall(text)))
-    amounts = sorted(set(AMOUNT_RE.findall(text)))
-    orgs = sorted(set(ORG_HINT_RE.findall(text)))
-    invoice_match = INVOICE_NO_RE.search(text)
-    names_raw = set(NAME_LINE_RE.findall(text))
-    names = sorted(n for n in names_raw if n.lower() not in NAME_BLOCKLIST and n not in orgs)[:10]
-
-    return {
-        "people": names,
-        "emails": emails,
-        "phones": phones,
-        "dates": dates,
-        "amounts": amounts,
-        "organizations": orgs,
-        "invoice_number": invoice_match.group(1) if invoice_match else None,
-    }
-
-
-def rule_based_extract(text: str, doc_type: str) -> dict:
-    entities = extract_entities(text)
-    result = {
-        "document_category": doc_type,
-        "summary": _summary(text),
-        "tags": _top_keywords(text, 8),
-        "keywords": _top_keywords(text, 15),
-        **entities,
-    }
-
-    # Type-specific shaping so the JSON looks purpose-built, not generic
-    if doc_type == "resume":
-        result["full_name"] = entities["people"][0] if entities["people"] else None
-        result["email"] = entities["emails"][0] if entities["emails"] else None
-        result["phone"] = entities["phones"][0] if entities["phones"] else None
-        skill_kw = ["python", "java", "sql", "react", "fastapi", "excel", "design",
-                    "leadership", "communication", "javascript", "aws", "docker"]
-        result["skills"] = [k for k in skill_kw if k in text.lower()]
-    elif doc_type in ("invoice", "receipt"):
-        result["total_amount"] = entities["amounts"][-1] if entities["amounts"] else None
-        result["vendor"] = entities["organizations"][0] if entities["organizations"] else None
-        result["date"] = entities["dates"][0] if entities["dates"] else None
-
-    # crude confidence: more populated fields = higher confidence
-    populated = sum(1 for v in result.values() if v)
-    result["confidence_score"] = round(min(0.95, 0.35 + populated * 0.05), 2)
-    return result
-
-
-def extract(text: str, doc_type: str = None, mode: str = "rules", model_path: str = None) -> dict:
-    """Main entry point. `mode` is 'rules' (default, always works offline)
-    or 'llm' (requires llama-cpp-python + a local GGUF model)."""
-    if not doc_type:
-        doc_type = classify_document(text)
-
-    if mode == "llm":
-        from . import llm_local
-        try:
-            return llm_local.extract_with_llm(text, doc_type, model_path)
-        except Exception as e:
-            # Always fall back to rule-based so the pipeline never hard-fails
-            result = rule_based_extract(text, doc_type)
-            result["_llm_error"] = str(e)
-            result["_fallback_used"] = "rules"
-            return result
-
-    return rule_based_extract(text, doc_type)

llm_local.py

@@ -1,73 +0,0 @@
-"""
-llm_local.py - Optional local LLM extraction via llama-cpp-python.
-
-This module is only imported when the user enables "LLM mode" in Settings.
-It is NOT required for the app to work - the default "rules" mode (see
-extractor.py) needs zero extra downloads.
-
-To enable real local-LLM extraction on your own machine (with internet
-access, just for the one-time setup):
-    pip install llama-cpp-python
-    Download a small instruction-tuned GGUF model, e.g.:
-      - TinyLlama-1.1B-Chat (~600MB)
-      - Phi-3-mini-4k-instruct (Q4_K_M, ~2.3GB)
-      - Qwen2.5-3B-Instruct (Q4_K_M, ~2GB)
-    Place the .gguf file in models/ and set its path in the Settings page
-    (or LOCAL_AI_MODEL_PATH env var).
-
-After that one-time setup, everything still runs 100% offline / CPU-only -
-no API keys, no cloud calls.
-"""
-import json
-import re
-import os
-
-_MODEL_CACHE = {}
-
-
-def _get_model(model_path: str):
-    if not model_path or not os.path.exists(model_path):
-        raise FileNotFoundError(
-            f"No GGUF model found at '{model_path}'. Download one and set "
-            f"its path in Settings (see backend/llm_local.py docstring)."
-        )
-    if model_path in _MODEL_CACHE:
-        return _MODEL_CACHE[model_path]
-
-    from llama_cpp import Llama  # lazy import - only needed for LLM mode
-    llm = Llama(model_path=model_path, n_ctx=4096, n_threads=os.cpu_count(), verbose=False)
-    _MODEL_CACHE[model_path] = llm
-    return llm
-
-
-def _extract_json(raw: str) -> dict:
-    """Models sometimes wrap JSON in markdown fences or add stray text -
-    pull out the first {...} block."""
-    raw = raw.strip()
-    raw = re.sub(r"^```(json)?", "", raw).strip()
-    raw = re.sub(r"```$", "", raw).strip()
-    match = re.search(r"\{.*\}", raw, re.DOTALL)
-    if not match:
-        raise ValueError("Model did not return parseable JSON")
-    return json.loads(match.group(0))
-
-
-def extract_with_llm(text: str, doc_type: str, model_path: str = None) -> dict:
-    from prompts.templates import build_prompt
-
-    model_path = model_path or os.environ.get("LOCAL_AI_MODEL_PATH", "models/model.gguf")
-    llm = _get_model(model_path)
-    prompt = build_prompt(doc_type, text)
-
-    output = llm(
-        prompt,
-        max_tokens=800,
-        temperature=0.1,
-        stop=["</s>", "```"],
-    )
-    raw_text = output["choices"][0]["text"]
-    result = _extract_json(raw_text)
-    result.setdefault("document_category", doc_type)
-    result.setdefault("confidence_score", 0.8)
-    result["_pipeline_mode"] = "llm"
-    return result

ocr.py

@@ -1,44 +0,0 @@
-"""
-ocr.py - Image OCR using Tesseract (offline, CPU-only).
-
-Tesseract is a mature, fully offline OCR engine - no internet or cloud
-calls are made. Requires the `tesseract` binary + `pytesseract` package.
-"""
-from PIL import Image
-import pytesseract
-import io
-
-
-def ocr_image(image_bytes: bytes, lang: str = "eng") -> dict:
-    """Run OCR on raw image bytes. Returns text + a rough confidence score."""
-    img = Image.open(io.BytesIO(image_bytes))
-    if img.mode != "RGB":
-        img = img.convert("RGB")
-
-    try:
-        text = pytesseract.image_to_string(img, lang=lang)
-    except pytesseract.TesseractNotFoundError:
-        raise RuntimeError(
-            "Tesseract OCR engine is not installed on this machine.\n"
-            "Install it, then restart the app:\n"
-            "  Windows: https://github.com/UB-Mannheim/tesseract/wiki\n"
-            "  macOS:   brew install tesseract\n"
-            "  Linux:   sudo apt install tesseract-ocr"
-        )
-
-    # image_to_data gives per-word confidence; average the valid ones
-    try:
-        data = pytesseract.image_to_data(img, lang=lang, output_type=pytesseract.Output.DICT)
-        confs = [int(c) for c in data["conf"] if c not in ("-1", -1)]
-        avg_conf = (sum(confs) / len(confs) / 100.0) if confs else 0.5
-    except Exception:
-        avg_conf = 0.5
-
-    return {"text": text.strip(), "confidence": round(avg_conf, 3)}
-
-
-def available_languages() -> list:
-    try:
-        return pytesseract.get_languages(config="")
-    except Exception:
-        return ["eng"]

pdf_extract.py

@@ -1,48 +0,0 @@
-"""
-pdf_extract.py - Extract text from PDFs (offline).
-
-Strategy:
-  1. Try pdfplumber for native text extraction (fast, accurate for text PDFs).
-  2. If a page has no extractable text (scanned/image PDF), rasterize it
-     and run it through Tesseract OCR.
-"""
-import io
-import pdfplumber
-from PIL import Image
-from . import ocr
-
-
-def extract_pdf(pdf_bytes: bytes, lang: str = "eng") -> dict:
-    pages_text = []
-    ocr_used = False
-    confidences = []
-
-    with pdfplumber.open(io.BytesIO(pdf_bytes)) as pdf:
-        for page in pdf.pages:
-            text = (page.extract_text() or "").strip()
-            if text:
-                pages_text.append(text)
-                confidences.append(0.95)  # native text is high-confidence
-            else:
-                # Scanned page -> rasterize + OCR
-                try:
-                    im = page.to_image(resolution=200).original
-                    buf = io.BytesIO()
-                    im.save(buf, format="PNG")
-                    result = ocr.ocr_image(buf.getvalue(), lang=lang)
-                    pages_text.append(result["text"])
-                    confidences.append(result["confidence"])
-                    ocr_used = True
-                except Exception as e:
-                    pages_text.append("")
-                    confidences.append(0.0)
-
-    full_text = "\n\n".join(p for p in pages_text if p)
-    avg_conf = (sum(confidences) / len(confidences)) if confidences else 0.0
-
-    return {
-        "text": full_text.strip(),
-        "page_count": len(pages_text),
-        "ocr_used": ocr_used,
-        "confidence": round(avg_conf, 3),
-    }

pipeline.py

@@ -1,110 +0,0 @@
-"""
-pipeline.py - Orchestrates the full Input -> Clean -> Extract -> JSON -> SQLite
-flow for every supported input type. Yields step-by-step status so the UI
-can show live progress.
-"""
-import time
-from . import ocr, pdf_extract, extractor, db
-
-STEPS_BY_TYPE = {
-    "image": ["Uploading", "Running OCR", "Cleaning text", "AI extracting", "Generating JSON", "Saving to database"],
-    "pdf": ["Uploading", "Extracting pages", "Cleaning text", "AI extracting", "Generating JSON", "Saving to database"],
-    "text": ["Uploading", "Cleaning text", "AI extracting", "Generating JSON", "Saving to database"],
-    "audio": ["Uploading", "Transcribing (Whisper)", "Cleaning text", "AI extracting", "Generating JSON", "Saving to database"],
-    "video": ["Uploading", "Extracting audio", "Transcribing (Whisper)", "Cleaning text", "AI extracting", "Generating JSON", "Saving to database"],
-}
-
-
-def _clean_text(text: str) -> str:
-    return "\n".join(line.strip() for line in text.splitlines() if line.strip())
-
-
-def file_kind(filename: str) -> str:
-    ext = filename.lower().rsplit(".", 1)[-1] if "." in filename else ""
-    if ext in ("jpg", "jpeg", "png"):
-        return "image"
-    if ext == "pdf":
-        return "pdf"
-    if ext == "txt":
-        return "text"
-    if ext in ("mp3", "wav", "m4a"):
-        return "audio"
-    if ext in ("mp4", "mov", "mkv"):
-        return "video"
-    return "unknown"
-
-
-def process_file(filename: str, file_bytes: bytes, mode: str = "rules",
-                  model_path: str = None, ocr_lang: str = "eng",
-                  status_callback=None):
-    """Runs the full pipeline. status_callback(step_name) is called for each
-    stage if provided (used by the Streamlit UI to show live progress)."""
-    start = time.time()
-    kind = file_kind(filename)
-    if kind == "unknown":
-        raise ValueError(f"Unsupported file type: {filename}")
-
-    def report(step):
-        if status_callback:
-            status_callback(step)
-
-    steps = iter(STEPS_BY_TYPE[kind])
-    report(next(steps))  # Uploading
-
-    file_hash = db.file_hash_bytes(file_bytes)
-    existing = db.find_by_hash(file_hash)
-    if existing:
-        return {"duplicate": True, "existing_id": existing["id"]}
-
-    raw_text = ""
-    ocr_confidence = 1.0
-
-    if kind == "image":
-        report(next(steps))
-        r = ocr.ocr_image(file_bytes, lang=ocr_lang)
-        raw_text, ocr_confidence = r["text"], r["confidence"]
-    elif kind == "pdf":
-        report(next(steps))
-        r = pdf_extract.extract_pdf(file_bytes, lang=ocr_lang)
-        raw_text, ocr_confidence = r["text"], r["confidence"]
-    elif kind == "text":
-        raw_text = file_bytes.decode("utf-8", errors="ignore")
-    elif kind == "audio":
-        report(next(steps))
-        from . import audio as audio_mod
-        r = audio_mod.transcribe_audio(file_bytes, filename_hint=filename)
-        raw_text, ocr_confidence = r["text"], r["confidence"]
-    elif kind == "video":
-        report(next(steps))
-        from . import video as video_mod
-        report(next(steps))
-        r = video_mod.transcribe_video(file_bytes, filename_hint=filename)
-        raw_text, ocr_confidence = r["text"], r["confidence"]
-
-    report(next(steps))  # Cleaning text
-    cleaned = _clean_text(raw_text)
-
-    report(next(steps))  # AI extracting
-    doc_type = extractor.classify_document(cleaned)
-    extracted = extractor.extract(cleaned, doc_type=doc_type, mode=mode, model_path=model_path)
-
-    report(next(steps))  # Generating JSON
-    confidence = round((ocr_confidence + extracted.get("confidence_score", 0.7)) / 2, 3)
-
-    report(next(steps))  # Saving to database
-    processing_time = round(time.time() - start, 3)
-    doc_id = db.insert_document(
-        filename=filename, file_hash=file_hash, doc_type=doc_type,
-        raw_text=cleaned, extracted=extracted, confidence=confidence,
-        processing_time=processing_time, pipeline_mode=mode,
-    )
-
-    return {
-        "duplicate": False,
-        "id": doc_id,
-        "doc_type": doc_type,
-        "raw_text": cleaned,
-        "extracted": extracted,
-        "confidence": confidence,
-        "processing_time": processing_time,
-    }

requirements.txt

@@ -1,20 +1,3 @@
-# Core app (required)
-streamlit>=1.35
+altair
 pandas
-openpyxl
-pillow
-pytesseract
-pdfplumber
-pypdf
-
-# Note: pytesseract requires the `tesseract` binary on your system:
-#   Ubuntu/Debian:  sudo apt install tesseract-ocr
-#   macOS:          brew install tesseract
-#   Windows:        https://github.com/UB-Mannheim/tesseract/wiki
-
-# ---- Optional: enable real local-LLM extraction mode (Settings > LLM) ----
-# llama-cpp-python
-
-# ---- Optional: enable audio/video transcription ----
-# openai-whisper
-# (also requires ffmpeg on your system PATH: sudo apt install ffmpeg)
+streamlit

video.py

@@ -1,34 +0,0 @@
-"""
-video.py - Extracts the audio track from a video file (using ffmpeg,
-which must be installed locally) and transcribes it via audio.py.
-
-Requires ffmpeg on PATH (e.g. `apt install ffmpeg` / `brew install ffmpeg`).
-No video data ever leaves the machine.
-"""
-import subprocess
-import tempfile
-import os
-from . import audio
-
-
-def transcribe_video(video_bytes: bytes, filename_hint: str = "video.mp4", model_size: str = "base") -> dict:
-    suffix = os.path.splitext(filename_hint)[1] or ".mp4"
-    with tempfile.NamedTemporaryFile(suffix=suffix, delete=False) as vtmp:
-        vtmp.write(video_bytes)
-        video_path = vtmp.name
-
-    audio_path = video_path + ".wav"
-    try:
-        subprocess.run(
-            ["ffmpeg", "-y", "-i", video_path, "-vn", "-ac", "1", "-ar", "16000", audio_path],
-            check=True, capture_output=True,
-        )
-        with open(audio_path, "rb") as f:
-            audio_bytes = f.read()
-        return audio.transcribe_audio(audio_bytes, filename_hint="extracted.wav", model_size=model_size)
-    except FileNotFoundError:
-        raise RuntimeError("ffmpeg not found. Install ffmpeg to enable video processing.")
-    finally:
-        for p in (video_path, audio_path):
-            if os.path.exists(p):
-                os.unlink(p)