Buckets:

---

EMTIAZZ

/

velaops-data-estate-bucket

TL;DR — get everything in one command

pip install -U huggingface_hub
hf download EMTIAZZ/velaops-data-estate --repo-type dataset --local-dir ./velaops-data-estate

That downloads the entire estate (every CSV, PDF, scanned image, email, and transcript) into ./velaops-data-estate. See Downloading the data for more ways.

---

The task

VelaOps builds internal AI assistants for businesses. A decade-old insurer has hired VelaOps to build one on top of their own data — a huge, messy pile collected over ten years: millions of claim documents and scanned forms (PDFs, images), years of CRM and ERP records, support emails, and call-center transcripts spread across several databases and cloud buckets. They want a RAG-based assistant so staff can ask natural-language questions about policies, claims, and customers, plus a fine-tuned domain LLM and a vision model that extracts fields from scanned forms.

Questions

1. How would you design the pipeline to ingest this large, heterogeneous data (structured DB/CRM/ERP records plus unstructured PDFs, images, and emails)? Where would you use batch vs. streaming, and which tools would you choose?
2. The source data is noisy, duplicated, and full of PII. How would you sanitize, deduplicate, and normalize it to meet the quality bar needed for retrieval and fine-tuning?
3. Walk through turning cleaned documents into retrieval-ready assets: chunking strategy, embedding pipeline, and vector store choice. How do you keep it efficient across millions of documents?
4. How would you prepare and version the datasets for fine-tuning both the LLM and the vision model? What does the pipeline and MLOps setup look like?
5. Once live, how do you monitor pipeline health, data drift, and retrieval quality? And how do you handle governance and lineage as schemas evolve?

This sample holds thousands of records, not millions. Design for millions and demonstrate the approach on this representative slice.

---

What's in here (data dictionary)

velaops-data-estate/
├── structured/                  # "the databases" — load these into Postgres
│   ├── crm_customers.csv         # CRM system: customers + PII
│   ├── erp_policies.csv          # ERP system: policies (DIFFERENT naming + date format)
│   ├── claims.csv                # claims, linked to policies + customers
│   └── postgres_dump.sql         # CREATE TABLE + INSERTs for all three
├── documents/                   # "the scanned forms / claim documents"
│   ├── pdf/<claim_no>.pdf         # 400 claim-form PDFs
│   └── scans/<claim_no>.png       # the same 400 forms as degraded "scanned" images
├── emails/thread_*.txt          # 600 support email threads (quoted history + PII signatures)
├── transcripts/call_*.txt       # 300 call-center transcripts (disfluencies + spoken PII)
├── real/                        # additional support-conversation datasets
│   ├── customer_support_faqs/data.csv
│   └── bitext_customer_support/data.csv
├── *_index.json                 # machine-readable indexes (emails/calls/documents)
├── MANIFEST.json                # counts + linkage + known quality issues
├── examples/quickstart.py       # runnable end-to-end example (load, join, OCR-ready, read text)
└── generate.py                  # the generator (full transparency on the synthetic data)

Structured tables (in structured/)

Unstructured data

• documents/scans/<claim_no>.png — claim forms rendered then degraded (grayscale, rotation, speckle noise, blur) to look like real scans. Run OCR / a vision model here.
• documents/pdf/<claim_no>.pdf — the clean PDF version of each form.
• emails/thread_*.txt — multi-turn support threads with quoted reply history; customer PII appears in signatures.
• transcripts/call_*.txt — agent/customer call-center dialogue with disfluencies ("um", "uh") and PII spoken aloud (phone, DOB).

How everything links (the important part)

crm_customers.customer_id  ─┐
                            ├─►  erp_policies.CustomerID
                            │        └─ erp_policies.PolicyNo ─► claims.policy_no
                            └─►  claims.customer_id
                                          └─ claims.claim_no ─► documents/{pdf,scans}/<claim_no>
                                                              ─► emails (subject + body reference)
                                                              ─► transcripts (header + body reference)

The same person appears across a claim, a scanned form, an email, and a transcript. That's what makes cross-source dedup / normalization / lineage a real exercise.

Deliberate quality problems (the whole point)

• Near-duplicate customer rows: different surrogate keys for the same person, casing / whitespace / typo drift, reformatted phone numbers, .dup@ emails, missing fields.
• Schema mismatch: CRM snake_case + ISO dates vs. ERP PascalCase + MM/DD/YYYY.
• PII in free text: emails, transcripts, and scanned forms.
• Noisy scans: real OCR difficulty.

Counts

---

Downloading the data

Option A — CLI, whole dataset (recommended)

pip install -U huggingface_hub
hf download EMTIAZZ/velaops-data-estate --repo-type dataset --local-dir ./velaops-data-estate

Option B — Python, whole dataset

from huggingface_hub import snapshot_download
path = snapshot_download(repo_id="EMTIAZZ/velaops-data-estate",
                         repo_type="dataset", local_dir="velaops-data-estate")
print("downloaded to:", path)

Option C — only part of it (e.g. just the scans, or just the tables)

from huggingface_hub import snapshot_download
# only the scanned images:
snapshot_download("EMTIAZZ/velaops-data-estate", repo_type="dataset",
                  local_dir="scans_only", allow_patterns=["documents/scans/*"])
# only the structured tables:
snapshot_download("EMTIAZZ/velaops-data-estate", repo_type="dataset",
                  local_dir="tables_only", allow_patterns=["structured/*"])

Option D — a single file, no download tooling

import pandas as pd
url = "https://huggingface.co/datasets/EMTIAZZ/velaops-data-estate/resolve/main/structured/claims.csv"
claims = pd.read_csv(url)          # loads directly over HTTP

# or with curl:
curl -L -o claims.csv "https://huggingface.co/datasets/EMTIAZZ/velaops-data-estate/resolve/main/structured/claims.csv"

Option E — git clone (needs git-lfs for the images/PDFs)

git lfs install
git clone https://huggingface.co/datasets/EMTIAZZ/velaops-data-estate

---

Using the data

Load + join the structured side

import pandas as pd
root = "velaops-data-estate"          # from Option A/B
cust   = pd.read_csv(f"{root}/structured/crm_customers.csv")
pol    = pd.read_csv(f"{root}/structured/erp_policies.csv")
claims = pd.read_csv(f"{root}/structured/claims.csv")

# join across the three "systems" (note the column-name mismatch):
full = (claims
        .merge(cust, on="customer_id", how="left")
        .merge(pol, left_on="policy_no", right_on="PolicyNo", how="left"))
print(full.shape)

Load the tables into Postgres ("several databases")

createdb velaops
psql -d velaops -f velaops-data-estate/structured/postgres_dump.sql

Read a scanned form + match it to ground truth

from PIL import Image
import pandas as pd, glob, os

root = "velaops-data-estate"
scan = sorted(glob.glob(f"{root}/documents/scans/*.png"))[0]
claim_no = os.path.basename(scan).replace(".png", "")
img = Image.open(scan)                       # feed this to your OCR / vision model
truth = pd.read_csv(f"{root}/structured/claims.csv").query("claim_no == @claim_no")
print(claim_no, img.size, "\nground truth:\n", truth.T)

Read an email thread / a transcript

print(open(f"{root}/emails/thread_0000.txt").read())
print(open(f"{root}/transcripts/call_0000.txt").read())

Or just run the bundled example

python velaops-data-estate/examples/quickstart.py velaops-data-estate

---

Hints (per question)

• Q1 (ingest): treat structured/ as CDC/batch DB loads, and documents/ emails/ transcripts/ as object-store/streaming sources. Land everything in a staging zone keyed by claim_no / customer_id.
• Q2 (clean/dedup/PII): the duplicate customers share a real identity but differ in key, casing, and formatting — fuzzy-match on name+dob+address, not on customer_id. PII detection must run over free text (emails/transcripts/scans), not just the SSN column.
• Q3 (retrieval): chunk by document type (long PDFs vs. short CRM rows vs. dialogue turns); embeddings + a vector store with per-tenant filtering; re-embed only changed docs.
• Q4 (fine-tune): the LLM trains on policy/claim/support text; the vision model trains on documents/scans/*.png with claims.csv + crm_customers.csv as field labels. Version every dataset slice.
• Q5 (monitor/govern): track lineage source → cleaned → chunk → embedding → answer; watch for new claim types / schema drift; keep an audit trail per record.