Hugging Face's logo

Spaces:
Vipooshanb
/
rag-data-platform
Runtime error

App Files Community
rag-data-platform / PROJECT_DEEP_DIVE.md
Vipooshanb's picture
Vipooshanb
Deploy Mozhii RAG Data Platform
af25178 verified
preview code
|
Raw
History Blame Contribute Delete
20.3 kB

Mozhii Platform Deep Dive

This document explains the full project in a teachable, implementation-level way:

  • Why it exists
  • How each layer works
  • How data moves through every stage
  • What each important function does
  • How frontend and backend are connected
  • What to watch out for while extending the code

1) What this project is

Mozhii Platform is a workflow system for preparing RAG training/retrieval data.

It is not an LLM chat app. It is a data pipeline UI and API where humans prepare content in 3 stages:

  1. Raw collection
  2. Cleaning
  3. Chunking

Each stage has review gates:

  • Submission goes to pending
  • Admin approves/rejects/edits
  • Approved data is stored separately
  • Approved data can be pushed to HuggingFace dataset repos

2) Tech stack and design choices

Backend

  • Python Flask
  • Blueprint routing modularization
  • Local file-based storage (text and JSON metadata)
  • HuggingFace Hub API integration

Frontend

  • Server-rendered HTML (single page with tabs)
  • Vanilla JavaScript modules
  • CSS-based component styling

Why this architecture

  • Simple to run locally
  • Low dependency complexity
  • Transparent file-level auditability
  • Easy to inspect pending/approved content in plain files

Tradeoff:

  • Local file system persistence means deployment must support persistent storage.

3) High-level component map

  • Entry point: run.py
  • Flask app factory: app/init.py
  • Config: app/config.py
  • Routes:
    • app/routes/main.py
    • app/routes/raw_data.py
    • app/routes/cleaning.py
    • app/routes/chunking.py
    • app/routes/admin.py
  • Services:
    • app/services/huggingface.py
    • app/services/storage.py (mostly scaffold, not primary route path currently)
  • Data schemas:
    • app/models/schemas.py
  • Frontend:
    • templates/index.html
    • static/js/main.js
    • static/js/raw-data.js
    • static/js/cleaning.js
    • static/js/chunking.js

4) Boot flow: from process start to browser ready

4.1 run.py

Main steps:

  1. Load .env values via dotenv
  2. Call create_app() from app package
  3. Read DEBUG and PORT env vars
  4. Start Flask server on 0.0.0.0:PORT

Key idea:

  • run.py is only startup wiring; most logic is in app package.

4.2 app/init.py create_app(config_name=None)

This is the factory pattern.

Function behavior:

  1. Build Flask instance with template and static folders
  2. Load Config object
  3. Enable CORS
  4. Ensure data folder tree exists
  5. Register route blueprints with URL prefixes
  6. Return configured app instance

Blueprint URL prefixes:

  • main: /
  • raw: /api/raw
  • cleaning: /api/cleaning
  • chunking: /api/chunking
  • admin: /api/admin

5) Configuration system

File: app/config.py

Config class centralizes:

  • Flask core settings: SECRET_KEY, DEBUG
  • HF credentials/repo names
  • Storage directory absolute paths
  • Admin defaults
  • Supported languages/categories/source types

Important operational detail:

  • Path constants resolve from project root, so file I/O stays stable regardless of where process starts.

6) Data storage model in detail

Data is split by stage and moderation status.

Directory layout:

  • data/pending/raw
  • data/pending/cleaned
  • data/pending/chunked
  • data/approved/raw
  • data/approved/cleaned
  • data/approved/chunked

For raw/cleaned item:

  • content file: filename.txt
  • metadata file: filename.meta.json

For chunked item:

  • folder per source file
  • chunk JSON files like chunk_01.json

Why this model works:

  • Human readable
  • Easy admin operations (move/delete)
  • Version-like trail in metadata fields

7) API and frontend integration model

Frontend calls backend using fetch through one wrapper function in static/js/main.js.

General request pattern:

  1. User action triggers handler in tab JS file
  2. Handler validates form state
  3. Handler calls api(endpoint, options)
  4. Flask route processes request
  5. Route writes/reads files and returns JSON
  6. Frontend shows toast and refreshes affected lists

This is straightforward request-response; no websockets or background queue.

8) Main routes explained function by function

File: app/routes/main.py

index()

  • Route: GET /
  • Returns index.html
  • This is the application shell page

health_check()

  • Route: GET /health
  • Returns service status, version, platform fields
  • Useful for uptime probes and deployment checks

get_config()

  • Route: GET /api/config
  • Returns safe client config (languages, categories, sourceTypes, repo names)
  • Does not expose secret tokens

9) Raw data routes explained function by function

File: app/routes/raw_data.py

generate_metadata(filename, language, source, content)

  • Utility function
  • Creates metadata object with id, timestamps, status, lengths

submit_raw_data()

  • Route: POST /api/raw/submit
  • Validates required fields: filename, language, source, content
  • Validates filename pattern
  • Rejects duplicate pending filename
  • Writes:
    • pending/raw/filename.txt
    • pending/raw/filename.meta.json
  • Returns submission id and success status

list_pending()

  • Route: GET /api/raw/pending
  • Reads all .meta.json files from pending raw
  • Sorts newest first by submitted_at

list_approved()

  • Route: GET /api/raw/approved
  • Reads all .meta.json from approved raw
  • Sorts newest first by approved_at

get_file_content(filename)

  • Route: GET /api/raw/file/
  • Checks pending first, then approved
  • Returns content + metadata + location

10) Cleaning routes explained function by function

File: app/routes/cleaning.py

list_raw_files()

  • Route: GET /api/cleaning/raw-files
  • Reads approved raw files only (important lineage rule)
  • Includes preview and full content for cleaning UI
  • Derives cleaning status by checking pending/approved cleaned files

submit_cleaned_data()

  • Route: POST /api/cleaning/submit
  • Requires filename and cleaned content
  • Verifies source raw file exists in approved raw
  • Loads original metadata to preserve language/source/lineage
  • Writes pending cleaned content + metadata

list_pending()

  • Route: GET /api/cleaning/pending
  • Lists pending cleaned metadata

list_approved()

  • Route: GET /api/cleaning/approved
  • Lists approved cleaned metadata

get_file_content(filename)

  • Route: GET /api/cleaning/file/
  • Returns pending or approved cleaned content and metadata

11) Chunking routes explained function by function

File: app/routes/chunking.py

generate_chunk_id(language, category, filename, index)

  • Utility for stable chunk id format
  • Combines language, short category, short filename, index

list_cleaned_files()

  • Route: GET /api/chunking/cleaned-files
  • Lists approved cleaned files as chunk sources
  • Includes counts of pending and approved chunks

get_chunks(filename)

  • Route: GET /api/chunking/chunks/
  • Reads pending and approved chunk json files
  • Adds status field per chunk
  • Sorts by chunk_index

submit_chunk()

  • Route: POST /api/chunking/submit
  • Validates required fields: filename, text, category
  • Verifies source cleaned file exists
  • Computes next chunk index from existing pending+approved count
  • Generates chunk object and writes pending chunk json file

submit_batch()

  • Route: POST /api/chunking/submit-batch
  • Accepts array of chunk payloads for same file
  • Computes sequential indices
  • Writes multiple pending chunk files

list_pending()

  • Route: GET /api/chunking/pending
  • Returns pending chunks grouped by source filename

delete_chunk(filename, chunk_index)

  • Route: DELETE /api/chunking/chunk//
  • Deletes pending chunk only

12) Admin routes explained function by function

File: app/routes/admin.py

get_all_pending()

  • Route: GET /api/admin/pending
  • Aggregates pending items across raw/cleaned/chunked
  • Returns per-stage arrays and total counters

get_pending_item()

  • Route: GET /api/admin/item
  • Query params: type, filename, optional chunk_index
  • Returns a specific pending item's content for edit flow

update_pending_item()

  • Route: POST /api/admin/update
  • Updates pending content/metadata/chunk data
  • Adds updated_at and updated_by audit fields

approve_submission()

  • Route: POST /api/admin/approve
  • For raw/cleaned: move files from pending to approved and update metadata
  • For chunk: move chunk file to approved folder and update chunk status fields

reject_submission()

  • Route: POST /api/admin/reject
  • Deletes pending item (or specific chunk)
  • Logs rejection with reason

approve_all()

  • Route: POST /api/admin/approve-all
  • Bulk approval by type:
    • raw all
    • cleaned all
    • chunks for one filename

get_stats()

  • Route: GET /api/admin/stats
  • Returns pending/approved counts for each stage + totals

push_to_huggingface()

  • Route: POST /api/admin/push-to-hf
  • Requires type and HF token (from body or env)
  • Iterates approved data and uploads via HuggingFaceService
  • Returns per-stage upload/failure counts

13) Frontend HTML structure and behavior

File: templates/index.html

Main UI blocks:

  • Header + admin panel toggle
  • Tab nav for raw, cleaning, chunking
  • Tab panel sections with forms and list panes
  • Admin sidebar for queue and HuggingFace push inputs
  • Generic toast container
  • Generic modal

Script load order:

  1. main.js
  2. raw-data.js
  3. cleaning.js
  4. chunking.js

Implication:

  • Shared helpers from main.js are available to tab modules.

14) Frontend shared engine functions

File: static/js/main.js

AppState object

Global in-memory state for current tab, config, pending counters.

api(endpoint, options)

  • Wrapper around fetch
  • Sets json headers
  • Parses response json
  • Throws standardized errors on non-2xx

showToast(title, message, type, duration)

  • Creates and inserts toast UI
  • Auto-removes with timer

showModal(title, content, buttons)

  • Generic modal renderer with dynamic footer actions

hideModal()

  • Hides modal overlay

initTabs()

  • Handles tab button click
  • Switches active panel
  • Triggers stage refresh for cleaning/chunking tabs

initAdminSidebar()

  • Handles open/close of sidebar
  • Triggers admin data refresh on open

refreshAdminData()

  • Calls /api/admin/pending and /api/admin/stats
  • Updates badge/stats and pending lists

renderPendingList(containerId, items, type)

  • Renders pending raw/cleaned item rows with edit/approve/reject actions

renderPendingChunks(containerId, chunkedFiles)

  • Renders grouped pending chunks and approve-all action

approveItem(type, filename)

  • Calls /api/admin/approve then refreshes UI

rejectItem(type, filename)

  • Confirms in modal then calls /api/admin/reject

approveAllChunks(filename)

  • Calls /api/admin/approve-all with type chunks

editItem(type, filename)

  • Active definition (later in file) uses:
    • GET /api/admin/item?type=...&filename=...
    • POST /api/admin/update
  • Opens modal, allows editing content, saves update

pushToHuggingFace()

  • Active definition (later in file) uses token/repo from sidebar inputs
  • Calls /api/admin/push-to-hf with type all
  • Displays upload result summary modal

initHFConfig()

  • Binds sync button
  • Loads/saves token and repo to localStorage

initModalHandlers()

  • Adds close handlers: close button, overlay click, Escape key

loadConfig()

  • Fetches /api/config and stores AppState.config

DOMContentLoaded initializer

  • Calls loadConfig
  • Initializes tabs, sidebar, modals, hf config
  • Refreshes admin badge data

Important code quality note:

  • main.js includes duplicate definitions for some functions (editItem, pushToHuggingFace, initHFConfig).
  • JavaScript uses the last definition encountered, so the later versions are active at runtime.

15) Raw tab frontend functions

File: static/js/raw-data.js

RawDataElements.init()

Caches dom elements for raw form controls.

validateFilename(filename)

Checks required, regex pattern, min and max length.

validateContent(content)

Checks required and min content length.

validateRawDataForm()

Runs filename/content validations and returns aggregate result.

updateCharCount()

Live character count and short-content warning style.

handleRawDataSubmit()

  • Validates form
  • Disables submit button
  • POST /api/raw/submit
  • On success: toast, clear form, refresh admin data
  • Re-enables button finally

clearRawDataForm()

Resets fields to defaults and focuses filename input.

initRawDataEventListeners()

Wires input, click, blur/focus, and Ctrl+Enter handlers.

DOMContentLoaded

Initializes element cache, listeners, and char count.

16) Cleaning tab frontend functions

File: static/js/cleaning.js

CleaningElements.init()

Caches cleaning pane dom nodes.

CleaningState

Holds file list, selected file, selected raw content.

refreshCleaningFiles()

Calls /api/cleaning/raw-files and updates list.

renderCleaningFileList()

Renders clickable file cards with cleaning status.

formatStatus(status)

Maps internal status values to user-facing labels.

selectCleaningFile(filename)

Selects file, fills raw preview, enables controls, clears cleaned textarea.

escapeHtml(text)

Prevents html injection in preview.

copyRawContent()

Copies selected raw content to clipboard with feedback.

updateCleaningCharCount()

Live cleaned text character count.

clearCleaningForm()

Clears cleaned textarea.

handleCleaningSubmit()

  • Validates selected file and content
  • POST /api/cleaning/submit
  • On success: toast + refresh lists + refresh admin badge

initCleaningEventListeners()

Binds refresh/copy/clear/submit/input/shortcut events.

DOMContentLoaded

Initializes dom refs and event listeners.

17) Chunking tab frontend functions

File: static/js/chunking.js

ChunkingElements.init()

Caches dom elements for chunking screen.

ChunkingState

Tracks cleaned files, selected file, chunks, next index.

refreshChunkingFiles()

Calls /api/chunking/cleaned-files and renders file list.

renderChunkingFileList()

Shows available cleaned files with chunk counts.

selectChunkingFile(filename)

Sets selected source, displays source text, loads existing chunks, enables form.

escapeHtml(text)

Prevents html injection in source display.

loadChunksForFile(filename)

Calls /api/chunking/chunks/, updates next index and list.

renderChunksList()

Renders existing chunk entries.

enableChunkingForm() and disableChunkingForm()

Control form interactivity state.

updateChunkCharCount()

Live count for chunk text field.

clearChunkForm()

Clears chunk text and overlap fields.

generateChunkId()

Client-side preview id generator.

previewChunk()

Shows JSON preview modal before submit.

handleChunkSubmit()

  • Validates file and text
  • POST /api/chunking/submit
  • On success: clear form, set overlap hint, reload chunks, refresh lists and admin badge

setupTextSelection()

Handles selecting source text to assist chunk drafting.

initChunkingEventListeners()

Binds refresh, preview, submit, input, and helper events.

18) Services and their practical role

18.1 HuggingFaceService

File: app/services/huggingface.py

Methods:

  • init: initializes token, API client, repo names
  • is_configured: quick readiness check
  • upload_raw_file: uploads txt + metadata json to raw repo
  • upload_cleaned_file: uploads txt + metadata json to cleaned repo
  • upload_chunk: uploads one chunk file under folder path
  • upload_chunks: uploads many chunks
  • list_raw_files: lists txt files from raw dataset repo
  • download_file: downloads a file from selected stage repo
  • sync_all_approved: batch local approved to HF

Runtime usage in current app:

  • Directly used by admin push endpoint.

18.2 StorageService

File: app/services/storage.py

Methods:

  • directory setup and path management
  • get/list/save/approve/delete/stats helpers

Runtime usage in current app:

  • Not currently used by route handlers (routes mostly do direct file I/O).
  • It can be used in future refactor to reduce repeated logic.

19) Schemas model package

File: app/models/schemas.py

Data classes:

  • RawDataSchema
  • CleanedDataSchema
  • ChunkSchema

Capabilities:

  • type-structured constructors
  • validation methods
  • to_dict and metadata helper transforms
  • chunk rag format helper

Runtime usage in current app:

  • Mostly unused by route handlers right now.
  • Current routes manually construct dictionaries.

This is a common intermediate state in growing projects: schema layer prepared, route code still imperative.

20) End-to-end call graphs (learning view)

20.1 Raw submission graph

  1. User enters file/content in RAW tab
  2. handleRawDataSubmit in raw-data.js
  3. api wrapper in main.js
  4. POST /api/raw/submit in raw_data.py
  5. Validation + write pending files
  6. JSON success response
  7. Toast + form reset + admin badge refresh

20.2 Cleaning submission graph

  1. User opens cleaning tab
  2. refreshCleaningFiles calls /api/cleaning/raw-files
  3. User selects file via selectCleaningFile
  4. User submits via handleCleaningSubmit
  5. POST /api/cleaning/submit in cleaning.py
  6. Pending cleaned file written
  7. UI refreshes statuses and admin counters

20.3 Chunk submission graph

  1. User opens chunking tab and selects cleaned file
  2. loadChunksForFile gets existing chunks
  3. User creates chunk and submits
  4. POST /api/chunking/submit in chunking.py
  5. Next index computed; pending chunk json written
  6. UI list and counts refreshed

20.4 Admin approval graph

  1. Admin opens sidebar; refreshAdminData fetches /api/admin/pending
  2. Admin approves or rejects
  3. approve_submission or reject_submission in admin.py
  4. File moves (approve) or deletes (reject)
  5. Sidebar counts and lists refresh

20.5 HuggingFace sync graph

  1. Admin enters HF token and repo in sidebar
  2. pushToHuggingFace in main.js
  3. POST /api/admin/push-to-hf
  4. admin route loops approved data and calls HuggingFaceService uploads
  5. Response includes per-stage success/failure counts
  6. UI shows completion summary

21) Known implementation mismatches and caveats

  1. Duplicate function definitions in static/js/main.js
  • Last function wins at runtime.
  • Can confuse maintenance and debugging.
  1. Some route/service abstractions are prepared but not fully unified
  • StorageService and schema classes are not the main execution path.
  1. Auth model is minimal
  • Admin routes are callable if endpoint is reachable.
  • For public deployment, add real auth/authorization.
  1. Local file storage limits horizontal scaling
  • Multi-instance deployment needs shared persistent volume/object storage.

22) How to safely extend this project

Recommended extension order:

  1. Add authentication to admin endpoints
  2. Refactor route file I/O into StorageService
  3. Replace manual dict validation with schema validate calls
  4. Remove duplicate JS function definitions
  5. Add audit log persistence beyond simple logger calls
  6. Add API tests for route contracts

23) Learning exercises for you

  1. Trace one request manually
  • Put console.log in handleRawDataSubmit
  • Put print/log in submit_raw_data
  • Observe request and file outputs
  1. Add a new source type end-to-end
  • config source list
  • UI select option
  • submit and metadata verification
  1. Add a new admin stat metric
  • compute backend metric in get_stats
  • display in sidebar
  1. Refactor one route to StorageService
  • Pick raw approve path first
  • Compare behavior before/after

24) Quick reference map

Backend route files:

  • app/routes/main.py
  • app/routes/raw_data.py
  • app/routes/cleaning.py
  • app/routes/chunking.py
  • app/routes/admin.py

Frontend behavior files:

  • static/js/main.js
  • static/js/raw-data.js
  • static/js/cleaning.js
  • static/js/chunking.js

Model/service files:

  • app/models/schemas.py
  • app/services/huggingface.py
  • app/services/storage.py

UI shell:

  • templates/index.html

25) Final summary

This project is a human-in-the-loop RAG data curation pipeline with:

  • stage-based processing,
  • moderation gates,
  • local persistent workflow state,
  • optional HuggingFace publication.

If you understand:

  • the pending/approved directory transitions,
  • the route called for each button click,
  • and the admin moderation and push endpoints, then you understand the core of the system.