data_extraction/README.md

# Raw Data Extraction

Most users start with messy HTML/TXT/PDF dumps.

This module converts raw files → structured JSON sessions compatible with the distill_rag indexer.

## Usage

1. Put raw files into a directory, e.g.:

   raw_corpus/file1.html
   raw_corpus/file2.txt

2. Run extraction:

   node data_extraction/walk_and_extract.js raw_corpus extracted_json

3. The output will be JSON files with:

   - title
   - session_date
   - turns[]

You can now index these with:

   JSON_DIR=extracted_json node indexing/index_distill_chunks.js