added data extractor

1972-03-01_llresearch.json

@@ -1,19 +0,0 @@
-{
-  "session_title": "March 1, 1972",
-  "session_date": "1972-03-01",
-  "session_type": "Channeling Session",
-  "synopsis": "Hatonn: I am speaking through this instrument. I will speak to you on the subject of maturity, for this you have requested. As you have already indicated, this concept may be somewhat different from that which is generally appreciated by your peoples.\n\nMaturity, my friends, is in truth a maturity of the spirit, for in truth there is nothing but the spirit. Physical illusion which you appreciate in your daily lives is of no consequence other than for its result in action upon the spiritual self.",
-  "turns": [
-    {
-      "role": null,
-      "speaker": null,
-      "content": "[The day of this channeling is unknown.] 1\n\n(Don channeling)"
-    },
-    {
-      "role": "assistant",
-      "speaker": "Hatonn",
-      "channeler": null,
-      "content": "I am Hatonn. I greet you, my friends, in the love and the light of our infinite Creator. It is a great privilege to be with you once more. I am always privileged to be with you. These are the thoughts of the one known as Hatonn.\n\nI am speaking through this instrument. I will speak to you on the subject of maturity, for this you have requested. As you have already indicated, this concept may be somewhat different from that which is generally appreciated by your peoples.\n\nMaturity, my friends, is in truth a maturity of the spirit, for in truth there is nothing but the spirit. Physical illusion which you appreciate in your daily lives is of no consequence other than for its result in action upon the spiritual self.\n\nMaturity, my friends, is first the realization of this fact. Secondly, maturity is the ability to control one's own consciousness in such a way so as to propagate the continuance of this maturing process.\n\nMaturity, my friends, then is realized in the ability to control one's consciousness. Unfortunately, upon the planet which you now enjoy, there is but very little true control of the basic consciousness. And, therefore, there is very, very little maturity. It is necessary first to realize the value of each thought you have, and then to reject those of little or no value. Most of the thoughts that we are able to discern occurring in the daily lives of those who dwell upon this planet lack maturity. For this reason, you might consider the planet upon which you live a planet of children. Their daily thoughts, communicated to one another, hold them in this state. It is a self-propagating thing, communicated from one to the other.\n\nIt is necessary to reject thoughts that continually infringe upon your mind from your present environment and to carefully select each thought that you generate in order to reach a state of true mental maturity.\n\nYou might ask how it is possible to select thoughts of value from thoughts that are meaningless, or of little value. It is very simple, my friends. All that is necessary is for you to analyze the thought with respect to the real objectives of your person. If the thought has true consequence, if the thought is of a true developmental nature-that is to say, if it develops either your consciousness or the consciousness of someone else with whom you are communicating-then it is a worthwhile thought. If it does not develop the consciousness, then it is probably of very little value.\n\nNow, how will a thought or concept develop into consciousness? There are several ways. One technique of development is simply evolving the ability of analyzing the merits of your thoughts. After this has been done, the thoughts themselves will act as generators of the maturing process.\n\nEach thought you have is important. It is important either in a negative or a positive sense. If it is a thought that is of no consequence, it is important to recognize this thought as being of no value. If it is a thought of consequence, then it is necessary that you amplify it and utilize it and communicate it, or it too will be of very little value.\n\nMaturity, my friends, is first the ability to think in this manner. It is second to act in the manner in which you think.\n\nThere are millions and millions of thoughts generated by the people of your planet each day. A very, very small percentage of these thoughts have to do with maturity. That is, a very, very small percentage of them have to do with creating a better environment for the growth of the spiritual self. By this I mean actively causing spiritual development.\n\nWhat is spiritual development? It is the process of maturing; the process of maturing, the process of analyzing everything that you are aware of in a true and unbiased sense. In order to do this, one must be able to recognize truth. It is only possible for one to recognize truth by the process of allowing truth to communicate the absolute base for truth which is ever present throughout the universe. This communication is accomplished primarily through the technique of meditation.\n\nThere is a separation of maturity into primarily two aspects: intellectual and spiritual maturity. They go hand in hand, and one generates the other. However, it is not necessary to acquire intellectual maturity in order to acquire spiritual maturity. It is, however, necessary to acquire spiritual maturity in order to acquire intellectual maturity, for the intellect cannot accurately evaluate concepts without a true spiritual basis.\n\nThere are three more things which I would like to speak about concerning maturity: the concept of infantile maturity; the concept of general or induced maturity; the concept of absolute or total maturity.\n\nThe concept of infantile maturity is highly misunderstood upon this planet. An infant, upon incarnating into your environment, has a certain amount of maturity that he normally brings with him. It is not necessary to induce this quality of maturity through any system of education to the infant. It is only necessary that he be alerted to the possibility of generating a continuance, through his own intellectual processes, of his own spiritual evolution and, consequently, spiritual maturity. Unfortunately, your religious systems do not provide, for the most part, this stimulus.\n\nIt is recommended that, in order for infantile maturity to progress at an acceptable rate, the infant be made aware at the earliest age possible of his responsibility in creating an intellectual communication with his total self. This is usually done through techniques of ritual and appreciation of the natural forces of the universe. The ritual that is employed by most of your religious systems upon your planet is highly ineffective, since it is generated primarily by force, and is not freely offered, to be accepted or rejected.\n\nThose, even in an extreme infantile state, who are appreciative, due to their previous growth of the proper ritualistic communications, will accept them, and continue, at their own pace, and should not be forced to attend weekly meetings at specific hours for these purposes, since they reach a peak of spiritual attunement that is a function of their own cyclical activities, and therefore should be able to seek out, at any time, spiritual communications and should be provided with a place for seeking. And this should be the limit of that which is expected of them. Your present system drives most of your people from spiritual seeking at a very early age due to the aspect of force which should be totally removed. This is what we have experienced, and what we have found to be most beneficial.\n\nThe second aspect of which I speak is that of induced maturity, occurring in most unusual aspect among the peoples of your present society. This maturity, which is a false maturity, is induced by the social systems which are presently in effect upon your planet. Each system intellectually communicates an aspect of assumed maturity, which has nothing to do with real or absolute maturity. Therefore, much strife and confusion is realized by those who attempt to orient their thinking so as to reach the accepted state or level of the assumed concept of the mature mind. This concept is usually heavily intellectual, for your society at present is primarily an intellectual society, with very, very little awareness of the existence or function of what you would call a spiritual society.\n\nTherefore, to mature within the boundaries of your present society and be accepted as a mature person, it is necessary to be able to communicate with it in its accepted intellectual jargon, which includes primarily a ridiculously long list of totally meaningless concepts. These should be, if one is to attain true maturity, rejected as meaningless, for they are extreme transients and have nothing to do with spiritual maturity.\n\nThe last aspect of maturity upon which I wish to speak is that of real maturity. My friends, there is only one way to reach real maturity: that is through meditation. We have said this many times. You cannot get there by intellectual mechanisms. You cannot get there by analyzing each of your thoughts, and labeling it either worthwhile or worthless. All of these things are aids, but without the foundation of daily meditation you cannot use this analysis, for the result of this analysis is insulated from the total self by a boundary. This boundary is permeable, but this boundary is only permeable when the mind is conditioned through meditation. Lack of meditation reduces this boundary to an impermeable state, all intellectual functions occurring on the surface, and having little effect upon the growth of the true self.\n\nSo you see, my friends, there is a dual process occurring. However, the meditation is always of the primary and greater importance. Once, however the art of meditation has been fully mastered, the intellectual mind becomes a useful tool in the development of spirit. It is of little consequence until this state of communication between the two is mastered. Therefore, my friends, all is of no avail until receptivity is made possible through daily meditation. This not only breaks down the barrier between the intellect and the spirit, it also breaks down all other barriers between the spirit and the one great All.\n\nI hope that I have been of some service to you in this discussion. I realize that it is difficult to speak to you on this subject because it is a difficult subject if one is to use the parameters of your present society. We consider it very, very simple. Therefore, any true discussion of the subject should require no more than a few sentences.\n\nIt has been a privilege to speak with you. Adonai, my friends. I am Hatonn.\n\n[At the end of this transcript is the following announcement.]"
-    }
-  ]
-}

LICENSE

@@ -0,0 +1,19 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+Copyright 2025 Andrii Zvorygin
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

cleanup.sh

@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+set -e
+
+echo "[cleanup] Starting distill_rag cleanup…"
+
+# Remove backup files
+echo "[cleanup] Removing *.bak files…"
+find . -type f -name "*.bak" -print -delete
+
+# Remove Node leftover logs
+echo "[cleanup] Removing npm debug logs…"
+find . -type f -name "npm-debug.log*" -print -delete
+
+# Clean old Elasticsearch index (if running)
+if [ -n "${ES_DISTILL_INDEX}" ]; then
+  echo "[cleanup] Deleting Elasticsearch index: ${ES_DISTILL_INDEX}"
+  curl -X DELETE "${ELASTICSEARCH_NODE:-http://localhost:9200}/${ES_DISTILL_INDEX}" || true
+else
+  echo "[cleanup] Skip index delete: ES_DISTILL_INDEX not set"
+fi
+
+# Remove build cache
+echo "[cleanup] Removing Jest cache…"
+rm -rf ./node_modules/.cache || true
+
+# Optional: remove node_modules entirely
+if [[ "$1" == "--deep" ]]; then
+  echo "[cleanup] Deep mode: removing node_modules…"
+  rm -rf node_modules
+fi
+
+echo "[cleanup] Done!"

data_extraction/README.md

@@ -0,0 +1,26 @@
+# Raw Data Extraction
+
+Most users start with messy HTML/TXT/PDF dumps.
+
+This module converts raw files → structured JSON sessions compatible with the distill_rag indexer.
+
+## Usage
+
+1. Put raw files into a directory, e.g.:
+
+   raw_corpus/file1.html
+   raw_corpus/file2.txt
+
+2. Run extraction:
+
+   node data_extraction/walk_and_extract.js raw_corpus extracted_json
+
+3. The output will be JSON files with:
+
+   - title
+   - session_date
+   - turns[]
+
+You can now index these with:
+
+   QUO_JSON_DIR=extracted_json node indexing/index_distill_chunks.js

data_extraction/clean_html.js

@@ -0,0 +1,28 @@
+// data_extraction/clean_html.js
+const cheerio = require("cheerio");
+
+function cleanHTML(html) {
+  // Load HTML into cheerio
+  const $ = cheerio.load(html);
+
+  const kill = [
+    "script",
+    "style",
+    "nav",
+    "header",
+    "footer",
+    ".ads",
+    ".advertisement",
+    "#sidebar",
+  ];
+
+  // Remove unwanted elements
+  kill.forEach(sel => $(sel).remove());
+
+  // Extract body text, normalize whitespace
+  let text = $("body").text();
+
+  return text.replace(/\s+/g, " ").trim();
+}
+
+module.exports = { cleanHTML };

data_extraction/convert_raw_to_sessions.js

@@ -0,0 +1,40 @@
+
+// data_extraction/convert_raw_to_sessions.js
+const fs = require("fs");
+const path = require("path");
+const { extractFromHTML } = require("./extractor");
+
+function convertFile(inputFile, outputFile) {
+  const turns = extractFromHTML(inputFile);
+
+  const session = {
+    title: path.basename(inputFile),
+    turns,
+  };
+
+  fs.writeFileSync(outputFile, JSON.stringify(session, null, 2));
+}
+
+// Convert all .html files in a directory
+function convertDir(inputDir, outputDir) {
+  if (!fs.existsSync(outputDir)) {
+    fs.mkdirSync(outputDir, { recursive: true });
+  }
+
+  const files = fs.readdirSync(inputDir);
+
+  files.forEach((file) => {
+    if (!file.endsWith(".html")) return;
+
+    const fullIn = path.join(inputDir, file);
+    const outName = file.replace(/\.html$/, ".json");
+    const fullOut = path.join(outputDir, outName);
+
+    convertFile(fullIn, fullOut);
+  });
+}
+
+module.exports = {
+  convertFile,
+  convertDir,
+};

data_extraction/example_corpus/examle_source_1.html

@@ -0,0 +1,6 @@
+<html>
+<body>
+<h1>Q: How do I serve others?</h1>
+<p>A: Service begins with small daily gestures of kindness.</p>
+</body>
+</html>

data_extraction/extractor.js

@@ -0,0 +1,34 @@
+// data_extraction/extractor.js
+const fs = require("fs");
+const cheerio = require("cheerio");
+
+function extractFromHTML(filePath) {
+  const raw = fs.readFileSync(filePath, "utf8");
+  const $ = cheerio.load(raw);
+
+  // Select paragraphs AND headings
+  const blocks = $("p, h1, h2, h3, h4, h5, h6")
+    .map((i, el) => $(el).text().trim())
+    .get()
+    .filter((t) => t.length > 0);
+
+  // If no structured blocks exist, fallback to body text split
+  let lines = blocks;
+  if (lines.length === 0) {
+    lines = $("body")
+      .text()
+      .split("\n")
+      .map((l) => l.trim())
+      .filter((l) => l.length > 0);
+  }
+
+  // Convert each block into a turn
+  const turns = lines.map((text, idx) => ({
+    role: idx === 0 ? "user" : "assistant",
+    content: text,
+  }));
+
+  return turns;
+}
+
+module.exports = { extractFromHTML };

data_extraction/walk_and_extract.js

@@ -0,0 +1,69 @@
+// data_extraction/walk_and_extract.js
+
+const fs = require("fs");
+const path = require("path");
+const { convertFile } = require("./convert_raw_to_sessions");
+
+// Recursively walk directory and return list of full file paths
+function walkDir(dir) {
+  let results = [];
+  const list = fs.readdirSync(dir);
+
+  list.forEach((file) => {
+    const filePath = path.join(dir, file);
+    const stat = fs.statSync(filePath);
+
+    if (stat && stat.isDirectory()) {
+      results = results.concat(walkDir(filePath));
+    } else {
+      results.push(filePath);
+    }
+  });
+
+  return results;
+}
+
+// Convert all HTML files from inputDir → outputDir
+function mainCLI() {
+  const args = process.argv.slice(2);
+
+  // ---------------------------
+  //  Usage print (fix for Jest)
+  // ---------------------------
+  if (args.length < 2) {
+    console.log("Usage: node walk_and_extract.js <input_dir> <output_dir>");
+    return; // no exit needed — Jest now sees output correctly
+  }
+
+  const inputDir = args[0];
+  const outputDir = args[1];
+
+  if (!fs.existsSync(inputDir)) {
+    console.error(`Input directory not found: ${inputDir}`);
+    process.exit(1);
+  }
+
+  if (!fs.existsSync(outputDir)) {
+    fs.mkdirSync(outputDir, { recursive: true });
+  }
+
+  const files = walkDir(inputDir).filter((f) => f.endsWith(".html"));
+
+  files.forEach((filePath) => {
+    const outName = path.basename(filePath).replace(/\.html$/, ".json");
+    const outPath = path.join(outputDir, outName);
+    convertFile(filePath, outPath);
+  });
+
+  console.log(`[walk_and_extract] Processed ${files.length} HTML files.`);
+}
+
+// Only run if executed directly
+if (require.main === module) {
+  mainCLI();
+}
+
+module.exports = {
+  walkDir,
+  mainCLI,
+};

jest.config.js

@@ -2,5 +2,8 @@
 module.exports = {
   testEnvironment: "node",
   verbose: true,
-  forceExit: true,
+  forceExit: true,            // force process exit
+  detectOpenHandles: false,   // explicitly disable warnings
+  silent: true,               // suppress worker chatter
+  testMatch: ["**/tests/**/*.test.js"]
 };

package-lock.json

@@ -12,6 +12,7 @@
         "@elastic/elasticsearch": "^8.17.0",
         "@tensorflow/tfjs-node": "^4.22.0",
         "@types/elasticsearch": "^5.0.43",
+        "cheerio": "^1.1.2",
         "dotenv": "^16.4.7",
         "elasticsearch": "^16.7.3",
         "fs-extra": "^11.3.0",
@@ -2163,6 +2164,11 @@
         "baseline-browser-mapping": "dist/cli.js"
       }
     },
+    "node_modules/boolbase": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/boolbase/-/boolbase-1.0.0.tgz",
+      "integrity": "sha512-JZOSA7Mo9sNGB8+UjSgzdLtokWAky1zbztM3WRLCbZ70/3cTANmQmOdR7y2g+J0e2WXywy1yS468tY+IruqEww=="
+    },
     "node_modules/brace-expansion": {
       "version": "1.1.12",
       "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.12.tgz",
@@ -2335,6 +2341,65 @@
         "node": ">=10"
       }
     },
+    "node_modules/cheerio": {
+      "version": "1.1.2",
+      "resolved": "https://registry.npmjs.org/cheerio/-/cheerio-1.1.2.tgz",
+      "integrity": "sha512-IkxPpb5rS/d1IiLbHMgfPuS0FgiWTtFIm/Nj+2woXDLTZ7fOT2eqzgYbdMlLweqlHbsZjxEChoVK+7iph7jyQg==",
+      "dependencies": {
+        "cheerio-select": "^2.1.0",
+        "dom-serializer": "^2.0.0",
+        "domhandler": "^5.0.3",
+        "domutils": "^3.2.2",
+        "encoding-sniffer": "^0.2.1",
+        "htmlparser2": "^10.0.0",
+        "parse5": "^7.3.0",
+        "parse5-htmlparser2-tree-adapter": "^7.1.0",
+        "parse5-parser-stream": "^7.1.2",
+        "undici": "^7.12.0",
+        "whatwg-mimetype": "^4.0.0"
+      },
+      "engines": {
+        "node": ">=20.18.1"
+      },
+      "funding": {
+        "url": "https://github.com/cheeriojs/cheerio?sponsor=1"
+      }
+    },
+    "node_modules/cheerio-select": {
+      "version": "2.1.0",
+      "resolved": "https://registry.npmjs.org/cheerio-select/-/cheerio-select-2.1.0.tgz",
+      "integrity": "sha512-9v9kG0LvzrlcungtnJtpGNxY+fzECQKhK4EGJX2vByejiMX84MFNQw4UxPJl3bFbTMw+Dfs37XaIkCwTZfLh4g==",
+      "dependencies": {
+        "boolbase": "^1.0.0",
+        "css-select": "^5.1.0",
+        "css-what": "^6.1.0",
+        "domelementtype": "^2.3.0",
+        "domhandler": "^5.0.3",
+        "domutils": "^3.0.1"
+      },
+      "funding": {
+        "url": "https://github.com/sponsors/fb55"
+      }
+    },
+    "node_modules/cheerio/node_modules/parse5": {
+      "version": "7.3.0",
+      "resolved": "https://registry.npmjs.org/parse5/-/parse5-7.3.0.tgz",
+      "integrity": "sha512-IInvU7fabl34qmi9gY8XOVxhYyMyuH2xUNpb2q8/Y+7552KlejkRvqvD19nMoUW/uQGGbqNpA6Tufu5FL5BZgw==",
+      "dependencies": {
+        "entities": "^6.0.0"
+      },
+      "funding": {
+        "url": "https://github.com/inikulin/parse5?sponsor=1"
+      }
+    },
+    "node_modules/cheerio/node_modules/undici": {
+      "version": "7.16.0",
+      "resolved": "https://registry.npmjs.org/undici/-/undici-7.16.0.tgz",
+      "integrity": "sha512-QEg3HPMll0o3t2ourKwOeUAZ159Kn9mx5pnzHRQO8+Wixmh88YdZRiIwat0iNzNNXn0yoEtXJqFpyW7eM8BV7g==",
+      "engines": {
+        "node": ">=20.18.1"
+      }
+    },
     "node_modules/chownr": {
       "version": "2.0.0",
       "resolved": "https://registry.npmjs.org/chownr/-/chownr-2.0.0.tgz",
@@ -2534,6 +2599,32 @@
         "urix": "~0.1.0"
       }
     },
+    "node_modules/css-select": {
+      "version": "5.2.2",
+      "resolved": "https://registry.npmjs.org/css-select/-/css-select-5.2.2.tgz",
+      "integrity": "sha512-TizTzUddG/xYLA3NXodFM0fSbNizXjOKhqiQQwvhlspadZokn1KDy0NZFS0wuEubIYAV5/c1/lAr0TaaFXEXzw==",
+      "dependencies": {
+        "boolbase": "^1.0.0",
+        "css-what": "^6.1.0",
+        "domhandler": "^5.0.2",
+        "domutils": "^3.0.1",
+        "nth-check": "^2.0.1"
+      },
+      "funding": {
+        "url": "https://github.com/sponsors/fb55"
+      }
+    },
+    "node_modules/css-what": {
+      "version": "6.2.2",
+      "resolved": "https://registry.npmjs.org/css-what/-/css-what-6.2.2.tgz",
+      "integrity": "sha512-u/O3vwbptzhMs3L1fQE82ZSLHQQfto5gyZzwteVIEyeaY5Fc7R4dapF/BvRoSYFeqfBk4m0V1Vafq5Pjv25wvA==",
+      "engines": {
+        "node": ">= 6"
+      },
+      "funding": {
+        "url": "https://github.com/sponsors/fb55"
+      }
+    },
     "node_modules/debug": {
       "version": "4.4.3",
       "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz",
@@ -2627,6 +2718,68 @@
       "integrity": "sha512-BSHWgDSAiKs50o2Re8ppvp3seVHXSRM44cdSsT9FfNEUUZLOGWVCsiWaRPWM1Znn+mqZ1OfVZ3z3DWEzSp7hRA==",
       "dev": true
     },
+    "node_modules/dom-serializer": {
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/dom-serializer/-/dom-serializer-2.0.0.tgz",
+      "integrity": "sha512-wIkAryiqt/nV5EQKqQpo3SToSOV9J0DnbJqwK7Wv/Trc92zIAYZ4FlMu+JPFW1DfGFt81ZTCGgDEabffXeLyJg==",
+      "dependencies": {
+        "domelementtype": "^2.3.0",
+        "domhandler": "^5.0.2",
+        "entities": "^4.2.0"
+      },
+      "funding": {
+        "url": "https://github.com/cheeriojs/dom-serializer?sponsor=1"
+      }
+    },
+    "node_modules/dom-serializer/node_modules/entities": {
+      "version": "4.5.0",
+      "resolved": "https://registry.npmjs.org/entities/-/entities-4.5.0.tgz",
+      "integrity": "sha512-V0hjH4dGPh9Ao5p0MoRY6BVqtwCjhz6vI5LT8AJ55H+4g9/4vbHx1I54fS0XuclLhDHArPQCiMjDxjaL8fPxhw==",
+      "engines": {
+        "node": ">=0.12"
+      },
+      "funding": {
+        "url": "https://github.com/fb55/entities?sponsor=1"
+      }
+    },
+    "node_modules/domelementtype": {
+      "version": "2.3.0",
+      "resolved": "https://registry.npmjs.org/domelementtype/-/domelementtype-2.3.0.tgz",
+      "integrity": "sha512-OLETBj6w0OsagBwdXnPdN0cnMfF9opN69co+7ZrbfPGrdpPVNBUj02spi6B1N7wChLQiPn4CSH/zJvXw56gmHw==",
+      "funding": [
+        {
+          "type": "github",
+          "url": "https://github.com/sponsors/fb55"
+        }
+      ]
+    },
+    "node_modules/domhandler": {
+      "version": "5.0.3",
+      "resolved": "https://registry.npmjs.org/domhandler/-/domhandler-5.0.3.tgz",
+      "integrity": "sha512-cgwlv/1iFQiFnU96XXgROh8xTeetsnJiDsTc7TYCLFd9+/WNkIqPTxiM/8pSd8VIrhXGTf1Ny1q1hquVqDJB5w==",
+      "dependencies": {
+        "domelementtype": "^2.3.0"
+      },
+      "engines": {
+        "node": ">= 4"
+      },
+      "funding": {
+        "url": "https://github.com/fb55/domhandler?sponsor=1"
+      }
+    },
+    "node_modules/domutils": {
+      "version": "3.2.2",
+      "resolved": "https://registry.npmjs.org/domutils/-/domutils-3.2.2.tgz",
+      "integrity": "sha512-6kZKyUajlDuqlHKVX1w7gyslj9MPIXzIFiz/rGu35uC1wMi+kMhQwGhl4lt9unC9Vb9INnY9Z3/ZA3+FhASLaw==",
+      "dependencies": {
+        "dom-serializer": "^2.0.0",
+        "domelementtype": "^2.3.0",
+        "domhandler": "^5.0.3"
+      },
+      "funding": {
+        "url": "https://github.com/fb55/domutils?sponsor=1"
+      }
+    },
     "node_modules/dotenv": {
       "version": "16.6.1",
       "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.6.1.tgz",
@@ -2744,6 +2897,29 @@
       "resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz",
       "integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A=="
     },
+    "node_modules/encoding-sniffer": {
+      "version": "0.2.1",
+      "resolved": "https://registry.npmjs.org/encoding-sniffer/-/encoding-sniffer-0.2.1.tgz",
+      "integrity": "sha512-5gvq20T6vfpekVtqrYQsSCFZ1wEg5+wW0/QaZMWkFr6BqD3NfKs0rLCx4rrVlSWJeZb5NBJgVLswK/w2MWU+Gw==",
+      "dependencies": {
+        "iconv-lite": "^0.6.3",
+        "whatwg-encoding": "^3.1.1"
+      },
+      "funding": {
+        "url": "https://github.com/fb55/encoding-sniffer?sponsor=1"
+      }
+    },
+    "node_modules/entities": {
+      "version": "6.0.1",
+      "resolved": "https://registry.npmjs.org/entities/-/entities-6.0.1.tgz",
+      "integrity": "sha512-aN97NXWF6AWBTahfVOIrB/NShkzi5H7F9r1s9mD3cDj4Ko5f2qhhVoYMibXF7GlLveb/D2ioWay8lxI97Ven3g==",
+      "engines": {
+        "node": ">=0.12"
+      },
+      "funding": {
+        "url": "https://github.com/fb55/entities?sponsor=1"
+      }
+    },
     "node_modules/error-ex": {
       "version": "1.3.4",
       "resolved": "https://registry.npmjs.org/error-ex/-/error-ex-1.3.4.tgz",
@@ -3290,6 +3466,24 @@
       "integrity": "sha512-H2iMtd0I4Mt5eYiapRdIDjp+XzelXQ0tFE4JS7YFwFevXXMmOp9myNrUvCg0D6ws8iqkRPBfKHgbwig1SmlLfg==",
       "dev": true
     },
+    "node_modules/htmlparser2": {
+      "version": "10.0.0",
+      "resolved": "https://registry.npmjs.org/htmlparser2/-/htmlparser2-10.0.0.tgz",
+      "integrity": "sha512-TwAZM+zE5Tq3lrEHvOlvwgj1XLWQCtaaibSN11Q+gGBAS7Y1uZSWwXXRe4iF6OXnaq1riyQAPFOBtYc77Mxq0g==",
+      "funding": [
+        "https://github.com/fb55/htmlparser2?sponsor=1",
+        {
+          "type": "github",
+          "url": "https://github.com/sponsors/fb55"
+        }
+      ],
+      "dependencies": {
+        "domelementtype": "^2.3.0",
+        "domhandler": "^5.0.3",
+        "domutils": "^3.2.1",
+        "entities": "^6.0.0"
+      }
+    },
     "node_modules/https-proxy-agent": {
       "version": "2.2.4",
       "resolved": "https://registry.npmjs.org/https-proxy-agent/-/https-proxy-agent-2.2.4.tgz",
@@ -3327,6 +3521,17 @@
         "ms": "^2.0.0"
       }
     },
+    "node_modules/iconv-lite": {
+      "version": "0.6.3",
+      "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.6.3.tgz",
+      "integrity": "sha512-4fCk79wshMdzMp2rH06qWrJE4iolqLhCUH+OiuIgU++RB0+94NlDL81atO7GX55uUKueo0txHNtvEyI6D7WdMw==",
+      "dependencies": {
+        "safer-buffer": ">= 2.1.2 < 3.0.0"
+      },
+      "engines": {
+        "node": ">=0.10.0"
+      }
+    },
     "node_modules/import-local": {
       "version": "3.2.0",
       "resolved": "https://registry.npmjs.org/import-local/-/import-local-3.2.0.tgz",
@@ -4627,6 +4832,17 @@
         "set-blocking": "^2.0.0"
       }
     },
+    "node_modules/nth-check": {
+      "version": "2.1.1",
+      "resolved": "https://registry.npmjs.org/nth-check/-/nth-check-2.1.1.tgz",
+      "integrity": "sha512-lqjrjmaOoAnWfMmBPL+XNnynZh2+swxiX3WUE0s4yEHI6m+AwrK2UZOimIRl3X/4QctVqS8AiZjFqyOGrMXb/w==",
+      "dependencies": {
+        "boolbase": "^1.0.0"
+      },
+      "funding": {
+        "url": "https://github.com/fb55/nth-check?sponsor=1"
+      }
+    },
     "node_modules/object-assign": {
       "version": "4.1.1",
       "resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz",
@@ -4745,6 +4961,51 @@
         "url": "https://github.com/sponsors/sindresorhus"
       }
     },
+    "node_modules/parse5-htmlparser2-tree-adapter": {
+      "version": "7.1.0",
+      "resolved": "https://registry.npmjs.org/parse5-htmlparser2-tree-adapter/-/parse5-htmlparser2-tree-adapter-7.1.0.tgz",
+      "integrity": "sha512-ruw5xyKs6lrpo9x9rCZqZZnIUntICjQAd0Wsmp396Ul9lN/h+ifgVV1x1gZHi8euej6wTfpqX8j+BFQxF0NS/g==",
+      "dependencies": {
+        "domhandler": "^5.0.3",
+        "parse5": "^7.0.0"
+      },
+      "funding": {
+        "url": "https://github.com/inikulin/parse5?sponsor=1"
+      }
+    },
+    "node_modules/parse5-htmlparser2-tree-adapter/node_modules/parse5": {
+      "version": "7.3.0",
+      "resolved": "https://registry.npmjs.org/parse5/-/parse5-7.3.0.tgz",
+      "integrity": "sha512-IInvU7fabl34qmi9gY8XOVxhYyMyuH2xUNpb2q8/Y+7552KlejkRvqvD19nMoUW/uQGGbqNpA6Tufu5FL5BZgw==",
+      "dependencies": {
+        "entities": "^6.0.0"
+      },
+      "funding": {
+        "url": "https://github.com/inikulin/parse5?sponsor=1"
+      }
+    },
+    "node_modules/parse5-parser-stream": {
+      "version": "7.1.2",
+      "resolved": "https://registry.npmjs.org/parse5-parser-stream/-/parse5-parser-stream-7.1.2.tgz",
+      "integrity": "sha512-JyeQc9iwFLn5TbvvqACIF/VXG6abODeB3Fwmv/TGdLk2LfbWkaySGY72at4+Ty7EkPZj854u4CrICqNk2qIbow==",
+      "dependencies": {
+        "parse5": "^7.0.0"
+      },
+      "funding": {
+        "url": "https://github.com/inikulin/parse5?sponsor=1"
+      }
+    },
+    "node_modules/parse5-parser-stream/node_modules/parse5": {
+      "version": "7.3.0",
+      "resolved": "https://registry.npmjs.org/parse5/-/parse5-7.3.0.tgz",
+      "integrity": "sha512-IInvU7fabl34qmi9gY8XOVxhYyMyuH2xUNpb2q8/Y+7552KlejkRvqvD19nMoUW/uQGGbqNpA6Tufu5FL5BZgw==",
+      "dependencies": {
+        "entities": "^6.0.0"
+      },
+      "funding": {
+        "url": "https://github.com/inikulin/parse5?sponsor=1"
+      }
+    },
     "node_modules/path-exists": {
       "version": "4.0.0",
       "resolved": "https://registry.npmjs.org/path-exists/-/path-exists-4.0.0.tgz",
@@ -4995,6 +5256,11 @@
         }
       ]
     },
+    "node_modules/safer-buffer": {
+      "version": "2.1.2",
+      "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz",
+      "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="
+    },
     "node_modules/secure-json-parse": {
       "version": "3.0.2",
       "resolved": "https://registry.npmjs.org/secure-json-parse/-/secure-json-parse-3.0.2.tgz",
@@ -5653,6 +5919,25 @@
       "resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-3.0.1.tgz",
       "integrity": "sha512-2JAn3z8AR6rjK8Sm8orRC0h/bcl/DqL7tRPdGZ4I1CjdF+EaMLmYxBHyXuKL849eucPFhvBoxMsflfOb8kxaeQ=="
     },
+    "node_modules/whatwg-encoding": {
+      "version": "3.1.1",
+      "resolved": "https://registry.npmjs.org/whatwg-encoding/-/whatwg-encoding-3.1.1.tgz",
+      "integrity": "sha512-6qN4hJdMwfYBtE3YBTTHhoeuUrDBPZmbQaxWAqSALV/MeEnR5z1xd8UKud2RAkFoPkmB+hli1TZSnyi84xz1vQ==",
+      "dependencies": {
+        "iconv-lite": "0.6.3"
+      },
+      "engines": {
+        "node": ">=18"
+      }
+    },
+    "node_modules/whatwg-mimetype": {
+      "version": "4.0.0",
+      "resolved": "https://registry.npmjs.org/whatwg-mimetype/-/whatwg-mimetype-4.0.0.tgz",
+      "integrity": "sha512-QaKxh0eNIi2mE9p2vEdzfagOKHCcj1pJ56EEHGQOVxp8r9/iszLUUV7v89x9O1p/T+NlTM5W7jW6+cz4Fq1YVg==",
+      "engines": {
+        "node": ">=18"
+      }
+    },
     "node_modules/whatwg-url": {
       "version": "5.0.0",
       "resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-5.0.0.tgz",

package.json

@@ -4,7 +4,7 @@
   "description": "",
   "main": "index.js",
   "scripts": {
-      "test": "jest"
+    "test": "jest"
   },
   "keywords": [],
   "author": "",
@@ -13,6 +13,7 @@
     "@elastic/elasticsearch": "^8.17.0",
     "@tensorflow/tfjs-node": "^4.22.0",
     "@types/elasticsearch": "^5.0.43",
+    "cheerio": "^1.1.2",
     "dotenv": "^16.4.7",
     "elasticsearch": "^16.7.3",
     "fs-extra": "^11.3.0",
@@ -24,4 +25,4 @@
     "jest": "^30.2.0",
     "supertest": "^7.1.4"
   }
-}
+}

tests/extraction/clean_html.test.js

@@ -0,0 +1,29 @@
+const { cleanHTML } = require("../../data_extraction/clean_html");
+
+describe("cleanHTML", () => {
+  test("removes scripts, styles, headers, footers, ads", () => {
+    const html = `
+      <html>
+        <head><style>.x{}</style></head>
+        <body>
+          <header>HEADER</header>
+          <script>alert("x")</script>
+          <div class="ads">Buy NOW</div>
+          <p>Hello world</p>
+          <footer>FOOTER</footer>
+        </body>
+      </html>
+    `;
+
+    const out = cleanHTML(html);
+
+    expect(out).toContain("Hello world");
+    expect(out).not.toMatch(/HEADER|FOOTER|Buy NOW|alert/);
+  });
+
+  test("collapses whitespace", () => {
+    const html = `<p>Text</p>\n\n<p>More</p>`;
+    const out = cleanHTML(html);
+    expect(out).toBe("Text More");
+  });
+});

tests/extraction/convert_raw_to_sessions.test.js

@@ -0,0 +1,38 @@
+const fs = require("fs");
+const path = require("path");
+const { convertDir } = require("../../data_extraction/convert_raw_to_sessions");
+
+describe("convert_raw_to_sessions", () => {
+  const rawDir = path.join(__dirname, "raw_data");
+  const outDir = path.join(__dirname, "out_data");
+
+  beforeAll(() => {
+    fs.mkdirSync(rawDir, { recursive: true });
+
+    fs.writeFileSync(
+      path.join(rawDir, "example.html"),
+      `
+        <h1>Q: How do I serve?</h1>
+        <p>A: Begin with kindness.</p>
+      `
+    );
+  });
+
+  afterAll(() => {
+    fs.rmSync(rawDir, { recursive: true });
+    fs.rmSync(outDir, { recursive: true });
+  });
+
+  test("converts raw HTML to JSON session", () => {
+    convertDir(rawDir, outDir);
+
+    const outFile = path.join(outDir, "example.json");
+    expect(fs.existsSync(outFile)).toBe(true);
+
+    const session = JSON.parse(fs.readFileSync(outFile, "utf8"));
+
+    expect(session.title).toBe("example.html");
+    expect(session.turns.length).toBe(2);
+    expect(session.turns[0].role).toBe("user");
+  });
+});

tests/extraction/extractor.test.js

@@ -0,0 +1,33 @@
+const fs = require("fs");
+const path = require("path");
+const { extractFromHTML } = require("../../data_extraction/extractor");
+
+describe("extractor", () => {
+  const sampleFile = path.join(__dirname, "sample.html");
+
+  beforeAll(() => {
+    fs.writeFileSync(
+      sampleFile,
+      `
+      <h1>Q: What is service?</h1>
+      <p>A: Service is love made visible.</p>
+    `
+    );
+  });
+
+  afterAll(() => {
+    fs.unlinkSync(sampleFile);
+  });
+
+  test("extracts paragraphs and roles", () => {
+    const turns = extractFromHTML(sampleFile);
+
+    expect(turns.length).toBe(2);
+
+    expect(turns[0].role).toBe("user");
+    expect(turns[0].content).toContain("What is service");
+
+    expect(turns[1].role).toBe("assistant");
+    expect(turns[1].content).toContain("love made visible");
+  });
+});

tests/extraction/walk_and_extract.test.js

@@ -0,0 +1,40 @@
+const fs = require("fs");
+const path = require("path");
+const { execSync } = require("child_process");
+
+describe("walk_and_extract CLI", () => {
+  const cli = path.resolve(__dirname, "../../data_extraction/walk_and_extract.js");
+  const rawDir = path.join(__dirname, "raw_cli");
+  const outDir = path.join(__dirname, "out_cli");
+
+  beforeAll(() => {
+    fs.mkdirSync(rawDir, { recursive: true });
+    fs.writeFileSync(
+      path.join(rawDir, "file1.html"),
+      `
+        <h1>Q: What is love?</h1>
+        <p>A: Love is unity.</p>
+      `
+    );
+  });
+
+  afterAll(() => {
+    fs.rmSync(rawDir, { recursive: true });
+    fs.rmSync(outDir, { recursive: true });
+  });
+
+  test("prints usage with no args", () => {
+    const output = execSync(`node ${cli}`, { encoding: "utf8" });
+    expect(output).toMatch(/Usage:/);
+  });
+
+  test("extracts files when given args", () => {
+    execSync(`node ${cli} ${rawDir} ${outDir}`);
+
+    const outFile = path.join(outDir, "file1.json");
+    expect(fs.existsSync(outFile)).toBe(true);
+
+    const json = JSON.parse(fs.readFileSync(outFile, "utf8"));
+    expect(json.turns.length).toBeGreaterThan(0);
+  });
+});