Spaces:
Sleeping
Sleeping
Commit ·
6e222b1
1
Parent(s): 7545a05
updated README
Browse files
README.md
CHANGED
|
@@ -10,7 +10,7 @@ short_description: Chunker simple from Sergii Makarevych's SDDW-based Chunker
|
|
| 10 |
|
| 11 |
# Chunker
|
| 12 |
|
| 13 |
-
[Chunker]
|
| 14 |
|
| 15 |
Chunker fulfills the base requirement of getting something along the lines of chunks and summaries as elaborated paragraphs from knowledge-source such as books. - though scaled up a little to include expanded Summary at Level 3 - worth hundreds of pages; called, although, "Jungian Chunker" it's usable for any pdf (though yet to be checked the max number of pages it can cover upto, but it probably runs into a few hundreds, and on the downside, its time-consuming)
|
| 16 |
|
|
@@ -26,14 +26,14 @@ The combined script - with two Phases, I (Extract and rewrite chunks - The "Leav
|
|
| 26 |
* The "Children" Key: In the final JSON, each summary block now lists which leaf chunks belong to it; making it a Navigable Tree
|
| 27 |
|
| 28 |
|
| 29 |
-
**
|
| 30 |
|
| 31 |
* L0 (Leaves): 1-2 pages of raw text rewritten
|
| 32 |
* L1 (Clusters/Branches): Summary of 5 Leaves (~10 pages)
|
| 33 |
* L2 (Chapters): Summary of 5 L1 Clusters/Branches (~50 pages)
|
| 34 |
* L3 (Volume): Summary of all L2 Nodes (The entire book)
|
| 35 |
|
| 36 |
-
**
|
| 37 |
|
| 38 |
> The Semantic Split: Instead of splitting - at fixed token boundaries, a cursor advances through the text and the LLM finds semantically complete split points, such as - at exactly 1000 characters, we give the LLM a 6000-character window and ask it to find the natural "Topic End" - locations where a topic is probable to end with another to probably begin - via "break_text".
|
| 39 |
|
|
|
|
| 10 |
|
| 11 |
# Chunker
|
| 12 |
|
| 13 |
+
[Chunker] (https://huggingface.co/spaces/prashantmatlani/chunker), a supposed substitute for RAG, is a low-level "browse a file and extract" tool of knowledge extraction via Progressive Disclosure from a large and loaded source such as a book (articles, papers work too) - taking a cue from Sergii Makarevych's [work] (https://www.linkedin.com/pulse/new-way-encode-documents-ai-agents-navigable-trees-sergii-makarevych-a6cof/), in turn derived from Andrej Karpathy's [LLM Wiki pattern] (https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f)
|
| 14 |
|
| 15 |
Chunker fulfills the base requirement of getting something along the lines of chunks and summaries as elaborated paragraphs from knowledge-source such as books. - though scaled up a little to include expanded Summary at Level 3 - worth hundreds of pages; called, although, "Jungian Chunker" it's usable for any pdf (though yet to be checked the max number of pages it can cover upto, but it probably runs into a few hundreds, and on the downside, its time-consuming)
|
| 16 |
|
|
|
|
| 26 |
* The "Children" Key: In the final JSON, each summary block now lists which leaf chunks belong to it; making it a Navigable Tree
|
| 27 |
|
| 28 |
|
| 29 |
+
**The Logic of the Knowledge-Pyramid:**
|
| 30 |
|
| 31 |
* L0 (Leaves): 1-2 pages of raw text rewritten
|
| 32 |
* L1 (Clusters/Branches): Summary of 5 Leaves (~10 pages)
|
| 33 |
* L2 (Chapters): Summary of 5 L1 Clusters/Branches (~50 pages)
|
| 34 |
* L3 (Volume): Summary of all L2 Nodes (The entire book)
|
| 35 |
|
| 36 |
+
**Phase I - Intelligent Chunking/Extract and rewrite chunks (The "Leaves")**
|
| 37 |
|
| 38 |
> The Semantic Split: Instead of splitting - at fixed token boundaries, a cursor advances through the text and the LLM finds semantically complete split points, such as - at exactly 1000 characters, we give the LLM a 6000-character window and ask it to find the natural "Topic End" - locations where a topic is probable to end with another to probably begin - via "break_text".
|
| 39 |
|