prashantmatlani commited on
Commit
6e222b1
·
1 Parent(s): 7545a05

updated README

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -10,7 +10,7 @@ short_description: Chunker simple from Sergii Makarevych's SDDW-based Chunker
10
 
11
  # Chunker
12
 
13
- [Chunker] [https://huggingface.co/spaces/prashantmatlani/chunker], a supposed substitute for RAG, is a low-level "browse a file and extract" tool of knowledge extraction via Progressive Disclosure from a large and loaded source such as a book (articles, papers work too) - taking a cue from Sergii Makarevych's [work] [https://www.linkedin.com/pulse/new-way-encode-documents-ai-agents-navigable-trees-sergii-makarevych-a6cof/], in turn derived from Andrej Karpathy's [LLM Wiki pattern] [https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f]
14
 
15
  Chunker fulfills the base requirement of getting something along the lines of chunks and summaries as elaborated paragraphs from knowledge-source such as books. - though scaled up a little to include expanded Summary at Level 3 - worth hundreds of pages; called, although, "Jungian Chunker" it's usable for any pdf (though yet to be checked the max number of pages it can cover upto, but it probably runs into a few hundreds, and on the downside, its time-consuming)
16
 
@@ -26,14 +26,14 @@ The combined script - with two Phases, I (Extract and rewrite chunks - The "Leav
26
  * The "Children" Key: In the final JSON, each summary block now lists which leaf chunks belong to it; making it a Navigable Tree
27
 
28
 
29
- ** The Logic of the Knowledge-Pyramid: **
30
 
31
  * L0 (Leaves): 1-2 pages of raw text rewritten
32
  * L1 (Clusters/Branches): Summary of 5 Leaves (~10 pages)
33
  * L2 (Chapters): Summary of 5 L1 Clusters/Branches (~50 pages)
34
  * L3 (Volume): Summary of all L2 Nodes (The entire book)
35
 
36
- ** Phase I - Intelligent Chunking/Extract and rewrite chunks (The "Leaves") **
37
 
38
  > The Semantic Split: Instead of splitting - at fixed token boundaries, a cursor advances through the text and the LLM finds semantically complete split points, such as - at exactly 1000 characters, we give the LLM a 6000-character window and ask it to find the natural "Topic End" - locations where a topic is probable to end with another to probably begin - via "break_text".
39
 
 
10
 
11
  # Chunker
12
 
13
+ [Chunker] (https://huggingface.co/spaces/prashantmatlani/chunker), a supposed substitute for RAG, is a low-level "browse a file and extract" tool of knowledge extraction via Progressive Disclosure from a large and loaded source such as a book (articles, papers work too) - taking a cue from Sergii Makarevych's [work] (https://www.linkedin.com/pulse/new-way-encode-documents-ai-agents-navigable-trees-sergii-makarevych-a6cof/), in turn derived from Andrej Karpathy's [LLM Wiki pattern] (https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f)
14
 
15
  Chunker fulfills the base requirement of getting something along the lines of chunks and summaries as elaborated paragraphs from knowledge-source such as books. - though scaled up a little to include expanded Summary at Level 3 - worth hundreds of pages; called, although, "Jungian Chunker" it's usable for any pdf (though yet to be checked the max number of pages it can cover upto, but it probably runs into a few hundreds, and on the downside, its time-consuming)
16
 
 
26
  * The "Children" Key: In the final JSON, each summary block now lists which leaf chunks belong to it; making it a Navigable Tree
27
 
28
 
29
+ **The Logic of the Knowledge-Pyramid:**
30
 
31
  * L0 (Leaves): 1-2 pages of raw text rewritten
32
  * L1 (Clusters/Branches): Summary of 5 Leaves (~10 pages)
33
  * L2 (Chapters): Summary of 5 L1 Clusters/Branches (~50 pages)
34
  * L3 (Volume): Summary of all L2 Nodes (The entire book)
35
 
36
+ **Phase I - Intelligent Chunking/Extract and rewrite chunks (The "Leaves")**
37
 
38
  > The Semantic Split: Instead of splitting - at fixed token boundaries, a cursor advances through the text and the LLM finds semantically complete split points, such as - at exactly 1000 characters, we give the LLM a 6000-character window and ask it to find the natural "Topic End" - locations where a topic is probable to end with another to probably begin - via "break_text".
39