tencent/Youtu-HiChunk

📃 License • 👨‍💻 Github • 📑 Paper

🎯 Introduction

Youtu-HiChunk is a hierarchical document chunking framework developed by Tencent Youtu Lab. Combined with the Auto-Merge retrieval algorithm, it can dynamically adjust the semantic granularity of retrieval fragments, mitigating issues of incomplete information caused by chunking.

• Hierarchical Document Structuring HiChunk is a hierarchical document structuring framework designed to address the limitations of traditional linear chunking methods in RAG systems. It focuses on modeling multi-level semantic granularity (e.g., sections, subsections, paragraphs) rather than flat text sequences, enabling RAG systems to retrieve information at contextually appropriate abstraction levels.

• Auto-Merge Retrieval Algorithm Auto-Merge Retrieval Algorithm dynamically adjusts chunk granularity via three complementary conditions, balancing semantic completeness and retrieval quality for both evidence-dense and sparse tasks.

📊 Performance

1. RAG piepline performance

2. Performance in various retrieval size

🚀 Quick Start

Install packages

uv venv hichunk --python 3.12
source hichunk/bin/activate
uv pip install torch==2.7.0 vllm==0.9.1 transformers==4.53.0 liger_kernel
uv pip install nltk
python -c "import nltk; nltk.download('punkt_tab')"

Then, you can deploy HiChunk model according link.

Usage

import os
os.environ['OPENAI_BASE_URL'] = "http://{serve_ip}:{serve_port}"
from HiChunk import HiChunkInferenceEngine, PROMPT

engine = HiChunkInferenceEngine(window_size=16*1024, line_max_len=100, max_level=10, prompt=PROMPT)
document_text = open('doc.txt', 'r').read()
chunked_document, chunks = engine.inference(document_text, recurrent_type=2)
print(chunked_document)

🎨 Visualization

Case 1

Case 2

🤝 Acknowledgements

The project is based on the excellent work of several open source projects:

• Youtu-LLM
• LongBench
• GraphRAG-Benchmark

📚 Citation

If you find our work useful in your research, please consider citing the following paper:

@misc{hi-chunk-2025,
  title={HiChunk: Evaluating and Enhancing Retrieval-Augmented Generation with Hierarchical Chunking},
  author={Tencent Youtu Lab},
  year={2025},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/TencentYoutuResearch/HiChunk.git}},
}