Spaces:
Running on Zero
Running on Zero
File size: 1,536 Bytes
4298e57 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 | """
makes sure that large text units are split into overlapping chunks without
losing source metadata, and that FAR section references can be identified from text.
Source metadata:
Metadata such as the source file name and section identifies where each
chunk originated. Preserving it allows retrieved chunks to remain linked
to their original document and section
Why:
Reliable chunk metadata improves traceability in retrieval workflows, while
FAR section detection helps organize and cite regulatory content accurately
"""
'''
params:
text: document content being processed
source_name: name of the source document, e.g. ``policy.txt``
source_type: file type of the source document, such as ``.TXT``
section: section metadata associated with the source text
chunk_size: max target size of each generated chunk, measured in characters
overlap: no. of characters repeated between adjacent chunks to preserve context
'''
from document_loader import TextUnit, infer_far_section, units_to_chunks
def test_chunking_preserves_source_metadata():
units = [TextUnit(text="A" * 2400, source_name="policy.txt", source_type="TXT", section="Section 1")]
chunks = units_to_chunks(units, chunk_size=1000, overlap=100)
assert len(chunks) >= 3
assert all(c.source_name == "policy.txt" for c in chunks)
assert all(c.section == "Section 1" for c in chunks)
def test_far_section_detection():
assert infer_far_section("FAR 10.001 requires market research") == "FAR 10.001"
|