Split text based on a count pattern.
purecpp_chunks_clean
Python package:
count_unit
, overlap
, and count_threshold
.
Parameter | Description |
---|---|
count_unit | The element to count before splitting (word, character, regex). |
overlap | Number of characters shared between consecutive chunks. |
count_threshold | Number of times the count_unit must appear before splitting. |
ProcessDocuments
method. Each resulting chunk will also be an instance of Document
.
The max_workers
parameter controls the number of concurrent threads used during processing.
Document
instances containing the chunked text data.