Split text into chunks and sort them based on similarity.
purecpp_chunks_clean
Python package:
chunk_size
, overlap
, and provide an embedding_model
. If using OpenAI embeddings, provide your openai_api_key
.
Parameter | Description |
---|---|
chunk_size | Maximum size of each chunk (in characters). |
overlap | Number of characters shared between consecutive chunks. |
embedding_model | Embedding model used for similarity calculation (HuggingFace or OpenAI). |
openai_api_key | API key required if using the OpenAI embedding model. |
embedding_model
parameter specifies which model to use for generating embeddings. The embeddings are used to calculate the similarity between document chunks. Two options are available:
0
): Uses the SentenceTransformer model (all-MiniLM-L6-v2
).1
): Uses OpenAI’s embedding model (text-embedding-ada-002
). Requires an openai_api_key
.OPENAI_API_KEY
environment variable in the terminal:
ProcessSingleDocument
ProcessSingleDocument
method processes a single document and sorts the chunks based on their similarity.
ProcessDocuments
ProcessDocuments
method processes multiple input documents and sorts the chunks based on their similarity.
Parameter | Description |
---|---|
items | List of LoaderDataStruct documents to process. |
max_workers | Number of parallel workers for processing. |