added
Chunking API
17 days ago by Cormick Marskamp
The Chunking API is now live - the final building block for fully programmatic knowledge base workflows. With this new API, you can automatically split (“chunk”) your text data in the way that best fits your needs, all without leaving your workflow.
The Chunking API supports six different chunking strategies, each suited to a different kind of content:
- Token chunker: Splits your text based on token count. Great for keeping chunks small enough for LLMs and for consistent embedding sizes.
- Sentence chunker: Breaks your text at sentence boundaries, so each chunk stays readable and sentences remain intact.
- Recursive chunker: Chunks text by working down a hierarchy (paragraphs, then sentences, then words) to maintain document structure.
- Semantic chunker: Groups together sentences that are topically related, so each chunk makes sense on its own.
- SDPM chunker: Uses advanced skip-gram patterns to find natural split points, especially helpful for technical or structured documents.
- Agentic chunker: Uses an LLM to determine the best split points, ideal for complex documents that need intelligent segmentation.
With this release, our RAG-as-a-service stack is complete. You can create knowledge bases, upload and chunk your data (or send pre-chunked data), search and filter content with chunk-level metadata, turn on agentic RAG, and apply reranker models. The knowledge base tester in the UI lets you experiment with search settings, chunk sizes, and thresholds to optimize retrieval for your use case.
Try the Chunking API out for yourself here: Chunking