Production RAG Chunking Strategies: Semantic, Recursive, and Parent-Document Retrieval Compared

The Fundamental Flaw in Naive RAG Chunking

Retrieval-Augmented Generation (RAG) has emerged as the standard architecture for grounding Large Language Models in proprietary enterprise data. However, the vast majority of retrieval failures in production RAG pipelines do not stem from poor embedding models or weak LLMs; they stem from naive document chunking.

Early RAG tutorials typically recommend splitting raw documents into fixed-character blocks (e.g., 500 characters with a 50-character overlap). This brute-force approach ignores the natural syntax and logical hierarchy of human language. Fixed-size windows routinely bisect sentences across chunk boundaries, sever markdown tables in half, isolate bullet points from their parent context, and dilute dense factual relationships. When vector search retrieves these mangled fragments, the LLM hallucinates or returns incomplete answers due to context starvation.

1. Comparative Analysis of RAG Chunking Strategies

Selecting the optimal chunking architecture requires balancing vector search precision against holistic context retention:

2. Implementing Recursive Character Chunking

Recursive chunking respects natural document structure by attempting to split along primary boundaries (paragraphs) first, only falling back to secondary boundaries (sentences, words) if a segment exceeds the target token limit:

# chunking/recursive.py
from typing import List

class RecursiveTextSplitter:
    def __init__(self, chunk_size: int = 800, chunk_overlap: int = 100):
        self.chunk_size = chunk_size
        self.chunk_overlap = chunk_overlap
        self.separators = ["

", "
", ". ", " ", ""]

    def split_text(self, text: str) -> List[str]:
        final_chunks = []
        # Splits along paragraphs first, preserving natural narrative boundaries
        paragraphs = text.split("

")
        current_chunk = ""

        for para in paragraphs:
            if len(current_chunk) + len(para) <= self.chunk_size:
                current_chunk += ("

" if current_chunk else "") + para
            else:
                if current_chunk:
                    final_chunks.append(current_chunk.strip())
                # Handle paragraphs that exceed chunk_size independently
                if len(para) > self.chunk_size:
                    sub_chunks = self._split_fallback(para)
                    final_chunks.extend(sub_chunks[:-1])
                    current_chunk = sub_chunks[-1] if sub_chunks else ""
                else:
                    current_chunk = para

        if current_chunk:
            final_chunks.append(current_chunk.strip())
        return final_chunks

    def _split_fallback(self, text: str) -> List[str]:
        # Fallback to sentence boundaries
        sentences = text.split(". ")
        chunks, buf = [], ""
        for s in sentences:
            if len(buf) + len(s) < self.chunk_size:
                buf += (". " if buf else "") + s
            else:
                if buf: chunks.append(buf)
                buf = s
        if buf: chunks.append(buf)
        return chunks

3. Semantic Chunking: Detecting Conceptual Shifts

Semantic chunking evaluates the conceptual similarity between consecutive sentences. When the cosine distance between the embedding of sentence i and sentence i+1 crosses a distance threshold (e.g., 95th percentile variance), a semantic boundary is created:

# chunking/semantic.py
import numpy as np

def detect_semantic_boundaries(sentence_embeddings: list[np.ndarray], threshold_percentile: int = 90) -> list[int]:
    # Computes cosine distances between consecutive sentence embeddings and returns
    # index locations where significant thematic shifts occur.
    distances = []
    for i in range(len(sentence_embeddings) - 1):
        vec_a = sentence_embeddings[i]
        vec_b = sentence_embeddings[i + 1]
        # Cosine distance = 1 - cosine_similarity
        dist = 1.0 - (np.dot(vec_a, vec_b) / (np.linalg.norm(vec_a) * np.linalg.norm(vec_b)))
        distances.append(dist)

    # Set dynamic split threshold based on dataset distribution
    cutoff = np.percentile(distances, threshold_percentile)
    split_indices = [i + 1 for i, d in enumerate(distances) if d > cutoff]
    return split_indices

4. The Gold Standard: Parent-Document Retrieval Architecture

The fundamental challenge of vector search is an inherent trade-off:

• Small Chunks (100 - 200 tokens): Produce precise embeddings that align closely with user queries, but lack sufficient context for the LLM to generate comprehensive answers.
• Large Chunks (1,000 - 2,000 tokens): Contain full context, but their embeddings are diluted across multiple topics, reducing vector retrieval accuracy.

Parent-Document Retrieval resolves this dichotomy by decoupling search indexing from context injection. Documents are split into large Parent Chunks (e.g., 1,500 tokens), and each Parent is subdivided into small Child Chunks (200 tokens). Child chunks are indexed into PostgreSQL via pgvector. When a query matches a Child Chunk, the retrieval layer fetches the full Parent Chunk from the database and passes it to the LLM context window.

Combine this strategy with Semantic Caching with Redis & pgvector to slash API costs. Explore our AI & RAG Engineering Services for custom enterprise search implementations.