Tuning Chunk Sizes: The Foundation of RAG Efficiency

In RAG (Retrieval-Augmented Generation), you don't send the whole document. You send Chunks.

But how big should those chunks be?

• 100 Tokens? (Small, precise, extremely cheap to send, but might lose the "Global Context" of the sentence).
• 1,000 Tokens? (Large, holds all the nuance, but costs 10x more per search result).

In this lesson, we learn that Chunk Size is a Financial Decision. We’ll explore the "Precision vs. Recall" trade-off of chunking, how to use "Context Wrapping" to make small chunks smarter, and how to tune your parameters for maximum token ROI.

---

1. The Chunking Paradox

• Small Chunks (e.g., 256 tokens):

  • Better for Niche Retrieval (finding a specific number).
  • Saves 75% on token costs per query.
  • Risk: "Lost in Context." The model might not know which project the "number" belongs to.

• Large Chunks (e.g., 1024 tokens):

  • Better for Summarization and Reasoning.
  • High Token Waste (sending 1,000 tokens to get a 10-token answer).
  • Benefit: High "Semantic Density."

---

2. Token-First Chunking Strategies

A. The "Parent-Child" Strategy (The Sweet Spot)

1. Index your document in large "Parent" chunks (1,000 tokens) for semantic understanding.
2. Break those into small "Child" chunks (200 tokens) for the actual search.
3. The Trick: Retrieve the Child, but if it has a high confidence score, send the Parent to the LLM.

B. Overlap Optimization

Standard chunking uses "Overlap" (e.g. 10% overlap between chunks). Token-Efficient Rule: Reduce overlap to < 5% if you are using a re-ranker. A re-ranker is smart enough to handle the edge cases, and you save 5-10% on your entire vector database storage and ingestion cost.

---

3. Implementation: Semantic Chunking (Python)

Instead of cutting text mid-sentence (which wastes tokens on "Garbage" at the edges), use a Semantic Chunker.

Python Code: Sentence-Boundary Chunking

import nltk
from typing import List

def get_semantic_chunks(text: str, max_tokens=300) -> List[str]:
    # Use NLTK to identify sentence boundaries
    sentences = nltk.sent_tokenize(text)
    chunks = []
    current_chunk = ""
    
    for sentence in sentences:
        # Check token count of current_chunk + sentence
        if len(current_chunk) + len(sentence) > max_tokens * 4: # Approximation
            chunks.append(current_chunk.strip())
            current_chunk = sentence
        else:
            current_chunk += " " + sentence
            
    if current_chunk:
        chunks.append(current_chunk.strip())
        
    return chunks

---

4. The "Metadata Inflation" Factor

In many RAG implementations, every chunk is prefixed with: "Document Name: X. Author: Y. Date: Z. Chunk ID: 123. Text: [Actual Content]"

If your metadata is 100 tokens and your chunk is 200 tokens, 33% of your RAG budget is being spent on metadata.

Optimization: Use the "Symbolic Reference" pattern from Module 2.4. Only send the "Signal" to the LLM. Keep the metadata in your Python app's memory to show the user after the AI responds.

---

5. Visualizing Token Waste in RAG

In a typical RAG system, the "Useful Answer" tokens are often a tiny fraction of the total "Context" tokens.

pie title "RAG Token Consumption"
    "Wait/Waste (Boring parts of chunk)" : 70
    "Actual Signal (The answer)" : 20
    "Metadata & Instructions" : 10

Your Goal: Through better chunking, move that "Actual Signal" slice from 20% to 60%.

---

6. Summary and Key Takeaways

1. Small is Cheap, Large is Smart: Choose based on whether your app is a "Fact Finder" (Small) or a "Reasoning Tool" (Large).
2. Semantic Boundaries: Never cut a sentence in half; it creates "Jagged Tokens" that confuse the model.
3. Parent-Child Patterns: Search small, read large.
4. Metadata Pruning: Don't pay for the same document ID five times in one prompt.

In the next lesson, Hybrid Search: Vector vs. Keyword Efficiency, we look at چگونه to use old-school search techniques to save new-school token costs.

---

Exercise: The Chunking Benchmark

1. Index the same 50-page PDF twice:
   • Version A: 100-token chunks.
   • Version B: 1.000-token chunks.
2. Ask 5 specific questions.
3. Record the Total Tokens and Accuracy Score for each.

• Does Version B cost 10x more? (Usually).
• Is Version B 10x more accurate? (Usually not).
• Find the "Profit Peak" where you get the most accurate answer for the lowest token price.

---

Congratulations on completing Module 7 Lesson 1! You are now a RAG architect.