Module 5 Wrap-up: The Data Engineer

You have learned that "AI" is 80% data cleaning and 20% modeling. By mastering Loaders and Splitters, you have built the "Eyes" of the system. You can now ingest everything from a simple blog post to a complex corporate PDF library.

---

Hands-on Exercise: The Doc-to-Chunk Machine

1. The Goal

Write a Python script that:

1. Loads a URL (pick a news article or blog post).
2. Splits the text into chunks of 500 characters with a 50-character overlap.
3. Prints the total number of chunks and the metadata of the first chunk.

2. The Implementation Plan

• Use WebBaseLoader.
• Use RecursiveCharacterTextSplitter.
• Review the docs[0].metadata to see the source URL attached to the chunk.

---

Module 5 Summary

• Loaders: Standardize format (PDF, Web, TXT).
• Documents: The universal object with content and metadata.
• Chunking: Breaking long text to fit AI memory.
• Splitters: Recursive vs. specialized (Code/Markdown).
• Overlap: Preserving context between "Broken" sentences.

Coming Up Next...

In Module 6, we turn these text chunks into Math. We will learn about Embeddings and Vector Stores, and how to store these chunks so we can "Search" them with the speed of a machine.

---

Module 5 Checklist

• I have installed pypdf and beautifulsoup4.
• I can describe the difference between split_text and split_documents.
• I understand why 1,000 characters is a common chunk size.
• I have verified that metadata travels with the chunks after splitting.
• I can explain why a Markdown splitter uses headers as cues.