Project 5: Building a Multimodal Search Platform

In this final project of Module 18, you will build a "Reverse Image Search" engine. This system will allow a user to either Type a description (e.g., "A photo of a mountain with a red tent") or Upload a photo to find similar ones in your database.

---

1. Project Requirements

• Model: OpenAI CLIP (clip-ViT-B-32).
• Data: A folder of at least 50 varied images (nature, people, objects).
• Goal: Implement both Text-to-Image and Image-to-Image search.

---

2. Technical Stack

1. Python Pillow: For image processing.
2. sentence-transformers: For generating CLIP embeddings.
3. ChromaDB: For vector storage.

---

3. Implementation (Python)

from sentence_transformers import SentenceTransformer
from PIL import Image

model = SentenceTransformer('clip-ViT-B-32')

# 1. Ingestion: Embed images
def ingest_images(image_paths, collection):
    for path in image_paths:
        img = Image.open(path)
        vec = model.encode(img).tolist()
        collection.add(
            embeddings=[vec],
            metadatas=[{"path": path}],
            ids=[path]
        )

# 2. Querying: Text-to-Image
def search_by_text(query_text, collection):
    vec = model.encode(query_text).tolist()
    return collection.query(query_embeddings=[vec], n_results=3)

# 3. Querying: Image-to-Image (Reverse Search)
def search_by_image(img_path, collection):
    img = Image.open(img_path)
    vec = model.encode(img).tolist()
    return collection.query(query_embeddings=[vec], n_results=3)

---

4. Evaluation Criteria

• Semantic Match: Does the text search "Mountain" find mountains instead of beaches?
• Visual Match: In reverse image search, does the engine find images with similar colors, shapes, and compositions?
• Scale Capability: If you had 1 million images, would your metadata storage strategy (Module 13.5) still work?

---

Deliverables

1. A Python file with the ingest, search_by_text, and search_by_image functions.
2. A video or screenshot demonstration of a specific search result showing a successful match.

---

Multimodal search is the frontier of AI application development. Congratulations on building your first one!