Images (PNG, JPG, Diagrams, Screenshots)

Images contain valuable information. Learn to process different image types for RAG.

Image Processing Pipeline

graph LR
    A[Image File] --> B{Image Type?}
    B -->|Text-heavy| C[OCR]
    B -->|Diagram| D[Vision Model]
    B -->|Photo| E[CLIP Embedding]
    B -->|Chart| F[Vision + Data Extract]
    
    C & D & E & F --> G[Indexed Content]

OCR for Text Images

import pytesseract
from PIL import Image

def extract_text_from_image(image_path):
    image = Image.open(image_path)
    
    # OCR
    text = pytesseract.image_to_string(image)
    
    # Get bounding boxes
    data = pytesseract.image_to_data(image, output_type=pytesseract.Output.DICT)
    
    return {
        'text': text,
        'layout': data
    }

Vision Model Analysis

def analyze_diagram(image_path):
    with open(image_path, 'rb') as f:
        image_b64 = base64.b64encode(f.read()).decode()
    
    response = claude.messages.create(
        model="claude-3-5-sonnet",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this diagram in detail"},
                {"type": "image", "source": {"type": "base64", "data": image_b64}}
            ]
        }]
    )
    
    return response.content[0].text

Image Embeddings (CLIP)

import clip

model, preprocess = clip.load("ViT-B/32")

def embed_image(image_path):
    image = Image.open(image_path)
    image_input = preprocess(image).unsqueeze(0)
    
    with torch.no_grad():
        image_embedding = model.encode_image(image_input)
    
    return image_embedding.numpy()

Processing Strategy by Type

Screenshots

• OCR for text
• Vision model for layout understanding
• Extract UI element positions

Diagrams

• Vision model for interpretation
• Extract relationships
• Identify components

Charts/Graphs

• Vision model for data extraction
• Convert to structured data
• Extract trends

Photos

• CLIP embedding for similarity
• Object detection if needed
• Caption generation

Complete Example

def process_image(image_path):
    # Classify image type
    image_type = classify_image(image_path)
    
    if image_type == 'text_heavy':
        result = extract_text_from_image(image_path)
    elif image_type in ['diagram', 'chart']:
        result = analyze_diagram(image_path)
    else:
        result = {
            'embedding': embed_image(image_path),
            'description': generate_caption(image_path)
        }
    
    return result

Next: Audio processing.