Integrating Vision

In the first half of this course, our agents were "Blind"—they only understood text. But some of the most powerful agentic use cases happen when an agent can See.

A "Quality Control Agent" looks at a photo of a widget on a factory line.
A "UI Testing Agent" looks at a screenshot of your website to find broken buttons.
A "Medical Agent" looks at a X-ray to find anomalies.

In this lesson, we will learn how to integrate Vision Models (GPT-4o, Claude 3.5 Sonnet) into your LangGraph workflows.

1. How Vision Models Work (Conceptually)

Vision models don't "Look" at the actual file. They use Vision Encoders (like CLIP) to translate the pixels into "Visual Tokens" that the LLM can understand alongside text tokens.

The Input Format

When sending an image to an agent, you usually send a Base64 String or a Signed URL.

Important: Images consume a LOT of tokens. A single high-resolution screenshot can cost as much as 1,000 to 2,000 text tokens.

2. Setting Up a Vision Node in LangGraph

A Vision Node is just a standard node where the LLM is configured to receive an image_url.

from langchain_core.messages import HumanMessage

async def vision_node(state):
    # Retrieve the image from the state (Module 10.3)
    image_data = state["current_image_base64"]
    
    message = HumanMessage(
        content=[
            {"type": "text", "text": "Describe any security risks in this office floor plan."},
            {
                "type": "image_url",
                "image_url": {"url": f"data:image/jpeg;base64,{image_data}"},
            },
        ]
    )
    
    response = await llm.ainvoke([message])
    return {"analysis": response.content}

3. Use Case: The "Web Voyager" (UI Interaction)

A popular agent pattern involves an agent that "Clips" its own screenshots to navigate a website.

Agent takes a screenshot of a browser.
Vision Node identifies the pixel coordinates of the "Login" button.
Execution Node calls a tool: click(x=450, y=200).
Repeat: Until the task is done.

4. Handling Multiple Images

Models have a "Context Window" for images too.

You can't send 50 images in one prompt without slowing down the model significantly.
Strategy: Sampling.
If you have a video, don't send every frame. Send 1 frame per second until the agent sees what it needs.

5. Vision Guardrails: The "Privacy Mask"

If an agent is looking at a user's screen or webcam, you must implement a PII Masking Tool.

Logic: Use a small, local model (like Rekognition or a YOLO detector) to find faces or credit card numbers.
Action: Blur those regions before the image is sent to the "Cloud" LLM.

6. Real-World Tip: OCR vs. Vision

If you want to read text from an image (like an invoice), you have two choices:

Cloud Vision (LLM): Great at understanding context (e.g., "What is the total after tax?").
Traditional OCR (Tesseract/AWS Textract): Great at raw data extraction but "Blind" to context. Best Pattern: Use traditional OCR to extract the text, and give the TEXT to the agent. It is 10x cheaper and 100% accurate on characters.

Summary and Mental Model

Think of a Vision-enabled Agent like a Collaborator on a Zoom Call.

They can hear you (Text).
But when you share your screen (Vision), they can suddenly understand Exactly what you are talking about.
Vision adds the "Spatial Context" that is missing from raw language.

Exercise: Vision Mapping

The Scenario: You are building an agent for Home Insurance. The user uploads a photo of a broken window.
- What are 3 specific questions the "Vision Node" should ask the model to answer? (e.g., "Is there glass inside or outside?")
Optimization: An image is 5MB.
- Should you resize it before sending it to the LLM?
- What resolution is typical for "Reading Text"? (Hint: Most models prefer 1000px on the longest side).
Privacy: Why is "Vision" a higher security risk than "Text"?
- Give an example of something a user might "Accidentally" show an agent in a background of a photo. Ready for sound? Next lesson: Voice and Audio Agents.

The Visionary Agent: Integrating Image Reasoning