Cost and Performance Monitoring

Building a working agent is an engineering success. Building a Profitable agent is a business success. Unlike traditional software, where your server cost is mostly fixed, an AI agent's cost is variable and uncapped. If an agent gets into a "Reasoning Loop," it can spend $100 in 10 minutes before you even notice.

In this lesson, we will learn how to monitor the "Vital Signs" of your agent's cost and performance.

---

1. Tracking the "Price-per-Session"

A single "Task" (e.g., "Summarize this repo") is not one API call. It might be 5 planning calls, 10 tool calls, and 1 final summary call. You must track the Cumulative Cost of the entire LangGraph thread.

Key Metric: Average Cost per Resolution (ACR).

• If your ACR is $0.50 but you only charge the user $0.10, your agent is a liability.

---

2. Hard Budget Guardrails (Kill Switches)

Your API layer (FastAPI) must enforce a "Token Quota" at the user and session level.

• Session Level: "If this specific thread exceeds 50,000 tokens, terminate the graph immediately and notify the user."
• User Level: "If User X has spent > $10 today, switch their agents to a cheaper model (e.g., Llama 3 8B)."

---

3. Monitoring Latency: The P99 Problem

Users care about Response Time.

• Average (P50): How long it take for a normal query (e.g., 2 seconds).
• Extreme (P99): How long it takes when the agent gets "Confused" and does 10 tool calls (e.g., 45 seconds).

The Target: Your P99 should be under 15 seconds for interactive agents. If it's higher, you need to simplify your Graph logic or optimize your Tools (Module 8.3).

---

4. Token Efficiency: The "Context Pruning" ROI

Every token you send in the "History" (Module 3.3) costs money.

• Optimization: If you implement "Summary Memory," you might reduce your history tokens by 50%.
• Impact: You just doubled your profit margin for that agent.

---

5. Performance Monitoring (DASHBOARD)

You should have a real-time dashboard (using Grafana or Datadog) that shows:

1. Total Spend today/this month.
2. Success Rate (based on Evals from Lesson 2).
3. Queue Depth: How many agents are currently waiting for a worker to become available?
4. API Provider Health: Are 429s (Rate Limits) or 503s (Downtimes) increasing for OpenAI?

---

6. Implementation Example: Tracking with Middleware

# Middleware to track total tokens in a LangGraph run
async def execute_agent(state):
    # Use callback handlers to capture tokens
    with get_openai_callback() as cb:
        result = await graph.ainvoke(state)
        # Log this to your analytics database
        log_cost(user_id=state['user_id'], cost=cb.total_cost, tokens=cb.total_tokens)
    return result

---

Summary and Mental Model

Think of Cost Monitoring like A Water Meter.

• If you don't look at it, you don't know there's a leak (A looping agent).
• By the time the bill arrives at the end of the month, the damage is done.

Real-time monitoring is the difference between a product and a prototype.

---

Exercise: Cost Analysis

1. The Math: An agent uses GPT-4o ($5 per 1M input tokens). Each user message sends 2,000 context tokens.
   • How much does it cost to handle 1,000 messages?
   • If you switch to GPT-4o-mini ($0.15 per 1M), how much do you save?
2. Strategy: Why is "First Token Latency" (TTFT) more important for user trust than "Total Task Latency"?
   • (Hint: Review Module 9.3 on Streaming).
3. Guardrails: Draft a "System Policy" for what happens when a user's credit card expires mid-way through a long-running agent task. Ready to automate the release? Next lesson: CI/CD for Agents.