Tracing and Debugging with LangSmith

In traditional coding, when a line fails, you get a "Stack Trace" that tells you exactly where the error occurred. In agentic coding, a failure doesn't look like an error—it looks like a weird answer or an infinite loop.

To debug an agent, you need to see the "Mental Steps" it took. In this lesson, we will learn how to use LangSmith (and the concept of Tracing) to visualize the lifecycle of a request.

---

1. What is Tracing?

Tracing is the process of recording every input, output, and hidden prompt for every step of an agentic chain.

A single Trace includes:

• The Raw Prompt: Exactly what was sent to the LLM (including system instructions).
• The Response: The raw JSON or text returned.
• The Tool Call: The arguments the model chose.
• The Tool Output: What the Python code returned to the model.
• Latency: How many ms each step took.

---

2. Setting Up LangSmith

LangSmith is the industry standard for AgentOps. It integrates natively with LangChain and LangGraph.

The configuration (Environment Variables)

export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_API_KEY="ls__..."
export LANGCHAIN_PROJECT="my-production-agent"

Once these are set, every ainvoke or astream call is automatically "Uploaded" to the LangSmith dashboard.

---

3. The "Infinite Loop" Debug

A common agent bug:

1. Agent calls Search.
2. Search results are a bit vague.
3. Agent calls Search again with the same query.
4. Repeat 50 times (Burning $5 in tokens).

In the Trace View: You will see a "Staircase" of duplicate nodes. This is the signal that your Termination Logic (Module 3.3) is failing.

---

4. Prompt Versioning and A/B Testing

One of the most powerful features of tracing is Contextual Debugging.

• "Yesterday the agent was perfect. Today it's hallucinating."
• LangSmith allows you to compare the trace from yesterday's version of the prompt with today's version side-by-side.

---

5. Capturing "User Feedback" in Traces

A trace is only useful if you know if it was "Good" or "Bad."

• The Star Rating: When a user clicks "Thumbs Down" in your UI (Module 9), you should send a Feedback signal to the specific trace ID in LangSmith.
• Why? It allows you to filter your dashboard for "All FAILED traces where the user was unhappy." This is your Priority Debug List.

---

6. Real-World Tip: Redacting PII from Traces

Never send raw user data to a 3rd party tracing service unless you have a Data Processing Agreement (DPA).

• Use a Log Scrubber (Module 7.4) to replace names, emails, and credit card numbers with [REDACTED] before the trace is sent to the cloud.

---

Summary and Mental Model

Think of Tracing like A Black Box Flight Recorder.

• You don't look at it when the flight is smooth.
• But if there's a "Crash" (a hallucination or a wrong tool call), the Black Box is the only way to find the "Maneuver" that caused the failure.

If you aren't tracing, you aren't engineering; you're just guessing.

---

Exercise: Debugging Challenge

1. Setup: Sign up for a free LangSmith account and set the environment variables in your terminal.
2. Analysis: Run a simple agent loop and find the trace in the dashboard.
   • Identify the Total Token Cost of that specific run.
   • Identify the Slowest Node in the graph.
3. Logic: If an agent successfully calls a tool but the output of the tool is None, how would you detect this in the trace?
   • (Hint: Look at the "Output" section of the tool node). Ready to automate your testing? Next lesson: Evaluation (Eval) Pipelines.