Module 13 Lesson 1: Log management for LLM applications

In traditional web apps, we log IP addresses and URLs. In AI apps, we must log Prompts and Responses. This is much more complex and sensitive.

1. What to Log (The Security List)

1. Raw Prompt: The exact text the user sent. (Crucial for after-the-fact injection analysis).
2. System Prompt Version: Which version of the "Instructions" was active during this conversation.
3. Model Metadata: The model name, version, temperature, and any "seed" values used.
4. Tool Calls: Which internal functions were called, and with what arguments.
5. Tokens Used: To detect "Wallet Draining" or Denial-of-Service attacks.

---

2. What NOT to Log (The Privacy List)

1. PII in Prompts: If you log everything, and a user enters their secret password or health info, that info is now stored in your logs forever.
   • The Fix: Use Redaction-on-Log. Scan the prompt for PII before it hits the log file and replace it with [STILL_PROTECTED_DATA].
2. Full Vector Content: Don't log the entire text of the documents retrieved from RAG, just the Document IDs.

---

3. Structured Logging for AI

Don't use "Text Logs" for AI. Use JSON Logs. A good AI log entry looks like this:

{
  "timestamp": "2024-01-01T12:00:00Z",
  "user_id": "user_123",
  "prompt_hash": "a1b2c3d4",
  "model": "gpt-4",
  "tool_calls": [{"name": "get_weather", "args": {"city": "NY"}}],
  "security_flags": ["high_similarity_to_known_injection"]
}

---

4. Centralized Observability

Use tools designed for LLM logging like LangSmith, PromptLayer, or Arize Phoenix. These tools allow you to "Visualize" the chain of thought and see exactly where a prompt injection might have occurred in a multi-step workflow.

---

Exercise: The Log Architect

1. Why is a prompt_hash useful for identifying repeat attackers without storing the full prompt?
2. You find a log file that contains clear-text credit card numbers from your users. What is your first step?
3. What is the "Retention Policy" for your security logs? (How long do you keep them?)
4. Research: What is "Traceability" in the context of AI Agent execution?

---

Summary

Logging is your Memory. Without it, you can't learn from attacks or prove that a breach happened. But if your logs are insecure, they become a goldmine for the very attackers you are trying to stop.

Next Lesson: Stop it now: Detecting prompt injection in real-time.