Module 13 Lesson 3: Anomaly detection for AI usage patterns

Attackers don't always use "Bad Words." Sometimes they use "Normal Words" in Abnormal Ways. This is where Anomaly Detection comes in.

1. Token Velocity (Denial of Wallet)

LLMs cost money per token.

• The Anomaly: A user who usually sends 50-word prompts suddenly starts sending 1,000 requests per minute with 5,000-word payloads.
• The Risk: This is a Denial of Wallet (DoW) attack designed to spike your cloud bill.
• The Detection: Set a "Moving Average" for token usage. If a user exceeds 5x their average within 10 minutes, trigger an alert.

---

2. Unusual Tool Sequences

AIs are designed to follow a "Happy Path."

• Normal Path: SearchDocs -> Summarize -> Respond.
• The Anomaly: A user triggers SearchDocs -> ListDatabaseTables -> SearchDocs -> ExportToCSV.
• The Risk: This sequence suggests the user is performing Reconnaissance and Data Exfiltration. The AI is being used as a "Crawler" for your internal networks.

---

3. "Strange" Embeddings

In a RAG system, you can monitor the Distance between the question and the documents.

• The Anomaly: A user asks a question that is "Mathematically Distant" from every document in your database, but they keep trying different variations.
• The Risk: The user is "Probing" the edges of your knowledge base to find "Gaps" or "Shadow Data" that they can exploit.

---

4. Response Time Shifting

If your AI normally takes 2 seconds to answer, but suddenly takes 30 seconds for a specific user:

• The Risk: The user might have found a "Complexity attack" (like a prompt that causes the model to enter a recursive loop). This is the AI version of "ReDoS" (Regular Expression Denial of Service).

---

Exercise: The Threat Hunter

1. Why is it harder to detect an "Anomaly" in a system with 1 million users than in a system with 10 users?
2. You see a user who only chats with the AI between 2 AM and 4 AM. Is this an anomaly? (Hint: Does it matter if they live in London vs. Tokyo?)
3. How can you use "Cluster Analysis" to find a group of bots all using the same attack pattern?
4. Research: What is "Outlier Detection" and how does it apply to high-dimensional embedding spaces?

---

Summary

Anomaly detection is about Context. You must know what "Normal" looks like before you can find the "Abnormal." By monitoring the behavior of the system rather than just the content, you can catch sophisticated attackers who know how to hide their tracks.

Next Lesson: Centralizing the fight: Building an AI security operation center (SOC).