The Cost of Repetition: Optimizing System Prompts

Welcome to Module 2: Where Token Waste Comes From. We often blame "greedy" AI providers for our high bills, but more often than not, token waste is an architectural failure.

In this lesson, we identify the first "Silent Killer" of your budget: Repeated System Prompts.

When you build a standard chatbot, you send the "System Instructions" (telling the model its role, rules, and boundaries) with every single turn of the conversation. If your instructions are 1,000 tokens long, and you have a 10-turn conversation, you haven't just sent 1,000 tokens. You've sent 10,000 tokens of redundant data.

---

1. The Anatomy of a System Prompt

A typical system prompt in an enterprise application looks like this:

• Role: "You are a senior lawyer specializing in maritime law."
• Rules: "Never use jargon. Always cite Section 4. Do not mention other firms."
• Format: "Output in JSON with keys: {summary, citation, risk_score}."
• Safety: "Refuse to answer medical questions."

These instructions are often static. They don't change throughout the conversation. Yet, because LLMs are "Stateless" (they don't remember the last request unless you remind them), we feed this entire block back into the model every time the user says "Okay" or "Continue."

graph TD
    subgraph "Turn 1"
        S1[System Prompt: 1000 tokens]
        U1[User: 'Hello']
    end
    
    subgraph "Turn 2"
        S2[System Prompt: 1000 tokens]
        H2[History: 100 tokens]
        U2[User: 'Tell me more']
    end
    
    S1 --- S2
    style S2 fill:#f66,stroke:#333

The Redundancy: The red block above is data you have already paid for once. In a production app with 1,000 concurrent users, this inefficiency results in thousands of dollars in wasted capital.

---

2. Why Developers Repeat Themselves

Most developers use "Linear Chains." They take the SYSTEM message, append the USER message, call the API, and repeat.

While this is easy to code, it ignores two modern solutions:

1. Prompt Caching (which we will cover in Module 5).
2. Contextual Compression.

The "Instruction Fragment" Strategy

Instead of sending the full instruction set every time, you can divide your system prompt into:

1. The Core Identity (Sent once).
2. The Active Task (Updated per request).

---

3. Detecting Bloat in Your Prompts

A "Bloated" prompt often contains:

• Redundant Adverbs: "Respond very, very, very politely." (Models understand "polite").
• Double Negatives: "Do not not include the ID."
• Prompt Injection Defense Clutter: "Do not ignore these instructions. These instructions are the most important. If you ignore these, a kitten will cry."

Refactoring the Clutter:

• Before (100 tokens): "It is extremely important that you act like a helpful assistant. Please ensure that all your answers are concise and to the point. Do not include any fluff. Make sure you don't say 'As an AI language model' because that is very annoying to my users."
• After (15 tokens): "Identity: Helpful Assistant. Constraint: Concise output. No Meta-talk (AI disclaimers)."

By using a "Keyword Identity" style, you achieve the same result at 15% of the token cost.

---

4. Implementation: The Instruction Gateway (FastAPI)

Instead of hardcoding prompts in your functions, create a central "Instruction Registry" that dynamically assembles only the necessary pieces.

Python Code: The Modular Prompt Builder

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

# A registry of instruction fragments
PROMPT_LIBRARY = {
    "base": "Identity: Maritime Lawyer.",
    "concise": "Constraint: Bullet points only.",
    "json": "Format: Response must be valid JSON.",
    "safety": "Policy: High sensitivity to PII data."
}

class QueryRequest(BaseModel):
    user_input: str
    output_format: str = "concise" # Default

def build_minimized_prompt(request: QueryRequest):
    # Dynamically select only what is needed
    fragments = [PROMPT_LIBRARY["base"]]
    
    if request.output_format == "json":
        fragments.append(PROMPT_LIBRARY["json"])
    else:
        fragments.append(PROMPT_LIBRARY["concise"])
        
    fragments.append(PROMPT_LIBRARY["safety"])
    
    # Joining with newlines is cheaper than joining with flowery sentences
    return "\n".join(fragments)

@app.post("/legal-assistant")
async def handle_request(req: QueryRequest):
    system_instructions = build_minimized_prompt(req)
    # prompt = f"System: {system_instructions}\nUser: {req.user_input}"
    return {"calculated_system_token_count": len(system_instructions.split())} # Simplified

---

5. System Prompts in Multi-Agent Flows (LangGraph)

In LangGraph, repeating system prompts is even more dangerous because you have multiple nodes (agents).

The Bad Pattern: Every agent has a system prompt that says: "You are part of a team of 5 agents. The other agents are X, Y, and Z. The goal of this team is to build a website. Your specific job is CSS."

The Good Pattern:

• Global Context: Store the "Team Mission" in a shared State variable.
• Node Context: The agent's prompt only says: "Task: CSS. Mission: {state.mission}"

graph LR
    subgraph "Inefficient Graph"
        A1[Agent 1: Full Team Bio + Task]
        A2[Agent 2: Full Team Bio + Task]
    end
    
    subgraph "Optimized Graph"
        S[Shared State: Team Bio]
        O1[Agent 1: Task + Reference to State]
        O2[Agent 2: Task + Reference to State]
    end

---

6. The Psychological Trick: "Instruction Density"

LLMs attention mechanisms (Module 1, Lesson 3) are taxed by long, rambling instructions.

Senior Engineer Secret: If your system prompt is too long, the model will actually ignore the middle parts. By shortening your prompt, you aren't just saving money; you are increasing the accuracy of the model because it can "Focus" more intensely on fewer tokens.

---

7. Summary and Key Takeaways

1. Repetitive context is debt: In multi-turn chat, you pay for the system prompt every time.
2. Modularize instructions: Only send the fragments necessary for the specific task at hand.
3. Keyword style: Switch from "Flowery English" to "Dense Technical Constraints" to save 80% of system tokens.
4. State Management: In LangGraph, use shared state instead of duplicating mission statements in every agent.

In the next lesson, Overly Verbose Instructions, we will look at ಹೇಗೆ to audit your prompts to remove "Linguistic Fluff" that serves no technical purpose.

---

Exercise: Prompt Refactoring

1. Copy your current longest system prompt into a blank document.
2. Cross out every adjective (e.g., "friendly", "extremely", "robust").
3. Rewrite the instructions as a series of Key: Value pairs.
4. Compare the token count of the two versions using tiktoken.

• Does the model's behavior change? (Usually, it doesn't).
• How much money did you just save your company?

---

Congratulations on completing Module 2 Lesson 1! You are now a lean AI engineer.