CI/CD for Agents

In traditional DevOps, "Continuous Integration" means running code tests. In Agentic DevOps, it means running Semantic Tests. When you update your system prompt, you are changing the "Firmware" of the brain. You must ensure that this change doesn't break existing behavior in unexpected ways.

In this lesson, we will learn how to build a CI/CD pipeline that treats Prompts as Code.

---

1. Version Everything: Prompt, Graph, and Model

You must never use latest in your production stack.

1. Model Versioning: Use gpt-4o-2024-05-13 instead of gpt-4o. (Providers update latest constantly, which can change your agent's behavior).
2. Prompt Versioning: Store your prompts in Git or a specialized "Prompt CMS" (like LangSmith or Portkey).
3. Graph Versioning: The structure of your nodes and edges must be tagged in your code repository.

---

2. The Pull Request (PR) Workflow for AI

When a developer submits a PR to change a prompt:

1. Linting: Check for basic "Red Flag" words or formatting errors.
2. Evaluation (CI): The CI runner (GitHub Actions) triggers the Eval Pipeline (Module 16.2).
3. The Scoreboard: The PR displays a summary:
   • "Accuracy: 95% (-2% from main)"
   • "Cost: $0.12/task (+5% from main)"
   • "Latency: 2.5s (No change)"

The PR should ONLY be merged if the accuracy doesn't drop significantly.

---

3. Canary Deployments for Agents

Never roll out a new agent version to 100% of users.

The Strategy:

1. Version A (Old): Continues to serve 95% of users.
2. Version B (New): Serves 5% of users.
3. Monitoring: Watch the Success Metric (e.g., "User clicked 'Thank You'") for Version B.
4. Promotion: If Version B performs better over 24 hours, roll it out to 100%.

---

4. Environment Parity

Your local "Dev" environment must match production exactly in terms of:

• Python Libraries (use requirements.lock).
• Tool Logic: The tools must return the same data schema in dev as they do in prod.
• Node Topology: Don't simplify the LangGraph for testing; test the exact structure that will live on the server.

---

5. Automated Rollbacks

If your production monitor (Module 16.3) detects a spike in Cost or Latency after a new release (e.g., the agent got stuck in a loop after a prompt change):

• The CI system should automatically "Revert" the API to the previous stable Docker image.

---

6. Implementation Example: GitHub Action Snippet

# .github/workflows/agent-eval.yml
jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - name: Run LangSmith Evaluators
        env:
          LANGCHAIN_API_KEY: ${{ secrets.LS_API_KEY }}
        run: python scripts/run_evals.py --branch ${{ github.head_ref }}

---

Summary and Mental Model

Think of CI/CD for Agents like Training a Team.

• You don't just hire a new person and let them talk to customers on day one.
• You Test their knowledge (CI).
• You let them shadow a pro (Canary).
• You review their performance daily (Monitoring).

Software becomes a living organism. Manage it accordingly.

---

Exercise: Deployment Design

1. Versioning: Why is it dangerous to change a Tool Schema (Module 11.2) without updating the Reasoning Node's System Prompt at the exact same time?
2. A/B Testing: You are testing two prompts for a Sales Agent.
   • Prompt A: Aggressive.
   • Prompt B: Helpful.
   • What is the "Success Metric" you would use in your CI/CD dashboard to decide which one to keep?
3. Rollback: Draft a "Disaster Recovery" plan for what happens if your LLM provider has an international outage.
   • (Hint: Review the Local Fallback pattern in Module 12.3). Ready to build something specialized? Next module: Specialized Agents: Coding and Automation.