Verification Loops

A Verification Loop is a second-pass process where a different LLM (or the same one with a different prompt) "audits" the first model's response. This is the "Double-Check" of AI engineering.

The Basic Loop Structure

1. Step 1: Generation: Model A generates the RAG response.
2. Step 2: Verification: Model B takes the Context + Response and checks for:
   • Hallucinations: Did the answer include facts not in the context?
   • Toxicity/Safety: Did the answer violate any rules?
   • Completeness: Did the answer actually address every part of the user query?
3. Step 3: Feedback: If it fails, send it back to Step 1 with the "Error Log" for a rewrite.

Implementation: The "Adversarial" Verifier

verifier_prompt = f"""
You are a factual auditor. 
Review the following Answer against the provided Context.
Flag any claim in the Answer that is NOT directly supported by the Context.

Answer: {model_a_response}
Context: {retrieved_chunks}

Reply with 'PASSED' or a list of 'ERRORS'.
"""

Self-Correction with "Critics"

In advanced RAG (like Self-RAG), the model itself critiques its work using special tokens:

• IsRel: Is the document relevant?
• IsSup: Is the answer supported?
• IsUse: Is the answer useful?

When to Use Verification Loops

• When Accuracy is Paramount: High-stakes use cases.
• When Cost is Less Critical: Verification adds an extra API call (cost/latency).

Monitoring the "Rewrite" Rate

Trace how many times a response has to be rewritten. A high rewrite rate means your Retrieval step is failing and providing poor data for the Generator.

Exercises

1. Why is it better to use a different model for verification (e.g., GPT-4 verifying Claude)?
2. Design a loop that stops after 3 failed verification attempts.
3. How can you use "Human-in-the-loop" for critical verification steps?