Feedback Loops & User Corrections

Your users are the best source of "Truth." Capturing their feedback allows you to move from a static system to an Evolving RAG pipeline.

Capturing Feedback

Implicit Feedback (Behavioral)

• Copy-Paste: The user copied the answer (indicator of success).
• Follow-up: The user asked a clarifiying question (indicator of ambiguity).
• Source Click: Did the user click the provided source link to verify?

Explicit Feedback (Rating)

• Thumbs Up / Down: The simplest metric.
• Correction: "Actually, the price is $40, not $50."

Using Corrections to Improve Quality

When a user provides a correction:

1. Log it: Store the query, the wrong answer, and the user's correction.
2. Review: Have a human (or a senior LLM) verify the correction.
3. Updating the Index: If the document was wrong, update the source file and re-index. If retrieval was wrong, use this as a new test case for your re-ranker.

RLHF (Reinforcement Learning from Human Feedback)

At a large scale, you can use these "Thumbs Up/Down" pairs to fine-tune your re-ranker model. A model that understands what your specific users like is far superior to a generic one.

Implementation: The Feedback API

# Simple endpoint for capturing user data
@app.post("/feedback")
async def capture_feedback(query_id: str, rating: int, comment: str):
    db.save_feedback(query_id, rating, comment)
    if rating == -1:
        trigger_manual_review(query_id)

Exercises

1. Why do users often forget to give "Explicit" (Thumbs Up) feedback?
2. How can you incentivize users to provide high-quality corrections?
3. What is the danger of "Malicious Feedback" (e.g., users intentionally training the bot to give wrong answers)?