Measuring the Immeasurable

Evaluating a "Translation" or a "Creative Story" is much harder than evaluating a math problem. There isn't always a "Correct" answer. However, if you are going to put an AI in front of your customers, you must be able to prove that it is High Quality.

On the AWS Certified AI Practitioner exam, you will be asked about Amazon Bedrock Model Evaluation. This allows you to test models against your specific benchmarks.

---

1. Two Ways to Evaluate

AWS provides two distinct "Paths" for evaluation:

A. Automatic Evaluation

The computer evaluates the computer. You provide a "Dataset" of prompts and "Reference" answers. The system uses mathematical scores to see how close the AI's answer is to the reference.

• Metrics:
  • Accuracy: Is the info correct?
  • Robustness: Does it stay safe even if the user is rude?
  • Toxicity: Does it use bad language?

B. Human Evaluation

The computer gives answers to a person who rates them.

• Metrics:
  • Helpfulness: Did it actually solve my problem?
  • Naturalness: Does it sound like a robot or a person?
  • Creative Quality: Is the writing good?

---

2. Using Amazon Bedrock Model Evaluation

This is a specific feature within the Bedrock console.

1. You select the Model(s) you want to test.
2. You select the Target Metric (Helpfulness, Accuracy, etc.).
3. You provide the Dataset (Prompts).
4. For human eval: Bedrock manages the "Rating UI" so your team (or a 3rd party team from AWS) can rank the responses.

---

3. The "Grounding" Check (RAG Evaluation)

If you are using RAG (from Module 6), you have a special metric called Grounding.

• Does the AI's answer only come from the provided documents?
• If the AI mentions a fact that isn't in the document, it has failed the grounding check (it is "Hallucinating" from its own internal memory).

---

4. Visualizing the Evaluation Loop

graph TD
    A[Prompt Dataset] --> B[Model A]
    A --> C[Model B]
    
    subgraph Evaluation_Engine
    B --> D[Automatic Scoring]
    C --> D
    B --> E[Human Review/Ranking]
    C --> E
    end
    
    D & E --> F[Comparison Report]
    F --> G{Decision}
    G -->|Model A is better| H[Deploy Model A]
    G -->|Fix Prompt| A

---

5. Summary: Continuous Testing

Evaluation is not a "One-time" event. You should re-evaluate your model:

• Every time you change your Prompt.
• Every time you update your Knowledge Base.
• Every time a New Model is released on Bedrock.

---

Exercise: Identify the Evaluation Type

A hotel chain wants to know if their new chatbot sounds "Friendly and Welcomig." They realize that a computer cannot measure "Friendliness" accurately. Which Bedrock feature should they use?

• A. Automatic Evaluation.
• B. Model Training Job.
• C. Human Evaluation.
• D. SageMaker Clarify.

The Answer is C! Human experience concepts like "Friendliness" or "Brand Voice" require human reviewers to provide valid rankings.

---

Knowledge Check

What's Next?

The model is tested and approved. Now it’s time to go live. In our final lesson of Module 13, we look at Deployment and monitoring pipelines.