The Unforgettable Record

In the final lesson of Domain 1, we focus on accountability. In a traditional application, if something goes wrong, you check the logs. In a Generative AI application, you must go deeper. You need to know:

1. Who made the request?
2. What specific prompt was sent?
3. Which model version was used?
4. How much did it cost?

In the AIP-C01 exam, you will be tested on your ability to configure Model Invocation Logging and CloudTrail to create a bulletproof audit trail.

---

1. Management vs. Data Events

Understanding the difference between these two types of logs is essential for the "Professional" mindset.

Management Events (CloudTrail)

• What: Actions like "CreateKnowledgeBase," "UpdateModelAccess," or "DeleteGuardrail."
• Why: To track who modified your AI infrastructure.
• Enabled By Default: Yes, usually for the last 90 days.

Data Events (Model Invocation Logging)

• What: The literal text of the prompt and the completion.
• Why: To debug hallucinations, audit safety compliance, and verify billing.
• Enabled By Default: NO. You must manually enable this in the Bedrock console.

---

2. Configuring Model Invocation Logging

You can send your AI logs to two main destinations:

1. Amazon CloudWatch Logs: Best for real-time monitoring and setting up alerts (e.g., "Alert me if the model generates a PII-Violation error").
2. Amazon S3: Best for long-term storage, auditing, and "Model Evaluation" (using logs to train better models later).

Pro Observation: Logging prompts and responses to S3 can be a security risk. You should ensure that the S3 bucket is encrypted with AWS KMS and that strict bucket policies are in place.

---

3. Metadata Tagging Strategy

Metadata is the "glue" that allows you to correlate an AI response with a real-world business event.

Metadata Field	Reason for Audit
user_id	To track which user is consuming the most budget.
task_type	To see if "Summarization" is more expensive than "Translation."
kb_id	To identify which knowledge base was used for a specific RAG call.
request_id	To link the model call back to your application's transaction logs.

---

4. Cost Allocation and Tagging

AI models are expensive. As a Developer Pro, you must implement Cost Allocation Tags.

• You can tag Bedrock resources (Knowledge Bases, Agents, Provisioned Throughput).
• The Result: At the end of the month, your Finance department can see exactly how much "Project Alpha" spent on AI compared to "Project Beta."

---

5. Visualizing the Audit Infrastructure

graph TD
    User[User] --> App[Your Application]
    App -->|Prompt + Metadata| B[Amazon Bedrock]
    
    subgraph Audit_Layer
    B -->|Management Events| CT[AWS CloudTrail]
    B -->|Prompt/Response| S3[S3 Audit Bucket]
    B -->|Metrics/Errors| CW[CloudWatch]
    end
    
    CW --> Alert[Admin Alert]

Architectural Pattern: A secure, observable AI workload.

---

6. Code Example: CloudWatch Alert for AI Failure

As a developer, you can use Boto3 to monitor "Throttling" errors and send a notification.

import boto3

# This is a conceptual snippet for setting up a CloudWatch Metric Filter
cw = boto3.client('logs')

def setup_alert():
    # Look for 429 errors in the Bedrock logs
    cw.put_metric_filter(
        logGroupName='/aws/bedrock/model-invocations',
        filterName='ThrottlingFilter',
        filterPattern='{ $.status = 429 }',
        metricTransformations=[
            {
                'metricName': 'BedrockThrottlingEvents',
                'metricNamespace': 'GenAI/Metrics',
                'metricValue': '1'
            }
        ]
    )
    print("CloudWatch filter for Throttling errors established.")

---

Knowledge Check: Test Your Audit Knowledge

---

Summary

You have officially finished Domain 1: Foundation Model Integration, Data Management, and Compliance. You know how to:

• Choose the best model (Module 2).
• Build the data pipelines (Module 3).
• Architect the knowledge bases (Module 4).
• Audit the entire system (Module 5).

Congratulations on reaching this milestone! In the next module, we leave the "Management and Data" phase and move to Domain 2: Implementation and Integration, where we learn to deploy these workloads at scale.

---

Next Module: The Deployment Dilemma: Bedrock vs. SageMaker vs. Custom