Establishing Metrics and Baseline Monitoring
How to establish metrics and baseline monitoring for your ML models using Vertex AI Model Monitoring.
If You Can't Measure It, You Can't Improve It
Before you can monitor your model for performance degradation, you need to establish a baseline. This baseline will serve as a point of comparison for all future predictions.
1. Choosing Your Metrics
The first step is to decide which metrics you want to monitor. The choice of metrics will depend on your specific use case. Some common metrics include:
- Accuracy: The percentage of predictions that are correct.
- Precision: The percentage of positive predictions that are correct.
- Recall: The percentage of actual positives that are correctly identified.
- AUC: The area under the ROC curve.
- Log loss: A measure of the error in a logistic regression model.
2. Creating a Baseline
Once you have chosen your metrics, you need to create a baseline. The baseline is a snapshot of your model's performance on a representative dataset.
You can use Vertex AI Model Monitoring to create a baseline for your model. To do this, you will need to provide:
- A training dataset: This dataset will be used to calculate the baseline statistics.
- A set of metrics: The metrics that you want to monitor.
- A set of thresholds: The thresholds for each metric that will trigger an alert.
3. Monitoring Your Model
Once you have created a baseline, you can start monitoring your model. Vertex AI Model Monitoring will automatically collect data from your model's predictions and compare it to the baseline. If any of the metrics exceed the thresholds that you have set, you will receive an alert.
Knowledge Check
?Knowledge Check
You are training a model to predict whether or not a customer will churn. You want to monitor the model's performance in production. Which of the following metrics would be most appropriate?