Squeezing Out Every Millisecond

In online prediction, every millisecond counts. A slow model can lead to a poor user experience and lost revenue. In this lesson, we'll cover some techniques for performance tuning and latency optimization.

---

1. Autoscaling

Vertex AI Prediction automatically scales the number of nodes in your endpoint based on CPU/GPU utilization. You can control the autoscaling behavior by setting the following parameters:

• min_replica_count: The minimum number of nodes to keep running at all times. Set this to a value greater than 0 to avoid cold starts.
• max_replica_count: The maximum number of nodes to scale up to.
• target_cpu_utilization_percentage or target_accelerator_duty_cycle_percentage: The target utilization for your nodes. If the utilization exceeds this value, the autoscaler will add more nodes.

Exam Tip: For predictable traffic patterns, use a scheduled scaling policy to increase the min_replica_count before the traffic spike.

---

2. Server-Side Batching

If you are receiving a high volume of requests, you can improve throughput by batching requests on the server-side. This allows you to process multiple requests in a single forward pass, which can be more efficient than processing each request individually.

You can enable server-side batching by setting the max_batch_size parameter when you deploy your model.

---

3. Model Optimization

You can also optimize the model itself to improve performance. Some common techniques include:

• Quantization: Converting model weights from 32-bit floating-point to 8-bit integers. This can reduce model size and increase inference speed, but may result in a small loss of accuracy.
• Pruning: Removing unimportant weights from the model to reduce its size and complexity.
• Knowledge Distillation: Training a smaller, faster model to mimic the behavior of a larger, more accurate model.

---

Knowledge Check