CPU vs. GPU for Serving

The choice of hardware for serving depends on the model architecture and the latency requirements.

---

1. When to Use CPUs

• Model Type: Traditional ML models like linear regression, logistic regression, and gradient boosted trees.
• Reasoning: These models are typically not computationally intensive and are often I/O bound. The overhead of moving data to a GPU can be greater than the benefit of the GPU's processing power.
• Example: A recommendation model that does a simple dot product between user and item embeddings.

---

2. When to Use GPUs

• Model Type: Deep learning models like CNNs and Transformers.
• Reasoning: These models involve a large number of matrix multiplications, which are highly parallelizable and can be significantly accelerated by GPUs.
• Example: An image classification model that uses a ResNet architecture.

GPU Selection for Serving

• NVIDIA T4: The most cost-effective GPU for serving. It provides a good balance of performance and cost.
• NVIDIA A100: A more powerful and expensive GPU. Use this for models with very low latency requirements or very large models that don't fit on a T4.

---

3. NVIDIA TensorRT (TF-TRT)

TensorRT is a library that optimizes TensorFlow graphs for inference on NVIDIA GPUs. It can provide a significant performance boost by:

• Fusing layers: Combining multiple layers into a single layer to reduce kernel launch overhead.
• Quantizing models: Converting model weights from 32-bit floating-point to 8-bit integers to reduce memory usage and increase inference speed.

Exam Tip: If you see a question about optimizing the performance of a deep learning model on a GPU, the answer is likely to involve TensorRT.

---

Knowledge Check