Scaling Chroma and Retrieval Layers

If your RAG system goes from 10 users to 10,000, your retrieval layer will become the primary bottleneck. Scaling Chroma (or any vector DB) involves distributed computing principles.

Vertical Scaling (The First Step)

• Increase the RAM of your Chroma server. HNSW search speed is bound by the amount of the index that can fit in memory.
• Use high-speed NVMe SSDs for the persistence layer.

Horizontal Scaling (The Next Step)

Read Replicas

You can have one "Write" node (where you add new documents) and multiple "Read" nodes. Each read node has a copy of the vector index and handles user queries.

• LB: Use a Load Balancer to distribute queries across read replicas.

Sharding

If your index is too large for one machine's RAM (e.g., 50GB index on 32GB RAM nodes), you must Shard the data.

• Shard A: Documents 1 - 500,000.
• Shard B: Documents 500,001 - 1,000,000.
• Query: Search both shards in parallel and combine (fuse) the results.

Moving to "Native" Cloud Vector DBs

At extreme scale (billions of vectors), you might outgrow Chroma and move to:

• Pinecone: Fully managed, specializes in extremely high-scale SaaS.
• Amazon OpenSearch: Better for complex hybrid search (vector + rich text).
• Weaviate: Scalable, open-source, with advanced multimodal support.

Monitoring for Scaling

• Latency (P99): If the slowest 1% of your searches are taking > 1s, you need more read replicas.
• RAM Usage: If RAM > 80%, you need more nodes (Sharding).
• CPU Usage: Intensive during re-indexing or re-ranking.

Exercises

1. Why is "Sharding" more complex than "Read Replication"?
2. How many dimensions can a single Chroma collection handle before performance degrades significantly?
3. Design a scaling plan for a RAG system that expects 1 million new vectors per week.