Disaster Recovery: Backups and Regional Failover

High Availability (Module 15.4) protects you from a server crash. Disaster Recovery (DR) protects you from a "Region-Wide" failure (e.g., California goes offline) or a database-wide corruption event (e.g., a buggy script deletes all your vectors).

In this final scaling lesson, we learn how to "Rewind" and "Rebuild."

---

1. RPO and RTO: The DR Metrics

• RPO (Recovery Point Objective): How much data can you afford to lose?
  • (e.g., "We take snapshots every 4 hours, so we can lose up to 4 hours of vectors.")
• RTO (Recovery Time Objective): How fast must you be back online?
  • (e.g., "We must restore the search engine within 1 hour.")

---

2. Stateless Backups vs. Vector Snapshots

There are two ways to back up a vector database:

1. Snapshot: A point-in-time binary copy of the HNSW index and vectors.
   • Pros: Fast to restore.
   • Cons: Proprietary format (hard to move from Pinecone to Chroma).
2. Re-Ingestion Source: A Parquet/JSON file in S3 containing the raw text and metadata.
   • Pros: Platform-agnostic. You can rebuild your entire index on a different database engine if needed.
   • Cons: Extremely slow to restore (you have to re-embed and re-index the whole set).

---

3. Cross-Region Failover

For "Mission Critical" apps, you maintain a "Warm Standby" in a different region (e.g., us-west-2).

• Active-Passive: All traffic goes to New York. If New York fails, you change your DNS to point to Oregon.
• Active-Active: Users in Europe hit an EU vector store; users in the US hit a US vector store. Data is replicated across the Atlantic.

---

4. Implementation: Triggering a Snapshot (Python)

Using a managed provider's API:

import pinecone

index = pinecone.Index("prod-index")

# Trigger a manual snapshot before a big data migration
snapshot_name = "pre-upgrade-backup-2024"
index.create_snapshot(name=snapshot_name)

print(f"Snapshot {snapshot_name} initiated.")

---

5. Summary and Key Takeaways

1. Snapshots are Life: Take periodic snapshots of your HNSW index.
2. S3 is the Source of Truth: Always keep a copy of your raw text and metadata in a standard file format (JSON/Parquet) outside the database.
3. Test your DR: A backup that has never been restored is not a backup.
4. Choose your RTO: High RTO (seconds) requires expensive "Warm Standbys"; Low RTO (hours) allows for cheaper "Cold Snapshots."

---

Exercise: The DR Planner

1. You have 100M customer support vectors.
2. Strategy A: Snapshot every 24 hours.
3. Strategy B: Real-time cross-region replication.
4. The Question: If your primary region goes down, what is the "Cost per Minute of Downtime" for your business? If it's more than $1,000, which strategy is the correct choice?

---

Congratulations on completing Module 15! You are now a master of vector database scaling.