Skip to content

Persistence & Backup

Data is precious, and Quiver takes data safety seriously. Let's explore how Quiver keeps your vectors safe and sound, even when things go wrong! 🛡️

Persistence

How Persistence Works

Quiver automatically persists your index to disk at regular intervals. This ensures that your data is safe even if the application crashes or is terminated unexpectedly.

The persistence process:

  1. Saves the HNSW graph structure
  2. Stores all vector data
  3. Persists metadata to DuckDB
  4. Updates the changelog

Configuring Persistence

You can configure how often Quiver persists data:

config := quiver.Config{
    // ... other settings ...
    PersistInterval: 5 * time.Minute, // Default is 5 minutes
}

A shorter interval means less data loss in case of a crash, but more I/O overhead.

Manual Persistence

You can also trigger persistence manually:

// Manually persist the index
err := idx.Save("./data/manual_save")
if err != nil {
    log.Fatalf("Failed to save index: %v", err)
}

This is useful before shutting down or when you want to ensure all data is saved at a specific point.

Incremental Persistence

Quiver uses incremental persistence to minimize I/O:

  1. Only new or changed vectors are persisted
  2. The changelog tracks which vectors have changed
  3. The persistence worker only writes what's necessary

This makes persistence efficient even with large indices.

Backup

Automatic Backups

Quiver can create automatic backups at regular intervals:

config := quiver.Config{
    // ... other settings ...
    BackupInterval:    1 * time.Hour,    // How often to create backups
    BackupPath:        "./backups",      // Where to store backups
    BackupCompression: true,             // Whether to compress backups
    MaxBackups:        5,                // Maximum number of backups to keep
}

Each backup includes:

  • The complete HNSW graph
  • All vector data
  • All metadata
  • Configuration settings

Manual Backups

You can also create backups manually:

// Create a full backup
err := idx.Backup("./backups/manual", false, true)
if err != nil {
    log.Fatalf("Backup failed: %v", err)
}

The parameters are:

  • Backup path
  • Whether to create an incremental backup
  • Whether to compress the backup

Incremental Backups

For large indices, incremental backups can save time and space:

// Create an incremental backup
err := idx.Backup("./backups/incremental", true, true)

Incremental backups only store what has changed since the last backup.

Backup Rotation

Quiver automatically manages backup rotation:

  1. When MaxBackups is reached, the oldest backup is deleted
  2. This prevents backups from consuming too much disk space
  3. You always have the most recent backups available

Recovery

Loading from Persistence

When you restart Quiver, it automatically loads from the persisted data:

// Load the index from disk
idx, err := quiver.Load(config, logger)
if err != nil {
    log.Fatalf("Failed to load index: %v", err)
}

This restores the index to its last persisted state.

Restoring from Backup

If you need to restore from a backup:

// Create a new index with the same configuration
idx, _ := quiver.New(config, logger)

// Restore from backup
err := idx.Restore("./backups/backup_20230615_120000")
if err != nil {
    log.Fatalf("Restore failed: %v", err)
}

This completely replaces the current index with the data from the backup.

Disaster Recovery

For disaster recovery scenarios:

  1. Keep backups in a separate location (ideally off-site)
  2. Regularly test the restore process
  3. Consider using cloud storage for backups
  4. Document the recovery procedure

Best Practices

Persistence Strategy

  • Use a shorter PersistInterval for critical data
  • Manually trigger persistence before shutdowns
  • Store data on reliable storage (SSD recommended)
  • Monitor disk space to prevent running out

Backup Strategy

  • Use compressed backups to save space
  • Keep enough backups to meet your recovery point objective (RPO)
  • Store backups in a different location than the primary data
  • Regularly test the restore process

Example Configuration

config := quiver.Config{
    // ... core settings ...

    // Persistence
    PersistInterval: 2 * time.Minute,

    // Backups
    BackupInterval:    6 * time.Hour,
    BackupPath:        "/mnt/backup/quiver",
    BackupCompression: true,
    MaxBackups:        24, // Keep 6 days worth
}

Monitoring

Quiver provides metrics to monitor persistence and backup:

// Get metrics
metrics := idx.CollectMetrics()

// Check persistence metrics
fmt.Printf("Last persist time: %v\n", metrics["last_persist_time"])
fmt.Printf("Vectors since last persist: %v\n", metrics["vectors_since_last_persist"])

// Check backup metrics
fmt.Printf("Last backup time: %v\n", metrics["last_backup_time"])
fmt.Printf("Backup count: %v\n", metrics["backup_count"])

Next Steps

Now that you've learned how to keep your data safe, check out: