Skip to content

Persistence

Quiver provides robust persistence capabilities to ensure your vector data and metadata are safely stored and quickly recoverable.

Overview

Persistence in Quiver includes:

  • Automatic periodic saving
  • Configurable persistence intervals
  • Safe atomic writes
  • Crash recovery
  • Optional encryption

Configuration

Basic Persistence Settings

config := quiver.Config{
    StoragePath: "/path/to/storage",
    PersistInterval: 5 * time.Minute,  // How often to persist
}

Advanced Configuration

config := quiver.Config{
    StoragePath: "/path/to/storage",
    PersistInterval: 5 * time.Minute,
    EncryptionEnabled: true,
    EncryptionKey: "your-32-byte-encryption-key",
}

How It Works

  1. Periodic Persistence
  2. Vector data and metadata are saved at configured intervals
  3. Changes are batched for efficiency

  4. Atomic writes ensure data integrity

  5. Storage Structure

  6. HNSW graph for vector indices
  7. DuckDB for metadata storage

  8. Separate backup files

  9. Background Processing

  10. Asynchronous persistence
  11. Non-blocking operations
  12. Configurable batch sizes

Data Safety Features

Atomic Writes

// Quiver uses atomic writes internally
if err := idx.persistToStorage(); err != nil {
    // Original data remains intact if persistence fails
}

Encryption

config := quiver.Config{
    EncryptionEnabled: true,
    EncryptionKey: "your-32-byte-encryption-key",
}

Crash Recovery

// Quiver automatically recovers on startup
idx, err := quiver.Load(config, logger)
if err != nil {
    // Handle recovery error
}

Best Practices

  1. Storage Configuration
  2. Use SSD for better performance
  3. Ensure sufficient disk space

  4. Monitor disk usage

  5. Persistence Settings

  6. Adjust interval based on update frequency
  7. Balance between safety and performance

  8. Consider backup strategy

  9. Monitoring

  10. Track persistence operations
  11. Monitor disk space
  12. Log persistence events

Performance Considerations

Batch Size

config := quiver.Config{
    BatchSize: 1000,  // Adjust based on your needs
}

Persistence Interval

config := quiver.Config{
    PersistInterval: 10 * time.Minute,  // Longer intervals for better performance
}

Error Handling

idx, err := quiver.New(config, logger)
if err != nil {
    switch {
    case errors.Is(err, ErrStoragePathNotFound):
        // Handle storage path issues
    case errors.Is(err, ErrEncryptionKeyInvalid):
        // Handle encryption key issues
    default:
        // Handle other errors
    }
}

Monitoring and Metrics

// Get persistence metrics
metrics := idx.CollectMetrics()
fmt.Printf("Last persist ID: %d\n", metrics["last_persist_id"])
fmt.Printf("Persist interval: %.2f seconds\n", metrics["persist_interval"])

Recovery Scenarios

Manual Recovery

// Force immediate persistence
if err := idx.persistToStorage(); err != nil {
    // Handle persistence error
}

Data Verification

// Verify index health
if err := idx.HealthCheck(); err != nil {
    // Handle health check failure
}

Next Steps