Skip to content

Configuration

Quiver is highly configurable to meet your specific needs. Let's dive into all the knobs and dials you can tweak to make Quiver purr like a well-tuned sports car! 🏎️

Configuration Options

Here's the full Config struct with all available options:

type Config struct {
    // Core settings
    Dimension       int            // Vector dimension
    StoragePath     string         // Base directory for storing index files
    Distance        DistanceMetric // Cosine or L2
    MaxElements     uint64         // Maximum number of vectors

    // HNSW algorithm parameters
    HNSWM           int            // Number of connections per element
    HNSWEfConstruct int            // Construction quality parameter
    HNSWEfSearch    int            // Search quality parameter

    // Performance settings
    BatchSize       int            // Number of vectors to batch before insertion

    // Persistence configuration
    PersistInterval time.Duration  // How often to persist index to disk

    // Backup configuration
    BackupInterval    time.Duration // How often to create backups
    BackupPath        string        // Path to store backups
    BackupCompression bool          // Whether to compress backups
    MaxBackups        int           // Maximum number of backups to keep

    // Security configuration
    EncryptionEnabled bool          // Whether to encrypt data at rest
    EncryptionKey     string        // Key for encrypting data at rest
}

Core Settings

Dimension

The dimension of your vectors. This must match the output dimension of your embedding model.

Dimension: 768, // For BERT embeddings

Immutable Setting

Once you create an index with a specific dimension, you cannot change it later.

StoragePath

Where Quiver will store its data. This can be:

  • A path to a directory (e.g., "./data")
  • A path to a specific file (e.g., "./data/quiver.db")
  • :memory: for an in-memory database (no persistence)
StoragePath: "./data/my_vectors.db",

Distance

The distance metric to use for similarity calculations:

Distance: quiver.Cosine, // Options: quiver.Cosine, quiver.L2
  • Cosine: Measures the cosine of the angle between vectors (good for text embeddings)
  • L2: Euclidean distance (good for image embeddings)

MaxElements

The maximum number of vectors your index will hold:

MaxElements: 1000000, // 1 million vectors

Setting this appropriately helps Quiver allocate memory efficiently.

HNSW Algorithm Parameters

These parameters control the HNSW graph that powers Quiver's lightning-fast searches.

HNSWM

The number of connections per element in the graph:

HNSWM: 16, // Default is 16

Higher values create more connections, which can improve search quality but increase memory usage and construction time.

Rule of Thumb

Values between 12-64 work well for most applications. Start with 16 and adjust if needed.

HNSWEfConstruct

Controls the quality of graph construction:

HNSWEfConstruct: 200, // Default is 200

Higher values create a better graph but slow down construction.

HNSWEfSearch

Controls the quality of search:

HNSWEfSearch: 100, // Default is 100

Higher values improve search quality but slow down search speed.

Balancing Act

For most applications, values between 50-200 provide a good balance. If you need more precision, increase this value.

Performance Settings

BatchSize

The number of vectors to accumulate before adding them to the index:

BatchSize: 1000, // Default is 1000

Larger batch sizes can improve throughput for bulk loading but increase memory usage.

Persistence Configuration

PersistInterval

How often to automatically save the index to disk:

PersistInterval: 5 * time.Minute, // Default is 5 minutes

Backup Configuration

BackupInterval

How often to create backups:

BackupInterval: 1 * time.Hour, // Default is 1 hour

BackupPath

Where to store backups:

BackupPath: "./backups", // Default is StoragePath/backups

BackupCompression

Whether to compress backups to save space:

BackupCompression: true, // Default is true

MaxBackups

Maximum number of backups to keep:

MaxBackups: 5, // Default is 5

Security Configuration

EncryptionEnabled

Whether to encrypt data at rest:

EncryptionEnabled: true, // Default is false

EncryptionKey

The key used for encryption:

EncryptionKey: "your-secret-key-at-least-32-bytes-long",

Key Length

The encryption key must be at least 32 bytes long. If it's longer, it will be truncated.

Configuration Examples

Minimal Configuration

config := quiver.Config{
    Dimension:   128,
    StoragePath: "./data",
}

Production Configuration

config := quiver.Config{
    // Core settings
    Dimension:       768,
    StoragePath:     "/var/lib/quiver/production.db",
    Distance:        quiver.Cosine,
    MaxElements:     10000000,

    // HNSW parameters
    HNSWM:           32,
    HNSWEfConstruct: 200,
    HNSWEfSearch:    100,

    // Performance
    BatchSize:       5000,

    // Persistence
    PersistInterval: 10 * time.Minute,

    // Backups
    BackupInterval:    1 * time.Hour,
    BackupPath:        "/var/backups/quiver",
    BackupCompression: true,
    MaxBackups:        24, // Keep a day's worth

    // Security
    EncryptionEnabled: true,
    EncryptionKey:     os.Getenv("QUIVER_ENCRYPTION_KEY"),
}

In-Memory Configuration (for Testing)

config := quiver.Config{
    Dimension:   128,
    StoragePath: ":memory:",
    BatchSize:   100,
}

Environment Variables

Quiver doesn't directly use environment variables, but you can use them in your configuration:

config := quiver.Config{
    Dimension:       128,
    StoragePath:     os.Getenv("QUIVER_STORAGE_PATH"),
    EncryptionKey:   os.Getenv("QUIVER_ENCRYPTION_KEY"),
    MaxElements:     parseUint64(os.Getenv("QUIVER_MAX_ELEMENTS")),
}

Next Steps

Now that you've configured Quiver to your liking, check out: