Skip to content

Configuration

Time to fine-tune your vector database! This guide will walk you through all the configuration options available in Quiver, helping you optimize for your specific use case. Let's make those vectors sing! 🎵

Configuration Overview

Quiver offers a wide range of configuration options to customize its behavior. These options control everything from basic settings like vector dimensions to advanced features like encryption and backup strategies.

Configuration Options

Quiver provides a variety of configuration options to customize the behavior of the vector database.

Basic Configuration

Option Description Default
dimension Dimension of the vectors Required
storage_path Path to store the index files Required
distance Distance metric to use (cosine or l2) cosine
max_elements Maximum number of vectors to store 1000000
hnsw_m HNSW hyperparameter M (number of connections per node) 16
hnsw_ef_construct HNSW hyperparameter efConstruction (size of the dynamic candidate list during construction) 200
hnsw_ef_search HNSW hyperparameter ef (size of the dynamic candidate list during search) 100
batch_size Number of vectors to batch before insertion 1000

Persistence Configuration

Option Description Default
persist_interval How often to persist index to disk 5m

Backup Configuration

Option Description Default
backup_interval How often to create backups 1h
backup_path Path to store backups {storage_path}/backups
backup_compression Whether to compress backups true
max_backups Maximum number of backups to keep 5

Security Configuration

Option Description Default
encryption_enabled Whether to encrypt data at rest false
encryption_key Key for encrypting data at rest (min 32 bytes) ""

Dimensionality Reduction Configuration

Option Description Default
enable_dim_reduction Whether to enable dimensionality reduction false
dim_reduction_method Method to use for dimensionality reduction (PCA, TSNE, UMAP) PCA
dim_reduction_target Target dimension for reduction dimension / 2
dim_reduction_adaptive Whether to use adaptive dimensionality reduction false
dim_reduction_min_variance Minimum variance to explain (0.0-1.0) for adaptive reduction 0.95

Configuration Examples

Minimal Configuration

The simplest configuration with just the required parameters:

config := quiver.Config{
    Dimension: 128,
    StoragePath: "/path/to/quiver.db",
}

In-Memory Configuration

For maximum performance with no persistence:

config := quiver.Config{
    Dimension: 128,
    StoragePath: ":memory:",
    HNSWM: 16,
    HNSWEfSearch: 100,
}

High-Throughput Configuration

Optimized for adding vectors quickly:

config := quiver.Config{
    Dimension: 128,
    StoragePath: "/path/to/quiver.db",
    BatchSize: 5000,
    PersistInterval: 10 * time.Minute,
}

High-Accuracy Configuration

Optimized for search accuracy:

config := quiver.Config{
    Dimension: 128,
    StoragePath: "/path/to/quiver.db",
    HNSWM: 32,
    HNSWEfConstruct: 300,
    HNSWEfSearch: 200,
}

Production-Ready Configuration

A balanced configuration for production use:

config := quiver.Config{
    Dimension: 128,
    StoragePath: "/path/to/quiver.db",
    Distance: quiver.Cosine,
    MaxElements: 10000000,
    HNSWM: 16,
    HNSWEfConstruct: 200,
    HNSWEfSearch: 100,
    BatchSize: 1000,
    PersistInterval: 5 * time.Minute,
    BackupInterval: 1 * time.Hour,
    BackupPath: "/path/to/backups",
    BackupCompression: true,
    MaxBackups: 7,
    EncryptionEnabled: true,
    EncryptionKey: "your-32-byte-encryption-key-here",
}

Environment Variables

When using the HTTP server, you can configure Quiver using environment variables:

Environment Variable Corresponding Config Example
QUIVER_DIMENSION Dimension QUIVER_DIMENSION=128
QUIVER_STORAGE_PATH StoragePath QUIVER_STORAGE_PATH=/data/quiver.db
QUIVER_DISTANCE Distance QUIVER_DISTANCE=cosine
QUIVER_MAX_ELEMENTS MaxElements QUIVER_MAX_ELEMENTS=1000000
QUIVER_HNSW_M HNSWM QUIVER_HNSW_M=16
QUIVER_HNSW_EF_CONSTRUCT HNSWEfConstruct QUIVER_HNSW_EF_CONSTRUCT=200
QUIVER_HNSW_EF_SEARCH HNSWEfSearch QUIVER_HNSW_EF_SEARCH=100
QUIVER_BATCH_SIZE BatchSize QUIVER_BATCH_SIZE=1000
QUIVER_PERSIST_INTERVAL PersistInterval QUIVER_PERSIST_INTERVAL=5m
QUIVER_BACKUP_INTERVAL BackupInterval QUIVER_BACKUP_INTERVAL=1h
QUIVER_BACKUP_PATH BackupPath QUIVER_BACKUP_PATH=/backups
QUIVER_BACKUP_COMPRESSION BackupCompression QUIVER_BACKUP_COMPRESSION=true
QUIVER_MAX_BACKUPS MaxBackups QUIVER_MAX_BACKUPS=5
QUIVER_ENCRYPTION_ENABLED EncryptionEnabled QUIVER_ENCRYPTION_ENABLED=true
QUIVER_ENCRYPTION_KEY EncryptionKey QUIVER_ENCRYPTION_KEY=your-key
QUIVER_API_KEY Server API key QUIVER_API_KEY=your-api-key
QUIVER_PORT Server port QUIVER_PORT=8080

Docker Configuration

When using the Docker image, you can configure Quiver using environment variables:

docker run -p 8080:8080 \
  -v /path/to/data:/data \
  -e QUIVER_DIMENSION=128 \
  -e QUIVER_STORAGE_PATH=/data/quiver.db \
  -e QUIVER_API_KEY=your-secret-key \
  weaviate/quiver:latest

Configuration File

You can also load configuration from a JSON file:

{
  "dimension": 128,
  "storage_path": "/path/to/quiver.db",
  "distance": "cosine",
  "max_elements": 1000000,
  "hnsw_m": 16,
  "hnsw_ef_construct": 200,
  "hnsw_ef_search": 100,
  "batch_size": 1000,
  "persist_interval": "5m",
  "backup_interval": "1h",
  "backup_path": "/path/to/backups",
  "backup_compression": true,
  "max_backups": 5,
  "encryption_enabled": true,
  "encryption_key": "your-32-byte-encryption-key-here",
  "enable_dim_reduction": true,
  "dim_reduction_method": "PCA",
  "dim_reduction_target": 64,
  "dim_reduction_adaptive": true,
  "dim_reduction_min_variance": 0.95
}

Load the configuration file in Go:

configBytes, err := os.ReadFile("config.json")
if err != nil {
    log.Fatalf("Failed to read config file: %v", err)
}

var config quiver.Config
if err := json.Unmarshal(configBytes, &config); err != nil {
    log.Fatalf("Failed to parse config: %v", err)
}

idx, err := quiver.New(config, logger)
if err != nil {
    log.Fatalf("Failed to create index: %v", err)
}

Python Client Configuration

When using the Python client, you can configure the connection:

from quiver import Client

# Connect to a local Quiver server
client = Client(
    url="http://localhost:8080",
    api_key="your-api-key",
    timeout=30,  # Request timeout in seconds
    retries=3    # Number of retries for failed requests
)

Next Steps

Now that you've configured Quiver, check out: