API Reference¶

Welcome to the Quiver API reference! Here you'll find detailed documentation for all the functions, methods, and types in Quiver. Whether you're using the Go library directly or interacting with the HTTP API, this guide has you covered. Let's dive in! 📚

Go Library API¶

Core Types¶

Config¶

The Config struct controls the behavior of the Quiver index:

type Config struct {
    Dimension       int            // Vector dimension
    StoragePath     string         // Path to store index files
    Distance        DistanceMetric // Distance metric (Cosine or L2)
    MaxElements     uint64         // Maximum number of vectors (0 = unlimited)
    HNSWM           int            // HNSW hyperparameter M (default: 16)
    HNSWEfConstruct int            // HNSW hyperparameter efConstruction (default: 200)
    HNSWEfSearch    int            // HNSW hyperparameter efSearch (default: 100)
    BatchSize       int            // Number of vectors to batch before insertion (default: 1000)

    // Persistence configuration
    PersistInterval time.Duration  // How often to persist index to disk (default: 5m)

    // Backup configuration
    BackupInterval    time.Duration // How often to create backups (default: 1h)
    BackupPath        string        // Path to store backups (default: StoragePath/backups)
    BackupCompression bool          // Whether to compress backups (default: true)
    MaxBackups        int           // Maximum number of backups to keep (default: 5)

    // Security configuration
    EncryptionEnabled bool   // Whether to encrypt data at rest
    EncryptionKey     string // Key for encrypting data at rest (min 32 bytes)
}

DistanceMetric¶

The DistanceMetric type defines the similarity metrics:

type DistanceMetric int

const (
    Cosine DistanceMetric = iota // Cosine distance (1 - cosine similarity)
    L2                           // Euclidean (L2) distance
)

SearchResult¶

The SearchResult struct represents a single search result:

type SearchResult struct {
    ID       uint64                 // Vector ID
    Distance float32                // Distance from query vector
    Metadata map[string]interface{} // Associated metadata
}

Index Creation and Management¶

New¶

Creates a new Quiver index:

func New(config Config, logger *zap.Logger) (*Index, error)

Example:

logger, _ := zap.NewProduction()
config := quiver.Config{
    Dimension:   128,
    StoragePath: "/path/to/storage.db",
    Distance:    quiver.Cosine,
    HNSWM:       16,
    BatchSize:   1000,
}
idx, err := quiver.New(config, logger)
if err != nil {
    log.Fatalf("Failed to create index: %v", err)
}
defer idx.Close()

Load¶

Loads an existing Quiver index from disk:

func Load(config Config, logger *zap.Logger) (*Index, error)

Example:

logger, _ := zap.NewProduction()
config := quiver.Config{
    StoragePath: "/path/to/storage.db",
}
idx, err := quiver.Load(config, logger)
if err != nil {
    log.Fatalf("Failed to load index: %v", err)
}
defer idx.Close()

Close¶

Closes the index and releases resources:

func (idx *Index) Close() error

Example:

err := idx.Close()
if err != nil {
    log.Fatalf("Failed to close index: %v", err)
}

Vector Operations¶

Add¶

Adds a vector to the index:

func (idx *Index) Add(id uint64, vector []float32, meta map[string]interface{}) error

Example:

id := uint64(1)
vector := []float32{0.1, 0.2, 0.3, ...} // 128-dimensional vector
metadata := map[string]interface{}{
    "name": "example",
    "category": "test",
    "tags": []string{"tag1", "tag2"},
}
err := idx.Add(id, vector, metadata)
if err != nil {
    log.Fatalf("Failed to add vector: %v", err)
}

BatchAdd¶

Adds multiple vectors to the index in a single batch:

func (idx *Index) BatchAdd(ids []uint64, vectors [][]float32, metadata []map[string]interface{}) error

Example:

ids := []uint64{1, 2, 3}
vectors := [][]float32{
    {0.1, 0.2, 0.3, ...}, // Vector 1
    {0.4, 0.5, 0.6, ...}, // Vector 2
    {0.7, 0.8, 0.9, ...}, // Vector 3
}
metadata := []map[string]interface{}{
    {"name": "vector1", "category": "A"},
    {"name": "vector2", "category": "B"},
    {"name": "vector3", "category": "C"},
}
err := idx.BatchAdd(ids, vectors, metadata)
if err != nil {
    log.Fatalf("Failed to batch add vectors: %v", err)
}

AppendFromArrow¶

Adds vectors from an Arrow record:

func (idx *Index) AppendFromArrow(rec arrow.Record) error

Example:

// Create Arrow record with vectors and metadata
builder := array.NewRecordBuilder(memory.DefaultAllocator, quiver.NewVectorSchema(128))
// ... populate the builder ...
record := builder.NewRecord()

// Add all vectors from the Arrow record
err := idx.AppendFromArrow(record)
if err != nil {
    log.Fatalf("Failed to append from Arrow: %v", err)
}

Search Operations¶

Search¶

Searches for similar vectors:

func (idx *Index) Search(query []float32, k, page, pageSize int) ([]SearchResult, error)

Parameters:

query: Query vector
k: Number of nearest neighbors to find
page: Page number (1-indexed)
pageSize: Number of results per page

Example:

query := []float32{0.1, 0.2, 0.3, ...} // 128-dimensional vector
results, err := idx.Search(query, 10, 1, 10) // Find 10 nearest neighbors, first page, 10 results per page
if err != nil {
    log.Fatalf("Failed to search: %v", err)
}

for i, result := range results {
    fmt.Printf("Result %d: ID=%d, Distance=%f, Metadata=%v\n", i, result.ID, result.Distance, result.Metadata)
}

SearchWithFilter¶

Searches for similar vectors with metadata filtering:

func (idx *Index) SearchWithFilter(query []float32, k int, filter string) ([]SearchResult, error)

Parameters:

query: Query vector
k: Number of nearest neighbors to find
filter: SQL WHERE clause for filtering metadata

Example:

query := []float32{0.1, 0.2, 0.3, ...} // 128-dimensional vector
filter := "category = 'test' AND score > 0.5"
results, err := idx.SearchWithFilter(query, 10, filter) // Find 10 nearest neighbors matching the filter
if err != nil {
    log.Fatalf("Failed to search with filter: %v", err)
}

for i, result := range results {
    fmt.Printf("Result %d: ID=%d, Distance=%f, Metadata=%v\n", i, result.ID, result.Distance, result.Metadata)
}

SearchWithNegatives¶

Searches for similar vectors with negative examples:

func (idx *Index) SearchWithNegatives(positiveQuery []float32, negativeQueries [][]float32, k, page, pageSize int) ([]SearchResult, error)

Parameters:

positiveQuery: Query vector to find similar vectors to
negativeQueries: Query vectors to find dissimilar vectors from
k: Number of nearest neighbors to find
page: Page number (1-indexed)
pageSize: Number of results per page

Example:

positiveQuery := []float32{0.1, 0.2, 0.3, ...} // 128-dimensional vector
negativeQueries := [][]float32{
    {0.9, 0.8, 0.7, ...}, // Negative example 1
    {0.6, 0.5, 0.4, ...}, // Negative example 2
}
results, err := idx.SearchWithNegatives(positiveQuery, negativeQueries, 10, 1, 10)
if err != nil {
    log.Fatalf("Failed to search with negatives: %v", err)
}

for i, result := range results {
    fmt.Printf("Result %d: ID=%d, Distance=%f, Metadata=%v\n", i, result.ID, result.Distance, result.Metadata)
}

FacetedSearch¶

Searches for similar vectors with faceted filtering:

func (idx *Index) FacetedSearch(query []float32, k int, facets map[string]string) ([]SearchResult, error)

Parameters:

query: Query vector
k: Number of nearest neighbors to find
facets: Map of facet field names to values

Example:

query := []float32{0.1, 0.2, 0.3, ...} // 128-dimensional vector
facets := map[string]string{
    "category": "electronics",
    "brand": "apple",
}
results, err := idx.FacetedSearch(query, 10, facets)
if err != nil {
    log.Fatalf("Failed to perform faceted search: %v", err)
}

for i, result := range results {
    fmt.Printf("Result %d: ID=%d, Distance=%f, Metadata=%v\n", i, result.ID, result.Distance, result.Metadata)
}

MultiVectorSearch¶

Searches for multiple query vectors in a single call:

func (idx *Index) MultiVectorSearch(queries [][]float32, k int) ([][]SearchResult, error)

Parameters:

queries: Array of query vectors
k: Number of nearest neighbors to find for each query

Example:

queries := [][]float32{
    {0.1, 0.2, 0.3, ...}, // Query 1
    {0.4, 0.5, 0.6, ...}, // Query 2
}
results, err := idx.MultiVectorSearch(queries, 10)
if err != nil {
    log.Fatalf("Failed to perform multi-vector search: %v", err)
}

for i, queryResults := range results {
    fmt.Printf("Results for query %d:\n", i)
    for j, result := range queryResults {
        fmt.Printf("  Result %d: ID=%d, Distance=%f\n", j, result.ID, result.Distance)
    }
}

Metadata Operations¶

QueryMetadata¶

Queries metadata using SQL:

func (idx *Index) QueryMetadata(query string) ([]map[string]interface{}, error)

Example:

query := "SELECT * FROM metadata WHERE category = 'test' ORDER BY score DESC LIMIT 10"
results, err := idx.QueryMetadata(query)
if err != nil {
    log.Fatalf("Failed to query metadata: %v", err)
}

for i, result := range results {
    fmt.Printf("Result %d: %v\n", i, result)
}

Persistence and Backup¶

Save¶

Saves the index to disk:

func (idx *Index) Save(saveDir string) error

Example:

err := idx.Save("/path/to/save/directory")
if err != nil {
    log.Fatalf("Failed to save index: %v", err)
}

Backup¶

Creates a backup of the index:

func (idx *Index) Backup(path string, incremental bool, compress bool) error

Parameters:

path: Path to store the backup
incremental: Whether to create an incremental backup
compress: Whether to compress the backup

Example:

err := idx.Backup("/path/to/backup", false, true) // Full backup with compression
if err != nil {
    log.Fatalf("Failed to create backup: %v", err)
}

Restore¶

Restores the index from a backup:

func (idx *Index) Restore(backupPath string) error

Example:

err := idx.Restore("/path/to/backup.zip")
if err != nil {
    log.Fatalf("Failed to restore from backup: %v", err)
}

Utility Functions¶

HealthCheck¶

Checks the health of the index:

func (idx *Index) HealthCheck() error

Example:

err := idx.HealthCheck()
if err != nil {
    log.Fatalf("Health check failed: %v", err)
}

CollectMetrics¶

Collects performance metrics:

func (idx *Index) CollectMetrics() map[string]interface{}

Example:

metrics := idx.CollectMetrics()
fmt.Printf("Vector count: %d\n", metrics["vector_count"])
fmt.Printf("Memory usage: %.2f MB\n", metrics["memory_usage_mb"])

HTTP API¶

Quiver provides a RESTful HTTP API for remote access. All endpoints return JSON responses.

Authentication¶

All API endpoints (except /health) require authentication using an API key:

Authorization: Bearer your-api-key

Endpoints¶

Health Check¶

GET /health

Checks if the server is running.

Response:

{
  "status": "ok"
}

Add Vector¶

POST /vectors

Adds a vector to the index.

Request Body:

{
  "id": 1,
  "vector": [0.1, 0.2, 0.3, ...],
  "metadata": {
    "name": "example",
    "category": "test",
    "tags": ["tag1", "tag2"]
  }
}

Response:

{
  "success": true,
  "id": 1
}

Batch Add Vectors¶

POST /vectors/batch

Adds multiple vectors to the index.

Request Body:

{
  "vectors": [
    {
      "id": 1,
      "vector": [0.1, 0.2, 0.3, ...],
      "metadata": {
        "name": "vector1",
        "category": "A"
      }
    },
    {
      "id": 2,
      "vector": [0.4, 0.5, 0.6, ...],
      "metadata": {
        "name": "vector2",
        "category": "B"
      }
    }
  ]
}

Response:

{
  "success": true,
  "count": 2
}

Search¶

POST /search

Searches for similar vectors.

Request Body:

{
  "vector": [0.1, 0.2, 0.3, ...],
  "k": 10,
  "page": 1,
  "page_size": 10
}

Response:

{
  "results": [
    {
      "id": 1,
      "distance": 0.1234,
      "metadata": {
        "name": "example",
        "category": "test"
      }
    },
    // More results...
  ]
}

Hybrid Search¶

POST /search/hybrid

Searches for similar vectors with metadata filtering.

Request Body:

{
  "vector": [0.1, 0.2, 0.3, ...],
  "k": 10,
  "filter": "category = 'test' AND score > 0.5"
}

Response:

{
  "results": [
    {
      "id": 1,
      "distance": 0.1234,
      "metadata": {
        "name": "example",
        "category": "test",
        "score": 0.75
      }
    },
    // More results...
  ]
}

Search with Negatives¶

POST /search/negatives

Searches for similar vectors with negative examples.

Request Body:

{
  "positive_vector": [0.1, 0.2, 0.3, ...],
  "negative_vectors": [
    [0.9, 0.8, 0.7, ...],
    [0.6, 0.5, 0.4, ...]
  ],
  "k": 10,
  "page": 1,
  "page_size": 10
}

Response:

{
  "results": [
    {
      "id": 1,
      "distance": 0.1234,
      "metadata": {
        "name": "example",
        "category": "test"
      }
    },
    // More results...
  ]
}

Faceted Search¶

POST /search/faceted

Searches for similar vectors with faceted filtering.

Request Body:

{
  "vector": [0.1, 0.2, 0.3, ...],
  "k": 10,
  "facets": {
    "category": "electronics",
    "brand": "apple"
  }
}

Response:

{
  "results": [
    {
      "id": 1,
      "distance": 0.1234,
      "metadata": {
        "name": "iPhone",
        "category": "electronics",
        "brand": "apple"
      }
    },
    // More results...
  ]
}

Query Metadata¶

POST /metadata/query

Queries metadata using SQL.

Request Body:

{
  "query": "SELECT * FROM metadata WHERE category = 'test' ORDER BY score DESC LIMIT 10"
}

Response:

{
  "results": [
    {
      "id": 1,
      "name": "example",
      "category": "test",
      "score": 0.95
    },
    // More results...
  ]
}

Backup¶

POST /admin/backup

Creates a backup of the index.

Request Body:

{
  "path": "/path/to/backup",
  "incremental": false,
  "compress": true
}

Response:

{
  "success": true,
  "path": "/path/to/backup.zip",
  "size_bytes": 1024000,
  "timestamp": "2023-06-15T14:30:00Z"
}

Restore¶

POST /admin/restore

Restores the index from a backup.

Request Body:

{
  "path": "/path/to/backup.zip"
}

Response:

{
  "success": true,
  "vector_count": 10000,
  "timestamp": "2023-06-15T14:35:00Z"
}

Get Metrics¶

GET /admin/metrics

Gets performance metrics.

Response:

{
  "vector_count": 10000,
  "memory_usage_mb": 256.5,
  "uptime_seconds": 3600,
  "search_latency_ms": 5.2,
  "add_latency_ms": 2.1
}

Python Client API¶

Quiver provides a Python client for easy integration with Python applications.

Installation¶

pip install quiver-db

Client Initialization¶

from quiver import Client

# Connect to a local Quiver server
client = Client("http://localhost:8080", api_key="your-api-key")

# Connect to a remote Quiver server
client = Client("https://quiver.example.com", api_key="your-api-key")

Vector Operations¶

Add Vector¶

# Add a single vector
client.add(
    id=1,
    vector=[0.1, 0.2, 0.3, ...],  # 128-dimensional vector
    metadata={
        "name": "example",
        "category": "test",
        "tags": ["tag1", "tag2"]
    }
)

Batch Add Vectors¶

# Add multiple vectors
vectors = [
    {
        "id": 1,
        "vector": [0.1, 0.2, 0.3, ...],
        "metadata": {"name": "vector1", "category": "A"}
    },
    {
        "id": 2,
        "vector": [0.4, 0.5, 0.6, ...],
        "metadata": {"name": "vector2", "category": "B"}
    }
]
client.batch_add(vectors)

Search Operations¶

Search¶

# Search for similar vectors
results = client.search(
    vector=[0.1, 0.2, 0.3, ...],
    k=10,
    page=1,
    page_size=10
)

for result in results:
    print(f"ID: {result['id']}, Distance: {result['distance']}, Metadata: {result['metadata']}")

Hybrid Search¶

# Search with metadata filtering
results = client.hybrid_search(
    vector=[0.1, 0.2, 0.3, ...],
    k=10,
    filter="category = 'test' AND score > 0.5"
)

for result in results:
    print(f"ID: {result['id']}, Distance: {result['distance']}, Metadata: {result['metadata']}")

Search with Negatives¶

# Search with negative examples
results = client.search_with_negatives(
    positive_vector=[0.1, 0.2, 0.3, ...],
    negative_vectors=[
        [0.9, 0.8, 0.7, ...],
        [0.6, 0.5, 0.4, ...]
    ],
    k=10,
    page=1,
    page_size=10
)

for result in results:
    print(f"ID: {result['id']}, Distance: {result['distance']}, Metadata: {result['metadata']}")

Faceted Search¶

# Search with faceted filtering
results = client.faceted_search(
    vector=[0.1, 0.2, 0.3, ...],
    k=10,
    facets={
        "category": "electronics",
        "brand": "apple"
    }
)

for result in results:
    print(f"ID: {result['id']}, Distance: {result['distance']}, Metadata: {result['metadata']}")

Metadata Operations¶

Query Metadata¶

# Query metadata using SQL
results = client.query_metadata(
    "SELECT * FROM metadata WHERE category = 'test' ORDER BY score DESC LIMIT 10"
)

for result in results:
    print(result)

Administration¶

Backup¶

# Create a backup
backup_info = client.backup(
    path="/path/to/backup",
    incremental=False,
    compress=True
)

print(f"Backup created at {backup_info['path']}, size: {backup_info['size_bytes']} bytes")

Restore¶

# Restore from a backup
restore_info = client.restore("/path/to/backup.zip")

print(f"Restored {restore_info['vector_count']} vectors")

Get Metrics¶

# Get performance metrics
metrics = client.metrics()

print(f"Vector count: {metrics['vector_count']}")
print(f"Memory usage: {metrics['memory_usage_mb']} MB")
print(f"Search latency: {metrics['search_latency_ms']} ms")

Error Handling¶

All API functions return errors that should be handled appropriately:

Go Library¶

result, err := idx.Search(query, 10, 1, 10)
if err != nil {
    if errors.Is(err, quiver.ErrDimensionMismatch) {
        // Handle dimension mismatch
    } else if errors.Is(err, quiver.ErrInvalidID) {
        // Handle invalid ID
    } else {
        // Handle other errors
    }
}

HTTP API¶

HTTP API errors are returned with appropriate status codes and error messages:

{
  "error": {
    "code": "dimension_mismatch",
    "message": "Vector dimension mismatch: expected 128, got 64"
  }
}

Python Client¶

try:
    results = client.search(vector, k=10)
except quiver.DimensionMismatchError:
    # Handle dimension mismatch
except quiver.InvalidIDError:
    # Handle invalid ID
except quiver.QuiverError as e:
    # Handle other errors
    print(f"Error: {e}")

Next Steps¶

Now that you're familiar with the Quiver API, check out:

Quick Start - Get started with Quiver
Examples - See practical examples of using Quiver
Performance Tuning - Optimize Quiver for your needs

Quiver Class¶

Constructor¶

Quiver(
    dimension: int,
    storage_path: str = ":memory:",
    distance: str = "cosine",
    max_elements: int = 1000000,
    hnsw_m: int = 16,
    hnsw_ef_construct: int = 200,
    hnsw_ef_search: int = 100,
    batch_size: int = 1000,
    persist_interval: str = "5m",
    backup_interval: str = "1h",
    backup_path: str = None,
    backup_compression: bool = True,
    max_backups: int = 5,
    encryption_enabled: bool = False,
    encryption_key: str = "",
    enable_dim_reduction: bool = False,
    dim_reduction_method: str = "PCA",
    dim_reduction_target: int = None,
    dim_reduction_adaptive: bool = False,
    dim_reduction_min_variance: float = 0.95,
)

Creates a new Quiver instance.

Parameters:

dimension (int): Dimension of the vectors.
storage_path (str, optional): Path to store the index files. Defaults to ":memory:".
distance (str, optional): Distance metric to use. Either "cosine" or "l2". Defaults to "cosine".
max_elements (int, optional): Maximum number of vectors to store. Defaults to 1000000.
hnsw_m (int, optional): HNSW hyperparameter M. Defaults to 16.
hnsw_ef_construct (int, optional): HNSW hyperparameter efConstruction. Defaults to 200.
hnsw_ef_search (int, optional): HNSW hyperparameter ef used during queries. Defaults to 100.
batch_size (int, optional): Number of vectors to batch before insertion. Defaults to 1000.
persist_interval (str, optional): How often to persist index to disk. Defaults to "5m".
backup_interval (str, optional): How often to create backups. Defaults to "1h".
backup_path (str, optional): Path to store backups. Defaults to None.
backup_compression (bool, optional): Whether to compress backups. Defaults to True.
max_backups (int, optional): Maximum number of backups to keep. Defaults to 5.
encryption_enabled (bool, optional): Whether to encrypt data at rest. Defaults to False.
encryption_key (str, optional): Key for encrypting data at rest. Defaults to "".
enable_dim_reduction (bool, optional): Whether to enable dimensionality reduction. Defaults to False.
dim_reduction_method (str, optional): Method to use for dimensionality reduction. Either "PCA", "TSNE", or "UMAP". Defaults to "PCA".
dim_reduction_target (int, optional): Target dimension for reduction. Defaults to dimension/2.
dim_reduction_adaptive (bool, optional): Whether to use adaptive dimensionality reduction. Defaults to False.
dim_reduction_min_variance (float, optional): Minimum variance to explain (0.0-1.0) for adaptive reduction. Defaults to 0.95.

Returns:

A new Quiver instance.

Example:

from quiver import Quiver

# Create a new Quiver instance with dimensionality reduction
index = Quiver(
    dimension=1536,
    storage_path="path/to/index",
    distance="cosine",
    enable_dim_reduction=True,
    dim_reduction_method="PCA",
    dim_reduction_target=768,
)

Methods¶

add¶

add(
    id: int,
    vector: List[float],
    metadata: Dict[str, Any] = None
) -> None

Adds a vector to the index.

Parameters:

id (int): Unique identifier for the vector.
vector (List[float]): Vector to add. If dimensionality reduction is enabled, this will be automatically reduced.
metadata (Dict[str, Any], optional): Metadata to associate with the vector. Defaults to None.

Returns:

None

Example:

# Add a vector to the index
index.add(
    id=1,
    vector=[0.1, 0.2, 0.3, ...],
    metadata={"text": "Example document", "category": "news"}
)

reduce_vectors¶

reduce_vectors(
    vectors: List[List[float]]
) -> List[List[float]]

Reduces the dimensionality of vectors without adding them to the index.

Parameters:

vectors (List[List[float]]): Vectors to reduce.

Returns:

List[List[float]]: Reduced vectors.

Example:

# Reduce vectors without adding them to the index
original_vectors = [
    [0.1, 0.2, 0.3, ...],  # 1536-dimensional vectors
    [0.4, 0.5, 0.6, ...],
]
reduced_vectors = index.reduce_vectors(original_vectors)
# reduced_vectors will be lower-dimensional

Note:

This method is only available if dimensionality reduction is enabled.
The reduction uses the same method and parameters as configured for the index.