Go Library¶

Quiver is primarily a Go library that you can embed directly in your applications. Let's dive into how to use Quiver as a library to supercharge your Go applications with vector search! 📚

Installation¶

First, add Quiver to your Go project:

go get github.com/TFMV/quiver

Basic Usage¶

Initializing Quiver¶

The first step is to create a new Quiver index:

package main

import (
    "log"

    "github.com/TFMV/quiver"
    "go.uber.org/zap"
)

func main() {
    // Create a logger
    logger, _ := zap.NewDevelopment()

    // Configure Quiver
    config := quiver.Config{
        Dimension:   128,            // Vector dimension
        StoragePath: "./data",       // Where to store the data
        Distance:    quiver.Cosine,  // Distance metric
    }

    // Create the index
    idx, err := quiver.New(config, logger)
    if err != nil {
        log.Fatalf("Failed to create index: %v", err)
    }
    defer idx.Close() // Don't forget to close when done!

    // Now you can use idx to add and search vectors
}

Adding Vectors¶

Add vectors to the index:

// Add a single vector
err := idx.Add(1, []float32{0.1, 0.2, 0.3, /* ... */}, map[string]interface{}{
    "category": "example",
    "name": "first vector",
})
if err != nil {
    log.Fatalf("Failed to add vector: %v", err)
}

// Add multiple vectors
for i := 2; i <= 100; i++ {
    vector := generateRandomVector(128) // Your function to generate vectors
    metadata := map[string]interface{}{
        "category": "example",
        "name": fmt.Sprintf("vector %d", i),
        "index": i,
    }

    if err := idx.Add(uint64(i), vector, metadata); err != nil {
        log.Printf("Failed to add vector %d: %v", i, err)
    }
}

Searching¶

Search for similar vectors:

// Create a query vector
queryVector := []float32{0.1, 0.2, 0.3, /* ... */}

// Search for the 10 most similar vectors
results, err := idx.Search(queryVector, 10, 1, 10)
if err != nil {
    log.Fatalf("Search failed: %v", err)
}

// Print the results
for i, result := range results {
    fmt.Printf("%d. ID: %d, Distance: %.4f, Name: %s\n", 
        i+1, result.ID, result.Distance, result.Metadata["name"])
}

Hybrid Search¶

Combine vector search with metadata filtering:

// Search with a metadata filter
results, err := idx.SearchWithFilter(queryVector, 10, 
    "category = 'example' AND index > 50")
if err != nil {
    log.Fatalf("Hybrid search failed: %v", err)
}

Advanced Usage¶

Working with Arrow¶

Quiver integrates with Apache Arrow for efficient data transfer:

import (
    "github.com/TFMV/quiver"
    "github.com/apache/arrow-go/v18/arrow"
    "github.com/apache/arrow/go/v18/arrow/array"
    "github.com/apache/arrow/go/v18/arrow/memory"
)

func addVectorsWithArrow(idx *quiver.Index, vectors [][]float32, metadata []map[string]interface{}) error {
    // Create a schema for the vectors
    schema := quiver.NewVectorSchema(len(vectors[0]))

    // Create a record builder
    builder := array.NewRecordBuilder(memory.DefaultAllocator, schema)
    defer builder.Release()

    // Get the builders for each column
    idBuilder := builder.Field(0).(*array.Uint64Builder)
    vectorBuilder := builder.Field(1).(*array.Float32Builder)
    metadataBuilder := builder.Field(2).(*array.MapBuilder)

    // Add data to the builders
    for i, vec := range vectors {
        // Add ID
        idBuilder.Append(uint64(i + 1))

        // Add vector
        for _, v := range vec {
            vectorBuilder.Append(v)
        }

        // Add metadata
        // ... (code to add metadata to the MapBuilder)
    }

    // Create the record
    record := builder.NewRecord()
    defer record.Release()

    // Add the record to the index
    return idx.AppendFromArrow(record)
}

Custom Distance Functions¶

Quiver supports custom distance functions:

// Define a custom distance function
func customDistance(a, b []float32) float32 {
    // Your custom distance calculation
    var sum float32
    for i := range a {
        diff := a[i] - b[i]
        sum += diff * diff
    }
    return float32(math.Sqrt(float64(sum)))
}

// Use it in your configuration
config := quiver.Config{
    Dimension:       128,
    StoragePath:     "./data",
    Distance:        quiver.CustomDistance(customDistance),
}

Persistence and Recovery¶

Save and load the index:

// Save the index to a specific location
err := idx.Save("./backup/my_index")
if err != nil {
    log.Fatalf("Failed to save index: %v", err)
}

// Load the index from disk
idx, err = quiver.Load(config, logger)
if err != nil {
    log.Fatalf("Failed to load index: %v", err)
}

Backup and Restore¶

Create and restore backups:

// Create a backup
err := idx.Backup("./backups/backup1", false, true)
if err != nil {
    log.Fatalf("Backup failed: %v", err)
}

// Restore from a backup
err = idx.Restore("./backups/backup1")
if err != nil {
    log.Fatalf("Restore failed: %v", err)
}

Integration Patterns¶

Embedding in a Web Service¶

Integrate Quiver with your web service:

package main

import (
    "encoding/json"
    "net/http"

    "github.com/TFMV/quiver"
    "go.uber.org/zap"
)

type SearchRequest struct {
    Vector []float32 `json:"vector"`
    K      int       `json:"k"`
}

type SearchResponse struct {
    Results []quiver.SearchResult `json:"results"`
}

func main() {
    // Create a logger
    logger, _ := zap.NewDevelopment()

    // Initialize Quiver
    idx, _ := quiver.New(quiver.Config{
        Dimension:   128,
        StoragePath: "./data",
    }, logger)
    defer idx.Close()

    // Define a search handler
    http.HandleFunc("/search", func(w http.ResponseWriter, r *http.Request) {
        var req SearchRequest
        if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
            http.Error(w, err.Error(), http.StatusBadRequest)
            return
        }

        results, err := idx.Search(req.Vector, req.K, 1, req.K)
        if err != nil {
            http.Error(w, err.Error(), http.StatusInternalServerError)
            return
        }

        json.NewEncoder(w).Encode(SearchResponse{Results: results})
    })

    // Start the server
    http.ListenAndServe(":8080", nil)
}

Background Processing¶

Use Quiver in a background worker:

package main

import (
    "time"

    "github.com/TFMV/quiver"
    "go.uber.org/zap"
)

func main() {
    // Create a logger
    logger, _ := zap.NewDevelopment()

    // Initialize Quiver
    idx, _ := quiver.New(quiver.Config{
        Dimension:   128,
        StoragePath: "./data",
        BatchSize:   10000, // Large batch size for bulk processing
    }, logger)
    defer idx.Close()

    // Process vectors in the background
    go func() {
        for {
            // Get vectors from somewhere (e.g., a queue)
            vectors, metadata := getVectorsFromQueue()

            // Add them to the index
            for i, vec := range vectors {
                idx.Add(uint64(i), vec, metadata[i])
            }

            // Sleep for a bit
            time.Sleep(1 * time.Second)
        }
    }()

    // Keep the main thread alive
    select {}
}

Best Practices¶

Memory Management¶

Close the index when done to release resources
Use appropriate batch sizes for your memory constraints
Consider using Arrow for bulk operations to reduce memory overhead

Concurrency¶

Quiver is thread-safe, but be mindful of:

Concurrent reads are fine
Concurrent writes are serialized
Heavy write loads might benefit from batching

Error Handling¶

Always check errors:

// Add with proper error handling
if err := idx.Add(id, vector, metadata); err != nil {
    // Log the error
    logger.Error("Failed to add vector", zap.Uint64("id", id), zap.Error(err))

    // Maybe retry
    if isRetryable(err) {
        // Retry logic
    }

    // Or return the error
    return err
}

Logging¶

Quiver uses zap for logging. Configure it appropriately:

// Development logger (verbose)
logger, _ := zap.NewDevelopment()

// Production logger (less verbose)
logger, _ := zap.NewProduction()

// Custom logger
logger, _ := zap.Config{
    Level:       zap.NewAtomicLevelAt(zap.InfoLevel),
    Development: false,
    Encoding:    "json",
    // ... other options
}.Build()

Next Steps¶

Now that you've learned how to use Quiver as a Go library, check out:

HTTP API - Use Quiver as a service
Examples - See more usage examples
Performance Tuning - Optimize Quiver for your needs