Skip to content

Extensions Overview

HNSW is designed to be extensible, allowing you to enhance its core functionality through modular extensions. This page provides an overview of the available extensions and how to use them.

Available Extensions

HNSW comes with several built-in extensions that provide additional functionality:

Metadata Extension

The Metadata Extension allows you to store and retrieve JSON metadata alongside vectors in the HNSW graph.

Key features:

  • Type-safe implementation using Go generics
  • Memory-efficient storage of metadata
  • Support for batch operations
  • Seamless integration with the core HNSW API

Learn more about the Metadata Extension

The Faceted Search extension enables filtering of search results based on facets (attributes) of the vectors.

Key features:

  • Support for multiple facet types (string, numeric, boolean, array)
  • Efficient filtering of search results
  • Combining multiple filters with AND/OR logic
  • Seamless integration with the core HNSW API

Learn more about Faceted Search

Parquet Extension

The Parquet Extension provides a persistent implementation of the HNSW graph algorithm using Apache Parquet for storage.

Key features:

  • Persistent storage of HNSW graph using Apache Parquet files
  • Efficient vector storage and retrieval
  • Memory-mapped file access for improved performance
  • Incremental updates with automatic compaction
  • Support for batch operations

Learn more about the Parquet Extension

Hybrid Extension

The Hybrid Extension provides a multi-strategy approach to approximate nearest neighbor search, combining the strengths of different algorithms.

Key features:

  • Multi-strategy approach combining HNSW with complementary techniques
  • Adaptive selection of optimal search strategy
  • Improved performance for diverse workloads
  • Better memory utilization and scalability
  • Real-time performance monitoring

Learn more about the Hybrid Extension

Using Extensions

To use an extension, you need to import it alongside the core HNSW package:

import (
    "github.com/TFMV/hnsw"
    "github.com/TFMV/hnsw/hnsw-extensions/meta"
    "github.com/TFMV/hnsw/hnsw-extensions/facets"
    "github.com/TFMV/hnsw/hnsw-extensions/parquet"
    "github.com/TFMV/hnsw/hnsw-extensions/hybrid"
)

Example: Using the Metadata Extension

// Create a new graph with metadata
graph := meta.NewMetaGraph[int, ProductMetadata]()

// Add a vector with metadata
err := graph.Add(hnsw.Node[int]{
    Key:   1,
    Value: []float32{0.1, 0.2, 0.3},
}, ProductMetadata{
    Name:  "Product 1",
    Price: 19.99,
    Tags:  []string{"electronics", "gadgets"},
})

// Search and retrieve metadata
results, err := graph.SearchWithMetadata(query, 5)
for _, result := range results {
    fmt.Printf("Product: %s, Price: %.2f\n", result.Metadata.Name, result.Metadata.Price)
}

Example: Using the Faceted Search Extension

// Create a new graph with facets
graph := facets.NewFacetedGraph[int]()

// Add a vector with facets
err := graph.Add(hnsw.Node[int]{
    Key:   1,
    Value: []float32{0.1, 0.2, 0.3},
}, map[string]interface{}{
    "category": "electronics",
    "price":    19.99,
    "inStock":  true,
})

// Search with a filter
filter := facets.And(
    facets.Eq("category", "electronics"),
    facets.Range("price", 10.0, 50.0),
    facets.Eq("inStock", true),
)
results, err := graph.SearchWithFilter(query, 5, filter)

Example: Using the Parquet Extension

// Create a configuration
config := parquet.DefaultParquetGraphConfig()
config.Storage.Directory = "/path/to/storage"
config.M = 16
config.EfSearch = 100
config.Distance = hnsw.CosineDistance

// Create a new graph
graph, err := parquet.NewParquetGraph[int](config)
if err != nil {
    // Handle error
}
defer graph.Close()

// Add vectors
nodes := []hnsw.Node[int]{
    {Key: 1, Value: []float32{0.1, 0.2, 0.3}},
    {Key: 2, Value: []float32{0.4, 0.5, 0.6}},
}
err = graph.Add(nodes...)

// Search
query := []float32{0.2, 0.3, 0.4}
results, err := graph.Search(query, 5)

Example: Using the Hybrid Extension

// Create a configuration
config := hybrid.DefaultHybridConfig()
config.ExactThreshold = 1000  // Use exact search for datasets smaller than 1000 vectors
config.Distance = hnsw.CosineDistance

// Create a new hybrid index
index, err := hybrid.NewHybridIndex[int](config)
if err != nil {
    // Handle error
}
defer index.Close()

// Add vectors
for i := 0; i < 1000; i++ {
    err := index.Add(i, generateVector(128))
    if err != nil {
        // Handle error
    }
}

// Search
query := []float32{0.2, 0.3, 0.4}
results, err := index.Search(query, 5)

Combining Extensions

You can combine multiple extensions to create a more powerful search system. For example, you can combine the Metadata Extension with the Faceted Search extension:

// Create a graph with both metadata and facets
graph := meta.NewMetaGraph[int, ProductMetadata]()
facetedGraph := facets.WrapGraph(graph)

// Add a vector with both metadata and facets
metadata := ProductMetadata{
    Name:  "Product 1",
    Price: 19.99,
    Tags:  []string{"electronics", "gadgets"},
}
facets := map[string]interface{}{
    "category": "electronics",
    "price":    19.99,
    "inStock":  true,
}
err := facetedGraph.Add(hnsw.Node[int]{
    Key:   1,
    Value: []float32{0.1, 0.2, 0.3},
}, metadata, facets)

// Search with a filter and retrieve metadata
filter := facets.Eq("category", "electronics")
results, err := facetedGraph.SearchWithFilterAndMetadata(query, 5, filter)

Creating Custom Extensions

You can create your own extensions to add custom functionality to HNSW. See the Creating Extensions guide for more information.

Next Steps