Skip to content

Metadata Extension

The Metadata Extension for HNSW allows you to store and retrieve JSON metadata alongside vectors in the graph. This is useful for applications where you need to associate additional information with each vector, such as product details, document content, or image attributes.

Features

  • Store arbitrary JSON metadata with each vector
  • Retrieve metadata along with search results
  • Type-safe implementation using Go generics
  • Memory-efficient storage
  • Support for batch operations
  • Seamless integration with the core HNSW API

Installation

The Metadata Extension is included in the HNSW package and can be imported as follows:

import (
    "github.com/TFMV/hnsw"
    "github.com/TFMV/hnsw/hnsw-extensions/meta"
)

Basic Usage

Creating a Metadata Graph

To create a graph with metadata support, use the NewMetaGraph function:

// Define a type for your metadata
type ProductMetadata struct {
    Name        string    `json:"name"`
    Category    string    `json:"category"`
    Price       float64   `json:"price"`
    Tags        []string  `json:"tags"`
    InStock     bool      `json:"in_stock"`
    ReleaseDate time.Time `json:"release_date"`
}

// Create a new graph with metadata support
graph := meta.NewMetaGraph[int, ProductMetadata]()

Adding Vectors with Metadata

To add a vector with metadata to the graph:

// Create a vector
vector := []float32{0.1, 0.2, 0.3, 0.4, 0.5}

// Create metadata
metadata := ProductMetadata{
    Name:        "Product 1",
    Category:    "Electronics",
    Price:       199.99,
    Tags:        []string{"gadget", "smartphone"},
    InStock:     true,
    ReleaseDate: time.Now(),
}

// Add the vector with metadata to the graph
err := graph.Add(hnsw.Node[int]{
    Key:   1,
    Value: vector,
}, metadata)

Searching and Retrieving Metadata

To search for vectors and retrieve their metadata:

// Create a query vector
query := []float32{0.15, 0.25, 0.35, 0.45, 0.55}

// Search for the 5 nearest neighbors and retrieve their metadata
results, err := graph.SearchWithMetadata(query, 5)
if err != nil {
    // Handle error
}

// Process the results
for _, result := range results {
    fmt.Printf("Key: %d, Distance: %f\n", result.Key, result.Dist)
    fmt.Printf("Name: %s, Price: %.2f\n", result.Metadata.Name, result.Metadata.Price)
}

Retrieving Metadata for a Specific Node

To retrieve the metadata for a specific node:

metadata, err := graph.GetMetadata(1)
if err != nil {
    // Handle error
}
fmt.Printf("Name: %s, Price: %.2f\n", metadata.Name, metadata.Price)

Updating Metadata

To update the metadata for an existing node:

// Get the existing metadata
metadata, err := graph.GetMetadata(1)
if err != nil {
    // Handle error
}

// Update the metadata
metadata.Price = 149.99
metadata.InStock = false

// Save the updated metadata
err = graph.UpdateMetadata(1, metadata)
if err != nil {
    // Handle error
}

Deleting Nodes with Metadata

To delete a node and its metadata:

err := graph.Delete(1)
if err != nil {
    // Handle error
}

Batch Operations

Adding Multiple Vectors with Metadata

To add multiple vectors with metadata in a batch:

// Create nodes with metadata
nodes := []hnsw.Node[int]{
    {Key: 1, Value: []float32{0.1, 0.2, 0.3}},
    {Key: 2, Value: []float32{0.2, 0.3, 0.4}},
    {Key: 3, Value: []float32{0.3, 0.4, 0.5}},
}

metadata := []ProductMetadata{
    {Name: "Product 1", Price: 199.99},
    {Name: "Product 2", Price: 299.99},
    {Name: "Product 3", Price: 399.99},
}

// Add the nodes with metadata to the graph
err := graph.BatchAdd(nodes, metadata)
if err != nil {
    // Handle error
}

Retrieving Metadata for Multiple Nodes

To retrieve metadata for multiple nodes:

keys := []int{1, 2, 3}
metadataMap, err := graph.GetMetadataBatch(keys)
if err != nil {
    // Handle error
}

for key, metadata := range metadataMap {
    fmt.Printf("Key: %d, Name: %s, Price: %.2f\n", key, metadata.Name, metadata.Price)
}

Advanced Usage

Custom Metadata Store

By default, the Metadata Extension uses an in-memory store for metadata. However, you can implement your own metadata store by implementing the MetadataStore interface:

type MetadataStore[K comparable, M any] interface {
    Add(key K, metadata M) error
    Get(key K) (M, error)
    GetBatch(keys []K) (map[K]M, error)
    Update(key K, metadata M) error
    Delete(key K) error
    DeleteBatch(keys []K) error
}

For example, you could implement a metadata store that uses a database or a file system to store metadata.

Combining with Other Extensions

The Metadata Extension can be combined with other extensions, such as the Faceted Search extension, to create a more powerful search system. See the Extensions Overview for more information.

Performance Considerations

The Metadata Extension is designed to be memory-efficient and performant. However, storing large amounts of metadata can increase memory usage and affect performance. Consider the following tips:

  • Keep metadata small and focused on essential information
  • Use batch operations for better performance when adding or retrieving multiple nodes
  • Consider implementing a custom metadata store for very large datasets or when persistence is required

Next Steps