A high-performance hybrid vector store written in Go. Comet brings together multiple indexing strategies and search modalities into a unified, hackable package. Hybrid retrieval with reciprocal rank fusion, autocut, pre-filtering, semantic search, full-text search, and multi-KNN searches, and multi-query operations -- all in pure Go.

Understand search internals from the inside out. Built for hackers, not hyperscalers. Tiny enough to fit in your head. Decent enough to blow it.

Choose from:

• Flat (exact), HNSW (graph), IVF (clustering), PQ (quantization), or IVFPQ (hybrid) storage backends
• Full-Text Search: BM25 ranking algorithm with tokenization and normalization
• Metadata Filtering: Fast filtering using Roaring Bitmaps and Bit-Sliced Indexes
• Ranking Programmability: Reciprocal Rank Fusion, Fixed size result sets, Threshold based result sets, Dynamic result sets etc.
• Hybrid Search: Unified interface combining vector, text, and metadata search

• Overview
• Features
• Installation
• Quick Start
• Architecture
• Core Concepts
• API Reference
• Examples
• Configuration
• API Details
• Use Cases
• Contributing
• License

Everything you need to understand how vector databases actually work--and build one yourself.

What's inside:

• 5 Vector Storage Types: Flat, HNSW, IVF, PQ, IVFPQ
• 3 Distance Metrics: L2, L2 Squared, Cosine
• Full-Text Search: BM25 ranking with Unicode tokenization
• Metadata Filtering: Roaring bitmaps + Bit-Sliced Indexes
• Hybrid Search: Combine vector + text + metadata with Reciprocal Rank Fusion
• Advanced Search: Multi-KNN queries, multi-query operations, autocut result truncation
• Production Features: Thread-safe, serialization, soft deletes, configurable parameters

Everything you need to understand how vector databases actually work--and build one yourself.

• Flat: Brute-force exact search (100% recall baseline)
• HNSW: Hierarchical navigable small world graphs (95-99% recall, O(log n) search)
• IVF: Inverted file index with k-means clustering (85-95% recall, 10-20x speedup)
• PQ: Product quantization for compression (85-95% recall, 10-500x memory reduction)
• IVFPQ: IVF + PQ combined (85-95% recall, 100x speedup + 500x compression)

• Vector Search: L2, L2 Squared, and Cosine distance metrics
• Full-Text Search: BM25 ranking with Unicode-aware tokenization
• Metadata Filtering: Boolean queries on structured attributes
• Hybrid Search: Combine all three with configurable fusion strategies

• Weighted Sum: Linear combination with configurable weights
• Reciprocal Rank Fusion (RRF): Scale-independent rank-based fusion
• Max/Min Score: Simple score aggregation

• HNSW Graphs: Multi-layer skip lists for approximate nearest neighbor search
• Roaring Bitmaps: Compressed bitmaps for metadata filtering (array, bitmap, run-length encoding)
• Bit-Sliced Index (BSI): Efficient numeric range queries without full scans
• Product Quantization Codebooks: Learned k-means centroids for vector compression
• Inverted Indexes: Token-to-document mappings for full-text search

• Quantization: Full precision, half precision, int8 precision
• Soft Deletes: Fast deletion with lazy cleanup
• Serialization: Persist and reload indexes
• Thread-Safe: Concurrent read/write operations
• Autocut: Automatic result truncation based on score gaps

Output:

Comet is organized into three main search engines that can work independently or together:

Manages vector storage and similarity search across multiple index types.

Common Interface:

Available Implementations:

Responsibilities:

• Vector preprocessing (normalization for cosine distance)
• Distance calculations (Euclidean, L22, Cosine)
• K-nearest neighbor search
• Serialization/deserialization
• Soft delete management with flush mechanism

Performance Characteristics:

• Flat: O(nxd) search, 100% recall
• HNSW: O(Mxefxlog n) search, 95-99% recall
• IVF: O(nProbesxn/kxd) search, 85-95% recall

Full-text search using BM25 ranking algorithm.

Inverted Index:

Term Frequencies:

Document Stats:

Responsibilities:

• Text tokenization (UAX#29 word segmentation)
• Unicode normalization (NFKC)
• Inverted index maintenance
• BM25 score calculation
• Top-K retrieval with heap

Performance Characteristics:

• Add: O(m) where m = tokens
• Search: O(qxd_avg) where q = query terms, d_avg = avg docs per term
• Memory: Compressed inverted index, no original text stored

Fast filtering using compressed bitmaps.

Categorical Fields (Roaring Bitmaps):

Numeric Fields (Bit-Sliced Index):

Document Universe:

Responsibilities:

• Bitmap index maintenance (Roaring compression)
• BSI for numeric range queries
• Boolean query evaluation (AND, OR, NOT)
• Existence checks
• Set operations (IN, NOT IN)

Performance Characteristics:

• Add: O(f) where f = fields
• Query: O(fxlog n) with bitmap operations
• Memory: Highly compressed, 1-10% of uncompressed

How a hybrid search request flows through the system:

Understanding how different index types use memory:

File Header (24 bytes):

Per-Node Storage:

Scaling Analysis:

Compression Overview:

Codebook Storage:

1M Vectors Comparison:

A vector store maintains high-dimensional embeddings and enables efficient similarity search. The choice of storage type determines the tradeoff between search speed, memory usage, and accuracy.

Example: How vectors are stored and searched

Given these input vectors:

The index computes distances and returns nearest neighbors:

Visual Representation: Storage Types Comparison

Benefits of Different Storage Types:

• Flat Index: Perfect recall, simple, no training. Use for small datasets (<100K vectors)
• HNSW: Excellent speed/accuracy tradeoff, no training. Use for most production workloads
• IVF: Fast filtering with clusters, scalable. Use for very large datasets (>10M vectors)
• PQ: Massive memory savings. Use when storage cost is a concern
• IVFPQ: Best of IVF + PQ. Use for extremely large, memory-constrained environments

BM25 (Best Matching 25) ranks documents by relevance to a text query using term frequency and inverse document frequency.

Test Corpus:

Query: "quick brown"

Parameters:

For term "quick":

For term "brown":

Doc 1 (9 words - longer than average):

Doc 2 (4 words):

Doc 3 (4 words - shorter than average):

Combine IDF x TF for each document:

Hybrid search combines vector similarity, text relevance, and metadata filtering. Different fusion strategies handle score normalization differently.

Formula: RRF_score = sum(1 / (K + rank_i)) where K=60

Doc 1 Calculation:

Doc 5 Calculation:

Doc 7 Calculation:

Doc 12 Calculation:

Choose your poison: Flat (exact), HNSW (graph), IVF (clustering), PQ (quantization), or IVFPQ (hybrid). Each trades speed, memory, and accuracy differently.

Full Deep Dive: See INDEX.md for complete algorithms, benchmarks, and implementation details.

The Simplest Approach: Compare query against EVERY vector. 100% recall, zero approximation.

The Graph Navigator: Build a multi-layer graph where each node connects to nearby vectors. Search by greedy navigation from layer to layer.

The Clustering Approach: Partition vectors into clusters using k-means. Search only the nearest clusters.

The Compression Master: Split vectors into subvectors, quantize each subvector to 256 codes. Reduce memory by 16-32x.

Best of Both Worlds: IVF clusters to reduce search space, PQ compression to reduce memory. The ultimate scalability index.

Step 1: IVF Clustering

Step 2: PQ Compression per Cluster

Rules of thumb:

• Small data (<10k): Flat - brute force is fast enough
• Medium data (10k-1M): HNSW - best recall/speed tradeoff
• Large data (1M-100M): IVF or PQ - choose speed (IVF) vs memory (PQ)
• Massive data (>100M): IVFPQ - only option that scales to billions

Latency targets:

• Flat: 1-100ms (depends on n)
• HNSW: 0.1-2ms (sub-millisecond possible)
• IVF: 0.5-5ms
• PQ: 1-10ms (fast scan but approximate)
• IVFPQ: 0.5-3ms (fastest for massive datasets)

For detailed API documentation, see API.md.

For practical examples and code samples, see EXAMPLE.md.

M (Connections Per Layer)

• Type: int
• Default: 16
• Range: 4 to 64
• Description: Number of bidirectional connections per node at each layer (except layer 0 which uses 2xM)

Effects of different values:

efConstruction (Build-Time Candidate List Size)

• Type: int
• Default: 200
• Range: 100 to 1000
• Description: Size of candidate list during index construction. Higher = better graph quality

Effects:

efSearch (Search-Time Candidate List Size)

• Type: int
• Default: 200
• Range: k to 1000 (must be >= k)
• Description: Size of candidate list during search. Can be adjusted dynamically.

Effects:

Example:

nClusters (Number of Clusters)

• Type: int
• Default: sqrt(n) where n = dataset size
• Range: 16 to n/100
• Description: Number of k-means clusters (Voronoi cells) to partition the space

nProbes (Search-Time Clusters to Probe)

• Type: int
• Default: 1
• Range: 1 to nClusters
• Description: Number of nearest clusters to search during query

Example:

Constructor:

Parameters:

Returns:

• *FlatIndex: Initialized flat index
• error: Error if parameters are invalid

Constructor:

Parameters:

Returns:

• *HNSWIndex: Initialized HNSW index
• error: Error if parameters are invalid

Parameter Guidelines:

Constructor:

Parameters:

Returns:

• *IVFIndex: Initialized IVF index
• error: Error if parameters are invalid

Parameter Guidelines:

Search Parameters:

Constructor:

Parameters:

Returns:

• *PQIndex: Initialized PQ index
• error: Error if parameters are invalid or constraints violated

Parameter Guidelines:

Constructor:

Parameters:

Returns:

• *IVFPQIndex: Initialized IVFPQ index
• error: Error if parameters are invalid

Parameter Guidelines:

Search Parameters:

Constructor:

Parameters: None (uses default BM25 parameters: k1=1.5, b=0.75)

Returns:

• *BM25SearchIndex: Initialized BM25 full-text search index

BM25 Parameters:

• k1 = 1.5: Term frequency saturation (typical range: 1.2-2.0)
• b = 0.75: Document length normalization (typical range: 0.5-0.9)

Constructor:

Parameters: None

Returns:

• *RoaringMetadataIndex: Initialized metadata filtering index using Roaring Bitmaps

Constructor:

Parameters:

Returns:

• HybridSearchIndex: Initialized hybrid search combining all three modalities

Creating a Search:

Methods:

HNSW-Specific Methods:

IVF-Specific Methods:

IVFPQ-Specific Methods:

Examples:

// Basic single-query search
results, err := index.NewSearch().
 WithQuery(queryVector).
 WithK(10).
 Execute()

// Multi-query with aggregation
results, _ := index.NewSearch().
 WithQuery(query1, query2, query3).
 WithK(20).
 WithScoreAggregation(comet.MeanAggregation).
 Execute()

// Node-based search (find similar to existing vectors)
results, _ := index.NewSearch().
 WithNodes(1, 5, 10). // IDs of indexed vectors
 WithK(10).
 Execute()

// Pre-filtered search
eligibleIDs := []uint32{100, 200, 300, 400, 500}
results, _ := index.NewSearch().
 WithQuery(queryVector).
 WithDocumentIDs(eligibleIDs...).
 WithK(5).
 Execute()

// Distance threshold filtering
results, _ := index.NewSearch().
 WithQuery(queryVector).
 WithK(100).
 WithThreshold(0.5). // Only distances <= 0.5
 Execute()

// Autocut (automatic result truncation)
results, _ := index.NewSearch().
 WithQuery(queryVector).
 WithK(100).
 WithAutocut(true). // Returns fewer if quality drops
 Execute()

// HNSW with custom efSearch
hnswIndex, _ := comet.NewHNSWIndex(384, comet.Cosine, 16, 200, 100)
results, _ := hnswIndex.NewSearch().
 WithQuery(queryVe