Vector Quantization in pgvector: Scaling to 50M+ Embeddings with Scalar (SQ8) & Product Quantization (PQ)

The Memory Wall of High-Dimensional Vector Search

As enterprise architectures integrate Retrieval-Augmented Generation (RAG), semantic code search, and real-time multimodal agents, PostgreSQL with the pgvector extension has become the default vector database for many teams. However, deploying high-dimensional embeddings at scale rapidly encounters a brutal physical constraint: the memory wall.

Modern embedding models (such as OpenAI's text-embedding-3-large, Cohere Embed v3, or open-source BAAI/bge models) output vectors with 1,536 to 3,072 dimensions. In standard single-precision floating-point format (float32), each dimension consumes 4 bytes:

• A single 1,536-dimensional vector requires 6,144 bytes (~6.1KB) of raw storage.
• A corpus of 50 million vectors requires approximately 307 Gigabytes of disk space just for raw vector data.
• To perform fast approximate nearest neighbor (ANN) lookups using Hierarchical Navigable Small World (HNSW) graphs, the entire index graph must reside permanently in RAM. An HNSW index for 50 million raw float32 vectors demands upwards of 450GB of RAM.

When the index exceeds PostgreSQL's available shared_buffers and OS page cache, query execution degrades from 8 milliseconds to over 800 milliseconds as the server thrashes random 8KB disk pages. To scale cost-effectively, teams must utilize Vector Quantization.

Scalar Quantization (SQ8): 75% Memory Reduction with Sub-1% Recall Loss

Scalar Quantization (SQ) compresses continuous 32-bit floating-point coordinates into discrete 8-bit signed or unsigned integers (int8). Rather than preserving 24 bits of floating-point mantissa precision, SQ maps the min-max coordinate space of each dimension onto 256 discrete integer buckets:

By mapping each 4-byte float down to a 1-byte integer, SQ8 immediately achieves a 4x compression ratio (75% memory reduction):

• The raw vector footprint drops from 6,144 bytes to 1,536 bytes.
• The in-memory HNSW index footprint for 50M embeddings shrinks from 450GB down to approximately 115GB.
• Distance computations (Euclidean distance or Cosine similarity) are accelerated using hardware SIMD instructions (such as AVX-512 or ARM Neon) that process 16 to 64 integer operations per clock cycle.

In empirical evaluation across standard enterprise benchmarks (MTEB and BEIR), SQ8 maintains greater than 99.1% search recall compared to raw float32 searches, making it a virtually lossless optimization for semantic retrieval.

Product Quantization (PQ): Extreme Compression via Sub-Vector Codebooks

When dataset sizes expand into hundreds of millions of embeddings, even SQ8 becomes cost-prohibitive. Product Quantization (PQ) overcomes this limitation by compressing groups of dimensions together into compact cluster codes:

1. Sub-Vector Slicing: A 1,536-dimensional vector is divided into $m$ smaller sub-vectors (e.g., $m = 96$ sub-vectors, each containing 16 dimensions).
2. Codebook Clustering: A training set of vectors is clustered using K-means to generate a codebook of centroids (typically $k = 256$ centroids per sub-space, requiring 1 byte per centroid index).
3. Asymmetric Distance Computation (ADC): When querying, the raw query vector is compared against the precomputed centroid distance lookup tables in RAM, requiring simple byte lookups and integer additions rather than floating-point dot products.

Product Quantization compresses a 1,536-dimensional vector into a tiny 96-byte code—achieving an astonishing 64x compression ratio. A 50-million-vector corpus that previously demanded 450GB of RAM can now reside comfortably inside 16GB of memory.

Implementing Quantized Indexing in pgvector 0.7+

Starting in pgvector 0.7.0, native support for halfvec (16-bit half-precision floats) and quantized vector indexing was introduced. The halfvec type halves memory consumption immediately with zero training overhead, while quantized expressions enable extreme compression.

1. Utilizing 16-Bit Floats (halfvec)

-- Enable the vector extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Define document embeddings using half-precision vectors
CREATE TABLE document_embeddings (
    id BIGSERIAL PRIMARY KEY,
    document_id UUID NOT NULL,
    chunk_index INT NOT NULL,
    embedding halfvec(1536) NOT NULL
);

-- Build an HNSW index directly on halfvec
CREATE INDEX idx_embeddings_hnsw ON document_embeddings 
USING hnsw (embedding halfvec_cosine_ops)
WITH (m = 16, ef_construction = 128);

2. Fast Ingestion & In-Memory HNSW Tuning

When indexing tens of millions of vectors, tuning PostgreSQL maintenance memory is critical to prevent spilling intermediate graph construction to disk:

-- Allocate sufficient memory for index builds
SET maintenance_work_mem = '32GB';
SET max_parallel_maintenance_workers = 8;

-- Execute parallel index creation
CREATE INDEX CONCURRENTLY idx_embeddings_hnsw_quantized 
ON document_embeddings 
USING hnsw (embedding halfvec_l2_ops)
WITH (m = 24, ef_construction = 200);

Inference Acceleration Context: For memory management strategies in conversational AI serving pipelines, see KV Cache Eviction & Prompt Prefix Caching in vLLM: Reducing TTFT by 80% Across Voice AI Dialogues.

Two-Stage Retrieval Architecture: Approximate Search with Exact Re-Ranking

To achieve the pinnacle of performance—sub-5-millisecond latency, minimal RAM footprint, and 99.9% recall accuracy—production architectures employ a Two-Stage Retrieval Pipeline directly in SQL:

1. Stage 1 (Coarse Filter in RAM): Use the quantized HNSW index to rapidly retrieve the top 60 nearest candidate vectors from memory-mapped cache.
2. Stage 2 (Fine Re-Ranking from Disk): Re-score only those 60 candidates using the original full-precision vectors to yield the final top 10 results.

-- High-Precision Two-Stage Retrieval Query
WITH candidates AS (
    -- Stage 1: Fast approximate retrieval via quantized HNSW index
    SELECT id, document_id, embedding
    FROM document_embeddings
    ORDER BY embedding <=> '[0.021, -0.045, ...]'::halfvec(1536)
    LIMIT 60
)
-- Stage 2: Final re-ranking
SELECT document_id, 
       (embedding <=> '[0.021, -0.045, ...]'::halfvec(1536)) AS exact_distance
FROM candidates
ORDER BY exact_distance ASC
LIMIT 10;

This hybrid retrieval strategy decouples RAM capacity from search precision: 99.9% of random disk I/O is eliminated, index memory requirements fall by up to 75%, and production systems scale effortlessly to tens of millions of embeddings on modest hardware.