LLM Engineering

A million 768-dimensional float32 embeddings take 3.1 GB, and a brute-force query costs 7.7×10⁸ multiply-adds. Product quantization to 64 one-byte codes per vector stores the same index in 0.064 GB. And in 768 dimensions unrelated vectors are nearly perpendicular — a mean absolute cosine of 0.029 — so “similar” scores need calibrating on real pairs.

• Approximate indexes trade recall for speed; measure recall against exact search on your data.
• Embedding similarity is topical, not logical: negations and numbers are often ignored.
• Changing the embedding model means re-embedding everything; vectors from two models are not comparable.

n, d = 1_000_000, 768
memory_gb = n * d * 4 / 1e9                      # float32
flops_per_query = n * d                          # brute force: one dot product per stored vector
pq_bytes = n * 64                                # product quantization to 64 one-byte codes
rng = np.random.default_rng(5)
X = rng.normal(size=(1_000, d)); X /= np.linalg.norm(X, axis=1, keepdims=True)
typical_cos = np.abs(X[:500] @ X[500:].T).mean()   # unrelated random vectors in 768-d

Assumes import numpy as np. Configurations named after real models use their published shapes; other numbers are made up or simulated, as the comments say.

import faiss
import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-base-en-v1.5")

# Index documents
docs = ["Machine learning basics", "Neural networks intro", ...]
embeddings = model.encode(docs, normalize_embeddings=True)
dim = embeddings.shape[1]

index = faiss.IndexFlatIP(dim)  # inner product (= cosine for normalized)
index.add(embeddings.astype('float32'))

# Search
query = model.encode(["How do neural nets work?"],
                     normalize_embeddings=True)
scores, indices = index.search(query.astype('float32'), k=5)
for i, (score, idx) in enumerate(zip(scores[0], indices[0])):
    print(f"{i+1}. [{score:.3f}] {docs[idx]}")Copy