search-lab

A focused proof-of-concept for hybrid search relevance engineering, built to exercise the skills in a "Senior Software Engineer — Search" role: OpenSearch/Elasticsearch index design, vector search, hybrid retrieval (keyword + vector), reranking, and — the differentiator — offline relevance evaluation with real metrics (NDCG/precision/recall).

This is the rigorous, measurable half of a two-project split. The other half is trippin (a personal travel-photo app) which demonstrates multimodal/image search as a product surface but can't be evaluated rigorously — its corpus is small, private, and has no labeled relevance judgments. search-lab uses public IR benchmarks that ship with judgments, so every pipeline change produces a comparable number against a published baseline. Same retrieval-engineering skills; this one has a scoreboard.

Why benchmark datasets (the core idea)

Public IR benchmarks are academic datasets (BEIR from UKP Lab/TU Darmstadt; MS MARCO from Microsoft) that let anyone building a retriever measure it against everyone else's on fixed data. Each dataset is three parts:

1. Corpus — the documents: {doc_id, text}.
2. Queries — {query_id, text}.
3. Qrels (query relevance judgments) — the human-labeled answer key: {query_id, doc_id, relevance}.

The qrels are the expensive hand-labeled part we can't cheaply produce ourselves — the benchmark hands them over for free. We build the retriever; the dataset grades it. The eval loop:

for each query:
    our_system.search(query.text, corpus)  ──►  ranked list of doc_ids
compare(ranked_lists, qrels)               ──►  NDCG@10, Recall@100, MRR

The metric reads our ranking, looks up which docs the qrels marked relevant, and rewards putting relevant docs near the top. We are not evaluating the dataset — the dataset is evaluating our pipeline.

Note: these benchmarks are text retrieval. That's fine — the point here is to prove out the retrieval engineering (index design, hybrid fusion, reranking, evaluation methodology), which then transfers to trippin's multimodal photos un-benchmarked.

Candidate datasets

• BEIR — suite of ~15 retrieval datasets with standard qrels. Start with small ones for a fast dev loop: SciFact (~5k docs), NFCorpus (~3.6k docs), FiQA (finance QA). "My hybrid setup gets X NDCG@10 on SciFact" is directly comparable to published numbers.
• MS MARCO passage — huge, the canonical reranking dataset. Use later for the reranking story; too big for the first loop.
• Travel-flavored (optional, to rhyme with trippin) — Yelp / Amazon reviews / TripAdvisor-style data; bigger and shareable but judgments must be synthesized, so weaker for rigorous eval. Secondary.

Recommendation: start with BEIR SciFact + NFCorpus. Small enough to iterate in seconds, published baselines to sanity-check against.

Architecture: a backend-agnostic retriever

Mirror the pattern already used in trippin (SearchBackend interface): the eval harness talks to a Retriever protocol; each backend is a swappable implementation. This makes the Elasticsearch-vs-Postgres comparison a config flag, not a rewrite.

BEIR loader ──► {corpus, queries, qrels}
                     │
        index corpus │ into each backend
                     ▼
   ┌─────────────────┴──────────────────┐
   ▼                                     ▼
ElasticRetriever                   PostgresRetriever
BM25 + dense_vector(kNN)           tsvector (FTS) + pgvector
hybrid via RRF / hybrid pipeline   hybrid via RRF in SQL
   │                                     │
   └──────────────► run queries ◄────────┘
                     │ ranked doc_ids
                     ▼
              eval harness (ir_measures)
              NDCG@10, Recall@100, MRR, MAP
                     │
                     ▼
              results table per (backend × mode)

# retriever protocol (sketch)
class Retriever(Protocol):
    def index(self, corpus: dict[str, dict]) -> None: ...
    def search(self, query: str, k: int, mode: Mode) -> list[tuple[str, float]]: ...
    # mode ∈ {"lexical", "semantic", "hybrid"}; returns (doc_id, score) ranked

Backends

• ElasticRetriever — primary. Index design (mappings/analyzers), a text field for BM25 and a dense_vector field (HNSW) for the embedding. Hybrid via native RRF (Elasticsearch) or a hybrid-query normalization pipeline (OpenSearch). This is the JD-named skill surface.
• PostgresRetriever — comparison baseline. tsvector + websearch_to_tsquery for lexical, pgvector cosine for semantic, fused with RRF in SQL (CTE per arm, 1/(k+rank) summed). Shows the "when do you actually need a dedicated search engine?" trade-off with numbers.

The comparison is itself an interview asset: "I ran the same hybrid pipeline on Elasticsearch and Postgres/pgvector and measured the relevance and latency delta on BEIR."

Eval harness — the deliverable that matters

Produce a results table after each pipeline change. The table is the story:

All rows are live numbers on BEIR SciFact (test, 5183 docs, 300 queries); latency is client-side per-query wall time (warm, single-threaded).

Backend	Mode	NDCG@10	Recall@100	MRR	p50 ms	p95 ms
Elastic	BM25 (lexical)	0.6537	0.8846	0.6236	21.3	57.0
Elastic	BM25 + rerank	0.6743	0.8846	0.6539	—	—
Elastic	vector (semantic)	0.6484	0.9250	0.6123	34.2	68.6
Elastic	hybrid (RRF)	0.6896	0.9543	0.6612	58.4	103.2
Elastic	hybrid + rerank	0.6862	0.9543	0.6611	—	—
Postgres	FTS (lexical)	0.2975	0.7459	0.2603	21.0	55.5
Postgres	vector (semantic)	0.6484	0.9250	0.6123	30.8	65.4
Postgres	hybrid (RRF-in-SQL)	0.5763	0.9260	0.5487	61.8	115.7

The Elastic BM25 row lands right on the published BEIR BM25 baseline for SciFact (~0.665 nDCG@10), sanity-checking the index design and eval harness end to end. The rerank rows' latency is dominated by the cross-encoder (~seconds/query on CPU), out of scale with the retrieval rows, so it's reported in the M3 writeup rather than here.

The ES-vs-Postgres read (M4): the dense arm is identical across backends — same bring-our-own vectors, both on HNSW+cosine → the same neighbors and the same 0.6484/0.9250. Latency is comparable at this corpus size (Postgres is not the slower store here). The whole gap is lexical: stock Postgres FTS (ts_rank_cd) scores less than half of Lucene BM25 (0.30 vs 0.65), and that weak arm drags Postgres's hybrid below its own semantic arm — RRF can't rescue a bad arm. So on this workload the reason to run Elasticsearch is BM25 quality, not speed. (ADR 0005)

The per-query evidence behind these rows — which queries each arm wins, where fusion rescues, why reranking is conditional — lives in notebooks/relevance-analysis.ipynb, which reads the committed run artifacts under runs/ and needs no backends running.

Rules of the road:

• Use standard metrics via a standard library so numbers are comparable to papers (don't hand-roll NDCG).
• Log per-query latency (p50/p95) alongside relevance — the JD cares about both.
• Keep the qrels/metric computation identical across backends; only the retriever changes.

Embeddings

Bring-our-own vectors to keep backends comparable (same vectors indexed into ES and Postgres):

• Reuse the OpenRouter OpenAI-compatible /v1/embeddings path already proven in trippin (google/gemini-embedding-2-preview, pin dimensions), or
• a local sentence-transformers model (e.g. all-MiniLM-L6-v2, 384-dim) for zero-cost offline iteration. Recommended for the first loop — no API cost, fully reproducible.

Later, optionally compare against Elastic's managed options (ELSER learned-sparse, Elastic Inference Service) to show awareness of turnkey relevance features.

Proposed stack (Python)

• Python 3.12+, uv for env/deps, ruff + pyright for lint/types (matches global tooling prefs).
• elasticsearch (elasticsearch-py) or opensearch-py — the search client.
• beir and/or ir_datasets — dataset loading (corpus/queries/qrels).
• ir_measures (or pytrec_eval) — standard metrics; do not hand-roll.
• sentence-transformers — local embeddings for the offline loop.
• psycopg + pgvector — the Postgres comparison backend.
• Elasticsearch/OpenSearch + Postgres via docker compose for local dev.

Milestones

1. Loader + baseline — pull BEIR SciFact; stand up Elasticsearch in docker; index the corpus; BM25-only search; wire ir_measures → first NDCG@10 number. Sanity-check against the published BM25 baseline.
2. Vector + hybrid — add local embeddings + dense_vector/kNN; add hybrid via RRF; extend the results table. This is the core "hybrid retrieval" deliverable.
3. Reranking — cross-encoder rerank of top-k on MS MARCO or a BEIR set; measure NDCG lift. Hits the JD's "reranking with measurable relevance."
4. Postgres comparison — implement PostgresRetriever (FTS + pgvector + RRF-in-SQL); same harness; compare relevance + latency to Elasticsearch.
5. Writeup — short results doc + runbook: what won, by how much, at what latency/cost. This is the artifact to show.

JD skill coverage (traceability)

• OpenSearch/ES index design (mappings/analyzers) → milestone 1–2, ElasticRetriever.
• Vector search / kNN + hybrid retrieval → milestone 2.
• Relevance experiments + offline metrics (NDCG/precision/recall) → eval harness, all milestones.
• Reranking / LLM-enabled retrieval → milestone 3 (+ optional query rewriting/RAG later).
• Evaluate & recommend technologies with trade-offs → milestone 4 (ES vs Postgres, with numbers).
• DevOps mindset (instrumentation, cost, runbooks) → latency logging throughout + milestone 5 writeup.

Quickstart

just setup                    # check prereqs, uv sync, wire hooks + .env
just up                       # start Elasticsearch (+ Postgres) via docker compose
uv sync --extra datasets --extra elastic --extra embeddings

uv run search-lab index beir/scifact/test                              # load + index corpus (text + vectors)
uv run search-lab eval  beir/scifact/test --mode lexical  # BM25       → results row
uv run search-lab eval  beir/scifact/test --mode semantic # dense kNN  → results row
uv run search-lab eval  beir/scifact/test --mode hybrid   # RRF fusion → results row
uv run search-lab eval  beir/scifact/test --mode hybrid --rerank # + cross-encoder rerank
uv run search-lab search "BRCA1 and DNA repair" --dataset beir/scifact/test --mode hybrid --k 5

# Postgres comparison backend (same corpus, same vectors, same harness):
uv sync --extra postgres
uv run search-lab index beir/scifact/test --backend postgres
uv run search-lab eval  beir/scifact/test --backend postgres --mode hybrid

# Native ES RRF retriever (licensed): ES_LICENSE=trial just up, then add --native to a hybrid eval.

just check runs format/lint/types; just validate mirrors CI (adds tests + coverage). just down stops services.

Status

Milestones 1–5 done — project complete. M1: BEIR SciFact loads via ir_datasets; Elasticsearch stands up in docker; the corpus indexes; BM25 (mode=lexical) → 0.6537 nDCG@10, on the published BM25 baseline. M2: local all-MiniLM-L6-v2 embeddings → dense_vector/kNN (mode=semantic) → hybrid via RRF (mode=hybrid). Hybrid beats BM25 on every metric — 0.6896 nDCG@10 (+3.6 pts), 0.9543 Recall@100 (+7.0 pts). Native Elasticsearch RRF is a licensed feature, so Python-side RRF is the reproducible default with the native rrf retriever as an opt-in --native path (ADR 0003); the two return identical numbers.

M3: a --rerank cross-encoder second stage (cross-encoder/ms-marco-MiniLM-L-6-v2) reorders the top-k pool over any first-stage mode (ADR 0004). The measured result is the honest one: reranking lifts the weak lexical stage (+2.1 pts nDCG@10, +3.0 pts MRR) but slightly regresses the already-strong hybrid (−0.34 pts) — a MS-MARCO cross-encoder can't out-order hybrid RRF on scientific-claim text, and hybrid alone still beats BM25+rerank. Both rerank rows leave Recall@100 exactly unchanged (0.8846 / 0.9543): reranking reorders the funnel, it can't widen it. Takeaway: on SciFact, hybrid RRF is the pick; a bolt-on cross-encoder isn't worth its latency here — the kind of measured trade-off M5 will report in full.

M4: a PostgresRetriever (--backend postgres) runs the same pipeline on Postgres — FTS (tsvector/ts_rank_cd) for lexical, pgvector HNSW for semantic, RRF-in-SQL for hybrid — through the identical harness, now instrumented with p50/p95 latency (ADR 0005). The comparison (full table above): the dense arm is identical across backends (same vectors → same 0.6484/0.9250), latency is comparable at this corpus size, but stock Postgres FTS scores less than half of Lucene BM25 (0.30 vs 0.65 nDCG@10) — and that weak lexical arm drags Postgres's hybrid below its own semantic arm (0.576 < 0.648), because RRF can't rescue a bad arm. The recommendation falls out with numbers: at this scale you reach for Elasticsearch for BM25 quality, not speed.

M5: the consolidated writeup — RESULTS.md (the scoreboard, the four findings, cost, and the recommendation) and docs/runbook.md (reproduce every row end to end). The read: hybrid RRF on Elasticsearch is the pick (0.6896 nDCG@10, p50 58 ms); ES's edge over Postgres is entirely BM25 quality, not speed; reranking pays only against a weak first stage. Related design context lives in trippin/wiki/search.md (the backend-abstraction pattern and an Elastic Serverless cost analysis).