Phase 2Weeks 5-79 machine-verified checks · 1 tutorial pack

Production RAG

From toy retrieval to retrieval you can bill on.

Why this phase exists

Chunk, embed, retrieve is the right first brick and a bad finished product. In a real product, wrong retrieval is a wrong quote, a wrong dosage, or a compliance breach. This phase is about knowing your retrieval is good before a customer finds out it is not.

What you learn

Query rewriting and intent extraction: fixing the question before you search
Hybrid search: keyword and vectors together, and when each one wins alone
Re-ranking with cross-encoders, and the latency you pay for it
Tenant filters and multi-tenant isolation as a correctness problem, not just a security one
Bad-chunk failure modes: split tables, orphaned headers, lost context
Retrieval metrics: hit rate, recall@k, MRR, nDCG, groundedness, measured before launch

Outcome: You ship RAG where wrong retrieval is caught by metrics, not by customers.

Tools you will actually touch

pgvectorQdrant / OpenSearchBM25Cross-encoder re-rankersRagas

The assignment pack

Scenario

A B2B supplier quotes on material grades, coatings, and norms. Retrieving the wrong spec produces a wrong price and a real financial loss. Build retrieval over their document set where the wrong-spec failure mode is measured and bounded.

Definition of done

Hybrid retrieval (keyword + vector) with a documented reason for the blend
Query rewriting step that measurably improves recall on the golden set
Re-ranking with a measured precision gain and a stated latency cost
Hard tenant isolation: a proven test that tenant A cannot retrieve tenant B content
A retrieval metrics report: recall@k and groundedness, before and after each change

How it is graded

Fail

Single vector search, no metrics, quality asserted by eyeballing a few queries.

Pass

Hybrid + re-ranking + tenant filters, with recall@k measured on a real golden set.

Strong pass

Every component change is justified by a metric delta, latency budget is explicit, and failure modes are documented with examples.

The tutorial pack

Take a toy RAG to a measured one

1Build the naive version first: chunk, embed, retrieve. Establish the baseline
2Construct a golden set of real questions with known correct sources
3Measure recall@5 on the baseline and write the number down
4Add BM25 and blend the scores; re-measure
5Add a cross-encoder re-ranker; re-measure and record the latency cost
6Add tenant filters and write the isolation test

The failure you will hit on purpose

Chunk a document so a table is split across two chunks, then ask a question whose answer needs the whole table. Watch a confident, wrong answer come back with high similarity. Then fix the chunker.

What you measure at the end

recall@5 and MRR, baseline versus final
p95 retrieval latency before and after re-ranking
Groundedness rate on the golden set
Tenant isolation test passes with zero leakage

Per-target lab variants

Same outcome, five stacks. You train on the one you will actually run.

AWSOpenSearch hybrid search + Bedrock embeddings + S3 corpus lifecycle

GCPVertex Vector Search or AlloyDB pgvector + Vertex embeddings

AzureAzure AI Search with semantic ranker + Azure OpenAI embeddings

On-premQdrant or pgvector with a locally served embedding model, no data egress

Agnosticpgvector + OSS re-ranker on any Kubernetes, portable by design

You must be able to answer

AI use is mandatory on every assignment. The integrity mechanism is defend-your-work, so these are the questions that decide whether the work counts.

“Your recall@5 is 0.82. Is that good? What would make it meaningful?”
“When does keyword search beat your embeddings, and did you verify that?”
“Show a query where re-ranking changed the answer. Was the trade worth the latency?”
“How do you know a new document did not silently poison retrieval?”

Phase 1 · The AI Control Plane

Phase 3 · Agent Control

Want to be walked through this phase?

The packs are delivered with live teaching, reviewed assignments, and a defence.

Train with me