AI Practitioner Certificate · M02.1

Production-Grade RAG

90 minutes · 39 planned slides · assembled 90.0 minutes · Stage 3 content master

Production RAG is seven owned decisions in a chain; reliable answers come from knowing where each decision starts, when its default stops working, and which component failed.

Ledger & open items

Topic	Status	Slides	Opening question	Kind	Closing link	Dated figures
T1	assembled — pending owner approval	1–5	If every answer cites a source, which failure could still make it unsafe?	show-of-hands risk vote	The first costly component decision is chunking.	—
T2	assembled — pending owner approval	6–13	Where would this policy lose its exception if it were cut every 512 tokens?	mark the break on a document	Chunks still require a model to define similarity.	Metadata update capability
T3	assembled — pending owner approval	14–18	Which would retrieve Japanese financial text better: the larger general model or the smaller specialist?	predict the benchmark winner	Embeddings need an operational home.	fig-t3-japanese-evidence; fig-t3-japanese-evidence; fig-t3-six-gates
T4	assembled — pending owner approval	19–25	If restricted content appears in retrieval, which component was supposed to stop it?	assign the control owner	Fast retrieval does not guarantee correct evidence.	fig-t4-operating-shapes; fig-t4-operating-shapes; fig-t4-operating-shapes; fig-t4-operating-shapes; fig-t4-filter-placement; fig-t4-permission-path; fig-t4-refresh-owner
T5	assembled — pending owner approval	26–32	Which retriever would find product code AC-417: one that understands meaning or one that matches characters?	choose the retriever	Advanced patterns require a measured unresolved failure.	fig-t5-dense-miss; fig-t5-rerank-boundary; fig-t5-three-retrieval-questions
T6	assembled — pending owner approval	33–34	Should a greenfield team start with agentic RAG because it can search again?	provocation withdrawn	Diagnose the baseline before adding variance.	—
T7	assembled — pending owner approval	35–39	One pipeline will fail four ways; can you locate each broken component before the reveal?	live diagnosis challenge	Change the model only after retrieval is measured and still short.	—

T1 · 11 minutes · slides 1–5

Why RAG, and the Pipeline Anatomy

Slide 1question1.5 min

If every answer cites a source, which failure could still make it unsafe?

Vote with one hand: unsupported claim, stale evidence, unauthorised evidence, or none of these.
By the end of this morning, you will know what each piece does — and, for each one, where to start.

Visual brief: Full-bleed show-of-hands risk vote with four equal response cards. Keep all cards visually neutral so the screen does not disclose that more than one answer can be unsafe.

Designer constraint: Do not reveal the answer through colour or emphasis before the room votes.

Speaker notes

Takeaway

A citation makes an answer traceable, but it does not prove that the answer is safe.

Key Points

Hold the vote before teaching so the room first exposes its own assumptions about citation safety.
A citation can point to evidence that is stale, unauthorised or unrelated to the generated claim.
Production safety depends on the complete evidence path rather than on the presence of a citation badge.

Prep Notes

Pause for 20 seconds and take the vote before commenting; keep every response option visually neutral and do not reveal that more than one failure can make the answer unsafe. Notice the spread of hands, because it provides a quick view of whether the room treats citation as proof or as a trace.

Steer toward the distinction that a citation helps a reviewer follow the answer back to evidence, but cannot establish by itself that the evidence is current, authorised or relevant to the claim. Use that distinction to introduce the session's central concern: production safety belongs to the whole evidence path, not to a citation badge.

Slide 2diagram3.5 min

The seven-stage pipeline separates what teams build once from what every question reruns.

Build once: Ingest → Chunk → Embed → Index.
Run per question: Retrieve → optional Re-rank → Generate.
Retrieve may combine semantic and lexical candidates; query embedding happens inside this stage.
The seven stages are a teaching spine, not a complete production architecture.

Preserve the build-once versus run-per-question split, the lexical leg inside Retrieve, and the dashed optional Re-rank stage; never flatten V01 into one undifferentiated chain.

Speaker notes

Takeaway

The seven-stage teaching spine separates durable build decisions from the stages rerun for every question.

Key Points

Ingest, Chunk, Embed and Index create the searchable evidence base, while Retrieve, optional Re-rank and Generate run per question.
Query embedding and any combination of semantic and lexical candidate retrieval belong inside Retrieve rather than forming an eighth stage.
The seven stages are Chartered Vectorial's teaching decomposition, not a claim that every production architecture has exactly seven components.

Prep Notes

Read the visual from the build-once side to the run-per-question side, preserving the boundary between them. Walk the stages in one line each: Ingest selects the corpus, Chunk sets evidence units, Embed encodes them, Index stores them, Retrieve finds candidates, optional Re-rank refines the shortlist, and Generate writes from the assembled evidence. Point out the lexical leg inside Retrieve and the dashed treatment of Re-rank so the audience does not mistake either semantic-only retrieval or re-ranking for a universal requirement.

Explain that query embedding belongs inside Retrieve, not as an eighth stage, and that the seven stages are Chartered Vectorial's teaching decomposition rather than an industry standard or a complete production architecture. Sources: Lewis et al., NeurIPS 2020; AWS, Microsoft Azure Architecture Center, and Google Cloud reference architectures; retrieved 27 July 2026.

Slide 3comparison2.5 min

RAG provides grounding and attribution; hallucination remains a generation risk.

Grounding: fetch evidence that can be updated outside the generator.
Attribution: retain the source used to produce the answer.
Permissions become enforceable at retrieval time — if you build the enforcement.
A citation does not prove that the answer is supported, current or authorised.

Keep the capability and the limit visible together; grounding must never be drawn as a guarantee of factuality or access control.

Speaker notes

Takeaway

RAG improves grounding and attribution, but it does not guarantee that a generated answer is correct, current or authorised.

Key Points

Retrieval quality is the ceiling on answer quality, yet good retrieval remains necessary rather than sufficient for a correct answer.
A model can still fail with good evidence and can sometimes answer correctly from insufficient evidence, so output quality alone does not diagnose the pipeline.
Permissions become enforceable at retrieval time — if you build the enforcement; RAG supplies the enforcement point, not access control itself.

Prep Notes

Keep the capability and its limit visible together. Grounding lets the system fetch evidence that can be updated outside the generator, and attribution lets it retain the source used for an answer; neither property makes a citation proof that the answer is supported, current or authorised. Emphasise the locked principle, “Retrieval quality is the ceiling on answer quality,” then qualify it immediately: good retrieval is necessary, not sufficient, because a model can still fail with good evidence and can sometimes answer correctly from insufficient evidence.

Treat access control with the same discipline. Say, “Permissions become enforceable at retrieval time — if you build the enforcement.” RAG supplies a place to apply access control, not access control itself, so the visual must not depict grounding as a guarantee of factuality or permission enforcement. Sources: Lewis et al., NeurIPS 2020; Joren et al., ICLR 2025; OWASP RAG Security Cheat Sheet, version date unverified; retrieved 27 July 2026.

Slide 4concept2 min

Each role owns a different failure surface, so production readiness cannot sit with the model team alone.

Developer: build retrieval behaviour, context assembly and failure visibility.
Infrastructure: own index refresh, rebuild, availability and service cost.
Risk/operations: set access rules, evidence retention and quality acceptance.
Shared: name an owner for every stage before treating the pipeline as production-ready.

Keep the three roles at equal visual weight and connect each to a concrete operating responsibility rather than a generic stakeholder label.

Speaker notes

Takeaway

Production readiness requires shared ownership because each role controls a different RAG failure surface.

Key Points

Developers own retrieval behaviour, context assembly, intermediate evidence visibility and their share of the latency budget.
Infrastructure owns the stateful index, including its availability, cost, refresh and rebuild paths.
Risk and operations define access, evidence retention and quality acceptance, while the team names an owner for every stage.

Prep Notes

Read the three roles at equal weight and connect each one to concrete operating responsibility. Tell developers that retrieval consumes part of the latency budget and must expose intermediate evidence; tell infrastructure that an index is stateful and its refresh and rebuild paths need owners; and tell risk and operations that citations support an audit trail only when evidence, permissions and acceptance criteria are retained. The point is not to assign the entire pipeline to one group, but to make the handoffs and shared accountability explicit before the system is treated as production-ready.

This is the session's single explicit role summary. Later topics should teach through the developer, infrastructure and risk/operations lenses without repeating this layout, so use this slide to establish the ownership vocabulary that those later implications will rely on.

Slide 5transition1.5 min

Chunking fixes the evidence units until teams re-index, making it the first costly component decision.

Chunking sets the boundaries of what retrieval can return as evidence.
Changing the strategy rebuilds derived chunks, embeddings and index entries.

Keep Chunk on the build-once side and show its downstream rebuild path; the handover must make the reversal cost visible.

Speaker notes

Takeaway

Chunking is the first costly component decision because changing evidence boundaries forces downstream artefacts to be rebuilt.

Key Points

Chunking chooses the evidence unit on the build-once side of the pipeline.
Changing the chunking strategy rebuilds derived chunks, embeddings and index entries rather than altering only one isolated setting.
The immediate question is where a document loses meaning when a cut separates an answer from its qualifying context.

Prep Notes

Close with the handover line: the first costly component decision is chunking. Point to Chunk on the build-once side, then follow its downstream path through embeddings and index entries so the reversal cost is visible. Do not teach chunking strategies on this slide; establish only that the team chooses the evidence unit during the build and that changing this decision rebuilds the derived artefacts.

The topic has established the complete pipeline and the difference between grounding and a guarantee. The next topic now examines the first component decision in detail by asking where a policy loses its exception when the cut is wrong.

T2 · 15 minutes · slides 6–13

Chunking Strategies

Slide 6question1 min

Where would this policy lose its exception if it were cut every 512 tokens?

Read the synthetic policy fragment before discussing a strategy.
Mark the boundary that would make the evidence incomplete.
Treat 512 tokens as an illustrative starting point, not a universal constant.

Keep the rule and exception verbatim and keep the generated demo answer hidden.

Speaker notes

Takeaway

A regular token boundary can make retrieved evidence incomplete by separating a rule from the exception that governs it.

Key Points

The failure mechanism is the break between the synthetic rule and its governing exception.
Participants should locate the break before discussing chunking strategies or solutions.
The 512-token boundary is an illustrative starting point, not a measured universal optimum.

Prep Notes

Give the room about 20 seconds in silence, then ask one person to point at the break rather than explain a solution. Steer toward the idea that a fixed boundary can separate a rule from the exception that governs it; do not ask what answer the demo system will generate.

The policy is synthetic and illustrative, and the 512-token framing is a starting point rather than a measured universal optimum. Preserve the no-spoiler boundary: participants may identify the severed clause, but the resulting demo answer remains hidden until D01.

Slide 7concept1.5 min

Whole documents retrieve too broadly, while smaller units expose the passage a question needs.

One vector for a long document compresses many unrelated meanings into one representation.
Smaller evidence units let retrieval match the passage the question actually needs.
Chunking still matters when an embedding model can accept the whole document.
The cost of a poor cut appears later as incomplete evidence, not as an ingestion error.

The durable argument is retrieval precision and semantic over-compression, not a model input limit.

Speaker notes

Takeaway

Chunking creates focused evidence units so retrieval can match the passage a question needs instead of compressing an entire document into one representation.

Key Points

A long document can contain unrelated meanings that become over-compressed when represented as one retrieval unit.
The chunk determines the record developers build, the index shape infrastructure operates and the evidence risk teams can later inspect.
This slide addresses retrieval precision; embedding-model context ceilings and silent truncation belong to T3.

Prep Notes

Keep this to the retrieval argument; T3 owns embedding-model context ceilings and silent truncation. Explain that chunking still matters even when an embedding model can technically accept the whole document, because input capacity does not guarantee a useful retrieval representation.

The Late Chunking paper states that shorter segments often retrieve better because their meaning is less over-compressed, but it is a vendor-authored preprint and is used here only for the mechanism, not as proof of a product capability. For developers the cut changes the record they build, for infrastructure it fixes the shape of the index, and for risk it fixes the evidence that can later be cited.

Slide 8diagram3 min

Fixed, semantic and hierarchical cuts give the same policy document three different retrieval futures.

Fixed-size creates a cheap, regular baseline but can split a governing exception.
Semantic chunking infers topic boundaries but has not shown a consistent gain over fixed-size.
Hierarchical retrieval matches a precise child and returns its verbatim parent for context.
Overlap and parent return are chunk-time mitigations, not downstream repairs.

Use the same synthetic clause in all three panels and in D01 Act 1; show the severed clause but never the demo answer.

Speaker notes

Takeaway

Fixed, semantic and hierarchical chunking reshape the same source into different evidence units, so each creates a different retrieval outcome and operating cost.

Key Points

Fixed-size is the cheap regular baseline, semantic chunking infers topic boundaries and hierarchical retrieval matches a child before returning its verbatim parent.
Overlap and parent return mitigate boundary failures when evidence extends beyond one child unit.
The available comparisons do not establish semantic chunking as consistently superior or hierarchical results as direct evidence for banking-policy retrieval.

Prep Notes

Walk the visual from left to right and keep the synthetic rule and exception constant across all three panels. Give semantic chunking less airtime because the controlled evidence is consistently weak or neutral, and show only the severed clause: the resulting D01 demo answer must remain hidden.

Qu, Tu and Bao, Findings of NAACL 2025, found fixed-size more efficient and reliable on real-document datasets; HiChunk, ACL 2026, supports hierarchical retrieval on evidence-dense questions but is not a direct banking-policy evaluation. Parent-child here means retrieving verbatim source text, not a RAPTOR-style tree of generated summaries.

Slide 9comparison1.5 min

No alternative reliably beats fixed-size for its cost, so measurement must justify moving from it.

Fixed-size is the reference baseline because it is cheap, predictable and easy to reproduce.
Semantic chunking must demonstrate a gain on the team's own corpus before adoption.
Hierarchical retrieval earns its complexity when relevant evidence routinely spans one child unit.
Compare retrieval outcomes and operating cost; do not choose from taxonomy or novelty.

The comparison must preserve fixed-size as the low-cost baseline without presenting it as universally best.

Speaker notes

Takeaway

Fixed-size is the cheapest defensible reference baseline, and a more complex strategy should be adopted only after it demonstrates a corpus-specific gain worth its cost.

Key Points

Fixed-size provides a cheap, predictable and reproducible baseline for comparison.
Semantic or hierarchical approaches must improve the team's own retrieval outcomes enough to justify their additional complexity and operating cost.
Controlled evidence has not established fixed-size with overlap as universally best or any alternative as consistently superior on real documents.

Prep Notes

Do not say that fixed-size with overlap is the configuration that research selected as best. The defensible claim is that controlled comparisons use fixed-size as the baseline and have not consistently shown an alternative paying for its additional cost on real documents.

The 2026 BGL BNP Paribas-linked financial-PDF study found only a small difference between plain token and its best neural chunker, while semantic scored below token chunking; keep the exact figures in reserve rather than on the slide. The decision should rest on measured retrieval outcomes and operating cost, not taxonomy or novelty.

Slide 10diagram2 min

The right chunk size is a corpus-specific window between precise matching and preserved context.

Smaller chunks improve topical precision but can fragment the evidence needed for one answer.
Larger chunks preserve context but dilute what one vector represents and add generation input.
Overlap protects some boundary cases while increasing redundant storage and retrieval noise.
The workable window moves with document structure, questions and the embedding model.

Keep the curves schematic: the figure shows a moving window, not a measured universal optimum.

Speaker notes

Takeaway

Chunk size and overlap must be tuned to the corpus because precision, context preservation and redundancy do not balance at one universal setting.

Key Points

Smaller chunks can improve topical precision but may fragment the evidence required for one answer.
Larger chunks preserve context while diluting the vector's meaning and adding generation input, and overlap protects boundaries at the cost of storage and noise.
The workable window changes with document structure, question patterns and the embedding model, so the diagram is schematic rather than a measured optimum.

Prep Notes

Read the figure as a moving trade-off window, not as a recommendation for one number. A 2025 multi-dataset preprint found different best sizes across datasets and embedding models, while the BGL BNP Paribas-linked ICSE-SEIP 2026 study found moderate overlap useful on financial PDFs; neither supports one universal setting.

If challenged on overlap, state that its 0/25/50 percent sweep used a neural chunker and that applying 25 percent to fixed-size rests on convergence with Microsoft's separate product guidance, not a direct fixed-size optimum study. Do not imply that the schematic curves represent measured values.

Slide 11concept1.5 min

Chunk-time schema must preserve provenance; only some metadata values can be back-filled without rebuilding.

Every chunk must retain a stable link to its source document, version and location.
Decide which fields require filtering, dates, jurisdiction and access classification before indexing.
Some stores let teams patch metadata values onto existing vector records.
Re-chunking, lost provenance or incompatible schema changes can still force a rebuild.

Distinguish patchable values from provenance and schema decisions that determine whether rebuilding is avoidable.

Speaker notes

Takeaway

Teams must design provenance and filterable metadata at chunk time, while distinguishing values that can be patched from structural changes that force rebuilding.

Key Points

Stable source, version and location links preserve the provenance needed to inspect and cite each chunk.
Some stores allow metadata values to be updated on existing vector records, but re-chunking, lost provenance or incompatible schema changes can still require a rebuild.
Access classification is control input rather than enforcement: Permissions become enforceable at retrieval time — if you build the enforcement.

Prep Notes

This corrects the brief's absolute statement that all metadata retrofitting requires a full reindex. Explain the operational distinction between patchable filter-only values and changes to provenance, chunk structure or index schema that alter how evidence is represented or queried.

Pinecone and Qdrant first-party documentation allowed metadata updates on existing records, while Microsoft Learn stated that changes such as making an existing Azure AI Search field filterable require a rebuild; all were retrieved 27 July 2026 and should be refreshed by 5 October 2026. Access classification is only data for a control: Permissions become enforceable at retrieval time — if you build the enforcement.

Slide 12workshop3 min

Which of your documents would break under fixed-size chunking?

Developer lens: identify where document structure carries meaning.
Infrastructure lens: identify what a re-chunk and rebuild would touch.
Risk / operations lens: identify where an omitted condition would change an accountable decision.
Produce one document type and the boundary that would make its evidence incomplete.

Keep the required prompt verbatim, show all three role lenses, and do not add trainer-supplied examples.

Speaker notes

Takeaway

Participants should identify one document they own and the structural boundary that could make its retrieved evidence incomplete.

Key Points

Developers should locate where headings, tables, clauses or layout carry meaning across a possible cut.
Infrastructure should identify the records, embeddings, index state and validation work that re-chunking would affect.
Risk and operations should identify where losing a condition, exception or effective version would change an accountable decision.

Prep Notes

Read the required prompt verbatim, give tables two minutes and protect one minute for report-out from exactly two tables. Ask each table for one document type and one break pattern; participants may describe its structure without sharing confidential content. Use the three role lenses so the discussion does not default to developers.

Do not rescue a quiet room with an example; the value is participants naming material they own, not repeating the synthetic policy. Do not debate chunk sizes, products or solution designs, and do not introduce printed material or participant tool use. Capture only the document type and the boundary that would make its evidence incomplete, then move directly to the default.

Slide 13transition1.5 min

Start fixed-size with overlap; move only when typical answers do not fit inside one chunk.

Use fixed-size with overlap as the cheapest defensible baseline.
Move to hierarchical retrieval when relevant evidence routinely spans a child chunk.
Adopt semantic chunking only after it shows a measured corpus-specific gain.
Once the evidence units are fixed, an embedding model must define which units count as similar.

The routing question decides when to leave the baseline; the next component decides similarity.

Speaker notes

Takeaway

Begin with fixed-size and overlap, then move only when measured evidence shows that typical answers cannot be preserved inside one chunk.

Key Points

Fixed-size with overlap is the cheapest defensible baseline rather than a universally best configuration.
Hierarchical retrieval fits evidence that routinely spans a child chunk, while semantic chunking requires a measured corpus-specific gain.
Once evidence units are fixed, the next decision is which embedding model will define their similarity.

Prep Notes

Close on the decision, then hand off with the exact line: Chunks still require a model to define similarity. The deciding question is supported by HiChunk's evidence-density finding, but the policy-specific application remains an inference rather than a direct banking-policy experiment.

Reinforce that overlap and parent return mitigate boundary failures at chunk time; do not repeat the disowned absolute claim that nothing downstream can ever recover a split. Keep the default bounded: move away from fixed-size with overlap only when typical answers do not fit inside one chunk or corpus-specific measurement demonstrates a better option.

Framework 2 · E01

Built activity

E01

Type

table talk

Owner

Pradeep Menon

Ready By

5 October 2026

Purpose

Make chunking concrete for a greenfield cohort by having participants identify a failure boundary in documents they already own.

Prompt

Which of your documents would break under fixed-size chunking?

Role Lenses

Developer	Identify where headings, tables, clauses or layout carry meaning that a regular token boundary would not see.
Infrastructure	Identify what records, embeddings, index state and validation work a re-chunk would force the team to rebuild.
Risk Operations	Identify which document carries an audit obligation and where losing a condition, exception or effective version would change the accountable decision.

Timing

Total	3 min
Table Discussion	2 min
Report Out	1 min

Instructions

Read the required prompt verbatim from the slide.
Ask each table to name one document type and one boundary that would make its evidence incomplete.
Use the three role lenses to stop the discussion defaulting to developers.
After two minutes, hear one concise response from exactly two tables and move on.

Output

One document type per reporting table and the specific structural boundary that would make a retrieved chunk incomplete.

Report Out

Two tables only; each gives the document type and break boundary in one sentence, with no plenary solution design.

Ground Rules

No printed material and no participant tool use.
The trainer does not supply examples, even if the room is initially quiet.
Participants may describe document structure without sharing confidential document content.
Do not debate chunk sizes or products; this exercise identifies failure shapes only.

Acceptance Criteria

The prompt appears verbatim on its own workshop slide.
Developer, infrastructure and risk / operations lenses appear on the slide.
The speaker note states two minutes at table, one minute report-out and a two-table cap.
The speaker note explicitly instructs the trainer not to provide examples.
At least one reported answer names both a document type and the boundary that could make its evidence incomplete.

T3 · 9 minutes · slides 14–18

Embedding Models

Slide 14question1 min

Which would retrieve Japanese financial text better: the larger general model or the smaller specialist?

Predict before the evidence is shown.
Assume both models were built for retrieval, not generation.

Keep both choices at equal visual weight and do not reveal a winner before the benchmark slide.

Speaker notes

Takeaway

Model size and specialisation cannot predict retrieval quality without evidence from the corpus being served.

Key Points

The opening prediction exposes the assumptions participants use when judging embedding models.
A defensible choice requires corpus-matched evidence rather than a debate about model names.
Technical fit, operating fit and evidential confidence must all be represented in the decision.

Prep Notes

Ask everyone to make a silent prediction and hold the room for about 30 seconds before sampling one rationale from each side; do not invite a model-name debate. The answer being steered toward is that neither size nor specialisation settles the result without corpus-matched evidence. Ask developers for the technical instinct, infrastructure for the operating instinct, and risk/operations for the evidence they would require.

Slide 15diagram1.5 min

An embedding model defines which passages count as similar; the corpus itself does not.

The model maps each query and chunk into its learned neighbourhood.
Two models can place the same passage beside different neighbours.
Retrieval inherits that judgement before generation sees any evidence.

The documents must stay fixed while the model changes the neighbourhood; do not turn this into a formula.

Speaker notes

Takeaway

An embedding model supplies the learned judgement of similarity that determines which evidence retrieval can surface.

Key Points

The model maps queries and chunks into a neighbourhood rather than discovering similarity inherent in the text.
Queries and chunks must use the same model version for their vector representations to be comparable.
A different embedding model can change the retrieved evidence even when the corpus is unchanged.

Prep Notes

Use the diagram as the complete no-maths definition: the embedding model decides what the system treats as similar. For developers, the operational rule is that queries and chunks must use the same model version; for risk/operations, similarity is a learned judgement rather than an intrinsic property of the text. Keep the explanation to one pass because the topic does not revisit vector mechanics.

Slide 16data2.5 min

General Japanese benchmarks favour specialised models, while Japanese financial retrieval nearly closes the gap.

JMTEB general Japanese retrieval: 78.1 specialist versus 66.4 general.
JFinTEB JaFIn financial retrieval: 86.3 specialist versus 84.3 general.
The panels use different models and benchmarks; compare only within each panel.
Your corpus decides; a leaderboard rank does not.

Keep the two benchmarks in separate equal-width panels, print both publication dates, and never encode the cross-benchmark comparison as one shared scale.

Speaker notes

Takeaway

Benchmark results show that specialisation can matter, but only the bank's own golden set can settle the model choice.

Key Points

JMTEB shows a material specialist advantage on general Japanese retrieval within that benchmark.
JFinTEB shows a much narrower gap on Japanese financial retrieval within its separate benchmark.
The two panels differ in models, benchmarks and provenance, so neither supports a cross-panel score comparison or replaces corpus-specific testing.

Prep Notes

The JMTEB values are text-embedding-3-small 66.39 and Ruri-v3-30m 78.08, published by the Ruri authors on an independently maintained benchmark in April 2025; retrieved 27 July 2026 and refresh by 5 October 2026. The JFinTEB JaFIn values are OpenAI-large 84.3 and Sarashina 86.3 from Suzuki and Sakaji, SIGIR 2026; retrieved 27 July 2026. State the provenance asymmetry aloud: the first table is author-reported, the second is peer-reviewed, and neither substitutes for a golden set drawn from the bank's own corpus.

Slide 17framework2.5 min

Six gates should decide an embedding model before a leaderboard rank does.

Language and Domain test whether the model understands the corpus you own.
Length and Width test whether chunks fit the model and vectors fit the operating store.
Hosting and Cost test whether the deployment is controllable and sustainable.
Default to one strong general model that clears all six; move only on corpus evidence.

Retain the name The 6 Gates Before You Index, the exact anchor order Language · Domain · Length · Width · Hosting · Cost, and the rule that evidence—not rank—opens the decision.

Speaker notes

Takeaway

Embedding selection is an architectural decision that must clear six corpus, compatibility and operating gates before benchmark rank matters.

Key Points

Language and Domain test whether the model represents the vocabulary and meaning present in the corpus.
Length and Width constrain compatibility with chunking, the vector index and the memory plan.
Hosting and Cost determine whether the chosen model can be governed, versioned and sustained in operation.

Prep Notes

Teach The 6 Gates Before You Index in order: Language, Domain, Length, Width, Hosting, Cost; the deciding question is whether the corpus carries language or domain vocabulary a general model mishandles. Length must exceed the chunk size, while Width must fit the index and memory plan; for example, pgvector's standard vector indexes support up to 2,000 dimensions and halfvec up to 4,000 in the pgvector README retrieved 27 July 2026, so re-verify before delivery. Hosting carries licence, residency and version-pinning for risk/operations, while Cost covers both the one-time corpus build and per-query embedding path.

Slide 18transition1.5 min

Assume an embedding change forces a corpus rebuild; budget the migration before choosing the index that stores it.

Version the embedding model as an architectural dependency.
Retain source text so every document can be re-encoded.
Name the owner, validation path and cutover plan before the corpus grows.

Show model replacement as re-encode → rebuild → validate → cut over; never present the old and new vector spaces as directly compatible.

Speaker notes

Takeaway

Teams should plan an embedding-model change as a corpus migration with explicit rebuild, validation and cutover ownership.

Key Points

The documented operating default is to re-encode the corpus and rebuild the ANN index when the embedding model changes.
Infrastructure, developers and risk/operations own different parts of capacity, reproducibility, validation, cutover and continuity evidence.
Research adapters are an exception rather than a standard production migration path, so the receiving vector store must support the planned transition.

Prep Notes

Treat a full re-encode and ANN index rebuild as the documented operating default, not an immutable law; Drift-Adapter, EMNLP 2025, reports a research bridge between spaces but not a standard production migration, retrieved 27 July 2026. Infrastructure owns capacity and cutover, developers own reproducibility and validation, and risk/operations owns continuity evidence. This hands the session to T4: embeddings now need an operational home whose residency, filtering and on-call model can carry that migration.

T4 · 12 minutes · slides 19–25

Vector Databases

Slide 19question1 min

If restricted content appears in retrieval, which component was supposed to stop it?

Assign one control owner before the architecture is revealed.
Distinguish identity, retrieval orchestration and storage.
A namespace or collection name is not yet an entitlement decision.

Keep all three possible owners visually equal and leave the control owner unresolved until the room commits.

Speaker notes

Takeaway

Preventing restricted retrieval is a control path shared by identity, retrieval orchestration and storage rather than the automatic duty of one component.

Key Points

Identity supplies the entitlement that determines what the caller is allowed to access.
The application enforces the entitlement for most stores, while PostgreSQL can enforce row policies in the data layer when Row Level Security is correctly configured.
Namespaces and collection names can isolate scope, but they do not make the authorization decision or authenticate a caller by themselves.

Prep Notes

Ask the room to choose one control owner, then hold the silence for about 30 seconds before taking one answer from a developer, one from infrastructure and one from risk/operations. Keep the three candidate owners visually equal, record the competing answers and do not resolve the architecture while participants are committing.

The direction is that there is no universal box that stops restricted retrieval on its own: identity supplies the entitlement, the application enforces it for most stores, and PostgreSQL can enforce row policies in the data layer when Row Level Security is correctly configured. Return to these answers on the permissions slide, where the full control path is made explicit.

Slide 20concept1.5 min

A production vector store must filter and isolate results around approximate search, not merely rank neighbours.

Approximate search ranks likely neighbours; business rules decide which neighbours are eligible.
Metadata filters constrain the candidate set before, during or after the index scan.
Namespaces, tenants and collections reduce cross-scope retrieval but do not authenticate a caller by themselves.
Refresh state determines whether the ranked evidence still represents the corpus.

Keep approximate ranking in the centre and show filtering, isolation and refresh as the operating contract around it.

Speaker notes

Takeaway

A production vector store must combine approximate ranking with explicit controls for eligibility, isolation and freshness.

Key Points

Approximate nearest-neighbour search ranks likely matches, while metadata and business rules determine which candidates are eligible.
Tenant, namespace or collection isolation reduces retrieval scope but does not authenticate the caller or replace per-chunk authorization checks.
Refresh state is part of the retrieval contract because correctly ranked evidence can still be stale.

Prep Notes

Read the visual from the ranking engine in the centre to the operating contract around it. The point is to move the definition beyond nearest-neighbour search: production retrieval combines ranking with eligibility, scope and freshness. Pan, Wang and Li's peer-reviewed vector-DBMS survey supplies the non-vendor filter taxonomy; the OWASP RAG Security Cheat Sheet requires per-chunk access metadata and retrieval-time checks, both reviewed on 3 August 2026.

For this greenfield room, the ownership test matters more than index internals. Developers carry metadata and policy context, infrastructure owns searchable state and refresh, and risk/operations defines the access boundary that must hold. Do not draw namespaces, tenants or collections as if they authenticate a caller by themselves.

Slide 21comparison2.5 min

The store you already operate is the default until residency, scale or filtering proves it inadequate.

Pinecone is managed: the provider runs the database while your team owns ingestion, authorization context and residency sign-off.
Weaviate runs as a hosted service or a self-managed instance; the chosen deployment decides who operates the cluster.
Chroma spans local, single-node, distributed and cloud production shapes; deployment mode, not a prototype label, sets the operating burden.
pgvector keeps retrieval in PostgreSQL, preserving the DBA, monitoring and policy path you already run.

Render all four stores at identical visual weight; position them only by operating shape and never imply a rank or score.

Speaker notes

Takeaway

The defensible default is the vector store the organisation can already operate until a measured residency, scale or filtering requirement defeats it.

Key Points

Pinecone is managed, Weaviate supports cloud and self-managed routes, Chroma spans local through cloud production shapes, and pgvector keeps retrieval inside PostgreSQL.
Deployment shape determines who owns ingestion, authorization context, backup, monitoring, audit, cluster operations and service recovery.
A listed public region, tenant boundary or unsourced vector-count threshold is not sufficient evidence for residency, authorization or migration decisions.

Prep Notes

Present these as operating shapes at equal visual weight, not as a procurement shortlist or ranking. Pinecone's current docs describe managed deployment, Weaviate documents cloud and self-managed routes, Chroma documents local, single-node and distributed modes plus Chroma Cloud, and pgvector is a PostgreSQL extension; all were rechecked on 3 August 2026. Explain that each shape moves operating responsibility differently: the provider may run a managed database, but the organisation still owns ingestion, authorization context and residency sign-off, while a self-managed route leaves the organisation operating the cluster.

Pinecone's public region table currently lists Singapore but not Tokyo and states that cloud and region cannot be changed after index creation, so a public region is not a substitute for contractual residency sign-off. Chroma's current architecture page also documents tenant-level isolation, but that is not evidence of per-user, per-document authorization across every deployment mode. The default is the system the organisation can already back up, monitor, audit and staff; move only when a measured operating condition defeats it, never at an unsourced vector-count threshold.

Slide 22diagram2 min

Filter placement changes both recall and confidentiality, so post-filtering is not harmless.

Pre-filtering restricts candidates first, but a selective predicate can disrupt graph traversal.
Single-stage filtering checks eligibility during the scan and may do more work to fill the result set.
Post-filtering can return fewer results because disallowed candidates consumed the shortlist.
It becomes a confidentiality incident only if restricted candidates or scores cross the control boundary.

Keep the recall failure and confidentiality failure separate; never show unsourced full-result counts for pre-filter or in-scan filtering.

Speaker notes

Takeaway

Filter placement changes search behaviour, but a thin result and a confidentiality breach are different failure modes.

Key Points

Pre-filtering, single-stage filtering and post-filtering impose different search costs, so pre-filtering should not be presented as universally free.
Post-scan filtering can reduce recall because ineligible candidates may consume the approximate-search shortlist before the filter is applied.
A confidentiality incident occurs only when restricted candidates or similarity information cross the control boundary, not merely when filtering produces fewer results.

Prep Notes

Read the figure as three distinct placements and keep the recall failure separate from the confidentiality failure. The three placements come from Pan, Wang and Li, The VLDB Journal 2024; each has a different search cost, so do not present pre-filtering as universally free. pgvector's README, rechecked 3 August 2026, documents the recall case: with approximate indexes the filter is applied after the scan, and a condition matching ten percent of rows with the default HNSW search setting yields four matches on average until the scan is widened. Do not add unsourced full-result counts for pre-filter or in-scan filtering.

The confidentiality case is different: OWASP prefers pre-retrieval enforcement because restricted similarity information must not become observable. A PostgreSQL WHERE clause or RLS policy can exclude rows correctly while the answer still becomes thin, so reduced recall is not by itself proof of a security incident. Label this separation as Chartered Vectorial's synthesis of the two sources, not as a vendor claim.

Slide 23diagram2 min

The corpus changes while the index stays still, so every deployment needs a named refresh owner.

Track source version, ingestion watermark, index build and validation state as separate facts.
Use incremental writes and deletes for ordinary corpus change; verify that every change reached the searchable state.
Re-chunking, embedding changes and some schema or index changes can require a controlled rebuild.
Name the owner, cadence, validation gate, cutover and rollback before launch.

The named owner must close the visible gap between changing source state and searchable index state; do not imply every metadata update requires a rebuild.

Speaker notes

Takeaway

A searchable index can silently diverge from its changing corpus unless a named owner controls refresh, validation, cutover and rollback.

Key Points

Source version, ingestion watermark, index build and validation state are separate facts that must be tracked independently.
Ordinary writes, deletes and patchable filter metadata may be updated incrementally, while re-chunking, embedding changes and some schema or index changes can require a controlled rebuild.
Product-specific rebuild behaviour demonstrates the need for ownership but does not support a universal refresh duration, cost or cross-product ranking.

Prep Notes

Use the figure to show the gap that can open between changing source state and searchable index state, then place the named owner across that gap. The owner needs a cadence, validation gate, cutover and rollback, and must verify that each incremental write or delete reached the searchable state. Do not imply that every metadata update requires a rebuild: distinguish schema or embedded metadata changes from patchable filter-only values.

No universal refresh duration or cost is asserted because the evidence pack found none for the four named stores. PostgreSQL documents the operational trade: REINDEX blocks writes, while REINDEX CONCURRENTLY permits writes but requires more work and takes longer; Weaviate documents that changing an existing vectorized property re-vectorizes an object, while adding a new property does not backfill existing objects, both rechecked 3 August 2026. Use those examples only to establish ownership, not to compare products, and flag this as a forward link to the later stale-index demonstration without revealing that act's answer or outcome.

Slide 24concept2 min

Among these four operating shapes, PostgreSQL is the database-enforced row-policy path; the others require application authorization.

Permissions become enforceable at retrieval time — if you build the enforcement.
Identity and entitlement data must travel with the query and with every stored chunk.
For most stores, the application authorizes the request and supplies the permitted candidate filter.
PostgreSQL Row Level Security can enforce rows in the database, but only when roles and policies are configured correctly.

Keep the shared authorization decision upstream, then distinguish application-enforced filtering from PostgreSQL row-policy enforcement without implying either is automatic.

Speaker notes

Takeaway

Retrieval-time authorization is an engineered control path, with PostgreSQL RLS offering the reviewed database-enforced row-policy option among these four shapes.

Key Points

Identity and entitlement data must accompany both the query and every stored chunk so the authorization decision has usable context.
Most reviewed store shapes rely on application authorization and permitted-candidate filters, while correctly configured PostgreSQL RLS can enforce row access in the database.
Neither path is automatic: scope isolation is not per-document authorization, and PostgreSQL table owners or BYPASSRLS roles can bypass row policies.

Prep Notes

Read the permission path from the shared authorization decision upstream, then distinguish application-enforced filtering from database-enforced row policies without implying that either happens automatically. Pinecone's January 2026 access-control guidance integrates an external authorization system and applies pre- or post-retrieval checks; current Weaviate and Chroma documentation describes platform, tenant or collection scopes rather than a native per-human, per-document row policy equivalent to PostgreSQL RLS, all rechecked 3 August 2026.

PostgreSQL's current documentation states that RLS can restrict returned rows per user and defaults to deny when enabled without a matching policy, but table owners and BYPASSRLS roles can bypass it. Therefore use the exact session wording: Permissions become enforceable at retrieval time — if you build the enforcement. Full authorization design, role mapping, bypass analysis and assurance belong to M07.

Slide 25transition1 min

A store can return results quickly without returning the right evidence, making retrieval quality the next design problem.

Operability decides where vectors live.
Filtering and isolation decide which evidence is eligible.
Refresh ownership decides whether that evidence is current.
None of those proves that the relevant passage was retrieved.

Show speed, eligibility and freshness as passed gates, then leave relevance as the unresolved gate handed to T5.

Speaker notes

Takeaway

Operability, eligibility and freshness are necessary retrieval gates, but none proves that the returned evidence is relevant.

Key Points

Use the store the organisation can already operate unless a signed-off residency requirement or measured scale or filtering demand defeats it.
Developers own query and metadata behaviour, infrastructure owns searchable state, and risk/operations owns authorization and residency gates.
After those gates pass, retrieval quality remains the unresolved question because fast, eligible and current results can still be wrong for the query.

Prep Notes

Close the topic with the default in one breath: use the store you can already operate; consider managed when time-to-production dominates and residency is signed off; consider a specialist only when measured scale or filtering demands exceed the current store. Point to speed, eligibility and freshness as passed gates, then leave relevance visibly unresolved.

Reinforce the ownership handoff: developers leave owning query and metadata behaviour, infrastructure leaves owning searchable state, and risk/operations leaves owning the authorization and residency gates. The unresolved question is whether eligible, current results contain the right evidence, which hands directly to T5's retrieval-quality design.

T5 · 13 minutes · slides 26–32

Retrieval Quality

Slide 26question1 min

Which retriever would find product code AC-417: one that understands meaning or one that matches characters?

Choose before the result is shown.
Assume the code appears verbatim in one document.

Keep Meaning and Characters at equal visual weight so the room must choose before sparse retrieval is revealed.

Speaker notes

Takeaway

Exact identifiers favour character matching, while meaning-based retrieval remains valuable when wording changes but intent does not.

Key Points

A verbatim product code creates an exact-match retrieval problem rather than a semantic-paraphrase problem.
The retriever choice affects development, a possible second index and query path, and the test corpus used to substantiate quality.
Dense and sparse retrieval address different query shapes, so neither should be presented as universally superior.

Prep Notes

Pause for about 30 seconds, then ask for one vote from each role before revealing the mechanism. Keep Meaning and Characters at equal visual weight until the vote, and steer toward the character-matching retriever for an exact code while preserving that dense retrieval is the better route when wording changes but the meaning does not.

Use the role responses to surface ownership: the developer chooses the retrieval mechanism, infrastructure anticipates the second index and query path, and risk/operations asks whether exact identifiers are represented in the test corpus. Do not disclose the answer before each role has committed to a choice.

Slide 27case1.5 min

Dense-only retrieval can miss exact identifiers; sparse matching covers that failure mode when literal strings matter.

Dense retrieval groups passages by semantic similarity.
Sparse retrieval preserves literal codes, dates, names and specialised jargon.
The two signals are complementary rather than substitutes.
The illustration is a mechanism, not a disclosed banking incident.

Retain the ILLUSTRATIVE label and use muted red only for the missed exact identifier; never present this as a real bank incident.

Speaker notes

Takeaway

Sparse matching complements dense retrieval by preserving literal signals such as codes, dates, names and specialised jargon.

Key Points

Dense retrieval groups passages by semantic similarity, whereas sparse retrieval preserves exact character matches.
Combining the signals covers different failure modes without treating either retriever as a universal substitute for the other.
AC-417 is an illustrative invented identifier, not a disclosed banking incident or a D01 example.

Prep Notes

Microsoft Learn's Azure AI Search hybrid overview, updated 19 February 2026 and retrieved 27 July 2026, identifies product codes, dates, names and specialised jargon as cases where keyword search can outperform vector search. Use the visual to contrast the dense semantic match with the sparse exact-string match, retaining the ILLUSTRATIVE label and using muted red only for the missed identifier.

AC-417 is an invented identifier used only for this teaching case; it is not a disclosed bank incident and does not appear in D01. Use the case to establish complementarity, not to disparage dense retrieval or imply that every exact string will be missed.

Slide 28framework2.5 min

The 3-Rung Retrieval Ladder adds sparse search before paying the recurring cost of re-ranking.

Dense finds meaning but exposes an exact-match gap.
Fuse combines dense and sparse lists as the first low-cost improvement.
Re-rank re-scores a shortlist only when measured quality justifies a model call.
Move one rung at a time against the same Golden set.

Retain the name The 3-Rung Retrieval Ladder, the anchor order Dense · Fuse · Re-rank, and the dashed optional treatment on Re-rank.

Speaker notes

Takeaway

Improve retrieval one measured rung at a time by adding fusion before accepting the recurring cost and boundary of re-ranking.

Key Points

Dense retrieval establishes the semantic baseline, Fuse adds sparse coverage, and optional Re-rank reorders the resulting shortlist.
Fusion is the first low-cost improvement because dense and sparse paths can run in parallel without another model inference.
Every rung must be evaluated against the same Golden set, and fusion should not be claimed as the largest raw quality gain.

Prep Notes

Teach the anchors exactly as Dense, Fuse, Re-rank, and read the ladder from the baseline upward. Preserve the dashed optional treatment on Re-rank: it is a measured escalation, not a mandatory architectural component. Fusion is the first low-cost improvement because dense and sparse retrieval can run in parallel and no additional model inference is required; do not call it the largest raw quality gain.

Reciprocal Rank Fusion is a defensible greenfield default because it needs no tuning data, while a tuned weighting becomes testable once the Golden set exists; Cormack et al., SIGIR 2009, and Bruch et al., ACM TOIS 2023, retrieved 27 July 2026.

Slide 29concept1.5 min

Re-ranking can improve the shortlist but cannot recover a relevant chunk that was never retrieved.

The retriever decides which candidates enter the shortlist.
The re-ranker changes only their order.
A missing candidate remains missing.
Choose shortlist depth from the Golden set, not a universal threshold.

Keep corpus membership outside the dashed Re-rank boundary; the re-ranker may reorder candidates but never add a missed chunk.

Speaker notes

Takeaway

Re-ranking can improve candidate order only after retrieval has supplied an adequate shortlist, because it cannot add a missing passage.

Key Points

The retriever determines shortlist membership, while the re-ranker changes only the order of existing candidates.
Shortlist depth and any claimed improvement must be tested on the fixed Golden set rather than chosen from a universal threshold.
Re-ranking is optional and adds query-path cost without repairing inadequate recall.

Prep Notes

Read the visual boundary literally: corpus membership and missed passages stay outside the dashed Re-rank layer. Re-ranking is optional rather than an architectural requirement, but it can improve ordering once recall is adequate. Microsoft Learn's semantic-ranking overview, updated 24 April 2026 and retrieved 27 July 2026, states the structural limit directly: its ranker works on the existing result set and cannot rerun the query over the corpus.

Make the ownership explicit. Infrastructure owns the added query-path cost, developers own candidate-depth testing, and risk/operations should require the claimed improvement to survive on the fixed Golden set.

Slide 30framework2.5 min

The three retrieval metrics answer different questions, so no single score proves quality.

Found? Recall@k asks whether agreed evidence entered the top k.
Clean? Precision@k asks how much returned material is relevant.
Early? MRR asks how high the first relevant result appears.
Use the same Golden set to compare versions, not to predict answer quality.

Retain the name The 3 Retrieval Questions and the anchor mapping Found? · Clean? · Early? to Recall@k · Precision@k · MRR.

Speaker notes

Takeaway

Recall, precision and reciprocal rank reveal different retrieval behaviours, so the team needs all three questions against one collaboratively authored Golden set.

Key Points

Recall@k asks whether agreed evidence was found, Precision@k asks whether returned material was clean, and MRR asks whether the first relevant result appeared early.
The Golden set is a fixed collection of representative questions and agreed answers written with domain owners, not by the build team alone.
These retrieval measures compare versions; no single score proves downstream answer quality or establishes a universal Golden-set size.

Prep Notes

Teach the anchors as Found?, Clean?, Early? and map them directly to Recall@k, Precision@k and MRR. The Golden set is the fixed set of representative questions and agreed answers, written with domain owners rather than by the build team alone. Manning, Raghavan and Schütze define precision and recall in Introduction to Information Retrieval (2008), while Craswell defines MRR in the Encyclopedia of Database Systems; sources retrieved 27 July 2026.

If size comes up, qualify the ARES finding precisely: approximately 150 human-annotated datapoints was a floor for distinguishing whole RAG systems with that framework, not a universal minimum for stable retrieval metrics; ARES, NAACL 2024, retrieved 27 July 2026.

Slide 31diagram2 min

Retrieve the chunk manually. Was the answer in it? If yes, generation failed. If no, retrieval failed.

Start with the evidence the system actually handed downstream.
Use the split as rapid triage, not as a final attribution.
If the answer was present, inspect context assembly and retrieval noise before blaming the model.

Preserve the locked binary as the main path and the dashed context-assembly/noise caveat as a subordinate third case.

Speaker notes

Takeaway

Manual inspection of the retrieved chunk is a rapid first triage that separates likely retrieval failure from likely generation failure without claiming final attribution.

Key Points

Begin with the evidence the system actually handed downstream and ask whether the answer was present in it.
If the answer was absent, investigate retrieval; if it was present, investigate generation while also checking context assembly and retrieval noise.
The binary is a first cut rather than a verdict, and this slide must not reveal D01's fixed queries or outcomes.

Prep Notes

Run the locked wording exactly as written and let the room answer the branch before explaining the dashed caveat: Retrieve the chunk manually. Was the answer in it? If yes, generation failed. If no, retrieval failed. Read the main path first so the method remains memorable, then qualify it as rapid triage rather than final attribution.

Barnett et al., CAIN 2024, identify a third case where retrieved evidence never reaches the model context, while RAGChecker, NeurIPS 2024, shows that retrieval noise can still be causal when the answer is present; both retrieved 27 July 2026. Keep the example generic here so Topic 7 can reveal its own fixed query and baseline outcomes during D01.

Slide 32transition2 min

Measure one change at a time; reach beyond the fixed pipeline only after the failure stays measurable.

Freeze the Golden set and baseline configuration.
Change one retrieval variable.
Re-run the same questions and keep or revert the change.
Escalate only when the same failure persists.

Keep the optional escalation gate after baseline, one change and re-measure; advanced patterns must not appear as a maturity destination.

Speaker notes

Takeaway

A team should escalate beyond a fixed RAG pipeline only after an isolated change and repeatable measurement show that the same failure remains unresolved.

Key Points

Freeze the Golden set and baseline so each retrieval change can be compared against the same evidence.
Change one variable, rerun the same questions and keep or revert the change according to the result.
Advanced patterns require a measured persistent failure, not curiosity, novelty or a vendor recommendation.

Prep Notes

Read the loop in order: baseline, one change, re-measure, then the optional escalation gate. Wang et al., EMNLP 2024, tested one method at a time while holding other RAG modules unchanged; Google Cloud's evaluation guidance published 19 December 2024 states the same discipline, both retrieved 27 July 2026.

Developers own the isolated change, infrastructure keeps the environment comparable, and risk/operations preserves the dated evidence trail. This establishes why T6 begins with a measured unresolved failure rather than curiosity, novelty or a vendor deck; advanced patterns must not be presented as a maturity destination.

T6 · 2 minutes · slides 33–34

Beyond Naive RAG

Slide 33question1 min

Should a greenfield team start with agentic RAG because it can search again?

Agentic RAG changes control flow: the model decides whether to retrieve, search again or stop. M03 builds this different design.
Graph RAG changes retrieval structure: relationships become retrievable across documents.
RAFT changes the model: fine-tuning teaches it to use relevant context and ignore distractors. M02.2 owns the fine-tuning depth.

Keep the three patterns at equal visual weight, with no arrows, steps or ordering between them; they change control flow, retrieval structure and the model respectively.

Speaker notes

Takeaway

Agentic RAG, graph RAG and RAFT are three distinct design changes, not successive stages of RAG maturity.

Key Points

Agentic RAG changes control flow, graph RAG changes retrieval structure, and RAFT changes the model.
A greenfield team should not choose agentic RAG merely because it can search repeatedly.
Each pattern changes technical failures, operational demands and the variance that teams must keep diagnosable.

Prep Notes

Pause for five seconds, then withdraw the provocation: no, a greenfield team should not start agentic merely because it can search again. The three patterns are different designs rather than maturity stages: agentic changes control flow, graph RAG changes retrieval structure, and RAFT changes the model. Developers should notice the changed failure surface, infrastructure the added operating surface, and risk/operations the extra variance that must remain diagnosable.

Slide 34transition1 min

Advanced patterns change different failure surfaces; measurement must identify the unresolved failure before escalation.

Start with the tuned fixed pipeline and its golden set.
Locate the failed component before changing the architecture.
Escalate one measured failure surface, then re-measure.
Chartered Vectorial recommendation.

The diagnosis gate must be the only jade element; the advanced-pattern box stays dashed and downstream, and the figure must point into T7 rather than imply an upgrade path.

Speaker notes

Takeaway

Teams should escalate beyond a tuned fixed pipeline only after measurement identifies the unresolved failure surface.

Key Points

Begin with a tuned fixed pipeline and use its golden set to locate the failed component.
Change one measured failure surface at a time, then re-measure before adding further complexity.
This diagnosis-first escalation rule is a Chartered Vectorial recommendation rather than a published industry standard.

Prep Notes

This escalation rule is a Chartered Vectorial recommendation, not a published industry standard; the evidence supports it because none of the advanced patterns is uniformly better. Developers change one surface at a time, infrastructure names the added cost and owner, and risk/operations keeps the failure attributable. Hand directly to T7: the demo now gives the room four chances to locate the broken component before choosing any mitigation.

T7 · 28 minutes · slides 35–39

The Demo: Break It Four Ways, plus Closing

Slide 35question1 min

One pipeline will fail four ways; can you locate each broken component before the reveal?

One illustrative synthetic corpus, one pipeline and one projected interface.
Four fixed query/baseline pairs; one deliberate break switch per act.
Capture-and-replay is the default; a live path remains available as proof.
There is no correct opening run: the reference is earned inside each act.

Keep the four component answers hidden on this surface; the room must diagnose each act before any stage is highlighted.

Speaker notes

Takeaway

The room must diagnose each failure from the answer and evidence trace before its originating component is revealed.

Key Points

All four acts use one illustrative synthetic corpus, pipeline and interface, with a different locked query and deliberate break for each act.
Diagnosis depends on comparing the answer with the evidence handed downstream rather than judging the answer's fluency.
These are four diagnosable failure modes, not a complete or prevalence-ranked taxonomy of RAG failures.

Prep Notes

Pause for about 20 seconds after the question and steer toward locating the failed component by comparing the answer with the evidence handed downstream, not by judging fluency. Say before the first act: Four failure modes, not the four. They were selected because four different component origins can be diagnosed in the room, not because they are complete or ranked by prevalence. Preserve the challenge by keeping all four component answers hidden and by not running a correct opening example before Act 1.

Slide 36workshop20 min

Each act earns its diagnosis only when the bad answer and retrieved evidence are shown side by side.

Run one fixed query/baseline pair through one deliberate break.
Hold about 60 seconds for the room to name the failed component.
Show the retrieved evidence beside the agreed evidence or known absence.
Reveal the origin and one mitigation, place it on V01, then reset.

Preserve the five beats and the side-by-side answer/evidence panes; trainer commentary is expendable, but the diagnosis pause and manual evidence check are not.

Speaker notes

Takeaway

Each demo act becomes diagnosable only when participants inspect the bad answer against the actual and agreed evidence before the reveal.

Key Points

Run four acts at roughly five minutes each through the same interface and protect about 60 seconds for diagnosis in every act.
Developers inspect configuration and traces, infrastructure checks index and execution state, and risk/operations tests the agreed evidence baseline.
Manual evidence inspection and reveal discipline are mandatory; trainer commentary is the first cut if time is lost.

Prep Notes

Run four acts at roughly five minutes each through the same interface, revealing one row of the board at a time. Developers inspect the changed configuration and trace, infrastructure checks the versioned index and replay/live path, and risk/operations tests the agreed baseline, effective date or justified abstention. Do not answer the room's question, do not describe every wrong answer as hallucination, and do not skip the manual evidence check when time is tight. Be explicit during preparation that D01's delivery-asset specification is complete for content approval, but the demo is not built: engineering ownership, implementation, trace acceptance, offline packaging and rehearsal remain delivery-stage work.

Slide 37diagram3 min

Four failures map to four components, but they are a diagnosable subset rather than a complete taxonomy.

Act 1: a governing exception is severed at Chunk.
Act 2: dense-only retrieval misses an exact identifier at Retrieve.
Act 3: a superseded policy remains answerable at Index.
Act 4: the corpus gap begins at Ingest; the failure to say so occurs at Generate.

Keep the four modes inside a dashed subset boundary and retain the two-stage Act 4 label; do not turn this into a complete or frequency-ranked taxonomy.

Speaker notes

Takeaway

The four staged failures demonstrate distinct diagnostic signatures without defining the full set of possible RAG failures.

Key Points

The staged origins are Chunk, Retrieve, Index and Generate because the room can distinguish their signatures.
Act 4 requires a two-stage attribution: the evidence gap begins at Ingest, while the failure to state the absence occurs at Generate.
Ingest, Embed and Re-rank remain legitimate failure surfaces, and index drift may be named without inventing a fifth act.

Prep Notes

Four failure modes, not the four. Chunk, Retrieve, Index and Generate are staged because they produce distinct signatures that a room can diagnose; Ingest, Embed and Re-rank remain legitimate failure surfaces even though they are not separate acts. Name index drift at Act 3 as stale data's systemic cousin, but do not stage or imply a fifth act.

Slide 38diagram2 min

The same fluent wrong answer can originate at Chunk, Retrieve, Index or Generate; the manual check separates them.

Chunk, Retrieve and Index can hand incomplete, wrong or superseded evidence downstream.
Generate can still fail when the agreed evidence is present.
Context assembly or retrieval noise can create a third case, so the check is first-cut triage.
Retrieval quality is the ceiling on answer quality.

V04 must retain V01's stage positions, build-once/run-per-question split, optional Re-rank treatment and Act 4's Ingest-gap/Generate-abstention distinction.

Speaker notes

Takeaway

Manual inspection of retrieved evidence is the fastest first-cut test for separating retrieval failure from generation failure.

Key Points

If the answer is present in the retrieved chunk, begin by investigating generation; if it is absent, begin with retrieval.
Context assembly and noisy retrieved evidence can create a third case, so this binary is triage rather than final attribution.
Retrieval quality is the ceiling on answer quality, but good retrieval remains necessary rather than sufficient.

Prep Notes

Retrieve the chunk manually. Was the answer in it? If yes, generation failed. If no, retrieval failed. Treat that binary as rapid triage: context assembly or noisy retrieved context can create a third case before final attribution. Good retrieval is necessary, not sufficient; a model can still fail with good evidence and can sometimes answer correctly from insufficient evidence.

Slide 39transition2 min

When retrieval is measured and still short, the next decision is whether to change the model rather than the pipeline.

M02.2 starts at the retrieval ceiling: change the model only after measured pipeline changes stop moving the result.
M02.3 later selects the model against explicit axes; it does not close unfinished RAG decisions.

Visual brief: A quiet two-line bridge: one jade arrow to M02.2 labelled 'retrieval measured, still short' and one smaller grey signpost to M02.3 labelled 'model selection on explicit axes'. Do not add a recap graphic or another pipeline redraw.

Designer constraint: Keep M02.2 as the primary bridge and M02.3 as a signpost; neither line may imply that M02.1 left a component choice unresolved.

Speaker notes

Takeaway

Model adaptation becomes the next decision only after measured retrieval improvements no longer close the remaining gap.

Key Points

M02.2 begins where a tuned and measured retrieval pipeline remains insufficient.
M02.3 later selects the model against explicit axes rather than reopening this session's RAG defaults.
The transition must preserve the distinction between changing the model and leaving a pipeline decision unfinished.

Prep Notes

Use the required M02.2 bridge line verbatim: When you've tuned the pipeline and you're still short, you stop changing the retrieval and start changing the model. Then signpost M02.3 in one sentence: the model underneath the pipeline is selected against explicit axes later in the day. Do not promise that M02.3 resolves an unanswered RAG decision; this session has already stated its chunking, embedding and store defaults.

Framework 2 · D01

The Staged RAG Diagnosis Demo

D01

Title

The Staged RAG Diagnosis Demo

Type

instructor-projected troubleshooting and diagnosis asset

Accountable Owner

Pradeep Menon

Engineering Builder

[VERIFY — TO ASSIGN]

Builder Gap

Assign one engineering builder who owns the corpus-to-trace implementation, replay/live interface, acceptance suite, offline package and one-page runbook. Pradeep Menon remains accountable for content, rehearsal sign-off and delivery.

Content Gate Status

The delivery-asset specification is complete for content approval. Implementation remains blocked until an engineering builder is assigned; content approval must not be represented as demo readiness.

Build Status

NOT BUILT — owner assignment, implementation, trace acceptance, offline packaging and rehearsal remain delivery-stage work.

Ready By

5 October 2026

Purpose

Let the room diagnose four upstream or generation failures from visible answer and evidence traces before the trainer reveals the originating component.

Delivery

Mode	Instructor-projected; no participant access
Default Path	Capture-and-replay through the same interface used by the live system
Proof Path	Live execution can be selected from the runbook if the room asks whether the pipeline is real
Opening Rule	Do not run a correct end-to-end example before Act 1
Reveal Rule	Reveal one act at a time; preserve the completed four-row board through consolidation

Shared System

Corpus	One versioned illustrative corpus of 24 invented source documents for a generic regulated institution. It contains no real institution policy text and no MUFG material. The builder may vary chunk count as required, but the 24 source documents, their versions and every locked baseline below remain fixed after owner sign-off.
Corpus Method	Derive a schema from realistic policy structure, generate varied invented documents from configurations, and derive the fixed questions and agreed answers from those configurations so ground truth is known by construction.
Required Corpus Features	The exact T2 illustrative rule: Role-change access reviews must be completed within 10 business days. The exact T2 illustrative exception: Exception: where privileged access is suspended, approval is required before access is restored. Rare, lexically similar exact identifiers including TS-999, which is reserved for D01 and never appears in the teaching sequence before slide 35. A superseded policy and replacement with effective-from plus effective-to or superseded-by metadata. An unanswerable gap surrounded by a densely covered neighbouring policy that shares vocabulary, entities and register. Document type, stable source/version/location, effective date, jurisdiction and access-class metadata.
Pipeline	One seven-stage teaching pipeline — Ingest, Chunk, Embed, Index, Retrieve, optional Re-rank, Generate — with query embedding inside Retrieve and intermediate output exposed at every stage. Break switches select versioned configurations or snapshots on this one visible instance; they do not create four separate applications.
Interface	One projected interface with V01 stage highlight, break control, fixed query ID, answer and citation pane, retrieved-chunk pane, agreed-evidence pane, and replay/live indicator. Captured traces and live traces render through the same components so replay is not visibly a video.
Invariants	Same synthetic corpus, pipeline implementation and projected interface across all four acts. A different fixed query/baseline pair is locked for each act; do not claim that one literal query runs across all four. The broken prompt carries no abstention instruction in any act because that is Act 4's revealed mitigation. Only the named switch or input condition changes within an act; the trainer says a change was made without naming its component.

Query Baseline Pairs

Act	Act 1
Pair Id	D01-A1
Query	What must happen after a role change, including when privileged access has been suspended?
Baseline	The agreed evidence and answer must preserve both exact T2 clauses: the review is completed within 10 business days, and where privileged access is suspended, approval is required before access is restored.
Manual Evidence	Show the retrieved rule chunk beside the separately located exception that should govern it.
Status	LOCKED FOR BUILD — the corpus writer must preserve both exact clauses and place the exception outside the broken fixed-size chunk.

Act	Act 2
Pair Id	D01-A2
Query	What is the approval route for TS-999?
Baseline	In the illustrative corpus, TS-999 requires the product owner to submit the release request to the Product Governance Committee and obtain recorded approval before release.
Manual Evidence	Show the TS-999 procedure that hybrid retrieval should return beside the topically related, wrong product-family chunks returned by dense-only retrieval.
Status	LOCKED FOR BUILD — TS-999 and its baseline are reserved for D01; the T5 teaching case uses AC-417.

Act	Act 3
Pair Id	D01-A3
Query	How often must high-risk customer files be reviewed under the policy currently in force?
Baseline	Every 12 months under the illustrative Customer Risk Review Policy v2, effective 1 July 2026; v1's superseded interval was 24 months.
Manual Evidence	Show the superseded and replacement policy passages side by side with both effective dates visible.
Status	LOCKED FOR BUILD — the corpus must include both policy versions and explicit effective-from, effective-to and superseded-by metadata.

Act	Act 4
Pair Id	D01-A4
Query	What retention period applies to biometric voiceprints used for branch queue analytics?
Baseline	The agreed answer is that the corpus does not contain enough evidence to answer. The gap originates at Ingest; the demonstrated failure is Generate not saying so.
Manual Evidence	Show every retrieved near-miss chunk and the recorded absence of an agreed supporting chunk; do not fabricate a should-have-been-retrieved passage.
Status	LOCKED FOR BUILD — no source document may answer this question; neighbouring policies may share retention, analytics and branch vocabulary but must not mention biometric voiceprints. Re-test if the delivery model changes.

Break Switches

Command	break=chunking
Act	Act 1
Change	Select the fixed-size chunk configuration whose boundary severs the T2 rule from its exception; lock overlap so it cannot silently repair the break.
Intended Bad Signature	The answer states the review rule while omitting the separately located restoration-approval condition.
Origin	Chunk
Mitigation	Use the cheapest defensible fixed-size-with-overlap baseline, then move to hierarchical retrieval when the typical answer spans a child chunk.

Command	break=dense_only
Act	Act 2
Change	Disable sparse retrieval and fusion while leaving the same index, pipeline and interface visible.
Intended Bad Signature	The system returns topically related product-family material but misses the exact TS-999 procedure.
Origin	Retrieve
Mitigation	Restore sparse retrieval and fuse it with dense results; re-ranking cannot recover a candidate that was never retrieved.

Command	break=stale_index
Act	Act 3
Change	Point the same pipeline at the pre-supersession index snapshot or disable the validity filter so the superseded policy remains answerable.
Intended Bad Signature	The answer is fluent, cited and out of date, and its citation resolves visibly to the superseded policy.
Origin	Index
Mitigation	Apply effective-date metadata at chunk time, filter validity at retrieval, propagate supersession and assign a refresh owner.

Command	break=out_of_corpus
Act	Act 4
Change	Select the fixed gap query while leaving the pipeline configuration unchanged; retrieval must return near-miss neighbours and the broken prompt must contain no abstention instruction.
Intended Bad Signature	Generate invents a plausible unsupported answer from near-miss evidence instead of stating that the corpus cannot answer.
Origin	The gap originates at Ingest; the failure to say so originates at Generate.
Mitigation	Add an abstention clause to the prompt contract and enforce citation support, while stating that neither control guarantees correctness.

Five Beats

Beat 1
Name Break
Timing about 20 sec
Instruction Trigger the act's named switch and say that one thing changed, without naming where.
Beat 2
Name Run
Timing about 40 sec
Instruction Run the act's fixed query through the same visible interface and show the captured bad answer and citation.
Beat 3
Name Diagnose
Timing about 60 sec
Instruction Ask which component failed and hold the silence; accept a concise hypothesis but do not answer it.

Beat	4
Name	Inspect
Timing	about 2 min
Instruction	Retrieve the evidence manually and show actual versus agreed evidence side by side. For Act 4, show near-miss chunks and the known absence of support rather than inventing a missing chunk.

Beat 5
Name Reveal
Timing about 1 min
Instruction Name the origin, pin it to V01, state one mitigation and reset to the next locked pair.

Timing

Topic Total	24 min
Setup	1 min; name the synthetic corpus, shared system, replay default and subset framing; do not show a correct run
Acts	20 min; four acts at roughly 5 min each, each protecting about 60 sec of diagnosis and the manual evidence check
Consolidation	3 min; completed four-row board plus subset qualification
Closing	4 min; 2 min V04/manual diagnostic and 2 min M02.2 bridge plus M02.3 signpost
Combined T7 And Closing	28 min

Execution Paths

Replay Default

Run every act live during build until the intended signature is produced and the accountable owner accepts the evidence pair.
Capture the complete trace: configuration and index version, query ID, retrieved chunks and scores, prompt, answer, citations, stage events and timing.
Replay those trace events locally through the same interface and stage-highlighting components used by live execution.
Record seed and sampling settings as build metadata only; fixed seeds do not guarantee hosted-model determinism.

Live Proof Path

Expose one runbook control that switches the same interface from replay to live without changing query IDs or stage labels.
Use live only on request or during rehearsal proof, state that output may vary, and return to the accepted replay trace for teaching.
Re-capture and re-accept affected traces after any model, embedding, vector-store, prompt or corpus change.

Acceptance Tests

Replay mode produces byte-identical trace output for each act in 20 consecutive runs, including chunks, prompt, answer, citations and stage events.
The same visible corpus, pipeline and interface are used across all four acts; each act has a distinct locked query/baseline pair and exactly one named break condition.
Act 1 returns the rule without the separately located exception, and its evidence pane reuses the exact illustrative clause shown in T2.
Act 2 returns topically related wrong material while the signed-off TS-999 evidence is recoverable through the baseline hybrid path.
Act 3 returns a fluent, cited, out-of-date answer whose citation opens the superseded policy with its effective date visible; the replacement policy is available in the agreed-evidence pane.
Act 4's corpus has no supporting answer, retrieval returns near-miss neighbours, and Generate fails to abstain; the test record states separately that the gap began at Ingest.
Run Act 4 live more than 20 times during build and again at rehearsal; any correct abstention blocks capture approval until the corpus/prompt is retuned and the test rerun. Repeat after any delivery-model change.
Every act completes inside roughly five minutes while preserving the diagnosis pause and manual evidence check; cut trainer commentary before either protected beat.
The V01 stage highlight and V04 placement use the same stage order, geometry, build/run split and dashed optional Re-rank treatment.
The full replay path passes with the network disabled, and the live proof control fails closed to replay rather than leaving the interface blank.
A venue-resolution fallback recording covers all four acts and can be launched locally if the interactive interface fails.
Pradeep Menon signs the four baselines, accepted wrong signatures, offline package and runbook at the rehearsal-ready gate on 5 October 2026.

Offline Fallback

Primary	Local replay package containing accepted trace files, assets and the same projected interface; no network call is required.
Secondary	One locally stored venue-resolution recording covering all four acts in order, including manual evidence panes and stage placements.
Preflight	Before rehearsal and delivery, disable the network, run all four replay traces, open the recording, verify fonts and resolution, and confirm that the live control falls back to replay with a visible status label.
Recovery Rule	If live or venue networking fails, state that replay is the designed default and continue from the current act; if the interface fails, switch to the recording at the same beat.

Runbook

Format	One page, available both printed and locally as PDF
Required Fields	Preflight checklist: network-off replay, recording, fonts, resolution, timer and V01 stage highlight. Act order and exact commands: break=chunking, break=dense_only, break=stale_index, break=out_of_corpus. For each act: pair ID, literal locked query, accepted bad-answer signature, retrieved/agreed evidence, origin stage, one-line mitigation and five-beat timing cues. Replay/live switch and the exact sentence explaining why replay is the default. Fallback point for every act and the local recording path. Reset step between acts and recovery steps for blank UI, wrong trace, accidental good answer or time overrun. Version manifest for corpus, chunker, embedding, index snapshot, retriever, optional re-ranker, prompt and model.
Trainer Rule	Do not improvise a query, baseline, answer or citation. If a trace does not match the accepted signature, switch to its accepted replay trace and continue.

Role Lenses

Developer	Own switch behaviour, trace completeness, intermediate-output visibility and the manual evidence comparison.
Infrastructure	Own versioned index snapshots, replay/live controls, network-off operation, local assets, latency headroom and recovery steps.
Risk Operations	Co-author and approve the baselines, effective-date logic, absence-of-support record, citation checks and rehearsal acceptance evidence.

Outputs

One versioned illustrative synthetic corpus and schema/configuration manifest.
Four signed-off fixed query/baseline pairs and four accepted captured traces.
One replay/live projected interface keyed to V01.
One 20-run-plus acceptance log with a separate Act 4 live firing log.
One offline replay package, one venue-resolution fallback recording and one-page runbook.

Scope Guards

No participant access, fifth act, opening success run, evaluation harness or UI polish beyond back-of-room legibility.
No agentic RAG, Graph RAG or RAFT in D01.
Do not present the four modes as complete, standard or frequency-ranked.
Do not use real institutional policy text, real MUFG material, a fabricated incident or unsupported literal query details.

Appendix — sources, verification and delivery notes

Dated evidence register

Topic	Figure	Value	Source	Refresh
T2	Metadata update capability	Metadata values can be patched in place in some stores, while incompatible schema changes can require an index rebuild.	Pinecone Update data; Qdrant Set payload; Microsoft Learn Update or rebuild an index; retrieved 27 July 2026.	5 October 2026
T3	fig-t3-japanese-evidence	JMTEB retrieval: text-embedding-3-small 66.39; Ruri-v3-30m 78.08	cl-nagoya Ruri-v3-30m model card, April 2025; JMTEB maintained by SB Intuitions; retrieved 27 July 2026	5 October 2026
T3	fig-t3-japanese-evidence	JFinTEB JaFIn retrieval: OpenAI-large 84.3; Sarashina 86.3	Suzuki and Sakaji, JFinTEB, SIGIR 2026, Table 4; retrieved 27 July 2026	5 October 2026
T3	fig-t3-six-gates	Trainer note: pgvector standard vector index limit 2,000 dimensions; halfvec 4,000	pgvector README limits cited by M02.1 evidence-base reconciliation; retrieved 27 July 2026	5 October 2026
T4	fig-t4-operating-shapes	Pinecone managed deployment; public serverless regions include Singapore but no Tokyo listing; cloud and region fixed after index creation	Pinecone Docs, Create an index; rechecked 3 August 2026	5 October 2026
T4	fig-t4-operating-shapes	Weaviate hosted and self-managed deployment routes	Weaviate Documentation, Deploying Weaviate; rechecked 3 August 2026	5 October 2026
T4	fig-t4-operating-shapes	Chroma local, single-node, distributed and managed-cloud shapes; current docs describe tenant-level isolation	Chroma Docs, Architecture Overview; rechecked 3 August 2026	5 October 2026
T4	fig-t4-operating-shapes	pgvector is an open-source vector similarity extension for PostgreSQL	pgvector project README; rechecked 3 August 2026	5 October 2026
T4	fig-t4-filter-placement	pgvector post-filter illustration: a 10% predicate with default HNSW ef_search 40 yields four matches on average	pgvector project README, Filtering; rechecked 3 August 2026	5 October 2026
T4	fig-t4-permission-path	Application authorization for most reviewed store shapes; PostgreSQL RLS is the reviewed row-policy enforcement path	Pinecone RAG with Access Control (8 January 2026); Weaviate RBAC docs; Chroma Architecture docs; PostgreSQL Row Security Policies; rechecked 3 August 2026	5 October 2026
T4	fig-t4-refresh-owner	PostgreSQL REINDEX operating modes and Weaviate property re-vectorisation behaviour are product-specific ownership examples, not cross-product benchmarks	PostgreSQL REINDEX documentation and Weaviate vectorisation documentation; rechecked 3 August 2026	5 October 2026
T5	fig-t5-dense-miss	Current Azure AI Search guidance identifies exact strings such as product codes as a hybrid-search use case	Microsoft Learn, Hybrid search overview; updated 19 February 2026; retrieved 27 July 2026	5 October 2026
T5	fig-t5-rerank-boundary	The semantic ranker reorders an existing result set and does not rerun retrieval over the corpus	Microsoft Learn, Semantic ranking overview; updated 24 April 2026; retrieved 27 July 2026	5 October 2026
T5	fig-t5-three-retrieval-questions	The ARES approximately-150 example is framework-specific and is not a universal Golden-set minimum	ARES, NAACL 2024; retrieved 27 July 2026	5 October 2026

Open verification items

T1 — V-T1-03 — Confirm the OWASP RAG Security Cheat Sheet version and date before delivery; until then, cite it without a printed date.
T2 — Confirm D01 Act 1 reuses the exact synthetic policy rule and exception defined in figs_T2.py; the generated bad answer must remain hidden until the demo.
T2 — Re-check Microsoft's 512-token / 25-percent-overlap guidance and the named-store metadata update documentation by 5 October 2026.
T2 — Check again for Japanese-language chunking evidence before delivery; none was located in the 27 July 2026 research pass.
T2 — No published enterprise-scale storage or query-cost comparison for hierarchical versus flat fixed-size retrieval was located; the trainer must say 'not published' rather than estimate.
T3 — By 5 October 2026, independently re-check the JMTEB rows for text-embedding-3-small 66.39 and Ruri-v3-30m 78.08; the current table is author-reported by the Ruri team.
T3 — By 5 October 2026, confirm the JFinTEB JaFIn 84.3 and 86.3 values against the final SIGIR 2026 proceedings or camera-ready Table 4.
T3 — By 5 October 2026, refresh the pgvector vector and halfvec index-dimension limits before retaining the trainer-note example.
T3 — Japanese cross-lingual retrieval and Chinese-specific retrieval degradation remain unsourced in this evidence pack; both are deliberately omitted from slide claims.
T4 — By 5 October 2026, re-check Pinecone's public region table and obtain contractual confirmation of the required data-residency location and commitments; Singapore is publicly listed and Tokyo is not listed as of 3 August 2026, but that does not settle the client's legal requirement.
T4 — By 5 October 2026, confirm whether Chroma's current tenant-level isolation and access-control statement applies consistently across local, self-hosted single-node, distributed and cloud modes; this topic does not treat it as per-user, per-document authorization.
T4 — By 5 October 2026, confirm the scope and commercial availability of Weaviate RBAC in the intended self-managed or cloud deployment; no row-level end-user enforcement claim appears on the slide.
T4 — By 5 October 2026, refresh all four operating-shape statements and the Pinecone, Weaviate and Chroma authorization scopes because these are perishable product claims.
T4 — No independent filtered-search benchmark, trustworthy pgvector vector-count exit threshold, product-comparable reindex duration, migration path, or four-store backup/DR comparison was found; none is asserted.
T7 — Assign D01's engineering builder and confirm they own corpus implementation, trace capture, replay/live interface, offline package, acceptance log and runbook.
T7 — The four D01 query/baseline pairs are locked as invented teaching content. The builder must encode them in the synthetic corpus exactly, preserve the declared absence in D01-A4, and obtain Pradeep Menon's sign-off before trace capture.
T7 — Implement exactly 24 versioned synthetic source documents; record the resulting chunk count after build and confirm the corpus remains legible in the projected evidence panes during rehearsal.
T7 — Empirically test the chosen vector store's index rebuild and search reproducibility; do not assume ANN determinism.
T7 — Run Act 4 more than 20 times on the exact delivery model, re-check it at rehearsal and repeat after any model change.
T7 — Refresh any provider determinism controls used in the builder runbook on 5 October 2026; no provider product claim belongs on a teaching slide.

Brief conflicts resolved or carried

T1 — Reworded the brief's permission claim to the locked form: enforcement is possible at retrieval time only when the team builds it.
T1 — Kept the required retrieval-ceiling line but removed the absolute claim that a stronger model can never recover from insufficient evidence.
T1 — Narrowed 'no retraining cycle' to updating facts outside the generator rather than implying that no component ever needs training.
T1 — Did not assert that RAG is the default pattern or use a dated adoption figure because the available measurement is thin and stale.
T1 — Widened Retrieve to admit semantic and lexical candidates and marked Re-rank as optional, as required by evidence-base reconciliation.
T2 — The brief's 'measured baseline' wording is corrected: fixed-size is the cheapest defensible reference baseline, not a configuration proven universally best.
T2 — The brief's 'nothing downstream recovers it' wording is corrected: overlap and parent-child retrieval mitigate boundary failures at chunk time, while re-ranking cannot add missing candidates.
T2 — The brief's 'retrofitting means a full reindex' wording is corrected: some metadata values are patchable, while provenance, re-chunking and schema changes can still force rebuilding.
T2 — Semantic chunking is treated as unproven until corpus-specific measurement shows a gain; it is not presented as a routinely superior strategy.
T2 — Hierarchical retrieval is positioned for evidence that spans one child unit, with policy applicability labelled as an inference rather than a directly measured banking-policy result.
T3 — The coverage brief sketches four slides, while the approved fleet map requires five: question, definition diagram, dated evidence, six-gate framework, and transition.
T3 — No sourced narrative Japanese retrieval example was found; the dated slide uses paired benchmark evidence instead.
T3 — The brief's implied blanket degradation of English-first models on Japanese is rejected: JFinTEB shows only a two-point gap on the financial QA retrieval task.
T3 — No defensible quality-versus-dimension curve was found; Width is treated as an operating compatibility and cost gate, not a quality proxy.
T3 — The absolute reindex claim is softened to an operating assumption because embedding-space adapters exist as a research exception.
T3 — Cross-lingual Japanese and Chinese-specific performance claims are omitted because this research pass did not source them.
T4 — The coverage brief sketches four slides, while the approved fleet map requires seven: question, production contract, operating shapes, filter placement, refresh ownership, permission enforcement, and transition.
T4 — Pinecone is described as managed without the unsourced claim that it is fastest to something working.
T4 — Weaviate is positioned by hosted versus self-managed deployment; any added operating burden is stated as a consequence of choosing self-management, not as a vendor attribute.
T4 — Chroma is not labelled prototype-only: current documentation covers local, single-node, distributed and cloud production shapes.
T4 — Access control is not presented as an automatic store feature: most reviewed paths require application authorization, while PostgreSQL RLS is the row-level database enforcement path among the four.
T4 — Namespace, tenant and collection separation is treated as scope isolation, not as a substitute for caller authentication and per-document authorization.
T4 — Post-filtering is split into a recall failure and a confidentiality failure rather than described as one universal security problem.
T4 — Residency is a signed-off deployment gate rather than a vendor score; the slide makes no claim that a listed cloud region satisfies the client's contractual requirement.
T4 — No vector-count threshold, store ranking, feature scorecard, price, performance superlative or cross-product reindex number is used.
T5 — The coverage brief sketches five slides, while the approved fleet map requires seven so the opening question, re-ranking boundary and transition remain distinct from the two named frameworks and the diagnostic.
T5 — The brief's highest-return superlative is rejected: hybrid is presented as the first low-cost improvement, while no claim is made that it supplies the largest raw quality gain.
T5 — The manual-retrieve binary is retained as locked wording and explicitly qualified as first-cut triage because context assembly and retrieval noise can create a third case.
T5 — Golden-set authorship is collaborative with domain owners rather than assigned to the business alone; the build team must not be its sole author.
T5 — The ARES approximately-150 figure appears only in the trainer note with its actual scope: distinguishing whole RAG systems in ARES, not setting a universal retrieval-evaluation minimum.
T5 — No real banking retrieval incident, account-number claim, benchmark ladder or latency number is fabricated; the exact-identifier case is labelled illustrative and sourced to mechanism-level guidance.
T6 — The brief says the retriever decides in agentic RAG; the evidence places control-flow decisions with the model or orchestrator, so the slide uses the corrected agency.
T6 — The brief groups three advanced patterns as parallel options; the slides instead distinguish control flow, retrieval structure and model change without ordering them as an upgrade ladder.
T6 — The measurement-gated escalation rule has no authoritative published source, so it is explicitly labelled Chartered Vectorial recommendation.
T6 — Graph RAG is described durably as relationship-based retrieval across documents; Microsoft's corpus-wide summarisation design is not mis-cited as general multi-hop evidence.
T7 — The coverage brief's one-query-across-four-acts instruction is replaced by four fixed query/baseline pairs; one literal query cannot credibly trigger four distinct failures.
T7 — Capture-and-replay is the default execution path; fixed seeds are recorded but do not guarantee hosted-model determinism, and live execution is the proof path.
T7 — Act 4's gap is surrounded by near-miss policy content rather than isolated, and the broken prompt carries no abstention instruction.
T7 — Act 4 is attributed in two parts: the missing evidence begins at Ingest, while the failure to state that absence occurs at Generate.
T7 — The four modes are framed as a diagnosable subset rather than a complete or frequency-ranked taxonomy.
T7 — The approved fleet map combines the coverage brief's three demo-support and two closing slides into exactly five slides, with no additional slide invented.

Honest timing and first cuts

T1 — Honest total: 11 minutes (1.5 + 3.5 + 2.5 + 2 + 1.5). First cut if the room runs long: shorten the spoken role examples while retaining one concrete ownership line per role; do not cut V01, the build-once/run-per-question explanation, or the grounding-versus-guarantee correction.
T2 — Reconciled total: 15 minutes with the eight-slide approved map and E01 protected at three minutes. The applied one-minute recovery caps the opening at one minute and the fixed-size evidence comparison at 1.5 minutes, while preserving V02, metadata, E01 and the closing default. The approved map is retained even though the coverage brief sketched five slides.
T3 — Honest total: 9 minutes (1 + 1.5 + 2.5 + 2.5 + 1.5). If one minute must be recovered, cut the spoken library-style analogy and one role prompt from the definition slide first; do not cut the dated provenance, the six-gate framework, or the migration handoff.
T4 — Honest total: 12 minutes (1 + 1.5 + 2.5 + 2 + 2 + 2 + 1). First cut if one minute is needed: remove the current-region detail from the operating-shapes narration and deliver each of the four shapes in one sentence; do not cut filter placement, refresh ownership, permission enforcement, or the T5 handoff.
T5 — Honest total: 13 minutes (1 + 1.5 + 2.5 + 1.5 + 2.5 + 2 + 2). If 90 seconds must be recovered, cut the RRF-versus-weighted-fusion note on The 3-Rung Retrieval Ladder first; do not cut the two named frameworks, the re-ranker boundary, the locked manual-retrieve diagnostic, its context/noise qualification, or the one-change transition.
T6 — Honest total: 2 minutes (1 + 1). First cut: T6 as a block. If an in-room overrun forces a partial delivery, show slide 1 and read slide 2's recommendation aloud; do not expand either slide with benchmarks, costs or pattern taxonomy.
T7 — Honest total: 28 minutes — 24 minutes for D01 and its consolidation, plus 4 minutes for V04 and the session bridge. The first cut is up to two minutes of trainer commentary after the four reveals: keep each mitigation to one sentence. Do not cut the approximately 60-second diagnosis pauses, manual evidence checks, three-minute completed-board consolidation, Act 4 origin distinction, or either closing bridge.