AI Practitioner Certificate · Codex comparison edition

M02.3 — Model Selection & Cost-Optimisation Framework

6 topics · 39 slides · 75 minutes · 11:00–12:15 · CONFIDENTIAL — CV / NUS AIDF Alliance

The one idea every topic servesStart with the task, remove infeasible options at hard gates, then optimise the feasible choices across five explicit axes.

Ledger — working state, hidden from print

Topic	Status	Slides	Opening question	Kind	Closing link	Dated figures
T1	written	1–7	Which starting point makes model selection defensible: candidates, benchmarks, or task constraints?	show of hands	Once the job is scored, deployment responsibility becomes the first decision the matrix must test.	—
T2	written	8–14	If processing must remain in jurisdiction, which deployment patterns are actually ruled out?	point at the map	Deployment constraints narrow the feasible set; the current market determines what remains worth testing.	fig-t2-shortlist-map
T3	written	15–19	Which fact on today’s model landscape is least likely to survive until delivery?	predict the expiry	A shortlist remains provisional until current prices are multiplied by the workload you will run.	fig-t3-landscape
T4	written	20–27	Which workload assumption is most likely to break your pilot cost estimate in production?	silent estimate	After accidental spend is removed, reasoning effort still needs a task-level justification.	fig-t4-two-caches; fig-t4-batch-contract
T5	written	28–31	When does more reasoning make an answer worse as well as slower?	provocation withdrawn	A targeted reasoning budget completes the method; the room must now defend a choice without a model list.	—
T6	written	32–39	Which axis will your role defend when the team cannot maximise all five?	role positioning	Changing model behaviour comes last because prompting, retrieval, selection, and operating choices should be tested first.	—

Brief conflicts applied

T1 — The brief's chatbot, AML narrative, and batch-extraction examples duplicate three E02 workloads and would pre-solve their dominant axes. The first replacements also remained structurally equivalent to E02, so the revision contract replaces them with trading-floor commentary summarisation, branch-network translation, and board-pack drafting.
T1 — Capability is treated as the candidate side of the accuracy threshold rather than a sixth axis. This preserves one vocabulary across V05, T2, and E02.
T1 — The brief's statement that improving one axis degrades another is bounded: the axes are coupled, but some optimisations improve two visible axes while shifting risk into precision, staleness, or operational control.
T1 — Data residency is phrased as where data may be processed and under whose jurisdiction, without interpreting regulation or claiming that residency requires self-operation.
T1 — The Slide Guide sketches five slides; the approved fleet map uses seven because the opening question and causal transition are mandatory and the capability correction needs its own concept slide.
T2 — Brief: The brief frames deployment as proprietary API versus self-hosted open-weight. · Resolution: Use a four-cell weight-licence by operating-responsibility map because managed open-weight endpoints and closed-weight perimeter appliances populate both omitted cells.
T2 — Brief: The brief implies residency commonly decides in favour of self-hosting. · Resolution: Treat residency as four technical checks; managed options can satisfy location constraints, while exact-version reproducibility may still require self-operation.
T2 — Brief: V06 includes capability as a comparison dimension alongside the five axes. · Resolution: Hold capability constant for the deployment comparison; capability remains the model side of the accuracy threshold, not a sixth use-case axis.
T2 — Brief: The Slide Guide sketches four slides for T2. · Resolution: Use seven slides because the owner requested an applied company map after the 2×2 framework; redistribute the existing teaching time so T2 remains nine minutes.
T2 — Brief: The shared rule quarantines visible model names to T3's dated landscape slide. · Resolution: Owner-approved exception: add one dated T2 applied example using a fictional company and first-party deployment evidence; label it as placement rather than capability ranking and refresh it before delivery.
T3 — The brief’s frontier / mid / small open-weight ladder conflates capability size with weight access. The content uses Small, Mid, and Frontier as equal capability bands and treats Open-weight as an orthogonal licence and deployment property.
T3 — The approved five-slide map adds a question and transition around the brief’s three-slide teaching core; the six-minute limit is preserved.
T3 — The brief’s static October refresh is insufficient because Anthropic published a 1 September 2026 price change; the content adds price, landscape, and delivery checkpoints.
T4 — The brief’s peak-versus-average multiplier is removed from the metered-token formula. Peak affects rate limits and self-operated capacity, not token-metered spend.
T4 — The brief’s single caching lever is split into prompt caching and response caching because they act on different terms and carry different risks.
T4 — The brief’s universal lever order is replaced by a prior Trim check and unnumbered peer moves for Cache, Batch, and Route, selected by workload eligibility and control cost.
T4 — The brief’s V07 mechanism is corrected: the waterfall operates on residual cost, not traffic volume, and savings compound conservatively rather than add.
T4 — The brief’s no-cost description of batching is replaced by its completion, queueing, streaming, expiry, and retention trade-offs.
T4 — The approved eight-slide map exceeds the brief’s five-slide sketch but remains within the ten-minute topic allocation.
T5 — The brief treats reasoning and standard as separate product families; current provider and open-weight documentation instead exposes reasoning as a per-request effort control on the selected model.
T5 — The brief implies a dearer model causes the cost premium; current billing normally keeps the rate card fixed while thinking expands billed output tokens.
T5 — The brief says reasoning is slower per call without locating the delay; the content makes clear that the wait is front-loaded before the first answer token rather than spread across answer generation.
T5 — The approved slide 29 wording used extraction and classification examples that exposed the operating shapes of E02 Teams 2 and 4. The revision replaces the headline, comparison, and adjacent question-note example with known-label message triage and multi-system incident diagnosis.
T5 — The approved four-slide map adds a question and transition around the brief’s three-slide teaching core while preserving the five-minute limit.
T6 — Brief: Topic 1 examples repeat three E02 workloads and reveal their dominant axes. · Resolution: Use the reconciled non-E02 examples in T1 so E02 remains a reasoning exercise.
T6 — Brief: Team 2 is accuracy-first despite measurable extraction thresholds and the brief's own cost-first batch example. · Resolution: Make Team 2 cost-first after its measurable accuracy gate; preserve Team 4 as accuracy-risk-first.
T6 — Brief: Team 3 uses capability as a sixth axis and prescribes a proprietary API. · Resolution: Use a high accuracy bar; allow either managed open- or closed-weight frontier tiers when the task and code-boundary gates are met.
T6 — Brief: Team 4 prescribes a separate reasoning-model product. · Resolution: Choose reasoning effort per request on a frontier tier and turn it down for straightforward work.
T6 — Brief: Team 5 implies that residency forces self-hosted open weights and that open weights provide explainability. · Resolution: Allow several verified in-jurisdiction patterns; use operational complexity and reproducibility as tiebreaks, and treat explainability as evidence-linking rather than a licence property.
T6 — Brief: Five sequential reports plus five challenges do not fit the original nine-minute report-out. · Resolution: Use the approved twelve-minute report-out: capture all five axis-and-selection answers first, then challenge the most instructive disagreements.

Agent notes

T1 — Honest timing is 11 minutes across the approved seven slides. If the session is running late, cut the verbal walk-through of the three examples on slide 6 to one sentence per card, saving about 45 seconds; do not cut slides 2 or 3 because they establish the method and the fillable V05 instrument. Revision applied across T1: the opening now remains neutral until commitment; V05 uses the use-case container as its sole jade emphasis and renders all five axes equally; slide 6 uses trading-floor commentary summarisation, branch-network translation, and board-pack drafting; generated SVG IDs are figure-prefixed for safe assembly. Four figures were regenerated and visually inspected.
T2 — Honest timing is 9 minutes: 1 + 1.5 + 1.25 + 1.5 + 1.25 + 2 + 0.5. The first cut under time pressure is the spoken elaboration on cost-curve assumptions; retain the labelled figure and the three checks—model class, sustained utilisation, and redundant capacity. Slides 8 and 9 intentionally reuse the deployment-map figure so the point-at-the-map hook resolves into the framework without a visual reset. The new dated company example is an owner-approved exception to the T3 model-name quarantine.
T3 — Honest time is 6.0 minutes across five slides. Protect slides 16 and 17 because they carry the durable skills. If a cut is forced, remove the named model chips from slide 15 and retain the dated three-band map, orthogonal open-weight lane, and refresh gates; do not compress either framework.
T4 — Honest time is 10.0 minutes across eight slides. The figures deliberately carry no provider names, current dollar prices, or workload hit-rate claims; volatile multipliers remain in notes and the appendix. First cut under time pressure: reduce slide 23 to a 30-second quality-gate read and let slide 24 carry Route as the summary, saving about 0.75 minute; do not cut the baseline, cache distinction, or residual-cost waterfall.
T5 — Honest time is 5.0 minutes across four slides. Protect slide 28 because it corrects both the product-category and latency-shape errors. If a cut is forced, remove the question slide’s figure and run the provocation verbally; do not shorten the comparison or the handover into E02. Revision applied: slide 27 remains neutral until commitment; slide 29 now uses known-label message triage versus multi-system incident diagnosis throughout; the measurement callout wraps with safe padding; every figure has one jade emphasis and figure-prefixed SVG IDs.
T6 — Honest timing is 34 minutes: briefing 3, team work 14, report-out 12, debrief 2, close 3. The first cut if the session is late is slide 33's spoken scenario-board orientation and then one report-out deep dive; do not cut team work, the all-team sweep, the debrief, or the close. Participant scenario figures are explicitly illustrative exercise assumptions and remain flagged for practitioner review before print.

T1 — The Five-Axis Decision Matrix

11 minutes · slides 1–7 · opens with show of hands · memory hooks: The 3-Step Selection Test · The 5-Axis Decision Matrix

1question1 min

Which starting point makes model selection defensible: candidates, benchmarks, or task constraints?

Show of hands: start with a candidate list, a benchmark table, or the task constraints?
Choose one starting artefact; hold all model names until the room commits.

1. Takeaway

A defensible model decision starts by specifying the job and its constraints, before any candidate enters the conversation.

2. Key Points

Listen for five kinds of constraint: accuracy, latency, cost, operating responsibility, and where processing may occur.
Starting from a favourite model contaminates the reasoning because the team begins adapting the job to the candidate.
M02.1 established that good enough belongs to the use case; this topic converts that idea into a repeatable selection method.

3. Prep Notes

Give the room ten seconds, then take a show of hands across the three starting points without asking anyone to name a model. If someone does name one, park it visibly and reveal that the exercise is testing whether the job can be specified before a shortlist shapes the criteria.

2framework1.5 min

The 3-Step Selection Test removes infeasible options before it optimises acceptable ones.

Gate: remove any option that violates a hard constraint.
Threshold: define the minimum result that counts as usable.
Optimise: compare only the survivors across the remaining trade-offs.
Score the use case before you look at a single model.

The sequence is non-negotiable: Gate, then Threshold, then Optimise; a strong score cannot compensate for a failed gate.

1. Takeaway

Selection becomes defensible when infeasible options are removed before the remaining choices are compared.

2. Key Points

A gate is non-compensatory: strength on another axis cannot repair a failed legal, security, or operational condition.
A threshold establishes the minimum acceptable performance for the use case rather than declaring one model universally best.
Optimisation is reserved for candidates that have already passed every gate and threshold.

3. Prep Notes

Introduce Gate, Threshold, Optimise as a CV teaching sequence, not a published model-selection standard. Use a right-to-work check as the analogy: an otherwise strong candidate who cannot legally work in the jurisdiction is removed before comparative scoring. Organisations may add a deciding axis without changing the order.

Figure source: CV original synthesis, informed by Carnegie Mellon SEI's Architecture Tradeoff Analysis Method report, retrieved 27 July 2026.

3framework2.25 min

The 5-Axis Decision Matrix scores the job because models cannot define their own acceptance criteria.

Accuracy: is it good enough for this task?
Latency and cost: is it useful in time and affordable at production volume?
Operational complexity: who runs, patches, and responds when it breaks?
Data residency: where may data be processed, and under whose jurisdiction?

V05 keeps the use-case container as its sole jade emphasis; all five fillable axes remain neutral and equal, with no total-score row.

1. Takeaway

The five-axis matrix specifies the job; it is not a scorecard that turns unlike constraints into one total.

2. Key Points

Complete the use-case row first so that every criterion exists before candidates are considered.
Record non-negotiable conditions as gates and minimum service levels as thresholds, keeping trade-offs visible for judgement.
Separate axes prevent a strong result in one area from disguising an unacceptable result in another.

3. Prep Notes

Walk from the use-case row down. Use document processing as the example: if data may be handled only in an approved jurisdiction, record that as a residency gate before comparing capability or cost. Emphasise that the matrix is not totalled.

Figure source: CV original V05 teaching instrument, developed from the coverage brief and T1 research dated 27 July 2026; it is not a published industry standard.

4concept1.25 min

Accuracy sets the task threshold; model capability determines whether a candidate can meet it.

The use case defines the consequence of error and the minimum acceptable result.
A golden set tests that threshold on the organisation's own work.
Capability belongs to the candidate, so it is not a sixth use-case axis.
A harder task raises the accuracy bar; it does not change the matrix.

1. Takeaway

Accuracy is a task-specific threshold, while capability is evidence about whether a candidate can meet it.

2. Key Points

The use case's golden set is the final test of good enough; a public benchmark is supporting evidence, not a substitute.
Benchmark familiarity can exaggerate apparent fitness when test items or task conditions differ from production.
A workload that needs advanced reasoning still records a demanding accuracy threshold; capability does not become a sixth axis.

3. Prep Notes

Make the M02.1 callback explicit, then use GSM1k as the warning: recreating a familiar mathematics benchmark with new questions produced materially lower results for some model families. Source: Zhang et al., GSM1k, arXiv:2405.00332, retrieved 27 July 2026. Bridge to E02 by saying that teams will defend thresholds with task evidence, not benchmark reputation.

5diagram1.75 min

Every apparent free improvement shifts its cost into precision, staleness, or operational burden.

Response caching can improve speed and cost while creating a staleness risk.
Quantisation can improve speed and cost while consuming precision headroom.
Self-operation can increase control while adding capacity, patching, and on-call obligations.
Re-score all five axes after an optimisation rather than celebrating one visible gain.

The benefit and its hidden price must remain paired; no improvement is presented as free across all five axes.

1. Takeaway

An optimisation move changes several parts of the decision, so the complete constraint set must be tested again.

2. Key Points

Coupling means a change aimed at cost or latency can also affect accuracy, operating effort, or deployment feasibility.
Some moves improve more than one visible axis, so avoid teaching every gain as an immediate one-for-one penalty.
The use case's golden set remains the acceptance test after optimisation.

3. Prep Notes

Use quantisation as the concrete example. Red Hat and Neural Magic reported substantial efficiency gains with high benchmark-score retention, but that vendor-published result still requires validation on the use case's golden set; evaluation dated 17 October 2024, retrieved 27 July 2026.

Figure source: CV original synthesis of the coupling mechanisms documented in T1 research, retrieved 27 July 2026.

6case2.25 min

The dominant axis changes with the workload, so one institution should reach different answers.

Trading-floor commentary summarisation is latency-first because late context loses its decision value.
Branch-network translation is cost-first because a large scheduled corpus can use a batch window.
Board-pack drafting is residency-first because confidential governance material has a processing boundary.
Same five axes, three different shapes: the job changed, not the method.

Each workload exposes one hard edge without reusing an E02 workload, output, or operational shape; the examples must not become exercise clues.

1. Takeaway

The dominant axis changes with the workload, even inside one institution.

2. Key Points

Interactive commentary summarisation is latency-led because usefulness decays while the user waits.
High-volume branch translation is cost-led because small unit-cost differences compound at scale.
Sensitive board-pack drafting is residency-led because processing location can determine feasibility before quality is compared.

3. Prep Notes

Move quickly across the three cards, then ask what condition would make another axis dominate. For example, turning an interactive commentary tool into an overnight digest removes the immediate latency constraint. Make clear that different answers follow from changed workloads, not vendor preference.

Figure source: CV original illustrative workloads created for this session; they are not named client cases and deliberately avoid E02 scenario clues.

7transition1 min

Once the job is scored, deployment responsibility becomes the first decision the matrix must test.

The matrix has exposed the hard gate, minimum threshold, and trade-off the workload can accept.
The next test separates what may be deployed from who must operate it.

T2 — Deployment Choices

9 minutes · slides 8–14 · opens with point at the map · memory hooks: The 2×2 Deployment Map · The 4 Residency Checks · The 5 Operating Obligations

8question1 min

If processing must remain in jurisdiction, which deployment patterns are actually ruled out?

Point to a cell before discussing any provider or model.
State whether the constraint is about storage, inference, retention, or reproducibility.

Do not collapse weight licence and operating responsibility into one proprietary-versus-open binary.

1. Takeaway

Residency alone does not identify a deployment pattern; it must be translated into separate technical controls.

2. Key Points

Storage location does not prove where inference runs.
Retention can vary by feature, endpoint, and account arrangement.
Reproducibility asks whether the validated model version can remain available, which is different from data location.

3. Prep Notes

Give the room ten seconds to point at one cell, then ask a risk or operations participant to state the constraint behind the choice before anyone names a provider. Reveal that no pattern is ruled out by the word residency alone; Rest, Run, Retain, and Reproduce must be checked separately.

Use a managed service that stores records in-jurisdiction while processing elsewhere. Sources: Anthropic Data residency, OpenAI Data controls in the OpenAI platform, and AWS Geographic cross-Region inference guidance, retrieved 27 July 2026. This is a technical-feasibility test, not regulatory permission. Figure source: CV original 2×2 teaching construct synthesised from the T2 research and those provider documents, retrieved 27 July 2026.

9framework1.5 min

The 2×2 Deployment Map separates weight licence from operating responsibility because both diagonals now exist.

Open weights can be self-operated or delivered through a managed endpoint.
Closed weights can sit behind a provider API or inside a managed perimeter appliance.
The useful comparison holds capability constant before it compares operating models.
No cell is the organisational default; the use case decides.

Keep all four cells equal in visual weight; the invariant is that licence and operator are independent axes.

1. Takeaway

Weight access and operating responsibility are independent deployment decisions, so all four cells of the map are possible.

2. Key Points

Open weights can be served by a provider rather than operated by the customer.
Closed weights can be delivered in a managed perimeter appliance rather than only through a public API.
Compare deployment patterns at a constant capability class so a smaller model does not disguise the cost of operating responsibility.

3. Prep Notes

Name the 2×2 Deployment Map and walk the axes aloud: Open or Closed weights crossed with Your team or Provider operation. Contrast a provider-operated open-weight endpoint with a managed closed-weight appliance. Sources: DigitalOcean Token Economics Across Traffic Profiles on Dedicated GPUs, 8 July 2026, and Google Cloud Run Gemini and AI on-prem with Google Distributed Cloud, retrieved 27 July 2026.

Figure source: CV original 2×2 teaching construct based on those provider deployment patterns; it is not an industry-standard taxonomy.

10case1.25 min

Meridian Bank’s shortlist spans three cells because deployment contracts—not model brands—determine placement.

gpt-oss-120b and Qwen3 can be evaluated as bank-operated open-weight candidates.
The same open-weight families can move to the provider-operated cell when a hosting provider runs the endpoint.
Claude Opus 4.8 sits in the proprietary provider-API cell for this illustrative shortlist.
An empty cell is a shortlist gap, not proof that the deployment pattern is impossible.

Placement describes weight access and operating responsibility—not capability, recommendation, or permanent product availability.

1. Takeaway

A model name does not determine a deployment cell; the licence, operator, region, and contract do.

2. Key Points

gpt-oss and Qwen3 publish open weights, so Meridian Bank could operate them directly or procure a provider-operated endpoint.
Claude Opus remains a proprietary hosted model in this example, so it is placed under provider operation rather than alongside downloadable weights.
The blank closed-weight, bank-operated cell records that the fictional shortlist has no verified candidate there; it does not erase contract-specific appliances or future arrangements.

3. Prep Notes

Introduce Meridian Bank as a fictional company and walk the badges rather than the quadrants. Point out that Qwen3 and gpt-oss appear in more than one cell because an open-weight family can be self-operated or hosted by a provider. Do not compare model capability on this slide; the purpose is to make the two axes operational.

Model-placement evidence retrieved 5 August 2026: OpenAI, Introducing gpt-oss and OpenAI open-weight models; QwenLM, Qwen3 repository; Alibaba Cloud, Model Studio; Anthropic, Claude Opus. Verify the exact model version, region, provider role, and contract at the decision date. Figure source: CV original illustrative company map based on those first-party sources.

11framework1.5 min

The 4 Residency Checks turn a vague location requirement into a hard deployment gate.

Rest: where inputs, outputs, and records are stored.
Run: where inference is executed and under which jurisdiction.
Retain: what each endpoint or feature may keep after the request.
Reproduce: whether the validated model version must remain runnable on your schedule.

Preserve all four checks: storage residency alone does not establish processing, retention, or version control.

1. Takeaway

A vague location requirement becomes testable only when Rest, Run, Retain, and Reproduce are answered separately.

2. Key Points

A storage-residency option may exist where regional inference processing does not.
Retention commitments can differ across features and endpoints within the same provider account.
Version availability matters when a validated deployment must be reproduced after a provider update.

3. Prep Notes

Have the room answer all four checks before calling residency satisfied. Use the July 2026 OpenAI example: Japan and Singapore were listed for storage residency but not regional processing, so Rest did not settle Run. Retention nuance is documented in Anthropic API and data retention and OpenAI Data controls guidance, retrieved 27 July 2026.

Figure source: CV original 4-check synthesis informed by those sources and AWS Geographic cross-Region inference guidance, all retrieved 27 July 2026. Apply it as a technical-feasibility gate, not regulatory interpretation.

12comparison1.25 min

API and self-operated cost curves cross only at a workload-specific volume.

Metered endpoints scale with actual use; self-operation commits capacity before demand arrives.
The comparison is invalid unless it holds the model class and service level constant.
Sustained utilisation must include nights, weekends, and redundant capacity.
The crossover is a calculated region, not a transferable benchmark.

The curves are illustrative shapes only; never add a crossover volume or imply that it transfers between workloads.

1. Takeaway

The API-versus-self-operated crossover is a workload-specific region, not a universal volume.

2. Key Points

Metered cost follows use, while committed capacity is paid for whether or not it is fully utilised.
Redundancy, nights and weekends, and failover reserves can move the crossover without changing the model or hardware.
A fair comparison must hold model class, service level, operating cost, and availability requirements explicit.

3. Prep Notes

Trace both shapes and point to the wide crossover region. Challenge the apparent intersection with a redundant replica kept idle for failover, which lowers sustained utilisation. State that the research found no vendor-neutral crossover analysis.

Figure source: CV original illustrative curves based on mechanisms in DigitalOcean Token Economics Across Traffic Profiles on Dedicated GPUs, 8 July 2026, and Inworld AI LLM Inference Cost at Scale, 7 July 2026, retrieved 27 July 2026. Both publishers sell inference, so cite them for the mechanism rather than a universal answer.

13framework2 min

Self-operation converts inference into five operating obligations that API prices conceal.

Capacity asks whether the service can handle demand; availability asks whether it can survive failure.
Patch changes the serving stack; re-evaluate proves the workload still meets its thresholds.
On-call assigns detection, escalation, and restoration to named people.
Each obligation needs an owner, a target, and evidence rather than a verbal promise.

Read each row as obligation → meaning → evidence; ownership is incomplete until all three are explicit.

1. Takeaway

Self-operation is an operating contract: the team must define, own, and evidence five obligations rather than treating inference as installed software.

2. Key Points

Capacity covers the resources needed to serve expected and peak demand; availability covers redundancy, failover, and recovery when part of the service fails.
Patch means safely changing drivers, runtimes, dependencies, and serving software; re-evaluate means proving after the change that task quality, latency, and reliability still pass their thresholds.
On-call completes the ownership loop by naming who detects, triages, escalates, restores, and learns from an incident; every obligation should leave auditable evidence.

3. Prep Notes

Walk the operating contract from top to bottom. For each row, name the obligation, read its definition, then point to the evidence column. Pause after Capacity versus Availability and Patch versus Re-evaluate because those are the two distinctions teams most often collapse. Use an inference-server security update as the example: the patch record proves the change was installed, while the golden-set report and performance test prove the service remains acceptable.

Ask the room which evidence already exists for one live service and who owns the missing item. Managed service changes the division of responsibility, but the client still needs monitoring, retry and escalation arrangements for provider-side failures. Datadog's 2026 State of AI Engineering identifies operational complexity as the principal barrier to reliable AI at scale, retrieved 27 July 2026. Figure source: CV original operating-contract synthesis informed by the T2 operations research and that Datadog evidence, retrieved 27 July 2026.

14transition0.5 min

Deployment constraints narrow the feasible set; the current market determines what remains worth testing.

Carry forward only the cells that pass the use case's hard gates.
Re-check the decision per workload rather than turning it into an organisational policy.

T3 — Reading the Model Landscape

6 minutes · slides 15–19 · opens with predict the expiry · memory hooks: The 4 Leaderboard Questions · The 6 Pricing Checks

15question0.75 min

Which fact on today’s model landscape is least likely to survive until delivery?

Choose one: model names, list prices, or leaderboard positions.
Commit before the date stamp and refresh gates are revealed.

Model names, list prices, and rankings must remain equally neutral until the room commits; only the durable reading method is emphasised.

16data1.25 min

Small, Mid, and Frontier bands remain useful after named-model rankings expire.

Small is the capability band for bounded tasks under strong cost and latency pressure.
Mid is the default working band where most production volume lands.
Frontier is the highest-capability band for tasks that justify the premium.
Open-weight cuts across all three bands; it is a licence and deployment property.

Keep Small, Mid, and Frontier as equal capability bands; keep Open-weight as an orthogonal property that can intersect every band, with equal treatment for every provider example.

1. Takeaway

Capability bands provide a stable reading structure even as named models, prices, and availability expire.

2. Key Points

Small, Mid, and Frontier are CV teaching bands, not an industry-standard ranking.
Open-weight is orthogonal because licence and weight access do not determine capability size.
Equal visual treatment across providers prevents the snapshot from implying a preferred vendor.

3. Prep Notes

Lead with the controls: research snapshot 27 July 2026, price check 1 September, full refresh 28 September, and delivery confirmation 5 October. Use the scheduled Sonnet 5 move from $2/$10 to $3/$15 per million input/output tokens on 1 September as the expiry proof.

Figure source: CV original dated-landscape synthesis adapted from Anthropic Pricing and Models overview, OpenAI API Pricing and Models, Google Gemini API Pricing and Models, Google Gemma releases, and OpenAI Introducing gpt-oss, all retrieved 27 July 2026. Open-weight examples are limited to releases and licences verified from first-party sources.

17framework1.4 min

The 4 Leaderboard Questions reduce a ranking to evidence rather than a decision.

Source — who ran the evaluation, and can the work be inspected?
Setting — is the entry a model or one configured run?
Hardness — does the test still separate candidates without leakage or saturation?
Signal — is the gap larger than prompt, harness, and run-to-run variation?

Keep all four questions equally neutral; the sole jade emphasis is the outcome that a leaderboard produces a shortlist, not a decision.

1. Takeaway

A leaderboard can support a shortlist, but it cannot supply acceptance for the organisation's task.

2. Key Points

Ask who ran the evaluation and which configuration was tested before treating a rank as candidate evidence.
Check whether the benchmark still separates candidates and whether the reported gap exceeds evaluation variation.
Even a banking-shaped benchmark does not use this cohort's documents, task conditions, or acceptance threshold.

3. Prep Notes

Apply the four questions aloud to one entry. Use Arena, Artificial Analysis, and Epoch AI only to show that leaderboards measure different things. If challenged, Singh et al., The Leaderboard Illusion, NeurIPS 2025, documents arena-distribution distortion from unequal access; do not characterise the unverified Arena rebuttal.

Figure source: CV original 4-question framework informed by arena.ai, Artificial Analysis Intelligence Index v4.1, Epoch AI Benchmarking Hub, and Singh et al. arXiv:2504.20879, all retrieved 27 July 2026.

18framework1.7 min

The 6 Pricing Checks expose the modifiers that list price hides.

Split records input and output prices separately; Shape applies the workload's actual token ratio.
Context checks whether prompt length enters a separately priced tier.
Levers test cache and batch eligibility; Terms apply region, endpoint, and data-use conditions.
Contract substitutes the signed commercial rate for the public list rate.

Each card defines the check before showing its cost effect; direction and magnitude remain workload- and provider-specific until the signed rate is applied.

1. Takeaway

The six checks do not apply one generic discount; each changes a different part of the estimate, and the final cost emerges only after workload eligibility and the actual commercial rate are known.

2. Key Points

Split creates separate input and output terms, while Shape weights those terms by the workload's actual token mix; together they determine cost per call.
Context can introduce a threshold step, while Levers and Terms apply only when the specific tokens, jobs, region, endpoint, or data-use arrangement qualifies.
Contract is the final substitution: replace the public list rate with the organisation's signed rate, without assuming the direction or size of the difference.

3. Prep Notes

Start with the equation at the top, then read the cards left to right. Split turns one rate into input and output. Shape changes their weighting. Context introduces a step only where a provider publishes a threshold. Levers shorten the applicable bar only for eligible cached tokens or asynchronous jobs. Terms fork the estimate by region or data-use configuration. Contract replaces the public rate with the signed rate.

Do not teach caching or batch economics here; Topic 4 owns those mechanisms. The purpose is to recognise where a pricing page can change an estimate and to avoid applying a discount to the whole workload. Procurement remains outside scope. Figure source: CV original 6-check cost-transformation framework informed by Anthropic Pricing, OpenAI API Pricing and Prompt Caching, and Google Gemini API Pricing and Context Caching, all retrieved 27 July 2026.

19transition0.9 min

A shortlist remains provisional until current prices are multiplied by the workload you will run.

The snapshot names feasible candidates; it does not select one.
The leaderboard supplies evidence; it does not define task acceptance.
The pricing page supplies rates; production workload turns them into economics.

1. Takeaway

A shortlist stays provisional until current market evidence is multiplied by the workload the organisation will actually run.

2. Key Points

Names, availability, rankings, and rates expire, while the leaderboard and pricing-page questions remain reusable.
A long-input, short-output workload cannot be costed from one headline rate.
The organisation's own volume, token mix, context conditions, and operating assumptions determine the useful estimate.

3. Prep Notes

Close the dated-landscape quarantine by separating expiring facts from reusable reading tools. Hand directly to Topic 4 with volume × tokens × rate, corrected for workload shape and operating assumptions. Provider pricing mechanisms referenced here were retrieved from Anthropic, OpenAI, and Google pricing documentation on 27 July 2026.

T4 — Cost Optimisation

10 minutes · slides 20–27 · opens with silent estimate · memory hooks: The 3+2 Cost Baseline · The 4 Cost Moves

20question0.75 min

Which workload assumption is most likely to break your pilot cost estimate in production?

Write one: call volume, input/output mix, retries, or growth.
Choose the assumption you would measure first, not the one easiest to estimate.

Keep all four assumptions and empty checkboxes visually equal; no option is selected until the room commits.

1. Takeaway

A pilot cost becomes unreliable when its workload assumptions are carried into production without correction.

2. Key Points

Production volume, token mix, retries, and growth can each invalidate a pilot estimate.
A pilot usually observes a narrow workload slice rather than the distribution that production will create.
Peak traffic affects rate limits and self-operated capacity, but it is not a direct multiplier on a metered token bill.

3. Prep Notes

Give everyone ten silent seconds to write one assumption, then ask for two choices and the reason behind each. Do not announce one universal answer. Use a pilot that counted user requests but missed retry calls as the concrete example.

Figure source: CV original neutral question visual based on the cost drivers in T4 research §§4.5 and 9; provider pricing and usage documentation retrieved 27 July 2026.

21framework1.6 min

The 3+2 Cost Baseline corrects the pilot’s volume, token-mix, retry, and growth assumptions.

Calls — count model calls per business task, not only user requests.
Tokens — separate input from output because their rates and workload shares differ.
Rate, then retries and growth — price the workload now and the workload you expect.

The baseline is three priced inputs multiplied by two workload corrections; peak sits outside the metered-token formula.

1. Takeaway

The 3+2 baseline converts pilot usage into a reproducible production-cost assumption set.

2. Key Points

Calls, Tokens, and Rate establish the metered baseline; Retry and Grow correct it for production behaviour.
Input and output must be costed separately because a long retrieved context can outweigh a shorter, dearer output.
Peak belongs in rate-limit or self-operated capacity planning, not as another multiplier on a metered token bill.

3. Prep Notes

Write calls × (input tokens × input rate + output tokens × output rate) × retries × growth, then apply it to a retrieval-grounded request. Name 3+2 as a CV memory device rather than a published costing standard.

Figure source: CV original 3+2 Cost Baseline, informed by Anthropic, OpenAI, and Google first-party pricing documentation and T4 research §§4.5–4.7, retrieved 27 July 2026.

22comparison1.35 min

Prompt caching discounts repeated input; response caching avoids inference only when it safely hits.

Prompt cache — the model still runs; only the stable input prefix receives a lower rate.
Response cache — a safe hit returns stored output and avoids the model call.
Prompt-cache misses waste a write; response-cache mistakes serve a stale or wrong answer.

Do not merge the two paths: prompt caching changes the price inside a call, while response caching may remove the call entirely.

1. Takeaway

Prompt caching discounts repeated input, while response caching bypasses inference only when a stored answer safely matches.

2. Key Points

A prompt-cache hit still invokes the model and normally leaves output fully billed.
A response-cache hit can avoid inference, but its apparently relevant answer may be stale or wrong for the current context.
Hit rates must come from the organisation's own traffic because no defensible financial-institution semantic-cache rate was found.

3. Prep Notes

Ask which cache someone means before discussing savings. Contrast a repeated system prompt with yesterday's stored policy answer. As of 27 July 2026, Anthropic, OpenAI, and Google each published a prompt-cache read at 0.1× base input while output remained fully billed. Preserve [VERIFY V-T4-04] and measure actual logs.

Figure source: CV original two-path comparison based on first-party provider pricing and prompt-caching documentation plus Regmi and Pun, GPT Semantic Cache, arXiv:2411.05276, all retrieved 27 July 2026.

23concept1.05 min

Batch pricing reduces unit cost only when the business accepts the completion and retention conditions.

Use it only when the work can wait and does not require streaming.
Build submission, status tracking, result reconciliation, and retry handling.
Confirm expiry and data-retention terms before moving regulated workloads.

The discount and the operating conditions are one contract; neither side may be presented without the other.

1. Takeaway

Batch pricing is a service contract: the discount is earned by accepting asynchronous completion and owning the queue.

2. Key Points

A batch workload must tolerate the provider's completion window rather than expecting an interactive response.
Submission, polling, reconciliation, and retry operations are part of the real cost of the saving.
Retention and zero-data-retention eligibility can differ from the synchronous path and must be checked before use.

3. Prep Notes

Use overnight document-quality review as the example. On 27 July 2026, Anthropic, OpenAI, and Google each published a 50% batch discount; Anthropic and OpenAI used 24-hour completion windows, while Google targeted 24 hours and expired work at 48 hours.

Figure source: CV original contract diagram based on those first-party batch documents, retrieved 27 July 2026. Retain [VERIFY V-T4-09] because batch zero-data-retention exclusions and Anthropic's reported 29-day retention require current first-party confirmation before delivery.

24concept1.25 min

Routing reduces spend only when the cheaper path meets a measured quality threshold.

Start with rules you can explain: task type, document shape, or service tier.
Shadow a sample against the stronger path and measure disagreement on your task.
Escalate or roll back when the measured threshold is missed.

The quality gate and feedback path are part of routing; a price split without measurement is not a routing system.

1. Takeaway

Routing captures a price spread only when the cheaper path continues to clear a measured quality gate.

2. Key Points

Start with an explainable routing rule and shadow representative traffic against the stronger path.
Disagreement above the workload's measured threshold is a rollback signal rather than an acceptable saving.
Tiering changes unit price between model paths; Topic 5 changes the work performed inside a request.

3. Prep Notes

Use document-format normalisation as the cheaper-path rule. LLMRouterBench, submitted 12 January 2026, evaluated 400,000 instances across 33 models and found several commercial routers did not reliably beat a simple baseline. Preserve [VERIFY V-T4-07] because banking-shaped routing evidence was not found.

Figure source: CV original quality-gate loop informed by LLMRouterBench arXiv:2601.07206, RouteLLM arXiv:2406.18665, and FrugalGPT arXiv:2305.05176, retrieved 27 July 2026.

25framework1.35 min

The 4 Cost Moves change different terms of the bill, so workload shape determines their order.

Trim is the prior check — remove unnecessary retrieved, repeated, or generated tokens.
Cache · Batch · Route are peer moves that change calls, input price, or unit price.
Choose among the peers by workload eligibility and control cost, not a universal sequence.

Keep Trim visibly separate as a prior check and keep Cache, Batch, and Route equal, unnumbered peers selected by workload eligibility and control cost.

26diagram1.55 min

An illustrative $100,000 monthly bill falls to $46,080—not $30,000—when four savings compound.

Trim saves 10% of $100,000, or $10,000, leaving $90,000.
Cache saves 20% of $90,000; Batch then saves 20% of the remaining $72,000.
Route saves 20% of $57,600, leaving a final monthly bill of $46,080.
The total saving is $53,920, or 53.9%; adding the percentages would incorrectly claim 70%.

Every amount and percentage is illustrative; each saving is calculated from the amount remaining in the previous card, not from the original $100,000.

1. Takeaway

Compounding means each later saving is calculated from a smaller remaining bill. In this illustrative sequence, $100,000 becomes $46,080, not the $30,000 implied by adding percentages.

2. Key Points

Trim removes $10,000 from the original $100,000; the Cache calculation therefore starts from $90,000 and saves $18,000 rather than $20,000.
Batch starts from $72,000 and Route starts from $57,600, producing a final bill of $46,080 and a total saving of $53,920, or 53.9%.
The sequence and percentages are illustrative only: real eligibility and hit rates must be recalculated after each move so the same traffic is not counted twice.

3. Prep Notes

Walk the cards left to right and say each calculation aloud: $100,000 minus $10,000 leaves $90,000; 20% of $90,000 is $18,000, leaving $72,000; 20% of $72,000 is $14,400, leaving $57,600; 20% of $57,600 is $11,520, leaving $46,080. Then compare the amber and jade summaries.

Emphasise that this is arithmetic teaching, not a forecast. Cache, Batch, and Route are not a recommended sequence, and 10% or 20% is not an expected saving. Figure source: CV original illustrative residual-cost calculation derived from the compounding method in T4 research §4.6; it uses no current provider prices or claimed workload hit rates.

27transition1.1 min

After accidental spend is removed, reasoning effort still needs a task-level justification.

The baseline makes production assumptions explicit.
The four moves reduce calls, tokens, or unit price without a universal order.
The remaining question is whether extra inference work improves this task enough to justify itself.

1. Takeaway

Once accidental spend is removed, extra reasoning work must still justify its token and latency cost at task level.

2. Key Points

Model tiering changes the price per token, while reasoning effort changes the work and output consumed within a request.
Keep the model tier fixed when testing whether more effort improves the golden-set result.
The acceptable setting is the lowest effort that clears the workload's quality threshold within its latency and cost bounds.

3. Prep Notes

Use a borderline task as the bridge: hold tier constant, vary effort, and measure whether quality improvement earns the added token and latency cost. Then hand to Topic 5 to decide when extra work is warranted. The distinction is grounded in reconciled T4/T5 research and first-party pricing and effort-control documentation retrieved 27 July 2026.

T5 — Reasoning Effort

5 minutes · slides 28–31 · opens with provocation withdrawn

28question0.75 min

When does more reasoning make an answer worse as well as slower?

Provocation: more reasoning improves the answer.
Commit first: agree, disagree, or it depends — then name the condition.

Agree, it depends, and disagree must remain visually equal with no response preselected before the room commits.

1. Takeaway

More reasoning is not a general upgrade; it must earn its cost on the task's measured quality threshold.

2. Key Points

Reasoning effort is a per-request control rather than a permanent model category.
Extra work can engage distractors on simple or noisy tasks and make an answer worse as well as slower.
Provider documentation treats thinking as billed output and places it before the first visible answer token.

3. Prep Notes

Read the provocation, take a quick commitment, and wait eight seconds before withdrawing the universal claim. Use known-label message triage with irrelevant thread history as the counterexample. Sources: Gema et al., Inverse Scaling in Test-Time Compute, TMLR 2025, plus Anthropic, OpenAI, and Google reasoning documentation, retrieved 27 July 2026.

Figure source: CV original neutral provocation diagram synthesised from the T5 research; keep all three response positions equally weighted until the room commits.

29diagram2 min

Reasoning effort is a per-request control whose benefit can peak before its cost does.

Set effort on the model already selected; do not treat reasoning as a separate product family.
Thinking consumes output-token budget even when the full scratchpad is not returned.
The wait accumulates before the first answer token, so streaming cannot remove it.
Measure quality, thinking-token use, and first-answer latency on representative traffic.

Keep the curve qualitative with no universal threshold, and keep the latency delay entirely before the first answer token.

1. Takeaway

Reasoning benefit can peak before token and latency cost do, so the setting must be tuned on representative tasks.

2. Key Points

The reviewed model families expose reasoning as an effort or thinking control applied per request.
Extra hidden work can increase the bill and wait even when the visible answer remains short.
No universal effort threshold exists; log task quality, thinking-token use, and time to first answer.

3. Prep Notes

Use a scratchpad analogy: the model does extra work before delivering the answer. Explain that research supports a useful middle band and possible harm on low-complexity work, while the measured latency comparison came from one model family only.

Figure source: CV original qualitative curve informed by Anthropic thinking and effort documentation, OpenAI reasoning guide, Google Gemini thinking documentation, DeepSeek thinking-mode guide, Snell et al. 2024, Shojaee et al. NeurIPS 2025, and Artificial Analysis, all retrieved 27 July 2026.

30comparison1.5 min

Known-label message triage rarely justifies the reasoning budget that multi-system incident diagnosis may require.

Start low when an established label set and known evidence patterns define the routing decision.
Consider higher effort when diagnosis must join systems and test dependent fault hypotheses.
More effort can amplify distractors and framing errors on deceptively simple work.
Adopt it only when a representative evaluation set proves the quality gain exceeds delay and token cost.

Known-label message triage and multi-system incident diagnosis remain equal in visual weight; the measurement gate alone is jade and decides whether higher effort earns its budget.

1. Takeaway

Effort should follow the structure of the task: matching known evidence differs from forming and eliminating linked hypotheses.

2. Key Points

Known-label message triage usually begins at low effort because the route and evidence pattern are already defined.
Multi-system incident diagnosis may justify more effort when evidence conflicts across dependent systems.
The M02.1 golden set, not the example label, decides whether the higher setting earns its place.

3. Prep Notes

Present the cards as CV practitioner examples, not an industry taxonomy. Ask whether the model is matching evidence to an established route or forming and eliminating linked fault hypotheses. Gema et al., TMLR 2025, found distraction and framing failures; Shojaee et al., NeurIPS 2025, supports standard behaviour on low-complexity tasks, but no current adaptive-model replication was found; sources retrieved 27 July 2026.

Figure source: CV original task-fit comparison synthesised from the T5 research; keep the measurement gate, not either card, as the emphasis.

31transition0.75 min

A targeted reasoning budget completes the method; the room must now defend a choice without a model list.

Reasoning effort is a per-request control and a targeted instrument, not a default upgrade.
The defensible setting follows the task’s accuracy, latency, and cost thresholds.
The exercise now tests whether teams can state those thresholds before selecting.

T6 — Selection Exercise and Close

34 minutes · slides 32–39 · opens with role positioning · memory hooks: The 4-Part Decision Defence

32question1 min

Which axis will your role defend when the team cannot maximise all five?

Developers name the quality threshold they refuse to lose.
Infrastructure roles name the service obligation they must operate.
Risk and operations roles name the gate the design must pass.

Keep the three role lenses connected to one shared decision; no role owns the answer alone.

1. Takeaway

Role disagreement is useful evidence because each function sees a different failure mode, but the team must reconcile it into one use-case decision.

2. Key Points

Developers, infrastructure, and risk or operations roles are expected to defend different axes.
No function owns the complete answer, so disagreement should be recorded rather than suppressed.
The team must still commit to one dominant axis before a model name is admissible.

3. Prep Notes

Ask each role group to point to the axis it expects to defend, then hold ten seconds so the positions remain visible. If three roles name three axes, write all three without adjudicating and say the exercise will force reconciliation.

Figure source: CV original role-lens map based on the E02 role design and T6 exercise-validation research dated 27 July 2026.

33workshop1 min

Teams must record one dominant axis before naming a deployment pattern or model tier.

Score the use case before you look at a single model.
Write one dominant axis before any candidate appears.
Record the role disagreement rather than smoothing it away.
Defend the reasoning; the key does not prescribe one answer.

The sequence is binding: Score, Commit, Record, then Select.

34workshop1 min

Five workload briefs create different hard edges without prescribing the answer.

Confirm your team number and assigned workload title.
Read the printed brief only after roles are assigned.
Do not compare briefs until every team has committed its decision.

The pre-work board shows team number and workload title only.

1. Takeaway

Scenario distribution must preserve independent judgement by revealing titles without leaking another team's constraints or likely answer.

2. Key Points

Assign roles before distributing the printed workload brief.
Use the board only to confirm team number and title; do not narrate or compare scenarios.
Volume, latency, accuracy, residency, audit, complexity, dominant axis, and likely selection remain private until commitment.

3. Prep Notes

Point once across the five titles, distribute briefs, and move immediately to team reading. Decline questions about another team's brief so the no-spoiler boundary remains structural.

Figure source: CV original title-only scenario board based on the revised E02 pack; the no-spoiler boundary was established by T6 research and cross-topic reconciliation dated 27 July 2026.

35workshop14 min

The 5-Axis Decision Matrix captures the gate, threshold, trade-off, and role disagreement behind the choice.

Mark every hard gate before comparing feasible options.
Write the minimum acceptable threshold for each binding axis.
Name one deployment pattern and one durable capability tier.
Record the trade-off accepted and the disagreement that remained.

The fillable matrix must preserve the five axes and the six required outputs in an A4-ready structure.

1. Takeaway

The workshop produces a traceable decision by reconciling role views into a gate, threshold, selection, and accepted trade-off.

2. Key Points

Task: complete the 5-Axis Decision Matrix and prepare the four-part defence.
Time: 14 minutes—3 for role scoring, 5 for reconciliation, 4 for selection and stress test, and 2 for defence preparation.
Output: one committed axis, explicit gate and threshold, selected pattern or tier, and recorded disagreement.

3. Prep Notes

Start a fourteen-minute timer and announce five minutes remaining and two minutes remaining. Circulate only to ask which axis was committed before selection and which gate removed an option. Do not suggest a tier, deployment pattern, or instructor-key landing; reconciliation replaces majority voting.

Figure source: CV original A4 V05 exercise matrix derived from the session's 5-Axis Decision Matrix and E02 output contract.

36framework1 min

The 4-Part Decision Defence makes each report comparable without forcing the same answer.

Axis: the requirement that dominated.
Gate: the option-removing constraint or minimum threshold.
Selection: the deployment pattern and capability tier.
Trade-off: the cost or risk the team accepts.

Preserve the four anchors in order: Axis, Gate, Selection, Trade-off.

37workshop11 min

Capturing every axis and answer first preserves time for the disagreements that matter.

Sweep all five teams for axis and selection before hearing justification.
Choose the two most instructive disagreements for deeper challenge.
Challenge weak reasoning; do not correct a defensible choice.
Capture one unresolved role tension per team.

The live board records all five decisions before any team receives extended airtime.

38concept2 min

Different workloads should produce different selections when the framework is working.

A shared method does not require a shared answer.
Accuracy can be a measurable threshold or an unbounded risk.
Operational complexity usually eliminates or breaks ties after the use case sets the field.
The room's divergence is evidence only when the reasoning remains defensible.

One shared method must remain visibly connected to several defensible outcomes.

1. Takeaway

Different workloads should create different defensible selections when the framework is being applied to the job rather than a preferred model.

2. Key Points

Use the room's actual divergence as evidence instead of reading out the instructor key.
Labelled extraction and open-ended regulatory analysis differ in measurable thresholds and late-discovered error risk.
Operational complexity usually removes an infeasible pattern or breaks a tie after the use case has set the field; it is not a sixth axis.

3. Prep Notes

If the teams' reasoning supports it, compare Teams 2 and 4 to show how answerability changes the decision. Debrief the quality of the reasoning rather than conformity to the key.

Figure source: CV original divergence synthesis based on the E02 instructor key and T6 exercise-validation research dated 27 July 2026.

39transition3 min

Changing model behaviour comes last because prompting, retrieval, selection, and operating choices should be tested first.

Prompt before adding knowledge.
Retrieve before changing the model.
Select and cost the feasible option before accepting operational burden.
Change model behaviour only when the earlier rungs cannot meet the threshold.

Keep the escalation order intact and hand the final Change rung to the fine-tuning sessions.

1. Takeaway

Changing model behaviour comes last, after lower-cost prompt, retrieval, selection, and operating interventions fail the use-case threshold.

2. Key Points

The completed matrix records the selection and operating decisions that have already been tested.
Moving up the ladder requires evidence that the current rung cannot clear the threshold.
Change hands off to M02.2 and M02.4; it is not an automatic recommendation to fine-tune.

3. Prep Notes

Trace Prompt, Retrieve, Select, Operate, and Change, then return to the spine: task first, hard gates second, optimisation third. Ask what evidence would justify moving up one rung and close by linking the matrix to the next M02 interventions.

Figure source: CV original escalation ladder synthesised from the M02 session sequence and the approved M02.3 closing handover.

E02 — Selection Exercise Pack

Type

case-study pack

Print Partition

Participant Only Blocks

participant_parameter_note
project_brief
deliverables
phase_timings
blank_matrix_instructions
participant_scenarios
role_cards
presentation_format
rubric
print_pack.participant_materials

Instructor Only Blocks

instructor_key
cold_test
owner
ready_by
status
print_pack.instructor_materials

Rule

Render participant-only and instructor-only blocks as separate print products; never place instructor-only content in the participant pack.

Owner

Content

Pradeep Menon

Production

Production support

Review And Signoff

Pradeep Menon

Ready By

2026-09-28

Status

Draft complete; practitioner parameter review and cold test required before print

Cohort

Five teams of five in a mixed developer, infrastructure, and risk/operations cohort

Participant Parameter Note

All scenario volumes and timing thresholds are illustrative exercise assumptions, not published industry benchmarks; replace the flagged values with reviewed practitioner estimates before print.

Project Brief

Context

Each team receives one banking workload and the same blank 5-Axis Decision Matrix.

Challenge

Choose and defend a feasible model capability tier and deployment pattern without starting from a model list.

Constraints

Name one dominant axis before any candidate.
Remove options that fail a hard gate before optimising the survivors.
Use durable tiers and deployment patterns; do not name models or providers.
Record one disagreement among the roles rather than hiding it.

What Good Looks Like

The team produces a coherent chain from use-case requirement to gate, threshold, selection, accepted trade-off, and recorded role disagreement.

Ground Rules

The instructor tests reasoning and does not enforce conformity to the key.
No internet lookup, model shortlist, price sheet, or benchmark table is used.
A defensible alternative is valid when it passes the gates and owns its trade-off.
One team member must be ready to state the decision in thirty seconds.

Deliverables

One completed A4 5-Axis Decision Matrix per team
One dominant axis
One hard gate and minimum acceptable threshold
One deployment pattern and durable model capability tier
One accepted trade-off
One recorded role disagreement
One 4-Part Decision Defence for report-out

Phase Timings

Blank Matrix Instructions

Format

One A4 page, landscape, completed in ink

Header Fields

Team and workload
One-sentence task
Dominant axis
Hard gate

Axis Rows

Accuracy
Latency
Cost
Operational complexity
Data residency

Columns

Use-case requirement
Pass/fail gate or minimum threshold
Evidence available
Trade-off permitted

Decision Fields

Feasible deployment pattern
Durable capability tier
Reasoning-effort choice where relevant
Cost lever where relevant
Accepted trade-off
Role disagreement and resolution

Completion Rule

The dominant axis and all hard gates must be inked before the selection fields are opened.

Participant Scenarios

Which condition removes an option before price or capability is discussed?

Presentation Format

First Sweep

Each team has 30 seconds to state Team, Axis, and Selection; the instructor fills the live board before hearing extended justification.

Deep Dive

The instructor selects the two most instructive disagreements and gives each up to three minutes for the 4-Part Decision Defence plus challenge.

Cross Team Compare

Use the remaining time to compare one threshold, one accepted trade-off, and one role disagreement across the other teams.

Speaker Order

The participant whose role initially disagreed with the final choice delivers the defence.

A tested answer-quality and escalation threshold approved for customer-facing use; no invented universal percentage.

Likely Selection

A low-latency small or mid capability tier on a managed deployment, with response or semantic caching only for approved repeatable answers and explicit invalidation.

Accepted Tradeoff

Lower capability and cache coverage in exchange for predictable interactive response; stale answers remain a controlled risk.

Defensible Alternatives

A mid-tier managed endpoint if the smaller tier misses the golden-set threshold.
Routing rare complex questions to a higher tier while keeping the common path fast.

Likely Wrong Turns

Naming the smallest tier first because the task sounds simple.
Treating traffic concentration as a guaranteed response-cache hit rate.

Prepared Challenge

Your cache misses on the questions customers ask least often. What happens to latency and answer safety on that path?

Role Disagreement To Surface

Developers want broader answer coverage, infrastructure wants a bounded fast path, and risk wants aggressive escalation and invalidation.

Team

Expected Reasoning

Accuracy is a measurable threshold with human fallback, so cost becomes the variable to optimise across a high-volume overnight workload.

Dominant Axis

Cost

Hard Gate

The extraction must clear the approved field-level threshold and finish inside the overnight window.

Minimum Threshold

A golden-set field threshold with every failure below it routed to a human queue.

Likely Selection

A mid capability tier in a scheduled throughput-oriented pattern, using batch processing only if its service conditions meet the overnight deadline.

Accepted Tradeoff

Longer completion time and a human exception queue in exchange for lower unit cost.

Defensible Alternatives

A smaller tier if it passes the same labelled evaluation and preserves the deadline.
A synchronous or provisioned managed pattern if the available batch service cannot meet the window.

Likely Wrong Turns

Treating expensive errors as proof that the highest capability tier is required.
Assuming batch tolerance automatically makes every batch product feasible.

Prepared Challenge

What evidence would let the tier below pass, and how much human fallback can the overnight process absorb?

Role Disagreement To Surface

Developers want schema reliability, infrastructure wants throughput economics, and risk sets the downstream error threshold.

Team

Expected Reasoning

The task has a high accuracy bar because failure on complex repository work destroys adoption; capability is the model side of that accuracy requirement, not a sixth axis.

Dominant Axis

Accuracy

Hard Gate

The deployment must satisfy the approved source-code processing boundary and the tier must pass representative internal coding tasks.

Minimum Threshold

A task-weighted internal evaluation covering the median request and the hardest repository changes.

Likely Selection

A frontier capability tier through a managed deployment that satisfies the code boundary, with lower-tier routing for measured routine work where justified.

Accepted Tradeoff

Higher cost and possible provider dependence in exchange for useful performance on complex tasks.

Defensible Alternatives

A managed open-weight frontier tier if it passes the same internal evaluation and boundary.
A mid-tier default with escalation when the task classifier can be evaluated reliably.

Likely Wrong Turns

Treating an internal tool as exempt from data-boundary decisions.
Choosing a leaderboard leader without testing the institution's task distribution.

Prepared Challenge

How would you prove the tier below is sufficient for the median request without hiding failures on the hardest work?

Role Disagreement To Surface

Developers prioritise difficult-task quality, infrastructure questions cost and operability, and risk constrains where proprietary code may be processed.

Team

Expected Reasoning

Accuracy dominates because there is no complete ground truth and errors emerge late; low volume makes the cost of additional reasoning non-binding rather than irrelevant.

Dominant Axis

Accuracy

Hard Gate

Every claimed obligation must link to source evidence and remain subject to specialist review.

Minimum Threshold

Traceable evidence coverage and review acceptance on a curated set of past changes; no fabricated universal accuracy score.

Likely Selection

A frontier capability tier at high reasoning effort for genuinely ambiguous cases, with lower effort on straightforward items and mandatory human review.

Accepted Tradeoff

Higher per-item cost and latency in exchange for margin on low-volume, hard-to-evaluate analysis.

Defensible Alternatives

A mid-tier model for straightforward notices with escalation to a frontier tier at higher effort.
A retrieval-led workflow that limits generation to evidence-linked synthesis before raising reasoning effort.

Likely Wrong Turns

Treating high stakes as sufficient proof that maximum effort always helps.
Skipping cost rather than showing that it does not bind at this volume.

Prepared Challenge

Which easy case becomes worse when reasoning effort is set high, and how will the workflow detect it?

Role Disagreement To Surface

Risk buys margin, developers demand evidence and evaluation, and infrastructure resists applying expensive effort to every item.

Team

Expected Reasoning

In-jurisdiction processing is a hard gate that leaves several managed and self-operated patterns; operational complexity and reproducibility then break the tie.

Dominant Axis

Data residency

Hard Gate

Inference must remain inside the permitted jurisdiction and the application must retain an evidence-linked production record.

Minimum Threshold

Documented processing location, retention behaviour, evidence linkage, and version identification for every narrative.

Likely Selection

Any verified in-jurisdiction pattern—regional managed inference, sovereign or perimeter-managed deployment, or self-operated weights—followed by an explicit operating-responsibility decision.

Accepted Tradeoff

A narrower provider and deployment field, possible regional cost or version constraints, and explicit operating ownership.

Defensible Alternatives

Regional managed inference when the operating team cannot support self-operation and contract controls satisfy the gate.
A perimeter-managed deployment when network isolation matters but the institution does not want to operate the model stack.
Self-operated open weights when version permanence is required and the operating capability exists.

Likely Wrong Turns

Treating self-operation as the only route through a residency constraint.
Assuming open weights explain why the narrative said what it said.

Prepared Challenge

Name three deployment patterns that pass the processing-location gate, then tell us which operating obligation removes two of them.

Role Disagreement To Surface

Risk sets the location and evidence gates, infrastructure decides which surviving pattern is operable, and developers own grounding and reproducibility.

Cold Test

Required

True

Readers

Reader Condition

Readers have not seen the instructor key, the research document, or another reader's work.

Procedure

Each reader completes all five scenarios using only the participant pack and the fourteen-minute team-work limit per scenario.
Record the order in which each reader named an axis and a candidate.
Compare dominant axis, gate, tier, deployment pattern, and accepted trade-off across scenarios.
Rewrite any scenario that names its answer, silences a role, or repeatedly collapses onto another scenario's reasoning.

Pass Conditions

Every scenario yields a defensible use-case-first chain without a model list.
The five workloads produce materially different selections or justifications rather than one organisational default.
Team 2 is distinguished from Team 4 by measurable threshold versus unbounded accuracy risk.
Team 5 permits more than one residency-satisfying pattern and makes operating responsibility the tiebreak.
All five role cards contribute a real decision in every scenario.
No participant can infer the instructor key from answer-signalling language.

Failure Action

Revise and repeat with three new cold readers; analysis by the content team does not substitute for the test.

Complete By

2026-09-28 before print

Print Pack

Participant Materials

Ten one-paragraph scenario briefs: one complete assigned set of five plus one complete spare set of five
Thirty blank A4 matrices: twenty-five participants plus five spares
One role-card set per team plus one spare set
Ten 4-Part Decision Defence reference strips: one per team plus five spares

Participant Quantities

Scenario Briefs

Blank A4 Matrices

Role Card Sets

Decision Defence Strips

Instructor Materials

One separately printed instructor key
One live report-out capture board
One cold-test record and sign-off sheet

Separation Rule

Instructor materials are never distributed or placed in the cohort's shared participant folder.

Appendix — dated figures and open items

Perishable provider detail is quarantined here and on T3's single dated landscape slide. Full evidence registers remain in the six topic research documents.

Dated figures

Topic	Figure	Value	Source	Refresh
T2	fig-t2-shortlist-map	Illustrative placement of gpt-oss-120b, Qwen3, and Claude Opus 4.8	OpenAI gpt-oss; QwenLM Qwen3; Alibaba Cloud Model Studio; Anthropic Claude Opus	5 October 2026
T3	fig-t3-landscape	Research snapshot 27 July 2026; Small, Mid, and Frontier capability bands with Open-weight shown orthogonally; Claude Sonnet 5 scheduled list-price change from $2/$10 to $3/$15 per million input/output tokens on 1 September 2026	Anthropic Pricing; supporting names and availability from Anthropic Models, OpenAI Models, Google Gemini API Models, Google Gemma releases, and OpenAI gpt-oss release pages; all retrieved 27 July 2026	1 September 2026 price check; 28 September 2026 full landscape refresh; 5 October 2026 delivery confirmation
T4	fig-t4-two-caches	Prompt-cache read multiplier published as 0.1× base input by Anthropic, OpenAI, and Google on 27 July 2026; output remained fully billed	Anthropic Pricing and Prompt Caching; OpenAI Pricing and Prompt Caching; Google Gemini API Pricing and Context Caching; retrieved 27 July 2026	5 October 2026
T4	fig-t4-batch-contract	Batch discount published as 50% by Anthropic, OpenAI, and Google on 27 July 2026; provider completion, expiry, and retention terms differ	Anthropic Batch Processing; OpenAI Batch API; Google Gemini Batch API; retrieved 27 July 2026	5 October 2026

[VERIFY] items

T2 — Id: G-T2-04 · Item: Check for a vendor-neutral API versus self-operation crossover analysis; if none exists by delivery, retain the explicit statement that the shape depends on disclosed workload assumptions. · Source: Academic venues, industry conference tracks, and cloud-economics research · Due: 2026-10-05
T3 — [VERIFY V-T3-01] Refresh every volatile model name, availability state, and price from first-party provider pages on 28 September 2026, with a final confirmation on 5 October 2026.
T3 — [VERIFY V-T3-02] Confirm on Anthropic’s pricing page that the Claude Sonnet 5 change to $3/$15 per million input/output tokens took effect on 1 September 2026.
T3 — [VERIFY V-T3-03/V-T3-09] Add no further open-weight example until its publisher’s release page and weight licence are both confirmed.
T3 — [VERIFY V-T3-05/V-T3-06] Recheck whether Gemini 3.1 Pro left Preview and whether an available gpt-5.6-pro variant exists before assigning frontier examples.
T3 — [VERIFY V-T3-08] Verify the Arena response to The Leaderboard Illusion before characterising its position; otherwise omit it.
T3 — [VERIFY V-T3-10] Confirm the exact October delivery date; the 28 September T-3-week checkpoint currently assumes 19 October 2026.
T4 — [VERIFY V-T4-01/T4-02] Recheck all provider prompt-cache and batch multipliers, limits, and expiry terms on 5 October 2026; coordinate with T3 so current prices cannot disagree.
T4 — [VERIFY V-T4-04] If no independent production semantic-cache evidence from a financial institution is found, retain no hit-rate percentage and teach measurement on the organisation’s own logs.
T4 — [VERIFY V-T4-05] Measure Japanese policy text with current provider token counters before delivery if the trainer wants to discuss language-specific token inflation; no multiplier is currently defensible.
T4 — [VERIFY V-T4-07] Treat routing performance on banking-shaped tasks as unverified; keep the quality gate and shadow measurement as the recommended control.
T4 — [VERIFY V-T4-09] Re-verify Batch API zero-data-retention exclusions and Anthropic’s reported 29-day retention directly from current first-party data-control documentation before putting the claim on screen.
T5 — [VERIFY V-T5-03] Check for a 2026 replication of overthinking or inverse-scaling findings on current adaptive-thinking models before delivery; if none exists, retain the explicit evidence limitation.
T5 — [VERIFY V-T5-05] Recheck provider pricing pages by 5 October 2026 for any newly separate reasoning or high-compute SKU before stating that effort normally shares the selected model’s rate card.
T5 — [VERIFY V-T5-06] Re-verify the qualitative front-loaded latency shape against a current same-model reasoning-on/off pair; do not add exact seconds unless refreshed and quarantined appropriately.
T5 — [VERIFY V-T5-08] Confirm that Team 4’s T6 instructor key accepts a frontier- or mid-tier model at high reasoning effort rather than requiring a separate reasoning-model product.
T6 — Id: V-T6-01 · Item: Replace or formally approve the illustrative 8,000-document weekday volume for Team 2. · Source: Practitioner in onboarding or operations · Due: Before print
T6 — Id: V-T6-02 · Item: Replace or formally approve the illustrative 18,000-request weekday volume for Team 3. · Source: Enterprise engineering-productivity research or practitioner estimate · Due: Before print
T6 — Id: V-T6-03 · Item: Validate the one-second first-response and four-second completion thresholds used as exercise assumptions for Team 1. · Source: T1 human-factors research and provider latency documentation · Due: 2026-09-30
T6 — Id: V-T6-06 · Item: Re-verify managed processing-residency patterns in the instructor key; do not add provider-specific facts to participant materials. · Source: Current provider deployment and data-control documentation · Due: 2026-10-05
T6 — Id: V-T6-08 · Item: Complete the required three-reader cold test and repeat after any material scenario rewrite. · Source: Build team cold-test record · Due: Before print

Sources & references

See module-02/02.research/CVNUS_APC_M02.3_T*_Research.md and CVNUS_APC_M02.3_RECONCILIATION.md.

CV-NUS_M02.3-Codex · Framework 1 + Framework 2 · Engineering Intelligence. Creating Value.