v0.4.0 Measurement Runbook¶

Status: COMPLETED 2026-07-24. All runs below were executed and the results published in athena-icd-snomed.md (ablation ×6 + USAGI via the UI-export bridge). This runbook is kept as the reproduction recipe.

The v0.4.0 code ships the harness; these runs produce the numbers. Nothing in the docs may replace a placeholder marker except output from these commands — no estimated or remembered figures.

Total compute: roughly a day of wall-clock on a laptop (model downloads on first run: SapBERT ~600 MB + cross-encoder ~90 MB).

Prerequisites¶

pip install -e ".[polars,quality]"      # dev install of the v0.4.0 branch
java -version                            # 17.x — required for USAGI only
export ATHENA_DIR=docs/notebooks_examples/example_assets/vocabulary_download_v5

Comparability guard: the published v0.3.x rows used Athena release 2026-04-30. If $ATHENA_DIR is a different release, DO NOT mix rows — re-run every backend below so the whole table shares one release, and set --athena-release-date accordingly on every invocation.

Step 1 — Reranker ablation (6 runs) + fixed-blend re-measurement¶

The v0.4.0 blending fix changes all three backend rows; these runs are both the ablation AND the Task-5 re-measurement.

OUT=src/portiere/benchmarks/athena_icd_snomed/expected_results_v040.json
for be in bm25s faiss hybrid; do
  portiere benchmark athena-icd-snomed --backend $be --athena-dir "$ATHENA_DIR" --out "$OUT"
  portiere benchmark athena-icd-snomed --backend $be --no-reranker --athena-dir "$ATHENA_DIR" --out "$OUT"
done

Fill: the ablation table in athena-icd-snomed.md (6 rows), and the routing field of each run feeds the throughput-proxy paragraph in docs/comparison.md.

Pre-registered interpretation (do not re-litigate after seeing numbers): - reranker hurts (off > on for ≥2 backends) → proceed to Step 3 with the swap as headline - neutral (|Δtop-1| < 0.01) → consider RerankerConfig(provider="none") default; document - helps → retrieval is the ceiling; throughput framing leads; Step 3 optional

Step 2 — USAGI baseline (1 run)¶

Note (as executed): no published USAGI JAR supports headless batch mode — the row was produced via the UI-export bridge instead:

python3 scripts/usagi_ui_bridge.py gen-input --athena-dir "$ATHENA_DIR" --out usagi_input.csv
# USAGI UI: import → auto-map (no corrections) → export source_to_concept_map
python3 scripts/usagi_ui_bridge.py score --athena-dir "$ATHENA_DIR" \
    --export usagi_export.csv --out "$OUT"

Fill: the USAGI TBD row. Then delete the "populated by a follow-up commit" paragraph and write one honest sentence about the result, whichever way it fell.

Step 3 — Domain reranker spike (TIMEBOXED)¶

Write the stop date here before starting: spike ends: ____-__-__ (suggested 2 weeks). Candidates (verify licence permits recommending in an Apache-2.0 project before evaluating):

ncbi/MedCPT-Cross-Encoder
a BioLORD-based cross-encoder
a SapBERT-derived cross-encoder

Each candidate, on the Step-1 winner backend (--reranker-model shipped in v0.4.0 exactly for this spike):

portiere benchmark athena-icd-snomed --backend bm25s \
    --reranker-model ncbi/MedCPT-Cross-Encoder \
    --athena-dir "$ATHENA_DIR" --out candidates_medcpt.json

Decision rule (pre-registered): adopt as default iff top-1 beats the incumbent by > 0.01 with MRR no worse; tie → keep incumbent. Update RerankerConfig.model in src/portiere/config.py + CHANGELOG migration note if adopted, then re-run Step 1's three reranker-on rows.

If nothing wins by the stop date: publish the negative result in the benchmark doc and stop. That outcome still ships.

Step 4 — Lock¶

Replace every placeholder marker in athena-icd-snomed.md and comparison.md. ✅
Promote expected_results_v040.json to expected_results.json (keep the old file as expected_results_v03x.json for the historical record).
Reconcile README / benchmark doc / CHANGELOG so all quote the same numbers.
Re-run the placeholder-marker grep gate → must return nothing before tagging the release.