Why one aggregate score doesn't tell you what to fix
A retrieval pipeline can lose the evidence a query needs at more than one stage: the chunker can separate a fact from the context that qualifies it, the retriever can rank the right chunk too low, or the final selection step can drop it. A single end-to-end retrieval score can't tell you which of those happened.
Project 43 addresses that by building a synthetic corpus where the evidence span each query needs is known in advance, so a miss can be attributed to a specific stage — chunking, retrieval or ranking — rather than reported as one undifferentiated failure rate.
A controlled comparison: heading context
One controlled comparison held the chunk boundaries fixed and varied only whether each chunk retained the heading context above it. Across the corpus, heading severances — cases where a fact was separated from the heading that qualified it — fell from 45 to 0 between the two configurations.
Because the chunk boundaries themselves did not change, the result isolates heading-context preservation as the variable that mattered in this comparison, rather than where the text happened to be split.
Registering predictions before measuring
The evaluation harness required stating an expectation before running the measurement that would confirm or refute it — an explicit predict-then-measure discipline rather than narrating results after the fact.
Of 12 predictions registered this way, 7 held and 5 were contradicted by the data. Keeping those contradictory findings visible makes the report auditable and limits post-hoc interpretation.
The statistical ceiling
The permutation-test budget used for significance testing could not clear the strictest multiple-comparison correction at the sample sizes involved. That bounds how confidently any single "improvement" can be claimed from this experiment — some comparisons are suggestive rather than statistically conclusive.
Scope
The retrieval methods evaluated are lexical and latent-semantic (BM25/TF-IDF-style scoring and LSA), not a trained neural retriever, and there is no answer-generation stage in scope — only retrieval and ranking are measured. The corpus is synthetic, built specifically to have known evidence spans.
What this demonstrates is a controlled attribution methodology for retrieval pipelines, built around the rqlab experiment-orchestration and reporting modules, not a measurement of production RAG answer quality.