Graph + Vector Hybrid Reasoning

Graph traversal vs. vector retrieval, measured

Java 21 · 133 tests · 6-stage verification · report regenerated from code

A question that gets asked as though it has an answer -- should we use a knowledge graph or a vector database? -- and a measurement showing it is the wrong question, because the two techniques answer different things and the working system needs both in specific, non-interchangeable roles.

The domain is sanctions screening over corporate ownership filings, chosen because it has the property that makes this interesting: the fact you need is never written down. No filing says "Ashford Components buys from a company controlled by a sanctioned person." That conclusion exists only as the composition of five separate assertions made by five parties who never spoke to each other.

The finding, in one paragraph

Retrieval finds text; traversal derives facts. On questions whose answer is written down, they are the same tool and retrieval is cheaper. On questions whose answer is a composition, retrieval fails -- not gradually, and not for want of a better encoder, but because it must retrieve a conjunction of passages that do not individually resemble the question. On questions outside the graph's schema, traversal cannot answer at all. And the graph itself is built by an embedding making similarity judgements about identity, so the two were never alternatives: the working architecture is vector for identity and graph for traversal, and the failure mode of the combination is a well-cited path between the wrong nodes.

What is measured

Nine predictions were registered in code before the measurement that settles them. Four held; five were contradicted. docs/results.md is generated by Experiments.java and byte-compared against a fresh run by the test suite, so it cannot drift from the code.

The experimental design, and why it is deliberately unfair to the graph

Both systems execute the same query plan through the same executor. They differ in exactly one thing: which graph they are handed. The graph system gets the full extracted graph; the vector system gets the subgraph induced by its top-k retrieved documents.

This makes the retriever's reader perfect. It never hallucinates, never misreads a passage, and composes facts flawlessly across everything it was given. Real readers do none of those things. So every number reported for the vector system is an upper bound on what any RAG pipeline over this corpus could achieve, and every failure below is a floor rather than an artefact of a weak baseline.

Three further choices push in the same direction:

• Plans are handed to both systems, not parsed. A parser's errors would be indistinguishable from a substrate's limits -- a wrong answer could mean "retrieval cannot support this question" or "the parser mangled it", and the report could not tell you which.
• The corpus contains an OPEN question class specifically so the graph can lose. The extractor knows five relations; the corpus contains facts outside that schema.
• Distractor documents are indexed alongside the factual ones, because a real index cannot tell them apart. An empty result is obvious; a confident irrelevant one is not.

Three results worth the click

The conjunction problem (§3). A four-hop question needs nine documents simultaneously inside one top-k window. The documents in the middle of a chain -- "Baltic Freight AG is controlled by Silverline Holdings SA" -- have no lexical or semantic relationship to the question that needs them. They are relevant by composition, and similarity is the only thing an index knows. Raising k improves the easy questions, does nothing for the hard ones, and costs monotonically more. §4 proves the cause is retrieval and not reasoning by handing the vector system an oracle: it then scores exactly what the graph scores.

The graph is built by an embedding (§5). You cannot build a graph over messy filings without deciding that "Halcyon Trading Co" and "Halcyon Trading Company" are one node and that "Meridian Shipping Ltd" and "Meridian Freight Services" are two. Sweeping the resolution threshold shows the two error types are not symmetric: across the whole sweep, thresholds that merge produced a false alarm; thresholds that only split produced none, ever. Splits fail by omission, merges fail by invention -- and in compliance a missing answer gets escalated while a fabricated one gets acted on.

Provenance is not the safety property it looks like (§9). Every edge carries the id of the document asserting it, so every answer arrives with a citable chain. But §5 produced a path whose every link was a real document and whose conclusion was false: the error was in the node identity, not in any edge. A provenance chain is evidence that the edges were asserted. It is not evidence that the entities were correctly resolved.

Running it

.\demo.ps1     # the argument in about 90 seconds
.\test.ps1     # 6 stages: compile, tests, report freshness, determinism, mutation, secrets

Requires JDK 21 and Maven. No network access is needed at runtime, no database, and no model API -- see docs/adr/001-lexical-embeddings.md for why the encoder is lexical and, more importantly, for the measurement showing that choice does not affect any conclusion drawn here.

Layout

Verification

test.ps1 runs six stages, each able to fail independently:

1. compile clean
2. 133 tests across 8 suites
3. report freshness -- docs/results.md byte-compared against a fresh generation
4. determinism -- three separate JVMs, output SHA-256'd and compared
5. mutation -- six single-token edits to production code, each removing a guarantee the report depends on; 6/6 killed
6. secrets scan

Stage 5 is the one that matters. A project whose entire argument is measure it, don't assume it cannot ship a test suite nobody has checked the sensitivity of. A seventh mutation candidate was tried, survived, and was kept in the script as a comment rather than deleted, because it survives for a provable reason -- it is an equivalent mutant. That distinction is recorded in docs/portfolio/04-what-the-tests-caught.md along with the four real defects the suite found.