What properties should the claim graph have?

I've been thinking a lot about the different properties we want out of the agentic system that produces the claim graph, and how to test for them.

Here is the list, with accompanying discussion of how I plan to test them and how I might approach it as a technical problem within Minerval's Constitutional-bureaucratic architecture.

Some principles come with hard mode, where you're dealing with bad faith/adversarial sources.

1. Idempotency

The same ingested document should produce approximately the same list of canonical claims, or, in Minerval's case, map every claim in the document onto the claims that got ingested and canonicalized the first time around.

Hard mode is when the inputs are small variations on the same sources and arguments, but not identical, and thus not amenable to mechanical deduplication. Claim stewards' judgments must be robust even to adversarial flooding of the zone. Logging of past decisions (as in Minerval) can help with this, but Minerval's rules here remain untested.

2. Reproducibility

Assessment/decomposition should produce approximately the same (in the eyes of an LLM-as-judge) arguments and subclaims, and arrive at approximately the same credence, if run multiple times independently.

3. Model convergence

Swapping the LLM doesn't drastically change outcomes. Some difference is fine, but systematic differences should at least be measured. Differences in credences between models put an upper bound on the calibration of the whole system. One can employ LLM-as-judge and credence differences to evaluate the smaller model, based on the strong prior that the frontier model strictly dominates on quality.

4. Path independence

Switching the order of contributions/ingestions does not substantially change the canonical claims and argument structures.

5. Cascade stability

This is relevant for any attempts at universal canonical claim graphs, as Minerval aims to be. While it is not that computationally expensive to mechanically propagate credences through an arbitrarily large Bayesian DAG, you can't do that if (as with Minerval) your credence metric for each claim is up to the judgment of an LLM agent.

Updates have to generally peter out after a few generations, as an update to a sub-sub-subclaim is likely too small to change the overall assessment and therefore notify dependent claims. This is a property that can ultimately be measured with statistics, but is likely highly sensitive to prompt changes and has effects on the cost of operating such a system.

6. Frame invariance

The same claim, framed in different ways, should produce the same canonical claim. Hard to get right, but should be easily testable and measurable using synthetic data (take claims, ask an LLM to produce versions with different biased framings, run canonicalization on each, take the cosines).

7. Source quality invariance

The claim graph's credence of a claim should not depend on the rhetorical or logical strength of the source in which that claim was first ingested or articulated. The claim must be assessed on its merits, not the merits of the argument in which it is raised.

While good arguments against them must be taken seriously into consideration, claims can be false without having any vocal contesters in the sources (as is common with, for example, conspiracy theories, where often there is simply no engagement from sources that could argue persuasively against them), and they can be true even if their proponents are otherwise discredited.

8. Internal coherence

A parent claim's credence is consistent with its children's under the argument structure. This is mechanically true wherever credences are computed with formal aggregation algorithms.

Where more than one argument for or against a claim exists, coherence is undecidable without the exercise of judgment on the merits of the arguments, unless arguments are rigorously logically structured. Such rigor is desirable in the limit, but I believe not feasible today on Horatio grounds, and risks brittleness.

9. Termination

Propagation of updates should not result in loops. This is managed the same way as cascade stability in Minerval; via prompts/constitutional principles that discourage notification/enqueuing of dependent claims' stewards over trivial changes.

10. Polarity invariance

A proposition and its negation must match to the same claim.

11. Conservation of expected evidence

Assignment of prior credences to claims that have not yet been assessed should be calibrated such that they are close to the credences after assessment. Systematic gaps here would be evidence of a problem.

12. Calibration

Intractable to mechanically test for in principle for almost all claims that matter, except for the narrow region of claimspace comprising claims that routinely get settled decisively: predictions. This makes FutureSearch's angle (grading epistemic systems on resolvable predictions) seem especially promising, since my hunch is that rationality is "holomorphic" in the sense that if an epistemic system can do really well at predicting near-future events, that is pretty strong evidence that it is broadly well-calibrated.