Ismail's Credibility Checker
From the documentation of the field
Ismail's Credibility Checker is an eight-rule protocol for judging whether a specific claim, theory, study or source is well supported. It separates validity from prestige, credentials and institutional affiliation, and it is packaged as a Claude Skill so that it can be run. It is one of two protocols in the IsmailsProtocols repository, and this site uses it on its own writing, as review discipline describes.
Why it exists
The protocol rests on two named failure patterns.
The first is epistemic trespassing, a term from the philosopher Nathan Ballantyne. A credential earned in one field is carried into another, where it brings an authority the holder has not earned. The second is theory-ladenness, an idea associated with Norwood Hanson. Experts in the same domain reach different conclusions from the same evidence, because their training shaped what they treat as relevant. In the first pattern a credential does not match the domain. In the second it matches the domain and still does not converge with an equally credentialed peer. In both, a credential is a poor guide to whether a particular claim is right, so the protocol evaluates the claim.
The eight rules
The rules are listed in dependency order, which is not the order in which they were first drafted. Each rule depends on the ones before it.
| Step | Rule | What it asks |
|---|---|---|
| 1 | Assume the null position | Treat the claim as an untested hypothesis, state exactly what it asserts, and judge it as if its source were anonymous. |
| 2 | Separate the medium from the message | Clarity or density of presentation says nothing about truth. Restate the claim in plain language before judging it. |
| 3 | Audit the definitions | Check that the claim's definitions map onto the thing it says it is about, and that each cited source bears on the exact point it is cited for. This rule gates everything after it. |
| 4 | Establish the hierarchy of verification | What kind of claim it is sets the ceiling. A deductive claim can reach machine-verified proof. An empirical claim can reach consilience, which is independent lines of evidence converging. Neither outranks the other, and applying a formal result to the real world is a separate empirical claim. |
| 5 | Recognise social signals as marketing, not truth | Citation counts, venue prestige, affiliation and the volume of discussion measure attention. They say nothing about accuracy. |
| 6 | Institutions do not confer infallibility | A credential shows what someone was trained in. It does not show that they are free of error. Weigh an objection on its content. |
| 7 | Burden of proof, and tools that shift it | Whoever asserts a claim carries the burden of supporting it. Tools that did not exist before now let an evaluator close some evidence gaps personally, once their output is checked against the source. |
| 8 | Maintain epistemic agility | Treat every rung, ceiling and threshold as revisable as better verification tools appear. |
Rule 3 runs before the hierarchy check because a high tier assigned to a mis-specified claim is a strong proof of the wrong thing.
Two depths
General Discovery is the default. It runs rules 1 to 7 against the claim and the sources it cites directly. First-Principles runs on request. It extends the same check recursively, taking every load-bearing source and the sources those rely on, and continuing toward primary sources or a stopping point. First-Principles contains General Discovery and goes deeper with the same checks. A report always names the depth it ran at, and a claim that clears General Discovery is reported as such, with an offer to go deeper.
Four severities
A report uses one of four outcomes. Disqualifying means the claim cannot be assessed as presented, usually because a definition fails rule 3. Evidentiary Gap means the definitions hold, but the claim falls short of the tier its own scope calls for. Caution means the claim clears its tier, but something about how it is supported deserves a second look. Unresolved means a First-Principles trace reached a depth, search or time limit before it reached a primary source, so the claim is neither confirmed nor disqualified.
Four outcomes are used instead of pass and fail because a binary collapses distinctions the rules already make. A clean result is reported as clearing every check run at that depth. It is never reported as the claim being true. A flag never blocks anything. It annotates the claim it concerns, and the rest of the response proceeds.
The evidence behind the rules
Every rule has a citation trail in the protocol's evidence audit. The audit includes the published research that supports each rule and the studies that complicate it. The protocol's rule content is released under CC BY 4.0.
| Tools | |
|---|---|
| Pages | Review DisciplineIsmail's Credibility CheckerIsmail AI Review |