Evaluation modules

HalluDX — Coming soon

Preview the ground-truth and consistency checks used to calculate Hallucination Risk Rate.

HalluDX is designed to detect, quantify, and localize factual hallucinations in LLMs and GenAI applications. It evaluates factual grounding across direct questions and multiple variants under black-box conditions, without requiring model internals.

What HalluDX measures

HalluDX combines a direct ground-truth check with a consistency check across different variants of the same input. Together, they measure whether the target covers the expected facts across direct and repeated tests.

Ground-truth check: Tests whether a response contains the expected claim for a specific subject and attribute in an approved source.
Consistency check: Tests expected-claim coverage across a user-selected set of prompt variants for one topic, typically 5–10 variants.
Expected claim: A factual detail extracted from the supplied ground-truth source and used as the evaluation reference.
Primary output: Hallucination Risk Rate, calculated from hallucinated or uncovered expected claims.

Ground-truth check

AIDX converts the supplied ground-truth document into subject-attribute-claim records. Each record identifies a subject and one expected factual claim. A question is generated from that record, sent to the target, and judged against the expected claim.

Check 01Ground-truth claim coverage

Ground-truth sourceApproved factual reference
Claim recordSubject · attribute · expected claim
Factual questionGenerated from the record
Test targetBlack-box request
ResponseCaptured for evaluation
Claim judgeCovered or not covered
Hallucination Risk RateLower is better

Consistency check

For one topic and its ground-truth claim bundle, the user chooses how many prompt variants to test, typically 5–10. Each response is evaluated for expected-claim coverage so HalluDX can identify omissions, contradictions, replacements, or unsupported expected claims across equivalent intents.

Check 02Claim coverage across prompt variants

Topic + truth bundleExpected response behavior
Prompt variantsUser-configured · typically 5–10
Test targetOne response per variant
Response setGrouped by topic
Claim coverageExpected facts present or absent
Hallucination Risk RateLower is better

How claims are judged

The evaluator receives the target response and the expected claim list. Each expected claim is marked as covered or not covered. Under this method, a claim is counted as hallucinated or uncovered when it is:

Outcome	Method treatment
Contradicted	The response conflicts with the expected factual claim.
Omitted	The response does not provide an expected claim.
Replaced	The response substitutes a different factual detail.
Not otherwise supported	The expected claim cannot otherwise be supported by the response.

Hallucination Risk Rate

Primary metricHallucination Risk Rate = hallucinated or uncovered expected claims ÷ total expected claims

The same claim-level ratio can be aggregated for a case, subject, topic, or the complete evaluation. Lower is better.

Result scope	Direction	Interpretation
Ground-truth check	Lower is better	Share of expected claims for a subject that are not correctly covered.
Consistency check	Lower is better	Share of expected claims omitted, contradicted, replaced, or otherwise unsupported across the selected variants.
Overall evaluation	Lower is better	Share of hallucinated or uncovered expected claims across every evaluated case.