Core concepts
Evaluation lifecycle
Follow an AIDX evaluation from a defined test boundary to traceable findings, a report, and controlled version comparison.
The AIDX evaluation lifecycle turns one-off testing into a repeatable evidence process. It connects the tested system, observed behavior, risk findings, and report so teams can act on results and measure what changes over time.
Why the lifecycle matters
One evidence chain
Bring test setup, execution records, findings, and reports into one structured process instead of separate scripts and risk records.
Evidence teams can use together
Give business, technical, security, and compliance teams a common record for review, remediation, and approval decisions.
More than a score
Turn summary metrics, risk dimensions, case evidence, conclusions, and suggestions into a risk profile teams can act on.
From one test to ongoing comparison
Reuse compatible evaluations to validate remediation, track version changes, and support repeatable governance evidence.
The evaluation process
Define the scope
Set the risk question, test target, and version boundary.
Configure the evaluation
Choose the evaluation method and relevant test material.
Execute the test cases
Send each test input to the target and preserve its trace.
Evaluate the responses
Apply the evaluator and record an outcome for every valid case.
Review the evidence
Move from the summary metric to the cases that shaped it.
Create the report and compare results
Package the evidence, tested versions, and limitations.
Define the scope
Scope defines which conclusions the evaluation can support. State the risk question first, then identify the exact system behavior that needs to be tested.
- Test target
- The model, chatbot, or application whose observable behavior is under test.
- Version boundary
- The model release and material application configuration represented by the result.
- Use boundary
- The users, tasks, languages, and scenarios the evidence is intended to represent.
Configure the evaluation
An evaluation applies one named method to one defined target and scope. Choose the method according to the risk question rather than the score you expect to produce.
- Evaluation method
- Defines the behavior under test, evaluator, metric calculation, and interpretation rules.
- Test material
- The selected datasets, prompts, attacks, categories, or standards used by the method.
- Method version
- Identifies the evaluator and scoring rules so the result can be reproduced and compared correctly.
Execute the test cases
Each case sends a test input to the target and records the returned response. The trace begins here and is completed during evaluation.
- Trace
- The linked test input, target response, execution metadata, evaluator outcome, and explanation for one case.
- Stable execution
- The target version, system instructions, retrieval, guardrails, and other material configuration remain unchanged during the run.
Evaluate the responses
The method-specific evaluator assesses each recorded response and attaches an outcome and explanation to its trace. Valid case outcomes can then be aggregated into metrics.
- Case outcome
- What the evaluator concluded about one response and the evidence supporting that conclusion.
- Execution error
- A timeout, malformed response, evaluator failure, or request that did not reach the target or produce an evaluable response. A valid safety refusal is a case outcome, not an execution error.
Review the evidence
The result combines case outcomes with the method-specific metrics aggregated from them. Review it from the overall signal down to the underlying evidence.
| Level | Use it to |
|---|---|
| Summary metric | Understand the overall method-specific signal |
| Dimension | Locate the broad area driving the result |
| Category | Identify a concentrated behavior or control weakness |
| Case | Inspect the input, response, outcome, and explanation |
Create the report and compare results
A report packages the result with the target, method, test-material, and evaluator versions, plus coverage, errors, and limitations. These details determine whether two reports can be compared.
Report comparison is meaningful only when both results use compatible scope, methods, test material, evaluators, metrics, and execution conditions.
| Comparison | Use it to | Reference | Key rule |
|---|---|---|---|
| Compare your own versions | Track improvement or regression across releases of the same target | The previous compatible report in the target’s version history | Change the target version while keeping the evaluation conditions fixed |
| Compare with a baseline | Measure the current result against a preselected reference point | A fixed approved target version or method-defined reference | Use only a baseline built with compatible scope, method versions, and metric definitions |
A previous report can also be designated as the baseline. The comparison question is different: version comparison tracks change over time within the same target, while baseline comparison measures distance from one fixed reference point.