Reference
Troubleshooting
Diagnose common target-connection, execution, scoring, and result problems without losing the original evaluation evidence.
Start with the earliest failing stage. A target connection problem, case-generation problem, and scoring problem can produce similar symptoms in the final view, but they require different fixes.
Use the diagnostic sequence
Identify the failing stage
Determine whether the run failed during target validation, case preparation, target execution, evaluator scoring, or aggregation.
Read the narrowest available error
Prefer a case-level or stage-level message over a generic evaluation status. Record the time and any request or case identifier shown.
Check what changed
Compare target version, credentials, endpoint, deployment status, rate limits, dataset, and evaluation settings with the last successful run.
Test the smallest safe unit
Validate target connectivity or retry a limited test before launching the complete evaluation again.
Create a traceable retry
Keep the method and evidence unchanged when testing an infrastructure fix. If the target or method changed, treat the retry as a new evaluated scope.
Target connection problems
| Symptom | Likely cause | Check |
|---|---|---|
| Authentication or unauthorized error | Missing, expired, or incorrectly scoped credential | Credential validity, header format, secret scope, and target provider |
| Target not found | Incorrect model, deployment, route, or base URL | Exact deployment name and final request URL |
| Connection timeout | Target unavailable, private network boundary, or timeout too low | Service health, network access, response latency, and timeout setting |
| Empty or unreadable response | Response shape differs from configured mapping | Raw response structure and expected response field |
| Works in a manual client but not AIDX | Headers, proxy, TLS, origin, or network path differs | Compare the complete request path without exposing credentials |
Execution problems
| Symptom | Interpretation | Recommended action |
|---|---|---|
| Run remains queued | No case has begun target execution | Confirm the run was started, then check platform status and workspace concurrency. |
| Progress stops after some cases | A repeated target or rate-limit failure may be blocking execution | Inspect the latest failed cases and target service limits before retrying. |
| Many timeout cases | Target latency exceeds the execution window or the service is overloaded | Stabilize the target, review rate and timeout settings, then create a controlled retry. |
| Rate-limit errors | The target rejects the evaluation request rate | Reduce concurrency or request rate when supported, or increase the target quota. |
| Responses differ unexpectedly | The target is stochastic or changed during the run | Confirm model/version and sampling settings; do not combine evidence from changed targets. |
Scoring and result problems
A run can complete target execution while evaluation scoring is still incomplete. Separate missing responses from missing evaluator outcomes.
| Symptom | Check first | Why it matters |
|---|---|---|
| No headline score | Whether all required cases and evaluator tasks completed | Aggregation may wait for every required outcome. |
| Unexpectedly low score | Directionality, included cases, and dominant weak categories | A valid aggregate can still be driven by one concentrated risk area. |
| Score changed after retry | Target version, sampling, dataset, method, and exclusions | The two runs are comparable only when their scope is equivalent. |
| Cases marked unscored | Response readability and evaluator availability | Unscored cases should not silently be treated as safe outcomes. |
When to contact AIDX
Contact AIDX when the target connection succeeds but platform processing repeatedly fails, a completed run cannot be scored, or the evidence shown is internally inconsistent.
Include the following non-secret information:
| Include | Do not include |
|---|---|
| Workspace or organization name | Passwords, API keys, access tokens, or full secret values |
| Evaluation name or identifier | Sensitive customer prompts unless an approved support channel is used |
| Target type and declared version | Unredacted personal or regulated data |
| Failing stage, status, timestamp, and error text | Credentials embedded in request examples or screenshots |
| Whether a retry produced the same outcome | Edited result data presented as original evidence |