Skip to content

Interpreting Test Failures

After running a test, the results come back as a FHIR TestReport. This guide explains how to read the report, understand failure categories, and decide what to fix.

Reading a test report

Quick summary

Get a high-level summary of results:

frog_report_summarise(reportPath: "run.json")

Output (example):

Run ID:        run-123-abc
TestScript:    http://example.org/TestScript/patient-search
Server:        http://127.0.0.1:8081/fhir
Status:        completed
Duration:      3.2 seconds
Result:        PASSED (3 tests, 0 failures)

Detailed report

Save the full report:

frog_run_report(runId: "run-123-abc", outputPath: "full-report.json")

Open full-report.json to see per-assertion detail:

{
  "resourceType": "TestReport",
  "id": "run-123-abc",
  "status": "completed",
  "result": "passed",
  "tally": {
    "passed": 3,
    "failed": 0,
    "skipped": 0
  },
  "testScriptReference": "http://example.org/TestScript/patient-search",
  "test": [
    {
      "name": "Search patients",
      "result": "passed",
      "action": [
        {
          "result": "pass",
          "message": "HTTP 200"
        },
        {
          "result": "pass",
          "message": "Resource type = Bundle"
        }
      ]
    }
  ]
}

Understanding failure types

FHIR Frog categorizes failures into two main types:

1. Execution errors

The test failed to run (infrastructure problem, not a test failure).

Examples: - Server did not respond to the request - Network timeout - Fixture file not found - Invalid TestScript syntax

Message pattern: Failed to execute: <error>

What to do: - Check frog_doctor output - Verify the server is running (frog_server_check) - Check that fixture files exist - Review the full error message for details

2. Assertion failures

The test ran, but the response did not match the expectation.

Examples: - Got HTTP 404 instead of 201 - Response type was not Patient - FHIRPath expression evaluated to false - Profile validation failed

Message pattern: Expected X but got Y

What to do: - Understand which assertion failed (responseCode, resource type, etc.) - Check if the server behaviour matches your test's assumption - Read the request/response detail in the report

Failure categories and hints

The frog_report_summarise tool groups related failures into categories with hints:

Category: test

The test itself has a problem (it will fail against any conformant server).

Hints: - "Fixture file not found" → ensure the file exists and the path is correct - "Invalid operation syntax" → review the TestScript grammar - "FHIRPath syntax error" → fix the FHIRPath expression

Fix: Correct the test definition.

Category: sut (System Under Test)

The server does not conform to your test's assumptions.

Hints: - "Expected HTTP 200 but got 404" → server does not have the resource; is it created? - "Profile validation failed" → server does not have the IG loaded, or the profile is wrong - "Could not find Patient/123" → resource does not exist on the server

Fix: Load the required data/IG on the server, or adjust the test to match the server's actual behaviour.

Category: environment

Infrastructure or configuration issue (not the test or server).

Hints: - "Docker container not running" → start the server - "Connection refused" → server is not listening on the expected port - "Stale registry credentials" → run docker logout registry.gitlab.com

Fix: Resolve the infrastructure issue.

Common failure scenarios

Scenario 1: "Failed to create: HTTP 400"

Message: "Failed to execute: HTTP 400 : HAPI-1094: Resource Patient/123 not found"

Likely cause: A prior operation (in Setup or a previous test) was supposed to create a resource, but it failed silently.

Investigation: 1. Check the Setup phase in the TestReport—did the create operation succeed? 2. Verify the fixture file contains valid FHIR JSON 3. Check the server logs: docker logs frog-sut

Fix: Either ensure the precondition resource exists, or skip the test.

Scenario 2: "Expected 201 but got 404"

Message: "Expected responseCode = 201 but got 404"

Likely cause: The server endpoint is not correct, or the server is not running.

Investigation: 1. Check the TestScript operation: is the resource type correct? 2. Verify the server is running: frog_server_check name=my-sut 3. Test manually: curl -X POST http://127.0.0.1:8081/fhir/Patient ...

Fix: Ensure the server is running and the endpoint is correct.

Scenario 3: "Profile validation failed: could not be found"

Message: "Validate profile: http://example.org/Profile/MyPatient — resource does not conform: could not be found"

Likely cause: The server does not have the IG loaded, so it cannot find the profile.

Investigation: 1. Is the IG loaded on the server? (Check server logs or /metadata) 2. Is the profile URL correct in the test?

Fix: - Load the IG on the server: frog_server_start_local name=my-sut igPackage="fhirfrog.au-core#4.2.0" confirm=true - Or remove the validateProfile assertion if the profile is not available

Scenario 4: "Expression evaluated to false"

Message: "Expression: Bundle.entry.count() > 0 — evaluated to false"

Likely cause: The search returned no results.

Investigation: 1. Is the data in the server? (Run a separate search test to verify) 2. Is the search parameter correct? 3. Is the resource type correct?

Fix: - Populate the server with test data (in Setup phase) - Adjust the assertion to allow for empty results: Bundle.entry.count() >= 0 - Verify the search parameters are correct

Interpreting the full TestReport JSON

The TestReport has this structure:

{
  "resourceType": "TestReport",
  "id": "<run-id>",
  "status": "completed",            // always "completed" (even if failed)
  "result": "passed|failed|skipped",
  "tally": {
    "passed": 3,
    "failed": 1,
    "skipped": 0
  },
  "test": [
    {
      "name": "Test name",
      "result": "passed|failed",
      "action": [
        {
          "operation": { ... },
          "result": "pass|fail|skip",
          "message": "Details"
        },
        {
          "assert": { ... },
          "result": "pass|fail",
          "message": "Expected X but got Y"
        }
      ]
    }
  ]
}

Key fields

  • status: completed means the run finished; does not indicate pass/fail
  • result: passed, failed, or skipped — the overall outcome
  • tally: count of passed/failed/skipped assertions (not tests)
  • test[].result: per-test outcome; a test fails if any action fails
  • test[].action[].result: per-action outcome (pass, fail, or skip)
  • test[].action[].message: human-readable detail (e.g. "HTTP 404", "Expected X but got Y")

Known limit: CLI reports

The fhir-frog CLI (when used directly, not through the MCP) generates TestReports without per-assertion detail. Use the MCP tools (frog_run_local, frog_run_remote) instead; they provide full detail.

Comparing runs

frog_report_compare is planned (v0.2). In v0.1, compare runs manually by examining the TestReport JSON files side by side.

This will be useful for debugging: "my test passed yesterday but fails today—what changed?"

Iterating on a failing test

  1. Understand the failure: Read the error message and categorize it (test, SUT, environment)
  2. Fix the root cause:
  3. If it is a test problem, update the TestScript
  4. If it is a SUT problem, ensure the data/IG is loaded
  5. If it is an environment issue, resolve it (start Docker, etc.)
  6. Recompile and re-run (if you changed TSH): 
    frog_tsh_compile(path: "my-test.tsh", output: "my-test.json")
frog_run_local(scriptPath: "my-test.json", server: "my-sut")
  7. Verify the fix: frog_report_summarise(runId: "<id>")

Exporting results

frog_report_export is planned (v0.2). In v0.1, use frog_run_report to download a report:

frog_run_report(
  runId: "<id>",
  format: "junit-xml",
  outputPath: "results.xml"
)

Supported formats for frog_run_report: - junit-xml — for CI systems (GitHub Actions, GitLab CI, Jenkins) - testreport-json — FHIR TestReport as JSON - testreport-xml — FHIR TestReport as XML - xlsx, csv, md, pdf, docx — other formats

See also