Running Tests: CLI¶

The fhir-frog CLI is a standalone executable for running FHIR TestScripts, TestPlans, and Test IG packages. This page documents the real flags and workflow.

Stub reports

The CLI generates TestReports without per-assertion detail. For detailed failure diagnosis, use frog_run_local or frog_run_remote (MCP tools) instead.

Quick start¶

# Download or build the JAR (see below)
java -jar fhir-frog-cli.jar my-test.json --server http://localhost:8080/fhir

Exit code: 0 (pass), 1 (fail or error).

Installation¶

Download a release¶

Releases are available from the GitLab package registry.

Build from source¶

git clone https://gitlab.com/fhirfrog/fhir-frog-cli.git
cd fhir-frog-cli
mvn -q -B -DskipTests package
# JAR is at: target/fhir-frog-cli-<version>.jar

Syntax¶

Test execution¶

java -jar fhir-frog-cli.jar <test-file> [options]

Test IG package execution¶

java -jar fhir-frog-cli.jar test-package <id#version> [options]

Report upload¶

java -jar fhir-frog-cli.jar upload-reports <directory> --server <url>

Test execution options¶

`--server <url>` (default: `http://localhost:8080/fhir`)¶

The FHIR server URL to run tests against.

java -jar fhir-frog-cli.jar my-test.json --server http://localhost:8080/fhir
java -jar fhir-frog-cli.jar my-test.json --server https://fhir.example.org/fhir

`--fixtures <path>` (default: `fixtures`)¶

Base directory where fixture files are located (relative to current directory or absolute).

java -jar fhir-frog-cli.jar my-test.json --fixtures ./fixtures
java -jar fhir-frog-cli.jar my-test.json --fixtures /absolute/path/to/fixtures

Note: When using test-package, the --fixtures option is ignored; fixtures come from the package's own fixtures/ directory.

`--report-dir <path>`¶

Save reports to a directory (in addition to stdout).

java -jar fhir-frog-cli.jar my-test.json \
  --server http://localhost:8080/fhir \
  --report-dir ./reports

Saved as <id>.json or <id>.xml depending on --report-format.

`--report-server <url>`¶

Upload reports to a FHIR server.

java -jar fhir-frog-cli.jar my-test.json \
  --server http://localhost:8080/fhir \
  --report-server http://report-server:8080/fhir

`--report-format <fmt>` (default: `json`)¶

Output format: json or xml.

java -jar fhir-frog-cli.jar my-test.json --report-format json
java -jar fhir-frog-cli.jar my-test.json --report-format xml

`--ig-package <id#version>`¶

An IG package containing profiles for validateProfile assertions.

java -jar fhir-frog-cli.jar my-test.json \
  --ig-package hl7.fhir.au.core#2.0.0

This tells the CLI to validate responses against the profiles in the IG. The package is resolved from the standard FHIR package cache (~/.fhir/packages).

`--terminology-server <url>`¶

External terminology server for code/ValueSet lookups not available locally.

java -jar fhir-frog-cli.jar my-test.json \
  --terminology-server https://api.healthterminologies.gov.au/integration/R4/fhir

Without this, validateProfile may fail with "not found in value set" for codes from IGs that are not loaded locally.

Examples¶

Single TestScript¶

java -jar fhir-frog-cli.jar my-test.json \
  --server http://localhost:8080/fhir \
  --report-dir ./reports \
  --report-format json

Single TestScript in TSH format¶

# Compile first (if using TSH)
frog_tsh_compile(path: "my-test.tsh", output: "my-test.json")

# Then run
java -jar fhir-frog-cli.jar my-test.json --server http://localhost:8080/fhir

TestPlan¶

java -jar fhir-frog-cli.jar my-testplan.json \
  --server http://localhost:8080/fhir \
  --report-dir ./reports

Test IG Package¶

java -jar fhir-frog-cli.jar test-package fhirfrog.au-core-tests#0.1.0 \
  --server http://localhost:8080/fhir \
  --ig-package hl7.fhir.au.core#2.0.0

The CLI will execute every TestScript in the package.

With IG and terminology server¶

java -jar fhir-frog-cli.jar my-test.json \
  --server http://localhost:8080/fhir \
  --ig-package hl7.fhir.au.core#2.0.0 \
  --terminology-server https://api.healthterminologies.gov.au/integration/R4/fhir \
  --report-dir ./reports \
  --report-format json

Upload reports to a server¶

java -jar fhir-frog-cli.jar upload-reports ./reports \
  --server http://report-server:8080/fhir

Exit codes¶

Code	Meaning
`0`	All tests passed
`1`	Test failed or execution error

Test files¶

Supported formats¶

TestScript: .json, .xml. A .tsh file is not executable: the CLI's own usage text says it is, but the CLI parses every TestScript as FHIR JSON/XML and fails on a .tsh file (first non-whitespace character was: 'T' (must be '{'); verified by running the CLI, 2026-10-21). Compile it first with the MCP tool frog_tsh_compile(path=..., output=...) and run the JSON.
TestPlan: .json, .xml

Example TestScript¶

{
  "resourceType": "TestScript",
  "id": "patient-search",
  "title": "Patient Search",
  "status": "draft",
  "url": "http://example.org/TestScript/patient-search",
  "fixture": [],
  "test": [
    {
      "name": "Search patients",
      "action": [
        {
          "operation": {
            "type": {"code": "search"},
            "resource": "Patient",
            "responseId": "search_resp",
            "encodeRequestUrl": true
          }
        },
        {
          "assert": {
            "warningOnly": false,
            "responseCode": "200"
          }
        }
      ]
    }
  ]
}

Fixtures¶

Fixtures are test data files (typically JSON resources) referenced by your TestScript.

Fixture layout¶

Place fixtures relative to the test file or in the --fixtures directory:

project/
  my-test.json
  fixtures/
    patient.json
    observation.json

Or, use absolute paths in the fixture reference.

Known limit: no auth options¶

The CLI does not support authentication (OAuth, Bearer token, mTLS). For authenticated servers, either:

Use the MCP tools (frog_run_local, frog_run_remote)
Register the server with auth in ~/.fhir-frog/backends.yaml and use it by name
Proxy the server through an unauthenticated endpoint for testing

Wiring into builds¶

Maven integration¶

Add the CLI JAR to your build and invoke it:

# In a Maven test phase, after the IG is built
mvn exec:java@frog-test -DscriptPath=input/tests/my-test.json

See deployment/maven.md (v0.2+) for full integration details.

Ant integration¶

Similarly, invoke from Ant:

<exec executable="java">
  <arg value="-jar"/>
  <arg value="${fhir-frog-cli.jar}"/>
  <arg value="input/tests/my-test.json"/>
  <arg value="--server"/>
  <arg value="http://localhost:8080/fhir"/>
</exec>

Known limits (documented)¶

Limitation	Impact	Workaround
No authentication	Cannot test authenticated servers	Use MCP tools, or proxy through unauth endpoint
Stub reports	No per-assertion detail in failures	Use MCP tools for detailed diagnosis
No `.tsh` execution	Must compile to JSON first	Use `frog_tsh_compile` before running
No report comparison	Cannot diff two runs easily	`frog_report_compare` planned (v0.2)

Running Tests: CLI¶

Quick start¶

Installation¶

Download a release¶

Build from source¶

Syntax¶

Test execution¶

Test IG package execution¶

Report upload¶

Test execution options¶

--server <url> (default: http://localhost:8080/fhir)¶

--fixtures <path> (default: fixtures)¶

--report-dir <path>¶

--report-server <url>¶

--report-format <fmt> (default: json)¶

--ig-package <id#version>¶

--terminology-server <url>¶

Examples¶

Single TestScript¶

Single TestScript in TSH format¶

TestPlan¶

Test IG Package¶

With IG and terminology server¶

Upload reports to a server¶

Exit codes¶

Test files¶

Supported formats¶

Example TestScript¶

Fixtures¶

Fixture layout¶

Known limit: no auth options¶

Wiring into builds¶

Maven integration¶

Ant integration¶

Known limits (documented)¶

See also¶

`--server <url>` (default: `http://localhost:8080/fhir`)¶

`--fixtures <path>` (default: `fixtures`)¶

`--report-dir <path>`¶

`--report-server <url>`¶

`--report-format <fmt>` (default: `json`)¶

`--ig-package <id#version>`¶

`--terminology-server <url>`¶