Skip to content

Choosing a TestScript Format

FHIR Frog can author and run tests in three formats. This page guides you to choose the right one for your use case.

Three authoring formats

Format Editing When to use Runnable with FHIR Frog
TSH (Shorthand) Human-friendly text Simple CRUD tests, readability, fast iteration Yes (after compile)
JSON TestScript Direct JSON Complex operations, search parameters, system operations Yes (direct)
FSH Instance FHIR Shorthand (FSH) Part of an IG definition, CI/CD integration Yes (after compile)

Quick decision tree

  1. Are you testing read/create/search with simple assertions?
    → TSH is best; it's concise and handles the common cases.

  2. Do you need search parameters in the operation URL (e.g. ?family=Smith)?
    → JSON TestScript is required; TSH cannot encode them.

  3. Do you need system-level operations ($expand, $validate, $lookup)?
    → JSON TestScript is required; TSH does not support them.

  4. Are you authoring tests as part of an IG (e.g. in input/tests/)?
    → FSH Instance of TestScript works (compile with frog_tsh_compile), or use JSON TestScript directly.

  5. Is your test suite large, with many tests to maintain?
    → TSH scales better for readability; otherwise, JSON is fine.

TSH (TestScript Shorthand)

What it compiles to

Every .tsh file is compiled to a JSON TestScript by frog_tsh_compile:

frog_tsh_compile(path: "my-test.tsh", output: "my-test.json")

Syntax overview

TestScript: PatientSearchTest
Title: "Patient Search"
Status: #draft
Url: "http://example.org/TestScript/patient-search"

Fixture: patient_data
  * resource = "patient.json"
  * autocreate = false
  * autodelete = false

Test: "Search patients by family name"
  * operation create Patient from "patient_data" as p1
  * assert responseCode = 201
  * operation search Patient as search_resp
  * assert responseCode = 200
  * assert resource = Bundle
  * assert expression "Bundle.entry.count() > 0"

Supported operations

  • create — POST with fixture body
  • search — GET to search endpoint (no parameters in TSH syntax)
  • read — does not run (see Known limits)
  • update — PUT with fixture body
  • delete — does not run

Supported assertions

  • responseCode (HTTP status)
  • resource (response type)
  • contentType (HTTP Content-Type header)
  • expression (FHIRPath)
  • minimumId (id comparison)
  • validateProfile (profile validation)

Pros

  • Readability: 60–70% shorter than JSON
  • Built-in guidance: compiler warns about unsupported constructs
  • Fixture management: simple fixture syntax
  • Test discovery: one fixture and one test per file is clear

Cons

  • Limited to CRUD: no search parameters, system operations, or parameterized requests
  • Compilation step: must compile to JSON before running
  • No variable substitution in URLs: variables work only in assertions

JSON TestScript

Syntax overview

A JSON TestScript directly in FHIR format (or XML):

{
  "resourceType": "TestScript",
  "id": "PatientSearchTest",
  "title": "Patient Search",
  "status": "draft",
  "url": "http://example.org/TestScript/patient-search",
  "profile": [],
  "fixture": [
    {
      "id": "patient_data",
      "autocreate": false,
      "autodelete": false,
      "resource": {
        "reference": "patient.json"
      }
    }
  ],
  "test": [
    {
      "name": "Search patients by family name",
      "action": [
        {
          "operation": {
            "type": {
              "system": "http://terminology.hl7.org/CodeSystem/testscript-operation-codes",
              "code": "create"
            },
            "resource": "Patient",
            "sourceId": "patient_data",
            "responseId": "p1",
            "encodeRequestUrl": true
          }
        },
        {
          "assert": {
            "warningOnly": false,
            "responseCode": "201"
          }
        }
      ]
    }
  ]
}

Supported operations

All FHIR TestScript operations:

  • create, search, read, update, delete (with full control)
  • $expand, $validate, $lookup (system-level operations)
  • Search with parameters: use "params": "?family=Smith" in the operation
  • Read by id: use "params": "/123" instead of encoding the id in the resource field

Example with search parameters:

{
  "operation": {
    "type": { "code": "search" },
    "resource": "Patient",
    "responseId": "search_result",
    "params": "?family=Smith",
    "encodeRequestUrl": true
  }
}

Supported assertions

All TestScript assertion types:

  • responseCode, resource, contentType, expression
  • minimumId, validateProfile
  • headerField, statusCode (detailed status checks)
  • Custom XPath/FHIRPath with full syntax

Pros

  • Full control: access every TestScript feature
  • Search parameters: encode parameters in URLs directly
  • System operations: $expand, $validate, etc.
  • No compilation: runs directly without a build step

Cons

  • Verbosity: more boilerplate than TSH
  • Manual management: no template or validation beyond linting
  • Fixture references: must manually manage fixture ids

FSH Instance of TestScript

Syntax overview

Define a TestScript as a FHIR Shorthand Instance within an IG:

Instance: PatientSearchTest
InstanceOf: TestScript
Title: "Patient Search"
Usage: #definition
* status = #draft
* url = "http://example.org/TestScript/patient-search"
* fixture[0].id = "patient_data"
* fixture[0].autocreate = false
* fixture[0].autodelete = false
* fixture[0].resource.reference = "patient.json"
* test[0].name = "Search patients"
* test[0].action[0].operation.type.code = #search
* test[0].action[0].operation.resource = #Patient
* test[0].action[0].operation.responseId = "search_result"

Authoring in an IG

Place FSH Instance files in input/fsh/tests/ or input/resources/:

your-ig/
  input/
    fsh/
      tests/
        patient-search-test.fsh

Then build with the IG Publisher, which emits JSON TestScript resources.

Compilation

If your IG uses FSH, the IG Publisher compiles instances to JSON automatically. Alternatively, use frog_tsh_compile if the file is in TSH format:

frog_tsh_compile(path: "patient-search.tsh", output: "patient-search.json")

Pros

  • IG integration: tests are part of the IG definition
  • Version control: tests versioned with the IG
  • Reusable profiles: reference IG profiles directly

Cons

  • Verbosity: even more boilerplate than JSON
  • Indirect authoring: property-by-property dot notation
  • Build dependency: requires the IG Publisher or SUSHI

Which to choose?

I'm a test author, just getting started

→ Use TSH. Fast to write, readable, good compiler feedback.

TestScript: MyFirstTest
Title: "First Test"
Status: #draft
Url: "http://example.org/TestScript/my-first"

Fixture: p1
  * resource = "patient.json"
  * autocreate = false
  * autodelete = false

Test: "Create a patient"
  * operation create Patient from "p1" as p
  * assert responseCode = 201
  * assert resource = Patient

I need to test an API with search filters or advanced operations

→ Use JSON TestScript. You have full control:

{
  "operation": {
    "resource": "Patient",
    "params": "?family=Smith&given=John"
  }
}

I'm adding tests to an existing IG

→ Use FSH Instance or JSON TestScript and place them in the IG's input/resources/ or input/fsh/tests/ directory. The IG Publisher will include them in the package.

I have a lot of tests and want to maintain them easily

→ Use TSH if you can; switch to JSON for tests that need features TSH lacks. Mix and match: keep simple tests in TSH for readability, complex tests in JSON.

Workflow summary

  1. Author in your chosen format (TSH, JSON, or FSH)
  2. Validate with frog_tsh_validate (TSH only) or linting
  3. Compile (if using TSH or FSH): frog_tsh_compile
  4. Lint the compiled JSON: frog_testscript_lint
  5. Run against a server: frog_run_local or frog_run_remote
  6. Interpret results with frog_report_summarise

See also