8. CI integration

Conformance is not a one-time event. Each release of your software can break the FHIR export — a renamed field, a new code mapping, a library upgrade. Add the sandbox to your continuous integration (CI) pipeline so the build fails when your output stops conforming.

The approach#

Your test suite makes your software generate FHIR resources from test fixtures and writes them to a folder, e.g. build/fhir/*.json.
A small script sends each file to the sandbox's $validate endpoint.
The script prints every issue and exits with a non-zero code if any resource has an error.

The validation script#

Save as scripts/npfs_validate.py in your repository. It uses only the Python standard library.

#!/usr/bin/env python3
"""Validate FHIR JSON files against NP Core using the NP FHIR IG Conformance Sandbox.

Usage: NPFS_BASE=... NPFS_API_KEY=... python scripts/npfs_validate.py build/fhir/*.json
Exit code 1 if any file has an error or fatal issue.
"""
import json
import os
import sys
import time
import urllib.error
import urllib.request

BASE = os.environ["NPFS_BASE"].rstrip("/")
KEY = os.environ["NPFS_API_KEY"]
FAIL_ON_WARNING = os.environ.get("NPFS_FAIL_ON_WARNING") == "1"


def validate(path):
    body = open(path, "rb").read()
    req = urllib.request.Request(
        f"{BASE}/fhir/$validate", data=body, method="POST",
        headers={"Authorization": f"Bearer {KEY}", "Content-Type": "application/fhir+json",
                 "Accept": "application/fhir+json"})
    for attempt in range(5):
        try:
            with urllib.request.urlopen(req, timeout=180) as r:
                return json.load(r)
        except urllib.error.HTTPError as e:
            if e.code == 429:                      # rate limited: back off and retry
                time.sleep(2 ** attempt)
                continue
            if e.code in (400, 422):               # OperationOutcome in the body
                return json.load(e)
            raise
    raise RuntimeError("rate limited too many times")


def main(files):
    failed = 0
    for path in files:
        oo = validate(path)
        issues = oo.get("issue", [])
        bad = [i for i in issues if i["severity"] in ("fatal", "error")
               or (FAIL_ON_WARNING and i["severity"] == "warning")]
        status = "FAIL" if bad else "ok"
        print(f"{status:4}  {path}")
        for i in issues:
            if i["severity"] == "information":
                continue
            where = ", ".join(i.get("expression") or i.get("location") or [])
            print(f"      [{i['severity']}] {where}: {i.get('diagnostics') or i.get('details', {}).get('text')}")
        failed += bool(bad)
    print(f"\n{len(files) - failed}/{len(files)} files conform to NP Core")
    return 1 if failed else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

Tip: set NPFS_FAIL_ON_WARNING=1 once you have cleared all errors, to keep warnings from creeping in.

GitHub Actions#

Add the API key as a repository secret named NPFS_API_KEY (Settings → Secrets and variables → Actions).

.github/workflows/np-core-conformance.yml:

name: NP Core conformance
on:
  push:
    branches: [main]
  pull_request:

jobs:
  fhir-conformance:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      # Replace with the command that makes your software export FHIR test resources.
      - name: Generate FHIR resources
        run: ./gradlew exportFhirFixtures   # e.g. writes build/fhir/*.json
      - name: Validate against NP Core
        env:
          NPFS_BASE: https://fhir-sandbox.sahasratech.com.np
          NPFS_API_KEY: ${{ secrets.NPFS_API_KEY }}
        run: python scripts/npfs_validate.py build/fhir/*.json

GitLab CI#

Add NPFS_API_KEY as a masked CI/CD variable (Settings → CI/CD → Variables).

.gitlab-ci.yml:

stages: [test]

np-core-conformance:
  stage: test
  image: python:3.12-slim
  variables:
    NPFS_BASE: "https://fhir-sandbox.sahasratech.com.np"
  script:
    - ./scripts/export-fhir-fixtures.sh      # your command; writes build/fhir/*.json
    - python scripts/npfs_validate.py build/fhir/*.json
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Scenarios from CI#

You can also submit conformance scenarios from CI so the conformance report always reflects your latest build:

curl -sS -X POST "$NPFS_BASE/api/scenarios/<scenario-id>/submit" \
  -H "Authorization: Bearer $NPFS_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"resource\": $(cat build/fhir/patient-registration.json)}"

The scenario ids are listed by GET /api/scenarios.

Good practice#

Use a dedicated API key per pipeline and name it accordingly, so you can revoke it independently.
Keep test fixtures synthetic; never export real records in CI.
Run the check on every pull request touching export code, and nightly on the main branch.
Pin your FHIR library versions; upgrades are a common source of new errors.
If the sandbox is unreachable, decide whether your pipeline should fail or skip — but do not silently pass.