Let’s Go: Start-to-End with the IBM FHIR Server and Bulk Data

Written by

In IBM FHIR Server 4.6.0, IBM FHIR Server has refactored the supported Bulk Data operations that support the HL7 FHIR BulkDataAccess IG: STU1 and the Proposal for $import Operation.

To demonstrate the new features in a quick script, use the following to setup an S3 demonstration from the start-to-end.

Setup Minio
Setup IBM FHIR Server
Load Sample Data
Run $import
Run $export

Setup Minio

Pull the minio docker image
Create a local data directory for data

mkdir -p minio-data/

Download demonstration only certificates/keys

curl -L -o private.key raw.githubusercontent.com/IBM/FHIR/main/build/docker/minio/private.key

curl -L -o public.crt https://raw.githubusercontent.com/IBM/FHIR/main/build/docker/minio/public.crt

Note, these are demonstration only Private Key and Public cert pair.

Create the Minio container

export MINIO_ROOT_PASSWORD=$(openssl rand -base64 20)
docker run -d --rm -p 9000:9000 \
  --name minio \
  -h minio \
  -e "MINIO_ROOT_USER=fhir-s3-admin" \
  -e "MINIO_ROOT_PASSWORD=${MINIO_ROOT_PASSWORD}" \
  -e "MINIO_ACCESS_KEY=fhirAccessKey" \
  -e "MINIO_SECRET_KEY=${MINIO_ROOT_PASSWORD}" \
  -v $(pwd)/minio-data:/data \
  -v $(pwd)/public.crt:/root/.minio/certs/public.crt \
  -v $(pwd)/private.key:/root/.minio/certs/private.key \
  minio/minio server /data

Note, the Minio container will create local certificates.

Using your MINIO_ROOT_PASSWORD echo ${MINIO_ROOT_PASSWORD}, log in to your Minio instance.

Create a Bucket (folder), look in the lower left. Name it fhirbulkdata.

You should see the bucket.

6. Download the [Sample NDJSON](https://github.com/IBM/FHIR/blob/main/fhir-server-test/src/test/resources/testdata/import-operation/test-import.ndjson)

`curl -L https://raw.githubusercontent.com/IBM/FHIR/main/fhir-server-test/src/test/resources/testdata/import-operation/test-import.ndjson -o test-import.ndjson`

We’re going to upload this file to the fhir-bulkdata bucket.

7. Click Upload File > Select the test-import.ndjson. Click Upload. Confirm you see the file in the bucket.

Setup IBM FHIR Server

Download the fhir-server-config.json

curl -L https://raw.githubusercontent.com/IBM/FHIR/main/fhir-server/liberty-config/config/default/fhir-server-config.json -o fhir-server-config.json

Update the fhir-server-config.json fhirServer/bulkdata with the following snippet to use hmac and bulkdata. Be sure to update fhirServer/bulkdata/storageProviders/minio/accessKeyId to fhirAccessKey and fhirServer/bulkdata/storageProviders/minio/secretAccessKey to your MINIO_ROOT_PASSWORD.

    "bulkdata": {
            "enabled": true,
            "core": {
                "api": {
                    "url": "https://localhost:9443/ibm/api/batch",
                    "user": "fhiradmin",
                    "password": "change-password",
                    "truststore": "resources/security/fhirTrustStore.p12",
                    "truststorePassword": "change-password", 
                    "trustAll": true
                },
                "cos" : { 
                    "partUploadTriggerSizeMB": 10,
                    "objectSizeThresholdMB": 200,
                    "objectResourceCountThreshold": 200000,
                    "useServerTruststore": true
                },
                "file" : { 
                    "writeTriggerSizeMB": 1,
                    "sizeThresholdMB": 200,
                    "resourceCountThreshold": 200000
                },
                "pageSize": 100,
                "batchIdEncryptionKey": "change-password",
                "maxPartitions": 3, 
                "maxInputs": 5
            },
            "storageProviders": {
                "default" : {
                    "type": "file",
                    "_type": "ibm-cos|aws-s3|file|https",
                    "validBaseUrls": [],
                    "fileBase": "/output/bulkdata",
                    "disableBaseUrlValidation": true,
                    "exportPublic": true,
                    "disableOperationOutcomes": true,
                    "duplicationCheck": false, 
                    "validateResources": false
                },
                "minio" : {
                    "type": "aws-s3",
                    "bucketName": "fhirbulkdata",
                    "location": "us",
                    "endpointInternal": "https://minio:9000",
                    "endpointExternal": "https://localhost:9000",
                    "auth" : {
                        "type": "hmac",
                        "accessKeyId": "minio",
                        "secretAccessKey": "change-password"
                    },
                    "disableBaseUrlValidation": true,
                    "exportPublic": true,
                    "disableOperationOutcomes": true,
                    "duplicationCheck": false, 
                    "validateResources": false, 
                    "create": false,
                    "presigned": true
                }
            }
        }

Create the /output/bulkdata folder

mkdir -p bulkdata

Download the ibm-fhir-server docker image

docker pull ibmcom/ibm-fhir-server:latest

$ docker pull ibmcom/ibm-fhir-server:latest
latest: Pulling from ibmcom/ibm-fhir-server
13897c84ca57: Pull complete 
64607cc74f9c: Pull complete 
16ac69f16a48: Pull complete 
54eaf4e89615: Pull complete 
215ce34bfb04: Pull complete 
23fc81994ef3: Pull complete 
358f256d50db: Pull complete 
2c5ee9f8d67e: Pull complete 
df1b462e3122: Pull complete 
64782415f212: Pull complete 
5871c358d552: Pull complete 
931b2daafdbb: Pull complete 
a77b0c08b2ad: Pull complete 
8405f4cee973: Pull complete 
a0d476c441e5: Pull complete 
d504580ab646: Pull complete 
b68755501c7d: Pull complete 
4071f846ff8d: Pull complete 
Digest: sha256:1829c8ab601355efecc190ef667fba93d92cbbd2e9305c109f736851c3ba97f4
Status: Downloaded newer image for ibmcom/ibm-fhir-server:latest
docker.io/ibmcom/ibm-fhir-server:latest

Start up the IBM FHIR Server with the fhir-server-config.json we just modified.

docker run --rm -d -p 9443:9443 -e BOOTSTRAP_DB=true \
    -v $(pwd)/fhir-server-config.json:/config/config/default/fhir-server-config.json \
    -v $(pwd)/bulkdata:/output/bulkdata \
    ibmcom/ibm-fhir-server

You’ll see the container id output 60a5f1cae6d677d80772f1736db1be74836a8a4845fcccc81286b7c557bc2d86.

Check that the applications are started using the container id.

$ docker logs 60a | grep -i started
[4/21/21, 20:55:58:449 UTC] 00000001 FrameworkMana I   CWWKE0002I: The kernel started after 1.43 seconds
[4/21/21, 20:55:58:464 UTC] 0000002a FeatureManage I   CWWKF0007I: Feature update started.
[4/21/21, 20:56:01:328 UTC] 00000030 AppMessageHel A   CWWKZ0001I: Application fhir-openapi started in 1.588 seconds.
[4/21/21, 20:56:03:141 UTC] 00000031 AppMessageHel A   CWWKZ0001I: Application fhir-bulkdata-webapp started in 3.402 seconds.
[4/21/21, 20:56:07:824 UTC] 0000002d AppMessageHel A   CWWKZ0001I: Application fhir-server-webapp started in 7.871 seconds.
[4/21/21, 20:56:07:868 UTC] 0000002a TCPPort       I   CWWKO0219I: TCP Channel defaultHttpEndpoint-ssl has been started and is now listening for requests on host *  (IPv4) port 9443.
[4/21/21, 20:56:07:880 UTC] 0000002a FeatureManage A   CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 10.885 seconds.

Download the Sample Data

curl -L https://raw.githubusercontent.com/IBM/FHIR/main/fhir-server-test/src/test/resources/testdata/everything-operation/Antonia30_Acosta403.json -o Antonia30_Acosta403.json

Load the Sample Data bundle to the IBM FHIR Server

curl -k --location --request POST 'https://localhost:9443/fhir-server/api/v4' \
--header 'Content-Type: application/fhir+json' \
--user "fhiruser:${DUMMY_PASSWORD}" \
--data-binary  "@Antonia30_Acosta403.json" -o response.json

Scan the response.json for any status that is not "status": "201". For example, the status is in the family of User Request Error or Server Side Error.
Run the $import operation

curl -k --location --request POST 'https://localhost:9443/fhir-server/api/v4/$import'     --header 'X-FHIR-TENANT-ID: default'     --header 'Content-Type: application/fhir+json'     --header 'X-FHIR-BULKDATA-PROVIDER: minio'     --header 'X-FHIR-BULKDATA-PROVIDER-OUTCOME: minio'     --user 'fhiruser:change-password'     --header 'Cookie: __cfduid=d02cd7c4e9e90b7f71c16bdd22da08fb41616718585'     --data-raw '{
        "resourceType": "Parameters",
        "id": "30321130-5032-49fb-be54-9b8b82b2445a",
        "parameter": [
            {
                "name": "inputSource",
                "valueUri": "https://my-server/source-fhir-server"
            },
            {
                "name": "inputFormat",
                "valueString": "application/fhir+ndjson"
            },
            {
                "name": "input",
                "part": [
                    {
                        "name": "type",
                        "valueString": "Patient"
                    },
                    {
                        "name": "url",
                        "valueUrl": "test-import.ndjson"
                    }
                ]
            },
            {
                "name": "storageDetail",
                "valueString": "ibm-cos"
            }
        ]
    }' -v

Grab the Location response header.

< HTTP/2 202
< content-location: https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=Zhkno6biH1Qwh2S6VDBT6w
< date: Fri, 23 Apr 2021 00:08:06 GMT
< content-length: 0
< content-language: en-US

Check the export status using the URL

curl -k --location --request GET 'https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=LqzauvqtHSmkpChVHo%2B1MQ' \
    --header 'X-FHIR-TENANT-ID: default' \
    --header 'Content-Type: application/fhir+json' \
    --user 'fhiruser:change-password'

View the response and confirm successful export. You’ll see 4 successful imports and no failures.

{
    "transactionTime": "2021-04-23T00:22:51.368Z",
    "request": "https://localhost:9443/fhir-server/api/v4/$import",
    "requiresAccessToken": false,
    "output": [
        {
            "type": "OperationOutcome",
            "url": "test-import.ndjson_oo_success.ndjson",
            "count": 4
        }
    ],
    "error": [
        {
            "type": "OperationOutcome",
            "url": "test-import.ndjson_oo_errors.ndjson",
            "count": 0
        }
    ]
}

Check the Docker Logs docker logs 7050fdf7905e1fb320fd23b731639a21e064589c68d384e8add038879bc7993

[4/23/21, 0:19:30:986 UTC] 0000002e Reporter      I   Operation Type: $import
[4/23/21, 0:19:30:987 UTC] 0000002e Reporter      I   Resource Type         failures              success               processed             totalRead             totalValidation       totalWrite            fileSize              Resource Size
[4/23/21, 0:19:30:987 UTC] 0000002e Reporter      I   Patient               0                     4                     4                     2410                  0                     4795                  4303                  1075.0
[4/23/21, 0:19:30:988 UTC] 0000002e Reporter      I    ---- Total: 4 ImportRate: 0.49 ----

Run the $export operation

curl -k -v --location --request GET 'https://localhost:9443/fhir-server/api/v4/$export?_outputFormat=application/fhir+ndjson&_type=Patient' \
    --header 'X-FHIR-TENANT-ID: default' \
    --header 'Content-Type: application/fhir+json' \
    --header 'X-FHIR-BULKDATA-PROVIDER: minio' \
    --header 'X-FHIR-BULKDATA-PROVIDER-OUTCOME: minio' \
    --user 'fhiruser:change-password'

Grab the Location response header, and check the export status

< HTTP/2 202
< content-location: https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=UKt4ESCnqOvAfxYWhdsfUg
< date: Fri, 23 Apr 2021 00:25:07 GMT
< content-length: 0
< content-language: en-US

Call the bulkdata-status endpoint

curl -k --location --request GET 'https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=UKt4ESCnqOvAfxYWhdsfUg' \
    --header 'X-FHIR-TENANT-ID: default' \
    --header 'Content-Type: application/fhir+json' \
    --user 'fhiruser:change-password'

View the response, and download the exported file.

{
    "transactionTime": "2021-04-23T00:25:07.782Z",
    "request": "https://localhost:9443/fhir-server/api/v4/$export?_outputFormat=application/fhir+ndjson&_type=Patient",
    "requiresAccessToken": false,
    "output": [
        {
            "type": "Patient",
            "url": "https://localhost:9000/fhirbulkdata/hnFKv4BahpfK2iCnlPHOvnVknZVCVbgrBKwMn2HWCgg/Patient_1.ndjson?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=fhirAccessKey%2F20210423%2Fus%2Fs3%2Faws4_request&X-Amz-Date=20210423T002551Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=21e52030687241b9380b8985d6ee4793328955a22bda3adabe0e724175947d58",
            "count": 4
        }
    ]
}

Download the file.

curl -k -o data.ndjson 'https://localhost:9000/fhirbulkdata/hnFKv4BahpfK2iCnlPHOvnVknZVCVbgrBKwMn2HWCgg/Patient_1.ndjson?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=fhirAccessKey%2F20210423%2Fus%2Fs3%2Faws4_request&X-Amz-Date=20210423T002551Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=21e52030687241b9380b8985d6ee4793328955a22bda3adabe0e724175947d58'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4266  100  4266    0     0  20809      0 --:--:-- --:--:-- --:--:-- 20708

Look at the content and you’ll see it’s an ndjson.

You now have a working environment start-to-end of the Bulk Data on IBM FHIR Server.

Addendum

File Provider

By changing X-FHIR-BULKDATA-PROVIDER to default, you can use the mapped volume /output/bulkdata to export and import from a local directory.

Logs

You can see the job logs using:

docker logs nervous_boyd

docker exec -it nervous_boyd cat /logs/joblogs/<path to job>

Networking issues

If you hit issues where the two images don’t talk to each other, please connect them to the same network:

docker network connect –alias fhir docker_default nervous_boyd

docker network connect –alias minio docker_default minio

ibm-fhir-server

Comments

2 responses to “Let’s Go: Start-to-End with the IBM FHIR Server and Bulk Data”

2021-07-11

Eliot Salant

Hi Paul. In step there is a typo: $(pwd)/fhir-server-config.json:/config/config/default/fhir-server-config.json should of course just be /config/default…
Also, step 17 requires some explanation…
Thanks, Eliot
1. 2021-07-13
  
  Paul
  
  Hi Eliot, In more recent releases, the method I shared is accurate. You’ll notice the pattern I use is `docker run -d -p 9443:9443 -e BOOTSTRAP_DB=true -v $(pwd)/fhir-server-config.json:/config/config/default/fhir-server-config.json ibmcom/ibm-fhir-server` in other blog entries.
  
  Let me explain 17, the IBM FHIR Server expects a parameters object as input to any operation per the specification.
  
  I’ll use pseudo-JsonPath to enumerate the paths:
  
  parameter[].where(name=’inputSource’) is an optional identification of the ‘_source’ of the FHIR ndjson’s that are going to be imported.
  
  parameter[].where(name=’inputFormat’) is a required parameter that identifies the input format, currently only application/fhir+ndjson, however it may be expanded in the future to use a blob, a bundle, or maybe even Parquet.
  
  parameter[].where(name=’input’) is a part (an array of sub parameters) which binds multiple types together in a single parameter, it requires a single type as a string that represents one of the 140+ FHIR Resource Types. The url points to the relative path on the source, or the absolute path of the https resource.
  
  parameter[].where(name=’storageDetail’) must be https, file, ibm-cos, aws-s3, or azure-blob and must match the storageProvider’s type in the fhir-server-config.json.
  
  { "resourceType": "Parameters", "id": "30321130-5032-49fb-be54-9b8b82b2445a", "parameter": [ { "name": "inputSource", "valueUri": "https://my-server/source-fhir-server" }, { "name": "inputFormat", "valueString": "application/fhir+ndjson" }, { "name": "input", "part": [ { "name": "type", "valueString": "Patient" }, { "name": "url", "valueUrl": "test-import.ndjson" } ] }, { "name": "storageDetail", "valueString": "ibm-cos" } ] }

Let’s Go: Start-to-End with the IBM FHIR Server and Bulk Data

Setup Minio

Setup IBM FHIR Server

Addendum

Comments

2 responses to “Let’s Go: Start-to-End with the IBM FHIR Server and Bulk Data”

More posts

Red Hat OpenShift Container Platform 4.18 Now Available on IBM Power

Nest Accelerator and Urandom… I think

Kernel Stack Trace

Nice article on name,version and references from Red Hat