Let’s Go: Start-to-End with the IBM FHIR Server and Bulk Data

In IBM FHIR Server 4.6.0, IBM FHIR Server has refactored the supported Bulk Data operations that support the HL7 FHIR BulkDataAccess IG: STU1 and the Proposal for $import Operation.

To demonstrate the new features in a quick script, use the following to setup an S3 demonstration from the start-to-end.

  • Setup Minio
  • Setup IBM FHIR Server
  • Load Sample Data
  • Run $import
  • Run $export

Setup Minio

  1. Pull the minio docker image

  2. Create a local data directory for data

mkdir -p minio-data/
  1. Download demonstration only certificates/keys

curl -L -o private.key raw.githubusercontent.com/IBM/FHIR/main/build/docker/minio/private.key

curl -L -o public.crt https://raw.githubusercontent.com/IBM/FHIR/main/build/docker/minio/public.crt

Note, these are demonstration only Private Key and Public cert pair.

  1. Create the Minio container
export MINIO_ROOT_PASSWORD=$(openssl rand -base64 20)
docker run -d --rm -p 9000:9000 \
  --name minio \
  -h minio \
  -e "MINIO_ROOT_USER=fhir-s3-admin" \
  -e "MINIO_ROOT_PASSWORD=${MINIO_ROOT_PASSWORD}" \
  -e "MINIO_ACCESS_KEY=fhirAccessKey" \
  -e "MINIO_SECRET_KEY=${MINIO_ROOT_PASSWORD}" \
  -v $(pwd)/minio-data:/data \
  -v $(pwd)/public.crt:/root/.minio/certs/public.crt \
  -v $(pwd)/private.key:/root/.minio/certs/private.key \
  minio/minio server /data

Note, the Minio container will create local certificates.

  1. Using your MINIO_ROOT_PASSWORD echo ${MINIO_ROOT_PASSWORD}, log in to your Minio instance.
  1. Create a Bucket (folder), look in the lower left. Name it fhirbulkdata.
Create Bucket

You should see the bucket.

Bucket

6. Download the [Sample NDJSON](https://github.com/IBM/FHIR/blob/main/fhir-server-test/src/test/resources/testdata/import-operation/test-import.ndjson)

`curl -L https://raw.githubusercontent.com/IBM/FHIR/main/fhir-server-test/src/test/resources/testdata/import-operation/test-import.ndjson -o test-import.ndjson`

We’re going to upload this file to the fhir-bulkdata bucket.

7. Click Upload File > Select the test-import.ndjson. Click Upload. Confirm you see the file in the bucket.

File Uploaded to MINIO

Setup IBM FHIR Server

  1. Download the fhir-server-config.json

curl -L https://raw.githubusercontent.com/IBM/FHIR/main/fhir-server/liberty-config/config/default/fhir-server-config.json -o fhir-server-config.json

  1. Update the fhir-server-config.json fhirServer/bulkdata with the following snippet to use hmac and bulkdata. Be sure to update fhirServer/bulkdata/storageProviders/minio/accessKeyId to fhirAccessKey and fhirServer/bulkdata/storageProviders/minio/secretAccessKey to your MINIO_ROOT_PASSWORD.
    "bulkdata": {
            "enabled": true,
            "core": {
                "api": {
                    "url": "https://localhost:9443/ibm/api/batch",
                    "user": "fhiradmin",
                    "password": "change-password",
                    "truststore": "resources/security/fhirTrustStore.p12",
                    "truststorePassword": "change-password", 
                    "trustAll": true
                },
                "cos" : { 
                    "partUploadTriggerSizeMB": 10,
                    "objectSizeThresholdMB": 200,
                    "objectResourceCountThreshold": 200000,
                    "useServerTruststore": true
                },
                "file" : { 
                    "writeTriggerSizeMB": 1,
                    "sizeThresholdMB": 200,
                    "resourceCountThreshold": 200000
                },
                "pageSize": 100,
                "batchIdEncryptionKey": "change-password",
                "maxPartitions": 3, 
                "maxInputs": 5
            },
            "storageProviders": {
                "default" : {
                    "type": "file",
                    "_type": "ibm-cos|aws-s3|file|https",
                    "validBaseUrls": [],
                    "fileBase": "/output/bulkdata",
                    "enableParquet": false,
                    "disableBaseUrlValidation": true,
                    "exportPublic": true,
                    "disableOperationOutcomes": true,
                    "duplicationCheck": false, 
                    "validateResources": false
                },
                "minio" : {
                    "type": "aws-s3",
                    "bucketName": "fhirbulkdata",
                    "location": "us",
                    "endpointInternal": "https://minio:9000",
                    "endpointExternal": "https://localhost:9000",
                    "auth" : {
                        "type": "hmac",
                        "accessKeyId": "minio",
                        "secretAccessKey": "change-password"
                    },
                    "enableParquet": false,
                    "disableBaseUrlValidation": true,
                    "exportPublic": true,
                    "disableOperationOutcomes": true,
                    "duplicationCheck": false, 
                    "validateResources": false, 
                    "create": false,
                    "presigned": true
                }
            }
        }
  1. Create the /output/bulkdata folder
mkdir -p bulkdata
  1. Download the ibm-fhir-server docker image
docker pull ibmcom/ibm-fhir-server:latest

$ docker pull ibmcom/ibm-fhir-server:latest
latest: Pulling from ibmcom/ibm-fhir-server
13897c84ca57: Pull complete 
64607cc74f9c: Pull complete 
16ac69f16a48: Pull complete 
54eaf4e89615: Pull complete 
215ce34bfb04: Pull complete 
23fc81994ef3: Pull complete 
358f256d50db: Pull complete 
2c5ee9f8d67e: Pull complete 
df1b462e3122: Pull complete 
64782415f212: Pull complete 
5871c358d552: Pull complete 
931b2daafdbb: Pull complete 
a77b0c08b2ad: Pull complete 
8405f4cee973: Pull complete 
a0d476c441e5: Pull complete 
d504580ab646: Pull complete 
b68755501c7d: Pull complete 
4071f846ff8d: Pull complete 
Digest: sha256:1829c8ab601355efecc190ef667fba93d92cbbd2e9305c109f736851c3ba97f4
Status: Downloaded newer image for ibmcom/ibm-fhir-server:latest
docker.io/ibmcom/ibm-fhir-server:latest
  1. Start up the IBM FHIR Server with the fhir-server-config.json we just modified.
docker run --rm -d -p 9443:9443 -e BOOTSTRAP_DB=true \
    -v $(pwd)/fhir-server-config.json:/config/config/default/fhir-server-config.json \
    -v $(pwd)/bulkdata:/output/bulkdata \
    ibmcom/ibm-fhir-server

You’ll see the container id output 60a5f1cae6d677d80772f1736db1be74836a8a4845fcccc81286b7c557bc2d86.

  1. Check that the applications are started using the container id.
$ docker logs 60a | grep -i started
[4/21/21, 20:55:58:449 UTC] 00000001 FrameworkMana I   CWWKE0002I: The kernel started after 1.43 seconds
[4/21/21, 20:55:58:464 UTC] 0000002a FeatureManage I   CWWKF0007I: Feature update started.
[4/21/21, 20:56:01:328 UTC] 00000030 AppMessageHel A   CWWKZ0001I: Application fhir-openapi started in 1.588 seconds.
[4/21/21, 20:56:03:141 UTC] 00000031 AppMessageHel A   CWWKZ0001I: Application fhir-bulkdata-webapp started in 3.402 seconds.
[4/21/21, 20:56:07:824 UTC] 0000002d AppMessageHel A   CWWKZ0001I: Application fhir-server-webapp started in 7.871 seconds.
[4/21/21, 20:56:07:868 UTC] 0000002a TCPPort       I   CWWKO0219I: TCP Channel defaultHttpEndpoint-ssl has been started and is now listening for requests on host *  (IPv4) port 9443.
[4/21/21, 20:56:07:880 UTC] 0000002a FeatureManage A   CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 10.885 seconds.
  1. Download the Sample Data

curl -L https://raw.githubusercontent.com/IBM/FHIR/main/fhir-server-test/src/test/resources/testdata/everything-operation/Antonia30_Acosta403.json -o Antonia30_Acosta403.json

  1. Load the Sample Data bundle to the IBM FHIR Server
curl -k --location --request POST 'https://localhost:9443/fhir-server/api/v4' \
--header 'Content-Type: application/fhir+json' \
--user "fhiruser:${DUMMY_PASSWORD}" \
--data-binary  "@Antonia30_Acosta403.json" -o response.json
  1. Scan the response.json for any status that is not "status": "201". For example, the status is in the family of User Request Error or Server Side Error.

  2. Run the $import operation

curl -k --location --request POST 'https://localhost:9443/fhir-server/api/v4/$import'     --header 'X-FHIR-TENANT-ID: default'     --header 'Content-Type: application/fhir+json'     --header 'X-FHIR-BULKDATA-PROVIDER: minio'     --header 'X-FHIR-BULKDATA-PROVIDER-OUTCOME: minio'     --user 'fhiruser:change-password'     --header 'Cookie: __cfduid=d02cd7c4e9e90b7f71c16bdd22da08fb41616718585'     --data-raw '{
        "resourceType": "Parameters",
        "id": "30321130-5032-49fb-be54-9b8b82b2445a",
        "parameter": [
            {
                "name": "inputSource",
                "valueUri": "https://my-server/source-fhir-server"
            },
            {
                "name": "inputFormat",
                "valueString": "application/fhir+ndjson"
            },
            {
                "name": "input",
                "part": [
                    {
                        "name": "type",
                        "valueString": "Patient"
                    },
                    {
                        "name": "url",
                        "valueUrl": "test-import.ndjson"
                    }
                ]
            },
            {
                "name": "storageDetail",
                "valueString": "ibm-cos"
            }
        ]
    }' -v
  1. Grab the Location response header.
< HTTP/2 202
< content-location: https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=Zhkno6biH1Qwh2S6VDBT6w
< date: Fri, 23 Apr 2021 00:08:06 GMT
< content-length: 0
< content-language: en-US
  1. Check the export status using the URL
curl -k --location --request GET 'https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=LqzauvqtHSmkpChVHo%2B1MQ' \
    --header 'X-FHIR-TENANT-ID: default' \
    --header 'Content-Type: application/fhir+json' \
    --user 'fhiruser:change-password'
  1. View the response and confirm successful export. You’ll see 4 successful imports and no failures.
{
    "transactionTime": "2021-04-23T00:22:51.368Z",
    "request": "https://localhost:9443/fhir-server/api/v4/$import",
    "requiresAccessToken": false,
    "output": [
        {
            "type": "OperationOutcome",
            "url": "test-import.ndjson_oo_success.ndjson",
            "count": 4
        }
    ],
    "error": [
        {
            "type": "OperationOutcome",
            "url": "test-import.ndjson_oo_errors.ndjson",
            "count": 0
        }
    ]
}
  1. Check the Docker Logs docker logs 7050fdf7905e1fb320fd23b731639a21e064589c68d384e8add038879bc7993
[4/23/21, 0:19:30:986 UTC] 0000002e Reporter      I   Operation Type: $import
[4/23/21, 0:19:30:987 UTC] 0000002e Reporter      I   Resource Type         failures              success               processed             totalRead             totalValidation       totalWrite            fileSize              Resource Size
[4/23/21, 0:19:30:987 UTC] 0000002e Reporter      I   Patient               0                     4                     4                     2410                  0                     4795                  4303                  1075.0
[4/23/21, 0:19:30:988 UTC] 0000002e Reporter      I    ---- Total: 4 ImportRate: 0.49 ----
  1. Run the $export operation
curl -k -v --location --request GET 'https://localhost:9443/fhir-server/api/v4/$export?_outputFormat=application/fhir+ndjson&_type=Patient' \
    --header 'X-FHIR-TENANT-ID: default' \
    --header 'Content-Type: application/fhir+json' \
    --header 'X-FHIR-BULKDATA-PROVIDER: minio' \
    --header 'X-FHIR-BULKDATA-PROVIDER-OUTCOME: minio' \
    --user 'fhiruser:change-password'
  1. Grab the Location response header, and check the export status
< HTTP/2 202
< content-location: https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=UKt4ESCnqOvAfxYWhdsfUg
< date: Fri, 23 Apr 2021 00:25:07 GMT
< content-length: 0
< content-language: en-US
  1. Call the bulkdata-status endpoint
curl -k --location --request GET 'https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=UKt4ESCnqOvAfxYWhdsfUg' \
    --header 'X-FHIR-TENANT-ID: default' \
    --header 'Content-Type: application/fhir+json' \
    --user 'fhiruser:change-password'
  1. View the response, and download the exported file.
{
    "transactionTime": "2021-04-23T00:25:07.782Z",
    "request": "https://localhost:9443/fhir-server/api/v4/$export?_outputFormat=application/fhir+ndjson&_type=Patient",
    "requiresAccessToken": false,
    "output": [
        {
            "type": "Patient",
            "url": "https://localhost:9000/fhirbulkdata/hnFKv4BahpfK2iCnlPHOvnVknZVCVbgrBKwMn2HWCgg/Patient_1.ndjson?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=fhirAccessKey%2F20210423%2Fus%2Fs3%2Faws4_request&X-Amz-Date=20210423T002551Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=21e52030687241b9380b8985d6ee4793328955a22bda3adabe0e724175947d58",
            "count": 4
        }
    ]
}
  1. Download the file.
curl -k -o data.ndjson 'https://localhost:9000/fhirbulkdata/hnFKv4BahpfK2iCnlPHOvnVknZVCVbgrBKwMn2HWCgg/Patient_1.ndjson?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=fhirAccessKey%2F20210423%2Fus%2Fs3%2Faws4_request&X-Amz-Date=20210423T002551Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=21e52030687241b9380b8985d6ee4793328955a22bda3adabe0e724175947d58'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4266  100  4266    0     0  20809      0 --:--:-- --:--:-- --:--:-- 20708
  1. Look at the content and you’ll see it’s an ndjson.
data.ndjson

You now have a working environment start-to-end of the Bulk Data on IBM FHIR Server.

Addendum

File Provider

By changing X-FHIR-BULKDATA-PROVIDER to default, you can use the mapped volume /output/bulkdata to export and import from a local directory.

Logs

You can see the job logs using:

docker logs nervous_boyd

docker exec -it nervous_boyd cat /logs/joblogs/<path to job>

Networking issues

If you hit issues where the two images don’t talk to each other, please connect them to the same network:

docker network connect –alias fhir docker_default nervous_boyd

docker network connect –alias minio docker_default minio

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.