Author: Paul

  • Recipe: IBM FHIR Server – Add Tenant Key, Revoke Tenant Key and Revoke All Tenant Keys

    The IBM FHIR Server supports many different persistence stores – Apache Derby, Postgres, and IBM Db2. The IBM Db2 offering includes a multi-tenancy feature with row-level permissions. Each tenant is assigned a unique internal id for the tenantName and tenantKey. The tenantKey on each JDBC connection is verified using a one-way hash, and supports allocating a tenant, adding a key to an existing tenant, revoking a key and revoking all tenant keys.

    Recipe

    1. Once you have created the schema on your database via the --update-schema, you can allocate a tenant.
    java -jar fhir-persistence-schema-*-cli.jar \
        --prop db.host=db \
        --prop db.port=50000 \
        --prop db.database=fhirdb \
        --prop user=db2inst1 \
        --prop password=change-password \
        --db-type db2 \
        --allocate-tenant acme32
    
    2021-06-22 11:19:13.226 00000001    INFO .common.JdbcConnectionProvider Opening connection to database: jdbc:db2://db:50000/fhirdb
    2021-06-22 11:19:14.346 00000001 WARNING ls.pool.PoolConnectionProvider Get connection took 1.120 seconds
    ...
    2021-06-22 11:22:59.846 00000001    INFO   com.ibm.fhir.schema.app.Main tenantId [29] is being pre-populated with lookup table data.
    2021-06-22 11:22:59.930 00000001    INFO   com.ibm.fhir.schema.app.Main Finished prepopulating the resource type and search parameter code/name tables tables
    2021-06-22 11:22:59.948 00000001    INFO   com.ibm.fhir.schema.app.Main Allocated tenant: acme32 [key=UsBglbyMJpSi/RjXwrkp0Bj2bAljUI+MixfAikdrcN0=] with Id = 29
    2021-06-22 11:22:59.949 00000001    INFO   com.ibm.fhir.schema.app.Main The tenantKey JSON follows:
    {"tenantKey": "UsBglbyMJpSi/RjXwrkp0Bj2bAljUI+MixfAikdrcN0="}
    2021-06-22 11:22:59.949 00000001    INFO   com.ibm.fhir.schema.app.Main Processing took: 226.765 s
    2021-06-22 11:22:59.950 00000001    INFO   com.ibm.fhir.schema.app.Main SCHEMA CHANGE: OK
    
    1. You can add a tenant key to an existing tenant.
    java -jar fhir-persistence-schema-*-cli.jar \
        --prop db.host=db \
        --prop db.port=50000 \
        --prop db.database=fhirdb \
        --prop user=db2inst1 \
        --prop password=change-password \
        --db-type db2 \
        --add-tenant-key acme32
    
    2021-06-22 11:25:53.254 00000001    INFO .common.JdbcConnectionProvider Opening connection to database: jdbc:db2://db:50000/fhirdb
    2021-06-22 11:25:54.197 00000001    INFO   com.ibm.fhir.schema.app.Main New tenant key: acme32 [key=UVFuDqD/V3v8d9S/XRjRQNu9eFTniksvxgIBbI6mEkg=]
    2021-06-22 11:25:54.201 00000001    INFO   com.ibm.fhir.schema.app.Main Processing took:   0.975 s
    2021-06-22 11:25:54.202 00000001    INFO   com.ibm.fhir.schema.app.Main SCHEMA CHANGE: OK
    
    1. List tenant to see allocated
    java -jar fhir-persistence-schema-*-cli.jar \
        --prop db.host=db \
        --prop db.port=50000 \
        --prop db.database=fhirdb \
        --prop user=db2inst1 \
        --prop password=change-password \
        --db-type db2 \
        --list-tenants
    
    2021-06-22 11:28:43.566 00000001    INFO .common.JdbcConnectionProvider Opening connection to database: jdbc:db2://db:50000/fhirdb
     TenantId     Status       TenantName Schema
           29  ALLOCATED           acme32 FHIRDATA
    2021-06-22 11:28:44.395 00000001    INFO   com.ibm.fhir.schema.app.Main Processing took:   0.854 s
    2021-06-22 11:28:44.396 00000001    INFO   com.ibm.fhir.schema.app.Main SCHEMA CHANGE: OK
    
    1. Revoking tenant key
    java -jar fhir-persistence-schema-*-cli.jar \
        --prop db.host=db \
        --prop db.port=50000 \
        --prop db.database=fhirdb \
        --prop user=db2inst1 \
        --prop password=change-password \
        --db-type db2 \
        --revoke-tenant-key acme32 \
        --tenant-key UVFuDqD/V3v8d9S/XRjRQNu9eFTniksvxgIBbI6mEkg=
    
    2021-06-22 11:27:17.280 00000001    INFO .common.JdbcConnectionProvider Opening connection to database: jdbc:db2://db:50000/fhirdb
    2021-06-22 11:27:18.112 00000001    INFO   com.ibm.fhir.schema.app.Main Tenant Key revoked for 'acme32' total removed=[1]
    2021-06-22 11:27:18.119 00000001    INFO   com.ibm.fhir.schema.app.Main Processing took:   0.867 s
    2021-06-22 11:27:18.120 00000001    INFO   com.ibm.fhir.schema.app.Main SCHEMA CHANGE: OK
    

    You’ll see one was removed.

    1. We’ll add a new key.
    java -jar fhir-persistence-schema-*-cli.jar \
        --prop db.host=db \
        --prop db.port=50000 \
        --prop db.database=fhirdb \
        --prop user=db2inst1 \
        --prop password=change-password \
        --db-type db2 \
        --add-tenant-key acme32
    
    2021-06-22 11:25:53.254 00000001    INFO .common.JdbcConnectionProvider Opening connection to database: jdbc:db2://db:50000/fhirdb
    2021-06-22 11:25:54.197 00000001    INFO   com.ibm.fhir.schema.app.Main New tenant key: acme32 [key=UVFuDqD/V3v8d9S/XRjRQNu9eFTniksvxgIBbI6mEkg=]
    2021-06-22 11:25:54.201 00000001    INFO   com.ibm.fhir.schema.app.Main Processing took:   0.975 s
    2021-06-22 11:25:54.202 00000001    INFO   com.ibm.fhir.schema.app.Main SCHEMA CHANGE: OK
    
    1. Revoking tenant key
    java -jar fhir-persistence-schema-*-cli.jar \
        --prop db.host=db \
        --prop db.port=50000 \
        --prop db.database=fhirdb \
        --prop user=db2inst1 \
        --prop password=change-password \
        --db-type db2 \
        --revoke-all-tenant-keys acme32
    
    2021-06-22 11:32:34.061 00000001    INFO .common.JdbcConnectionProvider Opening connection to database: jdbc:db2://db:50000/fhirdb
    2021-06-22 11:32:35.112 00000001    INFO   com.ibm.fhir.schema.app.Main Tenant Key revoked for 'acme32' total removed=[2]
    2021-06-22 11:32:35.144 00000001    INFO   com.ibm.fhir.schema.app.Main Processing took:   1.116 s
    2021-06-22 11:32:35.146 00000001    INFO   com.ibm.fhir.schema.app.Main SCHEMA CHANGE: OK
    

    You’ll see that two are removed.

    You now know the lifecycle for the IBM FHIR Server tenantKey – allocate, add and revoke.

    Reference

  • Recipe: IBM FHIR Server – Using Hard Delete with the $erase Operation

    The IBM FHIR Server supports hard delete using the custom FHIR Operation Framework. The operation is called $erase; you can always read the design document.

    The operation supports:

    By default, the $erase operation is not enabled, and must be set to true, and the roles allowed to execute the operation, can be tweaked to FHIROperationAdmin or FHIRUsers.

    PropertyTypeDescription
    fhirServer/operations/erase/enabledbooleanEnables the $erase operation
    fhirServer/operations/erase/allowedRoleslistThe list of allowed roles, allowed entries are: FHIRUsers every authenticated user, FHIROperationAdmin which is authenticated FHIRAdmin users
    Properties

    The fhir-server-config.json should be amended with a snippet like the following:

    "operations": {
                "erase": {
                    "enabled": true,
                    "allowedRoles": [
                        "FHIROperationAdmin",
                        "FHIRUsers"
                    ]
                }
            }

    As the $erase operation uses stored procedures for Db2 and functions for Postgres, the data schema must be updated to the 4.8.3 level. It’s also worth noting that Derby is also supported as it uses the same DAO as the Db2 and Postgres to run.

    A tip, you should search for the references to the Patient’s ResourceId, and delete the referenced resources in a Transaction Bundle. Net – Search, Assemble Bundle, Execute Batch.

    Let me show you how to run with it, and operate the $erase operation with a container (Docker: ibmcom/ibm-fhir-server).

    Recipe

    1. Download the fhir-server-config.json

    2. Start the Docker container, and capture the container id. It’s going to take a few moments to start up as it lays down the test database.

    docker run -d -p 9443:9443 -e BOOTSTRAP_DB=true -v $(pwd)/fhir-server-config.json:/config/config/default/fhir-server-config.json ibmcom/ibm-fhir-server
    cde61943964464b528eb77d132fd2a4952e0eaf43588da1c9b6bb2fa584f0608

    3. Check the logs until you see:

    docker logs cde61943964464b528eb77d132fd2a4952e0eaf43588da1c9b6bb2fa584f0608
    ...
    [6/16/21, 15:31:34:533 UTC] 0000002a FeatureManage A   CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 17.665 seconds.

    4. Download the Postman Collection for Version Specific Erase and All Versions Erase.

    5. Import the Postman Collections and Run

    You’ve seen how to use the $erase operation, and a bit more details on configuring and using it.

    Postman Collections

  • Recipe: IBM FHIR Server – Using Bulk Data with the Azure Blob Service

    The IBM FHIR Server has support for exporting and importing Bulk Data using extended operations for Bulk Data $import, $export and $bulkdata-status, which are implemented as Java Maven projects. The IBM FHIR Server uses JSR252 JavaBatch jobs running in the Open Liberty Java Batch Framework.

    The IBM FHIR Server supports storage providers S3 (aws-s3,ibm-cos), File System (file), and now Azure (azure-blob).

    I recently implemented support for the Azure Blob Service in IBM/FHIR Pull Request #2413. The code that supports the Azure client uses the OK Http library and not Netty to communicate with the backend. The storage provider is implemented as AzureProvider, and uses the Java – AppendBlobClient to facilitate export. To facilitate import, the code uses the BlobClient to read from a location in 10Kb blocks, and reassemble when a full block is reached. Importantly, this is done in a way to ensure full resources are returned up to a 2Gb limit.

    Overview Video

    If you want to use the IBM FHIR Server’s bulk data feature with the IBM FHIR Server, use the following recipe:

    Recipe – Azure Blob and Bulk Data

    1. Login to the Azure Portal

    2. Click Home

    3. Navigate the Azure services, and click on Storage Accounts.

    4. Click Create

    5. Enter the appropriate subscription and resource group

    6. Give it a name such as fhirintegrationit

    7. Click Review+Create

    8. Click Create

    You’ll see Deployment in Progress for a period of time.

    Once it is done, you see Your deployment is complete, proceed to the next steps.

    9. Click Go to Resource

    10. Click on Access Key

    11. Click Show keys and copy the key1’s Connection string. It’ll look like DefaultEndpointsProtocol=https;AccountName=fhirintegrationit;AccountKey=HIDDEN==;EndpointSuffix=core.windows.net

    12. Click on Data > Containers

    13. Click +Container

    14. Fill in the container details and call it bulkdata

    14. Click on the bulkdata container

    15. Upload blob Data – AllergyIntolerance.ndjson

    You should see the file uploaded and available.

    You are now ready to use this with the IBM FHIR Server from the main branch. If you need to clone the repo go to IBM/FHIR.

    16. Build the Maven Projects and the Docker Build. You should see “[INFO] BUILD SUCCESS” after each Maven build, and “=> naming to docker.io/ibmcom/ibm-fhir-server:latest” when the Docker build is successful.

    mvn clean install -f fhir-examples -B -DskipTests -ntp
    mvn clean install -f fhir-parent -B -DskipTests -ntp
    docker build -t ibmcom/ibm-fhir-server:latest fhir-install

    17. Download the fhir-server-config.json

    18. Open the fhir-server-config.json and update fhirServer/bulkdata/storageProviders/default entry to be the connection is updated to your Connection string from above.

    17. Start the Docker container, and capture the container id. It’s going to take a few moments to start up as it lays down the test database.

    docker run -d -p 9443:9443 -e BOOTSTRAP_DB=true -v $(pwd)/fhir-server-config.json:/config/config/default/fhir-server-config.json ibmcom/ibm-fhir-server
    cde61943964464b528eb77d132fd2a4952e0eaf43588da1c9b6bb2fa584f0608

    18. Check the logs until you see:

    docker logs cde61943964464b528eb77d132fd2a4952e0eaf43588da1c9b6bb2fa584f0608
    ...
    [6/16/21, 15:31:34:533 UTC] 0000002a FeatureManage A   CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 17.665 seconds.

    19. Create an $import request and capture the content-location header.

    20. Check and poll until you get response code 200 (you should only get 202 in the interim) until you see (note 99 failed validation which is expected):

    curl -v -k --user 'fhiruser:change-password' 'https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=UKt4ESCnqOvAfxYWhdsfUg'
    
    < HTTP/2 200
    < content-type: application/json
    < date: Wed, 16 Jun 2021 16:01:54 GMT
    < content-language: en-US
    < content-length: 501
    <
    {
        "transactionTime": "2021-06-16T15:59:53.086Z",
        "request": "https://localhost:9443/fhir-server/api/v4/$import",
        "requiresAccessToken": false,
        "output": [
            {
                "type": "OperationOutcome",
                "url": "AllergyIntolerance.ndjson_oo_success.ndjson",
                "count": 1600
            }
        ],
        "error": [
            {
                "type": "OperationOutcome",
                "url": "AllergyIntolerance.ndjson_oo_errors.ndjson",
                "count": 99
            }
        ]
    * Connection #0 to host localhost left intact
    }

    21. We can check to see if there is data on the server, and there are 1600 resources.

    curl -v -k --user 'fhiruser:change-password' 'https://localhost:9443/fhir-server/api/v4/AllergyIntolerance?_format=json'
    
    {"resourceType":"Bundle","id":"d485339f-49e5-4b11-a1be-4ae06edd0b16","type":"searchset","total":1600,"link"...}

    22. Now we run $export and capture the content-location:

    curl --location --request GET -k -v 'https://localhost:9443/fhir-server/api/v4/$export?_outputFormat=application/fhir+ndjson&_type=AllergyIntolerance' \
    --header 'X-FHIR-TENANT-ID: default' \
    --header 'Content-Type: application/fhir+json' \
    --header 'X-FHIR-BULKDATA-PROVIDER: default' \
    --header 'X-FHIR-BULKDATA-PROVIDER-OUTCOME: default' \
    --header 'Authorization: Basic ZmhpcnVzZXI6Y2hhbmdlLXBhc3N3b3Jk'
    
    < HTTP/2 202
    < content-location: https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=LqzauvqtHSmkpChVHo%2B1MQ
    < date: Wed, 16 Jun 2021 16:39:19 GMT
    < content-length: 0
    < content-language: en-US
    <
    * Connection #0 to host localhost left intact
    * Closing connection 0

    23. Poll and check the response 202 or 200 and the contents of the response. (empty until done)

    curl -k --user 'fhiruser:change-password' 'https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=LqzauvqtHSmkpChVHo%2B1MQ'
    {
        "transactionTime": "2021-06-16T16:39:26.551Z",
        "request": "https://localhost:9443/fhir-server/api/v4/$export?_outputFormat=application/fhir+ndjson&_type=AllergyIntolerance",
        "requiresAccessToken": false,
        "output": [
            {
                "type": "AllergyIntolerance",
                "url": "c7GRY0uFsu-hO_QlquVZJr_7hTg5c_m6SHlZ1z3Z7J4/AllergyIntolerance_1.ndjson",
                "count": 1600
            }
        ]
    }

    24. Check Azure Blob Service and you’ll see the folder and file on Azure Blob Service

    25. Download and check there are 1600 lines.

    You now know how to use Azure and Bulk Data.

  • IBM FHIR Server – Debug Drop Tablespace

    I’ve been debugging a drop tablespace issue on Db2 – IBM/FHIR: 2354. The core issue was a timing problem with lots of partitions dettaching. This adds a delay to the dropTablespace so the async operation can complete, and cleanly exit with a specific error code so downstream consumers can work around the issue.

    When debugging why a partition hasn’t dettached, I found that it’s worth checking the details when a drop tablespace fails:

    [db2inst1@53fe3a4d3ad2 ~]$ db2 list utilities show detail
    
    ID                               = 18435
    Type                             = ASYNCHRONOUS PARTITION DETACH
    Database Name                    = FHIRDB
    Member Number                    = 0
    Description                      = Finalize the detach for partition '3' of table 'FHIRDATA.PARAMETER_NAMES'
    Start Time                       = 06/08/2021 16:31:05.526513
    State                            = Executing
    Invocation Type                  = Automatic
    Progress Monitoring:
          Description                = Performing detach operation and
     making the target table available; new compilations blocked
          Start Time                 = 06/08/2021 16:31:10.836936
    

    I’ve attached a useful partition.sql to demo partitions and check the system catalog.

    Links

    Example Code

  • IBM FHIR Server – Debugging Tips

    Checking a Postgres Function Definition

    I needed to verify my postgres function.

    1. Shows the contents of the functions in the schema.
    SELECT pg_get_functiondef(f.oid)
    FROM pg_catalog.pg_proc f
    INNER JOIN pg_catalog.pg_namespace n ON (f.pronamespace = n.oid)
    WHERE n.nspname = 'fhirdata';
    
    1. Show all the details of the functions in the schema.
    SELECT *
    FROM pg_catalog.pg_proc f
    INNER JOIN pg_catalog.pg_namespace n ON (f.pronamespace = n.oid)
    WHERE n.nspname = 'fhirdata';
    

    Tracing Bulk Data with Cloud Object Storage

    I wanted to figure out why my code was failing to connect to the backend S3 bucket.

    I used the environment variable in my docker image called TRACE_SPEC. TRACE_SPEC is loaded into the logging as the traceSpecification.

    I set this to *=info:com.ibm.cloud.*=FINEST which spits out great detail to S3 using the IBM COS SDK.

    The output looks like:

    With this level of trace, you can really dive into the connection, and determine what is going on.

    Note, if you want the whole picture of what is happening with COS and JavaBatch and BulkData, use the following:

    *=info:com.ibm.fhir.*=finer:RRA=all:WAS.j2c=all:com.ibm.ws.jdbc.*=all:Transaction=all:ConnLeakLogic=all:Transaction=all:com.ibm.ws.transaction.services.WebAppTransactionCollaboratorImpl=all:RRA=all:com.ibm.cloud.*=FINEST

  • BulkData: rclone with S3

    This was super helpful for debugging S3 files/folders.

    brew install rclone
    rclone rcd --config rclone.conf  --rc-web-gui --no-check-certificate --rc-user rc --rc-pass rc
    
    [minio]
    type = s3
    env_auth = false
    access_key_id = minio
    secret_access_key = demooo-password
    region = us-east-1
    endpoint = https://localhost:9000
    location_constraint = 
    server_side_encryption = 
    

    rclone sync

    rclone sync --config rclone-s3.conf --no-check-certificate -i fhir-performance:fhir-performance $(pwd)
    
  • GitHub Actions: Paging file is too small

    If you hit this issue, per the GitHub community it’s a Windows issue that can be worked around with a Pagefile. link

    [ERROR] OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000700000000, 553648128, 0) failed; error='The paging file is too small for this operation to complete' (DOS error/errno=1455)

    I used the discussed GitHub Action configure-pagefile

    You can see the solution inline at GitHub Actions Workflow

        - name: Configure Pagefile
          uses: al-cheb/configure-pagefile-action@v1.2
          with:
            minimum-size: 8GB
            maximum-size: 8GB
            disk-root: "C:"
    
  • Some examples of FHIR Path

    Here are some examples I generated from a demo with the IBM FHIR Server

    Extract the references from the first subject

    Bundle.entry.resource.ofType(Observation).subject[0].reference.value

    Output Patient/ebb16c62-cb06-4ff8-8ce8-ccb865e7a240

    Extract the references from the first subject alternative

    Bundle.entry.resource.ofType(Observation).subject.first().reference.value

    Output Patient/ebb16c62-cb06-4ff8-8ce8-ccb865e7a240

    Select the first Patient reference where the Diastolic is over 90

    Bundle.entry.resource.ofType(Observation).where(component.where(code.coding.code = '8462-4').value.where( value > 90.0).exists()).subject.reference.value.first()

    Output Patient/5fabfccd-254e-42af-bed2-84199a5c05f2

    Select the Patient’s first name where the reference matches the logical id

    Bundle.entry.resource.ofType(Patient).where(id = '" + v.split("/")[1] + "').name.given.value

    Output John

  • Let’s Go: Start-to-End with the IBM FHIR Server and Bulk Data

    In IBM FHIR Server 4.6.0, IBM FHIR Server has refactored the supported Bulk Data operations that support the HL7 FHIR BulkDataAccess IG: STU1 and the Proposal for $import Operation.

    To demonstrate the new features in a quick script, use the following to setup an S3 demonstration from the start-to-end.

    • Setup Minio
    • Setup IBM FHIR Server
    • Load Sample Data
    • Run $import
    • Run $export

    Setup Minio

    1. Pull the minio docker image

    2. Create a local data directory for data

    mkdir -p minio-data/
    
    1. Download demonstration only certificates/keys

    curl -L -o private.key raw.githubusercontent.com/IBM/FHIR/main/build/docker/minio/private.key

    curl -L -o public.crt https://raw.githubusercontent.com/IBM/FHIR/main/build/docker/minio/public.crt

    Note, these are demonstration only Private Key and Public cert pair.

    1. Create the Minio container
    export MINIO_ROOT_PASSWORD=$(openssl rand -base64 20)
    docker run -d --rm -p 9000:9000 \
      --name minio \
      -h minio \
      -e "MINIO_ROOT_USER=fhir-s3-admin" \
      -e "MINIO_ROOT_PASSWORD=${MINIO_ROOT_PASSWORD}" \
      -e "MINIO_ACCESS_KEY=fhirAccessKey" \
      -e "MINIO_SECRET_KEY=${MINIO_ROOT_PASSWORD}" \
      -v $(pwd)/minio-data:/data \
      -v $(pwd)/public.crt:/root/.minio/certs/public.crt \
      -v $(pwd)/private.key:/root/.minio/certs/private.key \
      minio/minio server /data
    

    Note, the Minio container will create local certificates.

    1. Using your MINIO_ROOT_PASSWORD echo ${MINIO_ROOT_PASSWORD}, log in to your Minio instance.
    1. Create a Bucket (folder), look in the lower left. Name it fhirbulkdata.
    Create Bucket

    You should see the bucket.

    Bucket

    6. Download the [Sample NDJSON](https://github.com/IBM/FHIR/blob/main/fhir-server-test/src/test/resources/testdata/import-operation/test-import.ndjson)

    `curl -L https://raw.githubusercontent.com/IBM/FHIR/main/fhir-server-test/src/test/resources/testdata/import-operation/test-import.ndjson -o test-import.ndjson`

    We’re going to upload this file to the fhir-bulkdata bucket.

    7. Click Upload File > Select the test-import.ndjson. Click Upload. Confirm you see the file in the bucket.

    File Uploaded to MINIO

    Setup IBM FHIR Server

    1. Download the fhir-server-config.json

    curl -L https://raw.githubusercontent.com/IBM/FHIR/main/fhir-server/liberty-config/config/default/fhir-server-config.json -o fhir-server-config.json

    1. Update the fhir-server-config.json fhirServer/bulkdata with the following snippet to use hmac and bulkdata. Be sure to update fhirServer/bulkdata/storageProviders/minio/accessKeyId to fhirAccessKey and fhirServer/bulkdata/storageProviders/minio/secretAccessKey to your MINIO_ROOT_PASSWORD.
        "bulkdata": {
                "enabled": true,
                "core": {
                    "api": {
                        "url": "https://localhost:9443/ibm/api/batch",
                        "user": "fhiradmin",
                        "password": "change-password",
                        "truststore": "resources/security/fhirTrustStore.p12",
                        "truststorePassword": "change-password", 
                        "trustAll": true
                    },
                    "cos" : { 
                        "partUploadTriggerSizeMB": 10,
                        "objectSizeThresholdMB": 200,
                        "objectResourceCountThreshold": 200000,
                        "useServerTruststore": true
                    },
                    "file" : { 
                        "writeTriggerSizeMB": 1,
                        "sizeThresholdMB": 200,
                        "resourceCountThreshold": 200000
                    },
                    "pageSize": 100,
                    "batchIdEncryptionKey": "change-password",
                    "maxPartitions": 3, 
                    "maxInputs": 5
                },
                "storageProviders": {
                    "default" : {
                        "type": "file",
                        "_type": "ibm-cos|aws-s3|file|https",
                        "validBaseUrls": [],
                        "fileBase": "/output/bulkdata",
                        "disableBaseUrlValidation": true,
                        "exportPublic": true,
                        "disableOperationOutcomes": true,
                        "duplicationCheck": false, 
                        "validateResources": false
                    },
                    "minio" : {
                        "type": "aws-s3",
                        "bucketName": "fhirbulkdata",
                        "location": "us",
                        "endpointInternal": "https://minio:9000",
                        "endpointExternal": "https://localhost:9000",
                        "auth" : {
                            "type": "hmac",
                            "accessKeyId": "minio",
                            "secretAccessKey": "change-password"
                        },
                        "disableBaseUrlValidation": true,
                        "exportPublic": true,
                        "disableOperationOutcomes": true,
                        "duplicationCheck": false, 
                        "validateResources": false, 
                        "create": false,
                        "presigned": true
                    }
                }
            }
    
    1. Create the /output/bulkdata folder
    mkdir -p bulkdata
    
    1. Download the ibm-fhir-server docker image
    docker pull ibmcom/ibm-fhir-server:latest
    
    $ docker pull ibmcom/ibm-fhir-server:latest
    latest: Pulling from ibmcom/ibm-fhir-server
    13897c84ca57: Pull complete 
    64607cc74f9c: Pull complete 
    16ac69f16a48: Pull complete 
    54eaf4e89615: Pull complete 
    215ce34bfb04: Pull complete 
    23fc81994ef3: Pull complete 
    358f256d50db: Pull complete 
    2c5ee9f8d67e: Pull complete 
    df1b462e3122: Pull complete 
    64782415f212: Pull complete 
    5871c358d552: Pull complete 
    931b2daafdbb: Pull complete 
    a77b0c08b2ad: Pull complete 
    8405f4cee973: Pull complete 
    a0d476c441e5: Pull complete 
    d504580ab646: Pull complete 
    b68755501c7d: Pull complete 
    4071f846ff8d: Pull complete 
    Digest: sha256:1829c8ab601355efecc190ef667fba93d92cbbd2e9305c109f736851c3ba97f4
    Status: Downloaded newer image for ibmcom/ibm-fhir-server:latest
    docker.io/ibmcom/ibm-fhir-server:latest
    
    1. Start up the IBM FHIR Server with the fhir-server-config.json we just modified.
    docker run --rm -d -p 9443:9443 -e BOOTSTRAP_DB=true \
        -v $(pwd)/fhir-server-config.json:/config/config/default/fhir-server-config.json \
        -v $(pwd)/bulkdata:/output/bulkdata \
        ibmcom/ibm-fhir-server
    

    You’ll see the container id output 60a5f1cae6d677d80772f1736db1be74836a8a4845fcccc81286b7c557bc2d86.

    1. Check that the applications are started using the container id.
    $ docker logs 60a | grep -i started
    [4/21/21, 20:55:58:449 UTC] 00000001 FrameworkMana I   CWWKE0002I: The kernel started after 1.43 seconds
    [4/21/21, 20:55:58:464 UTC] 0000002a FeatureManage I   CWWKF0007I: Feature update started.
    [4/21/21, 20:56:01:328 UTC] 00000030 AppMessageHel A   CWWKZ0001I: Application fhir-openapi started in 1.588 seconds.
    [4/21/21, 20:56:03:141 UTC] 00000031 AppMessageHel A   CWWKZ0001I: Application fhir-bulkdata-webapp started in 3.402 seconds.
    [4/21/21, 20:56:07:824 UTC] 0000002d AppMessageHel A   CWWKZ0001I: Application fhir-server-webapp started in 7.871 seconds.
    [4/21/21, 20:56:07:868 UTC] 0000002a TCPPort       I   CWWKO0219I: TCP Channel defaultHttpEndpoint-ssl has been started and is now listening for requests on host *  (IPv4) port 9443.
    [4/21/21, 20:56:07:880 UTC] 0000002a FeatureManage A   CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 10.885 seconds.
    
    1. Download the Sample Data

    curl -L https://raw.githubusercontent.com/IBM/FHIR/main/fhir-server-test/src/test/resources/testdata/everything-operation/Antonia30_Acosta403.json -o Antonia30_Acosta403.json

    1. Load the Sample Data bundle to the IBM FHIR Server
    curl -k --location --request POST 'https://localhost:9443/fhir-server/api/v4' \
    --header 'Content-Type: application/fhir+json' \
    --user "fhiruser:${DUMMY_PASSWORD}" \
    --data-binary  "@Antonia30_Acosta403.json" -o response.json
    
    1. Scan the response.json for any status that is not "status": "201". For example, the status is in the family of User Request Error or Server Side Error.

    2. Run the $import operation

    curl -k --location --request POST 'https://localhost:9443/fhir-server/api/v4/$import'     --header 'X-FHIR-TENANT-ID: default'     --header 'Content-Type: application/fhir+json'     --header 'X-FHIR-BULKDATA-PROVIDER: minio'     --header 'X-FHIR-BULKDATA-PROVIDER-OUTCOME: minio'     --user 'fhiruser:change-password'     --header 'Cookie: __cfduid=d02cd7c4e9e90b7f71c16bdd22da08fb41616718585'     --data-raw '{
            "resourceType": "Parameters",
            "id": "30321130-5032-49fb-be54-9b8b82b2445a",
            "parameter": [
                {
                    "name": "inputSource",
                    "valueUri": "https://my-server/source-fhir-server"
                },
                {
                    "name": "inputFormat",
                    "valueString": "application/fhir+ndjson"
                },
                {
                    "name": "input",
                    "part": [
                        {
                            "name": "type",
                            "valueString": "Patient"
                        },
                        {
                            "name": "url",
                            "valueUrl": "test-import.ndjson"
                        }
                    ]
                },
                {
                    "name": "storageDetail",
                    "valueString": "ibm-cos"
                }
            ]
        }' -v
    1. Grab the Location response header.
    < HTTP/2 202
    < content-location: https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=Zhkno6biH1Qwh2S6VDBT6w
    < date: Fri, 23 Apr 2021 00:08:06 GMT
    < content-length: 0
    < content-language: en-US
    1. Check the export status using the URL
    curl -k --location --request GET 'https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=LqzauvqtHSmkpChVHo%2B1MQ' \
        --header 'X-FHIR-TENANT-ID: default' \
        --header 'Content-Type: application/fhir+json' \
        --user 'fhiruser:change-password'
    
    1. View the response and confirm successful export. You’ll see 4 successful imports and no failures.
    {
        "transactionTime": "2021-04-23T00:22:51.368Z",
        "request": "https://localhost:9443/fhir-server/api/v4/$import",
        "requiresAccessToken": false,
        "output": [
            {
                "type": "OperationOutcome",
                "url": "test-import.ndjson_oo_success.ndjson",
                "count": 4
            }
        ],
        "error": [
            {
                "type": "OperationOutcome",
                "url": "test-import.ndjson_oo_errors.ndjson",
                "count": 0
            }
        ]
    }
    
    1. Check the Docker Logs docker logs 7050fdf7905e1fb320fd23b731639a21e064589c68d384e8add038879bc7993
    [4/23/21, 0:19:30:986 UTC] 0000002e Reporter      I   Operation Type: $import
    [4/23/21, 0:19:30:987 UTC] 0000002e Reporter      I   Resource Type         failures              success               processed             totalRead             totalValidation       totalWrite            fileSize              Resource Size
    [4/23/21, 0:19:30:987 UTC] 0000002e Reporter      I   Patient               0                     4                     4                     2410                  0                     4795                  4303                  1075.0
    [4/23/21, 0:19:30:988 UTC] 0000002e Reporter      I    ---- Total: 4 ImportRate: 0.49 ----
    
    1. Run the $export operation
    curl -k -v --location --request GET 'https://localhost:9443/fhir-server/api/v4/$export?_outputFormat=application/fhir+ndjson&_type=Patient' \
        --header 'X-FHIR-TENANT-ID: default' \
        --header 'Content-Type: application/fhir+json' \
        --header 'X-FHIR-BULKDATA-PROVIDER: minio' \
        --header 'X-FHIR-BULKDATA-PROVIDER-OUTCOME: minio' \
        --user 'fhiruser:change-password'
    
    1. Grab the Location response header, and check the export status
    < HTTP/2 202
    < content-location: https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=UKt4ESCnqOvAfxYWhdsfUg
    < date: Fri, 23 Apr 2021 00:25:07 GMT
    < content-length: 0
    < content-language: en-US
    
    1. Call the bulkdata-status endpoint
    curl -k --location --request GET 'https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=UKt4ESCnqOvAfxYWhdsfUg' \
        --header 'X-FHIR-TENANT-ID: default' \
        --header 'Content-Type: application/fhir+json' \
        --user 'fhiruser:change-password'
    
    1. View the response, and download the exported file.
    {
        "transactionTime": "2021-04-23T00:25:07.782Z",
        "request": "https://localhost:9443/fhir-server/api/v4/$export?_outputFormat=application/fhir+ndjson&_type=Patient",
        "requiresAccessToken": false,
        "output": [
            {
                "type": "Patient",
                "url": "https://localhost:9000/fhirbulkdata/hnFKv4BahpfK2iCnlPHOvnVknZVCVbgrBKwMn2HWCgg/Patient_1.ndjson?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=fhirAccessKey%2F20210423%2Fus%2Fs3%2Faws4_request&X-Amz-Date=20210423T002551Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=21e52030687241b9380b8985d6ee4793328955a22bda3adabe0e724175947d58",
                "count": 4
            }
        ]
    }
    
    1. Download the file.
    curl -k -o data.ndjson 'https://localhost:9000/fhirbulkdata/hnFKv4BahpfK2iCnlPHOvnVknZVCVbgrBKwMn2HWCgg/Patient_1.ndjson?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=fhirAccessKey%2F20210423%2Fus%2Fs3%2Faws4_request&X-Amz-Date=20210423T002551Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=21e52030687241b9380b8985d6ee4793328955a22bda3adabe0e724175947d58'
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  4266  100  4266    0     0  20809      0 --:--:-- --:--:-- --:--:-- 20708
    
    1. Look at the content and you’ll see it’s an ndjson.
    data.ndjson

    You now have a working environment start-to-end of the Bulk Data on IBM FHIR Server.

    Addendum

    File Provider

    By changing X-FHIR-BULKDATA-PROVIDER to default, you can use the mapped volume /output/bulkdata to export and import from a local directory.

    Logs

    You can see the job logs using:

    docker logs nervous_boyd

    docker exec -it nervous_boyd cat /logs/joblogs/<path to job>

    Networking issues

    If you hit issues where the two images don’t talk to each other, please connect them to the same network:

    docker network connect –alias fhir docker_default nervous_boyd

    docker network connect –alias minio docker_default minio

  • Question: How to setup the IBM FHIR Server with fhir-audit?

    The IBM FHIR Server supports audit events for FHIR operations (CREATE-READ-UPDATE-DELETE-OPERATION) in Cloud Auditing Data Federation (CADF) and HL7 FHIR AuditEvent and pushing the events to an Apache Kafka backend. You can read more about it on the IBM FHIR Server site.

    Let’s spin up an IBM FHIR Server with fhir-audit and see what we get with a running container.

    1. Create an Event Streams instance on IBM Cloud Console

    2. Create a topic FHIR_AUDIT

    Create
    1. Click Service Credentials, click Create New Credential, Click Add
    Add a Credential
    1. Expand your new Service Credential, and copy the JSON and store locally, we’ll use it in a subsequent step.
    curl -L https://raw.githubusercontent.com/IBM/FHIR/main/fhir-server/liberty-config/config/default/fhir-server-config-audit-environment.json -o fhir-server-config-audit-environment.json

    The file contains a setting for loading the properties from the Environment, and some basic settings:

            "audit": {
                "serviceClassName" : "com.ibm.fhir.audit.impl.KafkaService",
                "serviceProperties" : {
                    "auditTopic": "FHIR_AUDIT",
                    "geoCity": "Dallas",
                    "geoState": "TX",
                    "geoCounty": "US"
                }
            }
    1. Download the docker image
    docker pull ibmcom/ibm-fhir-server:latest
    
    $ docker pull ibmcom/ibm-fhir-server:latest
    latest: Pulling from ibmcom/ibm-fhir-server
    13897c84ca57: Pull complete 
    64607cc74f9c: Pull complete 
    16ac69f16a48: Pull complete 
    54eaf4e89615: Pull complete 
    215ce34bfb04: Pull complete 
    23fc81994ef3: Pull complete 
    358f256d50db: Pull complete 
    2c5ee9f8d67e: Pull complete 
    df1b462e3122: Pull complete 
    64782415f212: Pull complete 
    5871c358d552: Pull complete 
    931b2daafdbb: Pull complete 
    a77b0c08b2ad: Pull complete 
    8405f4cee973: Pull complete 
    a0d476c441e5: Pull complete 
    d504580ab646: Pull complete 
    b68755501c7d: Pull complete 
    4071f846ff8d: Pull complete 
    Digest: sha256:1829c8ab601355efecc190ef667fba93d92cbbd2e9305c109f736851c3ba97f4
    Status: Downloaded newer image for ibmcom/ibm-fhir-server:latest
    docker.io/ibmcom/ibm-fhir-server:latest
    1. Make the JSON a single line
    cat << EOF | tr -d '\n'
    {
      "api_key": "credentialapikey",
      "apikey": "credentialapikey",
      "iam_apikey_description": "Auto-generated for key 01ba3165-85d9-410b-ad1a-1111",
      "iam_apikey_name": "Service credentials-1",
      "iam_role_crn": "crn:v1:bluemix:public:iam::::serviceRole:Manager",
      "iam_serviceid_crn": "crn:v1:bluemix:public:iam-identity::a/11111::serviceid:ServiceId-864b549f-ff90-4b24-84f2-1111",
      "instance_id": "6b00cc8c-26ec-4c42-a1e8-f4da5b9f71e7",
      "kafka_admin_url": "https://admin.eventstreams.cloud.ibm.com",
      "kafka_brokers_sasl": [
        "broker-3.eventstreams.cloud.ibm.com:9093",
        "broker-5.eventstreams.cloud.ibm.com:9093",
        "broker-1.eventstreams.cloud.ibm.com:9093",
        "broker-2.eventstreams.cloud.ibm.com:9093",
        "broker-4.eventstreams.cloud.ibm.com:9093",
        "broker-0.eventstreams.cloud.ibm.com:9093"
      ],
      "kafka_http_url": "https://admin.eventstreams.cloud.ibm.com",
      "password": "credentialapikey",
      "user": "token"
    }
    EOF
    

    The output is:

    {  "api_key": "credentialapikey",  "apikey": "credentialapikey",  "iam_apikey_description": "Auto-generated for key 01ba3165-85d9-410b-ad1a-1111",  "iam_apikey_name": "Service credentials-1",  "iam_role_crn": "crn:v1:bluemix:public:iam::::serviceRole:Manager",  "iam_serviceid_crn": "crn:v1:bluemix:public:iam-identity::a/11111::serviceid:ServiceId-864b549f-ff90-4b24-84f2-1111",  "instance_id": "6b00cc8c-26ec-4c42-a1e8-f4da5b9f71e7",  "kafka_admin_url": "https://admin.eventstreams.cloud.ibm.com",  "kafka_brokers_sasl": [    "broker-3.eventstreams.cloud.ibm.com:9093",    "broker-5.eventstreams.cloud.ibm.com:9093",    "broker-1.eventstreams.cloud.ibm.com:9093",    "broker-2.eventstreams.cloud.ibm.com:9093",    "broker-4.eventstreams.cloud.ibm.com:9093",    "broker-0.eventstreams.cloud.ibm.com:9093"  ],  "kafka_http_url": "https://admin.eventstreams.cloud.ibm.com",  "password": "credentialapikey",  "user": "token"}
    
    1. Export ES_CONFIG
    export ES_CONFIG='{  "api_key": "credentialapikey",  "apikey": "credentialapikey",  "iam_apikey_description": "Auto-generated for key 01ba3165-85d9-410b-ad1a-1111",  "iam_apikey_name": "Service credentials-1",  "iam_role_crn": "crn:v1:bluemix:public:iam::::serviceRole:Manager",  "iam_serviceid_crn": "crn:v1:bluemix:public:iam-identity::a/11111::serviceid:ServiceId-864b549f-ff90-4b24-84f2-1111",  "instance_id": "6b00cc8c-26ec-4c42-a1e8-f4da5b9f71e7",  "kafka_admin_url": "https://admin.eventstreams.cloud.ibm.com",  "kafka_brokers_sasl": [    "broker-3.eventstreams.cloud.ibm.com:9093",    "broker-5.eventstreams.cloud.ibm.com:9093",    "broker-1.eventstreams.cloud.ibm.com:9093",    "broker-2.eventstreams.cloud.ibm.com:9093",    "broker-4.eventstreams.cloud.ibm.com:9093",    "broker-0.eventstreams.cloud.ibm.com:9093"  ],  "kafka_http_url": "https://admin.eventstreams.cloud.ibm.com",  "password": "credentialapikey",  "user": "token"}'
    

    Note the single quotes surround the above.

    1. Start up the IBM FHIR Server with the EventStreams credentials and fhir-server-config.json we just downloaded.
    docker run --rm -d -p 9443:9443 -e BOOTSTRAP_DB=true \
        -v $(pwd)/fhir-server-config-audit-environment.json:/config/config/default/fhir-server-config.json \
        -e EVENT_STREAMS_AUDIT_BINDING="${ES_CONFIG}" ibmcom/ibm-fhir-server
    

    You’ll see the container id output 60a5f1cae6d677d80772f1736db1be74836a8a4845fcccc81286b7c557bc2d86.

    1. Check that the applications are started using the container id.
    $ docker logs 60a | grep -i started
    [4/21/21, 20:55:58:449 UTC] 00000001 FrameworkMana I   CWWKE0002I: The kernel started after 1.43 seconds
    [4/21/21, 20:55:58:464 UTC] 0000002a FeatureManage I   CWWKF0007I: Feature update started.
    [4/21/21, 20:56:01:328 UTC] 00000030 AppMessageHel A   CWWKZ0001I: Application fhir-openapi started in 1.588 seconds.
    [4/21/21, 20:56:03:141 UTC] 00000031 AppMessageHel A   CWWKZ0001I: Application fhir-bulkdata-webapp started in 3.402 seconds.
    [4/21/21, 20:56:07:824 UTC] 0000002d AppMessageHel A   CWWKZ0001I: Application fhir-server-webapp started in 7.871 seconds.
    [4/21/21, 20:56:07:868 UTC] 0000002a TCPPort       I   CWWKO0219I: TCP Channel defaultHttpEndpoint-ssl has been started and is now listening for requests on host *  (IPv4) port 9443.
    [4/21/21, 20:56:07:880 UTC] 0000002a FeatureManage A   CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 10.885 seconds.
    
    1. Download the Sample Data
    curl -L https://raw.githubusercontent.com/IBM/FHIR/main/fhir-server-test/src/test/resources/testdata/everything-operation/Antonia30_Acosta403.json -o Antonia30_Acosta403.json
    
    1. Load the Sample Data bundle to the IBM FHIR Server
    curl -k --location --request POST 'https://localhost:9443/fhir-server/api/v4' \
    --header 'Content-Type: application/fhir+json' \
    --header "Authorization: Basic ${DUMMY_PASSWORD}" \
    --data-binary  "@Antonia30_Acosta403.json" -o response.json
    
    1. Scan the response.json for any status that is not "status": "201". For example, the status is in the family of User Request Error or Server Side Error.

    Note: DUMMY_PASSWORD should be set to your environment.

    1. Download Kafka from Apache Kafka

    2. Create a client-ssl.properties from your service credentials.

    cat << EOF > client-ssl.properties
    bootstrap.servers=broker-2.mybroker.com:9093,broker-5.mybroker.com:9093,broker-0.mybroker.com:9093,broker-3.mybroker.com:9093,broker-4.mybroker.com:9093,broker-1.mybroker.com:9093
    sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="token" password="MYPASSWORD";
    sasl.mechanism=PLAIN
    security.protocol=SASL_SSL
    ssl.protocol=TLSv1.2
    EOF
    
    1. Unzip the kafka archive

    2. Check the Kafka Console Consumer

    bash bin/kafka-console-consumer.sh \
        --bootstrap-server broker-4.mybroker.com:9093,broker-5.mybroker.com:9093,broker-3.mybroker.com:9093,broker-2.mybroker.com:9093,broker-1.mybroker.com:9093,broker-0.mybroker.com:9093 \
        --topic FHIR_AUDIT --max-messages 25 --property print.timestamp=true --consumer.config client-ssl.properties --from-beginning
    

    We see up to 25 messages related to the generated data.

    You see the CADF audit messages, and you can change settings in the fhir-server-config.json to use AuditEvent. You can see an example at https://github.com/IBM/FHIR/blob/main/fhir-server/liberty-config/config/default/fhir-server-config-audit-config.json