Category: IBM FHIR Server

  • Recipe: Building a custom IBM FHIR Server container with Bulk Data Parquet Support

    Note: Parquet Support is now obsolete.

    The IBM FHIR Server has early support for Bulk Data export to the Apache Parquet format using the Apache Spark libraries. New as of version 4.4.0, the export to parquet feature requires:

    • Apache Spark v3.0 and the IBM Stocator adapter (version 1.1)
    • the configuration /fhirServer/bulkdata/storageProviders/(source)/enableParquet set to true

    The Parquet Bulk Data export is activated using a custom _outputFormat in the export request.

            {
                "name": "_outputFormat",
                "valueString": "application/fhir+parquet"
            },
    

    Let me show you how to build a custom IBM FHIR Server container with parquet support Docker: ibmcom/ibm-fhir-server. It is recommended to use 4.9.0 or higher.

    Recipe

    1. Prior to 4.9.0, build the Maven Projects and the Docker Build. You should see [INFO] BUILD SUCCESS after each Maven build, and docker.io/ibmcom/ibm-fhir-server:latest when the Docker build is successful.
    mvn clean install -f fhir-examples -B -DskipTests -ntp
    mvn clean install -f fhir-parent -B -DskipTests -ntp
    docker build -t ibmcom/ibm-fhir-server:latest fhir-install
    
    1. Download the dependency files for parquet and stocator.
    export WORKSPACE=~/git/wffh/2021/fhir
    bash ${WORKSPACE}/fhir-bulkdata-webapp/src/main/sh/cache-parquet-deps.sh
    
    1. Download the fhir-server-config.json
    curl -L -o fhir-server-config.json \
        https://raw.githubusercontent.com/IBM/FHIR/main/fhir-server/liberty-config/config/default/fhir-server-config.json
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  8423  100  8423    0     0  40495      0 --:--:-- --:--:-- --:--:-- 40301
    
    1. Update the fhir-server-config.json to use a IBM COS storage provider with parquet support. You’ll need to update with your HMAC id, internal and external URLs and parquet enabled.
    "storageProviders": {
                    "default" : {
                        "type": "ibm-cos",
                        "bucketName": "fhir-performance",
                        "location": "us-east",
                        "endpointInternal": "https://s3.us-east.cloud-object-storage.appdomain.cloud",
                        "endpointExternal": "https://s3.us-east.cloud-object-storage.appdomain.cloud",
                        "auth" : {
                            "type": "hmac",
                            "accessKeyId": "key",
                            "secretAccessKey": "secret"
                        },
                        "enableParquet": true,
                        "disableOperationOutcomes": true,
                        "duplicationCheck": false, 
                        "validateResources": false, 
                        "create": false
                    }
                }
    
    1. Start the Docker container, and capture the container id. It’s going to take a few moments to start up as it lays down the test database.
    docker run -d -p 9443:9443 -e BOOTSTRAP_DB=true \
      -v $(pwd)/fhir-server-config.json:/config/config/default/fhir-server-config.json \
      -v $(pwd)/deps:/config/userlib/ \
      ibmcom/ibm-fhir-server
    3f8e90f20cd42129adc58df8a0295efc3fb2a0f4507350589f71939a072999ae
    
    1. Check the logs until you see:
    docker logs 3f8e90f20cd42129adc58df8a0295efc3fb2a0f4507350589f71939a072999ae
    ...
    [6/16/21, 15:31:34:533 UTC] 0000002a FeatureManage A   CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 17.665 seconds.
    
    1. Download the Sample Data
    curl -L https://raw.githubusercontent.com/IBM/FHIR/main/fhir-server-test/src/test/resources/testdata/everything-operation/Antonia30_Acosta403.json \
    -o Antonia30_Acosta403.json
    
    1. Load the Sample Data bundle to the IBM FHIR Server
    curl -k --location --request POST 'https://localhost:9443/fhir-server/api/v4' \
    --header 'Content-Type: application/fhir+json' \
    --user "fhiruser:${DUMMY_PASSWORD}" \
    --data-binary  "@Antonia30_Acosta403.json" -o response.json
    

    Note, DUMMY_PASSWORD should be previously set.

    1. Scan the response.json for any status that is not "status": "201". For example, the status is in the family of User Request Error or Server Side Error.
    cat response.json | jq -r '.entry[].response.status' | sort -u
    201
    
    1. Call the export to Parquet operation, and grab the content-location.
    curl --location --request GET 'https://localhost:9443/fhir-server/api/v4/$export?_outputFormat=application/fhir%2Bparquet&_type=Patient' \
    --header 'X-FHIR-TENANT-ID: default' \
    --user "fhiruser:${DUMMY_PASSWORD}" \
    --header 'Content-Type: application/json' -k -v
    
    < content-location: https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=LqzauvqtHSmkpChVHo%2B1MQ
    
    1. Check the exprot status using the previous URL, and once you see a 200 response, you can go out and use your exported Parquet data.
    curl --location --request GET 'https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=LqzauvqtHSmkpChVHo%2B1MQ' \
    --header 'X-FHIR-TENANT-ID: default' \
    --user "fhiruser:${DUMMY_PASSWORD}" \
    --header 'Content-Type: application/json' -k
    
    {
        "transactionTime": "2021-08-09T00:34:11.594Z",
        "request": "https://localhost:9443/fhir-server/api/v4/$export?_outputFormat=application/fhir%2Bparquet&_type=Patient",
        "requiresAccessToken": false,
        "output": [
            {
                "type": "Patient",
                "url": "https://s3.us-east.cloud-object-storage.appdomain.cloud/fhir-performance/AZ0gsQS05_RqZnHPhj57AfhYSIHU8VzwmnWjDCQdi2I/Patient_1.parquet?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=fc85bf9cc1ac49e99e40085f9ba00f77%2F20210809%2Fus-east%2Fs3%2Faws4_request&X-Amz-Date=20210809T003601Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=6d54f677b91d92304caf889eb0a1efbc2b3ebe3d24cefd9c17169b21816d1cdf",
                "count": 1
            }
        ]
    }
    
    1. You can access the files via COS. cos://us-east/fhir-performance/AZ0gsQS05_RqZnHPhj57AfhYSIHU8VzwmnWjDCQdi2I/Patient_1.parquet/part-00000-dba6ec99-7fdb-4674-a202-0452d4435d18-c000-attempt_202108090034065817435166928016302_0003_m_000000_3.snappy.parquet
    List of the Parquet Files
    View of the Parquet File
  • Recipe: Reindexing with fhir-bucket and the IBM FHIR Server

    The IBM FHIR Server enables user defined and implementation guide defined SearchParameter definitions and is constantly improving Specification conformance. Given these dynamic changes, the IBM FHIR Server Search parameter values may require refresh to optimize the Search and Retrieval of the data in the operational data store.

    A quick tool to aid in this is fhir-bucket

    Let me show you how to run reindexing with the IBM FHIR Server container Docker: ibmcom/ibm-fhir-server. This feature requires 4.9.0 or higher.

    Recipe

    1. Prior to 4.9.0, build the Maven Projects and the Docker Build. You should see [INFO] BUILD SUCCESS after each Maven build, and docker.io/ibmcom/ibm-fhir-server:latest when the Docker build is successful.
    mvn clean install -f fhir-examples -B -DskipTests -ntp
    mvn clean install -f fhir-parent -B -DskipTests -ntp
    docker build -t ibmcom/ibm-fhir-server:latest fhir-install
    
    1. Download the fhir-server-config.json
    curl -L -o fhir-server-config.json \
        https://raw.githubusercontent.com/IBM/FHIR/main/fhir-server/liberty-config/config/default/fhir-server-config.json
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  8423  100  8423    0     0  40495      0 --:--:-- --:--:-- --:--:-- 40301
    
    1. Download the extension-search-parameters.json
    curl -L -o extension-search-parameters.json \
        https://raw.githubusercontent.com/IBM/FHIR/main/fhir-server/liberty-config/config/default/extension-search-parameters.json
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100   154  100   154    0     0    611      0 --:--:-- --:--:-- --:--:--   611
    
    1. Start the Docker container, and capture the container id. It’s going to take a few moments to start up as it lays down the test database.
    docker run -d -p 9443:9443 -e BOOTSTRAP_DB=true \
      -v $(pwd)/fhir-server-config.json:/config/config/default/fhir-server-config.json \
      -v $(pwd)/extension-search-parameters.json:/config/config/default/extension-search-parameters.json \
      ibmcom/ibm-fhir-server
    9c44ad49784a6ca6456799ab63aea460fc564cc4a9bd5f4fef1aeae4b82225bd
    
    1. Check the logs until you see:
    docker logs 9c44ad49784a6ca6456799ab63aea460fc564cc4a9bd5f4fef1aeae4b82225bd
    ...
    [6/16/21, 15:31:34:533 UTC] 0000002a FeatureManage A   CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 17.665 seconds.
    
    1. Download the Sample Data
    curl -L https://raw.githubusercontent.com/IBM/FHIR/main/fhir-server-test/src/test/resources/testdata/everything-operation/Antonia30_Acosta403.json \
    -o Antonia30_Acosta403.json
    
    1. Load the Sample Data bundle to the IBM FHIR Server
    curl -k --location --request POST 'https://localhost:9443/fhir-server/api/v4' \
    --header 'Content-Type: application/fhir+json' \
    --user "fhiruser:${DUMMY_PASSWORD}" \
    --data-binary  "@Antonia30_Acosta403.json" -o response.json
    

    Note, DUMMY_PASSWORD should be previously set.

    1. Scan the response.json for any status that is not "status": "201". For example, the status is in the family of User Request Error or Server Side Error.
    cat response.json | jq -r '.entry[].response.status' | sort -u
    201
    
    1. Search maiden-name = Graciela518 Badillo851, and you’ll see the reindexing completes and the search succeeds.
    curl --location --request GET 'https://localhost:9443/fhir-server/api/v4/Patient?maiden-name=Graciela518 Badillo851' \
    --header 'X-FHIR-TENANT-ID: default' \
    --user "fhiruser:${DUMMY_PASSWORD}" \
    --header 'Content-Type: application/json' -k
    
    { 
        "resourceType":"OperationOutcome",
        "id":"ac-11-0-3-14c966ad-8c8b-438d-86ad-de4b2fdb27b0",
        "issue":[
            {
                "severity":"fatal",
                "code":"invalid",
                "details": {
                    "text":"Search parameter 'maiden-name' for resource type 'Patient' was not found."
                },
                "expression":["<empty>"]
            }
        ]
    }
    
    1. Download the extension-search-parameters.json with an example expression.
    curl -L -o extension-search-parameters.json \
        https://gist.githubusercontent.com/prb112/03f0d77f72685180868d2f31a1070ebe/raw/14fda1cb86dda74095faa5e32fec6f9226ffb587/extension-search-parameters.json
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100   752  100   752    0     0   2231      0 --:--:-- --:--:-- --:--:--  2231
    
    1. Post 4.9.0’s release, download the fhir-bucket-cli
    curl -L -o fhir-bucket-4.9.0-cli.jar \
        https://repo1.maven.org/maven2/com/ibm/fhir/fhir-bucket/4.9.0/fhir-bucket-4.9.0-cli.jar
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100 46.7M  100 46.7M    0     0  12.2M      0  0:00:03  0:00:03 --:--:-- 12.2M
    
    1. Pre 4.9.0’s release, copy the fhir-bucket-cli from the .m2 repository.
    cp ~/.m2/repository/com/ibm/fhir/fhir-bucket/4.9.0-SNAPSHOT/fhir-bucket-4.9.0-SNAPSHOT-cli.jar fhir-bucket-4.9.0-cli.jar
    
    1. Download the client truststore
    curl -L -o fhirClientTrustStore.p12 \
        https://github.com/IBM/FHIR/raw/main/fhir-server-test/src/test/resources/fhirClientTrustStore.p12
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100   174  100   174    0     0    661      0 --:--:-- --:--:-- --:--:--   661
    100  5946  100  5946    0     0  12651      0 --:--:-- --:--:-- --:--:-- 12651
    
    1. Download the fhir-bucket logging.properties
    curl -L -o logging.properties \
        https://raw.githubusercontent.com/IBM/FHIR/main/fhir-bucket/logging.properties
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100   688  100   688    0     0   3245      0 --:--:-- --:--:-- --:--:--  3245
    
    1. Create the bucket.properties file
    cat << EOF > bucket.properties
    fhir.server.host=localhost
    fhir.server.port=9443
    fhir.server.user=fhiruser
    fhir.server.pass=change-password
    fhir.server.endpoint=/fhir-server/api/v4/
    truststore=fhirClientTrustStore.p12
    truststore.pass=change-password
    read.timeout=125000
    connect.timeout=20000
    pool.connections.max=400
    disable.hostname.verification=true
    EOF
    
    1. Execute the fhir-bucket reindex operation with the current date-time (e.g. now is 2021-12-01T00:00:00Z)
    export JAR="fhir-bucket-4.9.0-cli.jar"
    java -Djava.util.logging.config.file=logging.properties -jar "${JAR}" \
        --fhir-properties bucket.properties \
        --tenant-name default \
        --max-concurrent-fhir-requests 100 \
        --no-scan \
        --reindex-tstamp 2021-12-01T00:00:00Z \
        --reindex-resource-count 50 \
        --reindex-concurrent-requests 20 \
        --reindex-client-side-driven
    
    1. Wait until the reindexing threads complete.
    2021-08-05 15:28:05.629 00000001    INFO x.ClientDrivenReindexOperation Starting monitor thread
    2021-08-05 15:28:07.864 0000001b    INFO x.ClientDrivenReindexOperation called $retrieve-index: 200 OK [took 0.851 s]
    2021-08-05 15:28:07.872 0000001b    INFO x.ClientDrivenReindexOperation Index IDs available for processing - filling worker pool
    2021-08-05 15:28:09.145 0000001d    INFO x.ClientDrivenReindexOperation called $reindex (indexIds=2,...): 200 OK [took 1.268 s]
    ...
    2021-08-05 15:28:27.946 0000001b    INFO x.ClientDrivenReindexOperation called $retrieve-index (afterIndexId=1802): 200 OK [took 0.053 s]
    2021-08-05 15:28:27.949 0000001b    INFO x.ClientDrivenReindexOperation No more index IDs to retrieve
    2021-08-05 15:28:27.950 0000001b    INFO x.ClientDrivenReindexOperation Nothing left to do, so tell all the worker threads to exit
    2021-08-05 15:28:27.951 0000001b    INFO x.ClientDrivenReindexOperation Waiting for 20 threads to complete before exiting
    2021-08-05 15:28:32.952 0000001b    INFO x.ClientDrivenReindexOperation Reindexing was completed
    2021-08-05 15:29:32.891 0000001a    INFO   com.ibm.fhir.bucket.app.Main Stopping all services
    
    1. Search maiden-name = Graciela518 Badillo851, and you’ll see the reindexing completes and the search succeeds.
    curl --location --request GET 'https://localhost:9443/fhir-server/api/v4/Patient?maiden-name=Graciela518 Badillo851' \
    --header 'X-FHIR-TENANT-ID: default' \
    --user "fhiruser:${DUMMY_PASSWORD}" \
    --header 'Content-Type: application/json' -k
    
        {
                "id": "17b17befcde-8fb6aca7-9008-4682-ae7f-4c3c5a91c7e2",
                "fullUrl": "https://localhost:9443/fhir-server/api/v4/Patient/17b17befcde-8fb6aca7-9008-4682-ae7f-4c3c5a91c7e2",
                "resource": {
                    "resourceType": "Patient",
                    "id": "17b17befcde-8fb6aca7-9008-4682-ae7f-4c3c5a91c7e2",
                    "meta": {
                        "versionId": "1",
                        "lastUpdated": "2021-08-05T19:16:37.89885Z"
                    },
                    "text": {
                        "status": "generated",
                        "div": "<div xmlns=\"http://www.w3.org/1999/xhtml\">Generated by <a href=\"https://github.com/synthetichealth/synthea\">Synthea</a>.Version identifier: master-branch-latest-2-g8e7e92c\n .   Person seed: 7105486291024734541  Population seed: 0</div>"
                    },
                    "extension": [
                        {
                            "url": "http://hl7.org/fhir/StructureDefinition/patient-mothersMaidenName",
                            "valueString": "Graciela518 Badillo851"
                        },
                        {
                            "url": "http://hl7.org/fhir/StructureDefinition/patient-birthPlace",
                            "valueAddress": {
                                "city": "Caguas",
                                "state": "Puerto Rico",
                                "country": "PR"
                            }
                        },
                        {
                            "url": "http://synthetichealth.github.io/synthea/disability-adjusted-life-years",
                            "valueDecimal": 1.6165934938317164
                        },
                        {
                            "url": "http://synthetichealth.github.io/synthea/quality-adjusted-life-years",
                            "valueDecimal": 47.383406506168288
                        },
                        {
                            "url": "http://hl7.org/fhir/us/core/StructureDefinition/us-core-birthsex",
                            "valueCode": "F"
                        },
                        {
                            "url": "http://hl7.org/fhir/us/core/StructureDefinition/us-core-birthsex",
                            "valueCode": "M"
                        }
                    ],
                    "identifier": [
                        {
                            "system": "https://github.com/synthetichealth/synthea",
                            "value": "bd958c64-56b6-f206-b03d-8f4b8a417215"
                        },
                        {
                            "type": {
                                "coding": [
                                    {
                                        "system": "http://terminology.hl7.org/CodeSystem/v2-0203",
                                        "code": "MR",
                                        "display": "Medical Record Number"
                                    }
                                ],
                                "text": "Medical Record Number"
                            },
                            "system": "http://hospital.smarthealthit.org",
                            "value": "bd958c64-56b6-f206-b03d-8f4b8a417215"
                        },
                        {
                            "type": {
                                "coding": [
                                    {
                                        "system": "http://terminology.hl7.org/CodeSystem/v2-0203",
                                        "code": "SS",
                                        "display": "Social Security Number"
                                    }
                                ],
                                "text": "Social Security Number"
                            },
                            "system": "http://hl7.org/fhir/sid/us-ssn",
                            "value": "999-93-6090"
                        },
                }
            ]
    

    References

  • Recipe: Running the IBM FHIR Server behind a Reverse Proxy

    A common deployment pattern for the IBM FHIR Server is to run the Application Server behind a reverse proxy (e.g. OpenShift Route, Kubernetes Ingress, NGINX or API Gateway). By default, the IBM FHIR Server runs under the https://localhost:9443/fhir-server/api/v4 context root and URI. With a modest configuration change, one can change to a context root (baseUrl) or use the X-FHIR-FORWARDED-URL to forward the incoming url to the IBM FHIR Server (expected to be from a trusted reverse proxy).

    In #1965, the fhirServer/core/externalBaseUrl is a tenant aware configuration that sets the context root and base URL. Note, the fhirServer/core/externalBaseUrl overrides the incomingUrl from X-FHIR-FORWARDED-URL.

    This document outlines how to set the externalBaseUrl for the IBM FHIR Server.

    Let me show you how to add a set the externalBaseUrl to an IBM FHIR Server container Docker: ibmcom/ibm-fhir-server. This feature requires 4.9.0 or higher.

    Recipe

    1. Prior to 4.9.0, build the Maven Projects and the Docker Build. You should see [INFO] BUILD SUCCESS after each Maven build, and docker.io/ibmcom/ibm-fhir-server:latest when the Docker build is successful.
    mvn clean install -f fhir-examples -B -DskipTests -ntp
    mvn clean install -f fhir-parent -B -DskipTests -ntp
    docker build -t ibmcom/ibm-fhir-server:latest fhir-install</code></pre>
    
    1. Download the fhir-server-config.json
    curl -L -o fhir-server-config.json \
        https://raw.githubusercontent.com/IBM/FHIR/main/fhir-server/liberty-config/config/default/fhir-server-config.json
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  8423  100  8423    0     0  35095      0 --:--:-- --:--:-- --:--:-- 34950
    
    1. Update the fhir-server-config.json at path fhirServer/core/externalBaseUrl to https://chocolate.fudge.
    "externalBaseUrl": "https://chocolate.fudge"
    
    1. Start the Docker container, and capture the container id. It’s going to take a few moments to start up as it lays down the test database.
    docker run -d -p 9443:9443 -e BOOTSTRAP_DB=true \
      -v $(pwd)/fhir-server-config.json:/config/config/default/fhir-server-config.json \
      ibmcom/ibm-fhir-server
    a096978867195ff6e610c36cdba77ff423c31c0ad488a7390f42cef6e89e7fd0
    
    1. Check the logs until you see:
    docker logs a096978867195ff6e610c36cdba77ff423c31c0ad488a7390f42cef6e89e7fd0
    ...
    [6/16/21, 15:31:34:533 UTC] 0000002a FeatureManage A   CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 17.665 seconds.
    
    1. Download the Sample Data

    curl -L https://raw.githubusercontent.com/IBM/FHIR/main/fhir-server-test/src/test/resources/testdata/everything-operation/Antonia30_Acosta403.json -o Antonia30_Acosta403.json

    1. Load the Sample Data bundle to the IBM FHIR Server
    curl -k --location --request POST 'https://localhost:9443/fhir-server/api/v4' \
    --header 'Content-Type: application/fhir+json' \
    --user "fhiruser:${DUMMY_PASSWORD}" \
    --data-binary  "@Antonia30_Acosta403.json" -o response.json
    
    1. Scan the response.json for any status that is not "status": "201". For example, the status is in the family of User Request Error or Server Side Error.

    2. Check the patient and you’ll see the self and next relation links include https://chocolate.fudge

    curl -k --location --request GET 'https://localhost:9443/fhir-server/api/v4/Patient' \
    --header 'Content-Type: application/fhir+json' \
    --user "fhiruser:${DUMMY_PASSWORD}" \
    
    {
        "resourceType": "Bundle",
        "id": "37c5abc7-d3e7-4506-b596-9725c59f9b6b",
        "type": "searchset",
        "total": 23,
        "link": [
            {
                "relation": "self",
                "url": "https://chocolate.fudge/Patient?_count=10&_page=1"
            },
            {
                "relation": "next",
                "url": "https://chocolate.fudge/Patient?_count=10&_page=2"
            }
        ],
        "entry": [
            {
                "id": "17b123f9a79-bd2011c1-6606-4617-90ed-3187790955b8",
                "fullUrl": "https://chocolate.fudge/Patient/17b123f9a79-bd2011c1-6606-4617-90ed-3187790955b8",
                "resource": {
                    "resourceType": "Patient",
                    "id": "17b123f9a79-bd2011c1-6606-4617-90ed-3187790955b8",
                    "meta": {
                        "versionId": "1",
                        "lastUpdated": "2021-08-04T17:39:23.385314Z",
                        "tag": [
                            {
                                "system": "http://terminology.hl7.org/CodeSystem/v3-ActReason",
                                "code": "HTEST",
                                "display": "test health data"
                            }
                        ]
                    }
                }
            }
        ]
    }
    

    References

  • Postgres and Vacuum with the IBM FHIR Server: Debugging Details

    The IBM FHIR Server stores resources in the PostgreSQL database and uses a relational model to store historical FHIR Resource and enable search on the latest FHIR resources. The resource data is spread in a relational model that is occasionally tweaked in order to improve search or optimize the retrieval using the relational model.

    In the IBM FHIR Server Performance Guide, the guide outlines some important alterations to the tables that facilitate an optimized Postgres instance. The guide suggests altering, per your providers recommendation, autovacuum_vacuum_cost_limit, autovacuum_vacuum_scale_factor and autovacuum_vacuum_threshold in order to optimize the VACUUM process. With the IBM FHIR Server fhir-persistence-schema-cli, autovacuum_vacuum_scale_factor is not automatically configured, and not recommended on Databases for Postgres on IBM Cloud.

    As Postgres uses "multi-version concurrency control (MVCC) to ensure that data remains consistent and accessible in high-concurrency environments", each transaction runs on a snapshot, and needs to be reconciled so dead_rows are removed – vacuumed. The VACUUM process manages dead rows and disk usage (reuse). The VACUUM process (autovacuum) frequently runs – gathering statistics and reconciling the maintenance of the table statitstics and data.

    To check for tables that need vacuuming:

     SELECT relname AS "table_name",
            n_tup_ins AS "inserts",
            n_tup_upd AS "updates",
            n_tup_del AS "deletes",
            n_live_tup AS "live_tuples",
            n_dead_tup AS "dead_tuples"
       FROM pg_stat_user_tables
      WHERE schemaname = 'fhirdata'
        AND (relname = 'logical_resources' OR relname LIKE '%_values')
        AND n_dead_tup > 0;
    

    Then a database administrator runs – VACUUM FULL FHIRDATA.PROCEDURE_RESOURCE_TOKEN_REFS; to execute a vacuum, which runs in the background.

    While the VACUUM process is running, the pg_stat_progress_vacuum view can be queried to see worker process.

    SELECT * 
    FROM pg_stat_progress_vacuum;
    

    If you need to update a specific tables settings, you can run with --vacuum-table-name.

    java -jar ./fhir-persistence-schema-${VERSION}-cli.jar \
    --db-type postgresql --prop db.host=localhost --prop db.port=5432 \
    --prop db.database=fhirdb --schema-name fhirdata \
    --prop user=fhiradmin --prop password=passw0rd \
    --update-vacuum --vacuum-cost-limit 2000 --vacuum-threshold 1000 \
    --vacuum-scale-factor 0.01 --vacuum-table-name LOGICAL_RESOURCES
    

    To update all tables in a schema, you can run without the table parameter. If you omit any value, the defaults are picked as described in the Performance guide.

    If you hit a lock (ShareUpdateExclusiveLock), the VACUUM worker process is currently churning on the table, and the ALTER statement is waiting.

    • wait_type = Lock relation Waiting to acquire a lock on a relation.
    • wait_lock_type – ShareUpdateExclusiveLock Acquired by VACUUM and conflicts with ALTER

    CHeck for the Blocking PID, and grab the blocking_pid.

     -- list bad connections
       SELECT blocked_locks.pid     AS blocked_pid,
             blocked_activity.usename  AS blocked_user,
             blocking_locks.pid     AS blocking_pid,
             blocking_activity.usename AS blocking_user,
             blocked_activity.query    AS blocked_statement,
             blocking_activity.query   AS current_statement_in_blocking_process,
             blocked_activity.application_name AS blocked_application,
             blocking_activity.application_name AS blocking_application
       FROM  pg_catalog.pg_locks         blocked_locks
        JOIN pg_catalog.pg_stat_activity blocked_activity  ON blocked_activity.pid = blocked_locks.pid
        JOIN pg_catalog.pg_locks         blocking_locks 
            ON blocking_locks.locktype = blocked_locks.locktype
            AND blocking_locks.DATABASE IS NOT DISTINCT FROM blocked_locks.DATABASE
            AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
            AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page
            AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple
            AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid
            AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid
            AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid
            AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid
            AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid
            AND blocking_locks.pid != blocked_locks.pid
        JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid
       WHERE NOT blocked_locks.GRANTED and blocked_activity.usename = 'fhirserver'
    

    Try canceling the PID, SELECT pg_cancel_backend(205384);

    Else, cancel the current Transaction the blocked pid:

    -- cancel the blocking trannsaction/pid (hard stop)
    SELECT pg_cancel_backend(blocked_locks.pid)     AS blocked_pid,
             blocked_activity.usename  AS blocked_user,
             blocking_locks.pid     AS blocking_pid,
             blocking_activity.usename AS blocking_user,
             blocked_activity.query    AS blocked_statement,
             blocking_activity.query   AS current_statement_in_blocking_process,
             blocked_activity.application_name AS blocked_application,
             blocking_activity.application_name AS blocking_application
       FROM  pg_catalog.pg_locks         blocked_locks
        JOIN pg_catalog.pg_stat_activity blocked_activity  ON blocked_activity.pid = blocked_locks.pid
        JOIN pg_catalog.pg_locks         blocking_locks 
            ON blocking_locks.locktype = blocked_locks.locktype
            AND blocking_locks.DATABASE IS NOT DISTINCT FROM blocked_locks.DATABASE
            AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
            AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page
            AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple
            AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid
            AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid
            AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid
            AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid
            AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid
            AND blocking_locks.pid != blocked_locks.pid
        JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid
       WHERE NOT blocked_locks.GRANTED and blocked_activity.usename = 'fhirserver'
    

    Now, wait until the VACUUM finishes, and then execute a new ALTER.

    You should be all set at this point.

    Check wait_type

    -- Check Wait Type
    SELECT 
        waiting.locktype           AS waiting_locktype,
        waiting.relation::regclass AS waiting_table,
        waiting_stm.query          AS waiting_query,
        waiting.mode               AS waiting_mode,
        waiting.pid                AS waiting_pid,
        other.locktype             AS other_locktype,
        other.relation::regclass   AS other_table,
        other_stm.query            AS other_query,
        other.mode                 AS other_mode,
        other.pid                  AS other_pid,
        other.granted              AS other_granted
    FROM
        pg_catalog.pg_locks AS waiting
    JOIN
        pg_catalog.pg_stat_activity AS waiting_stm
        ON (
            waiting_stm.pid = waiting.pid
        )
    JOIN
        pg_catalog.pg_locks AS other
        ON (
            (
                waiting."database" = other."database"
            AND waiting.relation  = other.relation
            )
            OR waiting.transactionid = other.transactionid
        )
    JOIN
        pg_catalog.pg_stat_activity AS other_stm
        ON (
            other_stm.pid = other.pid
        )
    WHERE
        NOT waiting.granted
    AND
        waiting.pid <> other.pid   
    
    SELECT
      schemaname, relname,
      last_vacuum, last_autovacuum,
      vacuum_count, autovacuum_count, *
    FROM pg_stat_user_tables
    WHERE schemaname = 'fhirdata' AND relname = 'observation_date_values';
    

    Check with locks

    SELECT now()::time, query, backend_start, xact_start, query_start,
             state_change, state,
             now()::time - state_change::time AS locked_since,
             pid, wait_event_type, wait_event
      FROM pg_stat_activity
      WHERE wait_event_type IS NOT NULL and wait_event_type = 'Lock'
    ORDER BY locked_since DESC;
    

    Check a PID

    SELECT a.usename, a.application_name, a.datname, a.query,
             l.granted, l.mode, transactionid
        FROM pg_locks l
        JOIN pg_stat_activity a ON a.pid = l.pid
        WHERE granted = false AND a.pid = 327589;
    

    Check a tansaction

    SELECT a.usename, a.application_name, a.datname, a.query,
            l.granted, l.mode, transactionid,
            now()::time - a.state_change::time AS acquired_since,
            a.pid
       FROM pg_locks l
       JOIN pg_stat_activity a ON a.pid = l.pid
       WHERE granted = true AND transactionid = 3031;
    

    Reference

  • Recipe: Testing Restricted Policies with OpenShift and the IBM FHIR Server Schema Tool

    Docker: ibmcom/ibm-fhir-schematool supports onboarding and offboarding of a schema in support of the IBM FHIR Server. I am working on a project that runs on OpenShift using CodeReadyContainers, and I needed to test the image with a restricted policy.

    Note, these are roughly my notes from testing, and converted to a post.

    Recipe

    1. Start up Code Ready Containers or create your OpenShift environment.

    2. Connect to the Docker registry in CRC

    docker login -u kubeadmin -p $(oc whoami -t) default-route-openshift-image-registry.apps-crc.testing
    
    1. Build your Docker image with a new tag
    docker build . -t default/ibm-fhir-schematool:latest
    docker push default/ibm-fhir-schematool:latest
    
    1. Login to OpenShift
    oc login -u $(oc whoami) -p $(oc whoami -t)
    
    1. Create a new persistence.json using one of the examples as a template
    oc create secret generic persistence-json --from-file=persistence-onboard-example.json -n=default
    
    1. Add the restricted policy to the developer user.
    oc adm policy add-scc-to-user restricted developer
    
    1. Add the developer user to the default namespace.
    oc adm policy add-role-to-user edit developer -n=default
    
    1. Make the locak registry lookup use relative names
    oc set image-lookup  --all
    
    1. Create a new pod.yaml
    apiVersion: v1
    kind: Pod
    metadata:
      name: ibm-fhir-server-schematool
    spec:
      containers:
        - name: test-container
          image: default/ibm-fhir-schematool
          env:
            - name: ENV_TOOL_INPUT
              valueFrom:
                secretKeyRef:
                  name: persistence-json
                  key: persistence-onboard-example.json
    
    1. Create the pod.
    oc apply -f pod.yaml --as=developer  -n=default
    
    1. Check the logs, and you should see a successful run.
    oc logs pod/ibm-fhir-server-schematool
    

    References

  • Recipe: IBM FHIR Server – Adding a Custom Extended Operation to the IBM FHIR Server

    The IBM FHIR Server has support for extended operations beyond the standard C-R-U-D. The Extended Operations are supported at the System, Resource, and Instance levels. Operations are packaged as JAR files, and the IBM FHIR Server loads the specific Operation using the Java ServiceLoader framework at startup. More implementation specific details are at FHIROperationFramework.

    The IBM FHIR Server repository builds a number of operations, release to Maven Repository, and bundles a subset automatically with the Docker container. Some of the bundles operations include Healthcheck, Terminology, BulkData Import, BulkData Export, Reindex, Everything, Erase, Convert, Document and Validate. The IBM FHIR Server maintains the Capability Statement which advertises the System, Type and Instance operations on the metadata endpoint.

    The IBM FHIR Server can take any configured custom extended operation, such as HelloOperation which is part of the fhir-operation-test.

    This document outlines how to add a custom extended operation to the IBM FHIR Server.

    Let me show you how to add a custom extended operation to an IBM FHIR Server container Docker: ibmcom/ibm-fhir-server.

    Recipe

    1. Download the fhir-server-config.json
    curl -L -o fhir-server-config.json \
        https://raw.githubusercontent.com/IBM/FHIR/main/fhir-server/liberty-config/config/default/fhir-server-config.json
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  8423  100  8423    0     0  35095      0 --:--:-- --:--:-- --:--:-- 34950
    
    1. Download the fhir-operation-test-*-tests.jar. Note, you should pick the most recent release.
    curl -L -o fhir-operation-test-4.8.3.jar \
        https://repo1.maven.org/maven2/com/ibm/fhir/fhir-operation-test/4.8.3/fhir-operation-test-4.8.3-tests.jar
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100 12393  100 12393    0     0  58734      0 --:--:-- --:--:-- --:--:-- 58734
    
    1. Create a directory called userlib, don’t put the jar in the userlib yet.
    mkdir -p userlib
    
    1. Start the Docker container, and capture the container id. It’s going to take a few moments to start up as it lays down the test database.
    docker run -d -p 9443:9443 -e BOOTSTRAP_DB=true \
      -v $(pwd)/fhir-server-config.json:/config/config/default/fhir-server-config.json \
      -v $(pwd)/userlib/:/config/userlib/ ibmcom/ibm-fhir-server
    a096978867195ff6e610c36cdba77ff423c31c0ad488a7390f42cef6e89e7fd0
    
    1. Check the logs until you see:
    docker logs a096978867195ff6e610c36cdba77ff423c31c0ad488a7390f42cef6e89e7fd0
    ...
    [6/16/21, 15:31:34:533 UTC] 0000002a FeatureManage A   CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 17.665 seconds.
    
    1. Check the Type and Instance operations, you can check using this curl/jq command.
    curl -k --location --request GET 'https://localhost:9443/fhir-server/api/v4/metadata' \
        | jq -r '.rest[].resource[].operation[].name' | sort -u
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100 1096k    0 1096k    0     0   103k      0 --:--:--  0:00:10 --:--:--  259k
    apply
    closure
    convert
    document
    erase
    everything
    expand
    export
    lookup
    reindex
    subsumes
    translate
    validate
    validate-code
    

    Note, that the hello operation is not displayed. The CapabilityStatement is best looked at selectively, since it’s such a big document, thus the use of jq.

    1. To check the System Level operations, you can check using this curl/jq command.
    curl -k --location --request GET 'https://localhost:9443/fhir-server/api/v4/metadata' \
        | jq -r '.rest[].operation[].name'
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100 1096k    0 1096k    0     0  2592k      0 --:--:-- --:--:-- --:--:-- 2592k
    bulkdata-status
    closure
    convert
    export
    healthcheck
    import
    reindex
    

    Note, that the hello operation is not displayed.

    1. Copy over the fhir-operation-test-*-tests.jar to userlib/.

    2. Restart the docker container

    docker restart a096978867195ff6e610c36cdba77ff423c31c0ad488a7390f42cef6e89e7fd0
    
    1. Check the logs until you see:
    docker logs a096978867195ff6e610c36cdba77ff423c31c0ad488a7390f42cef6e89e7fd0
    ...
    [6/16/21, 15:31:34:533 UTC] 0000002a FeatureManage A   CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 17.665 seconds.
    
    1. To check the System Level operations, you can check using this curl/jq command.
    curl -k --location --request GET 'https://localhost:9443/fhir-server/api/v4/metadata' \
        | jq -r '.rest[].operation[].name'
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100 1096k    0 1096k    0     0  2562k      0 --:--:-- --:--:-- --:--:-- 2556k
    bulkdata-status
    closure
    convert
    export
    healthcheck
    hello
    import
    reindex
    

    You should see the hello operation listed.

    1. Now you can execute the hello operation.
    curl -k --location --request GET 'https://localhost:9443/fhir-server/api/v4/$hello' \
        -v -u fhiruser:change-password
    ...
    * Connection state changed (MAX_CONCURRENT_STREAMS == 200)!
    < HTTP/2 200 
    < date: Fri, 09 Jul 2021 23:48:33 GMT
    < content-length: 0
    < content-language: en-US
    < 
    * Connection #0 to host localhost left intact
    * Closing connection 0
    

    You’ve now seen how to add a jar file to the userlib in a running container that adds the custom operation HelloOperation.

    References

  • Recipe: IBM FHIR Server – Add Tenant Key, Revoke Tenant Key and Revoke All Tenant Keys

    The IBM FHIR Server supports many different persistence stores – Apache Derby, Postgres, and IBM Db2. The IBM Db2 offering includes a multi-tenancy feature with row-level permissions. Each tenant is assigned a unique internal id for the tenantName and tenantKey. The tenantKey on each JDBC connection is verified using a one-way hash, and supports allocating a tenant, adding a key to an existing tenant, revoking a key and revoking all tenant keys.

    Recipe

    1. Once you have created the schema on your database via the --update-schema, you can allocate a tenant.
    java -jar fhir-persistence-schema-*-cli.jar \
        --prop db.host=db \
        --prop db.port=50000 \
        --prop db.database=fhirdb \
        --prop user=db2inst1 \
        --prop password=change-password \
        --db-type db2 \
        --allocate-tenant acme32
    
    2021-06-22 11:19:13.226 00000001    INFO .common.JdbcConnectionProvider Opening connection to database: jdbc:db2://db:50000/fhirdb
    2021-06-22 11:19:14.346 00000001 WARNING ls.pool.PoolConnectionProvider Get connection took 1.120 seconds
    ...
    2021-06-22 11:22:59.846 00000001    INFO   com.ibm.fhir.schema.app.Main tenantId [29] is being pre-populated with lookup table data.
    2021-06-22 11:22:59.930 00000001    INFO   com.ibm.fhir.schema.app.Main Finished prepopulating the resource type and search parameter code/name tables tables
    2021-06-22 11:22:59.948 00000001    INFO   com.ibm.fhir.schema.app.Main Allocated tenant: acme32 [key=UsBglbyMJpSi/RjXwrkp0Bj2bAljUI+MixfAikdrcN0=] with Id = 29
    2021-06-22 11:22:59.949 00000001    INFO   com.ibm.fhir.schema.app.Main The tenantKey JSON follows:
    {"tenantKey": "UsBglbyMJpSi/RjXwrkp0Bj2bAljUI+MixfAikdrcN0="}
    2021-06-22 11:22:59.949 00000001    INFO   com.ibm.fhir.schema.app.Main Processing took: 226.765 s
    2021-06-22 11:22:59.950 00000001    INFO   com.ibm.fhir.schema.app.Main SCHEMA CHANGE: OK
    
    1. You can add a tenant key to an existing tenant.
    java -jar fhir-persistence-schema-*-cli.jar \
        --prop db.host=db \
        --prop db.port=50000 \
        --prop db.database=fhirdb \
        --prop user=db2inst1 \
        --prop password=change-password \
        --db-type db2 \
        --add-tenant-key acme32
    
    2021-06-22 11:25:53.254 00000001    INFO .common.JdbcConnectionProvider Opening connection to database: jdbc:db2://db:50000/fhirdb
    2021-06-22 11:25:54.197 00000001    INFO   com.ibm.fhir.schema.app.Main New tenant key: acme32 [key=UVFuDqD/V3v8d9S/XRjRQNu9eFTniksvxgIBbI6mEkg=]
    2021-06-22 11:25:54.201 00000001    INFO   com.ibm.fhir.schema.app.Main Processing took:   0.975 s
    2021-06-22 11:25:54.202 00000001    INFO   com.ibm.fhir.schema.app.Main SCHEMA CHANGE: OK
    
    1. List tenant to see allocated
    java -jar fhir-persistence-schema-*-cli.jar \
        --prop db.host=db \
        --prop db.port=50000 \
        --prop db.database=fhirdb \
        --prop user=db2inst1 \
        --prop password=change-password \
        --db-type db2 \
        --list-tenants
    
    2021-06-22 11:28:43.566 00000001    INFO .common.JdbcConnectionProvider Opening connection to database: jdbc:db2://db:50000/fhirdb
     TenantId     Status       TenantName Schema
           29  ALLOCATED           acme32 FHIRDATA
    2021-06-22 11:28:44.395 00000001    INFO   com.ibm.fhir.schema.app.Main Processing took:   0.854 s
    2021-06-22 11:28:44.396 00000001    INFO   com.ibm.fhir.schema.app.Main SCHEMA CHANGE: OK
    
    1. Revoking tenant key
    java -jar fhir-persistence-schema-*-cli.jar \
        --prop db.host=db \
        --prop db.port=50000 \
        --prop db.database=fhirdb \
        --prop user=db2inst1 \
        --prop password=change-password \
        --db-type db2 \
        --revoke-tenant-key acme32 \
        --tenant-key UVFuDqD/V3v8d9S/XRjRQNu9eFTniksvxgIBbI6mEkg=
    
    2021-06-22 11:27:17.280 00000001    INFO .common.JdbcConnectionProvider Opening connection to database: jdbc:db2://db:50000/fhirdb
    2021-06-22 11:27:18.112 00000001    INFO   com.ibm.fhir.schema.app.Main Tenant Key revoked for 'acme32' total removed=[1]
    2021-06-22 11:27:18.119 00000001    INFO   com.ibm.fhir.schema.app.Main Processing took:   0.867 s
    2021-06-22 11:27:18.120 00000001    INFO   com.ibm.fhir.schema.app.Main SCHEMA CHANGE: OK
    

    You’ll see one was removed.

    1. We’ll add a new key.
    java -jar fhir-persistence-schema-*-cli.jar \
        --prop db.host=db \
        --prop db.port=50000 \
        --prop db.database=fhirdb \
        --prop user=db2inst1 \
        --prop password=change-password \
        --db-type db2 \
        --add-tenant-key acme32
    
    2021-06-22 11:25:53.254 00000001    INFO .common.JdbcConnectionProvider Opening connection to database: jdbc:db2://db:50000/fhirdb
    2021-06-22 11:25:54.197 00000001    INFO   com.ibm.fhir.schema.app.Main New tenant key: acme32 [key=UVFuDqD/V3v8d9S/XRjRQNu9eFTniksvxgIBbI6mEkg=]
    2021-06-22 11:25:54.201 00000001    INFO   com.ibm.fhir.schema.app.Main Processing took:   0.975 s
    2021-06-22 11:25:54.202 00000001    INFO   com.ibm.fhir.schema.app.Main SCHEMA CHANGE: OK
    
    1. Revoking tenant key
    java -jar fhir-persistence-schema-*-cli.jar \
        --prop db.host=db \
        --prop db.port=50000 \
        --prop db.database=fhirdb \
        --prop user=db2inst1 \
        --prop password=change-password \
        --db-type db2 \
        --revoke-all-tenant-keys acme32
    
    2021-06-22 11:32:34.061 00000001    INFO .common.JdbcConnectionProvider Opening connection to database: jdbc:db2://db:50000/fhirdb
    2021-06-22 11:32:35.112 00000001    INFO   com.ibm.fhir.schema.app.Main Tenant Key revoked for 'acme32' total removed=[2]
    2021-06-22 11:32:35.144 00000001    INFO   com.ibm.fhir.schema.app.Main Processing took:   1.116 s
    2021-06-22 11:32:35.146 00000001    INFO   com.ibm.fhir.schema.app.Main SCHEMA CHANGE: OK
    

    You’ll see that two are removed.

    You now know the lifecycle for the IBM FHIR Server tenantKey – allocate, add and revoke.

    Reference

  • Recipe: IBM FHIR Server – Using Hard Delete with the $erase Operation

    The IBM FHIR Server supports hard delete using the custom FHIR Operation Framework. The operation is called $erase; you can always read the design document.

    The operation supports:

    By default, the $erase operation is not enabled, and must be set to true, and the roles allowed to execute the operation, can be tweaked to FHIROperationAdmin or FHIRUsers.

    PropertyTypeDescription
    fhirServer/operations/erase/enabledbooleanEnables the $erase operation
    fhirServer/operations/erase/allowedRoleslistThe list of allowed roles, allowed entries are: FHIRUsers every authenticated user, FHIROperationAdmin which is authenticated FHIRAdmin users
    Properties

    The fhir-server-config.json should be amended with a snippet like the following:

    "operations": {
                "erase": {
                    "enabled": true,
                    "allowedRoles": [
                        "FHIROperationAdmin",
                        "FHIRUsers"
                    ]
                }
            }

    As the $erase operation uses stored procedures for Db2 and functions for Postgres, the data schema must be updated to the 4.8.3 level. It’s also worth noting that Derby is also supported as it uses the same DAO as the Db2 and Postgres to run.

    A tip, you should search for the references to the Patient’s ResourceId, and delete the referenced resources in a Transaction Bundle. Net – Search, Assemble Bundle, Execute Batch.

    Let me show you how to run with it, and operate the $erase operation with a container (Docker: ibmcom/ibm-fhir-server).

    Recipe

    1. Download the fhir-server-config.json

    2. Start the Docker container, and capture the container id. It’s going to take a few moments to start up as it lays down the test database.

    docker run -d -p 9443:9443 -e BOOTSTRAP_DB=true -v $(pwd)/fhir-server-config.json:/config/config/default/fhir-server-config.json ibmcom/ibm-fhir-server
    cde61943964464b528eb77d132fd2a4952e0eaf43588da1c9b6bb2fa584f0608

    3. Check the logs until you see:

    docker logs cde61943964464b528eb77d132fd2a4952e0eaf43588da1c9b6bb2fa584f0608
    ...
    [6/16/21, 15:31:34:533 UTC] 0000002a FeatureManage A   CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 17.665 seconds.

    4. Download the Postman Collection for Version Specific Erase and All Versions Erase.

    5. Import the Postman Collections and Run

    You’ve seen how to use the $erase operation, and a bit more details on configuring and using it.

    Postman Collections

  • Recipe: IBM FHIR Server – Using Bulk Data with the Azure Blob Service

    The IBM FHIR Server has support for exporting and importing Bulk Data using extended operations for Bulk Data $import, $export and $bulkdata-status, which are implemented as Java Maven projects. The IBM FHIR Server uses JSR252 JavaBatch jobs running in the Open Liberty Java Batch Framework.

    The IBM FHIR Server supports storage providers S3 (aws-s3,ibm-cos), File System (file), and now Azure (azure-blob).

    I recently implemented support for the Azure Blob Service in IBM/FHIR Pull Request #2413. The code that supports the Azure client uses the OK Http library and not Netty to communicate with the backend. The storage provider is implemented as AzureProvider, and uses the Java – AppendBlobClient to facilitate export. To facilitate import, the code uses the BlobClient to read from a location in 10Kb blocks, and reassemble when a full block is reached. Importantly, this is done in a way to ensure full resources are returned up to a 2Gb limit.

    Overview Video

    If you want to use the IBM FHIR Server’s bulk data feature with the IBM FHIR Server, use the following recipe:

    Recipe – Azure Blob and Bulk Data

    1. Login to the Azure Portal

    2. Click Home

    3. Navigate the Azure services, and click on Storage Accounts.

    4. Click Create

    5. Enter the appropriate subscription and resource group

    6. Give it a name such as fhirintegrationit

    7. Click Review+Create

    8. Click Create

    You’ll see Deployment in Progress for a period of time.

    Once it is done, you see Your deployment is complete, proceed to the next steps.

    9. Click Go to Resource

    10. Click on Access Key

    11. Click Show keys and copy the key1’s Connection string. It’ll look like DefaultEndpointsProtocol=https;AccountName=fhirintegrationit;AccountKey=HIDDEN==;EndpointSuffix=core.windows.net

    12. Click on Data > Containers

    13. Click +Container

    14. Fill in the container details and call it bulkdata

    14. Click on the bulkdata container

    15. Upload blob Data – AllergyIntolerance.ndjson

    You should see the file uploaded and available.

    You are now ready to use this with the IBM FHIR Server from the main branch. If you need to clone the repo go to IBM/FHIR.

    16. Build the Maven Projects and the Docker Build. You should see “[INFO] BUILD SUCCESS” after each Maven build, and “=> naming to docker.io/ibmcom/ibm-fhir-server:latest” when the Docker build is successful.

    mvn clean install -f fhir-examples -B -DskipTests -ntp
    mvn clean install -f fhir-parent -B -DskipTests -ntp
    docker build -t ibmcom/ibm-fhir-server:latest fhir-install

    17. Download the fhir-server-config.json

    18. Open the fhir-server-config.json and update fhirServer/bulkdata/storageProviders/default entry to be the connection is updated to your Connection string from above.

    17. Start the Docker container, and capture the container id. It’s going to take a few moments to start up as it lays down the test database.

    docker run -d -p 9443:9443 -e BOOTSTRAP_DB=true -v $(pwd)/fhir-server-config.json:/config/config/default/fhir-server-config.json ibmcom/ibm-fhir-server
    cde61943964464b528eb77d132fd2a4952e0eaf43588da1c9b6bb2fa584f0608

    18. Check the logs until you see:

    docker logs cde61943964464b528eb77d132fd2a4952e0eaf43588da1c9b6bb2fa584f0608
    ...
    [6/16/21, 15:31:34:533 UTC] 0000002a FeatureManage A   CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 17.665 seconds.

    19. Create an $import request and capture the content-location header.

    20. Check and poll until you get response code 200 (you should only get 202 in the interim) until you see (note 99 failed validation which is expected):

    curl -v -k --user 'fhiruser:change-password' 'https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=UKt4ESCnqOvAfxYWhdsfUg'
    
    < HTTP/2 200
    < content-type: application/json
    < date: Wed, 16 Jun 2021 16:01:54 GMT
    < content-language: en-US
    < content-length: 501
    <
    {
        "transactionTime": "2021-06-16T15:59:53.086Z",
        "request": "https://localhost:9443/fhir-server/api/v4/$import",
        "requiresAccessToken": false,
        "output": [
            {
                "type": "OperationOutcome",
                "url": "AllergyIntolerance.ndjson_oo_success.ndjson",
                "count": 1600
            }
        ],
        "error": [
            {
                "type": "OperationOutcome",
                "url": "AllergyIntolerance.ndjson_oo_errors.ndjson",
                "count": 99
            }
        ]
    * Connection #0 to host localhost left intact
    }

    21. We can check to see if there is data on the server, and there are 1600 resources.

    curl -v -k --user 'fhiruser:change-password' 'https://localhost:9443/fhir-server/api/v4/AllergyIntolerance?_format=json'
    
    {"resourceType":"Bundle","id":"d485339f-49e5-4b11-a1be-4ae06edd0b16","type":"searchset","total":1600,"link"...}

    22. Now we run $export and capture the content-location:

    curl --location --request GET -k -v 'https://localhost:9443/fhir-server/api/v4/$export?_outputFormat=application/fhir+ndjson&_type=AllergyIntolerance' \
    --header 'X-FHIR-TENANT-ID: default' \
    --header 'Content-Type: application/fhir+json' \
    --header 'X-FHIR-BULKDATA-PROVIDER: default' \
    --header 'X-FHIR-BULKDATA-PROVIDER-OUTCOME: default' \
    --header 'Authorization: Basic ZmhpcnVzZXI6Y2hhbmdlLXBhc3N3b3Jk'
    
    < HTTP/2 202
    < content-location: https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=LqzauvqtHSmkpChVHo%2B1MQ
    < date: Wed, 16 Jun 2021 16:39:19 GMT
    < content-length: 0
    < content-language: en-US
    <
    * Connection #0 to host localhost left intact
    * Closing connection 0

    23. Poll and check the response 202 or 200 and the contents of the response. (empty until done)

    curl -k --user 'fhiruser:change-password' 'https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=LqzauvqtHSmkpChVHo%2B1MQ'
    {
        "transactionTime": "2021-06-16T16:39:26.551Z",
        "request": "https://localhost:9443/fhir-server/api/v4/$export?_outputFormat=application/fhir+ndjson&_type=AllergyIntolerance",
        "requiresAccessToken": false,
        "output": [
            {
                "type": "AllergyIntolerance",
                "url": "c7GRY0uFsu-hO_QlquVZJr_7hTg5c_m6SHlZ1z3Z7J4/AllergyIntolerance_1.ndjson",
                "count": 1600
            }
        ]
    }

    24. Check Azure Blob Service and you’ll see the folder and file on Azure Blob Service

    25. Download and check there are 1600 lines.

    You now know how to use Azure and Bulk Data.

  • IBM FHIR Server – Debug Drop Tablespace

    I’ve been debugging a drop tablespace issue on Db2 – IBM/FHIR: 2354. The core issue was a timing problem with lots of partitions dettaching. This adds a delay to the dropTablespace so the async operation can complete, and cleanly exit with a specific error code so downstream consumers can work around the issue.

    When debugging why a partition hasn’t dettached, I found that it’s worth checking the details when a drop tablespace fails:

    [db2inst1@53fe3a4d3ad2 ~]$ db2 list utilities show detail
    
    ID                               = 18435
    Type                             = ASYNCHRONOUS PARTITION DETACH
    Database Name                    = FHIRDB
    Member Number                    = 0
    Description                      = Finalize the detach for partition '3' of table 'FHIRDATA.PARAMETER_NAMES'
    Start Time                       = 06/08/2021 16:31:05.526513
    State                            = Executing
    Invocation Type                  = Automatic
    Progress Monitoring:
          Description                = Performing detach operation and
     making the target table available; new compilations blocked
          Start Time                 = 06/08/2021 16:31:10.836936
    

    I’ve attached a useful partition.sql to demo partitions and check the system catalog.

    Links

    Example Code