The IBM FHIR Server supports hard delete using the custom FHIR Operation Framework. The operation is called $erase; you can always read the design document.

The operation supports:

• Hard Delete All Versions of a Resource – removes all versions including the latest, and no longer available using Search
• Version Specific Hard Delete – the latest version may not be hard deleted, you must execute a soft DELETE, then a hard delete. Otherwise, all versions may be deleted.

By default, the $erase operation is not enabled, and must be set to true, and the roles allowed to execute the operation, can be tweaked to FHIROperationAdmin or FHIRUsers.

Property	Type	Description
fhirServer/operations/erase/enabled	boolean	Enables the $erase operation
fhirServer/operations/erase/allowedRoles	list	The list of allowed roles, allowed entries are: FHIRUsers every authenticated user, FHIROperationAdmin which is authenticated FHIRAdmin users

The fhir-server-config.json should be amended with a snippet like the following:

"operations": {
            "erase": {
                "enabled": true,
                "allowedRoles": [
                    "FHIROperationAdmin",
                    "FHIRUsers"
                ]
            }
        }

As the $erase operation uses stored procedures for Db2 and functions for Postgres, the data schema must be updated to the 4.8.3 level. It’s also worth noting that Derby is also supported as it uses the same DAO as the Db2 and Postgres to run.

A tip, you should search for the references to the Patient’s ResourceId, and delete the referenced resources in a Transaction Bundle. Net – Search, Assemble Bundle, Execute Batch.

Let me show you how to run with it, and operate the $erase operation with a container (Docker: ibmcom/ibm-fhir-server).

Recipe

1. Download the fhir-server-config.json

2. Start the Docker container, and capture the container id. It’s going to take a few moments to start up as it lays down the test database.

docker run -d -p 9443:9443 -e BOOTSTRAP_DB=true -v $(pwd)/fhir-server-config.json:/config/config/default/fhir-server-config.json ibmcom/ibm-fhir-server
cde61943964464b528eb77d132fd2a4952e0eaf43588da1c9b6bb2fa584f0608

3. Check the logs until you see:

docker logs cde61943964464b528eb77d132fd2a4952e0eaf43588da1c9b6bb2fa584f0608
...
[6/16/21, 15:31:34:533 UTC] 0000002a FeatureManage A   CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 17.665 seconds.

4. Download the Postman Collection for Version Specific Erase and All Versions Erase.

5. Import the Postman Collections and Run

You’ve seen how to use the $erase operation, and a bit more details on configuring and using it.

Postman Collections

The IBM FHIR Server has support for exporting and importing Bulk Data using extended operations for Bulk Data $import, $export and $bulkdata-status, which are implemented as Java Maven projects. The IBM FHIR Server uses JSR252 JavaBatch jobs running in the Open Liberty Java Batch Framework.

The IBM FHIR Server supports storage providers S3 (aws-s3,ibm-cos), File System (file), and now Azure (azure-blob).

I recently implemented support for the Azure Blob Service in IBM/FHIR Pull Request #2413. The code that supports the Azure client uses the OK Http library and not Netty to communicate with the backend. The storage provider is implemented as AzureProvider, and uses the Java – AppendBlobClient to facilitate export. To facilitate import, the code uses the BlobClient to read from a location in 10Kb blocks, and reassemble when a full block is reached. Importantly, this is done in a way to ensure full resources are returned up to a 2Gb limit.

Overview Video

If you want to use the IBM FHIR Server’s bulk data feature with the IBM FHIR Server, use the following recipe:

Recipe – Azure Blob and Bulk Data

1. Login to the Azure Portal

2. Click Home

3. Navigate the Azure services, and click on Storage Accounts.

4. Click Create

5. Enter the appropriate subscription and resource group

6. Give it a name such as fhirintegrationit

7. Click Review+Create

8. Click Create

You’ll see Deployment in Progress for a period of time.

Once it is done, you see Your deployment is complete, proceed to the next steps.

9. Click Go to Resource

10. Click on Access Key

11. Click Show keys and copy the key1’s Connection string. It’ll look like DefaultEndpointsProtocol=https;AccountName=fhirintegrationit;AccountKey=HIDDEN==;EndpointSuffix=core.windows.net

12. Click on Data > Containers

13. Click +Container

14. Fill in the container details and call it bulkdata

14. Click on the bulkdata container

15. Upload blob Data – AllergyIntolerance.ndjson

You should see the file uploaded and available.

You are now ready to use this with the IBM FHIR Server from the main branch. If you need to clone the repo go to IBM/FHIR.

16. Build the Maven Projects and the Docker Build. You should see “[INFO] BUILD SUCCESS” after each Maven build, and “=> naming to docker.io/ibmcom/ibm-fhir-server:latest” when the Docker build is successful.

mvn clean install -f fhir-examples -B -DskipTests -ntp
mvn clean install -f fhir-parent -B -DskipTests -ntp
docker build -t ibmcom/ibm-fhir-server:latest fhir-install

17. Download the fhir-server-config.json

18. Open the fhir-server-config.json and update fhirServer/bulkdata/storageProviders/default entry to be the connection is updated to your Connection string from above.

17. Start the Docker container, and capture the container id. It’s going to take a few moments to start up as it lays down the test database.

docker run -d -p 9443:9443 -e BOOTSTRAP_DB=true -v $(pwd)/fhir-server-config.json:/config/config/default/fhir-server-config.json ibmcom/ibm-fhir-server
cde61943964464b528eb77d132fd2a4952e0eaf43588da1c9b6bb2fa584f0608

18. Check the logs until you see:

docker logs cde61943964464b528eb77d132fd2a4952e0eaf43588da1c9b6bb2fa584f0608
...
[6/16/21, 15:31:34:533 UTC] 0000002a FeatureManage A   CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 17.665 seconds.

19. Create an $import request and capture the content-location header.

20. Check and poll until you get response code 200 (you should only get 202 in the interim) until you see (note 99 failed validation which is expected):

curl -v -k --user 'fhiruser:change-password' 'https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=UKt4ESCnqOvAfxYWhdsfUg'

< HTTP/2 200
< content-type: application/json
< date: Wed, 16 Jun 2021 16:01:54 GMT
< content-language: en-US
< content-length: 501
<
{
    "transactionTime": "2021-06-16T15:59:53.086Z",
    "request": "https://localhost:9443/fhir-server/api/v4/$import",
    "requiresAccessToken": false,
    "output": [
        {
            "type": "OperationOutcome",
            "url": "AllergyIntolerance.ndjson_oo_success.ndjson",
            "count": 1600
        }
    ],
    "error": [
        {
            "type": "OperationOutcome",
            "url": "AllergyIntolerance.ndjson_oo_errors.ndjson",
            "count": 99
        }
    ]
* Connection #0 to host localhost left intact
}

21. We can check to see if there is data on the server, and there are 1600 resources.

curl -v -k --user 'fhiruser:change-password' 'https://localhost:9443/fhir-server/api/v4/AllergyIntolerance?_format=json'

{"resourceType":"Bundle","id":"d485339f-49e5-4b11-a1be-4ae06edd0b16","type":"searchset","total":1600,"link"...}

22. Now we run $export and capture the content-location:

curl --location --request GET -k -v 'https://localhost:9443/fhir-server/api/v4/$export?_outputFormat=application/fhir+ndjson&_type=AllergyIntolerance' \
--header 'X-FHIR-TENANT-ID: default' \
--header 'Content-Type: application/fhir+json' \
--header 'X-FHIR-BULKDATA-PROVIDER: default' \
--header 'X-FHIR-BULKDATA-PROVIDER-OUTCOME: default' \
--header 'Authorization: Basic ZmhpcnVzZXI6Y2hhbmdlLXBhc3N3b3Jk'

< HTTP/2 202
< content-location: https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=LqzauvqtHSmkpChVHo%2B1MQ
< date: Wed, 16 Jun 2021 16:39:19 GMT
< content-length: 0
< content-language: en-US
<
* Connection #0 to host localhost left intact
* Closing connection 0

23. Poll and check the response 202 or 200 and the contents of the response. (empty until done)

curl -k --user 'fhiruser:change-password' 'https://localhost:9443/fhir-server/api/v4/$bulkdata-status?job=LqzauvqtHSmkpChVHo%2B1MQ'
{
    "transactionTime": "2021-06-16T16:39:26.551Z",
    "request": "https://localhost:9443/fhir-server/api/v4/$export?_outputFormat=application/fhir+ndjson&_type=AllergyIntolerance",
    "requiresAccessToken": false,
    "output": [
        {
            "type": "AllergyIntolerance",
            "url": "c7GRY0uFsu-hO_QlquVZJr_7hTg5c_m6SHlZ1z3Z7J4/AllergyIntolerance_1.ndjson",
            "count": 1600
        }
    ]
}

24. Check Azure Blob Service and you’ll see the folder and file on Azure Blob Service

25. Download and check there are 1600 lines.

You now know how to use Azure and Bulk Data.