Bulk Data: Using the SMART-on-FHIR Bulk Data Client to test $export

The SMART-on-FHIR team has built a node application to call Bulk Data $export and facilitate the calls to get a bearer token.

1. Clone the Bulk Data Client repository

/git/wffh/2023$ git clone -b main-pb --single-branch \
    https://github.com/prb112/bulk-data-client.git && \
    cd $(basename $_ .git)
Cloning into 'bulk-data-client'...
remote: Enumerating objects: 343, done.
remote: Counting objects: 100% (343/343), done.
remote: Compressing objects: 100% (188/188), done.
remote: Total 343 (delta 220), reused 261 (delta 146), pack-reused 0
Receiving objects: 100% (343/343), 407.93 KiB | 3.32 MiB/s, done.
Resolving deltas: 100% (220/220), done.
/git/wffh/2023/bulk-data-client$

Note I added a changes to support the IBM FHIR Server.

2. Install nvm

$ brew install nvm
Running `brew update --preinstall`...
==> Auto-updated Homebrew!
Updated 3 taps (homebrew/core, homebrew/cask and minio/stable).
==> New Formulae
cwb3                                                      erofs-utils                                               netmask                                                   scalingo                                                  zk
==> Updated Formulae
Updated 390 formulae.

...

nvm 0.35.3 is already installed but outdated (so it will be upgraded).
==> Downloading https://ghcr.io/v2/homebrew/core/nvm/manifests/0.39.1_1
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/nvm/blobs/sha256:6e14c8a2bf94212545c1ebac9a722df168c318d0e8af2fc75b729a07fea54efe
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:6e14c8a2bf94212545c1ebac9a722df168c318d0e8af2fc75b729a07fea54efe?se=2022-01-13T21%3A30%3A00Z&sig=52CYOAyuxabqLNYih9%2BjMQtHLFrN%2FVyHXf%2FgPuBgi6w%3D&sp=r&spr=https&sr=b&sv=2019-12-12
######################################################################## 100.0%
==> Upgrading nvm
  0.35.3 -> 0.39.1_1

==> Pouring nvm--0.39.1_1.all.bottle.tar.gz
...

Bash completion has been installed to:
  /usr/local/etc/bash_completion.d
==> Summary
🍺  /usr/local/Cellar/nvm/0.39.1_1: 9 files, 184.1KB
==> Running `brew cleanup nvm`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
Removing: /usr/local/Cellar/nvm/0.35.3... (7 files, 149.9KB)

3. Switch to Node 16

$ nvm use 16
Now using node v16.13.0 (npm v8.1.2)

4. Install Node 16

$ nvm install
Found '/git/wffh/2023/bulk-data-client/.nvmrc' with version <16>
Downloading and installing node v16.13.2...
Downloading https://nodejs.org/dist/v16.13.2/node-v16.13.2-darwin-x64.tar.xz...
############################################################## 100.0%
Computing checksum with sha256sum
Checksums matched!
Now using node v16.13.2 (npm v8.1.2)

5. Create the binaries

$ npm run clean && npm run build

> bulk-data-client@1.0.0 clean
> rm -rf ./built


> bulk-data-client@1.0.0 build
> tsc

5. Create the config/ibm-fhir-server.js – update the fhirUrl, tokenUrl and privateKey. I used the RS384 Key.

6. Run the application

$ AUTO_RETRY_TRANSIENT_ERRORS=1 SHOW_ERRORS=1 node . --config config/ibm-fhir-server.js  --_type=Patient
Kick-off started
Got new access token
Kick-off completed
Bulk Data export started
Bulk Data export completed in 10 seconds
Export manifest:  {
  transactionTime: '2022-01-14T20:56:31.553Z',
  request: 'https://bulk.cluster1-blue-x123456.containers.appdomain.cloud/fhir-server/api/v4/Patient/$export?_type=Patient',
  requiresAccessToken: false,
  output: [
    {
      type: 'Patient',
      url: 'https://x123456.cloud-object-storage.appdomain.cloud/x123456/f1wSMD84-3eRVNTUlyIwctWtNxA33Odz14KiQctry-U/Patient_1.ndjson?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=x123456%2F20220114%2Fus-east%2Fs3%2Faws4_request&X-Amz-Date=20220114T205641Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=abcdefgh',
      count: 2
    }
  ],
  error: []
}

Downloading exported files: ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 100%
          Downloaded Files: 1 of 1
            FHIR Resources: 2
               Attachments: 0
           Downloaded Size: 6.3 kB

Download completed in 0 seconds
Do you want to signal the server that this export can be removed? [Y/n]n

If you need to debug, you’ll see all the outputs:

NODE_DEBUG=* AUTO_RETRY_TRANSIENT_ERRORS=1 SHOW_ERRORS=1 node . \
   --config config/ibm-fhir-server.js

7. The files are downloaded to a local directory, and you can check the data. cat downloads/1.Patient.ndjson

{"resourceType":"Patient","id":"17e44643c84-7618d4fe-e1d3-48e5-8a66-923bd4780c17","meta":{"versionId":"1","lastUpdated":"2022-01-10T14:28:46.854836Z"},"text":{"status":"generated","div":"<div xmlns=\"http://www.w3.org/1999/xhtml\">Generated by <a href=\"https://github.com/synthetichealth/synthea\">Synthea</a>.Version identifier: master-branch-latest-2-g8e7e92c\n .   Person seed: 7105486291024734541  Population seed: 0</div>"},"extension":[{"url":"http://hl7.org/fhir/StructureDefinition/patient-mothersMaidenName","valueString":"Graciela518 Badillo851"},{"url":"http://hl7.org/fhir/StructureDefinition/patient-birthPlace","valueAddress":{"city":"Caguas","state":"Puerto Rico","country":"PR"}},{"url":"http://synthetichealth.github.io/synthea/disability-adjusted-life-years","valueDecimal":1.6165934938317164},{"url":"http://synthetichealth.github.io/synthea/quality-adjusted-life-years","valueDecimal":47.383406506168285},{"url":"http://hl7.org/fhir/us/core/StructureDefinition/us-core-birthsex","valueCode":"F"},{"url":"http://hl7.org/fhir/us/core/StructureDefinition/us-core-birthsex","valueCode":"M"}],"identifier":[{"system":"https://github.com/synthetichealth/synthea","value":"bd958c64-56b6-f206-b03d-8f4b8a417215"},{"type":{"coding":[{"system":"http://terminology.hl7.org/CodeSystem/v2-0203","code":"MR","display":"Medical Record Number"}],"text":"Medical Record Number"},"system":"http://hospital.smarthealthit.org","value":"bd958c64-56b6-f206-b03d-8f4b8a417215"},{"type":{"coding":[{"system":"http://terminology.hl7.org/CodeSystem/v2-0203","code":"SS","display":"Social Security Number"}],"text":"Social Security Number"},"system":"http://hl7.org/fhir/sid/us-ssn","value":"999-93-6090"},{"type":{"coding":[{"system":"http://terminology.hl7.org/CodeSystem/v2-0203","code":"DL","display":"Driver's License"}],"text":"Driver's License"},"system":"urn:oid:2.16.840.1.113883.4.3.25","value":"S99913499"},{"type":{"coding":[{"system":"http://terminology.hl7.org/CodeSystem/v2-0203","code":"PPN","display":"Passport Number"}],"text":"Passport Number"},"system":"http://standardhealthrecord.org/fhir/StructureDefinition/passportNumber","value":"X26583229X"}],"name":[{"use":"official","family":"Acosta403","given":["Antonia30"],"prefix":["Mrs."]},{"use":"maiden","family":"Armenta418","given":["Antonia30"],"prefix":["Mrs."]}],"telecom":[{"system":"phone","value":"555-457-3489","use":"home"}],"gender":"female","birthDate":"1970-09-06","address":[{"extension":[{"extension":[{"url":"latitude","valueDecimal":41.6497330022825},{"url":"longitude","valueDecimal":-71.17511064024423}],"url":"http://hl7.org/fhir/StructureDefinition/geolocation"}],"line":["635 Littel Esplanade Suite 65"],"city":"Fall River","state":"Massachusetts","postalCode":"02790","country":"US"}],"maritalStatus":{"coding":[{"system":"http://terminology.hl7.org/CodeSystem/v3-MaritalStatus","code":"M","display":"M"}],"text":"M"},"multipleBirthBoolean":false,"communication":[{"language":{"coding":[{"system":"urn:ietf:bcp:47","code":"es","display":"Spanish"}],"text":"Spanish"}}]}
{"resourceType":"Patient","id":"17e4f2f3dca-67724e2c-3991-4ac1-b6f9-51fc30209894","meta":{"versionId":"1","lastUpdated":"2022-01-12T16:46:43.149786Z"},"text":{"status":"generated","div":"<div xmlns=\"http://www.w3.org/1999/xhtml\">Generated by <a href=\"https://github.com/synthetichealth/synthea\">Synthea</a>.Version identifier: v2.4.0-404-ge7ce2295\n .   Person seed: -8292973307042192125  Population seed: 0</div>"},"extension":[{"extension":[{"url":"ombCategory","valueCoding":{"system":"urn:oid:2.16.840.1.113883.6.238","code":"2135-2","display":"Other"}},{"url":"text","valueString":"Other"}],"url":"http://hl7.org/fhir/us/core/StructureDefinition/us-core-race"},{"extension":[{"url":"ombCategory","valueCoding":{"system":"urn:oid:2.16.840.1.113883.6.238","code":"2186-5","display":"Not Hispanic or Latino"}},{"url":"text","valueString":"Not Hispanic or Latino"}],"url":"http://hl7.org/fhir/us/core/StructureDefinition/us-core-ethnicity"},{"url":"http://hl7.org/fhir/StructureDefinition/patient-mothersMaidenName","valueString":"Isabel214 Reséndez908"},{"url":"http://hl7.org/fhir/us/core/StructureDefinition/us-core-birthsex","valueCode":"M"},{"url":"http://hl7.org/fhir/StructureDefinition/patient-birthPlace","valueAddress":{"city":"La Paz","state":"Baja California","country":"MX"}},{"url":"http://synthetichealth.github.io/synthea/disability-adjusted-life-years","valueDecimal":0.13213216767824654},{"url":"http://synthetichealth.github.io/synthea/quality-adjusted-life-years","valueDecimal":34.86786783232176}],"identifier":[{"system":"https://github.com/synthetichealth/synthea","value":"d171d808-1f31-4ad3-aba5-e03a2fa921c7"},{"type":{"coding":[{"system":"http://terminology.hl7.org/CodeSystem/v2-0203","code":"MR","display":"Medical Record Number"}],"text":"Medical Record Number"},"system":"http://hospital.smarthealthit.org","value":"d171d808-1f31-4ad3-aba5-e03a2fa921c7"},{"type":{"coding":[{"system":"http://terminology.hl7.org/CodeSystem/v2-0203","code":"SS","display":"Social Security Number"}],"text":"Social Security Number"},"system":"http://hl7.org/fhir/sid/us-ssn","value":"999-83-6585"},{"type":{"coding":[{"system":"http://terminology.hl7.org/CodeSystem/v2-0203","code":"DL","display":"Driver's License"}],"text":"Driver's License"},"system":"urn:oid:2.16.840.1.113883.4.3.25","value":"S99914532"},{"type":{"coding":[{"system":"http://terminology.hl7.org/CodeSystem/v2-0203","code":"PPN","display":"Passport Number"}],"text":"Passport Number"},"system":"http://standardhealthrecord.org/fhir/StructureDefinition/passportNumber","value":"X18544570X"}],"name":[{"use":"official","family":"Hurtado459","given":["Marco Antonio298"],"prefix":["Mr."]}],"telecom":[{"system":"phone","value":"555-185-5373","use":"home"}],"gender":"male","birthDate":"1983-04-03","address":[{"extension":[{"extension":[{"url":"latitude","valueDecimal":42.49384661818001},{"url":"longitude","valueDecimal":-70.92858466579901}],"url":"http://hl7.org/fhir/StructureDefinition/geolocation"}],"line":["552 Rippin Port"],"city":"Revere","state":"Massachusetts","postalCode":"02151","country":"US"}],"maritalStatus":{"coding":[{"system":"http://terminology.hl7.org/CodeSystem/v3-MaritalStatus","code":"M","display":"M"}],"text":"M"},"multipleBirthBoolean":false,"communication":[{"language":{"coding":[{"system":"urn:ietf:bcp:47","code":"es","display":"Spanish"}],"text":"Spanish"}}]}

As you can see the bulk-data-client is a handy tool to test Bulk Data with $export with SMART Backend Services Authorization.

We opened a few issues with the bulk data client team:

1. Bearer token is expected to be capitalized Bearer. #1
2. User-Agent string is awkward #3
3. Output doesn’t give a lot of details on what resourceType was exported #5

And one we’re watching:

4. Provides token even if requiresAccessToken is false #2

We opened an Pull Request #1 to address 1,2,4.