Tag: fhir

  • Bulk Data: Using the SMART-on-FHIR Bulk Data Client to test $export

    I recently attended the HL7 FHIR Connectathon 29. For those that are not familiar with Connectathons, I think they are fairly unique events featuring standards enthusiasts, vendors and implementors doing hands-on standards development (FHIR) and testing. As an attendee I picked one of the tracksbulk data.

    This blog is part of a series on Bulk Data Setup and Testing based on my experience at HL7 FHIR Connectathon 29.

    The SMART-on-FHIR team has built a node application to call Bulk Data $export and facilitate the calls to get a bearer token.

    1. Clone the Bulk Data Client repository
    /git/wffh/2023$ git clone -b main-pb --single-branch \
        https://github.com/prb112/bulk-data-client.git && \
        cd $(basename $_ .git)
    Cloning into 'bulk-data-client'...
    remote: Enumerating objects: 343, done.
    remote: Counting objects: 100% (343/343), done.
    remote: Compressing objects: 100% (188/188), done.
    remote: Total 343 (delta 220), reused 261 (delta 146), pack-reused 0
    Receiving objects: 100% (343/343), 407.93 KiB | 3.32 MiB/s, done.
    Resolving deltas: 100% (220/220), done.
    /git/wffh/2023/bulk-data-client$
    

    Note I added a changes to support the IBM FHIR Server.

    1. Install nvm
    $ brew install nvm
    Running `brew update --preinstall`...
    ==> Auto-updated Homebrew!
    Updated 3 taps (homebrew/core, homebrew/cask and minio/stable).
    ==> New Formulae
    cwb3                                                      erofs-utils                                               netmask                                                   scalingo                                                  zk
    ==> Updated Formulae
    Updated 390 formulae.
    
    ...
    
    nvm 0.35.3 is already installed but outdated (so it will be upgraded).
    ==> Downloading https://ghcr.io/v2/homebrew/core/nvm/manifests/0.39.1_1
    ######################################################################## 100.0%
    ==> Downloading https://ghcr.io/v2/homebrew/core/nvm/blobs/sha256:6e14c8a2bf94212545c1ebac9a722df168c318d0e8af2fc75b729a07fea54efe
    ==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:6e14c8a2bf94212545c1ebac9a722df168c318d0e8af2fc75b729a07fea54efe?se=2022-01-13T21%3A30%3A00Z&sig=52CYOAyuxabqLNYih9%2BjMQtHLFrN%2FVyHXf%2FgPuBgi6w%3D&sp=r&spr=https&sr=b&sv=2019-12-12
    ######################################################################## 100.0%
    ==> Upgrading nvm
      0.35.3 -> 0.39.1_1
    
    ==> Pouring nvm--0.39.1_1.all.bottle.tar.gz
    ...
    
    Bash completion has been installed to:
      /usr/local/etc/bash_completion.d
    ==> Summary
    🍺  /usr/local/Cellar/nvm/0.39.1_1: 9 files, 184.1KB
    ==> Running `brew cleanup nvm`...
    Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
    Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
    Removing: /usr/local/Cellar/nvm/0.35.3... (7 files, 149.9KB)
    
    1. Switch to Node 16
    $ nvm use 16
    Now using node v16.13.0 (npm v8.1.2)
    
    1. Install Node 16
    $ nvm install
    Found '/git/wffh/2023/bulk-data-client/.nvmrc' with version <16>
    Downloading and installing node v16.13.2...
    Downloading https://nodejs.org/dist/v16.13.2/node-v16.13.2-darwin-x64.tar.xz...
    ############################################################## 100.0%
    Computing checksum with sha256sum
    Checksums matched!
    Now using node v16.13.2 (npm v8.1.2)
    
    1. Create the binaries
    $ npm run clean && npm run build
    
    > bulk-data-client@1.0.0 clean
    > rm -rf ./built
    
    
    > bulk-data-client@1.0.0 build
    > tsc
    
    1. Create the config/ibm-fhir-server.js – update the fhirUrl, tokenUrl and privateKey. I used the RS384 Key.
    1. Run the application
    $ AUTO_RETRY_TRANSIENT_ERRORS=1 SHOW_ERRORS=1 node . --config config/ibm-fhir-server.js  --_type=Patient
    Kick-off started
    Got new access token
    Kick-off completed
    Bulk Data export started
    Bulk Data export completed in 10 seconds
    Export manifest:  {
      transactionTime: '2022-01-14T20:56:31.553Z',
      request: 'https://bulk.cluster1-blue-x123456.containers.appdomain.cloud/fhir-server/api/v4/Patient/$export?_type=Patient',
      requiresAccessToken: false,
      output: [
        {
          type: 'Patient',
          url: 'https://x123456.cloud-object-storage.appdomain.cloud/x123456/f1wSMD84-3eRVNTUlyIwctWtNxA33Odz14KiQctry-U/Patient_1.ndjson?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=x123456%2F20220114%2Fus-east%2Fs3%2Faws4_request&X-Amz-Date=20220114T205641Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=abcdefgh',
          count: 2
        }
      ],
      error: []
    }
    
    Downloading exported files: β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰β–‰ 100%
              Downloaded Files: 1 of 1
                FHIR Resources: 2
                   Attachments: 0
               Downloaded Size: 6.3 kB
    
    Download completed in 0 seconds
    Do you want to signal the server that this export can be removed? [Y/n]n
    

    If you need to debug, you’ll see all the outputs:

    NODE_DEBUG=* AUTO_RETRY_TRANSIENT_ERRORS=1 SHOW_ERRORS=1 node . \
       --config config/ibm-fhir-server.js
    
    1. The files are downloaded to a local directory, and you can check the data. cat downloads/1.Patient.ndjson
    {"resourceType":"Patient","id":"17e44643c84-7618d4fe-e1d3-48e5-8a66-923bd4780c17","meta":{"versionId":"1","lastUpdated":"2022-01-10T14:28:46.854836Z"},"text":{"status":"generated","div":"<div xmlns=\"http://www.w3.org/1999/xhtml\">Generated by <a href=\"https://github.com/synthetichealth/synthea\">Synthea</a>.Version identifier: master-branch-latest-2-g8e7e92c\n .   Person seed: 7105486291024734541  Population seed: 0</div>"},"extension":[{"url":"http://hl7.org/fhir/StructureDefinition/patient-mothersMaidenName","valueString":"Graciela518 Badillo851"},{"url":"http://hl7.org/fhir/StructureDefinition/patient-birthPlace","valueAddress":{"city":"Caguas","state":"Puerto Rico","country":"PR"}},{"url":"http://synthetichealth.github.io/synthea/disability-adjusted-life-years","valueDecimal":1.6165934938317164},{"url":"http://synthetichealth.github.io/synthea/quality-adjusted-life-years","valueDecimal":47.383406506168285},{"url":"http://hl7.org/fhir/us/core/StructureDefinition/us-core-birthsex","valueCode":"F"},{"url":"http://hl7.org/fhir/us/core/StructureDefinition/us-core-birthsex","valueCode":"M"}],"identifier":[{"system":"https://github.com/synthetichealth/synthea","value":"bd958c64-56b6-f206-b03d-8f4b8a417215"},{"type":{"coding":[{"system":"http://terminology.hl7.org/CodeSystem/v2-0203","code":"MR","display":"Medical Record Number"}],"text":"Medical Record Number"},"system":"http://hospital.smarthealthit.org","value":"bd958c64-56b6-f206-b03d-8f4b8a417215"},{"type":{"coding":[{"system":"http://terminology.hl7.org/CodeSystem/v2-0203","code":"SS","display":"Social Security Number"}],"text":"Social Security Number"},"system":"http://hl7.org/fhir/sid/us-ssn","value":"999-93-6090"},{"type":{"coding":[{"system":"http://terminology.hl7.org/CodeSystem/v2-0203","code":"DL","display":"Driver's License"}],"text":"Driver's License"},"system":"urn:oid:2.16.840.1.113883.4.3.25","value":"S99913499"},{"type":{"coding":[{"system":"http://terminology.hl7.org/CodeSystem/v2-0203","code":"PPN","display":"Passport Number"}],"text":"Passport Number"},"system":"http://standardhealthrecord.org/fhir/StructureDefinition/passportNumber","value":"X26583229X"}],"name":[{"use":"official","family":"Acosta403","given":["Antonia30"],"prefix":["Mrs."]},{"use":"maiden","family":"Armenta418","given":["Antonia30"],"prefix":["Mrs."]}],"telecom":[{"system":"phone","value":"555-457-3489","use":"home"}],"gender":"female","birthDate":"1970-09-06","address":[{"extension":[{"extension":[{"url":"latitude","valueDecimal":41.6497330022825},{"url":"longitude","valueDecimal":-71.17511064024423}],"url":"http://hl7.org/fhir/StructureDefinition/geolocation"}],"line":["635 Littel Esplanade Suite 65"],"city":"Fall River","state":"Massachusetts","postalCode":"02790","country":"US"}],"maritalStatus":{"coding":[{"system":"http://terminology.hl7.org/CodeSystem/v3-MaritalStatus","code":"M","display":"M"}],"text":"M"},"multipleBirthBoolean":false,"communication":[{"language":{"coding":[{"system":"urn:ietf:bcp:47","code":"es","display":"Spanish"}],"text":"Spanish"}}]}
    {"resourceType":"Patient","id":"17e4f2f3dca-67724e2c-3991-4ac1-b6f9-51fc30209894","meta":{"versionId":"1","lastUpdated":"2022-01-12T16:46:43.149786Z"},"text":{"status":"generated","div":"<div xmlns=\"http://www.w3.org/1999/xhtml\">Generated by <a href=\"https://github.com/synthetichealth/synthea\">Synthea</a>.Version identifier: v2.4.0-404-ge7ce2295\n .   Person seed: -8292973307042192125  Population seed: 0</div>"},"extension":[{"extension":[{"url":"ombCategory","valueCoding":{"system":"urn:oid:2.16.840.1.113883.6.238","code":"2135-2","display":"Other"}},{"url":"text","valueString":"Other"}],"url":"http://hl7.org/fhir/us/core/StructureDefinition/us-core-race"},{"extension":[{"url":"ombCategory","valueCoding":{"system":"urn:oid:2.16.840.1.113883.6.238","code":"2186-5","display":"Not Hispanic or Latino"}},{"url":"text","valueString":"Not Hispanic or Latino"}],"url":"http://hl7.org/fhir/us/core/StructureDefinition/us-core-ethnicity"},{"url":"http://hl7.org/fhir/StructureDefinition/patient-mothersMaidenName","valueString":"Isabel214 ResΓ©ndez908"},{"url":"http://hl7.org/fhir/us/core/StructureDefinition/us-core-birthsex","valueCode":"M"},{"url":"http://hl7.org/fhir/StructureDefinition/patient-birthPlace","valueAddress":{"city":"La Paz","state":"Baja California","country":"MX"}},{"url":"http://synthetichealth.github.io/synthea/disability-adjusted-life-years","valueDecimal":0.13213216767824654},{"url":"http://synthetichealth.github.io/synthea/quality-adjusted-life-years","valueDecimal":34.86786783232176}],"identifier":[{"system":"https://github.com/synthetichealth/synthea","value":"d171d808-1f31-4ad3-aba5-e03a2fa921c7"},{"type":{"coding":[{"system":"http://terminology.hl7.org/CodeSystem/v2-0203","code":"MR","display":"Medical Record Number"}],"text":"Medical Record Number"},"system":"http://hospital.smarthealthit.org","value":"d171d808-1f31-4ad3-aba5-e03a2fa921c7"},{"type":{"coding":[{"system":"http://terminology.hl7.org/CodeSystem/v2-0203","code":"SS","display":"Social Security Number"}],"text":"Social Security Number"},"system":"http://hl7.org/fhir/sid/us-ssn","value":"999-83-6585"},{"type":{"coding":[{"system":"http://terminology.hl7.org/CodeSystem/v2-0203","code":"DL","display":"Driver's License"}],"text":"Driver's License"},"system":"urn:oid:2.16.840.1.113883.4.3.25","value":"S99914532"},{"type":{"coding":[{"system":"http://terminology.hl7.org/CodeSystem/v2-0203","code":"PPN","display":"Passport Number"}],"text":"Passport Number"},"system":"http://standardhealthrecord.org/fhir/StructureDefinition/passportNumber","value":"X18544570X"}],"name":[{"use":"official","family":"Hurtado459","given":["Marco Antonio298"],"prefix":["Mr."]}],"telecom":[{"system":"phone","value":"555-185-5373","use":"home"}],"gender":"male","birthDate":"1983-04-03","address":[{"extension":[{"extension":[{"url":"latitude","valueDecimal":42.49384661818001},{"url":"longitude","valueDecimal":-70.92858466579901}],"url":"http://hl7.org/fhir/StructureDefinition/geolocation"}],"line":["552 Rippin Port"],"city":"Revere","state":"Massachusetts","postalCode":"02151","country":"US"}],"maritalStatus":{"coding":[{"system":"http://terminology.hl7.org/CodeSystem/v3-MaritalStatus","code":"M","display":"M"}],"text":"M"},"multipleBirthBoolean":false,"communication":[{"language":{"coding":[{"system":"urn:ietf:bcp:47","code":"es","display":"Spanish"}],"text":"Spanish"}}]}
    

    As you can see the bulk-data-client is a handy tool to test Bulk Data with $export with SMART Backend Services Authorization.

  • jq fu

    Extracting a Resource from an Array

    Extracting a resource from a FHIR Bundle with over 10000 entries, and you know there is a problem at a specific resource, then you can use jq and array processing to extract the resource:

    single_patient_bundle-03-09-2020/9b3f6160-285d-4319-8d15-ac07ee3d3a8e.json \
        | jq '.entry[12672].resource'
    {
      "id": "99274e87-db14-43fa-9ada-2fcb6c1d68a6",
      "meta": {
        "profile": [
          "http://hl7.org/fhir/StructureDefinition/vitalspanel",
          "http://hl7.org/fhir/StructureDefinition/vitalsigns"
        ]
      },
      "status": "final",
      "resourceType": "Observation"
    }```

    Extracting two correlated values

    Extracting two correlated values, you can use the multiple selectors, such as the following:

    cat single_patient_bundle-03-09-2020/9b3f6160-285d-4319-8d15-ac07ee3d3a8e.json \ 
       | jq '.entry[] | "\(.status),\(.resourceType)"' | sort -u 
    final,Observation
    

    Checking the Supported Profiles on the IBM FHIR Server

    This is a handy curl to check what profiles are loaded on your IBM FHIR Server.

    Request

    curl -ks -u fhiruser:change-password https://localhost:9443/fhir-server/api/v4/metadata 2>&1 | jq -r '.rest[].resource[] | "\(.type),\(.supportedProfile)"'
    

    Processed Response

    PractitionerRole,["http://hl7.org/fhir/us/carin/StructureDefinition/carin-bb-practitionerrole|0.1.0","http://hl7.org/fhir/us/core/StructureDefinition/us-core-practitionerrole|3.1.0","http://hl7.org/fhir/us/davinci-pdex-plan-net/StructureDefinition/plannet-PractitionerRole|0.1.0"]
    Procedure,["http://hl7.org/fhir/us/core/StructureDefinition/us-core-procedure|3.1.0"]
    Provenance,["http://hl7.org/fhir/StructureDefinition/ehrsrle-provenance|4.0.1","http://hl7.org/fhir/StructureDefinition/provenance-relevant-history|4.0.1","http://hl7.org/fhir/us/core/StructureDefinition/us-core-provenance|3.1.0"]
    Questionnaire,["http://hl7.org/fhir/StructureDefinition/cqf-questionnaire|4.0.1"]
    QuestionnaireResponse,null
    RelatedPerson,["http://hl7.org/fhir/us/carin/StructureDefinition/carin-bb-relatedperson|0.1.0"]
    RequestGroup,["http://hl7.org/fhir/StructureDefinition/cdshooksrequestgroup|4.0.1"]
    ResearchDefinition,null
    ResearchElementDefinition,null
    ResearchStudy,null
    ResearchSubject,null
    

    Extracting Search Parameters with a Type Composite

    cat ./fhir-registry/definitions/search-parameters.json | jq -r '.entry[].resource | select(.type == "composite") | .expression' | sort -u

    ActivityDefinition.useContext
    CapabilityStatement.useContext | CodeSystem.useContext | CompartmentDefinition.useContext | ConceptMap.useContext | GraphDefinition.useContext | ImplementationGuide.useContext | MessageDefinition.useContext | NamingSystem.useContext | OperationDefinition.useContext | SearchParameter.useContext | StructureDefinition.useContext | StructureMap.useContext | TerminologyCapabilities.useContext | ValueSet.useContext
    ChargeItemDefinition.useContext
    DocumentReference.relatesTo
    EffectEvidenceSynthesis.useContext
    EventDefinition.useContext
    Evidence.useContext
    EvidenceVariable.useContext
    ExampleScenario.useContext
    Group.characteristic
    Library.useContext
    Measure.useContext
    MolecularSequence.referenceSeq
    MolecularSequence.variant
    Observation
    Observation | Observation.component
    Observation.component
    PlanDefinition.useContext
    Questionnaire.useContext
    ResearchDefinition.useContext
    ResearchElementDefinition.useContext
    RiskEvidenceSynthesis.useContext
    TestScript.useContext
    

    Extracting Composite Codes from Search Parameters

    cat ./fhir-registry/definitions/search-parameters.json | jq -r '.entry[].resource | select(.type == "composite") | .code'

    context-type-quantity
    context-type-value
    context-type-quantity
    context-type-value
    context-type-quantity
    context-type-value
    relationship
    ...
    chromosome-variant-coordinate
    chromosome-window-coordinate
    referenceseqid-variant-coordinate
    referenceseqid-window-coordinate
    code-value-concept
    code-value-date
    code-value-quantity
    code-value-string
    combo-code-value-concept
    combo-code-value-quantity
    component-code-value-concept
    component-code-value-quantity
    ...
    context-type-quantity
    context-type-value
    

    Handy Command to get Duplicate Search Parameters

  • Release 4.0.1 – Lessons Learned in Development

    My team just released IBM FHIR Server 4.0.1.

    A few things, I learned:

    To Reset a Tag

    git tag -d 4.0.1
    git push origin --delete 4.0.1
    git tag 4.0.1
    git push --tags
    

    End to End Windows Tests

    I used a powershell and added an archive step.

    # Create the Zip File
    $compress = @{
    LiteralPath= $pre_it_logs
    CompressionLevel = 'Fastest'
    DestinationPath = $zip_file
    }
    
    Compress-Archive @compress -Force
    

    Ignoring Self-Signed Certs with Powershell

    Also a step to test our server, but hit an issue with the certificate
    
    # Ignore Self Signed Certificates and use TLS
    # Reference https://stackoverflow.com/a/46254549/1873438
    $AllProtocols = [System.Net.SecurityProtocolType]'Ssl3,Tls,Tls11,Tls12'
    [System.Net.ServicePointManager]::SecurityProtocol = $AllProtocols
    Add-Type -TypeDefinition @"
    using System.Net;
    using System.Security.Cryptography.X509Certificates;
    public class TrustAllCertsPolicy : ICertificatePolicy {
    public bool CheckValidationResult(
    ServicePoint srvPoint, X509Certificate certificate,
    WebRequest request, int certificateProblem) {
    return true;
    }
    }
    "@
    $AllProtocols = [System.Net.SecurityProtocolType]'Ssl3,Tls,Tls11,Tls12'
    [System.Net.ServicePointManager]::SecurityProtocol = $AllProtocols
    [System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
    

    Break it down into individual parts

    Release Tricks and Tips break it down into different workflows.