Author: Paul

  • DockerHub API to Get Statistics

    I had to gather statistics for my team’s repo. Here is a small recipe to get the pull count for a specific repository .

    1. Setup the Bearer Token.
    export DOCKER_USERNAME="prb112"
    export DOCKER_PASSWORD="<<>>"
    
    export TOKEN=$(curl -s -H "Content-Type: application/json" \
       -X POST -d '{"username": "'${DOCKER_USERNAME}'", "password": "'${DOCKER_PASSWORD}'"}' \
       https://hub.docker.com/v2/users/login/ | jq -r .token)
    
    1. Pull the stats:
    curl -L -H "Authorization: Bearer $TOKEN" \
        https://hub.docker.com/v2/repositories/ibmcom/ibm-fhir-server \
        | jq -r '.pull_count'
    574725
    

    Thanks to Arthur Koziel’s Blog

  • Using the HL7 FHIR® Da Vinci Health Record Exchange $member-match operation in IBM FHIR Server

    HL7 FHIR® Da Vinci Health Record Exchange (HREX) is an FHIR Implementation Guide at version 0.2.0 – STU R1 – 2nd ballot. The HREX Implementation Guide is a foundational guide for all of the Da Vinci guides which support US payer, provider, member and HIPAA covered entity data exchange. The guide defines "FHIR profiles, operations" and depends on HL7 FHIR® US Core Implementation Guide STU3 Release 3.1.0. In an issue, I implemented this profile and operation.

    As members (Patient) move from one plan (Coverage) to another plan (Coverage) or provider (Provider). To faciliates this exchange across boundaries, HREX introduces the $member-match operation which allows one health plan to retrieve a unique identifier for a member from another health plan using a member’s demographic and coverage information. This identifier can then be used to perform subsequent queries and operations. Members implementing a deterministic match require a match on member id or subscriber id at a minimum.

    The IBM FHIR Server team has implemented the HREX Implementation Guide and Operation as two modules: fhir-ig-davinci-hrex HREX 0.2.0 Implementation Guide and fhir-operation-member-match. The operation depends on fhir-ig-us-core US Core 3.1.1. Note, in the main branch the fhir-ig-us-core supports 3.1.1 and 4.0.0. These three modules are to be released to Maven Central when the next version is tagged.

    The $member-match operation executes on the Resource Type – Patient/$member-match. The operation implements the IBM FHIR Server Extended Operation framework using the Java Service Loader.

    operation-framework

    The $member-match provides a default strategy to execute strategy executes a series of Searches on the local FHIR Server to find a Patient on the system with a Patient and Coverage (to-match). The strategy is extensible by extending the strategy.

    MemberMatch Framework

    If the default strategy is not used, the Java Service Loader must be used by the new strategy. To register a JAR, META-INF/services/com.ibm.fhir.operation.davinci.hrex.provider.strategy.MemberMatchStrategy the file must point to the package and class that implements MemberMatchStrategy. Alternatively, AbstractMemberMatch or DefaultMemberMatchStrategy may be used as a starting point.

    For implementers, there is an existing AbstractMemberMatch which provides a template and series of hooks to extend:

    MemberMatchResult implements a light-weight response which gets translated to Output Parameters or OperationOutcomes if there is no match or multiple matches.

    More advanced processing of the input and validation is shown in DefaultMemberMatchStrategy which processes the input resources to generate SearchParameter values to query the local IBM FHIR Server.

    It’s highly recommended to extend the default implementation and override the getMemberMatchIdentifier for the strategy:

    The $member-match operation is configured for each tenant using the respective fhir-server-config.json. The configuration is rooted under the path fhirServer/operations/membermatch.

    Name Default Description
    enabled true Enables or Disable the MemberMatch operation for the tenant
    strategy default The key used to identify the MemberMatchStrategy that is loaded using the Java Service Loader
    extendedProps true Used by custom MemberMatchStrategy implementations
    {
        "__comment": "",
        "fhirServer": {
            "operations": {
                "membermatch": {
                    "enabled": true,
                    "strategy": "default",
                    "extendedProps": {
                        "a": "b"
                    }
                }
            }
        }
    }
    

    Recipe

    1. Prior to 4.10.0, build the Maven Projects and the Docker Build. You should see [INFO] BUILD SUCCESS after each Maven build, and docker.io/ibmcom/ibm-fhir-server:latest when the Docker build is successful.
    mvn clean install -f fhir-examples -B -DskipTests -ntp
    mvn clean install -f fhir-parent -B -DskipTests -ntp
    docker build -t ibmcom/ibm-fhir-server:latest fhir-install
    
    1. Create a temporary directory for the dependencies that we’ll mount to userlib/, so it looks at:
    userlib\
        fhir-ig-us-core-4.10.0.jar
        fhir-ig-davinci-hrex-4.10.0.jar
        fhir-operation-member-match-4.10.0.jar
    
    export WORKSPACE=~/git/wffh/2021/fhir
    mkdir -p ${WORKSPACE}/tmp/userlib
    cp -p conformance/fhir-ig-davinci-hrex/target/fhir-ig-davinci-hrex-4.10.0-SNAPSHOT.jar ${WORKSPACE}/tmp/userlib/
    cp -p conformance/fhir-ig-us-core/target/fhir-ig-us-core-4.10.0-SNAPSHOT.jar ${WORKSPACE}/tmp/userlib/
    cp -p operation/fhir-operation-member-match/target/fhir-operation-member-match-4.10.0-SNAPSHOT.jar ${WORKSPACE}/tmp/userlib/
    

    Note, the use of snapshot as these are not yet released.

    1. Download the fhir-server-config.json.
    curl -L -o fhir-server-config.json \
        https://raw.githubusercontent.com/IBM/FHIR/main/fhir-server/liberty-config/config/default/fhir-server-config.json
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  8423  100  8423    0     0  40495      0 --:--:-- --:--:-- --:--:-- 40301
    
    1. Start the Docker container, and capture the container id. It’s going to take a few moments to start up as it lays down the test database.
    docker run -d -p 9443:9443 -e BOOTSTRAP_DB=true \
      -v $(pwd)/fhir-server-config.json:/config/config/default/fhir-server-config.json \
      -v ${WORKSPACE}/tmp/userlib:/config/userlib/ \
      ibmcom/ibm-fhir-server:latest
    4334334a3a6ad395c4b600e14c8563d7b8a652de1d3fdf14bc8aad9e6682cc02
    
    1. Check the logs until you see:
    docker logs 4334334a3a6ad395c4b600e14c8563d7b8a652de1d3fdf14bc8aad9e6682cc02
    ...
    [6/16/21, 15:31:34:533 UTC] 0000002a FeatureManage A   CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 17.665 seconds.
    
    1. Download and update the Sample Data
    curl -L 'https://raw.githubusercontent.com/IBM/FHIR/main/conformance/fhir-ig-davinci-hrex/src/test/resources/JSON/020/Parameters-member-match-in.json' \
    -o Parameters-member-match-in.json
    
    1. Split the resource out from the sample:
    cat Parameters-member-match-in.json | jq -r '.parameter[0].resource' > Patient.json
    cat Parameters-member-match-in.json | jq -r '.parameter[1].resource' > Coverage.json
    
    1. Load the Sample Data bundle to the IBM FHIR Server
    curl -k --location --request PUT 'https://localhost:9443/fhir-server/api/v4/Patient/1' \
    --header 'Content-Type: application/fhir+json' \
    --header 'Prefer: return=representation' \
    --user "fhiruser:${DUMMY_PASSWORD}" \
    --data-binary  "@Patient.json"
    
    curl -k --location --request PUT 'https://localhost:9443/fhir-server/api/v4/Coverage/9876B1' \
    --header 'Content-Type: application/fhir+json' \
    --header 'Prefer: return=representation' \
    --user "fhiruser:${DUMMY_PASSWORD}" \
    --data-binary  "@Coverage.json"
    

    Note, DUMMY_PASSWORD should be previously set to your server’s password.

    1. Execute the Member Match
    curl -k --location --request POST 'https://localhost:9443/fhir-server/api/v4/Patient/$member-match' \
    --header 'Content-Type: application/fhir+json' \
    --header 'Prefer: return=representation' \
    --user "fhiruser:${DUMMY_PASSWORD}" \
    --data-binary  "@Parameters-member-match-in.json" -o response.json
    

    When you execute the operation, it runs two visitors across the Parameters input to generate searches against the persistence store:

    • DefaultMemberMatchStrategy.MemberMatchPatientSearchCompiler – Enables the Processing of a Patient Resource into a MultivaluedMap, which is subsequently used for the Search Operation. Note there are no SearchParameters for us-core-race, us-core-ethnicity, us-core-birthsex these elements in the US Core Patient profile. The following fields are combined in a Search for the Patient:

        - Patient.identifier
        - Patient.name
        - Patient.telecom
        - Patient.gender
        - Patient.birthDate
        - Patient.address
        - Patient.communication
      
    • DefaultMemberMatchStrategy.MemberMatchCovergeSearchCompiler Coverage is a bit unique here. It’s the CoverageToMatch – details of prior health plan coverage provided by the member, typically from their health plan coverage card and has dubious provenance. The following fields are combined in a Search for the Coverage.

        - Coverage.identifier
        - Coverage.beneficiary
        - Coverage.payor
        - Coverage.subscriber
        - Coverage.subscriberId
      

    Best wishes with MemberMatch.

  • Checking fillfactor for Postgres Tables

    My teammate implemented Adjust PostgreSQL fillfactor for tables involving updates #1834, which adjusts the amount of data in each storefile.

    Per Cybertec, fillfactor is important to "INSERT operations pack table pages only to the indicated percentage; the remaining space on each page is reserved for updating rows on that page. This gives UPDATE a chance to place the updated copy of a row on the same page as the original, which is more efficient than placing it on a different page." Link as such my teammate implemented in a PR a change to adjust the fillfactor to co-locate INSERT/UPDATES into the same space.

    Query

    If you want to check your fillfactor settings, you can can check the pg_class admin table to see your settings using the following scripts:

    SELECT 
    	pc.relname as "Table Name", 
    	pc.reloptions As "Settings on Table",
    	pc.relkind as "Table Type"
    FROM pg_class AS pc
    INNER JOIN pg_namespace AS pns 
    	ON pns.oid = pc.relnamespace
    WHERE pns.nspname = 'test1234'
    	AND pc.relkind = 'r';
    

    Note

    1. relkind represents the object type char r is a table. A good reference is the following snippet: relkind char r = ordinary table, i = index, S = sequence, v = view, m = materialized view, c = composite type, t = TOAST table, f = foreign table
    2. nspname is the schema you are checking for the fillfactor values.

    Results

    You see the value:

    basic_resources,{autovacuum_vacuum_scale_factor=0.01,autovacuum_vacuum_threshold=1000,autovacuum_vacuum_cost_limit=2000,fillfactor=90},'r'
    

    References

  • GitHub Action Workflow Tips

    I went through a Knowledge Transfer to a teammate who is implementing GitHub Actions and workflows in their repository. My team has been working with GitHub actions since they became available to developers. You can see my team’s workflows at https://github.com/IBM/FHIR/tree/main/.github/workflows and our automation scripts at https://github.com/IBM/FHIR/tree/main/build

    Here are my tips:

    Pushing Changes to GH Pages

    If you need to push changes back to GitHub, I recommend you checkout your code to a working sub folder and build in that subfolder, and copy the artifacts back to another subfolder and then push those changes (after a git add and commit with signature back to git)

    Triggering GH Pages Build

    In my build and release process, I generate my own website artifacts. I then need to call the API to trigger the GH Pages workflow as it is not automatically triggered by pushing the artifacts directly to the gh-pages branch. This trick starts the deployment of the branch to the gh-pages environment. It uses curl and the git hubs api.

    Grabbing the Current Tag

    I found this helpful to grab the tag and inject it into the Git Hub Environment variables in subsequent Workflow Job steps.

    Conditionally Skip based on a Label

    You should be able to skip your workflow at any given point, and you can add a conditional to skip, for instance ci-skip which should be a label in your repo.

    Capture your logs and Upload no matter what

    Workflows are designed to skip dependent steps on failure, Step B fails because Step A failed. It’s worth adding at the end of your workflow a step to gather any debug logs and pack them up, upload in all conditions.

    The condition is set with if: always().

    Lock your ‘uses’ workflow versions

    Lock in your workflow’s uses on a specific version. For instance, you can lock in on action/upload-artifact or action/checkout, and use the organization/repository to check the documentation on GitHub. Here are some key Actions and the links to their Repos.

    action/checkoutcheckout
    action/upload-artifactupload artifacts
  • Protected: IBM FHIR® Server – v1.2.0 is now released

    This content is password protected. To view it please enter your password below:

  • Job and Bulk Data APIs

    Here are some short cut APIs for Open Liberty API and IBM FHIR Server’s batch feature.

  • IBM Digital Developer Conference: Hybrid Cloud – Integrating Healthcare Data in a Serverless World

    My lab is now live and available on the IBM Digital Developer Conference. In my session, developers integrate a healthcare data application using IBM FHIR Server with OpenShift serverless, to create and respond to actual healthcare scenarios.

    The lab materials Link to the lab material https://prb112.github.io/healthcare-serverless/ and you need to sign up for the lab using the following instructions.

    Signing up for the Lab

    To get access to the lab environment, follow the following instructions:
    1. Get added to the IBM Cloud account “DEGCLOUD” using the following app:

    https://account-invite.mybluemix.net/

    – Enter Lab key: welcome

    – Enter IBM ID: the email you used to sign up for the Digital Developer Conference

    2. You will then get an invite message in your email which you must accept to continue. It doesn’t matter if you see an “oops” message here.

    3. Then, you can use the following app to get access to a cluster.

    – Open Link https://ddc-healthcare-lab.mybluemix.net

    – Enter Lab key: oslab

    – Enter IBM ID: The email you used to create your IBM Cloud account

    To jump right to the session you can go right to the IBM website https://developer.ibm.com/conferences/digital-developer-conference-hybrid-cloud/track-5-labs/s2-healthcare-data-serverless/

    Video of the Walk Through

  • Digital Developer Conference – Hybrid Cloud: Integrating Healthcare Data in a Serverless World

    Recently I developed and presented this lab… which gets released in late September 2021.

    In this lab, developers integrate a healthcare data application using IBM FHIR Server with Red Hat OpenShift Serverless to create and respond to a healthcare scenario.

    This lab is a companion to the session Integrating Healthcare Data in a Serverless World at Digital Developer Conference – Hybrid Cloud.

    The content for this lab can be found at https://ibm.biz/ibm-fhir-server-healthcare-serverless.

    Have fun! Enjoy… Ask Questions… I’m here to help.

  • Playing with buildah and ubi-micro: Part 1

    buildah is an intriguing open source tool to build of Open Container Initiative (OCI) container images using a scripted approach versus a traditional Dockerfile. It’s fascinating and I’ve started to use podman and buildah to build my project’s images.

    I picked ubi-micro as my startingn point. Per Red Hat, ubi-microis the smallest possible image excludinng the package manager and all of its dependencies which are normally included in a container image. This approach is an alternative to the current release of the IBM FHIR Server image. The following only documents my first stages with Java testing.

    1. On Fedora, install the prerequisites.
    # sudo dnf install buildah -y
    Last metadata expiration check: 0:23:36 ago on Thu 02 Sep 2021 10:06:55 AM EDT.
    Dependencies resolved.
    =====================================================================================================================================================================
     Package                               Architecture                         Version                                      Repository                             Size
    =====================================================================================================================================================================
    Installing:
     buildah                               x86_64                               1.21.4-5.fc33                                updates                               7.9 M
    
    Transaction Summary
    =====================================================================================================================================================================
    Install  1 Package
    
    Total download size: 7.9 M
    Installed size: 29 M
    Downloading Packages:
    buildah-1.21.4-5.fc33.x86_64.rpm                                                                                                     7.2 MB/s | 7.9 MB     00:01
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Total                                                                                                                                6.2 MB/s | 7.9 MB     00:01
    Running transaction check
    Transaction check succeeded.
    Running transaction test
    Transaction test succeeded.
    Running transaction
      Preparing        :                                                                                                                                             1/1
      Installing       : buildah-1.21.4-5.fc33.x86_64                                                                                                                1/1
      Running scriptlet: buildah-1.21.4-5.fc33.x86_64                                                                                                                1/1
      Verifying        : buildah-1.21.4-5.fc33.x86_64                                                                                                                1/1
    
    Installed:
      buildah-1.21.4-5.fc33.x86_64
    
    Complete!
    
    1. Start the new image
    # microcontainer=$(buildah from registry.access.redhat.com/ubi8/ubi-micro)
    Trying to pull registry.access.redhat.com/ubi8/ubi-micro:latest...
    Getting image source signatures
    Copying blob 4f4fb700ef54 done
    Copying blob 098a109c8679 done
    Copying config c5ba898d36 done
    Writing manifest to image destination
    Storing signatures
    
    1. Confirm the container name.
    # echo $microcontainer
    ubi-micro-working-container
    
    1. Mount the layer locally and display the path.
    # micromount=$(buildah mount $microcontainer)
    # echo $micromount
    /var/lib/containers/storage/overlay/14c524d6a5ef0e94887bc52685dbe911b40a5a9e39a6df00dc3b02e5f5ad7796/merged
    
    1. Setup the AdoptOpennJdk repository.
    cat <<'EOF' > $micromount/etc/yum.repos.d/adoptopenjdk.repo
    [AdoptOpenJDK]
    name=AdoptOpenJDK
    baseurl=http://adoptopenjdk.jfrog.io/adoptopenjdk/rpm/rhel/8/$basearch
    enabled=1
    gpgcheck=1
    gpgkey=https://adoptopenjdk.jfrog.io/adoptopenjdk/api/gpg/key/public
    EOF
    
    1. Install to micromount without any ancillary dependencies.
    yum install \
        --installroot $micromount \
        --releasever 8 \
        --setopt install_weak_deps=false \
        --nodocs -y \
        adoptopenjdk-11-openj9xl.x86_64
    

    Results in:

    ------------------------------------------------------------------------------------------------------------------------------------
    Total                                                                                               8.9 MB/s | 193 MB     00:21
    warning: Found bdb Packages database while attempting sqlite backend: using bdb backend.
    warning: /var/lib/containers/storage/overlay/14c524d6a5ef0e94887bc52685dbe911b40a5a9e39a6df00dc3b02e5f5ad7796/merged/var/cache/dnf/AdoptOpenJDK-096a01411439d076/packages/adoptopenjdk-11-openj9xl-11.0.10+9.openj9-0.24.0-3.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 74885c03: NOKEY
    AdoptOpenJDK                                                                                         13 kB/s | 3.1 kB     00:00
    warning: Found bdb Packages database while attempting sqlite backend: using bdb backend.
    Importing GPG key 0x74885C03:
     Userid     : "AdoptOpenJDK (used for publishing RPM and DEB files) <adoptopenjdk@gmail.com>"
     Fingerprint: 8ED1 7AF5 D7E6 75EB 3EE3 BCE9 8AC3 B291 7488 5C03
     From       : https://adoptopenjdk.jfrog.io/adoptopenjdk/api/gpg/key/public
    
    1. Clean up the dependencies
    # yum clean all \
     --installroot $micromount
    warning: Found bdb Packages database while attempting sqlite backend: using bdb backend.
    61 files removed
    
    1. Unmount the container
    buildah umount $microcontainer
    
    1. Coommit the image
    buildah commit $microcontainer ubi-micro-java
    
    1. Confirm the image
    # buildah images
    REPOSITORY                                  TAG        IMAGE ID       CREATED          SIZE
    localhost/ubi-micro-java                    latest     334404b8ebf2   22 seconds ago   43 MB
    

    It’s about 40M smaller than the ubi-minimal as it has no docs and ancillary dependencies.

    Tip: Starting with the IBM FHIR Server

    To start with the IBM FHIR Server image, you can use:

    buildah from --pull docker.io/ibmcom/ibm-fhir-server:latest
    
    [root@localhost ~]# buildah from --pull docker.io/ibmcom/ibm-fhir-server:latest
    Trying to pull docker.io/ibmcom/ibm-fhir-server:latest...
    Getting image source signatures
    Copying blob e2bef77118c7 done
    Copying blob 45cc8b7f2b43 done
    Copying blob 5627e846e80f done
    Copying blob 5f6bf015319e done
    Copying blob 87212cfd39ea done
    Copying blob b89ea354ae59 done
    Copying blob 4a939b72e1c6 done
    Copying blob d3cbf41efb4e done
    Copying blob 4feff1abc28e done
    Copying blob 9ff4465d271b done
    Copying blob 5e41012b4001 done
    Copying blob 410af8b678f6 done
    Copying blob 2f26dc40d01f done
    Copying blob 1415c9c2e161 done
    Copying blob e374de62001e done
    Copying blob 94d978ce0b1f done
    Copying blob 1fabae8675b6 done
    Copying blob 7b088cbebf16 done
    Copying blob 4167c1ebbd85 done
    Copying config 637552c186 done
    Writing manifest to image destination
    Storing signatures
    ibm-fhir-server-working-container
    

    Tip: Pullinng Fedora

    If you need to use Fedora, you can use fedora-minimal.

    # buildah from --pull registry.fedoraproject.org/fedora-minimal
    

    To remove the image

    $ podman image rm registry.fedoraproject.org/fedora-minimal:34
    

    Tip: Runnning with SELINUX

    If you are running with SELINUX, you should set specific selinux permissions.

    1. set the permission
    $ setsebool -P container_manage_cgroup 1
    
    1. Confirm the permission
    $ getsebool container_manage_cgroup
    container_manage_cgroup --> on
    

    References

  • Recipe: Azure Postgres with IBM FHIR Server Bulk Data

    One of the prerequisites for setting up IBM FHIR Server Bulk Data is setting up max_prepared_transactions since the IBM FHIR Server leverages Open Liberty Java Batch which uses an XA Transaction.

    If you are using Azure, here are the steps for updating your Postgres resource.

    Navigate to the Azure Portal

    Find your Postgres resource

    Update your Server Parameters max_prepared_transactions to 200 (anything non-zero is recommended to enable XA)

    Click Save

    Click Overview

    Click Restart

    Click On Activity Log

    Wait until Postgres is restarted

    Restart your IBM FHIR Server, and you are ready to use the Bulk Data feature.

    If you don’t do the setup, you’ll see a log like the following:

    [9/2/21, 1:49:38:257 UTC] [step1 partition0] com.ibm.fhir.bulkdata.jbatch.listener.StepChunkListener        StepChunkListener: job[bulkexportfastjob/8/15] --- javax.transaction.RollbackException
                                     com.ibm.jbatch.container.exception.TransactionManagementException: javax.transaction.RollbackException
            at com.ibm.jbatch.container.transaction.impl.JTAUserTransactionAdapter.commit(JTAUserTransactionAdapter.java:108)
            at com.ibm.jbatch.container.controller.impl.ChunkStepControllerImpl.invokeChunk(ChunkStepControllerImpl.java:656)
            at com.ibm.jbatch.container.controller.impl.ChunkStepControllerImpl.invokeCoreStep(ChunkStepControllerImpl.java:795)
            at com.ibm.jbatch.container.controller.impl.BaseStepControllerImpl.execute(BaseStepControllerImpl.java:295)
            at com.ibm.jbatch.container.controller.impl.ExecutionTransitioner.doExecutionLoop(ExecutionTransitioner.java:118)
            at com.ibm.jbatch.container.controller.impl.WorkUnitThreadControllerImpl.executeCoreTransitionLoop(WorkUnitThreadControllerImpl.java:96)
            at com.ibm.jbatch.container.controller.impl.WorkUnitThreadControllerImpl.executeWorkUnit(WorkUnitThreadControllerImpl.java:178)
            at com.ibm.jbatch.container.controller.impl.WorkUnitThreadControllerImpl$AbstractControllerHelper.runExecutionOnThread(WorkUnitThreadControllerImpl.java:503)
            at com.ibm.jbatch.container.controller.impl.WorkUnitThreadControllerImpl.runExecutionOnThread(WorkUnitThreadControllerImpl.java:92)
            at com.ibm.jbatch.container.util.BatchWorkUnit.run(BatchWorkUnit.java:113)
            at com.ibm.ws.context.service.serializable.ContextualRunnable.run(ContextualRunnable.java:79)
            at com.ibm.ws.threading.internal.ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:238)
            at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
            at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
            at java.base/java.lang.Thread.run(Thread.java:866)
    Caused by: javax.transaction.RollbackException
            at com.ibm.tx.jta.impl.TransactionImpl.stage3CommitProcessing(TransactionImpl.java:978)
            at com.ibm.tx.jta.impl.TransactionImpl.processCommit(TransactionImpl.java:778)
            at com.ibm.tx.jta.impl.TransactionImpl.commit(TransactionImpl.java:711)
            at com.ibm.tx.jta.impl.TranManagerImpl.commit(TranManagerImpl.java:165)
            at com.ibm.tx.jta.impl.TranManagerSet.commit(TranManagerSet.java:113)
            at com.ibm.tx.jta.impl.UserTransactionImpl.commit(UserTransactionImpl.java:162)
            at com.ibm.tx.jta.embeddable.impl.EmbeddableUserTransactionImpl.commit(EmbeddableUserTransactionImpl.java:101)
            at com.ibm.ws.transaction.services.UserTransactionService.commit(UserTransactionService.java:72)
            at com.ibm.jbatch.container.transaction.impl.JTAUserTransactionAdapter.commit(JTAUserTransactionAdapter.java:101)

    Go back and enable max_prepared_transactions

    References

    Azure Docs