As of IBM FHIR Server 4.10.2…
A colleague of mine is entering into the depths of the IBM FHIR Server’s Bulk Data feature. Each tenant in the IBM FHIR Server may specify multiple storageProviders. The default tenant is assumed, unless specified with the Http Headers X-FHIR-BULKDATA-PROVIDER and X-FHIR-BULKDATA-PROVIDER-OUTCOME. Each tenant’s configuration may mix the different providers, however each provider is only of a single type. For instance, minio is aws-s3 and default is file and az is azure-blob.
Note, type http is only applicable to $import operations. Export is only supported with s3, azure-blob and file.
File Storage Provider Configuration
The file storage provider uses a directory local to the IBM FHIR Server. The fileBase is an absolute path that must exist. Each import inputUrl is a relative path from the fileBase. The File Provider is available for both import and export. Authentication is not supported.
{
"__comment": "IBM FHIR Server BulkData - File Storage configuration",
"fhirServer": {
"bulkdata": {
"__comment" : "The other bulkdata configuration elements are skipped",
"storageProviders": {
"default": {
"type": "file",
"fileBase": "/opt/wlp/usr/servers/fhir-server/output",
"disableOperationOutcomes": true,
"duplicationCheck": false,
"validateResources": false,
"create": false
}
}
}
}
}
An example request is:
{
"resourceType": "Parameters",
"id": "30321130-5032-49fb-be54-9b8b82b2445a",
"parameter": [
{
"name": "inputSource",
"valueUri": "https://my-server/source-fhir-server"
},
{
"name": "inputFormat",
"valueString": "application/fhir+ndjson"
},
{
"name": "input",
"part": [
{
"name": "type",
"valueString": "AllergyIntolerance"
},
{
"name": "url",
"valueUrl": "r4_AllergyIntolerance.ndjson"
}
]
},
{
"name": "storageDetail",
"valueString": "file"
}
]
}
HTTPS Storage Provider Configuration
The http storage provider uses a set of validBaseUrls to confirm $import inputUrl is acceptable. The Https Provider is available for import. Authentication is not supported.
{
"__comment": "IBM FHIR Server BulkData - Https Storage configuration (Import only)",
"fhirServer": {
"bulkdata": {
"__comment" : "The other bulkdata configuration elements are skipped",
"storageProviders": {
"default": {
"type": "https",
"__comment": "The whitelist of valid base urls, you can always disable",
"validBaseUrls": [],
"disableBaseUrlValidation": true,
"__comment": "You can always direct to another provider",
"disableOperationOutcomes": true,
"duplicationCheck": false,
"validateResources": false
}
}
}
}
}
An example request is:
{
"resourceType": "Parameters",
"id": "30321130-5032-49fb-be54-9b8b82b2445a",
"parameter": [
{
"name": "inputSource",
"valueUri": "https://my-server/source-fhir-server"
},
{
"name": "inputFormat",
"valueString": "application/fhir+ndjson"
},
{
"name": "input",
"part": [
{
"name": "type",
"valueString": "AllergyIntolerance"
},
{
"name": "url",
"valueUrl": "https://validbaseurl.com/r4_AllergyIntolerance.ndjson"
}
]
},
{
"name": "storageDetail",
"valueString": "https"
}
]
}
Azure Storage Provider Configuration
The azure-blob storage provider uses a connection string from the Azure Blob configuration. The bucketName is the blob storage name. The azure-blob provider supports import and export. Authentication and configuration are built into the Connection string.
"__comment": "IBM FHIR Server BulkData - Azure Blob Storage configuration",
"fhirServer": {
"bulkdata": {
"__comment" : "The other bulkdata configuration elements are skipped",
"storageProviders": {
"default": {
"type": "azure-blob",
"bucketName": "fhirtest",
"auth": {
"type": "connection",
"connection": "DefaultEndpointsProtocol=https;AccountName=fhirdt;AccountKey=ABCDEF==;EndpointSuffix=core.windows.net"
},
"disableBaseUrlValidation": true,
"disableOperationOutcomes": true
}
}
}
}
}```
An example request is:
{
"resourceType": "Parameters",
"id": "30321130-5032-49fb-be54-9b8b82b2445a",
"parameter": [
{
"name": "inputSource",
"valueUri": "https://my-server/source-fhir-server"
},
{
"name": "inputFormat",
"valueString": "application/fhir+ndjson"
},
{
"name": "input",
"part": [
{
"name": "type",
"valueString": "AllergyIntolerance"
},
{
"name": "url",
"valueUrl": "r4_AllergyIntolerance.ndjson"
}
]
},
{
"name": "storageDetail",
"valueString": "azure-blob"
}
]
}
S3 Storage Provider Configuration
The aws-s3 storage provider supports import and export. The bucketName, location, auth style (hmac, iam), endpointInternal, endpointExternal are separate values in the configuration. Note, enableParquet is obsolete.
{
"__comment": "IBM FHIR Server BulkData - AWS/COS/Minio S3 Storage configuration",
"fhirServer": {
"bulkdata": {
"__comment" : "The other bulkdata configuration elements are skipped",
"storageProviders": {
"default": {
"type": "aws-s3",
"bucketName": "myfhirbucket",
"location": "us-east-2",
"endpointInternal": "https://s3.us-east-2.amazonaws.com",
"endpointExternal": "https://myfhirbucket.s3.us-east-2.amazonaws.com",
"auth": {
"type": "hmac",
"accessKeyId": "AKIAAAAF2TOAAATMAAAO",
"secretAccessKey": "mmmUVsqKzAAAAM0QDSxH9IiaGQAAA"
},
"enableParquet": false,
"disableBaseUrlValidation": true,
"exportPublic": false,
"disableOperationOutcomes": true,
"duplicationCheck": false,
"validateResources": false,
"create": false,
"presigned": true,
"accessType": "host"
}
}
}
}
}
An example request is:
{
"resourceType": "Parameters",
"id": "30321130-5032-49fb-be54-9b8b82b2445a",
"parameter": [
{
"name": "inputSource",
"valueUri": "https://my-server/source-fhir-server"
},
{
"name": "inputFormat",
"valueString": "application/fhir+ndjson"
},
{
"name": "input",
"part": [
{
"name": "type",
"valueString": "AllergyIntolerance"
},
{
"name": "url",
"valueUrl": "r4_AllergyIntolerance.ndjson"
}
]
},
{
"name": "storageDetail",
"valueString": "aws-s3"
}
]
}
Note, you can exchange aws-s3 and ibm-cos as the parameter.where(name=’storageDetail’). These are treated interchangeably.
There are lots of configurations that are possible, I hope this helps you.
