Creating Data

Transactions and Submitting Bundles

• If you POST to [baseUrl]/Bundle you are submitting the bundle for storage as-is. In other words, the bundle is stored as a Bundle, and the contents inside aren't looked at by the server (aside from any validation that is enabled). This mode is generally used to store Bundle resources with Bundle.type values such as document, and collection. In its default configuration, Opala will prohibit storing a Bundle with a type value of transaction or batch as this is generally a sign that the client is attempting to perform the operations described below but with an incorrect request URL.
  
  Naming: This is not a FHIR Transaction, but instead is a simple resource create where the resource happens to be a Bundle resource.
• If you POST to [baseUrl] and your Bundle has a Bundle.type value of transaction you are performing a FHIR transaction operation, meaning that all of the individual resources inside the bundle will be processed. It is also possible to include other REST operations such as searches in this kind of bundle. The processing works as an atomic unit, meaning that if anything fails (e.g., invalid data in an individual element) the entire thing will be rolled back.
  
  Naming: This operation is referred to as a FHIR Transaction operation.
• If you POST to [baseUrl] and your Bundle has a Bundle.type value of batch, the same processing as the transaction applies, except that individual operations are executed in individual database transactions, so an individual failure doesn't cause the entire operation to be rolled back. In this case, the response Bundle returned by the server will include status entries indicating the outome for the individual operations within. Note that the batch operation does require the entire Bundle to be valid FHIR at a minimum. This means that it can't have non-existent resource types in it, malformed datatypes, etc.
  
  Naming: This operation is referred to as a FHIR Batch operation.

Auto-Creating Reference Targets

Often when batch processing data from multiple sources, you will have data from one source that has references to data from other sources.

For example, a collection of Observation resources could be imported from a lab system data source at the same time that a collection of Patient resources is created from a patient administration data source. The Observation resources would have references to the Patient resources. Under ideal conditions, the Patient resource would process first and be present for the Observation to link to. In the real world however, often it is hard to control the order in which transactions occur, and so it might be possible for an Observation to be processed before its Patient. By default this would cause an error since the Observation would have an invalid reference, and nothing would be stored.

This page describes several strategies for solving this issue:

• Transaction With Conditional Create
• Auto-Create Placeholder Reference Targets
• Auto-Create Placeholder Reference Targets with Identifier

Transaction With Conditional Create

The semantics of a conditional create are roughly described as "use an existing resource that matches specific criteria if one exists (and do not modify that resource), or create a new one if not". The specific criteria in question can be any set of FHIR search parameters that could be used to otherwise locate the resource. The resource identifier field/search parameter is often used for this purpose, but other search parameters can also be used.

In a FHIR Transaction operation, an upsert is performed using a conditional create .

To do this, you create a Transaction Bundle with the following properties (an Observation being created with a reference to a Conditionally Created Patient is being used for this example):

{
    "resourceType": "Bundle",
    "type": "transaction",
    "entry": [ {
      "request": {
        "method": "POST",
        "url": "Observation"
      },
      "resource": {
        "resourceType": "Observation",
        "status": "final",
        "code": {
            "coding": [ {
            "system": "http://loinc.org",
            "code": "789-8",
            "display": "Erythrocytes [#/volume] in Blood by Automated count"
          } ]
        },
        "subject": {
          "reference": "urn:uuid:3bc44de3-069d-442d-829b-f3ef68cae371"
        },
        "valueQuantity": {
          "value": 4.12,
          "unit": "10 trillion/L",
              "system": "http://unitsofmeasure.org",
              "code": "10*12/L"
            }
          }
        },{
      "request": {
        "method": "POST",
        "url": "Patient",
        "ifNoneExist": "identifier=http://acme.org/mrns|12345"
      },
      "resource": {
        "resourceType": "Patient",
        "identifier": [ {
          "system": "http://acme.org/mrns",
          "value": "12345"
        } ],
        "name": [ {
          "family": "Jameson",
          "given": [ "J", "Jonah" ]
        } ],
        "gender": "male"
      }
    } ]
}

Auto-Create Placeholder Reference Targets

If the Auto-Create Placeholder Reference Targets setting is enabled in the FHIR Storage module configuration, it is possible to have the server automatically create an empty placeholder resource with a pre-assigned ID.

This technique is somewhat less complex than the example above, since it does not require a transaction bundle to be created. With this technique, the ID (not the identifier) of the target resource must be known. For example, the following payload could be POSTed to [baseUrl]/Observation, and would result in the creation of an empty resource with the ID Patient/ABC if one does not already exist. Note that if you want to be able to use this technique with purely numeric resource IDs you will also need to adjust the Client ID Mode.

  {
    "resourceType": "Observation",
    "status": "final",
    "code": {
      "coding": [ {
        "system": "http://loinc.org",
        "code": "789-8"
      } ]
    },
    "subject": {
      "reference": "Patient/ABC"
    },
    "valueQuantity": {
      "value": 4.12,
      "system": "http://unitsofmeasure.org",
      "code": "10*12/L"
    }
  }

Auto-Create Placeholder Reference Targets with Identifier

If the Auto-Create Placeholder Reference Targets setting is enabled in the FHIR Storage module configuration (as shown above), and the Allow Inline Match URL References Enabled setting is also enabled, you can refine the behavior shown above further.

In this case, it is possible to use an inline match URL instead of a hardcoded resource ID, and you can then achieve similar behavior to the Transaction Bundle use case.

Consider the following Observation being POSTed to /Observation.

{
    "resourceType": "Observation",
    "status": "final",
    "code": {
      "coding": [ {
        "system": "http://loinc.org",
        "code": "789-8"
      } ]
    },
    "subject": {
      "reference": "Patient?identifier=http://foo|1234",
      "identifier": {
        "system": "http://foo",
        "value": "1234"
      }
    },
    "valueQuantity": {
      "value": 4.12,
      "system": "http://unitsofmeasure.org",
      "code": "10*12/L"
    }
}

In this case, the reference will be treated as a local search (in this example, for a Patient with the identifier included in the inline match URL) and executed as such. If the search finds zero results, a new Patient resource will be created. If the inline match URL uses an identifier as it does here, Patient.identifier will be populated with the inline match URL's identifer system and value. The reference will then be automatically replaced with a reference to this new Patient. If the search finds one result, the reference will be automatically replaced with a reference to the found Patient and no placeholder reference target will be created. Note that we use Patient as an example, this applies to any reference targets that include an Identifier element.

Optionally, Reference.identifier can also provide an identifier. This is useful where the inline match URL uses a SearchParameter other than identifier. When Reference.identifier is provided, Patient.identifier may be populated with the system and value provided (see below). If Reference.identifier is provided, it will be persisted within the referencing resource. This is an important consideration because it exposes information from the reference target in the referencing resource, and it may not reflect later changes to the reference target.

Evaluating Inline Match URLs and Reference Identifiers

Opala uses the following logic to determine which identifiers get added to the newly created placeholder reference target:

• If only an inline match URL with identifier (e.g., Patient?identifier=http://foo|123) is provided, this identifier will be added to the auto-created placeholder reference target.
• If only Reference.identifier is provided, its value will be added to the auto-created placeholder reference target.
• If both an inline match URL with identifier and Reference.identifier are provided, and they include the same system and value, only one copy of that identifier will be added to the auto-created placeholder reference target. If they differ, both will be added.
• If neither an inline match URL with identifier nor Reference.identifier are provided, no identifier will be added to the auto-created placeholder reference target.

Updating Auto-Created Placeholder Reference Targets by Identifier

If neither an inline match URL with identifier nor Reference.identifier are provided such that the auto-created placeholder reference target does not include an identifier, the reference target cannot be subsequently updated using a conditional update . For this reason, it is recommended to always provide at least one identifier for the reference target where the resource ID is unknown.

Auto-Created Placeholder Extension

A placeholder reference target is just another instance of a given resource type (e.g., Patient). FHIR itself does not have a notion of a "placeholder." Such a resource is minimally populated depending on the mechanism by which it was created. It may have a client-assigned ID or it may have a server-assigned ID and one or more identifiers. To help identify such resources, every auto-created placeholder reference target is populated with the following extension:

"extension": [ {
    "url": "http://hapifhir.io/fhir/StructureDefinition/resource-placeholder",
    "valueBoolean": true
} ],

When the placeholder is subsequently updated, the extension only appears in the first version of the resource.

Searching for Auto-Created Resources

In all cases, when the Auto-Create Placeholder Reference Targets setting is enabled, Opala will automatically load custom SearchParameters into the repository that allow searching for any resources that were auto-created as placeholders. The following is an example of such a query:

GET http://localhost:8000/Patient?resource-placeholder=true`

In all cases, the code of the SearchParameter is resource-placeholder.

Auto-Creating Reference Targets and Enforcing Uniqueness

See the topic Enforcing Uniqueness in the Smile CDR documentation.

Mass Ingestion Mode

A common part of a new rollout involves an initial backload of data (or a regular periodic batch load).

In order to maximize performance during this kind of activity, a setting called Mass Ingestion Mode is available. When enabled, this setting tunes the system for maximum write performance by tweaking a number of internal functions:

• Internal cache sizes are increased and set to have longer timeouts, with cache space being prioritized for functions that improve write performance (as opposed to the default, which is a balance of read/write performance).
• When updating resources, the system avoids fetching the current contents of the resource by only performing a comparison of a SHA-256 hash of the contents. This avoids an expensive lookup for every update operation.

This setting is controlled using the Mass Ingestion Mode property of the FHIR Storage (RDBMS) module.