Contact Opala's
Documentation Team

Deleting Data

The FHIR delete External Link operation performs a "logical" delete, meaning the data is not physically removed from the database.

For example, suppose a Patient resource with ID 123 is created (via an HTTP POST /Patient and subsequently deleted (via an HTTP DELETE Patient/123). This will cause a second version of the Patient/123 resource to be created with version Patient/123/_history/2 that is marked as deleted. This patient will no longer appear in search results, and attempts to read the resource (using an HTTP GET Patient/123) will fail with an HTTP 410 Gone response.

The original content of the resource is not destroyed however. It can still be found using two FHIR operations:

Using a FHIR version-specific read: GET Patient/123/_history/1.
Using a FHIR instance-history: GET Patient/123/_history.

The HTTP 410 Gone responses includes a Location header which contains the fully qualified resource ID as well as the version ID. For example:

410 Gone

                    Location: http://example.org/fhir/Patient/123/_history/12

In this example, you can see that the deleted version of the resource is version 12. This means that the last non-deleted version is version 11 and this version can be accessed using a version-specific read to the following URL:

http://example.org/fhir/Patient/123/_history/11

Deletes and Referential Integrity

A common problem — frequently in test systems but sometimes in production systems, as well — is cleaning up batches of interdependent data.

For example, suppose you have a CDR containing a patient resource with the ID Patient/A. Suppose this CDR also contains Encounter/1 and Encounter/2, as well as Observation/3 and MedicationAdministration/4, and all of these resources have a reference to the resource Patient/A. We will call these resources the child resources.

If you try to DELETE Patient/A (using a standard FHIR DELETE operation), this request will be denied, assuming that there is a resource link between the child resources and the patient (a resource link is a field that is indexed with an active SearchParameter).

This will result in an HTTP 409 Conflict with a response like the following:

{
    "resourceType": "OperationOutcome",
    "issue": [
      {
        "severity": "error",
        "code": "processing",
        "diagnostics": "Unable to delete Patient/1 because at least one resource has a reference to this resource. First reference
found was resource Observation/2 in path Observation.subject.where(resolve() is Patient)"
      }
    ]
}

If you want to force a delete of Patient/A, you have several options:

You can manually delete all of the child resources before trying to delete the Patient.
You can disable referential integrity checking in the CDR by changing the Enforce Referential Integrity on Delete setting on the FHIR Storage module. This means that any delete will be permitted as long as the user has appropriate permissions to actually perform the delete, even if other resources still have references left. This can be confusing for other users, since it leaves resources with invalid references, but there are valid cases for doing so.
You can use a transactional delete (see below).
You can use cascading deletes (see below).

Transactional Delete

The FHIR Transaction External Link operation can be used to delete multiple resources at the same time. This is useful if you have chains or collections of resources to delete at once, but also can be used to delete circular references.

To delete multiple resources in a transaction, POST a Bundle such as the following to the root of your FHIR endpoint.

{
    "resourceType": "Bundle",
    "type": "transaction",
    "entry": [
      {
        "request": {
          "method": "DELETE",
          "url": "Organization/1"
        }
      },
      {
        "request": {
          "method": "DELETE",
          "url": "Organization/2"
        }
      }
    ]
}

Referential Integrity

By default, Opala will block the deletion of a resource if any other resources have indexed references to the resource being deleted.

For example, suppose the resource Patient/123 has been saved in the repository, and a second resource Observation/456 is then saved as well, where the Observation.subject reference is a reference to the Patient. In this situation, attempts to delete Patient/123 will be blocked unless resources with references to this resource are deleted first (or are deleted as a part of the same transaction in the case of a transactional delete (see above)).

Disabling Checking Globally

You can disable this referential integrity checking globally using the Enforce Referential Integrity on Write configuration property on the FHIR Storage module.

Disabling Checking Selectively

You can also selectively disable referential integrity checking by configuring the system to disable referential integrity checking for specific paths. This is done using the Enforce Referential Integrity on Write setting.

The value for this setting is a set of one or more FHIRPath expressions, each one on a new line. For example, this setting could be set to Observation.subject in order to allow the deletion described above to proceed.

Cascading Deletes

With cascading deletes enabled, a user can perform the delete on Patient/A (per the example above) and all of the child resources will be deleted as well.

In order to perform a cascading delete, three things must occur:

The Cascading Deletes Enabled setting must be enabled on the FHIR Storage module.
The user performing the operation must have the FHIR_DELETE_CASCADE_ALLOWED permission, as well as a specific permission allowing the child resource to be deleted.
For example, you might grant the user the FHIR_DELETE_CASCADE_ALLOWED and FHIR_ALL_DELETE permissions.
To perform a cascaded delete, the client HTTP request must include either a special URL parameter (_cascade) or a special header to indicate that a cascading delete is desired.

The following example shows a delete using a URL parameter:

DELETE /Patient/123?_cascade=delete

The following example shows a delete using an HTTP header:

DELETE /Patient/123

                            X-Cascade: delete

Delete Child Resource Count

Depending on the number of child resources linked to a given resource, the cascading delete can potentially take several seconds or even minutes to complete, and can consume considerable memory and processing resources during that time, which may have unexpected impacts on Opala. To minimize the duration and impact of cascading deletes, the actual deletes are performed in batches of a pre-configured size. If after completing 10 passes of batched deletes, there are still child resources remaining, the deletes will be rolled back and Opala will return an error similar to:

Requested delete operation stopped before all conflicts were handled. May need to increase the configured Maximum Delete Conflict Query Count.

If this type of error occurs after attempting a Cascading Delete request, the batch size can be increased by increasing the configured value of Delete Child Resource Count.

The $expunge Operation

This method requires specific permissions in order to use different features. See the table below for information on which permissions are needed.

In some cases, You want to truly delete data. This might be because it was entered in error and should not be seeen, because it represents a privacy concern to leave it in place, or because your solution does not require long term retention of stale data.

The $expunge operation can physically delete old versions of resources, deleted resources, or even all data in a database.

Important. This operation is globally disabled by default as it is potentially dangerous. Change the Expunge Operation Enabled setting to enable it.

Input Parameters

Name	Type	Usage	Default	Permission Required
`Limit`	Number	This parameter specifies the maximum number of entries (resource versions and/or resources) that will be deleted in a single batch before exiting.	1000	N/A
`expungeDeletedResources`	Token (boolean value)	If set to `true`, deleted resources will be expunged (including all previous versions of the resource).	false	FHIR_EXPUNGE_DELETED
`expungePreviousVersions`	Token (boolean value)	If set to `true`, non-current versions of resources will be expunged.	false	FHIR_EXPUNGE_PREVIOUS_VERSIONS
`expungeEverything`	Token (boolean value)	If set to `true`, current versions of resources will also be expunged.	false	FHIR_EXPUNGE_EVERYTHING

Instance-Level Expunge

The $expunge operation can be invoked against a single resource instance, or even an individual version of a resource instance. If invoked at the instance level (shown below), previous versions of the resource may be deleted (if expungePreviousVersions is set to true) and the current version may be deleted (if the resource is deleted and expungeDeletedResources is set to true).

POST [base]/Patient/123/$expunge
Content-Type: application/fhir+json
                                
{
  "resourceType": "Parameters",
  "parameter": [
    {
      "name": "limit",
      "valueInteger": 1000
    },{
      "name": "expungeDeletedResources",
      "valueBoolean": true
    },{
      "name": "expungePreviousVersions",
      "valueBoolean": true
    }
  ]
}

The $expunge operation can also be invoked at the instance version level (shown below). This can be used to expunge an individual version of a resource without affecting other versions.

POST [base]/Patient/123/_history/2/$expunge
Content-Type: application/fhir+json
                                
{
  "resourceType": "Parameters",
  "parameter": [
    {
      "name": "expungeDeletedResources",
      "valueBoolean": true
    }
  ]
}

Type-Level Expunge

The $expunge operation can be invoked at the type level. In this mode, all resources of a given type will be processed with the same rules as at the instance level.

POST [base]/Patient/$expunge
Content-Type: application/fhir+json
                                
{
  "resourceType": "Parameters",
  "parameter": [
    {
      "name": "expungeDeletedResources",
      "valueBoolean": true
    },{
      "name": "expungePreviousVersions",
      "valueBoolean": true
    }
  ]
}

System-Level Expunge

The $expunge operation can be invoked at the system level. In this mode, all resources on the server will be processed with the same rules as at the instance level.

POST [base]/$expunge
Content-Type: application/fhir+json
                                
{
  "resourceType": "Parameters",
  "parameter": [
    {
      "name": "expungeDeletedResources",
      "valueBoolean": true
    },{
      "name": "expungePreviousVersions",
      "valueBoolean": true
    }
  ]
}

Drop All Data

The following operation will delete all data, including non-deleted resources.

POST [base]/$expunge
Content-Type: application/fhir+json
                                
{
  "resourceType": "Parameters",
  "parameter": [
    {
      "name": "expungeEverything",
      "valueBoolean": true
    }
  ]
}

Delete Expunge

If you need to quickly delete all data associated with a set of resources — for example in a test environment — you can combine the DELETE and $expunge operations into a single step. Opala offers two ways to do this. You can either call a usual DELETE with a special parameter _expunge=true, or you can POST a $delete-expunge operation. Both of these ways result in starting a Delete Expunge Batch Job that deletes and expunges the requested details in the background.

In order to perform a Delete Expunge, three settings need to be enabled on the Storage Module:

Expunge Enabled
Delete Expunge Enabled
Allow Multiple Delete Enabled

DELETE with _expunge=true

A user must have both FHIR_DELETE_ALL_OF_TYPE permission and FHIR_EXPUNGE_EVERYTHING permission to DELETE with _expunge=true

Here is an example of performing delete expunge using the DELETE method:

DELETE [base]/Observation?status=cancelled&_expunge=true

$delete-expunge operation (since 2021.05.R02)

A user must have FHIR_DELETE_EXPUNGE, FHIR_DELETE_ALL_OF_TYPE permission and FHIR_EXPUNGE_EVERYTHING permission to call this operation.

Here is an example of performing delete expunge with a POST to /$delete-expunge:

POST [base]/$delete-expunge
Content-Type: application/fhir+json
                                    
{
  "resourceType": "Parameters",
  "parameter": [ {
    "name": "url",
    "valueString": "Observation?subject.active=false"
  }, {
    "name": "url",
    "valueString": "Patient/?active=false"
  }, {
    "name": "batchSize",
    "valueDecimal": 1000
  } ]
}

Delete Expunge Batch Job

The DELETE _expunge=true and $delete-expunge operations create the same type of batch job. Batch jobs are stopped and restarted on the Batch Job Management page.

Delete Expunge and Partitions

The $delete-expunge operation is partition aware. The operation is performed only on the partition that was included in the request and the job is only started if the user is allowed to access that partition.

Performance

The delete expunge batch job is optimized to delete the resource records as quickly as possible. This means usual checks (authorization callbacks, logging, auditing, etc.) are skipped when performing batch delete expunges. If you require normal checks, then you should use the normal DELETE followed by $expunge operation.

The only check that is made before deleting the resources is referential integrity: resources are not delete expunged if there are other resources that refer to them. The URLs to delete expunge need to be ordered so that the child resources are removed before the parent resources are removed.

The resources are removed in batches, with all database records associated to those resources deleted in a single transaction. The batch size indicates the number of resources to delete together in single transaction. So for example, if the batch size is set to 100 and there are 15 records per resource, that results in removing 1500 records in a single transaction. Larger batch sizes performs faster but require more memory.

If the DELETE ?_expunge=true syntax is used to trigger the delete expunge, then the batch size is determined by the value of Expunge Batch Size property.

Top