When organisations base their document management solution on SharePoint Online, there is the matter of archiving and at some point, disposition. While the disposition review in the Purview Compliance portal provides decent capabilities, disposition reviews may require a bit more kick. That’s where the value of Power Automate comes in.
Setting the stage
Consider a large-scale document management and archiving solution using SharePoint Online as the core. Several departments have their own document repository and a associated archive site. Documents are managed in “dossier”-document sets and leverage features like Document ID’s, versioning, sharing etc. There are also some Power Automate flows in place to support archiving and applying retention labels, but these are not in scope for this post.
The document sets also uses metadata columns as part of the information architecture. This includes a “Status” column that determines if a document set is “Open” or “Closed”. Metadata is inherited by the underlying documents within the document set.
When a dossier is completed, the owner changes the “Status” to “Closed”. A scheduled flow checks once a day for closed dossiers and moves them and the underlying content to the associated archive site. Once the dossier arrives, the metadata determines the retention label to apply. This is also done using a flow. Depending on the type of dossier, retention periods may vary. Some need to be kept forever, while others may need to be retained. This could be e.g., 5, 10, 20 or 50 years.
The challenge
While the retention period is covered by the retention label capabilities in the Purview Compliance portal, the challenge is what to do when the end of the retention period is reached. Each department potentially has their own process for reviewing content that is eligible for disposition.
While at first, the disposition review capability in the Purview Compliance portal appeared to be a proper fit, the differences in how departments handle disposition required a different approach.
The departments process the items up for disposition review a couple of times per year where they should only look at their own content.
Also, the Purview Compliance portal is built around document retention and does not cover the concept of a document set which is the way the users work.
The solution should cover that concept and also gather all information about the dossier document set to support the decisions on disposition or e.g., having content re-labelled. The disposition review panels do not show the complete set of information that plays a part in the process:
- Dossier name
- Library name
- Site URL
- Start of retention period (when label was applied)
- End of the retention period
- Applied label
- Label description
- Status
- Comments (to support the decisions).
Based on this, an alternate solution was required, combining Power Automate flow and with retention labels.
The solution
To have a complete fit with business requirements, each archive site would have their own disposition review list, which would be populated by a Power Automate flow that triggers at the end of the retention period for a specific label. This trigger is available through a license that includes premium connectors for Power Automate. The trigger outputs the following data from the expired document that can be used in the flow:
- Site address (URL)
- Site Id
- Folder path
- Library name
- Document name
- Item unique Id
- Existing label Id.
There is more data, but this is what is needed in the disposition review process.
So the flow is stared at the end of the retention period specified for the label. The flow needs to be created first with the end of retention period trigger as a prerequisite to link it to a retention label.
It then retrieves the data from the dossier through the expired document within. It does so by using the FolderPath-data that is output by the trigger. Using an expression, the document set title is retrieved from the folder path, considering that the document sets are always created at the root level of the document library.
split(triggerOutputs()?['body/FolderPath'], '/')[1]
The expression splits the path of the FolderPath-element and uses the 2nd element from the output, which is the document set name. This also works if the document set has the use of folders enabled.
Next, the flow checks the disposition review list on the site if there is an item there with the same name as the document set using the “Get list items” SharePoint-action.
This part is especially interesting. A document set usually contains several documents which inherit the applied retention label from the document set. So these documents all expire at the same time, which may cause the check for an existing document set in the disposition review list to fail.
The solution for this, is to reduce the “Degree of Parallelism” of the trigger to ‘1’.
If no items are found, the flow can proceed to get all data and create a list item.
Calling the Microsoft Graph
Next, we must get all the details for the applied retention label using the Microsoft Graph. This requires an Azure App registration with consent for the following Application permissions:
- RecordsManagement.Read.All
- RecordsManagement.ReadWrite
The latter seems a bit strange, as we are only reading, but it will not work without that.
The app will work with a client secret, client id and tenant id, which are all sensitive data. This data is best stored in an Azure Vault and can be retrieved using the “Get secret” action. This is also part of premium connectors and basically retrieves specified data from the vault so it can be securely used for the Graph call that gets the retention label data. With this information a “HTTP request”-action can be performed. To ensure that the sensitive data remains secure, the action is best configured with “Secure inputs” enabled in the settings.
The Graph call uses the existing label Id that is also available from the trigger. The output contains the data we need:
{
"@odata.context": "https://graph.microsoft.com/beta/$metadata#security/labels/retentionLabels/$entity",
"displayName": "Process a - Contract with exernal party",
"descriptionForAdmins": "Dispose after 10 years",
"descriptionForUsers": "Displose after 10 years",
"isInUse": false,
"retentionTrigger": "dateLabeled",
"behaviorDuringRetentionPeriod": "retain",
"actionAfterRetentionPeriod": "none",
"createdDateTime": "2023-06-28T14:00:00Z",
"lastModifiedDateTime": "2033-06-28T14:00:00Z",
"labelToBeApplied": "",
"defaultRecordBehavior": "startLocked",
"id": "{guid}",
"retentionDuration": {
"@odata.type": "#microsoft.graph.security.retentionDurationInDays",
"days": 3650
},
"createdBy": {
"user": {
"id": null,
"displayName": "MOD Administrator"
}
},
"lastModifiedBy": {
"user": {
"id": null,
"displayName": "MOD Administrator"
}
},
"dispositionReviewStages@odata.context": "https://graph.microsoft.com/beta/$metadata#security/labels/retentionLabels('{guid}')/dispositionReviewStages",
"dispositionReviewStages": []
}
Next, the data from the dossier in the document library is needed, using the id from the first expired document to get this. This can be done by a “Send an HTTP request to SharePoint”-action specifying that the data is retrieved as text values:
_api/web/lists/getbytitle('Documents')/items('{ID}')/FieldValuesAsText
Creating the list item
Finally, all gathered information can be used to create a new list item in the disposition review list.
The “RetentionPeriodinDays” (int) can be used in the list to calculate the disposition date in a calculated column simply by adding the days to the “RetentionLabelApplied” (date) column.
So that populates the list for each dossier that is ready for disposition review on the site where the archived dossiers reside. The Site address (URL) is also included in the data coming from the trigger.
What happens next, depends on how the organisation wants to support the actual disposition review process. But the data to support this process is all there.
Conclusion
This post is meant to outline a somewhat high-level solution that could be used as an alternative to using the disposition review capability in the Purview Compliance portal. I think one of the main issues could be scalability as the flow is configured without concurrency. So for bigger environments, populating the disposition lists could be slow.
Also, the folderpath-trick to get the documentset is probably not the best approach. But, as long as document sets are not supported by the flow trigger, it’s the best solution I could find.
