Restrict Processing to Specific File Types Using Auto-Label Retention Policies
A reader asks if it’s possible to limit the processing of a Microsoft 365 retention policy to certain file types. In this case, they wish to remove any Word documents, Excel spreadsheets, and PDF files from the SharePoint Online sites used by Teams while leaving other items stored in SharePoint like the OneNote notebook and .mht files used by the Teams wiki.
Before answering the question, it’s important to understand that Teams retention policies only process messages from channel conversations (including private channels) and chat. If you want a retention policy to process the SharePoint content owned by a team, you need the policy to process Microsoft 365 Groups. This is because Teams uses Microsoft 365 Groups for its identity and membership service.
The challenge then is how to amend a retention policy for Microsoft 365 Groups to only process certain file types. However, Microsoft 365 retention policies operate on a “whole container” basis. In other words, they apply the same retention settings to all items found in the target locations. In this case, Microsoft 365 Groups. Although you can refine a policy to process selected groups, the GUI in the Microsoft 365 compliance center doesn’t allow you to modify a policy to perform selective processing by file type.
Assigning Content Queries with PowerShell
PowerShell cmdlets are available to manage Microsoft 365 standard and auto-label retention policies. To access these cmdlets, connect to the Exchange Online management endpoint and then run the Connect-IPPSSession cmdlet to connect to the compliance endpoint. You can then run the Get-RetentionCompliancePolicy cmdlet to view the set of retention policies defined in the tenant and the Get-RetentionComplianceRule to view the rule defined for each policy. Each retention policy divides into the general policy settings and the rule dictating the retention action performed by the policy.
If you look at retention policy rules, you’ll see a parameter called ContentMatchQuery. Checking Microsoft’s documentation for Set-RetentionComplianceRule, we discover that this parameter contains a content query in Keyword Query Language (KQL) to filter the content scanned by the policy. A KQL-based content query is a commonplace object within Microsoft 365 as it’s used for SharePoint search, eDiscovery content searches, and automatic labeling policies.
It is possible to update a standard retention policy to add a content query. However, because no GUI exists for this purpose, you’ve got to add the content query with PowerShell. For example, the content query added to the target retention policy finds any Word document, Excel worksheet, or PDF file:
Set-RetentionComplianceRule -Identity "Retention for Specific File Types" -ContentMatchQuery "filetype:doc* filetype:xls* filetype:pdf"
You can test a content query by inputting it to SharePoint Search. If the search returns the expected files, you know the filter is good and will work with a retention policy.
The Problem with Standard Retention Policies
Or can it? Well, the filter works, and the retention policy will process only the specified files, but Microsoft doesn’t support this scenario. At least, they don’t include this scenario in their testing as they introduce new functionality (like adaptive scopes), which means that something might break inadvertently. That’s not a good situation for a compliance solution.
In the earliest days of Office 365 retention policies, as they were then, you could attach a content query to a standard retention policy, but this has not been a supported scenario since Microsoft removed the GUI to enter KQL queries for retention policies. However, Microsoft left the ability to define a content query for a standard retention policy through PowerShell intact, and that’s what has created the inconsistency which exists today. To address the issue, Microsoft should update the documentation for Set-RetentionComplianceRule to say that the ContentMatchQuery parameter is unsupported with standard retention policies.
Use Auto-Label Policies
The approved solution is an auto-label retention policy. These policies accept KQL content queries like the example cited above. When SharePoint Online applies a retention policy with a content query, it finds matching files and assigns the retention label defined in the policy to the items. The retention period and action defined in the retention label controls what happens to the files. For example, the label might define a retention period of one year after which SharePoint Online should remove the labeled files.
One advantage of using an auto-label retention policy is that users see the retention label applied to matching files and can change the label if necessary. This doesn’t happen with standard retention policies because all processing happens in the background and users won’t necessarily be aware of the deletion of files.
The downside of auto-label retention policies is that any account which comes within the scope of the policy (for instance, all members of the Microsoft 365 groups processed by a policy) require Office 365 E5 or Microsoft 365 compliance licenses. This is not a problem if your organization already has these licenses, but there’s quite a step up from Office 365 E3 (needed for standard retention policies) to automatic processing, even before Microsoft’s March 1 price increase.
The bottom line is that you can live on the edge and use content queries with standard retention policies until something breaks (or Microsoft disables the capabilities) or use auto-label retention policies for selective processing based on file types. Going with a supported approach is always a better choice, even if it might cost some extra license fees.
Insight like this doesn’t come easily. You’ve got to know the technology and understand how to look behind the scenes. Benefit from the knowledge and experience of the Office 365 for IT Pros team by subscribing to the best eBook covering Office 365 and the wider Microsoft 365 ecosystem.
2 Replies to “Why Content Queries are Only Supported by Microsoft 365 Auto-Label Retention Policies”
Thank you for this article.
Regarding the necessity of a Microsoft 365 group retention policy to delete SharePoint files associated with a Teams team, the documentation from Microsoft is fuzzy.
This whole thing is further complicated by the green “Tip” text box on the same page I first mentioned, where it says, “You can apply a retention policy to the files of just a specific team when it’s not connected to a Microsoft 365 group by selecting the SharePoint site for the team, and the OneDrive accounts of users in the Team.” Again, we know that every Team is “connected to a Microsoft 365 group” so what is meant by this tip? I suspect it’s referring to teams that were created inside of Teams, from scratch, which didn’t start out as Microsoft 365 groups and so only need the SharePoint retention policy and not a Microsoft 365 retention policy, in order to delete/retain files.
I’ve been trying to test all of this for the last three months with help from Microsoft Support, but without any definitive results so far.
In a nutshell, if you have a SPO site connected to a Microsoft 365 group (including Teams), use the Microsoft 365 Groups location for retention policies. If they are standalone sites, use SharePoint Online.
{"id":null,"mode":"button","open_style":"in_modal","currency_code":"EUR","currency_symbol":"\u20ac","currency_type":"decimal","blank_flag_url":"https:\/\/office365itpros.com\/wp-content\/plugins\/tip-jar-wp\/\/assets\/images\/flags\/blank.gif","flag_sprite_url":"https:\/\/office365itpros.com\/wp-content\/plugins\/tip-jar-wp\/\/assets\/images\/flags\/flags.png","default_amount":100,"top_media_type":"featured_image","featured_image_url":"https:\/\/office365itpros.com\/wp-content\/uploads\/2022\/11\/cover-141x200.jpg","featured_embed":"","header_media":null,"file_download_attachment_data":null,"recurring_options_enabled":true,"recurring_options":{"never":{"selected":true,"after_output":"One time only"},"weekly":{"selected":false,"after_output":"Every week"},"monthly":{"selected":false,"after_output":"Every month"},"yearly":{"selected":false,"after_output":"Every year"}},"strings":{"current_user_email":"","current_user_name":"","link_text":"Virtual Tip Jar","complete_payment_button_error_text":"Check info and try again","payment_verb":"Pay","payment_request_label":"Office 365 for IT Pros","form_has_an_error":"Please check and fix the errors above","general_server_error":"Something isn't working right at the moment. Please try again.","form_title":"Office 365 for IT Pros","form_subtitle":null,"currency_search_text":"Country or Currency here","other_payment_option":"Other payment option","manage_payments_button_text":"Manage your payments","thank_you_message":"Thank you for supporting the work of Office 365 for IT Pros!","payment_confirmation_title":"Office 365 for IT Pros","receipt_title":"Your Receipt","print_receipt":"Print Receipt","email_receipt":"Email Receipt","email_receipt_sending":"Sending receipt...","email_receipt_success":"Email receipt successfully sent","email_receipt_failed":"Email receipt failed to send. Please try again.","receipt_payee":"Paid to","receipt_statement_descriptor":"This will show up on your statement as","receipt_date":"Date","receipt_transaction_id":"Transaction ID","receipt_transaction_amount":"Amount","refund_payer":"Refund from","login":"Log in to manage your payments","manage_payments":"Manage Payments","transactions_title":"Your Transactions","transaction_title":"Transaction Receipt","transaction_period":"Plan Period","arrangements_title":"Your Plans","arrangement_title":"Manage Plan","arrangement_details":"Plan Details","arrangement_id_title":"Plan ID","arrangement_payment_method_title":"Payment Method","arrangement_amount_title":"Plan Amount","arrangement_renewal_title":"Next renewal date","arrangement_action_cancel":"Cancel Plan","arrangement_action_cant_cancel":"Cancelling is currently not available.","arrangement_action_cancel_double":"Are you sure you'd like to cancel?","arrangement_cancelling":"Cancelling Plan...","arrangement_cancelled":"Plan Cancelled","arrangement_failed_to_cancel":"Failed to cancel plan","back_to_plans":"\u2190 Back to Plans","update_payment_method_verb":"Update","sca_auth_description":"Your have a pending renewal payment which requires authorization.","sca_auth_verb":"Authorize renewal payment","sca_authing_verb":"Authorizing payment","sca_authed_verb":"Payment successfully authorized!","sca_auth_failed":"Unable to authorize! Please try again.","login_button_text":"Log in","login_form_has_an_error":"Please check and fix the errors above","uppercase_search":"Search","lowercase_search":"search","uppercase_page":"Page","lowercase_page":"page","uppercase_items":"Items","lowercase_items":"items","uppercase_per":"Per","lowercase_per":"per","uppercase_of":"Of","lowercase_of":"of","back":"Back to plans","zip_code_placeholder":"Zip\/Postal Code","download_file_button_text":"Download File","input_field_instructions":{"tip_amount":{"placeholder_text":"How much would you like to tip?","initial":{"instruction_type":"normal","instruction_message":"How much would you like to tip? Choose any currency."},"empty":{"instruction_type":"error","instruction_message":"How much would you like to tip? Choose any currency."},"invalid_curency":{"instruction_type":"error","instruction_message":"Please choose a valid currency."}},"recurring":{"placeholder_text":"Recurring","initial":{"instruction_type":"normal","instruction_message":"How often would you like to give this?"},"success":{"instruction_type":"success","instruction_message":"How often would you like to give this?"},"empty":{"instruction_type":"error","instruction_message":"How often would you like to give this?"}},"name":{"placeholder_text":"Name on Credit Card","initial":{"instruction_type":"normal","instruction_message":"Enter the name on your card."},"success":{"instruction_type":"success","instruction_message":"Enter the name on your card."},"empty":{"instruction_type":"error","instruction_message":"Please enter the name on your card."}},"privacy_policy":{"terms_title":"Terms and conditions","terms_body":null,"terms_show_text":"View Terms","terms_hide_text":"Hide Terms","initial":{"instruction_type":"normal","instruction_message":"I agree to the terms."},"unchecked":{"instruction_type":"error","instruction_message":"Please agree to the terms."},"checked":{"instruction_type":"success","instruction_message":"I agree to the terms."}},"email":{"placeholder_text":"Your email address","initial":{"instruction_type":"normal","instruction_message":"Enter your email address"},"success":{"instruction_type":"success","instruction_message":"Enter your email address"},"blank":{"instruction_type":"error","instruction_message":"Enter your email address"},"not_an_email_address":{"instruction_type":"error","instruction_message":"Make sure you have entered a valid email address"}},"note_with_tip":{"placeholder_text":"Your note here...","initial":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"empty":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"not_empty_initial":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"saving":{"instruction_type":"normal","instruction_message":"Saving note..."},"success":{"instruction_type":"success","instruction_message":"Note successfully saved!"},"error":{"instruction_type":"error","instruction_message":"Unable to save note note at this time. Please try again."}},"email_for_login_code":{"placeholder_text":"Your email address","initial":{"instruction_type":"normal","instruction_message":"Enter your email to log in."},"success":{"instruction_type":"success","instruction_message":"Enter your email to log in."},"blank":{"instruction_type":"error","instruction_message":"Enter your email to log in."},"empty":{"instruction_type":"error","instruction_message":"Enter your email to log in."}},"login_code":{"initial":{"instruction_type":"normal","instruction_message":"Check your email and enter the login code."},"success":{"instruction_type":"success","instruction_message":"Check your email and enter the login code."},"blank":{"instruction_type":"error","instruction_message":"Check your email and enter the login code."},"empty":{"instruction_type":"error","instruction_message":"Check your email and enter the login code."}},"stripe_all_in_one":{"initial":{"instruction_type":"normal","instruction_message":"Enter your credit card details here."},"empty":{"instruction_type":"error","instruction_message":"Enter your credit card details here."},"success":{"instruction_type":"normal","instruction_message":"Enter your credit card details here."},"invalid_number":{"instruction_type":"error","instruction_message":"The card number is not a valid credit card number."},"invalid_expiry_month":{"instruction_type":"error","instruction_message":"The card's expiration month is invalid."},"invalid_expiry_year":{"instruction_type":"error","instruction_message":"The card's expiration year is invalid."},"invalid_cvc":{"instruction_type":"error","instruction_message":"The card's security code is invalid."},"incorrect_number":{"instruction_type":"error","instruction_message":"The card number is incorrect."},"incomplete_number":{"instruction_type":"error","instruction_message":"The card number is incomplete."},"incomplete_cvc":{"instruction_type":"error","instruction_message":"The card's security code is incomplete."},"incomplete_expiry":{"instruction_type":"error","instruction_message":"The card's expiration date is incomplete."},"incomplete_zip":{"instruction_type":"error","instruction_message":"The card's zip code is incomplete."},"expired_card":{"instruction_type":"error","instruction_message":"The card has expired."},"incorrect_cvc":{"instruction_type":"error","instruction_message":"The card's security code is incorrect."},"incorrect_zip":{"instruction_type":"error","instruction_message":"The card's zip code failed validation."},"invalid_expiry_year_past":{"instruction_type":"error","instruction_message":"The card's expiration year is in the past"},"card_declined":{"instruction_type":"error","instruction_message":"The card was declined."},"missing":{"instruction_type":"error","instruction_message":"There is no card on a customer that is being charged."},"processing_error":{"instruction_type":"error","instruction_message":"An error occurred while processing the card."},"invalid_request_error":{"instruction_type":"error","instruction_message":"Unable to process this payment, please try again or use alternative method."},"invalid_sofort_country":{"instruction_type":"error","instruction_message":"The billing country is not accepted by SOFORT. Please try another country."}}}},"fetched_oembed_html":false}
Thank you for this article.
Regarding the necessity of a Microsoft 365 group retention policy to delete SharePoint files associated with a Teams team, the documentation from Microsoft is fuzzy.
Here (https://docs.microsoft.com/en-us/microsoft-365/compliance/create-retention-policies?view=o365-worldwide#additional-retention-policy-needed-to-support-teams) the language is, “If you have teams that were created from a Microsoft 365 group (formerly Office 365 group), you should additionally configure a retention policy that includes that Microsoft 365 group by using the Microsoft 365 Groups location.” But we know that every Teams team is associated with a Microsoft 365 Group (see here https://docs.microsoft.com/en-us/microsoftteams/office-365-groups#how-microsoft-365-groups-work-with-teams). So what does is meant by “If you have a teams that were created from a Microsoft 365 group”? I take that to mean that a team created from scratch inside of Teams does not require such a policy, but a team that was created from an existing Microsoft 365 group would need a Microsoft 365 Group retention policy applied.
This whole thing is further complicated by the green “Tip” text box on the same page I first mentioned, where it says, “You can apply a retention policy to the files of just a specific team when it’s not connected to a Microsoft 365 group by selecting the SharePoint site for the team, and the OneDrive accounts of users in the Team.” Again, we know that every Team is “connected to a Microsoft 365 group” so what is meant by this tip? I suspect it’s referring to teams that were created inside of Teams, from scratch, which didn’t start out as Microsoft 365 groups and so only need the SharePoint retention policy and not a Microsoft 365 retention policy, in order to delete/retain files.
I’ve been trying to test all of this for the last three months with help from Microsoft Support, but without any definitive results so far.
In a nutshell, if you have a SPO site connected to a Microsoft 365 group (including Teams), use the Microsoft 365 Groups location for retention policies. If they are standalone sites, use SharePoint Online.