Exchange and SharePoint on-premises servers both included retention processing. Exchange’s Messaging Records Management (MRM 2.0) persists today in the form of mailbox retention policies. Microsoft 365 retention policies, first introduced in 2017, replaced SharePoint’s equivalent (here’s an example of SharePoint retention in use) as part of a strategy to implement workload-agnostic data governance. In other words, deploy Microsoft 365 retention processing that worked for all workloads.
The initial implementation of retention policies (and classification labels, as Microsoft then called retention labels) supported Exchange Online (including public folders), SharePoint Online, OneDrive for Business, and Skype for Business Online.
Compliance Records and Retention
Teams was the first Office 365 application to store compliance records in Exchange Online mailboxes. Compliance records are mail items created to capture details of chats and channel conversations, including messages sent by people who don’t have Exchange Online mailboxes like on-premises users, guest accounts, and external (federated) users (the substrate stores these messages in special cloud-only mailboxes). Yammer adopted the same mechanism in 2020, but only for networks configured in Microsoft 365 native mode. Planner announced that they would follow likewise but have not yet delivered this capability.
If you check the number of compliance items in your mailbox, you might find that you have more than you imagine. Most of my chatting happens in other tenants, but even so, I have quite a few compliance records for Teams chats (and fewer for Yammer conversations). Exchange Online doesn’t charge the storage consumed by these messages against user mailbox quotas.
Like any other Exchange Online content, Microsoft Search includes the compliance records in its indexes, which makes the records available for eDiscovery. Communications compliance policies also process compliance records to detect policy violations such as threatening or offensive behavior.
In addition to their role in eDiscovery, compliance records are the basis for retention processing for their originating workloads. Instead of having to create separate jobs to apply retention policies against the workload repositories, a single processing job runs against the compliance records and synchronizes policy actions such as removing items with the workloads.
Originally, the Exchange Managed Folder Assistant (MFA) processed the Teams compliance records. This was a pragmatic step because MFA could process the compliance records along with other mailbox content and avoided the need to create a special retention assistant for Teams. When MFA processed Teams content, it synchronized the results back to Teams, which applied the deletions to its Cosmos DB-based message store. Teams clients then picked up the changes from the message store and removed items in local caches.
The implementation worked well for Teams channel conversations and chat messages, but a better solution was needed to deal with the growth of workloads supporting retention policies and the need to process special cases like conversations in Teams private channels. To solve the problem, Microsoft implemented a specific background assistant to handle retention processing for all Microsoft 365 workloads except Exchange Online (Exchange is different because it supports both the older mailbox retention policies in addition to Microsoft 365 retention policies).
The Retention Assistant and the Microsoft 365 Substrate
Today, the retention assistant evaluates retention policies against OneDrive for Business, SharePoint Online, Yammer, and Teams using the “digital twins” of workload items stored in the Microsoft 365 substrate. Digital twins are copies of content stored in the Microsoft 365 substrate (in Exchange Online) to make it easier for shared services like Search and artificial intelligence services to process items. It’s much easier for a service to process items in a single place than having to deal with multiple repositories, each requiring a different interface (API).
Substrate items coming within the scope of retention policies are subject to retention periods and actions. As the assistant processes items, it checks if the item’s retention period has lapsed, and if this is the case, invokes the retention action. This could involve additional processing, like putting the item into a manual disposition cycle, or it could be a simple delete action. The assistant interacts with the underlying workloads to ensure compliance with retention policy actions. For instance, if the assistant determines that the retention period for a document stored in a SharePoint Online document library has expired, it instructs SharePoint Online to move the document into the site recycle bin.
Black Box Processing
Although it’s possible to track the processing done by MFA for mailbox items, Microsoft has not yet made an equivalent capability available for the retention assistant. This means that administrators who want to validate the effectiveness of retention processing need to make manual checks of content in repositories like SharePoint Online sites to ensure that items which should retention policies should remove are no longer present. As evident in this useful flowchart, understanding how retention policies process items can be a complex business, especially when locations like sites or teams come within the scope of multiple policies.
Making sense of complex retention policies is why the retention assistant exists. I’m sure it does its job very well. It would just be nice to be able to validate and understand exactly what actions the assistant takes for different locations. Is that too much to ask?
Learn more about how Office 365 really works on an ongoing basis by subscribing to the Office 365 for IT Pros eBook. Our monthly updates keep subscribers informed about what’s important across the Office 365 ecosystem.
{"id":null,"mode":"button","open_style":"in_modal","currency_code":"EUR","currency_symbol":"\u20ac","currency_type":"decimal","blank_flag_url":"https:\/\/office365itpros.com\/wp-content\/plugins\/tip-jar-wp\/\/assets\/images\/flags\/blank.gif","flag_sprite_url":"https:\/\/office365itpros.com\/wp-content\/plugins\/tip-jar-wp\/\/assets\/images\/flags\/flags.png","default_amount":100,"top_media_type":"featured_image","featured_image_url":"https:\/\/office365itpros.com\/wp-content\/uploads\/2022\/11\/cover-141x200.jpg","featured_embed":"","header_media":null,"file_download_attachment_data":null,"recurring_options_enabled":true,"recurring_options":{"never":{"selected":true,"after_output":"One time only"},"weekly":{"selected":false,"after_output":"Every week"},"monthly":{"selected":false,"after_output":"Every month"},"yearly":{"selected":false,"after_output":"Every year"}},"strings":{"current_user_email":"","current_user_name":"","link_text":"Virtual Tip Jar","complete_payment_button_error_text":"Check info and try again","payment_verb":"Pay","payment_request_label":"Office 365 for IT Pros","form_has_an_error":"Please check and fix the errors above","general_server_error":"Something isn't working right at the moment. Please try again.","form_title":"Office 365 for IT Pros","form_subtitle":null,"currency_search_text":"Country or Currency here","other_payment_option":"Other payment option","manage_payments_button_text":"Manage your payments","thank_you_message":"Thank you for supporting the work of Office 365 for IT Pros!","payment_confirmation_title":"Office 365 for IT Pros","receipt_title":"Your Receipt","print_receipt":"Print Receipt","email_receipt":"Email Receipt","email_receipt_sending":"Sending receipt...","email_receipt_success":"Email receipt successfully sent","email_receipt_failed":"Email receipt failed to send. Please try again.","receipt_payee":"Paid to","receipt_statement_descriptor":"This will show up on your statement as","receipt_date":"Date","receipt_transaction_id":"Transaction ID","receipt_transaction_amount":"Amount","refund_payer":"Refund from","login":"Log in to manage your payments","manage_payments":"Manage Payments","transactions_title":"Your Transactions","transaction_title":"Transaction Receipt","transaction_period":"Plan Period","arrangements_title":"Your Plans","arrangement_title":"Manage Plan","arrangement_details":"Plan Details","arrangement_id_title":"Plan ID","arrangement_payment_method_title":"Payment Method","arrangement_amount_title":"Plan Amount","arrangement_renewal_title":"Next renewal date","arrangement_action_cancel":"Cancel Plan","arrangement_action_cant_cancel":"Cancelling is currently not available.","arrangement_action_cancel_double":"Are you sure you'd like to cancel?","arrangement_cancelling":"Cancelling Plan...","arrangement_cancelled":"Plan Cancelled","arrangement_failed_to_cancel":"Failed to cancel plan","back_to_plans":"\u2190 Back to Plans","update_payment_method_verb":"Update","sca_auth_description":"Your have a pending renewal payment which requires authorization.","sca_auth_verb":"Authorize renewal payment","sca_authing_verb":"Authorizing payment","sca_authed_verb":"Payment successfully authorized!","sca_auth_failed":"Unable to authorize! Please try again.","login_button_text":"Log in","login_form_has_an_error":"Please check and fix the errors above","uppercase_search":"Search","lowercase_search":"search","uppercase_page":"Page","lowercase_page":"page","uppercase_items":"Items","lowercase_items":"items","uppercase_per":"Per","lowercase_per":"per","uppercase_of":"Of","lowercase_of":"of","back":"Back to plans","zip_code_placeholder":"Zip\/Postal Code","download_file_button_text":"Download File","input_field_instructions":{"tip_amount":{"placeholder_text":"How much would you like to tip?","initial":{"instruction_type":"normal","instruction_message":"How much would you like to tip? Choose any currency."},"empty":{"instruction_type":"error","instruction_message":"How much would you like to tip? Choose any currency."},"invalid_curency":{"instruction_type":"error","instruction_message":"Please choose a valid currency."}},"recurring":{"placeholder_text":"Recurring","initial":{"instruction_type":"normal","instruction_message":"How often would you like to give this?"},"success":{"instruction_type":"success","instruction_message":"How often would you like to give this?"},"empty":{"instruction_type":"error","instruction_message":"How often would you like to give this?"}},"name":{"placeholder_text":"Name on Credit Card","initial":{"instruction_type":"normal","instruction_message":"Enter the name on your card."},"success":{"instruction_type":"success","instruction_message":"Enter the name on your card."},"empty":{"instruction_type":"error","instruction_message":"Please enter the name on your card."}},"privacy_policy":{"terms_title":"Terms and conditions","terms_body":null,"terms_show_text":"View Terms","terms_hide_text":"Hide Terms","initial":{"instruction_type":"normal","instruction_message":"I agree to the terms."},"unchecked":{"instruction_type":"error","instruction_message":"Please agree to the terms."},"checked":{"instruction_type":"success","instruction_message":"I agree to the terms."}},"email":{"placeholder_text":"Your email address","initial":{"instruction_type":"normal","instruction_message":"Enter your email address"},"success":{"instruction_type":"success","instruction_message":"Enter your email address"},"blank":{"instruction_type":"error","instruction_message":"Enter your email address"},"not_an_email_address":{"instruction_type":"error","instruction_message":"Make sure you have entered a valid email address"}},"note_with_tip":{"placeholder_text":"Your note here...","initial":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"empty":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"not_empty_initial":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"saving":{"instruction_type":"normal","instruction_message":"Saving note..."},"success":{"instruction_type":"success","instruction_message":"Note successfully saved!"},"error":{"instruction_type":"error","instruction_message":"Unable to save note note at this time. Please try again."}},"email_for_login_code":{"placeholder_text":"Your email address","initial":{"instruction_type":"normal","instruction_message":"Enter your email to log in."},"success":{"instruction_type":"success","instruction_message":"Enter your email to log in."},"blank":{"instruction_type":"error","instruction_message":"Enter your email to log in."},"empty":{"instruction_type":"error","instruction_message":"Enter your email to log in."}},"login_code":{"initial":{"instruction_type":"normal","instruction_message":"Check your email and enter the login code."},"success":{"instruction_type":"success","instruction_message":"Check your email and enter the login code."},"blank":{"instruction_type":"error","instruction_message":"Check your email and enter the login code."},"empty":{"instruction_type":"error","instruction_message":"Check your email and enter the login code."}},"stripe_all_in_one":{"initial":{"instruction_type":"normal","instruction_message":"Enter your credit card details here."},"empty":{"instruction_type":"error","instruction_message":"Enter your credit card details here."},"success":{"instruction_type":"normal","instruction_message":"Enter your credit card details here."},"invalid_number":{"instruction_type":"error","instruction_message":"The card number is not a valid credit card number."},"invalid_expiry_month":{"instruction_type":"error","instruction_message":"The card's expiration month is invalid."},"invalid_expiry_year":{"instruction_type":"error","instruction_message":"The card's expiration year is invalid."},"invalid_cvc":{"instruction_type":"error","instruction_message":"The card's security code is invalid."},"incorrect_number":{"instruction_type":"error","instruction_message":"The card number is incorrect."},"incomplete_number":{"instruction_type":"error","instruction_message":"The card number is incomplete."},"incomplete_cvc":{"instruction_type":"error","instruction_message":"The card's security code is incomplete."},"incomplete_expiry":{"instruction_type":"error","instruction_message":"The card's expiration date is incomplete."},"incomplete_zip":{"instruction_type":"error","instruction_message":"The card's zip code is incomplete."},"expired_card":{"instruction_type":"error","instruction_message":"The card has expired."},"incorrect_cvc":{"instruction_type":"error","instruction_message":"The card's security code is incorrect."},"incorrect_zip":{"instruction_type":"error","instruction_message":"The card's zip code failed validation."},"invalid_expiry_year_past":{"instruction_type":"error","instruction_message":"The card's expiration year is in the past"},"card_declined":{"instruction_type":"error","instruction_message":"The card was declined."},"missing":{"instruction_type":"error","instruction_message":"There is no card on a customer that is being charged."},"processing_error":{"instruction_type":"error","instruction_message":"An error occurred while processing the card."},"invalid_request_error":{"instruction_type":"error","instruction_message":"Unable to process this payment, please try again or use alternative method."},"invalid_sofort_country":{"instruction_type":"error","instruction_message":"The billing country is not accepted by SOFORT. Please try another country."}}}},"fetched_oembed_html":false}
One Reply to “Microsoft 365 Retention Processing Gets a New Background Assistant”