Error in Retention Processing Removes Personal Chats
Reading Monday’s report in the Register about the problem KPMG suffered when erasing Teams personal chat data for 145,000 users, you might ask the question “how did this happen”? The answer is that an error was made in a change applied to an Office 365 retention policy applied to Teams personal chat. Instead of removing a user from the policy, the update applied the policy to the entire KPMG deployment.
Teams compliance records for personal chats are captured in user mailboxes. These records are used by content searches, eDiscovery cases, and other data governance features like communications compliance policies. Office 365 retention policies to control Teams data use the Exchange Managed Folder Assistant (MFA) to process the compliance records in mailboxes according to policy settings. Exchange Online synchronizes the deletions made by MFA to remove compliance records from the mailboxes to Teams, which then removes the items from its data store in Azure Cosmos DB. The cycle completes when the deletions synchronize from Teams to clients. The overall process used to take far longer than it used to.
KPMG Error
Errors happen in life and in IT. I don’t have direct knowledge of what happened in this case, but it looks very like an administrator updated the retention policy for Teams personal chat and applied it to everyone instead of excluding a user from the policy. It’s easy to do if you’re not paying attention (Figure 1).
Figure 1: Updating a Teams retention policy to cover chats for all users
Only messages in personal chats were affected. Files shared in chats remained unaffected in user OneDrive for Business accounts. Different settings in retention policies for Teams apply to channel conversations, so these messages were unaffected too.
Restrictive Policies for Chats
Many organizations apply restrictive retention policies to Teams personal chat, which is one of the reasons why Microsoft enabled a 1-day retention period for this data. The logic is that personal chat is much like the discussions in WhatsApp and all business discussion should be conducted through Teams channels. That’s a reasonable approach but one that founders on the simple fact that Teams supports group chats for up to 250 users. You can do a lot of business with 249 others, especially if you follow the advice to move debates into chats before presenting an agreed position in a team channel.
Avoiding Errors in Retention Processing
I’m sure KPMG has good change control policies in place to make sure that the right change is made at the right time. You could question making a change to retention policies manually in such a large organization, but on the face of it the proposed change seemed straightforward and easy.
Other large tenants develop and deploy PowerShell scripts to automate management operations and test and debug those processes in test tenants (as the Register report notes, “to automate service execution and remove human intervention in policy management”). However, it can be argued that the nature of retention policies is that they don’t change all that often, making the investment to automate this operation less attractive than others.
The basic fix for something like this is to make sure that anyone who makes a change understands the technology and knows what will happen if they update a retention policy. Asking someone qualified to check that the right change is being made before it is committed is also possible. Such is the benefit of hindsight.
Proponents of backup services for Office 365 will say that they could rescue the situation by being able to restore the deleted compliance records back to user mailboxes. This will certainly solve the eDiscovery gap. Regretfully, no API exists to allow restore back so that the missing data appears as chats, leaving users no better off than they are now. There’s no silver restore bullet available in this scenario.
Retention policies can be complicated and they work differently across Exchange Online, SharePoint Online, OneDrive for Business, and Teams. Learn how to master retention processing by reading Office 365 for IT Pros.
While I know no third-party backup could have fixed this particular issue, do you not see how this proves the point that Retention Policies are not that easy to use? Or at the very least, mistakes can be made that you can’t recover from? What if they had made this mistake with EOL and immediately wiped out all its retained data?
I’m sure you will mention that you can activate the “even you can’t delete history” feature, but I’m guessing most people don’t turn that on because they’re worried about not being able to change their mind if the storage costs get out of hand – which I’m hearing that they can be.
“Retention policies are not that easy to use”: Rather, retention policies are too easy to set up and change (which happened in this case). It’s the knowledge of how retention works and how retention policies are processed that causes the problems.
“Even you can’t delete history”: do you mean retention in SharePoint and its effect on storage quotas and potential cost? I have covered this extensively in the past (https://petri.com/how-retention-impacts-office-365-storage for instance), and as this doesn’t affect the issue under debate (Teams chats), it’s not covered here.
{"id":null,"mode":"button","open_style":"in_modal","currency_code":"EUR","currency_symbol":"\u20ac","currency_type":"decimal","blank_flag_url":"https:\/\/office365itpros.com\/wp-content\/plugins\/tip-jar-wp\/\/assets\/images\/flags\/blank.gif","flag_sprite_url":"https:\/\/office365itpros.com\/wp-content\/plugins\/tip-jar-wp\/\/assets\/images\/flags\/flags.png","default_amount":100,"top_media_type":"featured_image","featured_image_url":"https:\/\/office365itpros.com\/wp-content\/uploads\/2022\/11\/cover-141x200.jpg","featured_embed":"","header_media":null,"file_download_attachment_data":null,"recurring_options_enabled":true,"recurring_options":{"never":{"selected":true,"after_output":"One time only"},"weekly":{"selected":false,"after_output":"Every week"},"monthly":{"selected":false,"after_output":"Every month"},"yearly":{"selected":false,"after_output":"Every year"}},"strings":{"current_user_email":"","current_user_name":"","link_text":"Virtual Tip Jar","complete_payment_button_error_text":"Check info and try again","payment_verb":"Pay","payment_request_label":"Office 365 for IT Pros","form_has_an_error":"Please check and fix the errors above","general_server_error":"Something isn't working right at the moment. Please try again.","form_title":"Office 365 for IT Pros","form_subtitle":null,"currency_search_text":"Country or Currency here","other_payment_option":"Other payment option","manage_payments_button_text":"Manage your payments","thank_you_message":"Thank you for supporting the work of Office 365 for IT Pros!","payment_confirmation_title":"Office 365 for IT Pros","receipt_title":"Your Receipt","print_receipt":"Print Receipt","email_receipt":"Email Receipt","email_receipt_sending":"Sending receipt...","email_receipt_success":"Email receipt successfully sent","email_receipt_failed":"Email receipt failed to send. Please try again.","receipt_payee":"Paid to","receipt_statement_descriptor":"This will show up on your statement as","receipt_date":"Date","receipt_transaction_id":"Transaction ID","receipt_transaction_amount":"Amount","refund_payer":"Refund from","login":"Log in to manage your payments","manage_payments":"Manage Payments","transactions_title":"Your Transactions","transaction_title":"Transaction Receipt","transaction_period":"Plan Period","arrangements_title":"Your Plans","arrangement_title":"Manage Plan","arrangement_details":"Plan Details","arrangement_id_title":"Plan ID","arrangement_payment_method_title":"Payment Method","arrangement_amount_title":"Plan Amount","arrangement_renewal_title":"Next renewal date","arrangement_action_cancel":"Cancel Plan","arrangement_action_cant_cancel":"Cancelling is currently not available.","arrangement_action_cancel_double":"Are you sure you'd like to cancel?","arrangement_cancelling":"Cancelling Plan...","arrangement_cancelled":"Plan Cancelled","arrangement_failed_to_cancel":"Failed to cancel plan","back_to_plans":"\u2190 Back to Plans","update_payment_method_verb":"Update","sca_auth_description":"Your have a pending renewal payment which requires authorization.","sca_auth_verb":"Authorize renewal payment","sca_authing_verb":"Authorizing payment","sca_authed_verb":"Payment successfully authorized!","sca_auth_failed":"Unable to authorize! Please try again.","login_button_text":"Log in","login_form_has_an_error":"Please check and fix the errors above","uppercase_search":"Search","lowercase_search":"search","uppercase_page":"Page","lowercase_page":"page","uppercase_items":"Items","lowercase_items":"items","uppercase_per":"Per","lowercase_per":"per","uppercase_of":"Of","lowercase_of":"of","back":"Back to plans","zip_code_placeholder":"Zip\/Postal Code","download_file_button_text":"Download File","input_field_instructions":{"tip_amount":{"placeholder_text":"How much would you like to tip?","initial":{"instruction_type":"normal","instruction_message":"How much would you like to tip? Choose any currency."},"empty":{"instruction_type":"error","instruction_message":"How much would you like to tip? Choose any currency."},"invalid_curency":{"instruction_type":"error","instruction_message":"Please choose a valid currency."}},"recurring":{"placeholder_text":"Recurring","initial":{"instruction_type":"normal","instruction_message":"How often would you like to give this?"},"success":{"instruction_type":"success","instruction_message":"How often would you like to give this?"},"empty":{"instruction_type":"error","instruction_message":"How often would you like to give this?"}},"name":{"placeholder_text":"Name on Credit Card","initial":{"instruction_type":"normal","instruction_message":"Enter the name on your card."},"success":{"instruction_type":"success","instruction_message":"Enter the name on your card."},"empty":{"instruction_type":"error","instruction_message":"Please enter the name on your card."}},"privacy_policy":{"terms_title":"Terms and conditions","terms_body":null,"terms_show_text":"View Terms","terms_hide_text":"Hide Terms","initial":{"instruction_type":"normal","instruction_message":"I agree to the terms."},"unchecked":{"instruction_type":"error","instruction_message":"Please agree to the terms."},"checked":{"instruction_type":"success","instruction_message":"I agree to the terms."}},"email":{"placeholder_text":"Your email address","initial":{"instruction_type":"normal","instruction_message":"Enter your email address"},"success":{"instruction_type":"success","instruction_message":"Enter your email address"},"blank":{"instruction_type":"error","instruction_message":"Enter your email address"},"not_an_email_address":{"instruction_type":"error","instruction_message":"Make sure you have entered a valid email address"}},"note_with_tip":{"placeholder_text":"Your note here...","initial":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"empty":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"not_empty_initial":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"saving":{"instruction_type":"normal","instruction_message":"Saving note..."},"success":{"instruction_type":"success","instruction_message":"Note successfully saved!"},"error":{"instruction_type":"error","instruction_message":"Unable to save note note at this time. Please try again."}},"email_for_login_code":{"placeholder_text":"Your email address","initial":{"instruction_type":"normal","instruction_message":"Enter your email to log in."},"success":{"instruction_type":"success","instruction_message":"Enter your email to log in."},"blank":{"instruction_type":"error","instruction_message":"Enter your email to log in."},"empty":{"instruction_type":"error","instruction_message":"Enter your email to log in."}},"login_code":{"initial":{"instruction_type":"normal","instruction_message":"Check your email and enter the login code."},"success":{"instruction_type":"success","instruction_message":"Check your email and enter the login code."},"blank":{"instruction_type":"error","instruction_message":"Check your email and enter the login code."},"empty":{"instruction_type":"error","instruction_message":"Check your email and enter the login code."}},"stripe_all_in_one":{"initial":{"instruction_type":"normal","instruction_message":"Enter your credit card details here."},"empty":{"instruction_type":"error","instruction_message":"Enter your credit card details here."},"success":{"instruction_type":"normal","instruction_message":"Enter your credit card details here."},"invalid_number":{"instruction_type":"error","instruction_message":"The card number is not a valid credit card number."},"invalid_expiry_month":{"instruction_type":"error","instruction_message":"The card's expiration month is invalid."},"invalid_expiry_year":{"instruction_type":"error","instruction_message":"The card's expiration year is invalid."},"invalid_cvc":{"instruction_type":"error","instruction_message":"The card's security code is invalid."},"incorrect_number":{"instruction_type":"error","instruction_message":"The card number is incorrect."},"incomplete_number":{"instruction_type":"error","instruction_message":"The card number is incomplete."},"incomplete_cvc":{"instruction_type":"error","instruction_message":"The card's security code is incomplete."},"incomplete_expiry":{"instruction_type":"error","instruction_message":"The card's expiration date is incomplete."},"incomplete_zip":{"instruction_type":"error","instruction_message":"The card's zip code is incomplete."},"expired_card":{"instruction_type":"error","instruction_message":"The card has expired."},"incorrect_cvc":{"instruction_type":"error","instruction_message":"The card's security code is incorrect."},"incorrect_zip":{"instruction_type":"error","instruction_message":"The card's zip code failed validation."},"invalid_expiry_year_past":{"instruction_type":"error","instruction_message":"The card's expiration year is in the past"},"card_declined":{"instruction_type":"error","instruction_message":"The card was declined."},"missing":{"instruction_type":"error","instruction_message":"There is no card on a customer that is being charged."},"processing_error":{"instruction_type":"error","instruction_message":"An error occurred while processing the card."},"invalid_request_error":{"instruction_type":"error","instruction_message":"Unable to process this payment, please try again or use alternative method."},"invalid_sofort_country":{"instruction_type":"error","instruction_message":"The billing country is not accepted by SOFORT. Please try another country."}}}},"fetched_oembed_html":false}
Hello, again, Tony.
While I know no third-party backup could have fixed this particular issue, do you not see how this proves the point that Retention Policies are not that easy to use? Or at the very least, mistakes can be made that you can’t recover from? What if they had made this mistake with EOL and immediately wiped out all its retained data?
I’m sure you will mention that you can activate the “even you can’t delete history” feature, but I’m guessing most people don’t turn that on because they’re worried about not being able to change their mind if the storage costs get out of hand – which I’m hearing that they can be.
“Retention policies are not that easy to use”: Rather, retention policies are too easy to set up and change (which happened in this case). It’s the knowledge of how retention works and how retention policies are processed that causes the problems.
“Even you can’t delete history”: do you mean retention in SharePoint and its effect on storage quotas and potential cost? I have covered this extensively in the past (https://petri.com/how-retention-impacts-office-365-storage for instance), and as this doesn’t affect the issue under debate (Teams chats), it’s not covered here.