Captured by the Substrate and Stored in Exchange Online
Updated 7 June, 2022
The Microsoft 365 substrate captures Teams compliance records as users chat and post messages to channels. Individual records are created for each message in a thread or conversation. The substrate stores the compliance records in Exchange Online mailboxes:
Compliance records for personal chats and private channel conversations are in user mailboxes. When guest and federated users participate in chats, the substrate captures compliance records for these users in special cloud mailboxes that are inaccessible to normal client interfaces.
Records for regular channel conversations are in the group mailbox of the hosting team.
Records for shared channel conversations are in a special cloud mailbox
In all cases, the compliance records are in the TeamsMessagesData folder in the mailbox. Figure 1 shows compliance records in the TeamsMessagesData folder as viewed using the MFCMAPI utility. The text of the message is highlighted in the item properties.
Figure 1: Teams Compliance Records stored in Exchange Online
Some backup vendors claim that including the compliance records is enough for Teams backup. The biggest issue with this stance is it’s impossible to recreate a chat or channel conversation from compliance records because the records are imperfect copies (aka “digital twins”) of the original messages created purely for compliance purposes. It’s one of the reasons why Teams is such a difficult application to backup.
Exchange Online indexes the compliance records to ensure that they are available for eDiscovery. Other Microsoft 365 services consume compliance records. For instance, communication compliance relies on these records to detect policy violations such as obscenities or threats in Teams conversations.
Accumulating Since 2017
The substrate has captured Teams compliance records since 2017. Since that time, considerable volumes of these records have accumulated unless Microsoft 365 retention policies for Teams control how long records remain in mailboxes. These policies can remove records after one day. In a notable incident in 2020, an error in the settings of a retention policy resulted in the removal of records for the entire KPMG organization.
You don’t need to worry about Teams compliance records. They exist, do a job, and won’t go away unless retention policies are active. And they are helpful in assessing how much data needs to be moved in a tenant-to-tenant migration.
Migrations for Microsoft 365 tenants involve many different types of data from emails to documents to Teams messages. Migrating emails and documents are well-understood challenges, but the API support for moving Teams information is less developed. Knowing how much information must be moved is a starting point. Although imprecise, one way to get an idea of what’s involved is to check how many Teams compliance records exist in the tenant.
Checking Chats
To begin, we can use the Get-ExoMailboxFolderStatistics cmdlet to return the number of records in user mailboxes. Here’s some code to do the job. You can see that we’re checking the Non-IPM part of the mailbox, which holds the folders used for system purposes that are invisible to users.
$Report | ft user, teamsitems, newestitem
User TeamsItems NewestItem
---- ---------- ----------
Andy Ruth (Director) 11257 03/05/2022 13:32:11
Ben Owens (DCPG) 145354 13/05/2022 15:12:51
Ben James 7257 23/05/2022 15:12:51
Brian Weakliam (Operations) 9866 31/05/2022 12:46:53
James Ryan 22446 29/05/2022 14:39:34
Chris Bishop 11178 28/05/2022 14:39:34
Kim Akers 9533 29/05/2022 10:41:57
James Abrahams 11127 01/06/2022 13:32:11
Tony Redmond 16398 01/06/2022 17:02:47
A total for Teams compliance records in user mailboxes is easily found with a command like:
($Report | measure-object teamsitems -Sum).Sum
Remember that the information spans personal and group chats and conversations in private channels. It’s also true that multiple copies of messages exist. For example, every message posted to a group chat involving ten people results in ten copies of the message (one per mailbox for each participant). The same is true for messages posted to conversations in private channels. It’s hard to set a duplication factor, but it’s likely to be in the region of 3-4, depending on the size of the organization (the larger the organization, the more likely it is that multiple people are involved in chats).
Checking Regular Channels
Channel conversations are easier to move than personal chats because only a single copy of each message exists, and they’re owned by the group rather than individual users. The same kind of approach should work for group mailboxes to report the compliance records gathered for conversations in regular channels.
Here’s a version of the code for user mailboxes that finds all group mailboxes used for teams and runs Get-MailboxFolderStatistics for each mailbox.
Connect-MgGraph
Select-MgProfile Beta
$Report = [System.Collections.Generic.List[Object]]::new()
[array]$Teams = Get-MgGroup -Filter "resourceProvisioningOptions/Any(x:x eq 'Team')" -All
If (!($Teams)) {Write-Host "No Teams found - exiting" ; break }
ForEach ($T in $Teams) {
[array]$Stats = Get-MailboxFolderStatistics -Identity $T.Id -FolderScope NonIPM -IncludeOldestAndNewestItems -ResultSize Unlimited | ? {$_.Name -eq "TeamsMessagesData"}
If ($Stats.Count -ne 0) {
$ReportLine = [PSCustomObject][Ordered]@{ # Write out details of the mailbox
"User" = $T.DisplayName
UPN = $M.Mail
TeamsItems = $Stats.ItemsInFolder
TeamsItemsSize = $Stats.FolderSize.Split("(")[0]
NewestItem = $Stats.NewestItemReceivedDate
TotalMbxFolderSize = [math]::Round($TotalMbxsize/1Mb,2) }
$Report.Add($ReportLine)
Write-Host "Team" $T.DisplayName "Teams channel messages" $Stats.ItemsInFolder }
Else {Write-Host "Can't figure out the Teams data for" $T.DisplayName }
}
Up to June 7, 2022, a problem existed in that both Get-MailboxFolderStatistics and Get-ExoMailboxFolderStatistics could return a maximum of 1,000 folders. The issue was first noted in May 2022. Microsoft addressed the problem by adding the ResultSize Unlimited parameter to the Get-MailboxFolderStatistics cmdlet. So far, they have not given Get-ExoMailboxFolderStatistics the same capability, possibly because of the different way that this cmdlet interacts with data (the cmdlet uses a REST API, which imply pagination).
Not many mailboxes have more than a thousand folders. However, I recently noticed the appearance of hundreds of folders named SPOOLS_830, SPOOLS_831, SPOOLS_832, etc. in some group mailboxes. The presence of these folders meant that some group mailboxes easily exceeded the 1,000 folder limit. That’s why I use Get-MailboxFolderStatistics instead of Get-ExoMailboxFolderStatistics in the code shown above. When Microsoft adds the -ResultSize Unlimited parameter to Get-ExoMailboxFolderStatistics, I’ll use that instead.
Interestingly, the Graph MailFolders API does not return the TeamsMessagesData folder in the set of folders it finds in the Non-IPM part of mailboxes.
Poking Around to Find Information
It’s always interesting to poke around Microsoft 365 to find out how things work. In this case, we’ve discussed how to report the number of Teams compliance records stored in user and group mailboxes, something that might be useful if you’re ever involved in a tenant-to-tenant migration project.
We also highlighted the upper limit of the Get-ExoMailboxFolderStatistics cmdlet. You’re unlikely to meet the limit in normal operations unless users have been very busy creating folders. I don’t know why so many SPOOLS folders exist in some group mailboxes, but I shall try to find out.
So much change, all the time. It’s a challenge to stay abreast of all the updates Microsoft makes across Office 365. Subscribe to the Office 365 for IT Pros eBook to receive monthly insights into what happens, why it happens, and what new features and capabilities mean for your tenant.
{"id":null,"mode":"button","open_style":"in_modal","currency_code":"EUR","currency_symbol":"\u20ac","currency_type":"decimal","blank_flag_url":"https:\/\/office365itpros.com\/wp-content\/plugins\/tip-jar-wp\/\/assets\/images\/flags\/blank.gif","flag_sprite_url":"https:\/\/office365itpros.com\/wp-content\/plugins\/tip-jar-wp\/\/assets\/images\/flags\/flags.png","default_amount":100,"top_media_type":"featured_image","featured_image_url":"https:\/\/office365itpros.com\/wp-content\/uploads\/2022\/11\/cover-141x200.jpg","featured_embed":"","header_media":null,"file_download_attachment_data":null,"recurring_options_enabled":true,"recurring_options":{"never":{"selected":true,"after_output":"One time only"},"weekly":{"selected":false,"after_output":"Every week"},"monthly":{"selected":false,"after_output":"Every month"},"yearly":{"selected":false,"after_output":"Every year"}},"strings":{"current_user_email":"","current_user_name":"","link_text":"Virtual Tip Jar","complete_payment_button_error_text":"Check info and try again","payment_verb":"Pay","payment_request_label":"Office 365 for IT Pros","form_has_an_error":"Please check and fix the errors above","general_server_error":"Something isn't working right at the moment. Please try again.","form_title":"Office 365 for IT Pros","form_subtitle":null,"currency_search_text":"Country or Currency here","other_payment_option":"Other payment option","manage_payments_button_text":"Manage your payments","thank_you_message":"Thank you for supporting the work of Office 365 for IT Pros!","payment_confirmation_title":"Office 365 for IT Pros","receipt_title":"Your Receipt","print_receipt":"Print Receipt","email_receipt":"Email Receipt","email_receipt_sending":"Sending receipt...","email_receipt_success":"Email receipt successfully sent","email_receipt_failed":"Email receipt failed to send. Please try again.","receipt_payee":"Paid to","receipt_statement_descriptor":"This will show up on your statement as","receipt_date":"Date","receipt_transaction_id":"Transaction ID","receipt_transaction_amount":"Amount","refund_payer":"Refund from","login":"Log in to manage your payments","manage_payments":"Manage Payments","transactions_title":"Your Transactions","transaction_title":"Transaction Receipt","transaction_period":"Plan Period","arrangements_title":"Your Plans","arrangement_title":"Manage Plan","arrangement_details":"Plan Details","arrangement_id_title":"Plan ID","arrangement_payment_method_title":"Payment Method","arrangement_amount_title":"Plan Amount","arrangement_renewal_title":"Next renewal date","arrangement_action_cancel":"Cancel Plan","arrangement_action_cant_cancel":"Cancelling is currently not available.","arrangement_action_cancel_double":"Are you sure you'd like to cancel?","arrangement_cancelling":"Cancelling Plan...","arrangement_cancelled":"Plan Cancelled","arrangement_failed_to_cancel":"Failed to cancel plan","back_to_plans":"\u2190 Back to Plans","update_payment_method_verb":"Update","sca_auth_description":"Your have a pending renewal payment which requires authorization.","sca_auth_verb":"Authorize renewal payment","sca_authing_verb":"Authorizing payment","sca_authed_verb":"Payment successfully authorized!","sca_auth_failed":"Unable to authorize! Please try again.","login_button_text":"Log in","login_form_has_an_error":"Please check and fix the errors above","uppercase_search":"Search","lowercase_search":"search","uppercase_page":"Page","lowercase_page":"page","uppercase_items":"Items","lowercase_items":"items","uppercase_per":"Per","lowercase_per":"per","uppercase_of":"Of","lowercase_of":"of","back":"Back to plans","zip_code_placeholder":"Zip\/Postal Code","download_file_button_text":"Download File","input_field_instructions":{"tip_amount":{"placeholder_text":"How much would you like to tip?","initial":{"instruction_type":"normal","instruction_message":"How much would you like to tip? Choose any currency."},"empty":{"instruction_type":"error","instruction_message":"How much would you like to tip? Choose any currency."},"invalid_curency":{"instruction_type":"error","instruction_message":"Please choose a valid currency."}},"recurring":{"placeholder_text":"Recurring","initial":{"instruction_type":"normal","instruction_message":"How often would you like to give this?"},"success":{"instruction_type":"success","instruction_message":"How often would you like to give this?"},"empty":{"instruction_type":"error","instruction_message":"How often would you like to give this?"}},"name":{"placeholder_text":"Name on Credit Card","initial":{"instruction_type":"normal","instruction_message":"Enter the name on your card."},"success":{"instruction_type":"success","instruction_message":"Enter the name on your card."},"empty":{"instruction_type":"error","instruction_message":"Please enter the name on your card."}},"privacy_policy":{"terms_title":"Terms and conditions","terms_body":null,"terms_show_text":"View Terms","terms_hide_text":"Hide Terms","initial":{"instruction_type":"normal","instruction_message":"I agree to the terms."},"unchecked":{"instruction_type":"error","instruction_message":"Please agree to the terms."},"checked":{"instruction_type":"success","instruction_message":"I agree to the terms."}},"email":{"placeholder_text":"Your email address","initial":{"instruction_type":"normal","instruction_message":"Enter your email address"},"success":{"instruction_type":"success","instruction_message":"Enter your email address"},"blank":{"instruction_type":"error","instruction_message":"Enter your email address"},"not_an_email_address":{"instruction_type":"error","instruction_message":"Make sure you have entered a valid email address"}},"note_with_tip":{"placeholder_text":"Your note here...","initial":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"empty":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"not_empty_initial":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"saving":{"instruction_type":"normal","instruction_message":"Saving note..."},"success":{"instruction_type":"success","instruction_message":"Note successfully saved!"},"error":{"instruction_type":"error","instruction_message":"Unable to save note note at this time. Please try again."}},"email_for_login_code":{"placeholder_text":"Your email address","initial":{"instruction_type":"normal","instruction_message":"Enter your email to log in."},"success":{"instruction_type":"success","instruction_message":"Enter your email to log in."},"blank":{"instruction_type":"error","instruction_message":"Enter your email to log in."},"empty":{"instruction_type":"error","instruction_message":"Enter your email to log in."}},"login_code":{"initial":{"instruction_type":"normal","instruction_message":"Check your email and enter the login code."},"success":{"instruction_type":"success","instruction_message":"Check your email and enter the login code."},"blank":{"instruction_type":"error","instruction_message":"Check your email and enter the login code."},"empty":{"instruction_type":"error","instruction_message":"Check your email and enter the login code."}},"stripe_all_in_one":{"initial":{"instruction_type":"normal","instruction_message":"Enter your credit card details here."},"empty":{"instruction_type":"error","instruction_message":"Enter your credit card details here."},"success":{"instruction_type":"normal","instruction_message":"Enter your credit card details here."},"invalid_number":{"instruction_type":"error","instruction_message":"The card number is not a valid credit card number."},"invalid_expiry_month":{"instruction_type":"error","instruction_message":"The card's expiration month is invalid."},"invalid_expiry_year":{"instruction_type":"error","instruction_message":"The card's expiration year is invalid."},"invalid_cvc":{"instruction_type":"error","instruction_message":"The card's security code is invalid."},"incorrect_number":{"instruction_type":"error","instruction_message":"The card number is incorrect."},"incomplete_number":{"instruction_type":"error","instruction_message":"The card number is incomplete."},"incomplete_cvc":{"instruction_type":"error","instruction_message":"The card's security code is incomplete."},"incomplete_expiry":{"instruction_type":"error","instruction_message":"The card's expiration date is incomplete."},"incomplete_zip":{"instruction_type":"error","instruction_message":"The card's zip code is incomplete."},"expired_card":{"instruction_type":"error","instruction_message":"The card has expired."},"incorrect_cvc":{"instruction_type":"error","instruction_message":"The card's security code is incorrect."},"incorrect_zip":{"instruction_type":"error","instruction_message":"The card's zip code failed validation."},"invalid_expiry_year_past":{"instruction_type":"error","instruction_message":"The card's expiration year is in the past"},"card_declined":{"instruction_type":"error","instruction_message":"The card was declined."},"missing":{"instruction_type":"error","instruction_message":"There is no card on a customer that is being charged."},"processing_error":{"instruction_type":"error","instruction_message":"An error occurred while processing the card."},"invalid_request_error":{"instruction_type":"error","instruction_message":"Unable to process this payment, please try again or use alternative method."},"invalid_sofort_country":{"instruction_type":"error","instruction_message":"The billing country is not accepted by SOFORT. Please try another country."}}}},"fetched_oembed_html":false}
3 Replies to “How Many Teams Compliance Records Are in Your Tenant?”