Search for What Participants Say During Teams Meetings
Microsoft message center notification MC260749 (last updated August 12) titled Microsoft Search: Find a meeting recording based on what was said is both technically interesting and important. Described in Microsoft 365 roadmap item 82003, the roll-out was delayed several times, but the way is now clear for Office 365 tenants to be able to search videos using spoken text along with a bunch of other changes to make Teams meeting recordings more accessible and useful. While it’s hard to say exactly when individual tenants will have all the functionality described here, I expect worldwide deployment to be complete by the end of October 2021.
Everything in OneDrive
Exposing the content of meeting recordings for search is important because it starts the process to close a major compliance gap. Up to now, transcripts for online meetings have not been searchable. The problem first surfaced when Teams stored its recordings in Stream. When the meeting finished, Stream processed the recording and created the transcript. However, the transcript remained in the Stream Azure service and was inaccessible to Microsoft Search. If something can’t be indexed by Microsoft Search, its content cannot be found by a search.
To fill the functionality gap, Microsoft introduced a transcription capability for Teams meeting recordings (a recent update means that if you record a Teams meeting now, you generate a transcript automatically). However, the issue of searchability remained. Because ODSP stores the recording files, Search could index file metadata like the name of the recording, but that’s about all.
The gap in indexing and searchability is now closed. Teams stores the spoken text captured during a meeting (including speaker attribution so you know who said what) and meeting metadata in the Exchange Online mailbox of the meeting organizer. Capturing the spoken text in mailboxes allows Microsoft Search to index the data and therefore makes it possible for searches to find this information. And as we’ll see, ODSP also holds a copy of the transcript to allow the words in the transcript to connect with segments in a meeting recording.
Exchange Mailbox Storage for Transcript Information
Teams stores transcript information in a folder called ApplicationDataRoot/93c8660e-1330-4e40-8fda-fd27f9eafe10/MeetingTranscriptCollection in the non-IPM part of the mailbox. Hidden means that the folder isn’t available to users through clients like Outlook, but its contents are available to administrative interfaces like Microsoft Search and programs like MFCMAPI.
Transcripts are captured as mail items. Examining the captured items with MFCMAPI, it looks like two properties for are most interesting:
TranscriptJsonBlob: stores the spoken text captured during the meeting. In Figure 1, you can see some captured text, including the name of the speaker. When users view the transcript in Teams, the information is displayed in a nicer format. It’s also possible to download transcripts in VTT or Word (DOCX) format.
TranscriptMetadataJsonBlob: stores metadata about the call.
Figure 1: Spoken text from a Teams meeting transcript stored in Exchange Online
Linking Words to Videos
The original implementation for Teams meeting recordings stored in Stream classic supported transcription, including the ability to edit the transcript to correct obvious errors. To allow Microsoft Search to find the MP4 file for a meeting recording based on words spoken during a meeting, a background process copies the transcript data captured in Exchange Online and indexes it against the recording to match segments of the video with the spoken words.
Replication of transcript data from Exchange Online to ODSP can take anything from 15 minutes to a day after the meeting ends. Once the process completes, you can search for text spoken in meetings and find recordings using the transcript (Figure 2).
Figure 2: Microsoft Search finds spoken text in a Teams meeting recording
Transcript Playback
Matching words in the transcript with meeting recordings (and eventually, any video stored in ODSP) allows concurrent playback for the two elements. Microsoft 365 roadmap item 82057, rolling out in September 2021, delivers a transcript pane for video playback (Figure 3). No ability is yet available for a video owner to edit the transcript.
Figure 3: Viewing a transcript of a Teams meeting recording alongside the video
Curiously, closed captions are available for only 60 days from the date of recording. In addition, Microsoft says that “Closed captions aren’t fully supported” if you move or copy a recording from its original location. Presumably, this is because the move might affect the link to the transcript data.
Making Transcription Available to More Teams Users
The option to transcribe meetings used to be restricted to accounts with enterprise E3/E5 and Business Premium/Standard. In early July, Microsoft made live transcription available (MC260564) for other licenses, including the E1, F1, academic, and Business basic SKUs, noting that this step improves the accessibility of Teams and makes meetings more inclusive for those who are hard of hearing. Microsoft followed up with MC280258 (August 24), to announce support for transcripts and captions in 27 additional languages (Figure 4) to join the previous support for U.S. English.
Figure 4: The new languages supported by Teams meeting recordings
Another Compliance Gap Nearly Closed
All the information shared during Teams meetings is gradually coming within the scope of compliance policies. eDiscovery can already find chat, presentations, and documents, and the advent of indexed speech means that spoken comments should soon come within the scope of eDiscovery searches. This hasn’t happened yet, probably because of the work needed to export transcripts and videos in eDiscovery cases, but I am sure this capability is high on Microsoft’s agenda.
Although the captured text is sometimes inaccurate, capturing any record of spoken comments is better than nothing. As time goes by, the artificial intelligence technology used to analyze speech to create the transcript will improve in terms of accuracy and ability to handle accents.
So much change, all the time. It’s a challenge to stay abreast of all the updates Microsoft makes across Office 365. Subscribe to the Office 365 for IT Pros eBook to receive monthly insights into what’s happening.
{"id":null,"mode":"button","open_style":"in_modal","currency_code":"EUR","currency_symbol":"\u20ac","currency_type":"decimal","blank_flag_url":"https:\/\/office365itpros.com\/wp-content\/plugins\/tip-jar-wp\/\/assets\/images\/flags\/blank.gif","flag_sprite_url":"https:\/\/office365itpros.com\/wp-content\/plugins\/tip-jar-wp\/\/assets\/images\/flags\/flags.png","default_amount":100,"top_media_type":"featured_image","featured_image_url":"https:\/\/office365itpros.com\/wp-content\/uploads\/2022\/11\/cover-141x200.jpg","featured_embed":"","header_media":null,"file_download_attachment_data":null,"recurring_options_enabled":true,"recurring_options":{"never":{"selected":true,"after_output":"One time only"},"weekly":{"selected":false,"after_output":"Every week"},"monthly":{"selected":false,"after_output":"Every month"},"yearly":{"selected":false,"after_output":"Every year"}},"strings":{"current_user_email":"","current_user_name":"","link_text":"Virtual Tip Jar","complete_payment_button_error_text":"Check info and try again","payment_verb":"Pay","payment_request_label":"Office 365 for IT Pros","form_has_an_error":"Please check and fix the errors above","general_server_error":"Something isn't working right at the moment. Please try again.","form_title":"Office 365 for IT Pros","form_subtitle":null,"currency_search_text":"Country or Currency here","other_payment_option":"Other payment option","manage_payments_button_text":"Manage your payments","thank_you_message":"Thank you for supporting the work of Office 365 for IT Pros!","payment_confirmation_title":"Office 365 for IT Pros","receipt_title":"Your Receipt","print_receipt":"Print Receipt","email_receipt":"Email Receipt","email_receipt_sending":"Sending receipt...","email_receipt_success":"Email receipt successfully sent","email_receipt_failed":"Email receipt failed to send. Please try again.","receipt_payee":"Paid to","receipt_statement_descriptor":"This will show up on your statement as","receipt_date":"Date","receipt_transaction_id":"Transaction ID","receipt_transaction_amount":"Amount","refund_payer":"Refund from","login":"Log in to manage your payments","manage_payments":"Manage Payments","transactions_title":"Your Transactions","transaction_title":"Transaction Receipt","transaction_period":"Plan Period","arrangements_title":"Your Plans","arrangement_title":"Manage Plan","arrangement_details":"Plan Details","arrangement_id_title":"Plan ID","arrangement_payment_method_title":"Payment Method","arrangement_amount_title":"Plan Amount","arrangement_renewal_title":"Next renewal date","arrangement_action_cancel":"Cancel Plan","arrangement_action_cant_cancel":"Cancelling is currently not available.","arrangement_action_cancel_double":"Are you sure you'd like to cancel?","arrangement_cancelling":"Cancelling Plan...","arrangement_cancelled":"Plan Cancelled","arrangement_failed_to_cancel":"Failed to cancel plan","back_to_plans":"\u2190 Back to Plans","update_payment_method_verb":"Update","sca_auth_description":"Your have a pending renewal payment which requires authorization.","sca_auth_verb":"Authorize renewal payment","sca_authing_verb":"Authorizing payment","sca_authed_verb":"Payment successfully authorized!","sca_auth_failed":"Unable to authorize! Please try again.","login_button_text":"Log in","login_form_has_an_error":"Please check and fix the errors above","uppercase_search":"Search","lowercase_search":"search","uppercase_page":"Page","lowercase_page":"page","uppercase_items":"Items","lowercase_items":"items","uppercase_per":"Per","lowercase_per":"per","uppercase_of":"Of","lowercase_of":"of","back":"Back to plans","zip_code_placeholder":"Zip\/Postal Code","download_file_button_text":"Download File","input_field_instructions":{"tip_amount":{"placeholder_text":"How much would you like to tip?","initial":{"instruction_type":"normal","instruction_message":"How much would you like to tip? Choose any currency."},"empty":{"instruction_type":"error","instruction_message":"How much would you like to tip? Choose any currency."},"invalid_curency":{"instruction_type":"error","instruction_message":"Please choose a valid currency."}},"recurring":{"placeholder_text":"Recurring","initial":{"instruction_type":"normal","instruction_message":"How often would you like to give this?"},"success":{"instruction_type":"success","instruction_message":"How often would you like to give this?"},"empty":{"instruction_type":"error","instruction_message":"How often would you like to give this?"}},"name":{"placeholder_text":"Name on Credit Card","initial":{"instruction_type":"normal","instruction_message":"Enter the name on your card."},"success":{"instruction_type":"success","instruction_message":"Enter the name on your card."},"empty":{"instruction_type":"error","instruction_message":"Please enter the name on your card."}},"privacy_policy":{"terms_title":"Terms and conditions","terms_body":null,"terms_show_text":"View Terms","terms_hide_text":"Hide Terms","initial":{"instruction_type":"normal","instruction_message":"I agree to the terms."},"unchecked":{"instruction_type":"error","instruction_message":"Please agree to the terms."},"checked":{"instruction_type":"success","instruction_message":"I agree to the terms."}},"email":{"placeholder_text":"Your email address","initial":{"instruction_type":"normal","instruction_message":"Enter your email address"},"success":{"instruction_type":"success","instruction_message":"Enter your email address"},"blank":{"instruction_type":"error","instruction_message":"Enter your email address"},"not_an_email_address":{"instruction_type":"error","instruction_message":"Make sure you have entered a valid email address"}},"note_with_tip":{"placeholder_text":"Your note here...","initial":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"empty":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"not_empty_initial":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"saving":{"instruction_type":"normal","instruction_message":"Saving note..."},"success":{"instruction_type":"success","instruction_message":"Note successfully saved!"},"error":{"instruction_type":"error","instruction_message":"Unable to save note note at this time. Please try again."}},"email_for_login_code":{"placeholder_text":"Your email address","initial":{"instruction_type":"normal","instruction_message":"Enter your email to log in."},"success":{"instruction_type":"success","instruction_message":"Enter your email to log in."},"blank":{"instruction_type":"error","instruction_message":"Enter your email to log in."},"empty":{"instruction_type":"error","instruction_message":"Enter your email to log in."}},"login_code":{"initial":{"instruction_type":"normal","instruction_message":"Check your email and enter the login code."},"success":{"instruction_type":"success","instruction_message":"Check your email and enter the login code."},"blank":{"instruction_type":"error","instruction_message":"Check your email and enter the login code."},"empty":{"instruction_type":"error","instruction_message":"Check your email and enter the login code."}},"stripe_all_in_one":{"initial":{"instruction_type":"normal","instruction_message":"Enter your credit card details here."},"empty":{"instruction_type":"error","instruction_message":"Enter your credit card details here."},"success":{"instruction_type":"normal","instruction_message":"Enter your credit card details here."},"invalid_number":{"instruction_type":"error","instruction_message":"The card number is not a valid credit card number."},"invalid_expiry_month":{"instruction_type":"error","instruction_message":"The card's expiration month is invalid."},"invalid_expiry_year":{"instruction_type":"error","instruction_message":"The card's expiration year is invalid."},"invalid_cvc":{"instruction_type":"error","instruction_message":"The card's security code is invalid."},"incorrect_number":{"instruction_type":"error","instruction_message":"The card number is incorrect."},"incomplete_number":{"instruction_type":"error","instruction_message":"The card number is incomplete."},"incomplete_cvc":{"instruction_type":"error","instruction_message":"The card's security code is incomplete."},"incomplete_expiry":{"instruction_type":"error","instruction_message":"The card's expiration date is incomplete."},"incomplete_zip":{"instruction_type":"error","instruction_message":"The card's zip code is incomplete."},"expired_card":{"instruction_type":"error","instruction_message":"The card has expired."},"incorrect_cvc":{"instruction_type":"error","instruction_message":"The card's security code is incorrect."},"incorrect_zip":{"instruction_type":"error","instruction_message":"The card's zip code failed validation."},"invalid_expiry_year_past":{"instruction_type":"error","instruction_message":"The card's expiration year is in the past"},"card_declined":{"instruction_type":"error","instruction_message":"The card was declined."},"missing":{"instruction_type":"error","instruction_message":"There is no card on a customer that is being charged."},"processing_error":{"instruction_type":"error","instruction_message":"An error occurred while processing the card."},"invalid_request_error":{"instruction_type":"error","instruction_message":"Unable to process this payment, please try again or use alternative method."},"invalid_sofort_country":{"instruction_type":"error","instruction_message":"The billing country is not accepted by SOFORT. Please try another country."}}}},"fetched_oembed_html":false}
5 Replies to “How Microsoft Search Finds Spoken Text in Teams Meeting Transcripts”