Azure AD Authentication Failure Stops Users Working
By now, you’ve probably heard about the second large Azure AD authentication outage since September. The March 15 incident calmed down after a few hours, but while it was ongoing users were unable to connect to Microsoft 365 applications when authentication was necessary. It wasn’t a happy experience. Microsoft plans to set a new SLA of 99.99% availability for Azure AD authentication on April 1, 2021. Perhaps they were making a few tweaks to the Azure AD infrastructure to prepare the ground for the upgraded SLA when things went wrong.
The current 99.9% SLA applies to the Azure AD tier for Office 365, but a Microsoft comment posted to the announcement for the new SLA said that the 99.99% level will only apply to those with Azure AD Premium licenses. I guess we shall have to wait and see the details of the SLA when Microsoft publishes the text of the agreement on April 1.
Microsoft 365 applications continued working during the outage unless authentication was necessary. Because they’re built on the Microsoft Graph APIs, the Teams clients authenticate hourly, so they were heavily affected. Outlook desktop stayed online throughout, and users reported varying degrees of useability for other apps.
Working in Word
While the outage progressed, I worked on a Word document for my blog post. All my Word documents are either in SharePoint Online document libraries or OneDrive for Business, so the OneDrive sync client is kept busy. The sync client is responsible for the differential synchronization of files up to the new 250 GB limit. Office apps autosave to capture changes. Not only does autosave ensure that you should never lose much if an app or workstation crashes, it’s also way changes get to other copies of Office documents open for co-authoring. And it’s why SharePoint Online keeps a minimum of 100 versions of documents. If you use the Office desktop apps heavily and store files online, the OneDrive sync client is busy.
OneDrive Sync Client Goes Nuts
Until that is, the OneDrive sync client decides that it should remove all the local copies of files from a SharePoint folder. This was a rather bizarre side effect of the Azure AD outage. At least, although I can’t prove that the outage caused the OneDrive sync client to do something very strange, the problem happened at the same time.
I noticed the issue when File Explorer reported nothing in the local folder which holds the synchronized copies of SharePoint files. The folder usually holds hundreds of files (423 as I write), so something had clearly happened. I opened the OneDrive sync client (build 21.041.0228.0001) and discovered that the client had removed the local files an hour ago (Figure 1), meaning that the client decided to remove the files at around 21:45 UTC, during the period when Microsoft was rolling out remediation for the Azure AD outage.
Figure 1: The OneDrive sync client removes a bunch of files
The problem was easily fixed by going to SharePoint Online and choosing to synchronize the folder again (Figure 2).
Figure 2: Opting to synchronize a SharePoint Online folder
The OneDrive sync client started to download local copies immediately (Figure 3) and a full set of documents was soon on my local drive.
Figure 3: The OneDrive sync client downloads files from SharePoint Online
Curious and Problematic Synchronization
You can argue that all’s well that ends well, but no good reason exists for the OneDrive sync client to do what it did. Perhaps the Azure AD authentication problem caused the client to believe that it was no longer allowed to download files from the SharePoint site. If so, it would be better if the client issued a warning to say what’s about to happen and offered the user a chance to authenticate with their credentials rather than concluding that everything should be removed now.
Failure to authenticate is the logical root cause which lead to the mass deletion of local files. Every document in the folder has a retention label to stop SharePoint removing documents (set as a default label for the library). The normal course of events is that you can remove a local copy of a file from File Explorer only for the OneDrive sync client to restore the file once it discovers the deletion block imposed by the retention label. Despite the presence of the retention labels, the OneDrive sync client removed all the local files. If my theory holds, the OneDrive sync client concluded that the user had no access to SharePoint Online, so it should remove the local copies as this wouldn’t impact the retained file in SharePoint.
What’s also curious is that just one folder was affected. The OneDrive sync client left everything else alone. My conclusion is that the folder was in active use because I had a Word document stored in that folder open at the time, and autosaved changes were flowing back to SharePoint Online. No need existed for the OneDrive sync client to go near my other folders (like those holding files for the Office 365 for IT Pros eBook), so it left them alone.
It’s not just me who has encountered odd synchronization issues leading to mass removal of files. Fellow MVPs Vasil Michev and Paul Robichaux have also had difficulties. It seems like Microsoft has some work to do to smoothen how the OneDrive sync client handles what could be transient authentication issues.
Update March 18: Microsoft has two advisories linked to the problem. SP244708 (SharePoint) and OD244709 (OneDrive). The symptoms experienced by people are different, but the root cause is the same.
4 Replies to “OneDrive Sync Client Has Meltdown During Azure AD Outage”
Tony did the synced folder contain a synced OneNote notebook? I saw similar issues with about 20% of our fleet and I believe the sync was halted/broken on folders that contained a OneNote notebook.
So I have the same issues, but a bit different. One Device got no errors; the other device have some errors. (it was about midnight, so decided to do nothing. I shut down this device. Next day al things are good. No error.
The thing in your blog post is, it was not a real deletion. Because the deletion was not synced to SharePoint. Otherwise the files would not be found on SharePoint Online (only in the recycle bin).
and your last comment is also not correct, because it was not a user action, and therefore die GPO does not work, even it was enabled
The same situation in another scenario:
OneDrive Personal
You buy additional storage, bringing all your personal documents into the Vault (Passport, ID Card, Vaccination, ….)
The automatic scan found a picture with too much skin, (you do not know that)
Microsoft locks your account
Maybe I need access to the Vaccination paper from your vault
But I cannot open your vault, because I’m not able to log in (even if my files include with the vault are store on my local device)
Interesting, standing in the front of my bank, they do not let me in, so no change to reach the vault room, even if I have the key
Microsoft has no process -No telephone number, nothing, to get back to my PAID storage
{"id":null,"mode":"button","open_style":"in_modal","currency_code":"EUR","currency_symbol":"\u20ac","currency_type":"decimal","blank_flag_url":"https:\/\/office365itpros.com\/wp-content\/plugins\/tip-jar-wp\/\/assets\/images\/flags\/blank.gif","flag_sprite_url":"https:\/\/office365itpros.com\/wp-content\/plugins\/tip-jar-wp\/\/assets\/images\/flags\/flags.png","default_amount":100,"top_media_type":"featured_image","featured_image_url":"https:\/\/office365itpros.com\/wp-content\/uploads\/2022\/11\/cover-141x200.jpg","featured_embed":"","header_media":null,"file_download_attachment_data":null,"recurring_options_enabled":true,"recurring_options":{"never":{"selected":true,"after_output":"One time only"},"weekly":{"selected":false,"after_output":"Every week"},"monthly":{"selected":false,"after_output":"Every month"},"yearly":{"selected":false,"after_output":"Every year"}},"strings":{"current_user_email":"","current_user_name":"","link_text":"Virtual Tip Jar","complete_payment_button_error_text":"Check info and try again","payment_verb":"Pay","payment_request_label":"Office 365 for IT Pros","form_has_an_error":"Please check and fix the errors above","general_server_error":"Something isn't working right at the moment. Please try again.","form_title":"Office 365 for IT Pros","form_subtitle":null,"currency_search_text":"Country or Currency here","other_payment_option":"Other payment option","manage_payments_button_text":"Manage your payments","thank_you_message":"Thank you for supporting the work of Office 365 for IT Pros!","payment_confirmation_title":"Office 365 for IT Pros","receipt_title":"Your Receipt","print_receipt":"Print Receipt","email_receipt":"Email Receipt","email_receipt_sending":"Sending receipt...","email_receipt_success":"Email receipt successfully sent","email_receipt_failed":"Email receipt failed to send. Please try again.","receipt_payee":"Paid to","receipt_statement_descriptor":"This will show up on your statement as","receipt_date":"Date","receipt_transaction_id":"Transaction ID","receipt_transaction_amount":"Amount","refund_payer":"Refund from","login":"Log in to manage your payments","manage_payments":"Manage Payments","transactions_title":"Your Transactions","transaction_title":"Transaction Receipt","transaction_period":"Plan Period","arrangements_title":"Your Plans","arrangement_title":"Manage Plan","arrangement_details":"Plan Details","arrangement_id_title":"Plan ID","arrangement_payment_method_title":"Payment Method","arrangement_amount_title":"Plan Amount","arrangement_renewal_title":"Next renewal date","arrangement_action_cancel":"Cancel Plan","arrangement_action_cant_cancel":"Cancelling is currently not available.","arrangement_action_cancel_double":"Are you sure you'd like to cancel?","arrangement_cancelling":"Cancelling Plan...","arrangement_cancelled":"Plan Cancelled","arrangement_failed_to_cancel":"Failed to cancel plan","back_to_plans":"\u2190 Back to Plans","update_payment_method_verb":"Update","sca_auth_description":"Your have a pending renewal payment which requires authorization.","sca_auth_verb":"Authorize renewal payment","sca_authing_verb":"Authorizing payment","sca_authed_verb":"Payment successfully authorized!","sca_auth_failed":"Unable to authorize! Please try again.","login_button_text":"Log in","login_form_has_an_error":"Please check and fix the errors above","uppercase_search":"Search","lowercase_search":"search","uppercase_page":"Page","lowercase_page":"page","uppercase_items":"Items","lowercase_items":"items","uppercase_per":"Per","lowercase_per":"per","uppercase_of":"Of","lowercase_of":"of","back":"Back to plans","zip_code_placeholder":"Zip\/Postal Code","download_file_button_text":"Download File","input_field_instructions":{"tip_amount":{"placeholder_text":"How much would you like to tip?","initial":{"instruction_type":"normal","instruction_message":"How much would you like to tip? Choose any currency."},"empty":{"instruction_type":"error","instruction_message":"How much would you like to tip? Choose any currency."},"invalid_curency":{"instruction_type":"error","instruction_message":"Please choose a valid currency."}},"recurring":{"placeholder_text":"Recurring","initial":{"instruction_type":"normal","instruction_message":"How often would you like to give this?"},"success":{"instruction_type":"success","instruction_message":"How often would you like to give this?"},"empty":{"instruction_type":"error","instruction_message":"How often would you like to give this?"}},"name":{"placeholder_text":"Name on Credit Card","initial":{"instruction_type":"normal","instruction_message":"Enter the name on your card."},"success":{"instruction_type":"success","instruction_message":"Enter the name on your card."},"empty":{"instruction_type":"error","instruction_message":"Please enter the name on your card."}},"privacy_policy":{"terms_title":"Terms and conditions","terms_body":null,"terms_show_text":"View Terms","terms_hide_text":"Hide Terms","initial":{"instruction_type":"normal","instruction_message":"I agree to the terms."},"unchecked":{"instruction_type":"error","instruction_message":"Please agree to the terms."},"checked":{"instruction_type":"success","instruction_message":"I agree to the terms."}},"email":{"placeholder_text":"Your email address","initial":{"instruction_type":"normal","instruction_message":"Enter your email address"},"success":{"instruction_type":"success","instruction_message":"Enter your email address"},"blank":{"instruction_type":"error","instruction_message":"Enter your email address"},"not_an_email_address":{"instruction_type":"error","instruction_message":"Make sure you have entered a valid email address"}},"note_with_tip":{"placeholder_text":"Your note here...","initial":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"empty":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"not_empty_initial":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"saving":{"instruction_type":"normal","instruction_message":"Saving note..."},"success":{"instruction_type":"success","instruction_message":"Note successfully saved!"},"error":{"instruction_type":"error","instruction_message":"Unable to save note note at this time. Please try again."}},"email_for_login_code":{"placeholder_text":"Your email address","initial":{"instruction_type":"normal","instruction_message":"Enter your email to log in."},"success":{"instruction_type":"success","instruction_message":"Enter your email to log in."},"blank":{"instruction_type":"error","instruction_message":"Enter your email to log in."},"empty":{"instruction_type":"error","instruction_message":"Enter your email to log in."}},"login_code":{"initial":{"instruction_type":"normal","instruction_message":"Check your email and enter the login code."},"success":{"instruction_type":"success","instruction_message":"Check your email and enter the login code."},"blank":{"instruction_type":"error","instruction_message":"Check your email and enter the login code."},"empty":{"instruction_type":"error","instruction_message":"Check your email and enter the login code."}},"stripe_all_in_one":{"initial":{"instruction_type":"normal","instruction_message":"Enter your credit card details here."},"empty":{"instruction_type":"error","instruction_message":"Enter your credit card details here."},"success":{"instruction_type":"normal","instruction_message":"Enter your credit card details here."},"invalid_number":{"instruction_type":"error","instruction_message":"The card number is not a valid credit card number."},"invalid_expiry_month":{"instruction_type":"error","instruction_message":"The card's expiration month is invalid."},"invalid_expiry_year":{"instruction_type":"error","instruction_message":"The card's expiration year is invalid."},"invalid_cvc":{"instruction_type":"error","instruction_message":"The card's security code is invalid."},"incorrect_number":{"instruction_type":"error","instruction_message":"The card number is incorrect."},"incomplete_number":{"instruction_type":"error","instruction_message":"The card number is incomplete."},"incomplete_cvc":{"instruction_type":"error","instruction_message":"The card's security code is incomplete."},"incomplete_expiry":{"instruction_type":"error","instruction_message":"The card's expiration date is incomplete."},"incomplete_zip":{"instruction_type":"error","instruction_message":"The card's zip code is incomplete."},"expired_card":{"instruction_type":"error","instruction_message":"The card has expired."},"incorrect_cvc":{"instruction_type":"error","instruction_message":"The card's security code is incorrect."},"incorrect_zip":{"instruction_type":"error","instruction_message":"The card's zip code failed validation."},"invalid_expiry_year_past":{"instruction_type":"error","instruction_message":"The card's expiration year is in the past"},"card_declined":{"instruction_type":"error","instruction_message":"The card was declined."},"missing":{"instruction_type":"error","instruction_message":"There is no card on a customer that is being charged."},"processing_error":{"instruction_type":"error","instruction_message":"An error occurred while processing the card."},"invalid_request_error":{"instruction_type":"error","instruction_message":"Unable to process this payment, please try again or use alternative method."},"invalid_sofort_country":{"instruction_type":"error","instruction_message":"The billing country is not accepted by SOFORT. Please try another country."}}}},"fetched_oembed_html":false}
Tony did the synced folder contain a synced OneNote notebook? I saw similar issues with about 20% of our fleet and I believe the sync was halted/broken on folders that contained a OneNote notebook.
Nope. No OneNote (I never really got into OneNote…). This was an authentication glitch provoked by autosaving…
So I have the same issues, but a bit different. One Device got no errors; the other device have some errors. (it was about midnight, so decided to do nothing. I shut down this device. Next day al things are good. No error.
The thing in your blog post is, it was not a real deletion. Because the deletion was not synced to SharePoint. Otherwise the files would not be found on SharePoint Online (only in the recycle bin).
and your last comment is also not correct, because it was not a user action, and therefore die GPO does not work, even it was enabled
The same situation in another scenario:
OneDrive Personal
You buy additional storage, bringing all your personal documents into the Vault (Passport, ID Card, Vaccination, ….)
The automatic scan found a picture with too much skin, (you do not know that)
Microsoft locks your account
Maybe I need access to the Vaccination paper from your vault
But I cannot open your vault, because I’m not able to log in (even if my files include with the vault are store on my local device)
Interesting, standing in the front of my bank, they do not let me in, so no change to reach the vault room, even if I have the key
Microsoft has no process -No telephone number, nothing, to get back to my PAID storage
The article makes it clear that the deletions were for local files, not in SharePoint.
As to the comment about the first file delete warning, that was a tongue in cheek remark if you understand the English saying…