Table of Contents
Debugging FUD
A reader of this blog contacted me to discuss a report they received from a well-known software vendor. The report advances six reasons why Office 365 backups are critical. I was asked what I thought of the arguments. After reading the document, my answer was “not much.”
The document makes the bold claim that:
“The misconception that Microsoft fully backs up your data on your behalf is quite common, and without a shift in mindset, could have damaging repercussions when this responsibility is left unattended.”
I’m not sure where this misconception exists. Microsoft is quite clear that, apart from SharePoint Online, they do not perform Office 365 backups. Instead, Microsoft depends on functionality built into the software (like Exchange Online’s Native Data Protection) coupled with hardware resilience to protect customer data. It has been this way since Microsoft launched Office 365 in June 2011.
Nevertheless, projecting the view that it’s important to backup data is goodness for backup vendors, especially when they can infuse customers with the feeling that they have a responsibility to backup their Office 365 data (because Microsoft won’t). Let’s review the reasons advanced to justify Office 365 backups.
The Hazards of the Recycle Bin
The document proclaiming the need for Office 365 backup begins its discussion about the six reasons by focusing on SharePoint Online and the 93-day standard processing cycle for deleted items through its recycle bin The report notes that “the average length of time from data compromise to discovery is over 140 days” and, somewhat breathlessly, says that this is “a shockingly large gap.” (assuming the gap is the 57 days difference).
This is good example of classic FUD. A picture is painted to tell customers that they might lose important data and the only solution is to deploy software that the vendor just happens to have available. What’s ignored is the simple fact that the tenant might have deployed retention policies to ensure that important information is kept for much longer. The author doesn’t seem to have heard about the SharePoint Preservation Hold Library, where documents coming under the scope of retention policies are kept safe and available for eDiscovery and recovery (if necessary). The downside of using retention policies for documents is that data held in the Preservation Hold Library is charged against the tenant’s SharePoint storage quota. Normally this isn’t a problem, but it might force the tenant to buy more storage from Microsoft.
Tenants do get compromised and data can be stolen by attackers, but tenants can do an awful lot to help prevent compromise. Making sure that accounts are protected with multi-factor authentication for a start, removing the ability for clients to connect using basic authentication, and deploying conditional access policies are three steps to halt attackers in their steps. Microsoft is helping by removing basic authentication for protocols like POP3, IMAP4, Exchange Web Services, Exchange ActiveSync, and PowerShell by mid-2021, but tenants can do this now without waiting.
Update: The deprecation of basic authentication for the email connection protocols starts in October 2022.
Accidental Deletion
I wonder what kind of technical review the document through. For instance, accidental deletion is the report’s #1 reason why backups are needed. I’d be more inclined to believe that this is a problem if I didn’t see text like “If you delete a user, whether you meant to or not, that deletion is replicated across the network, along with the deletion of their personal SharePoint site and their OneDrive data.” The focus here is on SharePoint Online, but it’s confused. OneDrive for Business is a user’s personal site, and the workflow used to remove accounts allows an administrator to assign access to the deleted user’s OneDrive for Business account as part of the process. If you want to keep someone’s mailbox, put it on hold before deleting the account and Exchange Online will make the mailbox inactive and keep it until the hold expires. In any case, for all account deletions, even if no hold exists, you can recover the account for up to 30 days following the deletion, which is never mentioned. Again, more FUD.
The authors use a splendid turn of phrase when they state, “Office 365 has geo-redundantly deleted the data forever.” I can’t find what geo-redundantly means, but it seems like they’re making a lot of the fact that user data is removed along with their account (subject to the 30-day recovery period for deleted accounts).
Later, discussion turns to the “Recoverable Items mailbox,” which is a folder rather than a mailbox. They forget to tell people that Exchange Online automatically increases the storage quota for Recoverable Items to 100 GB when mailboxes are on hold to ensure that email held by retention policies can be kept online and available.
Retention Policy Gaps and Confusion
Reason #2 is “Retention policy gaps and confusion.” Retention planning and execution can be confusing unless it’s planned, especially when several Office 365 workloads are involved, but let’s investigate further. The report says: “Office 365 has limited backup and retention policies that can only fend off situational data loss.”
Leaving aside the odd phrasing of “fending off situational data loss,” the fact is that SharePoint Online is the only Office 365 workload that takes backups, so they’re right about limited backup. But that’s by a deliberate design decision to rely on the ability of software to construct robust data protection schemes, like Exchange Native Data Protection.
Microsoft’s decision to use Native Data Protection for Exchange Online is rooted in practicality. In general, the amount of data generated by Office 365 applications would create significant challenges and cost if Microsoft were to attempt to take backups. In the case of Exchange Online , consider how long it would take and how much storage is needed to back up the mailboxes of 345 million Office 365 users when the average size of a cloud mailbox is much larger than its on-premises counterpart due to larger mailbox quotas, the recoverable items quota, and expandable archives. Lacking Office 365 backups is only a weakness in the eyes of backup vendors; it can be a valid and defendable choice for tenants who elect to exploit the features incorporated in the software.
The report says: “In the case of a catastrophic issue, a backup solution can provide the ability to roll back to a previous point-in-time prior to this issue and saving the day.” I wonder if this statement is provable. If Office 365 suffers a catastrophic failure, what is the restore target? Will the customer have on-premises servers to restore data to? Or can they move data to another platform? The multiple datacenters within Office 365 regions means that it would take a truly catastrophic incident to remove service from tenants for more than a working day – and the history of Office 365 operations proves this to be the case.
At what point during an incident would a tenant decide that it’s time to restore back to a point in time? And what data can be restored? Teams is cloud-only, as is Planner and Yammer, so what happens for these applications – where their data be restored to in such a way that users can work with the data? Email and documents can be restored to another location, but that creates other problems like client reconfigurations and email and documents protected with sensitivity labels where users might not be able to authenticate to access the content. While micro-level restores (like a single message, document, or mailbox) are eminently possible if Office 365 is operational, the challenge of restoring all mailboxes (including archives) and all sites for a complete tenant is more challenging. In most cases, the best advice for an Office 365 tenant experiencing an outage is to wait and let Microsoft fix the problem and resume normal operations.
I appreciate that the availability of backups could allow granular restores or point in time restores of objects like an Exchange Online mailbox, a SharePoint Online site, or a OneDrive for Business account, and can be valuable in scenarios like a ransomware attack. Outside the base workloads, all bets are off because no backup and restore APIs exist for applications like Teams and Planner, a fact that no ISV cares to mention. To be fair, Planner data is easier to deal with because it’s reasonably limited in quantity and a Planner Graph API exists that can extract and rewrite data if needed.
The Teams Question
Backing up the Teams compliance records created in Exchange Online is not a Teams backup, no matter how loudly a vendor proclaims this to be true. Including these records along with other mailbox data is an imperfect and fundamentally flawed answer; the records are incomplete and can’t be restored into Teams chats or channels.
Some backup vendors, like Code Two, say that they backup Teams data and then clarify that they’re only referring to the SharePoint Online, OneDrive for Business, and Exchange Online (calendar) data. Although this is better than leading customers to believe that Teams chat and channel conversations are backed up, I prefer the clarity of other vendors like Spanning, who concentrate on the well-known and fully understood challenge of backing up Exchange Online, SharePoint Online, and OneDrive for Business.
Update: It’s been pointed out to me that Keepit uses the beta migration API for Teams to extract channel messages for backup and to restore messages back. This is the same API used by tenant to tenant migration products like those created by Quest, AvePoint, and BitTitan to move Teams data from one tenant to another and has the same problems, such as no support for personal and group chats and complete threads being written into channels instead of separate topics and replies. It’s a better solution than depending on Teams compliance records, but it’s not true backup and restore.
Internal Security Threats
Internal security threats is reason #3. It’s true that someone with administrative permissions could attempt to remove or compromise data before they leave an organization. The “rogue admin” scenario is much beloved of backup companies, but the threat is less in the cloud than it is on-premises. Retention policies (including those that can’t be amended by administrators) can ameliorate the potential effect of someone who deletes all around them before they are escorted off the premises and features like Privileged Access Management can moderate the ability of administrators to wreak havoc on mailboxes. And confidential information can be protected against casual browsing by administrators by encrypting them with sensitivity labels.
None of these features is available on-premises, which is why they might be overlooked. Another thing is that isn’t available on-premises is a comprehensive audit log to capture details of what someone does. It exists in the cloud as the Office 365 audit log.
The author’s worry about “an employee strategically deleting incriminating emails or files” is easily addressed by deploying retention policies. If an employee, a group, sites, or teams come within the scope of retention policies, any attempt to remove incriminating evidence will seem successful, but copies of the items will be kept and available for investigators to find.
External Security Threats
The prospect of users downloading infected files or succumbing to a phishing attack is very real. User training helps, as do good mail hygiene defences like Advanced Threat Protection or an equivalent email cleansing service, but threat of infection exists and can have horrible consequences. It’s worth noting here that SharePoint Online and OneDrive for Business both can restore files up to 30 days back and can cure an infection in this manner. Problems have been noted with these restores, but Microsoft has improved how the feature works since. No other Office 365 app has a point in time restore capability.
The report reassures customers that reason #4 for backups will help, saying, ”Regular backups will help ensure a separate copy of your data is uninfected and that you can recover quickly.” This statement holds true if you can restore a backup to a location where the uninfected data can be recovered. I’m happy that this works well for Exchange Online and SharePoint Online; I wonder what will happen with Teams, Planner, Yammer, and so on.
Legal and Compliance Requirements
The report raises the need to be able to retrieve information to satisfy legal or compliance requirements as reason #5. They note: “Microsoft has built in a couple safety nets, (Litigation Hold) but again, these are not a robust backup solution capable of keeping your company out of legal trouble” and trot out the prospect (again) of losing SharePoint data when a user account is deleted.
Litigation hold is an Exchange construct where complete mailboxes are put on hold. The feature works on both on-premises and cloud platforms. Office 365 retention policies and in-place holds are much more powerful and comprehensive than simple litigation holds and cover Exchange Online, SharePoint Online, OneDrive for Business, Teams, Microsoft 365 Groups, and Skype for Business Online. I guess the authors didn’t get the memo about retention policies. Or they didn’t read it.
Managing Hybrid Email Deployments
There’s not much to say about reason #6, which is that it is an advantage to use the same backup tools for on-premises and cloud email platforms, saying: “The right Office 365 backup solution should be able to handle hybrid email deployments, and treat exchange [sic] data the same, making the source location irrelevant.”
The text comingles Exchange with Office 365. Exchange Online is the biggest workload in Office 365 but it’s not Office 365. This problem of assuming that what works for an individual workload applies across a huge suite exhibits a lack of awareness of the integrated nature of the service. On-premises deployments focus on individual applications like Exchange or SharePoint, but Office 365 is more integrated, broader, more complex, and its data is less accessible to ISVs. I understand how people can take their on-premises experience and transpose it to Office 365, but it’s not right. A different attitude, more knowledge, and a deep understanding of how technologies are interconnected in the cloud is needed, and that just isn’t evident in this document.
Confusing Office 365 with a Limited Application Set
It seems that backup vendors often refer to Office 365 when they really mean Exchange Online and SharePoint Online (including OneDrive for Business), which are the workloads they can process. Claims are advanced for Microsoft 365 Groups, which is fine when you look through the narrow lens of being able to process the group mailbox and team site. Things get more problematic for apps like Teams, Yammer, and Planner. Microsoft hasn’t delivered suitable APIs to allow vendors to build backup and restore tools for these apps, so ISVs cannot be blamed for being unable to stream data from these apps to suitable backup locations.
It’s important that vendors tell potential customers exactly what their software can do. Regretfully, a lack of precision and accuracy is often evident in the narrative spun by backup vendors.
Survey Data Not Informative
A survey of customers is cited as justification that backups are essential. But no detail is available about the experience of the 1,000 IT Professionals surveyed. The assertion is that these people had suffered some data loss in the cloud, but we don’t know about what cloud platforms are involved. The implication is that the problems were with Office 365, but without probing the exact circumstances of the loss for each of the individuals surveyed, we don’t know. If the loss was related to Office 365, we don’t know what applications were involved or what licenses the tenants owned (Office 365 E3 is a base requirement for retention policies and E5 brings a lot more protection). The survey is just FUD.
Further solace is sought from a statement taken from an IDC study (May 2019 – possibly funded by the vendor) that six out of ten organizations don’t have any form of Office 365 data protection sounds horrible, but we don’t know if these are tenants operating in the small to medium businesses, education, or enterprise sectors, so we don’t know what functionality is available to these tenants. I’d be more worried if I knew that six of ten Office 365 enterprise tenants with E3 or E5 licenses had no protection, but my experience is that this is not the case. Lack of protection is usually a problem with small tenants, not large ones.
Saying that organizations don’t have “any form of protection” is an inaccurate claim because tenants automatically gain protection for Exchange (Native Data Protection), SharePoint Online (backups) and OneDrive for Business (restore my files), not to mention inactive mailboxes, retention policies, etc. I also note that the IDC study is focused on email and collaboration (aka Exchange and SharePoint) and doesn’t go near the problematic area of other Office 365 applications where backup isn’t possible. Furthermore, leading statements like “Microsoft offers a 90-day retention policy” don’t offer any evidence for such a conclusion (what data, what app, under what conditions?) and undermine the credibility of the report.
Poor, Badly Argued, Incomplete
I am underimpressed at the quality of the arguments advanced in the report. Poor technical awareness and understanding of the functionality available to Office 365 enterprise tenants is evident in the attempt to convince customers that backups are a necessity. That’s not true. Backups are something that you should consider after thoroughly evaluating the threats facing your tenant and assessing how to mitigate threats using Office 365 standard functionality.
Streaming information to a backup location is a well-known science for Exchange Online and SharePoint Online. Restoring data is where backup vendors differentiate their offerings and earn their money. If you decide to use a backup service, ask questions about just what data is covered and how realistic it is to restore data after a catastrophic outage. If the vendor can’t come up with compelling and evidence-based answers, try someone else.
Upgrading to more advanced (and expensive) Office 365 licenses might be a better investment than committing to a third-party backup service, especially when you consider the problems of a) backing up several Office 365 apps and b) the difficulty of restoring data (apart from email and documents) to some other platform should a disaster happen, unlikely as that might be.
Note: AvePoint’s view on the issues raised in this article are available here.
Want more practical tips about the administration of Office 365? Subscribe to the Office 365 for IT Pros eBook.