Azure AD My Sign-Ins Activity Report Now Generally Available

Check Your Azure AD Sign-Ins for Problems

Microsoft VP Alex Simons exclaimed himself to be excited that “End users can now report “This wasn’t me” for unusual sign-in activity” when he announced the general availability of Azure AD My Sign-Ins, a new page that allows users to see details of their sign-in activity for Azure AD and detect:

• If anyone is trying to guess their password.
• If an attacker successfully signed into their account from a strange location. (in other words, somewhere that the user knows they weren’t at the time).
• What apps the attacker accessed.

Accessing My Sign-ins

Although people can use the URL for the My Sign-ins page, most Office 365 users will probably find it through their My Account page (Figure 1), which is accessed when a user selects the View account option after clicking their avatar in an Office 365 app.

When we covered the feature in preview in October 2019, we suggested that tenants create a custom tile to expose My Sign-Ins in the Office 365 menu. That still seems like a good idea to make this data more accessible to users.

Understanding the Data

Microsoft’s hope is that users will review the sign-in data to discover problems that can then be reported internally. But when you review data, you need to understand what the data means. As the Sign-in Activity page says, “You should recognise each of these recent activities.”

I looked at my recent sign-in activities (Figure 1). Each sign-in is described in terms of:

• Time.
• Location.
• Operating System
• Browser.
• Client IP address.
• App.
• Account.
• Sign-in status (if you see lots of unsuccessful attempts to sign into your account, a problem exists, even if it’s just your inability to type your password).

Issues with Sign-in Data

The first issue is that some client activity is not browser based. I couldn’t find a way to easily identify sign-ins from Outlook desktop or Outlook mobile. The next is that the app names used bear little or no relation to what users think of the applications they use. I found the following names:

• Office Online Client AAD- Loki
• Office Online Client AAD- Augmentation Loop
• SharePoint Online Web Client Extensibility
• Skype Web Experience On Office 365
• ProjectWorkManagement
• Office 365

I think ProjectWorkManagement is Planner. I don’t use Skype for Business Online, so I have no idea what the Skype Web Experience is, nor what I was doing to extend SharePoint Online. There’s no evidence of OWA, which I use heavily, nor do I see any mention of Teams in the records I reviewed going back over the last week. Yet anyone who has ever examined evidence of Teams sign-in activity in the Office 365 audit log knows that a record is logged hourly when accounts are active. I can’t account for why these events aren’t easily discernible in this data. Maybe OWA and Teams are in the catch-all “Office 365” app.

Some apps do a good job of identifying themselves, like the Graph Explorer and the Microsoft 365 Security and Compliance Center. These easily-understood and relevant names leave no doubt in user minds as to what they signed into.

I think the root cause here is that the internal strings used by Office 365 applications to identity themselves when they authenticate with Azure AD were never designed to be exposed to end users. To be useful, the strings need to be changed to clearly indicate the application in user terms, not internal Microsoft speak. This is likely to take some time to achieve, but if they cannot interpret and understand the data, users cannot safely attest that they signed into an app at such a time from a certain location.

Goodness but Maybe Still Not of Great Interest to Users

Giving people the tools to understand when their account might be compromised is goodness, but it depends on users knowing how to check their sign-in history and being able to detect anomalous activity based on what they see on the page. And of course, users must be interested in checking the data too, something that’s hard to assure when so many demands exist on time.

Microsoft responded to some of the criticisms exposed by the preview, notably by highlighting suspicious activities at the top of the list and making the feature more accessible through users’ Office 365 profiles. However, I still fear that this is another Azure AD feature that will only be used by technologists, which might not be the cohort that Microsoft is trying to help.

---

The general availability of an Azure AD feature is a small but important change for Office 365 tenants. Don’t miss out on detail. Subscribe to the Office 365 for IT Pros eBook and let us help you to stay current.