Measuring KPIs like Response Times for Shared Mailboxes

Shared Mailboxes Are Not CRM Systems: Limited Measurements Are Possible

Although they’re not designed to act like a customer relationship management (CRM), many organizations use shared mailboxes to capture and respond to customer interactions. Shared mailboxes don’t consume Exchange Online licenses unless they use more than the standard 50 GB mailbox quota, have an archive. On the other hand, shared mailboxes might need Microsoft Defender for Office 365 (MDO) licenses if they handle external email. Either way, the cost of shared mailboxes is low compared to the available functionality.

Because shared mailboxes are used to handle customer interactions, questions like this arise (from the Microsoft Technical Community):

“I am looking to create report for management for our KPI. Management want to know how quick teams are replying to email once it’s landed to mailbox. Also, average reply time for the particular mailbox for a day or week or month.”

Here is a similar example from Microsoft Answers.

The easy response is to explain that a shared mailbox is a repository for email that can be shared by a team such as a set of customer agents but that the mailbox comes with none of the accompanying infrastructure needed to track aspects like responsiveness, agent activity, or customer satisfaction. This doesn’t mean that organizations or ISVs cannot design and implement structures around shared mailboxes to capture information that might be useful to measure productivity and to ensure that customer queries are dealt with efficiently. But you still run into the issue that shared mailboxes are simply not designed for this purpose.

Analyzing Inbox Traffic for Shared Mailboxes

The Office 365 for IT Pros eBook team has customers, and we use shared mailboxes to handle queries about book availability, duplicate purchases, lost download codes, and so on. We also receive a bunch of messages from people who would like to post articles on Office365itpros.com. Because we have shared mailboxes with real-life customer communications, it seemed like a good idea to test what information could be extracted using the Microsoft Graph and PowerShell to measure aspects like the response time for inbound messages.

There’s even a specific Graph permission (Mail.Read.Shared) to handle access to items in shared mailboxes. Signed-in users with full access to shared mailboxes can use the permission to read items with cmdlets like Get-MgUserMailFolderMessage to retrieve items from shared mailboxes.

It’s impossible to know how people use shared mailboxes or what they want to extract in terms of analysis, so the script I wrote to explore the principles of retrieving information from shared mailboxes focuses on measuring the speed of responses to inbound messages. Figure 1 shows the result with a list of queries arriving in the mailbox and the average time for a first response and time to final response. The time for the first response seems unacceptable at over six hours, but some of the queries came in from Australia and New Zealand when no one was monitoring the mailbox!

The script can be downloaded from the Office 365 for IT Pros GitHub repository. Feel free to amend the code to meet specific organizational requirements. The script can analyze one or multiple shared mailboxes over the last 180 days (configurable).

Conversations and Threads

When Exchange Online delivers a message to a mailbox, it assigns a conversation identifier to the message. If Exchange can match the message as a reply to an existing conversation, it takes the identifier for that conversation. Otherwise, Exchange creates a new identifier. The messages that share a conversation identifier form a thread composed of the original message and its replies.

Messages in a conversation can exist in several folders (Inbox, Sent Items, and Deleted Items are common), but the Get-MgUserMessage cmdlet can use a filter to find all messages sharing a conversation identifier. For example:

[array]$ThreadItems = Get-MgUserMessage -UserId $Mailbox -Filter "conversationID eq '$ConversationId'" 

The script works by tracking conversations and measuring the elapsed between a message arriving, the first response, and the last message in the thread. We assume that the last message marks a successful conclusion to the conversation.

Tracking messages via conversation identifiers is not foolproof. New threads can be created by users adding or removing addresses from responses. However, conversation identifiers are probably the most effective out-of-the-box method to track messages from initial enquiry to final response.

Finding Messages in Other Ways

Outside of conversation identifiers, you could try to tie messages together in different ways. For example, you could add some form of unique identifier to the subject of replies generated. It’s possible to filter messages by subject, so this would find the items in a set except the initial message, which would have to be added through a separate search. Another approach might inject a unique identifier into the message body to enable searching message bodies for the identifier. For example:

Get-MgUserMessage -UserId $Mailbox -Search "Body:IdentifierXXX”

Inserting identifiers in message subjects or bodies is something that could be done manually. However, an add-in would do a more reliable job. I didn’t investigate this topic any further.

Determining Who Responds to a Query

The script attempts to extract the email addresses of agents who respond to messages. However, this is not easy. Multiple people might be involved in a conversation from inside and outside the organization. Knowing who are agents whose activity should be measured is highly specific to an organization.

Users with access to a shared mailbox can add one or more categories to items, so items can be tagged with categories to capture information about the assigned agent. However, the categories can be overwritten by anyone with full access to the mailbox, so depending on categories is not reliable.

It’s possible to know if someone other than the shared mailbox responds to a query if they use the Send on Behalf Of mailbox permission rather than the Send As permission. In the first case, Outlook captures the agent’s email address as the sender while in the second Outlook records the name of the shared mailbox as the sender. Outlook is working as designed here because the Send As permission essentially allows someone to impersonate the shared mailbox.

An Example, Not a Solution

I don’t pretend that the solution presented in the script will satisfy anyone seeking detailed statistics about agent performance in answering inbound client queries that arrive in a shared mailbox. That’s not what I set out to do. Instead. The script explores the principles involved in extracting items from shared mailboxes and using that data to answer questions. What you do with those principles is up to you.

---

Insight like this doesn’t come easily. You’ve got to know the technology and understand how to look behind the scenes. Benefit from the knowledge and experience of the Office 365 for IT Pros team by subscribing to the best eBook covering Office 365 and the wider Microsoft 365 ecosystem.