Table of Contents
Using Audit Events to Determine Level of Copilot Activity
Updated 9-Apr-2024
In an article reporting how Copilot usage had dropped off after the initial novelty of having a digital assistant, MVP Joe Stocker described how he used ChatGPT to analyze information about Copilot interactions using Copilot for Microsoft 365 audit events retrieved from the unified audit log. It’s an interesting technique that caused me to investigate the information captured for Copilot interactions.
Running an audit search to find Copilot interactions returns the audit events captured when Microsoft 365 apps use Copilot. Outlook is an exception. Neither OWA nor Outlook for Windows capture Copilot audit events when drafting or refining message text or summarizing a message thread. I wasn’t too surprised at this because Microsoft had said that the Monarch client would be the only supported email client for Copilot until recently.
Interestingly, despite Monarch or any other Outlook client failing to log audit events for Copilot interactions, usage data shows up for Copilot for Outlook in the usage reports section of the Microsoft 365 admin center. The usage data does not describe which Outlook client is used. I assume that the lack of audit events for Outlook interactions is an oversight that Microsoft will address soon.
Update: According to message center notification MC773621 (9 Apr 2024), Microsoft plans to ship support for eDiscovery of Copilot interactions in Outlook (all clients) in mid-April. This implies that audit events will become available.
What’s in the Copilot for Microsoft 365 Audit Events
The unified audit log is a terrific resource for Microsoft 365 administrators to learn about what really happens inside a tenant. Audit events have two parts. The first is fixed and is consistent across all workloads. This part contains properties like the record identifier, user identifier, and timestamp. The second part contains the AuditData multi-value property and is under the control of the workload developers. Audit events generated by SharePoint Online are different to those generated by Exchange Online or Teams. The inconsistency in audit content means that it takes time to understand and interpret the information contained in audit events.
Microsoft defines a schema for Copilot audit events. Regretfully, the formatting of the information written into the AuditData property is inconsistent across the different clients. Some differences are expected because of the variation in content processed by Copilot, but the inconsistencies make it more difficult to understand and analyze what happened than it should be.
Adjusting Audit Search Results for Accuracy
When reviewing audit events generated for Copilot interactions, the fundamental principle is that every prompt processed by Copilot results in an audit event. For instance, if a user asks Copilot for Microsoft Chat a question and follows up with a couple of clarifications, the audit log ingests three audit events. The overall total of Copilot interaction audit events is therefore not the total of discrete user conversations with Copilot. Rather, it is the count of Copilot prompts.
It’s also important to remove duplicates of audit events from the set returned by an audit search. To improve the accuracy of audit searches, Microsoft is introducing a high completeness mode for the Search-UnifiedAuditLog cmdlet. If you don’t use a high completeness search (which takes much longer to complete), make sure to remove duplicates before analyzing the search results. I usually sort the audit search results by unique values in the identity property to remove duplicates. The Purview audit search feature does the same when it downloads audit search results.
After some trial and error, I created a PowerShell script to extract Copilot audit events from the unified audit log and parse the information contained in the AuditData property. An example of the output is shown in Figure 1.
You can download the script I used to find and analyze Copilot for Microsoft 365 audit events from GitHub.
Using Copilot for Excel to Analyze Audit Events
Given that we’re exploring Copilot for Microsoft 365, it made sense to try to use Copilot for Excel to interpret the results. Copilot for Excel can only work with tables and the workbooks must be enabled for AutoSave. In other words, the workbooks must be stored in OneDrive for Business or SharePoint Online. I used the ImportExcel PowerShell module to generate a worksheet with a table of audit events with the following command:
$Report | Export-Excel -Path c:\Temp\Copilot.xlsx -WorksheetName "Copilot Interactions" -Title "Copilot for Microsoft 365 Interactions" -TitleBold -TableName "CopilotEvents"
Figure 2 shows some of the analysis generated by Copilot for Excel. It’s important to say that Copilot for Excel is still in preview, which accounts for the difficulties reported by many users when prompting Copilot for Excel to do something simple, like create a chart.
Some Gaps in Copilot Audit Coverage
Microsoft touts the data security and compliance coverage for Copilot in Purview solutions like auditing. Not capturing audit events for Outlook clients seems like a big gap that must be filled ASAP. Improving the consistency of the information included in the audit events to make it easier to understand and use the data. But I have been waiting for various Microsoft development groups to improve their audit events since 2016 and I suspect that I’ll be waiting before I see better Copilot events.
Learn how to exploit the data available to Microsoft 365 tenant administrators through the Office 365 for IT Pros eBook. We love figuring out how things work.