Table of Contents
Can the MCP Server for Enterprise Help Microsoft 365 Tenant Administrators?
Along with agents, Model Context Protocol (MCP) servers received a lot of attention at the recent Microsoft Ignite event. An MCP server is a secure intermediary that lets AI models access data, like Entra ID, using a standardized approach. It all sounds very promising.
Microsoft has released a set of MCP servers, including the MCP Server for Enterprise, which can convert natural language queries into Microsoft Graph API calls. Other MCP servers are available, including the Microsoft 365 admin center, user profile, Dataverse, Outlook mail, Outlook calendar, SharePoint Lists, Teams, and so on (see the Agent 365 tooling servers overview for details – Figure 1). All the servers are in preview, and they all accept natural language queries against their respect knowledge sources.
From a Microsoft 365 tenant administrator perspective, the big question is whether MCP servers can do a job for them. Given that I know something about Graph API queries, I decided to test the MCP Server for Enterprise.
MCP servers have tools to help them work. Among the tools in the MCP Server for Enterprise are ones to extract intent from the user query and convert the query into a Graph API HTTP request that is then executed to answer the question with retrieved data.
Provisioning the MCP Server with Visual Studio Code
I followed the directions to provision the MCP server with Visual Studio Code. This action only needs to be done once for a tenant. Interestingly, the directions use a cmdlet from the Entra PowerShell module instead of the Microsoft Graph PowerShell SDK to configure two service principals (enterprise applications) for the server (“Microsoft MCP Server for Enterprise”) and client (“Visual Studio Code”). As far as I can tell, all the work could have been done with the Microsoft Graph PowerShell SDK, which is what most administrators use to deal with apps and service principals.
After setting up the service principals, I installed the MCP Server for Enterprise into Visual Studio Code and authenticated with my administrator account. The final step is to open Copilot Chat in Agent mode to create prompts for the server to process. This is where things went wrong. I think it’s because I sign into Visual Studio Code with the address that I have always used with GitHub (and GitHub Copilot). That address is different to my Entra ID account, and the net result was that although the MCP server reported that it was connected to my Entra ID account, the client couldn’t authenticate to send prompts (401 errors). But then things started to work, which is even weirder.
Prompting the MCP Server for Enterprise
By working, I mean that I could ask questions and the MCP server responded. In Figure 2, I asked how many security-enabled groups are in the tenant. The server contemplated the question, looked for matching examples, and found a Graph request that it issued. The response came back and is formatted to answer the question nicely.
Asking about security-related groups is a simple question in Graph terms. Let’s ask the server to tell us how many SharePoint sites are connected to Teams. The response to this prompt took longer, but the right answer popped out (Figure 3).
The server also generated a reasonable answer when I asked it to summarize the domains where guest accounts come from. Then I asked the server to tell me about users with Microsoft 365 E5 licenses that aren’t making full use of the license. The server took even longer to respond (it’s a more difficult question) before coming back with some PowerShell to run, noting that “full use” is not exposed directly (presumably as an account property).
Nevertheless, the server suggested that a practical way to figure out the answer is to measure the activity of users over the last 30 days based on usage reports (a developed version of the approach is explored here).
While the approach is sound, I wasn’t happy with the suggested code because it didn’t work. The code contained a throwback to profile selection (Select-MgProfile -Name 1.0) from V1 of the Microsoft Graph PowerShell SDK (V2 appeared in July 2023) and the command to find E5 licenses using the Get-MgSubscribedSKU cmdlet could not work because of the many variations of SKU part number used by Microsoft for E5 licenses. Then the call to the Get-MgUser cmdlet to find users and their licenses used a client-side filter to extract the set of licensed users (and didn’t filter to retrieve just member accounts). I could go on.
No Ready for Serious Work
The upshot is that the MCP Server for Enterprise is certainly capable of answering simple questions. However, once queries get more complicated, the server’s ability to answer prompts is hampered by the corpus of knowledge in its LLM.
It seems like the model was built from some old and limited examples of Graph API requests (the documentation cites “over 500 real-world examples”).Perhaps many of the examples came from Graph API documentation (in which the examples are never very complex). In short, the preview version of the MCP Server for Enterprise is a nice tool to demo at a technology exhibition, but the current build isn’t ready for serious work.
Keep up to date with developments like agents and MCP servers by subscribing to the Office 365 for IT Pros eBook. Our monthly updates make sure that our subscribers understand the most important changes happening across Office 365.
If you find the time, it may be interesting to compare the results of your queries with Looka/Jan
(https://practical365.com/practical-ai-ai-exploration-of-microsoft-graph-with-lokka/)
Good thought. I have lots lined up, but maybe some time would be well spent looking at lokka.
Is not clear from the article why it sometimes takes too long to get answer. Mcp server provides only queries. LLM model creates PowerShell scripts etc. By the way those 500 queries should be provides by teams who owns the APIs.
Or the 500 queries come from Microsoft documentation or other examples. I don’t think engineers sit down to figure out the PowerShell required to answer very complex queries.