In any case, the point is that Azure AD app permissions exist in a tenant and it’s good to know what the apps are, why they have permissions, and if they are still needed. An increasing number of ISV and other apps use the Graph APIs to interact with Microsoft 365 data. Each of these apps needs an OAuth 2.0 consent to interact with the Graph and ends up as an Azure AD integrated app. By running Vasil’s script, I found 58 apps in my tenant. Based on what I see in other tenants, this is not uncommon.
Reviewing Integrated Apps
Although you could review the set of Azure AD integrated apps through the Azure AD admin center (Figure 1), it’s often easier to perform a review using a shared resource like the CSV file generated by the script.
Figure 1: Enterprise apps in the Azure AD admin center
To make the data easier to work with, after running the script to generate the CSV files, I converted the CSV to an Excel worksheet formatted as a table and imported it into Microsoft Lists in Teams. Storing the data in a list accessed through a tab in a channel makes the information very accessible to people who might know what function apps serve (if any). I added a couple of fields to track the apps during the review, including creating a category to classify the apps and a notes field to capture comments made by reviewers. Here are the set of categories I used:
Microsoft apps.
Trial apps installed for testing purposes.
ISV apps still in use.
Tenant Apps registered to use PowerShell to call Microsoft Graph APIs.
Apps requiring further investigation.
Unwanted apps which can be removed.
Figure 2 shows how the list of apps for review appears in Teams.
Figure 2: Reviewing details of Azure AD integrated apps in a list in Teams
Looking through the set of apps uncovered some interesting items. For instance, a bunch of apps exist to help with registering users for conferences. If you’ve ever attended a Microsoft event like Ignite, you’ll probably find an app called “Microsoft Events” with permissions to read user profiles. Sessionize.com has an app with the same permissions to help people like me submit sessions to conferences, while the EventPoint sign-in app seems to serve the same purpose while demanding access to users’ email addresses. And finally, the Nubelus app is, I think, used by the European Collaboration Summit, but limits itself to delegated permissions for selected users (me, in this case).
Each app needs careful examination to understand its purpose, who uses the app, and the permissions it holds. Bringing the information about the apps into the list made that review quicker and easier.
Focusing on Problem Apps
The highlighted app (CXP Previews Portal is a good example of a questionable app. Examining details of the app (Figure 3), we discover that its home page is http://bf.net.nz/, located in New Zealand and that its creation date (in the tenant) was 20 December 2016. Access is valid until 18 June 2017, so it is obvious that this app is unused and a prime candidate for renewal. The other information captured for the app makes me think that this app is used to gain access to some Microsoft previews (CXP is a Microsoft acronym for Customer Experience Program). All in all, this app is a great candidate for removal.
Figure 3: Examining details of an Azure AD integrated app
In total, the review highlighted 16 unwanted apps which could be removed immediately along with several others which needed more investigation. These apps belong to trials that I had signed up for in the past (like the four apps registered for Office365mon.com), others for services I looked at but never used, like Microsoft FastTrack, and some were old Microsoft pilot apps, like CollabDB, part of the Project Osaka initiative from 2017. I remembered some apps, while others needed an internet search to fill in the gaps. In many cases, several years (going back to 2015) had lapsed since the app was granted permissions.
Removing Apps
To remove an app, go to the Enterprise applications section of the Azure AD admin center and select the app. Click properties in the left-hand pane to reveal the option to delete the app (Figure 4). Click Delete and confirm to remove the app. The Azure AD admin center only lists 50 apps in its UI, so if your tenant has more than 50 apps, you must search using the app id in to view its properties.
Figure 4: Removing an Azure Ad integrated app
If you remove an app in error, it’s easy for an administrator to grant consent to the app and its required permissions the next time the app is needed.
After removing the 16 unwanted apps, my set of Azure AD integrated apps is now down to 42. I’m now gathering information about the seven apps which need further investigation (if I were bold, I would delete the apps to see what happens, but that’s seldom a good plan).
Time for an App Spring Clean
What this exercise proves is that the set of apps integrated with Azure AD tends to grow over time and is not managed in any way by Azure AD. It’s up to administrators to audit the set of apps in their tenant and decide which apps remain useful and which can be discarded. Apart from cleaning out old apps, the purpose of the audit is to ensure that bad actors can’t leave highly permissioned apps behind to use after an initial visit.
The script described in Vasil’s article is a good starting point for an audit. Putting the results of the script into a Microsoft list makes the app more accessible and easier to work with. At the end of the day, humans must decide what apps to keep. Based on my experience, it should be possible to remove between 30-40% from a tenant. Your mileage may vary!
Apart from users and groups, it’s often surprising how little attention the contents of Azure AD receives from tenant administrators. Learn more by subscribing to the Office 365 for IT Pros eBook. We might not cover everything there, but what we do cover is important…
Any idea on what the “Valid until (delegate)” value holds exactly? How can we use this value to determine candidates for removal? Documentation on this property (expiryTime) is kind of vague in my opinion.
Hi Tony. Script is excellent but you can not rely on “Valid until (delegate)” parameter only. You should also check “User Sign-Ins” (interactive/non-interactive), “Service principal sign-ins” and “Managed identity sign-ins” which should be the most important parameter that determines whether a given application can be removed.
Loading...
I think it’s Vasil’s script… He wrote it; I just report on it. The sign in information that you refer to possibly wasn’t easily available at the time (managed identities certainly were not). But I think that an administrator who’s reviewing a list of apps can highlight ones that need to be checked and that’s what the script sets out to do. How the administrator then performs the checks is up to them.
Loading...
Tony, any idea why some of the apps, like Fast Track, from Microsoft are not “Verified”?
{"id":null,"mode":"button","open_style":"in_modal","currency_code":"EUR","currency_symbol":"\u20ac","currency_type":"decimal","blank_flag_url":"https:\/\/office365itpros.com\/wp-content\/plugins\/tip-jar-wp\/\/assets\/images\/flags\/blank.gif","flag_sprite_url":"https:\/\/office365itpros.com\/wp-content\/plugins\/tip-jar-wp\/\/assets\/images\/flags\/flags.png","default_amount":100,"top_media_type":"featured_image","featured_image_url":"https:\/\/office365itpros.com\/wp-content\/uploads\/2022\/11\/cover-141x200.jpg","featured_embed":"","header_media":null,"file_download_attachment_data":null,"recurring_options_enabled":true,"recurring_options":{"never":{"selected":true,"after_output":"One time only"},"weekly":{"selected":false,"after_output":"Every week"},"monthly":{"selected":false,"after_output":"Every month"},"yearly":{"selected":false,"after_output":"Every year"}},"strings":{"current_user_email":"","current_user_name":"","link_text":"Virtual Tip Jar","complete_payment_button_error_text":"Check info and try again","payment_verb":"Pay","payment_request_label":"Office 365 for IT Pros","form_has_an_error":"Please check and fix the errors above","general_server_error":"Something isn't working right at the moment. Please try again.","form_title":"Office 365 for IT Pros","form_subtitle":null,"currency_search_text":"Country or Currency here","other_payment_option":"Other payment option","manage_payments_button_text":"Manage your payments","thank_you_message":"Thank you for supporting the work of Office 365 for IT Pros!","payment_confirmation_title":"Office 365 for IT Pros","receipt_title":"Your Receipt","print_receipt":"Print Receipt","email_receipt":"Email Receipt","email_receipt_sending":"Sending receipt...","email_receipt_success":"Email receipt successfully sent","email_receipt_failed":"Email receipt failed to send. Please try again.","receipt_payee":"Paid to","receipt_statement_descriptor":"This will show up on your statement as","receipt_date":"Date","receipt_transaction_id":"Transaction ID","receipt_transaction_amount":"Amount","refund_payer":"Refund from","login":"Log in to manage your payments","manage_payments":"Manage Payments","transactions_title":"Your Transactions","transaction_title":"Transaction Receipt","transaction_period":"Plan Period","arrangements_title":"Your Plans","arrangement_title":"Manage Plan","arrangement_details":"Plan Details","arrangement_id_title":"Plan ID","arrangement_payment_method_title":"Payment Method","arrangement_amount_title":"Plan Amount","arrangement_renewal_title":"Next renewal date","arrangement_action_cancel":"Cancel Plan","arrangement_action_cant_cancel":"Cancelling is currently not available.","arrangement_action_cancel_double":"Are you sure you'd like to cancel?","arrangement_cancelling":"Cancelling Plan...","arrangement_cancelled":"Plan Cancelled","arrangement_failed_to_cancel":"Failed to cancel plan","back_to_plans":"\u2190 Back to Plans","update_payment_method_verb":"Update","sca_auth_description":"Your have a pending renewal payment which requires authorization.","sca_auth_verb":"Authorize renewal payment","sca_authing_verb":"Authorizing payment","sca_authed_verb":"Payment successfully authorized!","sca_auth_failed":"Unable to authorize! Please try again.","login_button_text":"Log in","login_form_has_an_error":"Please check and fix the errors above","uppercase_search":"Search","lowercase_search":"search","uppercase_page":"Page","lowercase_page":"page","uppercase_items":"Items","lowercase_items":"items","uppercase_per":"Per","lowercase_per":"per","uppercase_of":"Of","lowercase_of":"of","back":"Back to plans","zip_code_placeholder":"Zip\/Postal Code","download_file_button_text":"Download File","input_field_instructions":{"tip_amount":{"placeholder_text":"How much would you like to tip?","initial":{"instruction_type":"normal","instruction_message":"How much would you like to tip? Choose any currency."},"empty":{"instruction_type":"error","instruction_message":"How much would you like to tip? Choose any currency."},"invalid_curency":{"instruction_type":"error","instruction_message":"Please choose a valid currency."}},"recurring":{"placeholder_text":"Recurring","initial":{"instruction_type":"normal","instruction_message":"How often would you like to give this?"},"success":{"instruction_type":"success","instruction_message":"How often would you like to give this?"},"empty":{"instruction_type":"error","instruction_message":"How often would you like to give this?"}},"name":{"placeholder_text":"Name on Credit Card","initial":{"instruction_type":"normal","instruction_message":"Enter the name on your card."},"success":{"instruction_type":"success","instruction_message":"Enter the name on your card."},"empty":{"instruction_type":"error","instruction_message":"Please enter the name on your card."}},"privacy_policy":{"terms_title":"Terms and conditions","terms_body":null,"terms_show_text":"View Terms","terms_hide_text":"Hide Terms","initial":{"instruction_type":"normal","instruction_message":"I agree to the terms."},"unchecked":{"instruction_type":"error","instruction_message":"Please agree to the terms."},"checked":{"instruction_type":"success","instruction_message":"I agree to the terms."}},"email":{"placeholder_text":"Your email address","initial":{"instruction_type":"normal","instruction_message":"Enter your email address"},"success":{"instruction_type":"success","instruction_message":"Enter your email address"},"blank":{"instruction_type":"error","instruction_message":"Enter your email address"},"not_an_email_address":{"instruction_type":"error","instruction_message":"Make sure you have entered a valid email address"}},"note_with_tip":{"placeholder_text":"Your note here...","initial":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"empty":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"not_empty_initial":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"saving":{"instruction_type":"normal","instruction_message":"Saving note..."},"success":{"instruction_type":"success","instruction_message":"Note successfully saved!"},"error":{"instruction_type":"error","instruction_message":"Unable to save note note at this time. Please try again."}},"email_for_login_code":{"placeholder_text":"Your email address","initial":{"instruction_type":"normal","instruction_message":"Enter your email to log in."},"success":{"instruction_type":"success","instruction_message":"Enter your email to log in."},"blank":{"instruction_type":"error","instruction_message":"Enter your email to log in."},"empty":{"instruction_type":"error","instruction_message":"Enter your email to log in."}},"login_code":{"initial":{"instruction_type":"normal","instruction_message":"Check your email and enter the login code."},"success":{"instruction_type":"success","instruction_message":"Check your email and enter the login code."},"blank":{"instruction_type":"error","instruction_message":"Check your email and enter the login code."},"empty":{"instruction_type":"error","instruction_message":"Check your email and enter the login code."}},"stripe_all_in_one":{"initial":{"instruction_type":"normal","instruction_message":"Enter your credit card details here."},"empty":{"instruction_type":"error","instruction_message":"Enter your credit card details here."},"success":{"instruction_type":"normal","instruction_message":"Enter your credit card details here."},"invalid_number":{"instruction_type":"error","instruction_message":"The card number is not a valid credit card number."},"invalid_expiry_month":{"instruction_type":"error","instruction_message":"The card's expiration month is invalid."},"invalid_expiry_year":{"instruction_type":"error","instruction_message":"The card's expiration year is invalid."},"invalid_cvc":{"instruction_type":"error","instruction_message":"The card's security code is invalid."},"incorrect_number":{"instruction_type":"error","instruction_message":"The card number is incorrect."},"incomplete_number":{"instruction_type":"error","instruction_message":"The card number is incomplete."},"incomplete_cvc":{"instruction_type":"error","instruction_message":"The card's security code is incomplete."},"incomplete_expiry":{"instruction_type":"error","instruction_message":"The card's expiration date is incomplete."},"incomplete_zip":{"instruction_type":"error","instruction_message":"The card's zip code is incomplete."},"expired_card":{"instruction_type":"error","instruction_message":"The card has expired."},"incorrect_cvc":{"instruction_type":"error","instruction_message":"The card's security code is incorrect."},"incorrect_zip":{"instruction_type":"error","instruction_message":"The card's zip code failed validation."},"invalid_expiry_year_past":{"instruction_type":"error","instruction_message":"The card's expiration year is in the past"},"card_declined":{"instruction_type":"error","instruction_message":"The card was declined."},"missing":{"instruction_type":"error","instruction_message":"There is no card on a customer that is being charged."},"processing_error":{"instruction_type":"error","instruction_message":"An error occurred while processing the card."},"invalid_request_error":{"instruction_type":"error","instruction_message":"Unable to process this payment, please try again or use alternative method."},"invalid_sofort_country":{"instruction_type":"error","instruction_message":"The billing country is not accepted by SOFORT. Please try another country."}}}},"fetched_oembed_html":false}
Hi Tony,
Great article,
Any idea on what the “Valid until (delegate)” value holds exactly? How can we use this value to determine candidates for removal? Documentation on this property (expiryTime) is kind of vague in my opinion.
Thanks in advance,
/Kenneth
I don’t know… I have never delved into that detail. There’s always something to research…
Hi Tony. Script is excellent but you can not rely on “Valid until (delegate)” parameter only. You should also check “User Sign-Ins” (interactive/non-interactive), “Service principal sign-ins” and “Managed identity sign-ins” which should be the most important parameter that determines whether a given application can be removed.
I think it’s Vasil’s script… He wrote it; I just report on it. The sign in information that you refer to possibly wasn’t easily available at the time (managed identities certainly were not). But I think that an administrator who’s reviewing a list of apps can highlight ones that need to be checked and that’s what the script sets out to do. How the administrator then performs the checks is up to them.
Tony, any idea why some of the apps, like Fast Track, from Microsoft are not “Verified”?
No idea. Probably some delay in issuing new versions which are verified.
hi
is there a way to list / delete enterprise applications using the python SDK?
i’d like to automate this.
Thanks.
Use the Graph API: https://docs.microsoft.com/en-us/graph/api/application-list?view=graph-rest-1.0&tabs=http