The shifting sands of cloud services caught me out on Monday when I spoke at the UK Evolve conference. My topic was how to use PowerShell to manage Office 365 Groups and Teams (aka “Hacking your way to Happiness” – you can download a PDF of the deck here). During the session, I use several demos to show people how easy PowerShell really is and how quickly it lets you get real work done. All went well and then I came to an example where I look for records in the Office 365 audit log for “add group” events.
Presenting at Evolve – Before the code problem struck (photo: Matt Ellis via Twitter)
To make sure that demos run smoothly I use a cheat sheet of PowerShell code snippets in a Word document. Cutting and pasting known good code is faster and saves the embarrassment of getting code wrong in front of large audiences. I was therefore nonplussed to see an error from code that “used” to work perfectly well and is from an example in Chapter 21 of the Office 365 for IT Pros eBook.
Cannot index into a null array.
At line:1 char:7
+ $ReportLine = [PSCustomObject][Ordered]@{
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : NullArray
Add Group is Truncated
Looking at the code, I found that the problem happened after I used the Search-UnifiedAuditLog cmdlet to search for audit records for the “add group” event. The idea is that you can extract these audit records from the log and then analyze them to figure out who’s creating new Office 365 Groups and Teams. Audit records hold a lot of interesting information in the JSON content held in the AuditData field, so we need to unpack the content to extract information about the name of the newly created groups. The resulting $AuditData variable for an unpacked record contains the data shown below:
ObjectId : Not Available
UserId : Tony.Redmond@office365itpros.com
ClientIP :
Id : a6ddc5dc-bfce-4d7c-b39b-775cba7b48ae
RecordType : 8
CreationTime : 2018-07-11T14:36:54
Operation : Add group.
OrganizationId : b662313f-14fc-43a2-9a7a-d2e27f4f3478
UserType : 0
UserKey : 1003BFFD805C87B0@office365itpros.com
Workload : AzureActiveDirectory
ResultStatus : Success : Record Truncated
Version : 1
Seeing “Record Truncated” in a status field doesn’t create a feeling of confidence that the audit records are complete. In fact, the data is missing two fields that used to be available called Actor and Target, the latter being the field that the group identifier and group name were available in. When my code tried to access the information in $AuditData with references to $AuditData.Target[0].Id and $AuditData.Target[1].Id, the null array error happened because that array was never populated by the JSON extract.
Something Changed
Something had clearly changed in the audit records generated for “add group” events since I used the same demo code at the European Collaboration Summit in late May. I looked at all the audit events I could find for “add group” using the Audit log viewer in the Security and Compliance Center and found that every one of the events lack the data.
Office 365 only keeps audit log records for 90 days, so I could only go back to early July. I found good records on July 5 and bad records after July 11. It therefore seems likely that the format of the audit records captured for Azure Active Directory events ingested into the Office 365 audit log changed sometime between July 5 and 11. I can’t be more definite than that.
Changing Audit Records is a Bad Thing
As it transpires, the problem of truncated information in Office 365 audit records exists for other Azure Active Directory group operations like add and remove user (see below) and group removal and updates. This is a real problem. Customers depend on this information to understand what happens with groups inside their tenant.
Office 365 audit record is truncated for remove member operation
Looking at the Azure audit log, the details of the group (name and identifier) are present, so it seems likely that the problem occurs when Office 365 extracts information from Azure Active Directory and normalizes the data before ingestion into the audit log.
Activity
Date : 9/9/2018, 4:45:32 PM
Name : Remove member from group
CorrelationId : 3669b365-5346-4fb7-aeb5-0260a1e64305
Source : AzureAD
Category : Core Directory
Activity Status
Status : Success
Reason :
Initiated By (Actor)
Type : User
Name : Microsoft Teams Services
ObjectId : eff4cd58-1bb8-4899-94de-795f656b4a18
Upn : Tony.Redmond@office365itpros.com
IpAddress :
Target(s)
Target
Type : User
ObjectId : bd8ad08e-c964-41e0-b5e9-456ab487a0c1
Upn : O365-ExchangeConnections2015@office365itpros.onmicrosoft.com
Modified Properties
Name : Group.ObjectID
New Value : "37991751-f6dd-48e5-bc86-1967181a7e53"
Name : Group.DisplayName
New Value : "All R &A Users"
Name : Group.WellKnownObjectName
Target
Type : Group
ObjectId : 37991751-f6dd-48e5-bc86-1967181a7e53
Additional Details
Not a Good Situation
I don’t know why Microsoft decided to change the format of the Azure Active Directory audit records as they were ingested into the Office 365 audit log. I do know that they messed up by removing essential data from the records. Where once it was possible to easily determine the name of a newly created group, now it is not. The same is true when trying to find out who was added or removed from groups, or accounts that are added or removed from the tenant. Losing this information is not good and it doesn’t give you confidence in the testing regime used to validate code changes.
I also don’t like when changes happen to data that might be used for compliance purposes without any warning or documentation. It doesn’t help people who roll their own analysis with PowerShell and it doesn’t help the ISVs who extract audit data on behalf of Office 365 tenants and store that data for longer than the 90-day default retention period.
All the Office 365 for IT Pros writing team can do is to keep checking to make sure that the code examples we include in the book continue to work over time. I’m happy that I found this problem and have been able to report it to Microsoft; it just wasn’t so good to run smack into the issue when doing a live demo.
Any app that fetches Azure AD audit records direct (like Cloud App Security) get the full records. It’s the Office 365 ingestion process to bring the records into the Audit log that’s broken.
{"id":null,"mode":"button","open_style":"in_modal","currency_code":"EUR","currency_symbol":"\u20ac","currency_type":"decimal","blank_flag_url":"https:\/\/office365itpros.com\/wp-content\/plugins\/tip-jar-wp\/\/assets\/images\/flags\/blank.gif","flag_sprite_url":"https:\/\/office365itpros.com\/wp-content\/plugins\/tip-jar-wp\/\/assets\/images\/flags\/flags.png","default_amount":100,"top_media_type":"featured_image","featured_image_url":"https:\/\/office365itpros.com\/wp-content\/uploads\/2022\/11\/cover-141x200.jpg","featured_embed":"","header_media":null,"file_download_attachment_data":null,"recurring_options_enabled":true,"recurring_options":{"never":{"selected":true,"after_output":"One time only"},"weekly":{"selected":false,"after_output":"Every week"},"monthly":{"selected":false,"after_output":"Every month"},"yearly":{"selected":false,"after_output":"Every year"}},"strings":{"current_user_email":"","current_user_name":"","link_text":"Virtual Tip Jar","complete_payment_button_error_text":"Check info and try again","payment_verb":"Pay","payment_request_label":"Office 365 for IT Pros","form_has_an_error":"Please check and fix the errors above","general_server_error":"Something isn't working right at the moment. Please try again.","form_title":"Office 365 for IT Pros","form_subtitle":null,"currency_search_text":"Country or Currency here","other_payment_option":"Other payment option","manage_payments_button_text":"Manage your payments","thank_you_message":"Thank you for supporting the work of Office 365 for IT Pros!","payment_confirmation_title":"Office 365 for IT Pros","receipt_title":"Your Receipt","print_receipt":"Print Receipt","email_receipt":"Email Receipt","email_receipt_sending":"Sending receipt...","email_receipt_success":"Email receipt successfully sent","email_receipt_failed":"Email receipt failed to send. Please try again.","receipt_payee":"Paid to","receipt_statement_descriptor":"This will show up on your statement as","receipt_date":"Date","receipt_transaction_id":"Transaction ID","receipt_transaction_amount":"Amount","refund_payer":"Refund from","login":"Log in to manage your payments","manage_payments":"Manage Payments","transactions_title":"Your Transactions","transaction_title":"Transaction Receipt","transaction_period":"Plan Period","arrangements_title":"Your Plans","arrangement_title":"Manage Plan","arrangement_details":"Plan Details","arrangement_id_title":"Plan ID","arrangement_payment_method_title":"Payment Method","arrangement_amount_title":"Plan Amount","arrangement_renewal_title":"Next renewal date","arrangement_action_cancel":"Cancel Plan","arrangement_action_cant_cancel":"Cancelling is currently not available.","arrangement_action_cancel_double":"Are you sure you'd like to cancel?","arrangement_cancelling":"Cancelling Plan...","arrangement_cancelled":"Plan Cancelled","arrangement_failed_to_cancel":"Failed to cancel plan","back_to_plans":"\u2190 Back to Plans","update_payment_method_verb":"Update","sca_auth_description":"Your have a pending renewal payment which requires authorization.","sca_auth_verb":"Authorize renewal payment","sca_authing_verb":"Authorizing payment","sca_authed_verb":"Payment successfully authorized!","sca_auth_failed":"Unable to authorize! Please try again.","login_button_text":"Log in","login_form_has_an_error":"Please check and fix the errors above","uppercase_search":"Search","lowercase_search":"search","uppercase_page":"Page","lowercase_page":"page","uppercase_items":"Items","lowercase_items":"items","uppercase_per":"Per","lowercase_per":"per","uppercase_of":"Of","lowercase_of":"of","back":"Back to plans","zip_code_placeholder":"Zip\/Postal Code","download_file_button_text":"Download File","input_field_instructions":{"tip_amount":{"placeholder_text":"How much would you like to tip?","initial":{"instruction_type":"normal","instruction_message":"How much would you like to tip? Choose any currency."},"empty":{"instruction_type":"error","instruction_message":"How much would you like to tip? Choose any currency."},"invalid_curency":{"instruction_type":"error","instruction_message":"Please choose a valid currency."}},"recurring":{"placeholder_text":"Recurring","initial":{"instruction_type":"normal","instruction_message":"How often would you like to give this?"},"success":{"instruction_type":"success","instruction_message":"How often would you like to give this?"},"empty":{"instruction_type":"error","instruction_message":"How often would you like to give this?"}},"name":{"placeholder_text":"Name on Credit Card","initial":{"instruction_type":"normal","instruction_message":"Enter the name on your card."},"success":{"instruction_type":"success","instruction_message":"Enter the name on your card."},"empty":{"instruction_type":"error","instruction_message":"Please enter the name on your card."}},"privacy_policy":{"terms_title":"Terms and conditions","terms_body":null,"terms_show_text":"View Terms","terms_hide_text":"Hide Terms","initial":{"instruction_type":"normal","instruction_message":"I agree to the terms."},"unchecked":{"instruction_type":"error","instruction_message":"Please agree to the terms."},"checked":{"instruction_type":"success","instruction_message":"I agree to the terms."}},"email":{"placeholder_text":"Your email address","initial":{"instruction_type":"normal","instruction_message":"Enter your email address"},"success":{"instruction_type":"success","instruction_message":"Enter your email address"},"blank":{"instruction_type":"error","instruction_message":"Enter your email address"},"not_an_email_address":{"instruction_type":"error","instruction_message":"Make sure you have entered a valid email address"}},"note_with_tip":{"placeholder_text":"Your note here...","initial":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"empty":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"not_empty_initial":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"saving":{"instruction_type":"normal","instruction_message":"Saving note..."},"success":{"instruction_type":"success","instruction_message":"Note successfully saved!"},"error":{"instruction_type":"error","instruction_message":"Unable to save note note at this time. Please try again."}},"email_for_login_code":{"placeholder_text":"Your email address","initial":{"instruction_type":"normal","instruction_message":"Enter your email to log in."},"success":{"instruction_type":"success","instruction_message":"Enter your email to log in."},"blank":{"instruction_type":"error","instruction_message":"Enter your email to log in."},"empty":{"instruction_type":"error","instruction_message":"Enter your email to log in."}},"login_code":{"initial":{"instruction_type":"normal","instruction_message":"Check your email and enter the login code."},"success":{"instruction_type":"success","instruction_message":"Check your email and enter the login code."},"blank":{"instruction_type":"error","instruction_message":"Check your email and enter the login code."},"empty":{"instruction_type":"error","instruction_message":"Check your email and enter the login code."}},"stripe_all_in_one":{"initial":{"instruction_type":"normal","instruction_message":"Enter your credit card details here."},"empty":{"instruction_type":"error","instruction_message":"Enter your credit card details here."},"success":{"instruction_type":"normal","instruction_message":"Enter your credit card details here."},"invalid_number":{"instruction_type":"error","instruction_message":"The card number is not a valid credit card number."},"invalid_expiry_month":{"instruction_type":"error","instruction_message":"The card's expiration month is invalid."},"invalid_expiry_year":{"instruction_type":"error","instruction_message":"The card's expiration year is invalid."},"invalid_cvc":{"instruction_type":"error","instruction_message":"The card's security code is invalid."},"incorrect_number":{"instruction_type":"error","instruction_message":"The card number is incorrect."},"incomplete_number":{"instruction_type":"error","instruction_message":"The card number is incomplete."},"incomplete_cvc":{"instruction_type":"error","instruction_message":"The card's security code is incomplete."},"incomplete_expiry":{"instruction_type":"error","instruction_message":"The card's expiration date is incomplete."},"incomplete_zip":{"instruction_type":"error","instruction_message":"The card's zip code is incomplete."},"expired_card":{"instruction_type":"error","instruction_message":"The card has expired."},"incorrect_cvc":{"instruction_type":"error","instruction_message":"The card's security code is incorrect."},"incorrect_zip":{"instruction_type":"error","instruction_message":"The card's zip code failed validation."},"invalid_expiry_year_past":{"instruction_type":"error","instruction_message":"The card's expiration year is in the past"},"card_declined":{"instruction_type":"error","instruction_message":"The card was declined."},"missing":{"instruction_type":"error","instruction_message":"There is no card on a customer that is being charged."},"processing_error":{"instruction_type":"error","instruction_message":"An error occurred while processing the card."},"invalid_request_error":{"instruction_type":"error","instruction_message":"Unable to process this payment, please try again or use alternative method."},"invalid_sofort_country":{"instruction_type":"error","instruction_message":"The billing country is not accepted by SOFORT. Please try another country."}}}},"fetched_oembed_html":false}
Any app that fetches Azure AD audit records direct (like Cloud App Security) get the full records. It’s the Office 365 ingestion process to bring the records into the Audit log that’s broken.