Chatbot Delivers Answers, But Usable Responses Are a Different Matter
The launch of the free research version OpenAI’s ChatGPT project generated lots of reactions, with some journalists predicting that using AI in this manner could mark the end of Google search results. According to Chris Johns, an economist whose podcast I subscribe to, the chatbot is capable of producing answers that meet the standard of first year university exams. Closer to home, MVP Doug Finke (author of the ImportExcel PowerShell module) thought the results generated for PowerShell questions were impressive (here’s his YouTube video).
Given the opinions voiced, I decided to sign up to test ChatGPT. My conclusion is that the chatbot is an idiot savant when it comes to technology. The answers generated by ChatGPT are plausible and cogent in some areas, but once it goes outside its area of comfort, the answers become weaker and weaker.
The Need for Good Source Material
By its very nature, AI depends on the source material used to train models. Inside Microsoft 365, a trainable classifier doesn’t work in scenarios like auto-label policies unless the set of source documents used to create the model underpinning the classifier are good enough. In the case of ChatGPT, OpenAI admit that the material used to build the model comes from 2021 or earlier. Given the nature of technology, especially cloud services, out-of-date information leads to bad answers.
A problem also arises when source material is wrong or contains information that might be accurate at a point in time but will be superseded by developments. This happens all the time in blog posts. For example, if you search for something like “How to update Azure AD accounts with PowerShell,” you’ll get a bunch of responses describing how to perform the task using cmdlets from the Azure AD or Microsoft Online Services (MSOL) modules. Posts published last week that I know of still reference these cmdlets, but people working in this space know that Microsoft plans to deprecate both modules in June 2023. The upshot is that the answer is right, works today, but is flawed because the code will stop working in six months. The lack of awareness of context is a flaw of AI and that shows through in its answers.
Asking About Azure AD Accounts
Take the example shown in Figure 1. The chatbot response to the question is inaccurate for two reasons: I asked about finding Azure AD accounts with the Microsoft Graph. The response is to use the soon-to-be-deprecated Azure AD module. There’s no trace of a Graph API request or the Microsoft Graph PowerShell SDK cmdlets.
Figure 1: Asking ChatGPT about finding Azure AD accounts
I have no idea why my question might have violated OpenAI’s content policy. That’s just a glitch. The important thing is that the code generated by ChatGPT works. Even though I wouldn’t use the Azure AD module now, the code runs perfectly and is a valid answer to the question
The Microsoft Graph PowerShell SDK existed in 2021, so I decided to check what the chatbot knew about the SDK. Figure 2 is the result. I think this is a good example of the ability of ChatGPT to generate a reasonably cogent (if wordy) answer in response to a question. The text is rather like the response you’d get from a Microsoft marketing person, but that’s another story.
Figure 2: ChatCPT discusses the Microsoft Graph PowerShell SDK
Testing a Real-Life Question
As a test of a real-life question, I took one about mailbox archiving from Practical365.com and input it to ChatGPT. The answer (Figure 3) is just plain wrong. First, only Exchange Online mailbox retention policies operate against archive mailboxes. Second, neither Microsoft 365 nor Exchange Online retention policies (there is no such thing as an online archiving policy) operate on the basis of mailbox size. Retention, including move to archive, is driven by item age. Like any assertion from a consultant, the confident nature of the response means that it might be accepted by someone who doesn’t know the technology. It seems like the text might be influenced by the way that Exchange Online expandable archives work, but the context is all wrong and the answer isn’t at all helpful.
Figure 3: ChatGPT gets mailbox archiving wrong
Finally, I asked about the world’s best Office 365 book. I was amused that ChatGPT recommended Office 365 for IT Pros but got the authors wrong. I have never met Ben Curry and he’s never been involved with the book, but hey, it’s still a highly plausible answer.
Figure 4: Who’s the Ben Curry guy that ChatGPT thinks wrote the Office 365 for IT Pros book?
Interesting but Flawed
The bottom line is captured in OpenAI’s admission that “ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers.” This, allied to the other flaw that “The model is often excessively verbose and overuses certain phrases, such as restating that it’s a language model trained by OpenAI” means that you can’t trust the chat bot’s responses to any question about technology that evolves quickly. Answering some basic PowerShell questions is fine. Seeking help to administer Office 365 is quite another matter.
ChatGPT is interesting and worthwhile technology that points to the way we might seek information in the future. Based on a $1 billion investment, Microsoft and OpenAI have been working since 2019 and OpenAI trained the ChatGPT model on Azure. With that kind of backing, I’m sure that OpenAI will improve the model and increase the accuracy of the answers that it generates. But for now, I think I shall stick with querying Google and sorting the wheat out of whatever chaff Google replies with.
Stay updated with developments across the Microsoft 365 ecosystem by subscribing to the Office 365 for IT Pros eBook. We do the research to make sure that our readers understand the technology.
{"id":null,"mode":"button","open_style":"in_modal","currency_code":"EUR","currency_symbol":"\u20ac","currency_type":"decimal","blank_flag_url":"https:\/\/office365itpros.com\/wp-content\/plugins\/tip-jar-wp\/\/assets\/images\/flags\/blank.gif","flag_sprite_url":"https:\/\/office365itpros.com\/wp-content\/plugins\/tip-jar-wp\/\/assets\/images\/flags\/flags.png","default_amount":100,"top_media_type":"featured_image","featured_image_url":"https:\/\/office365itpros.com\/wp-content\/uploads\/2022\/11\/cover-141x200.jpg","featured_embed":"","header_media":null,"file_download_attachment_data":null,"recurring_options_enabled":true,"recurring_options":{"never":{"selected":true,"after_output":"One time only"},"weekly":{"selected":false,"after_output":"Every week"},"monthly":{"selected":false,"after_output":"Every month"},"yearly":{"selected":false,"after_output":"Every year"}},"strings":{"current_user_email":"","current_user_name":"","link_text":"Virtual Tip Jar","complete_payment_button_error_text":"Check info and try again","payment_verb":"Pay","payment_request_label":"Office 365 for IT Pros","form_has_an_error":"Please check and fix the errors above","general_server_error":"Something isn't working right at the moment. Please try again.","form_title":"Office 365 for IT Pros","form_subtitle":null,"currency_search_text":"Country or Currency here","other_payment_option":"Other payment option","manage_payments_button_text":"Manage your payments","thank_you_message":"Thank you for supporting the work of Office 365 for IT Pros!","payment_confirmation_title":"Office 365 for IT Pros","receipt_title":"Your Receipt","print_receipt":"Print Receipt","email_receipt":"Email Receipt","email_receipt_sending":"Sending receipt...","email_receipt_success":"Email receipt successfully sent","email_receipt_failed":"Email receipt failed to send. Please try again.","receipt_payee":"Paid to","receipt_statement_descriptor":"This will show up on your statement as","receipt_date":"Date","receipt_transaction_id":"Transaction ID","receipt_transaction_amount":"Amount","refund_payer":"Refund from","login":"Log in to manage your payments","manage_payments":"Manage Payments","transactions_title":"Your Transactions","transaction_title":"Transaction Receipt","transaction_period":"Plan Period","arrangements_title":"Your Plans","arrangement_title":"Manage Plan","arrangement_details":"Plan Details","arrangement_id_title":"Plan ID","arrangement_payment_method_title":"Payment Method","arrangement_amount_title":"Plan Amount","arrangement_renewal_title":"Next renewal date","arrangement_action_cancel":"Cancel Plan","arrangement_action_cant_cancel":"Cancelling is currently not available.","arrangement_action_cancel_double":"Are you sure you'd like to cancel?","arrangement_cancelling":"Cancelling Plan...","arrangement_cancelled":"Plan Cancelled","arrangement_failed_to_cancel":"Failed to cancel plan","back_to_plans":"\u2190 Back to Plans","update_payment_method_verb":"Update","sca_auth_description":"Your have a pending renewal payment which requires authorization.","sca_auth_verb":"Authorize renewal payment","sca_authing_verb":"Authorizing payment","sca_authed_verb":"Payment successfully authorized!","sca_auth_failed":"Unable to authorize! Please try again.","login_button_text":"Log in","login_form_has_an_error":"Please check and fix the errors above","uppercase_search":"Search","lowercase_search":"search","uppercase_page":"Page","lowercase_page":"page","uppercase_items":"Items","lowercase_items":"items","uppercase_per":"Per","lowercase_per":"per","uppercase_of":"Of","lowercase_of":"of","back":"Back to plans","zip_code_placeholder":"Zip\/Postal Code","download_file_button_text":"Download File","input_field_instructions":{"tip_amount":{"placeholder_text":"How much would you like to tip?","initial":{"instruction_type":"normal","instruction_message":"How much would you like to tip? Choose any currency."},"empty":{"instruction_type":"error","instruction_message":"How much would you like to tip? Choose any currency."},"invalid_curency":{"instruction_type":"error","instruction_message":"Please choose a valid currency."}},"recurring":{"placeholder_text":"Recurring","initial":{"instruction_type":"normal","instruction_message":"How often would you like to give this?"},"success":{"instruction_type":"success","instruction_message":"How often would you like to give this?"},"empty":{"instruction_type":"error","instruction_message":"How often would you like to give this?"}},"name":{"placeholder_text":"Name on Credit Card","initial":{"instruction_type":"normal","instruction_message":"Enter the name on your card."},"success":{"instruction_type":"success","instruction_message":"Enter the name on your card."},"empty":{"instruction_type":"error","instruction_message":"Please enter the name on your card."}},"privacy_policy":{"terms_title":"Terms and conditions","terms_body":null,"terms_show_text":"View Terms","terms_hide_text":"Hide Terms","initial":{"instruction_type":"normal","instruction_message":"I agree to the terms."},"unchecked":{"instruction_type":"error","instruction_message":"Please agree to the terms."},"checked":{"instruction_type":"success","instruction_message":"I agree to the terms."}},"email":{"placeholder_text":"Your email address","initial":{"instruction_type":"normal","instruction_message":"Enter your email address"},"success":{"instruction_type":"success","instruction_message":"Enter your email address"},"blank":{"instruction_type":"error","instruction_message":"Enter your email address"},"not_an_email_address":{"instruction_type":"error","instruction_message":"Make sure you have entered a valid email address"}},"note_with_tip":{"placeholder_text":"Your note here...","initial":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"empty":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"not_empty_initial":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"saving":{"instruction_type":"normal","instruction_message":"Saving note..."},"success":{"instruction_type":"success","instruction_message":"Note successfully saved!"},"error":{"instruction_type":"error","instruction_message":"Unable to save note note at this time. Please try again."}},"email_for_login_code":{"placeholder_text":"Your email address","initial":{"instruction_type":"normal","instruction_message":"Enter your email to log in."},"success":{"instruction_type":"success","instruction_message":"Enter your email to log in."},"blank":{"instruction_type":"error","instruction_message":"Enter your email to log in."},"empty":{"instruction_type":"error","instruction_message":"Enter your email to log in."}},"login_code":{"initial":{"instruction_type":"normal","instruction_message":"Check your email and enter the login code."},"success":{"instruction_type":"success","instruction_message":"Check your email and enter the login code."},"blank":{"instruction_type":"error","instruction_message":"Check your email and enter the login code."},"empty":{"instruction_type":"error","instruction_message":"Check your email and enter the login code."}},"stripe_all_in_one":{"initial":{"instruction_type":"normal","instruction_message":"Enter your credit card details here."},"empty":{"instruction_type":"error","instruction_message":"Enter your credit card details here."},"success":{"instruction_type":"normal","instruction_message":"Enter your credit card details here."},"invalid_number":{"instruction_type":"error","instruction_message":"The card number is not a valid credit card number."},"invalid_expiry_month":{"instruction_type":"error","instruction_message":"The card's expiration month is invalid."},"invalid_expiry_year":{"instruction_type":"error","instruction_message":"The card's expiration year is invalid."},"invalid_cvc":{"instruction_type":"error","instruction_message":"The card's security code is invalid."},"incorrect_number":{"instruction_type":"error","instruction_message":"The card number is incorrect."},"incomplete_number":{"instruction_type":"error","instruction_message":"The card number is incomplete."},"incomplete_cvc":{"instruction_type":"error","instruction_message":"The card's security code is incomplete."},"incomplete_expiry":{"instruction_type":"error","instruction_message":"The card's expiration date is incomplete."},"incomplete_zip":{"instruction_type":"error","instruction_message":"The card's zip code is incomplete."},"expired_card":{"instruction_type":"error","instruction_message":"The card has expired."},"incorrect_cvc":{"instruction_type":"error","instruction_message":"The card's security code is incorrect."},"incorrect_zip":{"instruction_type":"error","instruction_message":"The card's zip code failed validation."},"invalid_expiry_year_past":{"instruction_type":"error","instruction_message":"The card's expiration year is in the past"},"card_declined":{"instruction_type":"error","instruction_message":"The card was declined."},"missing":{"instruction_type":"error","instruction_message":"There is no card on a customer that is being charged."},"processing_error":{"instruction_type":"error","instruction_message":"An error occurred while processing the card."},"invalid_request_error":{"instruction_type":"error","instruction_message":"Unable to process this payment, please try again or use alternative method."},"invalid_sofort_country":{"instruction_type":"error","instruction_message":"The billing country is not accepted by SOFORT. Please try another country."}}}},"fetched_oembed_html":false}