Restriction forces Customers with New Tenants to Ask Microsoft Support to Enable Inbound Connectors
In an unannounced change, Microsoft recently disabled the ability of new Exchange Online tenants to activate newly-created inbound connectors. The text in the inbound connector FAQ (refreshed on 15 February) says:
“When you create an Inbound connector of OnPremises type manually, you may see the warning message Inbound connector for this service offering is created in a disabled state. Contact Support to enable it.”
The FAQ goes on to say that the customer must contact Microsoft support and provide a business justification for why their tenant needs to use an inbound connector. Microsoft attempts to reassure everyone by saying “Legitimate usage is approved, and the connector is enabled by our service engineers.” I’m sure that’s true, if you manage to speak to a level 1 support engineer who knows about why Microsoft disabled inbound connectors and understand what to do to release the block.
The EX505293 Incident
The problem with creating and updating connectors surfaced in incident EX505293 (January 27). Microsoft determined the root cause to be “A recent change to the service introduced a regression that may have prevented some admins from creating or modifying Exchange email transport connectors” and applied a fix that was operational by February 6, 2023. Because the Exchange Hybrid Connection Wizard (HCW) creates connectors, it was affected by the problem.
Figure 1: The EX505293 incident for inbound connectors
Microsoft doesn’t describe the precise nature of the fix in EX505293, but it seems like it allows tenants to create new connectors (in a disabled state). Moreover, the text in EX505293 indicates that the restriction applies only to tenants created from 2023 onwards. Microsoft’s FAQ doesn’t mention why they’ve clamped down on newly-created tenants, but it’s possible that it’s easier for an attacker to spin up a new tenant and create connectors to do bad things than to break in and compromise an existing tenant to take over its connectors.
Good Reasons Exist to Disable Inbound Connectors
Good reasons exist for why Microsoft should block inbound connectors. First, an inbound connector is not required for normal mail flow. The usual reason is that an organization wants to use a third-party solution to process their email. For instance, you might want to route messages through a third-party service to apply corporate email signatures before sending the messages to their final destination. Many of the specialized email signature ISVs like Code Two Software use inbound connectors to bring traffic back into an organization after inserting appropriate email signatures into messages.
Second, attackers who compromise a tenant might create connectors to route email through their services in an attempt to either use the tenant to send spam or to inject malware into user mailboxes (see this report about an attack on a Microsoft 365 tenant). Placing newly-created connectors into an inactive state until the owning tenant justifies the use of the connector closes off this attack vector.
I don’t know why Microsoft decided to restrict newly-created connectors. My gut feel is that something happened to cause immediate action, such as emerging evidence of a new attack technique involving connectors. We won’t know and Microsoft won’t say. Such is the nature of the security war between attackers and defenders playing out every day across IT infrastructures.
The Effect on ISVs
Even if closing off an attack vector stops attackers dead, doing so without due consideration of legitimate usage by ISVs is bad practice, especially when Microsoft didn’t warn ISVs or announce the change in a post in the EHLO blog, post a notification to the Microsoft 365 message center, or publish plans as a Microsoft 365 roadmap item. The change appeared without warning, perhaps to surprise potential attackers. It certainly surprised the affected ISVs.
Instead of being able to install their products and configure everything needed to make their software work, ISVs are now forced to perform a partial installation and ask their customers to contact Microsoft support to enable the disabled inbound connector. Microsoft has left customer-facing ISVs in an invidious position.
Not Good Business
I don’t criticize anything Microsoft does to protect Exchange Online against attack. Too many people depend on Exchange Online to risk potential compromise of user mailboxes. My criticism is entirely focused on the total lack of communication since Microsoft introduced the change referred to in EX505293 and fixed in early February. Operating in a vacuum is good for no-one, especially when Microsoft leaves ISVs out to dry and doesn’t tell them why their code no longer works.
Failure to communicate is always bad for business. It increases costs for ISVs and creates friction between ISVs and their customers. It generates an increased number of calls (and cost) for Microsoft Support to deal with. It slows business productivity where the cloud is supposed to speed things up. It’s a great example of a solution that makes perfect sense when sketched out by engineers on a whiteboard that runs headlong into problems in the real world. All in all, even if it fixed a potential security hole, forcing customers to go to Microsoft support to justify their use of a connector is a poor plan that Microsoft snuck in without saying anything to anyone.
Microsoft might say that they made the change because they want to protect Exchange Online customers. I accept their bona fides, but I expect better from the world’s largest software company, especially in how they deal with ISVs. After all, ISVs help the Microsoft cloud work better for Microsoft customers. They can’t do that if Microsoft changes the rules without saying anything.
Support the work of the Office 365 for IT Pros team by subscribing to the Office 365 for IT Pros eBook. Your support pays for the time we need to track, analyze, and document the changing world of Microsoft 365 and Office 365.
8 Replies to “Exchange Online Disables New Inbound Connectors”
Presumably this is just limited to creating/enabling new connectors? i.e. assume re-running an existing HCW or adding IPs into an existing connector would not be impacted?
It may actually be worse than just not being able to create connectors. With Exclaimer, the “old” interface gives an error when trying to add the connectors, but on the “new” interface, the process fails the first time, but if you repeat it, the connector and rules are actually added in 365, it just blocks outgoing emails…
Because the connector is disabled. You can create a connector, but it remains in a disabled state until you provide a business justification to Microsoft.
They aren’t disabled in the sense that we know from the console, but in the back end somehow, aren’t they? When I got the connectors added from Exclaimer they showed to me as though they were enabled. Disabling connectors is one thing but in this state of being able to have connectors added yet being disabled in the back end somewhere which causes transit issues is more than just lack of communication.
Loading...
Actually I think the connectors that Exclaimer is trying to create and either failing or for which there is some hidden block by Microsoft are Outbound connectors and not Inbound in this instance. They pass outgoing emails through the Exclaimer smarthost.
Loading...
Could be. It’s a long time since I looked at how Exclaimer works.
{"id":null,"mode":"button","open_style":"in_modal","currency_code":"EUR","currency_symbol":"\u20ac","currency_type":"decimal","blank_flag_url":"https:\/\/office365itpros.com\/wp-content\/plugins\/tip-jar-wp\/\/assets\/images\/flags\/blank.gif","flag_sprite_url":"https:\/\/office365itpros.com\/wp-content\/plugins\/tip-jar-wp\/\/assets\/images\/flags\/flags.png","default_amount":100,"top_media_type":"featured_image","featured_image_url":"https:\/\/office365itpros.com\/wp-content\/uploads\/2022\/11\/cover-141x200.jpg","featured_embed":"","header_media":null,"file_download_attachment_data":null,"recurring_options_enabled":true,"recurring_options":{"never":{"selected":true,"after_output":"One time only"},"weekly":{"selected":false,"after_output":"Every week"},"monthly":{"selected":false,"after_output":"Every month"},"yearly":{"selected":false,"after_output":"Every year"}},"strings":{"current_user_email":"","current_user_name":"","link_text":"Virtual Tip Jar","complete_payment_button_error_text":"Check info and try again","payment_verb":"Pay","payment_request_label":"Office 365 for IT Pros","form_has_an_error":"Please check and fix the errors above","general_server_error":"Something isn't working right at the moment. Please try again.","form_title":"Office 365 for IT Pros","form_subtitle":null,"currency_search_text":"Country or Currency here","other_payment_option":"Other payment option","manage_payments_button_text":"Manage your payments","thank_you_message":"Thank you for supporting the work of Office 365 for IT Pros!","payment_confirmation_title":"Office 365 for IT Pros","receipt_title":"Your Receipt","print_receipt":"Print Receipt","email_receipt":"Email Receipt","email_receipt_sending":"Sending receipt...","email_receipt_success":"Email receipt successfully sent","email_receipt_failed":"Email receipt failed to send. Please try again.","receipt_payee":"Paid to","receipt_statement_descriptor":"This will show up on your statement as","receipt_date":"Date","receipt_transaction_id":"Transaction ID","receipt_transaction_amount":"Amount","refund_payer":"Refund from","login":"Log in to manage your payments","manage_payments":"Manage Payments","transactions_title":"Your Transactions","transaction_title":"Transaction Receipt","transaction_period":"Plan Period","arrangements_title":"Your Plans","arrangement_title":"Manage Plan","arrangement_details":"Plan Details","arrangement_id_title":"Plan ID","arrangement_payment_method_title":"Payment Method","arrangement_amount_title":"Plan Amount","arrangement_renewal_title":"Next renewal date","arrangement_action_cancel":"Cancel Plan","arrangement_action_cant_cancel":"Cancelling is currently not available.","arrangement_action_cancel_double":"Are you sure you'd like to cancel?","arrangement_cancelling":"Cancelling Plan...","arrangement_cancelled":"Plan Cancelled","arrangement_failed_to_cancel":"Failed to cancel plan","back_to_plans":"\u2190 Back to Plans","update_payment_method_verb":"Update","sca_auth_description":"Your have a pending renewal payment which requires authorization.","sca_auth_verb":"Authorize renewal payment","sca_authing_verb":"Authorizing payment","sca_authed_verb":"Payment successfully authorized!","sca_auth_failed":"Unable to authorize! Please try again.","login_button_text":"Log in","login_form_has_an_error":"Please check and fix the errors above","uppercase_search":"Search","lowercase_search":"search","uppercase_page":"Page","lowercase_page":"page","uppercase_items":"Items","lowercase_items":"items","uppercase_per":"Per","lowercase_per":"per","uppercase_of":"Of","lowercase_of":"of","back":"Back to plans","zip_code_placeholder":"Zip\/Postal Code","download_file_button_text":"Download File","input_field_instructions":{"tip_amount":{"placeholder_text":"How much would you like to tip?","initial":{"instruction_type":"normal","instruction_message":"How much would you like to tip? Choose any currency."},"empty":{"instruction_type":"error","instruction_message":"How much would you like to tip? Choose any currency."},"invalid_curency":{"instruction_type":"error","instruction_message":"Please choose a valid currency."}},"recurring":{"placeholder_text":"Recurring","initial":{"instruction_type":"normal","instruction_message":"How often would you like to give this?"},"success":{"instruction_type":"success","instruction_message":"How often would you like to give this?"},"empty":{"instruction_type":"error","instruction_message":"How often would you like to give this?"}},"name":{"placeholder_text":"Name on Credit Card","initial":{"instruction_type":"normal","instruction_message":"Enter the name on your card."},"success":{"instruction_type":"success","instruction_message":"Enter the name on your card."},"empty":{"instruction_type":"error","instruction_message":"Please enter the name on your card."}},"privacy_policy":{"terms_title":"Terms and conditions","terms_body":null,"terms_show_text":"View Terms","terms_hide_text":"Hide Terms","initial":{"instruction_type":"normal","instruction_message":"I agree to the terms."},"unchecked":{"instruction_type":"error","instruction_message":"Please agree to the terms."},"checked":{"instruction_type":"success","instruction_message":"I agree to the terms."}},"email":{"placeholder_text":"Your email address","initial":{"instruction_type":"normal","instruction_message":"Enter your email address"},"success":{"instruction_type":"success","instruction_message":"Enter your email address"},"blank":{"instruction_type":"error","instruction_message":"Enter your email address"},"not_an_email_address":{"instruction_type":"error","instruction_message":"Make sure you have entered a valid email address"}},"note_with_tip":{"placeholder_text":"Your note here...","initial":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"empty":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"not_empty_initial":{"instruction_type":"normal","instruction_message":"Attach a note to your tip (optional)"},"saving":{"instruction_type":"normal","instruction_message":"Saving note..."},"success":{"instruction_type":"success","instruction_message":"Note successfully saved!"},"error":{"instruction_type":"error","instruction_message":"Unable to save note note at this time. Please try again."}},"email_for_login_code":{"placeholder_text":"Your email address","initial":{"instruction_type":"normal","instruction_message":"Enter your email to log in."},"success":{"instruction_type":"success","instruction_message":"Enter your email to log in."},"blank":{"instruction_type":"error","instruction_message":"Enter your email to log in."},"empty":{"instruction_type":"error","instruction_message":"Enter your email to log in."}},"login_code":{"initial":{"instruction_type":"normal","instruction_message":"Check your email and enter the login code."},"success":{"instruction_type":"success","instruction_message":"Check your email and enter the login code."},"blank":{"instruction_type":"error","instruction_message":"Check your email and enter the login code."},"empty":{"instruction_type":"error","instruction_message":"Check your email and enter the login code."}},"stripe_all_in_one":{"initial":{"instruction_type":"normal","instruction_message":"Enter your credit card details here."},"empty":{"instruction_type":"error","instruction_message":"Enter your credit card details here."},"success":{"instruction_type":"normal","instruction_message":"Enter your credit card details here."},"invalid_number":{"instruction_type":"error","instruction_message":"The card number is not a valid credit card number."},"invalid_expiry_month":{"instruction_type":"error","instruction_message":"The card's expiration month is invalid."},"invalid_expiry_year":{"instruction_type":"error","instruction_message":"The card's expiration year is invalid."},"invalid_cvc":{"instruction_type":"error","instruction_message":"The card's security code is invalid."},"incorrect_number":{"instruction_type":"error","instruction_message":"The card number is incorrect."},"incomplete_number":{"instruction_type":"error","instruction_message":"The card number is incomplete."},"incomplete_cvc":{"instruction_type":"error","instruction_message":"The card's security code is incomplete."},"incomplete_expiry":{"instruction_type":"error","instruction_message":"The card's expiration date is incomplete."},"incomplete_zip":{"instruction_type":"error","instruction_message":"The card's zip code is incomplete."},"expired_card":{"instruction_type":"error","instruction_message":"The card has expired."},"incorrect_cvc":{"instruction_type":"error","instruction_message":"The card's security code is incorrect."},"incorrect_zip":{"instruction_type":"error","instruction_message":"The card's zip code failed validation."},"invalid_expiry_year_past":{"instruction_type":"error","instruction_message":"The card's expiration year is in the past"},"card_declined":{"instruction_type":"error","instruction_message":"The card was declined."},"missing":{"instruction_type":"error","instruction_message":"There is no card on a customer that is being charged."},"processing_error":{"instruction_type":"error","instruction_message":"An error occurred while processing the card."},"invalid_request_error":{"instruction_type":"error","instruction_message":"Unable to process this payment, please try again or use alternative method."},"invalid_sofort_country":{"instruction_type":"error","instruction_message":"The billing country is not accepted by SOFORT. Please try another country."}}}},"fetched_oembed_html":false}
Presumably this is just limited to creating/enabling new connectors? i.e. assume re-running an existing HCW or adding IPs into an existing connector would not be impacted?
You should be OK. The problem is limited to new connectors in new tenants created after 1 January 2023.
It may actually be worse than just not being able to create connectors. With Exclaimer, the “old” interface gives an error when trying to add the connectors, but on the “new” interface, the process fails the first time, but if you repeat it, the connector and rules are actually added in 365, it just blocks outgoing emails…
Because the connector is disabled. You can create a connector, but it remains in a disabled state until you provide a business justification to Microsoft.
They aren’t disabled in the sense that we know from the console, but in the back end somehow, aren’t they? When I got the connectors added from Exclaimer they showed to me as though they were enabled. Disabling connectors is one thing but in this state of being able to have connectors added yet being disabled in the back end somewhere which causes transit issues is more than just lack of communication.
Actually I think the connectors that Exclaimer is trying to create and either failing or for which there is some hidden block by Microsoft are Outbound connectors and not Inbound in this instance. They pass outgoing emails through the Exclaimer smarthost.
Could be. It’s a long time since I looked at how Exclaimer works.