Migration from onPremise to Office 365 – Part 4 -Troubleshoot Exchange Hybrid

In Part 4 we will see how to troubleshoot Exchange Hybrid. This Blog Post Series consists of 6 parts. So if you missed one check them out as follows.

If you encounter the error HCW8064 or HCW8110 after finishing the Hybrid Configuration Wizard (HCW), you can also check out my following post.

This post is split into multiple parts

Part 1 … will cover the prerequisites like synchronize your onPrem users to Office 365 with Azure AD Connect.

Part 2 … will cover migration from Exchange onPrem to Exchange Online and here especially Exchange Hybrid classic full.

Part 3 … will cover moving user mailboxes from onPrem to Exchange Online.

Part 4 … will cover troubleshooting Exchange Hybrid

Part 5 … will cover migration from Skype for Business onPrem users to Skype for Business Online and Teams.

Part 6 … will cover Skype for Business Hybrid Connectivity and Teams Direct Routing

Part 7 … will cover troubleshooting Skype for Business Hybrid

Mailflow issues between Exchange Online and on-premise

After moving an user to Office 365, I tried to send an email to an external user and also I tried to send from this external user an email to my Office 365 user.

At this time, I tested and checked Enable centralized mail transport (CMT) on the Edge Server, also the MX Record still points to the onPrem environment, so all traffic outbound and inbound to and from Office 365, will traverse through the onPrem Edge Server.

Both mails stuck in queue, inbound to Office 365 on the Edge queue and the outbound from Office 365 on the Exchange Online queue 🙁

Here you can see the mail from an external user tried to send to my Office 365 user. This is the queue on my onPrem Edge Server which is responsible for the secure bi-directional mail transport between onPrem and Office 365.

451 4.4.395 Target host responded with error. -> 421 4.4.1 Connection timed out

Target host: braintestingde-mail-onmicrosoft-com.mail.protection.outlook.com
IP 104.47.7.138

Also at Exchange Online (EAC), here under mailflow – message trace, I can see that my mail from the Office 365 user mailbox send to an external user is in status pending.

Clicking on Details (the pen) will reveal more information about why it is in status pending.

Reason: [{LED=450 4.4.317 Cannot connect to remote server [Message=451 5.7.3 STARTTLS is required to send mail]

As you can see there are some problems regarding the secure TLS connection between Office 365 and my onPrem Edge Server.

Generally the Edge Server supports STARTTLS, so I suppose there are some problems regarding the certificate.

Because here Office 365 wants to connect to my Edge Server, I have to check the certificate from the Receive Connector on my Edge Server.

There we go, no certificate is assigned to the Receive Connector.

So we want to change this immediately, determine the thumbprint of your responsible certificate and assign it to the Receive Connector, in my case:

$Cert = Get-ExchangeCertificate -Thumbprint 9ae241799965c16866bc341bcdedd5f6d75c83a4
$TLSCertificateName = "<i>$($Cert.Issuer)<s>$($Cert.Subject)"
$TLSCertificateName

# also check that the certificate is enabled for SMTP
Enable-ExchangeCertificate -Thumbprint 9ae241799965c16866bc341bcdedd5f6d75c83a4 -Services SMTP

Set-ReceiveConnector "MAILGATEDefault internal receive connector MAILGATE" -TlsCertificateName $TLSCertificateName

Looks good 🙂

Mailflow issue from Exchange On-Prem to Office 365

https://docs.microsoft.com/en-in/archive/blogs/lalitbisht/mailflow-issue-from-exchange-on-prem-to-office-36

WHAT ARE YOUR EXCHANGE HYBRID OPTIONS?

https://www.enowsoftware.com/solutions-engine/whatareyourexchangehybridoptions

A local loop was detected on the Edge Server

If you renew your Edge Subscription, you have to rerun the Hybrid Configuration Wizard, otherwise you will run into the A local loop was detected error on your Edge Server, when you try to send an email from an on-premise mailbox to an online mailbox.

After the Hybrid Configuration Wizard runs successfully, the mails from the internal queues of the Mailbox Servers should immediately send out as they get the changes from the wizard also immediately, in order the Edge Server is also getting the changes immediately and will be able to send out the queued mails, you should run the Start-EdgeSynchronization command on one of the internal Mailbox Servers.

Target mailbox doesn’t have an smtp proxy matching in a mailbox migration

If you migrate an on-premise mailbox to Exchange Online and the migration batch stops and failed with the follwing exception:

Error: MigrationPermanentException: The target mailbox doesn‎’t have an SMTP proxy matching ‘<domain>.mail.onmicrosoft.com‎’.

One of the follwing conditions can be the cause:

• The source mailbox isn’t stamped to have a <domain>.mail.onmicrosoft.com smtp address.
• The proxy address <domain>.mail.onmicrosoft.com is not synced to Office 365 on the corresponding cloud mail-user object.

To resolve this issue will be detailed described under
https://docs.microsoft.com/en-us/exchange/troubleshoot/move-or-migrate-mailboxes/no-smtp-proxy-matching#solution

User can still access their on-premises mailbox after it is moved to Exchange Online in a hybrid deployment | Mailbox is shown up in Exchange Online and in on-premise it shows up as User Mailbox Type instead Office 365

https://docs.microsoft.com/en-us/exchange/troubleshoot/user-and-shared-mailboxes/still-can-access-mailbox-after-moved

Assume that you have a hybrid on-premises deployment of Exchange Server and Exchange Online in which you moved the on-premises mailboxes to the Exchange Online organization. After this move, one or more users are still able to access their on-premises mailbox. Additionally, the on-premises mailboxes of the users continue to receive mail.

CAUSE
As part of the remote move process, several changes are made to the on-premises user mailbox. After these changes are made, mail is delivered only to the Exchange Online mailbox and can be accessed only from this mailbox. If these changes do not occur, or if they occur incorrectly, mail is not routed to the Exchange Online mailbox as expected.

This problem occurs because the on-premises mailbox type is not correctly changed from UserMailbox to RemoteMailbox. Therefore, the mailbox cannot correctly route mail to Exchange Online.

Solution

To resolve this problem, follow these steps:

1. Restore the Exchange Online mailbox to your on-premises Exchange organization. To do this, see the following Microsoft website, and then follow the steps in the Move Exchange Online mailboxes to the on-premises organization section in major step 3 (Use the EAC to move mailboxes):Move mailboxes between on-premises and Exchange Online organizations in hybrid deployments
2. Force a full directory synchronization.
3. Restore the affected user mailbox to Exchange Online. To do this, see website from step 1, and then follow the steps in the Move on-premises mailboxes to Exchange Online section of major step 3.

If this is not possible and runs into an error, you can also disable synchronization for the affected user. In this case the online user and assigned online mailbox will be deleted.

Not to mention that you should first check that the on-premise mailbox works fine and is still running in UserMailbox Recipient Type and therefore independent from the failed online mailbox.

The deleted user and mailbox in this case will still reside in the Azure Active Directory recycle bin for less than 30 days.

Therefore we should delete the user from the deleted users folder in the Azure AD Portal or with PowerShell and:

Remove-MsolUser –UserPrincipalName [username] –RemoveFromRecycleBin –Force

# for all users in the recycle bin
Get-MsolUser -ReturnDeletedUsers | Remove-MsolUser -RemoveFromRecycleBin -Force

After that we again enable synchronization for this user, assign an Office 365 licence back and will repeat the move from the on-premise mailbox to Exchange Online.

A better way to solve this problem, if the online mailbox doesn’t still store important content and mails, is to use the Set-User cmdlet in Exchange Online PowerShell with the switch -PermanentlyClearPreviousMailboxInfo.

Permanently Clear Previous Mailbox Info
https://techcommunity.microsoft.com/t5/exchange-team-blog/permanently-clear-previous-mailbox-info/ba-p/607619

The new parameter in the user cmdlet will allow tenant admin to clean up Exchange Online Jon’s object without having to delete it. To clean the object, you can run the following command:

Set-User Jon@contoso.com -PermanentlyClearPreviousMailboxInfo

Executing this leaves you with a clean object that can be re-licensed without causing the 2-mailbox problem. Now you can on-board Jon’s on-premises mailbox following the usual steps.

Set-User cmdlet
https://docs.microsoft.com/en-us/powershell/module/exchange/set-user?view=exchange-ps

Troubleshooting Migration Endpoints

Get-MigrationEndpoint | fl

Test-MigrationServerAvailability
https://docs.microsoft.com/de-de/powershell/module/exchange/test-migrationserveravailability

This cmdlet is available in on-premises Exchange and in the cloud-based service. Some parameters and settings may be exclusive to one environment or the other.

Use the Test-MigrationServerAvailability cmdlet to test the availability of the target server in preparation to perform cross-forest mailbox moves, migration of on-premises mailboxes to Exchange Online, or to migrate on-premises mailbox data from an IMAP server to Exchange Online mailboxes. For all migration types, the cmdlet attempts to verify the connection settings used to connect to the target server.


Test-MigrationServerAvailability -ExchangeRemoteMove -Autodiscover -EmailAddress John.Doe@braintesting.net -Credentials (get-credential BRAINTESTINGAdministrator)

Test-MigrationServerAvailability -ExchangeRemoteMove –RemoteServer mail.braintesting.de -Credentials(get-credential BRAINTESTINGAdministrator)

Test-MigrationServerAvailability -Endpoint <Identity of the Endpoint>
This example verifies the connection settings to a remote server using the settings stored in an existing migration endpoint in Exchange Online.

Mapping from Exchange Online Cloud only user mailbox to an on-premise AD user account

If you assign a valid Exchange Online licence to a user in Office 365, there will be automatically and immediately provisioned a corresponding cloud only user mailbox, if Office 365 cannot determine that an on-premise user mailbox is still assigned to this user.

This can happen, if the on-premise user is synchronized to Office 365 immediately after creation in on-premise and before an on-premise mailbox was assigned to this user.

In this case, you have no chance to manage and see the user mailbox in the on-premise EAC nor you can offboarding the user mailbox from Office 365 to on-premise (move it from Exchange Online to on-premise).

The following example is using the cloud only user mailbox jnokes@braintesting.de.

In order to correct this situation we have to map this cloud only user mailbox to the corresponding on-premise AD user account.

To map the cloud only user mailbox we first have to determine her GUID in Office 365.

Get-Mailbox -Identity jnokes@braintesting.de | fl ExchangeGUID,Name,Alias

Next we need to determine the Remote Routing Address Domain. This can be done with the on-premise Exchange Management Shell and an user mailbox which is still moved from on-premise to Exchange Online.

Get-RemoteMailbox -Identity <UserPrincipalName> | fl RemoteRoutingAddress

Get-RemoteMailbox -Identity jdoe@braintesting.de | fl RemoteRoutingAddress

… is returning …

RemoteRoutingAddress : SMTP:jdoe@braintestingde.mail.onmicrosoft.com

So this is braintestingde.mail.onmicrosoft.com.

At the moment our user in on-premise AD even isn’t mail-enabled in Exchange. So we want to change this and therefore need to know the Remote Routing Address determined the step before. Run the following in the on-premise Exchange Management Shell.

Enable-RemoteMailbox -Identity jnokes@braintesting.de -RemoteRoutingAddress jnokes@braintestingde.mail.onmicrosoft.com

From now on you can see the user jnokes also in the on-premise EAC and is also mail-enabled in AD.

Finally we can now map the cloud only user mailbox to our on-premise mail-enabled AD user account as follows.

Set-RemoteMailbox -Identity jnokes@braintesting.de -ExchangeGUID <GUID determined above>

To let Office 365 know this immediately initiate an Azure AD Connect.

Start-ADSyncSyncCycle -PolicyType Initial
or
Start-ADSyncSyncCycle -PolicyType Delta

Granting Mailbox Folder Permissions – The user was found in Active Directory but isn’t valid to use for permissions.

If you want to add Mailbox Folder Permissions for an on-premises Mailbox to an User in Exchange Online, you will run into this error message.

Regarding this issue please read my following post.

Set up SPF to help prevent spoofing

https://docs.microsoft.com/en-us/microsoft-365/security/office-365-security/set-up-spf-in-office-365-to-help-prevent-spoofing

include:spf.protection.outlook.com

Links

Troubleshooting Hybrid Migration Endpoints in Classic and Modern Hybrid
http://thewindowsupdate.com/2019/10/25/troubleshooting-hybrid-migration-endpoints-in-classic-and-modern-hybrid/

In Part 5 … which is coming soon we will migrate our Skype for Business onPrem user to Teams …