Migration from onPremise to Office 365 – Part 7 -Troubleshoot Skype for Business Hybrid

In Part 7 we will see how to troubleshoot Skype for Business Hybrid. This Blog Post Series consists of 7 parts. So if you missed one check them out as follows.

This post is split into multiple part

Part 1 … will cover the prerequisites like synchronize your onPrem users to Office 365 with Azure AD Connect.

Part 2 … will cover migration from Exchange onPrem to Exchange Online and here especially Exchange Hybrid classic full.

Part 3 … will cover moving user mailboxes from onPrem to Exchange Online.

Part 4 … will cover troubleshooting Exchange Hybrid

Part 5 … will cover migration from Skype for Business onPrem users to Skype for Business Online and Teams.

Part 6 … will cover Skype for Business Hybrid Connectivity and Teams Direct Routing

Part 7 … will cover troubleshooting Skype for Business Hybrid

Federation with Office 365 is not configured error

The Hybrid Setup wizard tells me that Federation with Office 365 and Shared SIP address space is not configured.

Further he kindly tells us, that if we select Next, he will configure our Skype for Business Server and Office 365 tenant with these required setting, so click on Next.

Hmmm, the Federation with Office 365 was not configured by the wizard.

You will find hints in the web, that the reason for this error will result from entries in the CSAllowedDomains or CSBlockedDomains list. In my case I didn’t configured any allowed or blocked domains and there were no entries in my environment, on-premise and online.

Under the following Federation requirements, they say that the Blocked domains and Allowed domains list in the on-premise deployment must exactly match with the online tenant.

Federation requirements
https://docs.microsoft.com/en-us/skypeforbusiness/hybrid/plan-hybrid-connectivity?toc=/SkypeForBusiness/toc.json&bc=/SkypeForBusiness/breadcrumb/toc.json#federation-requirements

So what’s the reason in my case?

After finishing the Hybrid configuration with PowerShell as described in Part 5 … , you will get the following entries for the Hosting Provider.

After running the Hybrid Setup wizard again, I get the same error with Federation with Office 365 was not configured by the wizard.

Further after checking it again with the Skype for Business Server Management Shell and the Get-CSHostingProvider Cmdlet, I will get the following output.

The Hybrid Setup wizard is setting IsLocal to False and deletes the AutodiscoverUrl.

So finally I configured Skype for Business hybrid with the PowerShell Cmdlets described in Part 5 …

Trying to run the Hybrid Setup wizard the next day will run fine without the Federation with Office 365 is not configured error message.

Further I discovered that the Hybrid Setup wizard in my version is setting a different new AutodiscoveryUrl and FQDN for the Hosting Provider as in the actual documentation from Microsoft under https://docs.microsoft.com/en-us/skypeforbusiness/hybrid/configure-federation-with-skype-for-business-online

You will see below that also the Skype for Business Online admin center is running under this FQDN https://webdired4.online.lync.com/LSCP.

The reason for this different Autodiscover URL is tenant specific!

So for my tenant the Audiscover URL is

https://webdired4.online.lync.com/Autodiscover/AutodiscoverService.svc/root

and the Hybrid Setup wizard can determine this specific URL at logon to the Office 365 tenant.

You will see later how to determine this tenant specific URLs.

So no clue why the Hybrid Setup wizard now runs without issues. I suppose this was due to some Office 365 synchronisation delays.

Skype for Business Hybrid configuration will break Open Federation

In my production environment I have configured Open Federation which allows to communicate with each external SIP domain, so called SIP Federated Domains.

If you do not select this option, federated user access is enabled only for users in the domains that you include in the allowed domains list.

More about Open Federation at the end of this post.

But after configuring Skype for Business Hybrid, external contacts in my Skype for Business desktop client appears with Updating … and finally ends in Presence unknown status.

The external contacts in contrast can see my presence status, even they wil be able to call me.

So what happens and break my Federation?

By the way it doesn’t matter if the user was moved to Skype for Business Online or still homed in on-premise, federation is broken.

The following parameter configured from the Hybrid wizard in on-premise with True is one of the reasons for breaking the federation in my case but not the only one, we will see further down.

After changing it back to false, SIP federation starts working again. But this is no solution as it will break Hybrid configuration.

# Change it for testing back to false, can only be set in conjunction with HostsOCSusers parameter
Set-CsHostingProvider -Identity LyncOnline -EnabledSharedAddressSpace $false -HostsOCSUsers $false

EnabledSharedAddressSpace: True

The reason for broken federation was:

A missing DNS SRV record _sipfederationtls._tcp.<my sipdomain>, the Edge Server will requesting when EnabledSharedAddressSpace $True and Open Federation is configured.

My Edge Server is configured to request the DNS servers in the perimeter network.


Solution:

Adding the following DNS SRV record to the perimeter network DNS servers:
_sipfederationtls._tcp.braincourt.com 0 0 5061 sip.braincourt.com

Configured on DNS Servers in the perimeter network, the Edge Server will request.

EnabledSharedAddressSpace: True means that Sykpe for Business users with the same SIP domain can be homed in on-premise or Office 365 (Skype for Business Online or Teams).

Why does the Edge Server request it’s own SRV Record for SIP Federation in case EnabledSharedAddressSpace $True with Open Federation?

That’s because of the way we configured the Access Edge Configuration to federate with Office 365 described in the following link.

Set-CSAccessEdgeConfiguration -AllowOutsideUsers $True -AllowFederatedUsers $True –EnablePartnerDiscovery $True –UseDnsSrvRouting

If ‘-EnablePartnerDiscovery‘ value is set to $True, we also need UseDnsSrvRouting, here Skype for Business Server will use DNS records to try and discover partner domains not listed in the AllowedDomains list. If the value is set to $False , Skype for Business Server will only federate with domains found on the AllowedDomains list. This parameter is required if you use DNS service routing.

If you configure AllowedDomains list in contrast, you will enter the domain name (SIP Domain) and the corresponding FQDN of the Access Edge in charge for this domain, therefore the Edge doesn’t need to use DNS records.

So as we configured ‘-EnablePartnerDiscovery‘ with $True, what is so called Open Federation, and also configured Hybrid ‘-EnabledSharedAddressSpace‘ with $True, the Edge Server not only will use DNS records to discover foreign partner domains but also our Office 365 tenant federation domain which should resolved into our on-premises Access Edge public IP.

To see what happens under the hood, I tried to reproduce this in my lab environment.

Enclosed you will see a trace from my lab environment SIP user account john.doe@braintesting.net who adds the SIP user account mra-testing02@braincourt.com from my production environment.

Therefore I removed the concerned DNS SRV record in the lab environment. In the production environment the record is meanwhile set and everything works fine. 🙂

# explicitly removed DNS SRV record in lab environment
_sipfederationtls._tcp.braintesting.net 0 0 5061 sip.braintesting.net

The following trace shows the startup from the Skype for Business client for john.doe@braintesting.net in the lab environment.

In advance it doesn’t matter if the user account from John Doe is on-premise or in Skype for Business Online (or Teams). With the missing on-premise SRV record for the Edge Server and EnabledSharedAddressSpace with $true (mandatory for Hybrid), the external contact mra-testing02@braincourt.com first stuck at Updating … and ends in Presence unknown as follows.

Here you can see the trace with trying to subscribe the external contact mra-testing02@braincourt.com. (John Doe homed in Skype for Business online)

ms-diagnostics: 1008;reason=”Unable to resolve DNS SRV record“;domain=”braintesting.net“;dns-srv-result=”NegativeResult”;dns-source=”InternalCache”;source=”sip.braintesting.net”

# targetname is the FQDN from the Online front end pool .
targetname=”ne0ed402FES03.infra.lync.com”

ms-split-domain-info: ms-traffic-type=SplitFedIn;ms-remote-fqdn=sip.braintesting.net

As you can see, the Edge Server performs a lookup for this SRV record but failed and doesn’t go further to establish a connection with the federated domain resp. external contact.

Here the same (but John Doe homed on-premise)

ms-split-domain-info only if the user is home online!

# targetname is the FQDN from the on-premise front end pool.
targetname=”sfb.braintesting.de”

Okay, now the same procedure after adding the missing DNS SRV record

_sipfederationtls._tcp.braintesting.net 0 0 5061 sip.braintesting.net

in the lab environment and the DNS servers in the perimter network the Edge server requests.

ET Voila!

Here the corresponding trace from the successful SUBSCRIBE.

Move-CsUser : HostedMigration fault: Error=(201), Description=(Cannot find user in Active Directory with the following SIP URI: “sip:John.Nokes@braintesting.net”)

https://docs.microsoft.com/en-us/skypeforbusiness/hybrid/move-users-from-on-premises-to-skype-for-business-online#move-users-with-move-csuser

Move-CsUser is available from an on-premises Skype for Business Management Shell PowerShell window. You must have sufficient privileges in both the on-premises environment as well as in the Microsoft 365/Office 365 organization as described in Required administrative credentials. You can either use a single account that has privileges in both environments, or you can start an on-premises Skype for Business Server Management Shell window with on-premises credentials, and use the -Credential parameter to specify credentials for a Microsoft 365 or Office 365 account with the necessary administrative role.

The following cmdlet sequence can be used to move a user to Skype for Business Online. It assumes the Microsoft 365 or Office 365 credential is a separate account and supplied as input for the Get-Credential prompt.

$cred=Get-Credential
$url=”https://admin1a.online.lync.com/HostedMigration/hostedmigrationService.svc”

Move-CsUser -Identity username@contoso.com -Target sipfed.online.lync.com -Credential $cred -HostedMigrationOverrideUrl $url

So in my case the first try to move a user with the Move-CsUser cmdlet ends as follows 🙁 .

Error Message
Move-CsUser : HostedMigration fault: Error=(201), Description=(Cannot find user in Active Directory with the following SIP URI: “sip:John.Nokes@braintesting.net”)

This error normally occurs if the msRTCSIP Active Directory attribute was not synchronized to your Azure AD. So if this was the issue, you can force the synchronization with the following two powershell cmdlets from your Azure AD Connect server:

To trigger a delta sync run

Start-ADSyncSyncCycle -PolicyType Delta

or to trigger a full (initial) sync run

Start-ADSyncSyncCycle -PolicyType Initial

But in my case the users Active Directory attributes was still correct synced with my Azure AD as you can see below.

To check that all attributes synced correctly run the follwing cmdlets:

Connect-AzureAD
Get-AzureADUser -ObjectId <UserPrincipalName> | fl

Get-AzureADUser -ObjectId jnokes@braintesting.de | fl ObjectID,ObjectType,AccountEnabled,DirSyncEnabled,DisplayName,LastDirSyncTime,Mail,SipProxyAddress,
UserPrincipalName

In my case I used the wrong Hosted Migration Service URL which was documented as follows

https://docs.microsoft.com/en-us/skypeforbusiness/hybrid/move-users-from-on-premises-to-skype-for-business-online#move-users-with-move-csuser

$cred=Get-Credential
$url=”https://admin1a.online.lync.com/HostedMigration/hostedmigrationService.svc”
Move-CsUser -Identity username@contoso.com -Target sipfed.online.lync.com -Credential $cred -HostedMigrationOverrideUrl $url

Under https://docs.microsoft.com/en-us/skypeforbusiness/hybrid/move-users-between-on-premises-and-cloud you will find the following important note and reason why I run into this error.

The value of the hosted migration override URL is a variant of the following URL:
https://adminXX.online.lync.com/HostedMigration/hostedmigrationService.svc

In the above URL, replace the XX with either two or three characters, determined as follows:

Connect to a Skype for Business Online PowerShell session, run the following cmdlet:

$credential = Get-Credential
$sfboSession = New-CsOnlineSession -Credential $credential
Import-PSSession $sfboSession

Get-CsTenant | fl identity

The resulting value will be in the following format:

OU=,OU=OCS Tenants,DC=lyncXX001,DC=local

The two- or three-digit code is the XX contained in the section, DC=lyncXX001. If it’s a two-character code, it will be a digit followed by a number (such as 0a). If it’s a three-character code, it will be two letters followed by a digit (such as jp1). In all cases, you’ll see 001 immediately after the XX code.

The hosted migration service is the service in Office 365 that performs user moves. By default, there is no need to specify a value for this parameter, as long as the hosting provider has its AutoDiscover URL properly configured and you are using an admin account the ends in .onmicrosoft.com. If you are using a user account from on-premises that synchronized to the cloud, you must specify this parameter.

Source: https://docs.microsoft.com/en-us/powershell/module/skype/move-csuser

If you are using Move-CsUser in PowerShell, you can either use an account that ends in .onmicrosoft.com, or you can use any on-premises account that is synchronized into Azure AD, provided that you also specify the HostedMigrationOverrideUrl parameter in the cmdlet

To determine the HostedMigrationOverrideUrl please read the following article from Microsoft.

Source: https://docs.microsoft.com/en-us/skypeforbusiness/hybrid/move-users-between-on-premises-and-cloud#required-administrative-credentials

So finally I changed the hosted migration override URL to my tenant specific URL and it works.

If both cases are not the issue, please read my following post, maybe it is also related to some synchronization delays in the back end from Office 365.

Move-CsUser: HostedMigration Fault: Error=(511), Description=(The user could not be moved because he or she is enabled for dial-in conferencing on-premises, but has not been assigned an Audio Conferencing license in Office 365. Users must be licensed before they can be moved to Teams or Skype for Business Online

The reason here is obviously, the user is enabled for dial-in conferencing in on-premises and doesn’t have a valid Audio Conferencing license assigned in Office 365.

To solve the problem, you can either use the switch -BypassAudioConferencingCheck, assign an Audio Conferencing license in Office 365 to the user, or disable the user in on-premises for dial-in conferencing as follows.

Enable or disable dial-in conferencing in Skype for Business Server
https://docs.microsoft.com/en-us/skypeforbusiness/manage/conferencing/enable-or-disable

Set-CsConferencingPolicy -Identity “No Dialin Conference Policy” -EnableDialInConferencing $false

Assign the new policy to the affected user.

Enable Open Federation in Skype for Business on-premise

To enable Open Federation you have to check Enable partner domain discovery in the on-premise Skype for Business Server Control Panel or using a PowerShell Cmdlet using the Management Shell.

https://docs.microsoft.com/en-us/skypeforbusiness/manage/federation-and-external-access/access-edge/enable-or-disable-federation-and-public-im-connectivity

Enable partner domain discovery   If you enable this option, Skype for Business Server uses Domain Name System (DNS) records to try to discover domains not listed in the allowed domains list, automatically evaluating incoming traffic from discovered federated partners and limiting or blocking that traffic based on trust level, amount of traffic, and administrator settings. If you do not select this option, federated user access is enabled only for users in the domains that you include on the allowed domains list. Whether or not you select this option, you can specify that individual domains to be blocked or allowed, including restricting access to specific servers running the Access Edge service in the federated domain. For details about controlling access by federated domains, see Configure support for allowed external domains.

In contrast to Open Federation, you can configure Allowed Domains list as follows. The following on my lab environment (SIP Domain: braintesting.net) will add the production SIP Federated Domain.

Links

Plan hybrid connectivity between Skype for Business Server and Microsoft 365 or Office 365
https://docs.microsoft.com/en-us/skypeforbusiness/hybrid/plan-hybrid-connectivity

Set-CsAccessEdgeConfiguration
https://docs.microsoft.com/en-us/powershell/module/skype/set-csaccessedgeconfiguration

Enable or disable discovery of federation partners in Skype for Business Server
https://docs.microsoft.com/en-us/skypeforbusiness/manage/federation-and-external-access/access-edge/enable-or-disable-discovery-of-federation-partners

Configure your on-premises Edge service to federate with Microsoft 365 or Office 365
https://docs.microsoft.com/en-us/skypeforbusiness/hybrid/configure-federation-with-skype-for-business-online#configure-your-on-premises-edge-service-to-federate-with-microsoft-365-or-office-365

Configure Skype for Business hybrid
https://docs.microsoft.com/en-us/skypeforbusiness/hybrid/configure-federation-with-skype-for-business-online

DNS requirements for Skype for Business Server
https://docs.microsoft.com/en-us/skypeforbusiness/plan-your-deployment/network-requirements/dns