Exchange 2013 Untrusted Cross-Forest Availability Intermittently Working

Goal:
I’m attempting to configure cross-forest availability for Exchange 2013 using the instructions here:
http://technet.microsoft.com/en-us/library/bb125182%28v=exchg.150%29.aspx
At the very bottom of the page are three different methods.  I have tried the first (per-user) and the third (untrusted) methods, with identical results.  For various unfortunate reasons, I am unable to use the Microsoft Federated Gateway for availability information (although that is configured in the production domain and I would use it if it were possible). 

Situation:
When attempting to view availability information in either OWA or Outlook, the free/busy information typically isn’t visible.  If you open and close Outlook a few times, creating meetings with the users in other domains, sometimes the other user’s information will be visible, and sometimes it will not.  When it is not, the area is filled with diagonal lines and hovering over it says “No Information”.  The situation is the same in both Adatum trying to access Contoso, and in Contoso trying to access either Adatum or Fabrikam.

I’m currently close to finishing up my third week with Microsoft Support on this issue, and am starting over with a third first level support person.  They are quickly eroding what little confidence I had in them already.  I’m posting here because I’m desperate, and web searches for my errors turn up zero results.  I fear this method of availability sharing doesn’t actually work correctly in Exchange 2013 as Microsoft is pushing organizations to use the Microsoft Federated Gateway, but I’d love to heave about anyone getting this to work, or not.

Setup:
There are three separate domains I am working with (names changed to protect the innocent).  Contoso.local is the production domain, containing Exchange 2007 and Exchange 2013 SP1 servers.  Adatum.local is a test domain set up fresh with Exchange 2013 SP1.  Fabrikam.com is a remote Exchange system that I others are connecting to without issue using Exchange 2010.

The Contoso and Adatum domain controllers are running Windows Server 2008 R2 SP1 and are running at a 2008 R2 functional levels.  The Exchange 2013 servers are all at SP1 (results were the same prior to SP1), and the OS is Windows Server 2012.  Contoso has two sites, connected via 10Gbps links, and ~10ms latency, with Exchange 2013 CAS and mailbox servers in both sites.  Adatum has a single site, and has two CAS and two mailbox servers.  Fabrikam has one internet facing server to connect to.  A handful of contacts have been created in both Contoso and Adatum for the other domains, to select to view availability.

Contoso and Adatum domains sit on different subnets, but there is no firewall or filtering between their subnets.  Routing between them is completely unimpeded.  The Fabrikam server sits on another network across the internet, but firewalls have been configured and I can browse the availability website from the Contoso CAS servers.

The CAS servers were originally set up to be load balanced, but working with Microsoft they’ve had me specify a single CAS server for autodiscover/EWS/ECP/OWA/etc in both Contoso and Adatum.  The number of actual users on Exchange 2013 in Contoso is ~10.  In Adatum, there are only a handful of mailboxes configured.  The Exchange 2007 servers in Contoso are using Public Folders for free/busy replication for other domains right now, and we don’t care at the moment if they can use the 2013 availability.  None of our testing/configurations have involved the Exchange 2007 servers.  There are no SPNs configured for the other domains in AD.

Errors:
There are three basic errors that are returned in Outlook diagnostics.  The first is the timeout error.  For a given mailbox server, the first time it is queried for availability information for a remote domain (after some amount of time of being idle) it might not respond for 70 seconds (actually somewhere between 69 and 70 seconds each time when viewing the IIS logs), and eventually fails with the timeout error.  If it doesn’t timeout, then it will respond with the Correct Response.

Once a particular mailbox server has timed out, it will typically immediately return the first Availability Error for all subsequent calls.  Less frequently, it will return Availability Error 2.  If a mailbox server returns the first Availability Error, then it will continue to return that error until it times out again or starts working.  Similarly, if a mailbox server returns the second Availability Error, then it will continue to return that error until it times out again or starts working.

If an IISRESET is performed on a mailbox server, then it will either timeout at the next cross-forest availability request, or work.  There is never an issue accessing availability information for users in the same domain as the request.

If the remote Exchange is in an errored state, then the response includes the error.  For example, if the mailbox servers in the remote domain are turned off, and the local mailbox server that you are querying happens to be responding correctly for the remote domain, then it will return an error about how no mailbox servers are available in adatum.local to service the request.

There are no Event Log errors that correspond to failed requests of any type.  IIS logs don’t show anything beyond what is shown in the Outlook diagnostics.  There are no DNS or Active Directory Replication errors in the Event Logs.

Timeout error:

CalendarEvents       : {}
ViewType             : None
MergedFreeBusyStatus : {}
WorkingHours         :
Result               : Error
ErrorCode            : ErrorTimeoutExpired
ErrorMessage         : Microsoft.Exchange.InfoWorker.Common.Availability.TimeoutExpiredException: Request could not be processed in time. Timeout occurred during 'LookupRecipientsBatchBegin'.
                       . Name of the server where exception originated: Mailbox01
ErrorDetails         : {}
ErrorProperties      : {}

Availability Error:

CalendarEvents       : {}
ViewType             : None
MergedFreeBusyStatus : {}
WorkingHours         :
Result               : Error
ErrorCode            : ErrorProxyRequestProcessingFailed
ErrorMessage         : Unable to send cross-forest request for mailbox <Free BusyTest>SMTP:freebusytest@adatum.local because of invalid configuration., inner exception: Microsoft.Exchange.InfoWorker.Common.Availability.AutoDiscoverFailedException: AvailabilityAddressSpace 'adatum.local' couldn't be used because the Autodiscover endpoint couldn't be discovered.
                       . Name of the server where exception originated: Mailbox01
ErrorDetails         : {}
ErrorProperties      : {}

Availability Error 2:

CalendarEvents       : {}
ViewType             : None
MergedFreeBusyStatus : {}
WorkingHours         :
Result               : Error
ErrorCode            : ErrorProxyRequestProcessingFailed
ErrorMessage         : Unable to send cross-forest request for mailbox <Free BusyTest>SMTP:freebusytest@adatum.local because of invalid configuration., inner exception: Microsoft.Exchange.InfoWorker.Common.Availability.AddressSpaceNotFoundException: Configuration information for forest/domain swelab.wayad.corp.wayport.net could not be found in Active Directory.
                          at Microsoft.Exchange.InfoWorker.Common.Availability.TargetForestConfigurationCache.FindByDomain(OrganizationId organizationId, String domainName)
                          at Microsoft.Exchange.InfoWorker.Common.Availability.QueryGenerator.GetTargetForestConfiguration(EmailAddress emailAddress)
                       . Name of the server where exception originated: Mailbox02
ErrorDetails         : {}
ErrorProperties      : {}

Working:

CalendarEvents       : {Microsoft.Exchange.WebServices.Data.CalendarEvent}
ViewType             : FreeBusyMerged
MergedFreeBusyStatus : {Free, Free, Free, Free...}
WorkingHours         : Microsoft.Exchange.WebServices.Data.WorkingHours
Result               : Success
ErrorCode            : NoError
ErrorMessage         :
ErrorDetails         : {}
ErrorProperties      : {}

Start : 04/09/2014 00:00:00
End : 04/12/2014 00:00:00
Subject :
Location :

Testing Methodologies:
While it is possible to dig through Outlook diagnostics and OWA, we ended up scripting out these requests to save time.  Microsoft support refuses to use the scripts, but they produce the same output that it takes them days to find in the logs, so I’ll post them here to help anyone in the future.

Through reading the documentation and experimenting, it appears that the Exchange 2013 CAS servers really do just proxy availability requests from the client to the mailbox servers.  At least by default, it seems to pick a mailbox server in the same site, but which mailbox server in the site appears to be random.  It will typically pick the same one repeatedly for a while.

The first script uses the Microsoft Exchange Web Services Managed API 2.1.
http://www.microsoft.com/en-us/download/details.aspx?id=42022
You specify a source email address, and a target address in the remote domain, and it creates a SOAP request that it sends to a CAS server of the source email address.  The CAS proxies the request to the mailbox server which either responds with a failure or the free/busy data.

The second script takes the XML SOAP request generated by the first script, and uses that to query a mailbox server directly.  That allows you to test specific mailbox servers that are working or failing, instead of randomly using whichever mailbox server the CAS happens to select.  I generated a SOAP request with the first script that I knew had some data, and then copy/pasted it into the second script to verify if data was being returned.

I’ve deleted and recreated the availability address spaces in Contoso and Adatum for each other and Fabrikam multiple times.  I’ve reset the password in the OrgWideAccount in both Adatum and Contoso, and viewed the lastBadPassword attribute in both ADs to verify it wasn’t failing authentication.  (A failed authentication also generates a 401 error that is returned to the client.)  I can access the availability site of the other domain using the credentials of the OrgWideAccount without any errors ever.

First Script:

#  Import the Exchange Web Services module
Import-Module -Name "C:\Program Files (x86)\Microsoft\Exchange\Web Services\2.1\Microsoft.Exchange.WebServices.dll"

#  Create the services object used to connect to Exchange
#  You can specify a specific Exchange version, which I had to do to connect to 2007
#  Exchange2007_SP1
#  Exchange2010
#  Exchange2010_SP1
#  Exchange2010_SP2
#  Exchange2013
#  $ExchangeVersion = [Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2007_SP1
#  $Service = New-Object Microsoft.Exchange.WebServices.Data.ExchangeService($ExchangeVersion) 
  $Service = New-Object Microsoft.Exchange.WebServices.Data.ExchangeService
$Service.UseDefaultCredentials = $true

#  Specify an SMTP address.  The autodiscover URL from the associated mailbox will be used to connect to Exchange
#  This is used to distinguish resolving from the 2007 server versus 2013
#$Service.AutodiscoverUrl("testuser1@contoso.local")  #  For Exchange 2007
$Service.AutodiscoverUrl("testuser2@contoso.local")  #  For Exchange 2013

#  Increase the amount output at the end to include the SOAP commands
$Service.TraceEnabled = $true

#  Specify time frame to get free/busy for
$StartTime = [DateTime]::Parse([DateTime]::Now.ToString("yyyy-MM-dd 0:00"))  
$EndTime = $StartTime.AddDays(7)  

#  Create the various objects needed to perform the EWS request
$drDuration = new-object Microsoft.Exchange.WebServices.Data.TimeWindow($StartTime,$EndTime)  
$AvailabilityOptions = new-object Microsoft.Exchange.WebServices.Data.AvailabilityOptions  
$AvailabilityOptions.RequestedFreeBusyView = [Microsoft.Exchange.WebServices.Data.FreeBusyViewType]::DetailedMerged  
$Attendeesbatch = New-Object "System.Collections.Generic.List[Microsoft.Exchange.WebServices.Data.AttendeeInfo]" 
$attendee = New-Object Microsoft.Exchange.WebServices.Data.AttendeeInfo($userSMTPAddress) 

#  Specify SMTP addresses of accounts to request availability for
#$Attendeesbatch.Add("testuser3@fabrikam.com")
$Attendeesbatch.Add("freebusytest@adatum.local")
#$Attendeesbatch.Add("testuser1@contoso.local")
#$Attendeesbatch.Add("testuser2@contoso.local")

#  Clear out old results so that a failed request doesn't show information still
$availresponse = ""
#  Request the availability information from Exchange
$availresponse = $service.GetUserAvailability($Attendeesbatch,$drDuration,[Microsoft.Exchange.WebServices.Data.AvailabilityData]::FreeBusy,$AvailabilityOptions)

#  Show summary information that would include errors
$availresponse.AttendeesAvailability

#  Show all of the appointments in the requested time period
foreach($avail in $availresponse.AttendeesAvailability){
    foreach($cvtEnt in $avail.CalendarEvents){
        "Start : " + $cvtEnt.StartTime"End : " + $cvtEnt.EndTime"Subject : " + $cvtEnt.Details.Subject"Location : " + $cvtEnt.Details.Location""
    }
}

Second Script:

#  Change the server in this URL to specify which mailbox server to access
$url = 'https://mailbox01.contoso.local:444/EWS/Exchange.asmx'

#  Uncomment the below lines if you want to query EWS using credentials other than
#  the ones used to run the script.
#If(!(Test-Path variable:global:cred))
#{
#    $cred = Get-Credential
#}

function Execute-SOAPRequest 
( 
        [Xml]    $SOAPRequest, 
        [String] $URL 
) 
{ 
        write-host "Sending SOAP Request To Server: $URL" 
        $soapWebRequest = [System.Net.WebRequest]::Create($URL) 

        #  These appear to be the only things needed in the headers when making the request
        $soapWebRequest.ContentType = 'text/xml;charset="utf-8"' 
        $soapWebRequest.Accept      = "text/xml" 
        $soapWebRequest.Method      = "POST" 
        If(Test-Path variable:global:cred)
        {
            $soapWebRequest.Credentials = $cred
        }
        Else
        {
            $soapWebRequest.UseDefaultCredentials = $true
        }
        write-host "Initiating Send." 
        $requestStream = $soapWebRequest.GetRequestStream() 
        $SOAPRequest.Save($requestStream) 
        $requestStream.Close() 
        write-host "Send Complete, Waiting For Response." 
        $resp = $soapWebRequest.GetResponse() 
        $responseStream = $resp.GetResponseStream() 
        $soapReader = [System.IO.StreamReader]($responseStream) 
        $ReturnXml = [Xml] $soapReader.ReadToEnd() 
        $responseStream.Close() 
        write-host "Response Received."

        return $ReturnXml 
}


#  The specing and line returns in the below variable are important for some reason
#  For example, there must be a line return after the @' on the first line, or it's invalid...
#  Change the line with this:
#  <t:Address>freebusytest@adatum.local</t:Address>
#  to the email address in the domain you want to query
$soap = [xml]@'<?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><t:RequestServerVersion Version="Exchange2013_SP1" /><t:TimeZoneContext><t:TimeZoneDefinition Name="(UTC-06:00) Central Time (US &amp; Canada)" Id="Central Standard Time"><t:Periods><t:Period Bias="P0DT6H0M0.0S" Name="Standard" Id="Std" /><t:Period Bias="P0DT5H0M0.0S" Name="Daylight" Id="Dlt/1" /><t:Period Bias="P0DT5H0M0.0S" Name="Daylight" Id="Dlt/2007" /></t:Periods><t:TransitionsGroups><t:TransitionsGroup Id="0"><t:RecurringDayTransition><t:To Kind="Period">Dlt/1</t:To><t:TimeOffset>P0DT2H0M0.0S</t:TimeOffset><t:Month>4</t:Month><t:DayOfWeek>Sunday</t:DayOfWeek><t:Occurrence>1</t:Occurrence></t:RecurringDayTransition><t:RecurringDayTransition><t:To Kind="Period">Std</t:To><t:TimeOffset>P0DT2H0M0.0S</t:TimeOffset><t:Month>10</t:Month><t:DayOfWeek>Sunday</t:DayOfWeek><t:Occurrence>-1</t:Occurrence></t:RecurringDayTransition></t:TransitionsGroup><t:TransitionsGroup Id="1"><t:RecurringDayTransition><t:To Kind="Period">Dlt/2007</t:To><t:TimeOffset>P0DT2H0M0.0S</t:TimeOffset><t:Month>3</t:Month><t:DayOfWeek>Sunday</t:DayOfWeek><t:Occurrence>2</t:Occurrence></t:RecurringDayTransition><t:RecurringDayTransition><t:To Kind="Period">Std</t:To><t:TimeOffset>P0DT2H0M0.0S</t:TimeOffset><t:Month>11</t:Month><t:DayOfWeek>Sunday</t:DayOfWeek><t:Occurrence>1</t:Occurrence></t:RecurringDayTransition></t:TransitionsGroup></t:TransitionsGroups><t:Transitions><t:Transition><t:To Kind="Group">0</t:To></t:Transition><t:AbsoluteDateTransition><t:To Kind="Group">1</t:To><t:DateTime>2007-01-01T06:00:00.000Z</t:DateTime></t:AbsoluteDateTransition></t:Transitions></t:TimeZoneDefinition></t:TimeZoneContext></soap:Header><soap:Body><m:GetUserAvailabilityRequest><m:MailboxDataArray><t:MailboxData><t:Email><t:Address>freebusytest@adatum.local</t:Address></t:Email><t:AttendeeType>Required</t:AttendeeType><t:ExcludeConflicts>false</t:ExcludeConflicts></t:MailboxData></m:MailboxDataArray><t:FreeBusyViewOptions><t:TimeWindow><t:StartTime>2014-04-03T00:00:00</t:StartTime><t:EndTime>2014-04-10T00:00:00</t:EndTime></t:TimeWindow><t:MergedFreeBusyIntervalInMinutes>30</t:MergedFreeBusyIntervalInMinutes><t:RequestedView>DetailedMerged</t:RequestedView></t:FreeBusyViewOptions></m:GetUserAvailabilityRequest></soap:Body></soap:Envelope>
'@

$ret = Execute-SOAPRequest $soap $url

#  Uncomment out one of the below two lines to get output in different alternative formats
#$ret | Export-Clixml  c:\temp\1.xml;Get-Content c:\temp\1.xml
#$ret.InnerXml

#  If the request is successful, show the appointments, otherwise show the failure message
If ($ret.Envelope.Body.GetUserAvailabilityResponse.FreeBusyResponseArray.FreeBusyResponse.ResponseMessage.ResponseClass -eq 'Success')
{
    $ret.Envelope.Body.GetUserAvailabilityResponse.FreeBusyResponseArray.FreeBusyResponse.FreeBusyView.CalendarEventArray.CalendarEvent
}
Else
{
    $ret.Envelope.Body.GetUserAvailabilityResponse.FreeBusyResponseArray.FreeBusyResponse.ResponseMessage
}