Help!
We are in the middle of migrating from Exchange 2010 SP3 to Exchange 2013 SP1 and we are being plagued with HTTP 500 errors on every Exchange 2013 server now.
I have two Client Access servers installed (EXCHFE01 & EXCHFE02) and one backend mailbox server (EXCHMB03) with the Client Access role also installed (for troubleshooting purposes). Once I get everything working I'll install two backend 2013 servers
with the mailbox role only for production (load-balanced, clustered, etc). I still have two Exchange 2010 mailbox servers left since one has a single large mailbox left on it and the other hosts public folders. I'll get to migrating those later. All Exchange
2013 servers are running Windows Server 2012. All are on the same LAN and no firewalls are between them.
On all three 2013 servers (EXCHFE01, EXCHFE02, EXCHMB03) I'm getting the dreaded HTTP 500 error whether I try to get to OWA or ECP. This did work at one point where I could at least login to the mailbox server (EXCHMB03) and migrate all my mailboxes over
to it. Sometime this morning after the two Client Access servers were installed the mailbox server stopped working (HTTP 500 errors). This is very frustrating because I can't even login to a console to manage Exchange from any server at this point.
Both Client Access servers aren't really showing anything of value in the event logs or the IIS logs. The mailbox server (EXCHMB03) does show the following event 3002:
Protocol /Microsoft-Server-ActiveSync failed to process request from identity NT AUTHORITY\SYSTEM. Exception: Microsoft.Exchange.Security.Authentication.BackendRehydrationException: Rehydration failed. Reason: Source server 'NT AUTHORITY\SYSTEM' does not have token serialization permission. at Microsoft.Exchange.Security.Authentication.BackendRehydrationModule.ProcessRequest(HttpContext httpContext) at Microsoft.Exchange.Security.Authentication.BackendRehydrationModule.OnAuthenticateRequest(Object source, EventArgs args).
...and I get the following when I run the Exchange Management Shell tool on the same mailbox server (along with lots of other red HTML text):
More Information: This error means that there was a problem while processing the request. The request was received by the Web server, but during processing a fatal error occurred, causing the 500 error.
The Client Access servers show the following errors:
Performance counter updating error. Counter name is Location cache update time, category name is MSExchange Active Manager Client. Optional code: 3. Exception: The exception thrown is : System.InvalidOperationException: The requested Performance Counter is not a custom counter, it has to be initialized as ReadOnly. at System.Diagnostics.PerformanceCounter.InitializeImpl() at System.Diagnostics.PerformanceCounter.set_RawValue(Int64 value) at Microsoft.Exchange.Diagnostics.ExPerformanceCounter.set_RawValue(Int64 value) Last worker process info : System.ArgumentException: Process with an Id of 4040 is not running. at System.Diagnostics.Process.GetProcessById(Int32 processId) at Microsoft.Exchange.Diagnostics.ExPerformanceCounter.GetLastWorkerProcessInfo() Processes running while Performance counter failed to update: 888 svchost 264 smss 532 conhost 2576 Microsoft.Exchange.UM.CallRouter 1596 WMSvc 3932 dllhost 4020 svchost 2308 dwm 5060 taskhostex 844 svchost 1972 Microsoft.Exchange.Directory.TopologyService 1228 inetinfo 1316 vmtoolsd 3896 svchost 780 svchost 1580 rundll32 1668 mqsvc 816 dwm 3840 slui 3532 rdpclip 3980 WmiPrvSE 4200 SppExtComObj 4288 winlogon 2140 SMSvcHost 676 svchost 1564 svchost 1296 Microsoft.Exchange.Diagnostics.Service 1472 Microsoft.Exchange.ServiceHost 1204 svchost 1292 ServerManagerLauncher 488 winlogon 664 svchost 572 lsass 3236 dllhost 2972 WmiPrvSE 4484 w3wp 388 csrss 1600 rundll32 564 services 1260 msdtc 5032 sppsvc 1448 MSExchangeHMHost 824 LogonUI 4204 VSSVC 2524 userinit 2700 MSExchangeFrontendTransport 1708 SMSvcHost 728 svchost 460 wininit 904 svchost 992 svchost 4996 TSTheme 4240 slui 4904 taskhost 452 csrss 4340 csrss 4 System 1164 spoolsv 1988 slClient 2760 MSExchangeHMWorker 0 Idle Performance Counters Layout information: FileMappingNotFoundException for category MSExchange Active Manager Client : Microsoft.Exchange.Diagnostics.FileMappingNotFoundException: Cound not open File mapping for name Global\netfxcustomperfcounters.1.0msexchange active manager client. Error Details: 2 at Microsoft.Exchange.Diagnostics.FileMapping.OpenFileMapping(String name, Boolean writable) at Microsoft.Exchange.Diagnostics.PerformanceCounterMemoryMappedFile.Initialize(String fileMappingName, Boolean writable) at Microsoft.Exchange.Diagnostics.ExPerformanceCounter.GetAllInstancesLayout(String categoryName)
... and...
A transient failure has occurred. The problem may resolve itself. Diagnostic information: Microsoft.Exchange.Data.DataSourceOperationException: The request failed. The remote server returned an error: (503) Server Unavailable. ---> Microsoft.Exchange.WebServices.Data.ServiceRequestException: The request failed. The remote server returned an error: (503) Server Unavailable. ---> System.Net.WebException: The remote server returned an error: (503) Server Unavailable. at System.Net.HttpWebRequest.GetResponse() at Microsoft.Exchange.WebServices.Data.EwsHttpWebRequest.Microsoft.Exchange.WebServices.Data.IEwsHttpWebRequest.GetResponse() at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.GetEwsHttpWebResponse(IEwsHttpWebRequest request) --- End of inner exception stack trace --- at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.GetEwsHttpWebResponse(IEwsHttpWebRequest request) at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.ValidateAndEmitRequest(IEwsHttpWebRequest& request) at Microsoft.Exchange.WebServices.Data.ExchangeService.InternalFindFolders(IEnumerable`1 parentFolderIds, SearchFilter searchFilter, FolderView view, ServiceErrorHandling errorHandlingMode) at Microsoft.Exchange.WebServices.Data.ExchangeService.FindFolders(FolderId parentFolderId, SearchFilter searchFilter, FolderView view) at Microsoft.Exchange.Data.Storage.Management.EwsStoreDataProvider.InvokeServiceCall[T](Func`1 callback) --- End of inner exception stack trace --- at Microsoft.Exchange.Data.Storage.Management.EwsStoreDataProvider.InvokeServiceCall[T](Func`1 callback) at Microsoft.Exchange.Data.Storage.Management.EwsStoreDataProvider.GetOrCreateFolderCore(String folderName, FolderId parentFolder, Func`1 creator) at Microsoft.Exchange.Data.Storage.Management.AsyncOperationNotificationDataProvider.GetDefaultFolder() at Microsoft.Exchange.Data.Storage.Management.EwsStoreDataProvider.<>c__DisplayClass1b`1.<InternalFindPaged>b__13() at Microsoft.Exchange.Data.Storage.Management.EwsStoreDataProvider.InvokeServiceCall[T](Func`1 callback) at Microsoft.Exchange.Data.Storage.Management.EwsStoreDataProvider.<InternalFindPaged>d__21`1.MoveNext() at Microsoft.Exchange.Data.Storage.Management.AsyncOperationNotificationDataProvider.<GetNotificationDetails>d__4.MoveNext() at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext() at Microsoft.Exchange.Servicelets.CertificateNotificationServicelet.RemoveAllNotification() at Microsoft.Exchange.Servicelets.CertificateNotificationServicelet.UpdateDataInMbx(List`1 certificates) at Microsoft.Exchange.Servicelets.CertificateNotificationServicelet.Work()
I've tried all sorts of recommended fixes but none have worked so far:
- ADSIEdit and remove msExchCanaryData0 data
- Reinstall Exchange multiple times
- Rebooted multiple times
- This article: http://social.technet.microsoft.com/Forums/exchange/en-US/08d3777c-dc03-4411-8c87-7db37d2f406a/exchange-2013-owa-login-error-http-500?forum=exchangesvrclients
- Rebooted DCs multiple times (no exchange servers are DCs)
I'm sure I've left out a few troubleshooting steps since I've been working at this for two days. Can anybody offer any advice? I'm at my wits end here...