Good day ,
I have 4 Exchange Servers 2013 SP1 installed on Windows Server 2012 R2 , 2 CAS and 2 Mailbox
I'm having delay problem on CAS Servers , time-taken is high most of the time and sometimes more than 20 seconds
Users have logs like 2 faild logon records ( SC-Status 401) then 2 successful logon records (SC-Status 200)
The delay is affecting some third party applications functionality , like journalling ,
there is no problem regarding hardware performance
Authentication methods used are Basic and Windows Authentication ,
virtual directory is RPC
I ran this command on Exchange CAS server
%windir%\system32\inetsrv\appcmd list requests /elapsed:10000
this is part the result
REQUEST "270000008000108e" (url:RPC_IN_DATA /rpc/rpcproxy.dll?6508eb47-7e27-4445-a96f-971de59a3d49@domain.com:6001 time:157770907 msec stage:ExecuteRequestHandler module:ManagedPipelineHandler) REQUEST "9500000080003efa" (url:RPC_OUT_DATA /rpc/rpcproxy.dll?6508eb47-7e27-4445-a96f-971de59a3d49@domain.com:6001 time:157770875 msec stage:ExecuteRequestHandler module:ManagedPipelineHandler) REQUEST "580000008000121d" (url:RPC_IN_DATA /rpc/rpcproxy.dll?27a66302-2312-436a-9dec-7087e739f1e2@domain.com:6001 time:157769907 msec stage:ExecuteRequestHandler module:ManagedPipelineHandler) REQUEST "ab000000800018b6" (url:RPC_IN_DATA /rpc/rpcproxy.dll?f525763b-b738-4c24-bef9-39e749a53746@domain.com:6001 time:157769907 msec stage:ExecuteRequestHandler module:ManagedPipelineHandler) REQUEST "a20000008000307a" (url:RPC_OUT_DATA /rpc/rpcproxy.dll?27a66302-2312-436a-9dec-7087e739f1e2@domain.com:6001 time:157769891 msec stage:ExecuteRequestHandler module:ManagedPipelineHandler) REQUEST "e400000480005933" (url:RPC_OUT_DATA /rpc/rpcproxy.dll?f525763b-b738-4c24-bef9-39e749a53746@domain.com:6001 time:157769875 msec stage:ExecuteRequestHandler module:ManagedPipelineHandler REQUEST "9a00000080002c87" (url:RPC_IN_DATA /rpc/rpcproxy.dll?09607927-b736-4002-9cba-8b0677acd142@domain.com:6001 time:157769735 msec stage:ExecuteRequestHandler module:ManagedPipelineHandler) REQUEST "6b00000080002837" (url:RPC_OUT_DATA /rpc/rpcproxy.dll?09607927-b736-4002-9cba-8b0677acd142@domain.com:6001 time:157769704 msec stage:ExecuteRequestHandler module:ManagedPipelineHandler) REQUEST "f700000180000613" (url:RPC_IN_DATA /rpc/rpcproxy.dll?f6b5075a-a817-4d24-8642-a315c246ca86@domain.com:6001 time:157768563 msec stage:ExecuteRequestHandler module:ManagedPipelineHandler) REQUEST "4700000080001cc9" (url:RPC_OUT_DATA /rpc/rpcproxy.dll?f6b5075a-a817-4d24-8642-a315c246ca86@domain.com:6001 time:157768547 msec stage:ExecuteRequestHandler module:ManagedPipelineHandler) REQUEST "410000008000330d" (url:RPC_IN_DATA /rpc/rpcproxy.dll?1464c4d3-7f26-4a91-945d-96cefbbd2cc3@domain.com:6001 time:157767563 msec stage:ExecuteRequestHandler module:ManagedPipelineHandler) REQUEST "e500000080006eb1" (url:RPC_IN_DATA /rpc/rpcproxy.dll?e86b0f73-8963-41c5-82d7-130d787b9d4f@domain.com:6001 time:157767532 msec stage:ExecuteRequestHandler module:ManagedPipelineHandler) REQUEST "e0000004800076d2" (url:RPC_OUT_DATA /rpc/rpcproxy.dll?1464c4d3-7f26-4a91-945d-96cefbbd2cc3@domain.com:6001 time:157767516 msec stage:ExecuteRequestHandler module:ManagedPipelineHandler) REQUEST "9000000080007028" (url:RPC_OUT_DATA /rpc/rpcproxy.dll?e86b0f73-8963-41c5-82d7-130d787b9d4f@domain.com:6001 time:157767500 msec stage:ExecuteRequestHandler module:ManagedPipelineHandler) REQUEST "740000008000397f" (url:RPC_IN_DATA /rpc/rpcproxy.dll?d18abcfc-8f8c-4fab-b67a-0b9ad11f87ef@domain.com:6001 time:157767141 msec stage:ExecuteRequestHandler module:ManagedPipelineHandler) REQUEST "8300000080003ffc" (url:RPC_OUT_DATA /rpc/rpcproxy.dll?d18abcfc-8f8c-4fab-b67a-0b9ad11f87ef@domain.com:6001 time:157767110 msec stage:ExecuteRequestHandler module:ManagedPipelineHandler) REQUEST "2e00000080000fca" (url:RPC_IN_DATA /rpc/rpcproxy.dll?4ad13598-850a-4604-9ced-82af31760a43@domain.com:6001 time:157766266 msec stage:ExecuteRequestHandler module:ManagedPipelineHandler) REQUEST "f3000004800050fe" (url:RPC_OUT_DATA /rpc/rpcproxy.dll?4ad13598-850a-4604-9ced-82af31760a43@domain.com:6001 time:157766235 msec stage:ExecuteRequestHandler module:ManagedPipelineHandler) REQUEST "4500000080002efa" (url:RPC_IN_DATA /rpc/rpcproxy.dll?38ac5b1a-3a66-40c8-87f5-3f6a3fff66fc@domain.com:6001 time:157763032 msec stage:ExecuteRequestHandler module:ManagedPipelineHandler) REQUEST "af0000008000302b" (url:RPC_OUT_DATA /rpc/rpcproxy.dll?38ac5b1a-3a66-40c8-87f5-3f6a3fff66fc@domain.com:6001 time:157762969 msec stage:ExecuteRequestHandler module:ManagedPipelineHandler)
i removed clients ip columns and changed the domain name for security reasons
clients ips are different , i mean its not the same client
Also ran the same command on one of the Exchange Mailbox Servers
REQUEST "cd000009800008e2" (url:RPC_IN_DATA /rpc/rpcproxy.dll?MBXSRV01.domain.local:6001 time:158343000 msec stage:ExecuteRequestHandler module:IsapiModule) REQUEST "aa0000008000065b" (url:RPC_OUT_DATA /rpc/rpcproxy.dll?MBXSRV01.domain.local:6001 time:158286625 msec stage:ExecuteRequestHandler module:IsapiModule) REQUEST "8000000880000fac" (url:RPC_IN_DATA /rpc/rpcproxy.dll?MBXSRV01.domain.local:6001 time:158286625 msec stage:ExecuteRequestHandler module:IsapiModule) REQUEST "b900000980000189" (url:RPC_OUT_DATA /rpc/rpcproxy.dll?MBXSRV01.domain.local:6001 time:158286312 msec stage:ExecuteRequestHandler module:IsapiModule) REQUEST "cd00000080000c06" (url:RPC_IN_DATA /rpc/rpcproxy.dll?MBXSRV01.domain.local:6001 time:158286250 msec stage:ExecuteRequestHandler module:IsapiModule) REQUEST "9a000009800003b9" (url:RPC_OUT_DATA /rpc/rpcproxy.dll?MBXSRV01.domain.local:6001 time:158286156 msec stage:ExecuteRequestHandler module:IsapiModule) REQUEST "e800000080000683" (url:RPC_IN_DATA /rpc/rpcproxy.dll?MBXSRV01.domain.local:6001 time:158286156 msec stage:ExecuteRequestHandler module:IsapiModule) REQUEST "8f0000008000026b" (url:RPC_OUT_DATA /rpc/rpcproxy.dll?MBXSRV01.domain.local:6001 time:158284562 msec stage:ExecuteRequestHandler module:IsapiModule) REQUEST "ed00000880000601" (url:RPC_IN_DATA /rpc/rpcproxy.dll?MBXSRV01.domain.local:6001 time:158284562 msec stage:ExecuteRequestHandler module:IsapiModule) REQUEST "fb00000080000512" (url:RPC_OUT_DATA /rpc/rpcproxy.dll?MBXSRV01.domain.local:6001 time:158282859 msec stage:ExecuteRequestHandler module:IsapiModule) REQUEST "960000098000076c" (url:RPC_IN_DATA /rpc/rpcproxy.dll?MBXSRV01.domain.local:6001 time:158282859 msec stage:ExecuteRequestHandler module:IsapiModule) REQUEST "f70000088000118f" (url:RPC_OUT_DATA /rpc/rpcproxy.dll?MBXSRV01.domain.local:6001 time:158282343 msec stage:ExecuteRequestHandler module:IsapiModule) REQUEST "8100000080000e89" (url:RPC_IN_DATA /rpc/rpcproxy.dll?MBXSRV01.domain.local:6001 time:158282343 msec stage:ExecuteRequestHandler module:IsapiModule) REQUEST "6000000080000323" (url:RPC_OUT_DATA /rpc/rpcproxy.dll?MBXSRV01.domain.local:6001 time:158280656 msec stage:ExecuteRequestHandler module:IsapiModule) REQUEST "b300000880002760" (url:RPC_IN_DATA /rpc/rpcproxy.dll?MBXSRV01.domain.local:6001 time:158280656 msec stage:ExecuteRequestHandler module:IsapiModule) REQUEST "a900000980001b50" (url:RPC_OUT_DATA /rpc/rpcproxy.dll?MBXSRV01.domain.local:6001 time:158280562 msec stage:ExecuteRequestHandler module:IsapiModule) REQUEST "cc0000008000044c" (url:RPC_IN_DATA /rpc/rpcproxy.dll?MBXSRV01.domain.local:6001 time:158280546 msec stage:ExecuteRequestHandler module:IsapiModule) REQUEST "8d00000180000463" (url:RPC_OUT_DATA /rpc/rpcproxy.dll?MBXSRV01.domain.local:6004 time:158277640 msec stage:ExecuteRequestHandler module:IsapiModule) REQUEST "f9000008800007ce" (url:RPC_IN_DATA /rpc/rpcproxy.dll?MBXSRV01.domain.local:6004 time:158277640 msec stage:ExecuteRequestHandler module:IsapiModule) REQUEST "db00000880000531" (url:RPC_OUT_DATA /rpc/rpcproxy.dll?MBXSRV01.domain.local:6001 time:158258187 msec stage:ExecuteRequestHandler module:IsapiModule)
Client IPs here are actually CAS Servers IPs
99% of the urls are /rpc/rpcproxy.dll ...
some of them are EWS and ActiveSync , but those are only 1 or 2 percent
if you noticed , domains are different between CAS and Mailbox , domain.com and domain.local
because users have to mail addresses one internal and one external
im not sure if this is related ,
does anyone have any idea about this issue ?
Regards
Best Regards Bisher Shbib