We are experiencing intermittent 500.0 errors, around 10K out of 2Million (page hits).
Windows Server 2008 R2 x64, IIS 7.5
Logging at IIS at logs we have noticed that this problem is much larger than what we thought was happening. They appear all over the place The appear on static files as well as on the application (php files). They are also spread out during the day, with peaks when I have most activity. The server also run several sites, but all of them have the same type of issues.
We have setup failed request logging and really can't make out what might be the cause.
Any help highly appreciated.
Line 30 seems to be the issue. (no client Certificates)
| <div>No.</div> | EventName | Details | Time | |
| 1 | i | GENERAL_REQUEST_START | SiteId="2", AppPoolId="app001", ConnId="1610645359", RawConnId="0", RequestURL="https://www.xxxxx.com.co:443/Blank.htm", RequestVerb="GET" | 26:42.0 |
| 2 | | PRE_BEGIN_REQUEST_START | ModuleName="MfeEngine" | 26:42.0 |
| 3 | | PRE_BEGIN_REQUEST_END | ModuleName="MfeEngine", NotificationStatus="NOTIFICATION_CONTINUE" | 26:42.0 |
| 4 | | PRE_BEGIN_REQUEST_START | ModuleName="FailedRequestsTracingModule" | 26:42.0 |
| 5 | | PRE_BEGIN_REQUEST_END | ModuleName="FailedRequestsTracingModule", NotificationStatus="NOTIFICATION_CONTINUE" | 26:42.0 |
| 6 | | PRE_BEGIN_REQUEST_START | ModuleName="RequestMonitorModule" | 26:42.0 |
| 7 | | PRE_BEGIN_REQUEST_END | ModuleName="RequestMonitorModule", NotificationStatus="NOTIFICATION_CONTINUE" | 26:42.0 |
| 8 | | PRE_BEGIN_REQUEST_START | ModuleName="IsapiFilterModule" | 26:42.0 |
| 9 | i | FILTER_PREPROC_HEADERS_START | 26:42.0 | |
| 10 | i | FILTER_START | FilterName="C:\Windows\Microsoft.NET\Framework64\v2.0.50727\aspnet_filter.dll" | 26:42.0 |
| 11 | | GENERAL_SET_REQUEST_HEADER | HeaderName="AspFilterSessionId", HeaderValue="", Replace="true" | 26:42.0 |
| 12 | i | FILTER_SET_REQ_HEADER | HeaderName="AspFilterSessionId:", HeaderValue="" | 26:42.0 |
| 13 | i | FILTER_END | NotificationStatus="SF_STATUS_REQ_NEXT_NOTIFICATION" | 26:42.0 |
| 14 | i | FILTER_PREPROC_HEADERS_END | 26:42.0 | |
| 15 | | PRE_BEGIN_REQUEST_END | ModuleName="IsapiFilterModule", NotificationStatus="NOTIFICATION_CONTINUE" | 26:42.0 |
| 16 | | PRE_BEGIN_REQUEST_START | ModuleName="WebDAVModule" | 26:42.0 |
| 17 | | PRE_BEGIN_REQUEST_END | ModuleName="WebDAVModule", NotificationStatus="NOTIFICATION_CONTINUE" | 26:42.0 |
| 18 | | PRE_BEGIN_REQUEST_START | ModuleName="RewriteModule" | 26:42.0 |
| 19 | | PRE_BEGIN_REQUEST_END | ModuleName="RewriteModule", NotificationStatus="NOTIFICATION_CONTINUE" | 26:42.0 |
| 20 | | PRE_BEGIN_REQUEST_START | ModuleName="AdvancedLoggingModule" | 26:42.0 |
| 21 | | PRE_BEGIN_REQUEST_END | ModuleName="AdvancedLoggingModule", NotificationStatus="NOTIFICATION_CONTINUE" | 26:42.0 |
| 22 | i | GENERAL_ENDPOINT_INFORMATION | RemoteAddress="190.85.208.242", RemotePort="32018", LocalAddress="173.193.234.251", LocalPort="443" | 26:42.0 |
| 23 | i | GENERAL_REQUEST_HEADERS | Headers="Connection: keep-alive | 26:42.0 |
| Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 | ||||
| Accept-Encoding: gzip,deflate,sdch | ||||
| Accept-Language: es-419,es;q=0.8 | ||||
| Cookie: PHPSESSID=00a6ab1a381d7a797f836101495e51a6; __utmt=1; __utma=46866085.1527617756.1434727931.1435324820.1435332600.3; __utmb=46866085.76.10.1435332600; __utmc=46866085; __utmz=46866085.1434727931.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none) | ||||
| Host: sab2.refinancia.com.co | ||||
| Referer: https://wwwxxxxx.com.co/GestionComercial.php | ||||
| User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/34.0.1847.116 Chrome/34.0.1847.116 Safari/537.36 | ||||
| " | ||||
| 24 | i | URL_CACHE_ACCESS_START | RequestURL="/Blank.htm" | 26:42.0 |
| 25 | i | URL_CACHE_ACCESS_END | PhysicalPath="", URLInfoFromCache="true", URLInfoAddedToCache="false", ErrorCode="The operation completed successfully. | 26:42.0 |
| (0x0)" | ||||
| 26 | i | GENERAL_GET_URL_METADATA | PhysicalPath="", AccessPerms="545" | 26:42.0 |
| 27 | i | HANDLER_CHANGED | OldHandlerName="", NewHandlerName="StaticFile", NewHandlerModules="StaticFileModule,DefaultDocumentModule,DirectoryListingModule", NewHandlerScriptProcessor="", NewHandlerType="" | 26:42.0 |
| 28 | | NOTIFY_MODULE_START | ModuleName="IsapiFilterModule", Notification="MAP_PATH", fIsPostNotification="false" | 26:42.0 |
| 29 | | NOTIFY_MODULE_END | ModuleName="IsapiFilterModule", Notification="MAP_PATH", fIsPostNotificationEvent="false", NotificationStatus="NOTIFICATION_CONTINUE" | 26:42.0 |
| 30 | r | MODULE_SET_RESPONSE_ERROR_STATUS | ModuleName="IIS Web Core", Notification="BEGIN_REQUEST", HttpStatus="500", HttpReason="Internal Server Error", HttpSubStatus="0", ErrorCode="The I/O operation has been aborted because of either a thread exit or an application request. | 26:42.4 |
| Warning | (0x800703e3)", ConfigExceptionInfo="" | |||
| 31 | | NOTIFY_MODULE_START | ModuleName="CustomLoggingModule", Notification="LOG_REQUEST", fIsPostNotification="false" | 26:42.4 |
| 32 | | NOTIFY_MODULE_END | ModuleName="CustomLoggingModule", Notification="LOG_REQUEST", fIsPostNotificationEvent="false", NotificationStatus="NOTIFICATION_CONTINUE" | 26:42.4 |
| 33 | | NOTIFY_MODULE_START | ModuleName="ClientLoggingHandler", Notification="END_REQUEST", fIsPostNotification="false" | 26:42.4 |
| 34 | | NOTIFY_MODULE_END | ModuleName="ClientLoggingHandler", Notification="END_REQUEST", fIsPostNotificationEvent="false", NotificationStatus="NOTIFICATION_CONTINUE" | 26:42.4 |
| 35 | | NOTIFY_MODULE_START | ModuleName="WebDAVModule", Notification="SEND_RESPONSE", fIsPostNotification="false" | 26:42.4 |
| 36 | | NOTIFY_MODULE_END | ModuleName="WebDAVModule", Notification="SEND_RESPONSE", fIsPostNotificationEvent="false", NotificationStatus="NOTIFICATION_CONTINUE" | 26:42.4 |
| 37 | | NOTIFY_MODULE_START | ModuleName="HttpLoggingModule", Notification="SEND_RESPONSE", fIsPostNotification="false" | 26:42.4 |
| 38 | | NOTIFY_MODULE_END | ModuleName="HttpLoggingModule", Notification="SEND_RESPONSE", fIsPostNotificationEvent="false", NotificationStatus="NOTIFICATION_CONTINUE" | 26:42.4 |
| 39 | | NOTIFY_MODULE_START | ModuleName="IsapiModule", Notification="SEND_RESPONSE", fIsPostNotification="false" | 26:42.4 |
| 40 | | NOTIFY_MODULE_END | ModuleName="IsapiModule", Notification="SEND_RESPONSE", fIsPostNotificationEvent="false", NotificationStatus="NOTIFICATION_CONTINUE" | 26:42.4 |
| 41 | | NOTIFY_MODULE_START | ModuleName="DigestAuthenticationModule", Notification="SEND_RESPONSE", fIsPostNotification="false" | 26:42.4 |
| 42 | | NOTIFY_MODULE_END | ModuleName="DigestAuthenticationModule", Notification="SEND_RESPONSE", fIsPostNotificationEvent="false", NotificationStatus="NOTIFICATION_CONTINUE" | 26:42.4 |
| 43 | | NOTIFY_MODULE_START | ModuleName="WindowsAuthenticationModule", Notification="SEND_RESPONSE", fIsPostNotification="false" | 26:42.4 |
| 44 | | NOTIFY_MODULE_END | ModuleName="WindowsAuthenticationModule", Notification="SEND_RESPONSE", fIsPostNotificationEvent="false", NotificationStatus="NOTIFICATION_CONTINUE" | 26:42.4 |
| 45 | | NOTIFY_MODULE_START | ModuleName="BasicAuthenticationModule", Notification="SEND_RESPONSE", fIsPostNotification="false" | 26:42.4 |
| 46 | | NOTIFY_MODULE_END | ModuleName="BasicAuthenticationModule", Notification="SEND_RESPONSE", fIsPostNotificationEvent="false", NotificationStatus="NOTIFICATION_CONTINUE" | 26:42.4 |
| 47 | | NOTIFY_MODULE_START | ModuleName="ProtocolSupportModule", Notification="SEND_RESPONSE", fIsPostNotification="false" | 26:42.4 |
| 48 | | NOTIFY_MODULE_END | ModuleName="ProtocolSupportModule", Notification="SEND_RESPONSE", fIsPostNotificationEvent="false", NotificationStatus="NOTIFICATION_CONTINUE" | 26:42.4 |
| 49 | | NOTIFY_MODULE_START | ModuleName="IsapiFilterModule", Notification="SEND_RESPONSE", fIsPostNotification="false" | 26:42.4 |
| 50 | | NOTIFY_MODULE_END | ModuleName="IsapiFilterModule", Notification="SEND_RESPONSE", fIsPostNotificationEvent="false", NotificationStatus="NOTIFICATION_CONTINUE" | 26:42.4 |
| 51 | | NOTIFY_MODULE_START | ModuleName="DynamicCompressionModule", Notification="SEND_RESPONSE", fIsPostNotification="false" | 26:42.4 |
| 52 | i | DYNAMIC_COMPRESSION_START | 26:42.4 | |
| 53 | i | DYNAMIC_COMPRESSION_NOT_SUCCESS | Reason="NOT_SUCCESS_STATUS" | 26:42.4 |
| 54 | i | DYNAMIC_COMPRESSION_END | 26:42.4 | |
| 55 | | NOTIFY_MODULE_END | ModuleName="DynamicCompressionModule", Notification="SEND_RESPONSE", fIsPostNotificationEvent="false", NotificationStatus="NOTIFICATION_CONTINUE" | 26:42.4 |
| 56 | | NOTIFY_MODULE_START | ModuleName="HttpCacheModule", Notification="SEND_RESPONSE", fIsPostNotification="false" | 26:42.4 |
| 57 | | NOTIFY_MODULE_END | ModuleName="HttpCacheModule", Notification="SEND_RESPONSE", fIsPostNotificationEvent="false", NotificationStatus="NOTIFICATION_CONTINUE" | 26:42.4 |
| 58 | | NOTIFY_MODULE_START | ModuleName="RewriteModule", Notification="SEND_RESPONSE", fIsPostNotification="false" | 26:42.4 |
| 59 | | NOTIFY_MODULE_END | ModuleName="RewriteModule", Notification="SEND_RESPONSE", fIsPostNotificationEvent="false", NotificationStatus="NOTIFICATION_CONTINUE" | 26:42.4 |
| 60 | | NOTIFY_MODULE_START | ModuleName="CustomErrorModule", Notification="SEND_RESPONSE", fIsPostNotification="false" | 26:42.4 |
| 61 | i | GENERAL_SEND_CUSTOM_ERROR | HttpStatus="500", HttpSubStatus="0", FileNameOrURL="500.htm" | 26:42.4 |
| 62 | i | FILE_CACHE_ACCESS_START | FileName="C:\inetpub\custerr\en-US\500.htm", UserName="", DomainName="" | 26:42.4 |
| 63 | i | FILE_CACHE_ACCESS_END | Successful="true", FileFromCache="true", FileAddedToCache="false", FileDirmoned="true", LastModCheckErrorIgnored="true", ErrorCode="The operation completed successfully. | 26:42.4 |
| (0x0)", LastModifiedTime="Sat, 17 Dec 2011 19:21:03 GMT" | ||||
| 64 | | NOTIFY_MODULE_END | ModuleName="CustomErrorModule", Notification="SEND_RESPONSE", fIsPostNotificationEvent="false", NotificationStatus="NOTIFICATION_CONTINUE" | 26:42.4 |
| 65 | i | HTTPSYS_CACHEABLE | HttpsysCacheable="false", Reason="NO_PIPELINE_ENABLE", CachePolicy="NO_CACHE", TimeToLive="0" | 26:42.4 |
| 66 | i | GENERAL_FLUSH_RESPONSE_START | 26:42.4 | |
| 67 | i | GENERAL_RESPONSE_HEADERS | Headers="Content-Type: text/html | 26:42.4 |
| Server: Microsoft-IIS/7.5 | ||||
| X-Powered-By: ASP.NET | ||||
| X-UA-Compatible: Edge | ||||
| " | ||||
| 68 | | GENERAL_RESPONSE_ENTITY_BUFFER | Buffer="<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> | 26:42.4 |
| <html xmlns="http://www.w3.org/1999/xhtml"> | ||||
| <head> | ||||
| <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/> | ||||
| <title>500 - Internal server error.</title> | ||||
| <style type="text/css"> | ||||
| <!-- | ||||
| body{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;} | ||||
| fieldset{padding:0 15px 10px 15px;} | ||||
| h1{font-size:2.4em;margin:0;color:#FFF;} | ||||
| h2{font-size:1.7em;margin:0;color:#CC0000;} | ||||
| h3{font-size:1.2em;margin:10px 0 0 0;color:#000000;} | ||||
| #header{width:96%;margin:0 0 0 0;padding:6px 2% 6px 2%;font-family:"trebuchet MS", Verdana, sans-serif;color:#FFF; | ||||
| background-color:#555555;} | ||||
| #content{margin:0 0 0 2%;position:relative;} | ||||
| .content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;position:relative;} | ||||
| --> | ||||
| </style> | ||||
| </head> | ||||
| <body> | ||||
| <div id="header"><h1>Server Error</h1></div> | ||||
| <div id="content"> | ||||
| <div class="content-container"><fieldset> | ||||
| <h2>500 - Internal server error.</h2> | ||||
| <h3>There is a problem with the resource you are looking for, and it cannot be displayed.</h3> | ||||
| </fieldset></div> | ||||
| </div> | ||||
| </body> | ||||
| </html> | ||||
| " | ||||
| 69 | i | GENERAL_FLUSH_RESPONSE_END | BytesSent="0", ErrorCode="An operation was attempted on a nonexistent network connection. | 26:42.4 |
| (0x800704cd)" | ||||
| 70 | i | GENERAL_REQUEST_END | BytesSent="0", BytesReceived="701", HttpStatus="500", HttpSubStatus="0" | 26:42.4 |