Windows Server 2008 R2 STD BSOD crash reason

suvaldykit · Oct 13, 2012

Hi Everyone,
Recently our Server started to crash. I have managed to get DMP file analized, but now i am not sure what is actually causing it to crash, can anyone help me with this DMP file, i need to find out is this specialized software we use "TP Shell" is causing crashes or this file win32k.sys is causing problems. If this win32k.sys file crashes, i please for help what is this file about? And possible fix senarios. Thank you. I cant really restore server to earlier point of time it will cause all production a hassle.
Here is my dmp file:

Code:

Microsoft (R) Windows Debugger Version 6.2.9200.16384 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Windows\MEMORY.DMP]
Kernel Summary Dump File: Only kernel address space is available

Symbol search path is: SRV*f:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows 7 Kernel Version 7601 (Service Pack 1) MP (16 procs) Free x64
Product: LanManNt, suite: TerminalServer
Built by: 7601.17835.amd64fre.win7sp1_gdr.120503-2030
Machine Name:
Kernel base = 0xfffff800`01856000 PsLoadedModuleList = 0xfffff800`01a9a670
Debug session time: Sun Oct 7 07:22:32.513 2012 (UTC + 3:00)
System Uptime: 2 days 9:56:46.865
Loading Kernel Symbols
...............................................................
................................................................
.................................
Loading User Symbols
PEB is paged out (Peb.Ldr = 000007ff`fffdd018). Type ".hh dbgerr001" for details
Loading unloaded module list
..........
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 3B, {c0000005, fffff960000bd010, fffff8800e684c10, 0}

Page 1632 not present in the dump file. Type ".hh dbgerr004" for details
Probably caused by : win32k.sys ( win32k!xxxProcessEventMessage+1c0 )

Followup: MachineOwner
---------

12: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c0000005, Exception code that caused the bugcheck
Arg2: fffff960000bd010, Address of the instruction which caused the bugcheck
Arg3: fffff8800e684c10, Address of the context record for the exception that caused the bugcheck
Arg4: 0000000000000000, zero.

Debugging Details:
------------------


EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

FAULTING_IP: 
win32k!xxxProcessEventMessage+1c0
fffff960`000bd010 017308 add dword ptr [rbx+8],esi

CONTEXT: fffff8800e684c10 -- (.cxr 0xfffff8800e684c10)
rax=fffff8800e685640 rbx=0000000000000000 rcx=fffff8800e685598
rdx=fffff900c0a0c010 rsi=0000000000000001 rdi=0000000000000000
rip=fffff960000bd010 rsp=fffff8800e6855f0 rbp=fffff8800e685790
r8=0000000000000000 r9=0000000000000000 r10=0000000000000000
r11=fffff8800e685510 r12=fffff900c2059c20 r13=fffff8800e685b68
r14=fffff900c2059c20 r15=0000000000000001
iopl=0 nv up ei ng nz na pe nc
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00010282
win32k!xxxProcessEventMessage+0x1c0:
fffff960`000bd010 017308 add dword ptr [rbx+8],esi ds:002b:00000000`00000008=????????
Resetting default scope

DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT

BUGCHECK_STR: 0x3B

PROCESS_NAME: TP.Shell.XAF.W

CURRENT_IRQL: 0

LAST_CONTROL_TRANSFER: from fffff960000d0b6d to fffff960000bd010

STACK_TEXT: 
fffff880`0e6855f0 fffff960`000d0b6d : 00000000`00000001 00000000`00000000 00000000`00000001 00000000`00000018 : win32k!xxxProcessEventMessage+0x1c0
fffff880`0e6856c0 fffff960`0010b747 : fffff900`c2059c20 fffff880`0e685b68 fffff900`c0a066e0 00000000`00000400 : win32k!xxxScanSysQueue+0x575
fffff880`0e685a00 fffff960`0010bbd5 : 00000000`00000400 fffff800`000020c8 00000000`00000400 fffffa80`00007fff : win32k!xxxRealInternalGetMessage+0x453
fffff880`0e685ae0 fffff960`00104627 : 00000000`00000000 00000000`00000000 fffff900`c0a09c30 fffff900`c01194a0 : win32k!xxxInternalGetMessage+0x35
fffff880`0e685b20 fffff800`018d4453 : fffffa80`13d75b00 00000000`001dc6b8 fffff880`0e685bc8 00000000`00000000 : win32k!NtUserPeekMessage+0x77
fffff880`0e685bb0 00000000`772f908a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
00000000`001dc698 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x772f908a


FOLLOWUP_IP: 
win32k!xxxProcessEventMessage+1c0
fffff960`000bd010 017308 add dword ptr [rbx+8],esi

SYMBOL_STACK_INDEX: 0

SYMBOL_NAME: win32k!xxxProcessEventMessage+1c0

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: win32k

IMAGE_NAME: win32k.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 5006fd0d

STACK_COMMAND: .cxr 0xfffff8800e684c10 ; kb

FAILURE_BUCKET_ID: X64_0x3B_win32k!xxxProcessEventMessage+1c0

BUCKET_ID: X64_0x3B_win32k!xxxProcessEventMessage+1c0

Followup: MachineOwner
---------

Looking forward to your comments, if we agree that its Tp Shell software fault, i will handle problem to vendor of this software, but i need to be sure, if its not, i will have to contact MS support, but i rather not.

Help is very much appreciated.

niemiro · Oct 13, 2012

Hello, and welcome to Sysnative :)

Can I ask you to run the posting instructions here, and post back in this thread? https://www.sysnative.com/forums/sh...D)-Posting-Instructions-Windows-8-7-amp-Vista

We can often get more out of the dump than just analyze -v through the use of other more complicated commands, which is why we ask for the dump files and other requested info to be uploaded to us.

Thank you very much,

Richard

Other threads: http://social.msdn.microsoft.com/Forums/en-US/windbg/thread/6f26acb4-f987-407f-a6f9-60f82a197a0f

usasma · Oct 13, 2012

Is this a production server? Can you afford downtime while you run tests that we may ask for?

If you can afford the downtime - then please run Driver Verifier according to these instructions: http://www.carrona.org/verifier.html
This should cause the system to crash - and we hope that verifier will force the system to give up the driver name during the crash.

suvaldykit · Oct 15, 2012

niemiro said:
Hello, and welcome to Sysnative :)

Can I ask you to run the posting instructions here, and post back in this thread? https://www.sysnative.com/forums/sh...D)-Posting-Instructions-Windows-8-7-amp-Vista

We can often get more out of the dump than just analyze -v through the use of other more complicated commands, which is why we ask for the dump files and other requested info to be uploaded to us.

Thank you very much,

Richard

Other threads: http://social.msdn.microsoft.com/Forums/en-US/windbg/thread/6f26acb4-f987-407f-a6f9-60f82a197a0f

Hi niemiro,

I have done these tests, i have them uploaded to http://www.suvaldykit.lt/tests.zip

· OS - Windows Server 2010 R2 STD SP1
· x64
· What was original installed OS on system?
None.
· Is the OS an OEM version (came pre-installed on system) or full retail version (YOU purchased it from retailer)?
No it is OLP.
· Age of system (hardware)
2 years
· Age of OS installation - have you re-installed the OS?
no its 2 years aswell.
· CPU
2 x Intel Xeon E5520
· Video Card
Ati ES1000
· MotherBoard
It HP Proliant ML350 G6 server.
· Power Supply - brand & wattage
2 x Original PSU from HP
· System Manufacturer
HP
· Exact model number (if laptop, check label on bottom)
487930-421

I hope that helps.

usasma · Oct 15, 2012

Perfmon /report shows that HP NC360T PCIe DP Gigabit Server Adapter is disabled. Is this deliberate? If so, why?
Please:
- enable the device
- update the drivers
- disable the device again (if so desired)

It's better to remove the device from the system if at all possible.
Next best is to disable the device in the BIOS
Last option is to disable within Windows (as sometimes the drivers load before the device is disabled).

The dumps don't show anything specific - so we'll have to ask you to run Driver Verifier. Please follow these instructions when doing so: http://www.carrona.org/verifier.html

My first suspicion is the Acronis software, you might want to temporarily remove it to see if that helps (it sometimes causes BSOD's on Win7 systems).

Your ATI video drivers date from 2009, please update them to the latest available version at http://ati.amd.com

This is outside of my abilities, but it may be significant:

Event[68]:
Log Name: System
Source: Microsoft-Windows-GroupPolicy
Date: 2012-10-15T07:49:21.787
Event ID: 1058
Task: N/A
Level: Error
Opcode: Start
Keyword: N/A
User: S-1-5-21-2631394776-2534484819-2867778882-1664
User Name: ABGEONAFTA\vdo
Computer: geo-srv-01.abgeonafta.local
Description:
The processing of Group Policy failed. Windows attempted to read the file \\abgeonafta.local\sysvol\abgeonafta.local\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\gpt.ini from a domain controller and was not successful. Group Policy settings may not be applied until this event is resolved. This issue may be transient and could be caused by one or more of the following:
a) Name Resolution/Network Connectivity to the current domain controller.
b) File Replication Service Latency (a file created on another domain controller has not replicated to the current domain controller).
c) The Distributed File System (DFS) client has been disabled.

Lot's of other networking errors in the System event log file also.

I'm going to ask one of our networking guru's to see what they can find.....

Analysis:
The following is for informational purposes only.

Code:

**************************Sun Oct  7 00:22:32.513 2012 (UTC - 4:00)**************************
Loading Dump File [C:\Users\John\_jcgriff2_\dbug\__Kernel__\100712-40092-01.dmp]
Windows 7 Kernel Version 7601 (Service Pack 1) MP (16 procs) Free x64
Built by: 7601.17835.amd64fre.win7sp1_gdr.120503-2030
System Uptime:2 days 9:56:46.865
BugCheck Code: BugCheck 3B, {c0000005, fffff960000bd010, fffff8800e684c10, 0}
Probably caused by :win32k.sys ( win32k!xxxProcessEventMessage+1c0 )
BugCheck Info: SYSTEM_SERVICE_EXCEPTION (3b)
DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT_SERVER
BUGCHECK_STR:  0x3B
PROCESS_NAME: TP.Shell.XAF.W
FAILURE_BUCKET_ID: X64_0x3B_win32k!xxxProcessEventMessage+1c0
CPUID:        "Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz"
MaxSpeed:     2270
CurrentSpeed: 2266
  BIOS Version                  D22
  BIOS Release Date             05/05/2011
  Manufacturer                  HP
  Product Name                  ProLiant ML350 G6
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
**************************Thu Oct  4 00:44:01.057 2012 (UTC - 4:00)**************************
Loading Dump File [C:\Users\John\_jcgriff2_\dbug\__Kernel__\100412-41012-01.dmp]
Windows 7 Kernel Version 7601 (Service Pack 1) MP (16 procs) Free x64
Built by: 7601.17835.amd64fre.win7sp1_gdr.120503-2030
System Uptime:0 days 10:53:34.134
BugCheck Code: BugCheck 3B, {c0000005, fffff9600012cfe0, fffff8800f555c10, 0}
Probably caused by :win32k.sys ( win32k!xxxProcessEventMessage+1c0 )
BugCheck Info: SYSTEM_SERVICE_EXCEPTION (3b)
DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT_SERVER
BUGCHECK_STR:  0x3B
PROCESS_NAME: TP.Shell.XAF.W
FAILURE_BUCKET_ID: X64_0x3B_win32k!xxxProcessEventMessage+1c0
CPUID:        "Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz"
MaxSpeed:     2270
CurrentSpeed: 2266
  BIOS Version                  D22
  BIOS Release Date             05/05/2011
  Manufacturer                  HP
  Product Name                  ProLiant ML350 G6
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
**************************Wed Oct  3 13:46:06.641 2012 (UTC - 4:00)**************************
Loading Dump File [C:\Users\John\_jcgriff2_\dbug\__Kernel__\100312-38906-01.dmp]
Windows 7 Kernel Version 7601 (Service Pack 1) MP (16 procs) Free x64
Built by: 7601.17835.amd64fre.win7sp1_gdr.120503-2030
System Uptime:6 days 0:03:19.847
BugCheck Code: BugCheck 3B, {c0000005, fffff960000acfe0, fffff8800df0bc10, 0}
Probably caused by :win32k.sys ( win32k!xxxProcessEventMessage+1c0 )
BugCheck Info: SYSTEM_SERVICE_EXCEPTION (3b)
DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT_SERVER
BUGCHECK_STR:  0x3B
PROCESS_NAME: TP.Shell.XAF.W
FAILURE_BUCKET_ID: X64_0x3B_win32k!xxxProcessEventMessage+1c0
CPUID:        "Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz"
MaxSpeed:     2270
CurrentSpeed: 2266
  BIOS Version                  D22
  BIOS Release Date             05/05/2011
  Manufacturer                  HP
  Product Name                  ProLiant ML350 G6
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
**************************Thu Sep 27 13:39:20.673 2012 (UTC - 4:00)**************************
Loading Dump File [C:\Users\John\_jcgriff2_\dbug\__Kernel__\092712-31855-01.dmp]
Windows 7 Kernel Version 7601 (Service Pack 1) MP (16 procs) Free x64
Built by: 7601.17835.amd64fre.win7sp1_gdr.120503-2030
System Uptime:0 days 3:43:51.674
BugCheck Code: BugCheck 3B, {c0000005, fffff960000ccfe0, fffff8800f837c10, 0}
Probably caused by :win32k.sys ( win32k!xxxProcessEventMessage+1c0 )
BugCheck Info: SYSTEM_SERVICE_EXCEPTION (3b)
DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT_SERVER
BUGCHECK_STR:  0x3B
PROCESS_NAME: TP.Shell.XAF.W
FAILURE_BUCKET_ID: X64_0x3B_win32k!xxxProcessEventMessage+1c0
CPUID:        "Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz"
MaxSpeed:     2270
CurrentSpeed: 2266
  BIOS Version                  D22
  BIOS Release Date             05/05/2011
  Manufacturer                  HP
  Product Name                  ProLiant ML350 G6
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
**************************Wed Sep 26 13:24:42.363 2012 (UTC - 4:00)**************************
Loading Dump File [C:\Users\John\_jcgriff2_\dbug\__Kernel__\092612-38345-01.dmp]
Windows 7 Kernel Version 7601 (Service Pack 1) MP (16 procs) Free x64
Built by: 7601.17835.amd64fre.win7sp1_gdr.120503-2030
System Uptime:7 days 0:05:22.279
BugCheck Code: BugCheck 3B, {c0000005, fffff9600007cfe0, fffff8800eb31c10, 0}
Probably caused by :win32k.sys ( win32k!xxxProcessEventMessage+1c0 )
BugCheck Info: SYSTEM_SERVICE_EXCEPTION (3b)
DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT_SERVER
BUGCHECK_STR:  0x3B
PROCESS_NAME: TP.Shell.XAF.W
FAILURE_BUCKET_ID: X64_0x3B_win32k!xxxProcessEventMessage+1c0
CPUID:        "Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz"
MaxSpeed:     2270
CurrentSpeed: 2266
  BIOS Version                  D22
  BIOS Release Date             05/05/2011
  Manufacturer                  HP
  Product Name                  ProLiant ML350 G6
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
**************************Wed Sep 19 13:15:09.041 2012 (UTC - 4:00)**************************
Loading Dump File [C:\Users\John\_jcgriff2_\dbug\__Kernel__\091912-54007-01.dmp]
Windows 7 Kernel Version 7601 (Service Pack 1) MP (16 procs) Free x64
Built by: 7601.17835.amd64fre.win7sp1_gdr.120503-2030
System Uptime:36 days 23:38:24.068
BugCheck Code: BugCheck 3B, {c0000005, fffff960000dcfe0, fffff8800c73ac10, 0}
Probably caused by :win32k.sys ( win32k!xxxProcessEventMessage+1c0 )
BugCheck Info: SYSTEM_SERVICE_EXCEPTION (3b)
DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT_SERVER
BUGCHECK_STR:  0x3B
PROCESS_NAME: TP.Shell.XAF.W
FAILURE_BUCKET_ID: X64_0x3B_win32k!xxxProcessEventMessage+1c0
CPUID:        "Intel(R) Xeon(R) CPU           E5520  @ 2.27GHz"
MaxSpeed:     2270
CurrentSpeed: 2266
  BIOS Version                  D22
  BIOS Release Date             05/05/2011
  Manufacturer                  HP
  Product Name                  ProLiant ML350 G6
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
**************************Sun Oct  7 00:22:32.513 2012 (UTC - 4:00)**************************

3rd Party Drivers:
The following is for information purposes only.
Any drivers in red should be updated or removed from your system. And should have been discussed in the body of my post.

Code:

**************************Sun Oct  7 00:22:32.513 2012 (UTC - 4:00)**************************
ati2mtag.sys                Wed Jun 24 23:33:44 2009 (4A42F018)
intelppm.sys                Mon Jul 13 19:19:25 2009 (4A5BC0FD)
amdxata.sys                 Fri Mar 19 12:18:18 2010 (4BA3A3CA)
HpSAMD.sys                  Tue Apr 20 14:32:18 2010 (4BCDF332)
timntr.sys                  Thu Jul 29 13:29:24 2010 (4C51BA74)
snapman.sys                 Wed Oct 27 07:36:16 2010 (4CC80EB0)
dfsrro.sys                  Sat Nov 20 04:26:37 2010 (4CE7944D)
passthruparser.sys          Sat Nov 20 04:57:28 2010 (4CE79B88)
vhdparser.sys               Sat Nov 20 04:57:30 2010 (4CE79B8A)
epfwwfpr.sys                Thu Dec  9 00:26:09 2010 (4D006871)
eamonm.sys                  Thu Dec  9 00:29:20 2010 (4D006930)
ehdrv.sys                   Thu Dec  9 00:29:58 2010 (4D006956)
hpqilo2.sys                 Thu Feb 17 18:16:32 2011 (4D5DAC50)
vmswitch.sys                Sat May 14 00:56:59 2011 (4DCE0B9B)
cpqcidrv.sys                Tue Sep 13 13:17:38 2011 (4E6F9032)
dump_HpCISSs2.sys           Wed Feb  1 16:24:29 2012 (4F29AD8D)
HpCISSs2.sys                Wed Feb  1 16:24:29 2012 (4F29AD8D)
b57nd60a.sys                Thu Feb  9 22:32:25 2012 (4F348FC9)
cpqteam.sys                 Wed Apr 25 11:32:31 2012 (4F98190F)
hvboot.sys                  Wed Aug 22 11:10:27 2012 (5034F663)
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
**************************Thu Oct  4 00:44:01.057 2012 (UTC - 4:00)**************************
cpqcidrv.sys                Mon May 11 18:08:05 2009 (4A08A1C5)
b57nd60a.sys                Fri May 21 01:48:07 2010 (4BF61E97)
dump_HpCISSs2.sys           Tue Aug 10 16:47:59 2010 (4C61BAFF)
HpCISSs2.sys                Tue Aug 10 16:47:59 2010 (4C61BAFF)
cpqteam.sys                 Wed Jan 26 19:17:27 2011 (4D40B997)
hvboot.sys                  Mon Mar 14 22:35:23 2011 (4D7ED06B)
SCDEmu.SYS                  Wed Jun 15 04:29:27 2011 (4DF86D67)

http://www.carrona.org/drivers/driver.php?id=ati2mtag.sys
http://www.carrona.org/drivers/driver.php?id=intelppm.sys
http://www.carrona.org/drivers/driver.php?id=amdxata.sys
http://www.carrona.org/drivers/driver.php?id=HpSAMD.sys
http://www.carrona.org/drivers/driver.php?id=timntr.sys
http://www.carrona.org/drivers/driver.php?id=snapman.sys
dfsrro.sys - this driver hasn't been added to the DRT as of this run. Please search Google/Bing for the driver if additional information is needed.
passthruparser.sys - this driver hasn't been added to the DRT as of this run. Please search Google/Bing for the driver if additional information is needed.
vhdparser.sys - this driver hasn't been added to the DRT as of this run. Please search Google/Bing for the driver if additional information is needed.
http://www.carrona.org/drivers/driver.php?id=epfwwfpr.sys
http://www.carrona.org/drivers/driver.php?id=eamonm.sys
http://www.carrona.org/drivers/driver.php?id=ehdrv.sys
hpqilo2.sys - this driver hasn't been added to the DRT as of this run. Please search Google/Bing for the driver if additional information is needed.
vmswitch.sys - this driver hasn't been added to the DRT as of this run. Please search Google/Bing for the driver if additional information is needed.
cpqcidrv.sys - this driver hasn't been added to the DRT as of this run. Please search Google/Bing for the driver if additional information is needed.
dump_HpCISSs2.sys - this driver hasn't been added to the DRT as of this run. Please search Google/Bing for the driver if additional information is needed.
HpCISSs2.sys - this driver hasn't been added to the DRT as of this run. Please search Google/Bing for the driver if additional information is needed.
http://www.carrona.org/drivers/driver.php?id=b57nd60a.sys
cpqteam.sys - this driver hasn't been added to the DRT as of this run. Please search Google/Bing for the driver if additional information is needed.
hvboot.sys - this driver hasn't been added to the DRT as of this run. Please search Google/Bing for the driver if additional information is needed.
cpqcidrv.sys - this driver hasn't been added to the DRT as of this run. Please search Google/Bing for the driver if additional information is needed.
http://www.carrona.org/drivers/driver.php?id=b57nd60a.sys
dump_HpCISSs2.sys - this driver hasn't been added to the DRT as of this run. Please search Google/Bing for the driver if additional information is needed.
HpCISSs2.sys - this driver hasn't been added to the DRT as of this run. Please search Google/Bing for the driver if additional information is needed.
cpqteam.sys - this driver hasn't been added to the DRT as of this run. Please search Google/Bing for the driver if additional information is needed.
hvboot.sys - this driver hasn't been added to the DRT as of this run. Please search Google/Bing for the driver if additional information is needed.
http://www.carrona.org/drivers/driver.php?id=SCDEmu.SYS

suvaldykit · Oct 16, 2012

Acronis is a backup solution doesnt really want to remove it, i have disabled all services of it.

Removed network card which was disabled.

Newer ATI driver is not available.

I have ran memtest last night didnt finnished,one night wasnt enough. So will have to repeat this over wewkend.

I will do driver verifier as well on saturday.

usasma · Oct 16, 2012

There's a number of KB articles listed at this link that may apply to your situation: http://www.carrona.org/bsodindx.html#0x0000003B
I'm not that familiar with Server OS's (and the terminology).

Try uninstalling the video drivers and then reinstalling a freshly downloaded copy.

I don't know if this is applicable, but I found this description:

Solution
No, the onboard ATI ES1000 does not support DirectX and ATI ES1000 VGA is 2D Graphic Chip. Therefore, it cannot support Direct3D or DirectDraw function.

From this link: http://support.asus.com/Search/KDetail.aspx?SLanguage=en&no=AB5E31C1-F18D-A325-099F-C3E12269CA8D&t=2

Good luck!

If it BSOD's again, please zip up and upload the memory dumps.

Vir Gnarus · Oct 17, 2012

All the crashdumps show completely identical cases involving this special software you're referring too (TP.Shell). Yes, I would think it has something to do with it. I would definitely consider working with the vendor on this one. Provide them information like the kernel dump (MEMORY.DMP in Windows directory) if necessary.

All the crashes involve processing a message in the message queue for an application window. Messages are basically methods for which the system will inform an application of input being sent to the application and provide it that input (anything from clicking a button on the application window to typing something in a form). Apparently there's some mishandling of the message queue, which in effect will conk out win32k.sys, which is required to maintain system stability. I can't figure things out further without a kernel dump. A full dump may even be necessary in this case.

Again, you'll wanna check with the vendor on this. It is far too consistent here to be dealing with something like some bad dereferenced pointer or some other floating bug. Granted, it can still be caused by some sort of conflict, but one of the elements of that conflict is indeed this TP.Shell, so it's best to check on that first.

suvaldykit · Oct 17, 2012

Vir Gnarus said:
All the crashdumps show completely identical cases involving this special software you're referring too (TP.Shell). Yes, I would think it has something to do with it. I would definitely consider working with the vendor on this one. Provide them information like the kernel dump (MEMORY.DMP in Windows directory) if necessary.

All the crashes involve processing a message in the message queue for an application window. Messages are basically methods for which the system will inform an application of input being sent to the application and provide it that input (anything from clicking a button on the application window to typing something in a form). Apparently there's some mishandling of the message queue, which in effect will conk out win32k.sys, which is required to maintain system stability. I can't figure things out further without a kernel dump. A full dump may even be necessary in this case.

Again, you'll wanna check with the vendor on this. It is far too consistent here to be dealing with something like some bad dereferenced pointer or some other floating bug. Granted, it can still be caused by some sort of conflict, but one of the elements of that conflict is indeed this TP.Shell, so it's best to check on that first.

I will ask vendor guy to join the chat, as he is more technical about its software.

suvaldykit · Oct 17, 2012

THese two actually related to our inviroment but i am not sure should i try installing them, would that may cause more problems than resolutions?

http://support.microsoft.com/kb/2359223/en-US
http://support.microsoft.com/kb/2260182

TP Shell software is runing throug APP-V so second one is most likely applicable but i am not keen to install it if it can make more mess than help what you think?

Vir Gnarus · Oct 17, 2012

Those both look very relevant, with the first describing very much what I hypothesized is happening. You should approach these hotfixes as an elevated update (maybe critical) for the server as defined in your company's SLA policy. If you don't have an SLA developed (you should!), then approach it with the usual care of any critical update for a server: backup system, create snapshot if available, form a recovery plan, inform customers of scheduled downtime, allot a time for this in your maintenance schedule, etc. etc. Like dealing with any production environment, approach this methodically and with care, as it's not just a PC sitting at home.

I'd start with the first patch listed, test, evaluate, then if it does not resolve the issue, rollback and then apply the other patch, test, evaluate. All of that should be done in the timespan of the scheduled maintenance you originally planned for this. If you cannot manually recreate the crashes, then apply both patches, but still test and evaluate the server, application and service's stability afterwards for any potential side effects.

suvaldykit · Oct 17, 2012

usasma,

I dont like this bit in verifier:
If you can't get into Safe Mode, try using System Restore from your installation DVD to set the system back to the previous restore point that you created.

Its production server, so i would like to have it runing after testing not trying to recover :), i just read about metal restore so i will make backup and maybe try this verifier thing this week.
Thank you

Vir Gnarus · Oct 17, 2012

Yes, I would not recommend using Driver Verifier unless you plan on an extended maintenance period for the server. However, to alleviate your concerns some, Driver Verifier is never active during Safe Mode. If you cannot get into Safe Mode after Driver Verifier has been turned on, it most likely means a hardware issue that has been exacerbated by Driver Verifier's more intense driver scrutiny and has ended up showing up symptoms in Safe Mode as well. In any other case you can confidently enter Safe Mode and turn off DV from there. But again, DV is designed to make a system more unstable by adding extra driver checks that will BSOD the system when a drive fails to adhere to one. Unless the time is appropriate and everything has been sufficiently prepared, it should not be ran on a production environment.

martys · Oct 18, 2012

Vir Gnarus said:
Again, you'll wanna check with the vendor on this. It is far too consistent here to be dealing with something like some bad dereferenced pointer or some other floating bug. Granted, it can still be caused by some sort of conflict, but one of the elements of that conflict is indeed this TP.Shell, so it's best to check on that first.

Hi, from the Software vendor :)
Speaking about the application "Tp.Shell", it is a .Net framework C# application, which is kinda memory hungry.

My suspicion is that it's some kind of the memory problem, or the .Net Framework error.
First because crash dump says, it cannot access some page in memory, and second, that the application is a managed code application. Meaning that .Net Framework takes care of managing resources for it ect.. Maybe if the app is running for a long time, and the garbage collector kicks in and cleans the memory, then smth happnes.. Anyways - It's a WinForms application, which uses 3-rd party compnents from DevExpress (who I contacted to make sure that there are not "unmanged" libs used in the process).

Vir Gnarus · Oct 18, 2012

Hi martys,

Thanks for the assistance and correspondence. As for the failed to access a memory page, The faulting instruction itself appears to have failed due to a null pointer reference when attempting to process a message from the message queue (ProcessEventMessage). What generated the null pointer, though, appears to be beyond the scope of these minidumps. The same can also be said of determining if we're suffering from a lack of memory, as that information is unavailable to us here.

I'd be happy to try and debug this further but it'd probably be moot given you already have source code and full symbols. I would like to at least look at the kernel dump though, but that can be a security risk on a production system. Regardless, it appears it may be in your hands now. We'll do what we can from this end, so if an assistance is needed you can contact us about it. Though personally, I despise .NET debugging!

Keep us up-to-date on this if you'd like. I'd be happy to know what would become of it!

martys · Oct 18, 2012

I'd be happy to debug it in Visual Studio, If I knew How to reproduce the problem locally, or at which point in the program the error happens..

Vir Gnarus · Oct 18, 2012

Unless you can work with the client to create a breakpoint (perhaps conditional?) of some sort, by having them setup a xperf report, or by having them install and setup Application Verifier, the only other way I see in getting the necessary info here is with a complete dump file, as you'll need access to userland memory. Obviously a full dump from a server environment is not going to be pretty. :|

If you still have concerns this may be due to memory exhaustion, we can also look at a kernel dump to see if that's the case. It might not point you directly to the code responsible, but it'll at least point blame to which module(s) are involved and what they were doing.

suvaldykit · Oct 24, 2012

Vir Gnarus said:
Those both look very relevant, with the first describing very much what I hypothesized is happening. You should approach these hotfixes as an elevated update (maybe critical) for the server as defined in your company's SLA policy. If you don't have an SLA developed (you should!), then approach it with the usual care of any critical update for a server: backup system, create snapshot if available, form a recovery plan, inform customers of scheduled downtime, allot a time for this in your maintenance schedule, etc. etc. Like dealing with any production environment, approach this methodically and with care, as it's not just a PC sitting at home.

I'd start with the first patch listed, test, evaluate, then if it does not resolve the issue, rollback and then apply the other patch, test, evaluate. All of that should be done in the timespan of the scheduled maintenance you originally planned for this. If you cannot manually recreate the crashes, then apply both patches, but still test and evaluate the server, application and service's stability afterwards for any potential side effects.

None of these hotfixes are applicable, not sure why.. Tryed to install both, got message that this update is not applicable.. Confused

Vir Gnarus · Oct 24, 2012

Then either a previous Windows update incorporated them or they simply don't fit your current OS, which is kind of a bummer. Have you still been corresponding with martys on this?

martys · Nov 6, 2012

Well, speaking of a Null pointers, I would agree with You it the app was C++ or similar.
To my understanding .Net is by definition, a "managed runtime" language, and by it's definitions, means that .Net Frameworks is managing all the resources needed (memory in this case).

My guess is that - it may be so that some part of .Net frameworks got corrupted ? (first one was a memory leak, but we weren't able to complete a Full Memtest yet.)

P.S. is there any way we could examine the dump via "remote"/coference/skype/share'd screen together?

Windows Server 2008 R2 STD BSOD crash reason

Member

Senior Administrator, Windows Update Expert

Retired Admin

Member

Retired Admin

Member

Retired Admin

BSOD Kernel Dump Expert

Member

Member

BSOD Kernel Dump Expert

Member

BSOD Kernel Dump Expert

New member

BSOD Kernel Dump Expert

New member

BSOD Kernel Dump Expert

Member

BSOD Kernel Dump Expert

New member