Vista notebook, 0x9F, "Too many Irp stacks to be believed (>30)!!"

satrow · Jun 11, 2013

I had hands on a Vista SP2 notebook a little earlier, any clues to be gleaned from this single dump? Excuse me if I don't reply until tomorrow, I have a sudden urge to try to sleep off the Vista experience :sleep2:

View attachment Mini061113-01.zip

Code:

Microsoft (R) Windows Debugger Version 6.11.0001.404 X86
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [E:\Mini061113-01.dmp]


Mini Kernel Dump File: Only registers and stack trace are available

Symbol search path is: G:\symbols;srv*c:\Symbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows Server 2008/Windows Vista Kernel Version 6002 (Service Pack 2) MP (2 procs) Free x86 compatible
Product: WinNt, suite: TerminalServer SingleUserTS Personal
Built by: 6002.18805.x86fre.vistasp2_gdr.130308-1436
Machine Name:
Kernel base = 0x81e47000 PsLoadedModuleList = 0x81f5ec70
Debug session time: Tue Jun 11 11:51:50.786 2013 (GMT+1)
System Uptime: 4 days 22:55:16.408
Loading Kernel Symbols
...............................................................
................................................................
.......................
Loading User Symbols
Loading unloaded module list
....
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 9F, {4, 258, 83fa2580, 0}

Probably caused by : ntkrpamp.exe ( nt!PopBuildDeviceNotifyListWatchdog+34 )

Followup: MachineOwner
---------

0: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

DRIVER_POWER_STATE_FAILURE (9f)
A driver is causing an inconsistent power state.
Arguments:
Arg1: 00000004, The power transition timed out waiting to synchronize with the Pnp
	subsystem.
Arg2: 00000258, Timeout in seconds.
Arg3: 83fa2580, The thread currently holding on to the Pnp lock.
Arg4: 00000000

Debugging Details:
------------------


DRVPOWERSTATE_SUBCODE:  4

FAULTING_THREAD:  83fa2580

CUSTOMER_CRASH_COUNT:  1

DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT

BUGCHECK_STR:  0x9F

PROCESS_NAME:  System

CURRENT_IRQL:  2

LAST_CONTROL_TRANSFER:  from 82121096 to 81f14af7

STACK_TEXT:  
81f3cb4c 82121096 0000009f 00000004 00000258 nt!KeBugCheckEx+0x1e
81f3cb68 81ef12eb 8836fa78 8836fa68 ad861620 nt!PopBuildDeviceNotifyListWatchdog+0x34
81f3cc88 81ef0f21 81f3ccd0 81ef1f02 81f3ccd8 nt!KiTimerListExpire+0x367
81f3cce8 81ef1615 00000000 00000000 01a2bfef nt!KiTimerExpiration+0x2a0
81f3cd50 81eef87d 00000000 0000000e 00000000 nt!KiRetireDpcList+0xba
81f3cd54 00000000 0000000e 00000000 00000000 nt!KiIdleLoop+0x49


STACK_COMMAND:  .thread 0xffffffff83fa2580 ; kb

FOLLOWUP_IP: 
nt!PopBuildDeviceNotifyListWatchdog+34
82121096 cc              int     3

SYMBOL_STACK_INDEX:  1

SYMBOL_NAME:  nt!PopBuildDeviceNotifyListWatchdog+34

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: nt

IMAGE_NAME:  ntkrpamp.exe

DEBUG_FLR_IMAGE_TIMESTAMP:  513a901f

FAILURE_BUCKET_ID:  0x9F_nt!PopBuildDeviceNotifyListWatchdog+34

BUCKET_ID:  0x9F_nt!PopBuildDeviceNotifyListWatchdog+34

Followup: MachineOwner
---------

0: kd> !irp 83fa2580
Irp is active with 177 stacks 38 is current (= 00000000)
 No Mdl: Master Irp=83fa2588: Thread 00010001:  Too many Irp stacks to be believed (>30)!!

mgrzeg · Jun 11, 2013

Parameter 1 = 0x4, and Parameter 3 is a thread address, not an irp :)

Code:

0: kd> !thread 83fa2580
GetPointerFromAddress: unable to read from 81f7e878
THREAD 83fa2580  Cid 0000.0000  Teb: 00000000 Win32Thread: 00000000 RUNNING on processor 0
Not impersonating
GetUlongFromAddress: unable to read from 81f56394
Owning Process            81f43900       Image:         <Unknown>
Attached Process          83f59a38       Image:         System
ffdf0000: Unable to get shared data
Wait Start TickCount      0            
Context Switch Count      64772990       IdealProcessor: 0             
ReadMemory error: Cannot get nt!KeMaximumIncrement value.
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address nt!KiIdleLoop (0x81eef834)
Stack Init 81f3d000 Current 81f3cd4c Base 81f3d000 Limit 81f3a000 Call 0
Priority 0 BasePriority 0 PriorityDecrement 0 IoPriority 0 PagePriority 0
ChildEBP RetAddr  Args to Child              
81f3cb4c 82121096 0000009f 00000004 00000258 nt!KeBugCheckEx+0x1e
81f3cb68 81ef12eb 8836fa78 8836fa68 ad861620 nt!PopBuildDeviceNotifyListWatchdog+0x34
81f3cc88 81ef0f21 81f3ccd0 81ef1f02 81f3ccd8 nt!KiTimerListExpire+0x367
81f3cce8 81ef1615 00000000 00000000 01a2bfef nt!KiTimerExpiration+0x2a0
81f3cd50 81eef87d 00000000 0000000e 00000000 nt!KiRetireDpcList+0xba
81f3cd54 00000000 0000000e 00000000 00000000 nt!KiIdleLoop+0x49 (FPO: [0,0,0])

m.g.

satrow · Jun 11, 2013

Ah, I knew I was tired!

Thanks m.g.

So am I right in thinking the thread leads to a dead-end and I fall back to updating drivers, disabling power saving for the network components, removing USB hardware etc?

Bug Check 0x9F: DRIVER_POWER_STATE_FAILURE

P1 0x4 (Windows Vista and later)
P2 Time-out value, in seconds.
P3 The thread currently holding onto the Plug-and-Play (PnP) lock.
P4 Reserved

The power state transition timed out waiting to synchronize with the PnP subsystem.

There is a suspicion of malware, the System logs after the restart showed an entry from MSE to the effect that "infected files and logs have been cleared" or similar. I'll need to rule that out as well.

Tekno Venus · Jun 11, 2013

A USB issue seems to be the cause of this crash IMO. . .

Code:

81f3cb68  81f3cc88 nt!KiDoubleFaultStack+0x2c88
81f3cb6c  81ef12eb nt!KiTimerListExpire+0x367
81f3cb70  8836fa78
81f3cb74  8836fa68
81f3cb78  ad861620
81f3cb7c  01ce6691
81f3cb80  81f45a70 nt!KiTimerTableListHead+0x1ef0
81f3cb84  00001ef0
81f3cb88  81f3fe38 nt!KiInitialPCR+0x638
81f3cb8c  00000006
81f3cb90  8836fa78
81f3cb94  82121062 nt!PopBuildDeviceNotifyListWatchdog
81f3cb98  8836fa68
81f3cb9c  8593fc3c
81f3cba0  90609034 usbhub!UsbhDmTimerDpc
81f3cba4  8593f028
81f3cba8  858c8c3c
81f3cbac  90609034 usbhub!UsbhDmTimerDpc
81f3cbb0  858c8028
81f3cbb4  85374828
81f3cbb8  807b3864 USBPORT!USBPORT_DM_TimerDpc
81f3cbbc  85374028
81f3cbc0  00000002
81f3cbc4  81f3cbe8 nt!KiDoubleFaultStack+0x2be8
81f3cbc8  853b00e0
81f3cbcc  00000002
81f3cbd0  853b0d02
81f3cbd4  81f3cbe4 nt!KiDoubleFaultStack+0x2be4
81f3cbd8  81e1b70c hal!KfLowerIrql+0x64
81f3cbdc  00000001
81f3cbe0  853b0d02
81f3cbe4  81f3cbf0 nt!KiDoubleFaultStack+0x2bf0
81f3cbe8  81e17f6b hal!KfReleaseSpinLock+0xb
81f3cbec  81e1d97f hal!HalpQueryHpetCount+0x4b
81f3cbf0  ffd070f0
81f3cbf4  00000020
81f3cbf8  87591cc8
81f3cbfc  00009968
81f3cc00  a3b46a6b
81f3cc04  000005b7
81f3cc08  81f3cc10 nt!KiDoubleFaultStack+0x2c10
81f3cc0c  81e1ddc1 hal!HalpHpetQueryPerformanceCounter+0x1d
81f3cc10  81f3cc48 nt!KiDoubleFaultStack+0x2c48
81f3cc14  81e8bea5 nt!EtwpGetPerfCounter+0x8
81f3cc18  00000000
81f3cc1c  81ef2530 nt!EtwpReserveTraceBuffer+0xce
81f3cc20  0000001c
81f3cc24  0000000c
81f3cc28  81e8e702 nt!ExfInterlockedPushEntryList+0x52
81f3cc2c  84181000
81f3cc30  00010000
81f3cc34  00009968
81f3cc38  84181008
81f3cc3c  81e1d97f hal!HalpQueryHpetCount+0x4b
81f3cc40  ffd070f0
81f3cc44  00000160
81f3cc48  87591cc8
81f3cc4c  c943c91d
81f3cc50  00000000
81f3cc54  00000000
81f3cc58  598298d5
81f3cc5c  000005bd
81f3cc60  000003e4
81f3cc64  81f3cc74 nt!KiDoubleFaultStack+0x2c74
81f3cc68  81ef46b9 nt!KeUpdateSystemTime+0x129
81f3cc6c  ffffff02
81f3cc70  000000d1
81f3cc74  81f3f920 nt!KiInitialPCR+0x120
81f3cc78  81f3f920 nt!KiInitialPCR+0x120
81f3cc7c  00000001
81f3cc80  00000001
81f3cc84  00000000
81f3cc88  81f3cce8 nt!KiDoubleFaultStack+0x2ce8
81f3cc8c  81ef0f21 nt!KiTimerExpiration+0x2a0
81f3cc90  81f3ccd0 nt!KiDoubleFaultStack+0x2cd0
81f3cc94  81ef1f02 nt!KiDeferredReadyThread+0x6ae
81f3cc98  81f3ccd8 nt!KiDoubleFaultStack+0x2cd8
81f3cc9c  00000010
81f3cca0  81f3f920 nt!KiInitialPCR+0x120
81f3cca4  81f41300 nt!KiInitialPCR+0x1b00
81f3cca8  00000000
81f3ccac  00000002
81f3ccb0  00000010
81f3ccb4  00001ef0
81f3ccb8  853ca500
81f3ccbc  81ef1f02 nt!KiDeferredReadyThread+0x6ae
81f3ccc0  01a2bfef
81f3ccc4  00000518
81f3ccc8  01a2bfef
81f3cccc  81f3f920 nt!KiInitialPCR+0x120
81f3ccd0  81f3ccd0 nt!KiDoubleFaultStack+0x2cd0
81f3ccd4  81f3ccd0 nt!KiDoubleFaultStack+0x2cd0
81f3ccd8  ad861620
81f3ccdc  01ce6691
81f3cce0  c943c91d
81f3cce4  000003e4
81f3cce8  81f3cd50 nt!KiDoubleFaultStack+0x2d50
81f3ccec  81ef1615 nt!KiRetireDpcList+0xba
81f3ccf0  00000000
81f3ccf4  00000000
81f3ccf8  01a2bfef
81f3ccfc  00000000
81f3cd00  81f43640 nt!KiInitialThread
81f3cd04  00000000
81f3cd08  81f3f800 nt!KiInitialPCR

Have you tried https://www.sysnative.com/forums/ha...driver-reset-usb-ports-power-state-reset.html

cluberti · Jun 11, 2013

Yeah, I'd wager it's a driver loaded in the UMDF stack somewhere, and could be a USB device given the stack. Unfortunately, a minidump has given you about all it has to give. Also, given it's probably a user-mode driver, figuring it out would require a full dump, not a kernel-only as well.

jcgriff2 · Jun 11, 2013

Any idea what this unloaded driver is?

Code:

Unloaded modules:
9eb1f000 9eb25000   MpKsldc63e5a
    Timestamp: unavailable (00000000)
    Checksum:  00000000
    ImageSize:  00006000

Or these loaded drivers? They show up 2x in the sole dump; different timestamps -

Code:

77504612.sys  Thu Oct 22 05:53:55 2009 (4AE02BB3)
50587522.sys  Thu Oct 22 05:53:55 2009 (4AE02BB3)

77504611.sys  Fri Sep 25 09:59:00 2009 (4ABCCCA4)
50587521.sys  Fri Sep 25 09:59:00 2009 (4ABCCCA4)

Also, your SAS is 2010 version -

Code:

SASKUTIL.SYS  Mon May 10 13:15:22 2010 (4BE83F2A)
SASDIFSV.SYS  Wed Feb 17 13:19:19 2010 (4B7C3327)

http://www.sysnative.com/drivers/driver.php?id=SASKUTIL.sys

I can't say that any of the above is causing the BSODs; just curious.

EDIT:

Code:

FAILURE_BUCKET_ID:  0x9F_4_nt!PopBuildDeviceNotifyListWatchdog+34

I know 'watchdog' can be other then video, but I noticed ATI video drivers are O-L-D --

Code:

atikmdag.sys  Fri Jul 27 23:36:35 2007 (46AAB9C3)

satrow · Jun 11, 2013

Thanks for looking in and helping with this one guys :thumbsup2: I think I'm awake now.

Some background, the owner is something of a technophobe and popups about updating anything seem to send him into something of a panic, I'd tried to keep the notebook simple as he would normally only visit a very small number of regular sites each day. He's recently become a convert of Skype ... which I think was the conduit for this problem.

It's looking like an infection behind all this now; JCGriff2 picked up on the odd drivers, the MpKs* driver is likely to be MSE - could the USB link mean that malware tried to take over the webcam?

As to other possible USB device involvement, there's a wireless mouse, an SD card reader that's rarely, if ever, used and a possibility that the internal wireless card is connected internally via USB.

I should be able to gain access to the machine later today for an hour or so, I'll concentrate on the malware side of things then, if I can keep it longer, I may be able to fully update drivers, etc. as well. Then I need to figure out a simple, regular maintenance procedure for him to follow.

I hope Corrine will be around in ~ 10 hours time, I may need her expertise :rose:

Corrine · Jun 16, 2013

I've been standing by but gather you haven't needed me.

The MSE driver is MPFilter.sys not MpKs*. SystemLookup - MpFilter

tom982 · Jun 16, 2013

Code:

77504612.sys  Thu Oct 22 05:53:55 2009 (4AE02BB3)
50587522.sys  Thu Oct 22 05:53:55 2009 (4AE02BB3)

77504611.sys  Fri Sep 25 09:59:00 2009 (4ABCCCA4)
50587521.sys  Fri Sep 25 09:59:00 2009 (4ABCCCA4)

I might be wrong but I think these are part of the Kaspersky Virus Removal Tool: Latest Versions | Virus Removal Tool | Kaspersky Lab

They're actually in the DRT :)

Driver Reference Table (DRT)

Second and third entries.

satrow · Jun 16, 2013

Sorry folks, time's not my own these days, likely to be worse over the next week to 10 days.

MpKsl*.sys: Multiple files starting with MpKsl keep trying to run at startup - - Microsoft Community (I recall chasing this around for hours on a different notebook about 2.5 years ago!).

Yes, Tom the 77504612.sys and 50587522.sys were Kaspersky verified.

Updated the BIOS and some of the drivers and software, the latest ATI drivers for it were 2008; sorry Stephen, I forgot to try the USB reset.

Corrine, I failed to get DDS to run, everything else I tried ran fine and came up clean as far as I could tell; I forgot to pull the logs over onto my thumb drive

Several Skype and MSE updating errors in the logs, and signs of a CPU problem on both cores.

Attached the BSOD app outputs x2 (2nd folder inside the first) along with the analysis *txts; I got it to BSOD a 2nd time

looks like the graphics were to blame.

x BlueRobot · Jun 17, 2013

It does seem that the graphics card driver is causing the issue here, and it is a very old driver too.

Code:

0: kd> !irp 86c8e5e8
Irp is active with 4 stacks 2 is current (= 0x86c8e67c)
 No Mdl: No System Buffer: Thread 00000000:  Irp stack trace.  
     cmd  flg cl Device   File     Completion-Context
 [  0, 0]   0  0 00000000 00000000 00000000-00000000    

			Args: 00000000 00000000 00000000 00000000
>[ 16, 2]   0 e1 87671c70 00000000 00000000-00000000    pending
	       \Driver\atikmdag
			Args: 00000000 00000001 00000001 00000000
 [ 16, 2]   0 e1 85be71a0 00000000 81e1a2d2-86af2230 Success Error Cancel pending
	       \Driver\monitor	nt!PopRequestCompletion
			Args: 00000000 00000001 00000001 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-86af2230    

			Args: 00000000 00000000 00000000 00000000

Code:

0: kd> lmvm atikmdag
start    end        module name
8bc06000 8c119000   atikmdag T (no symbols)           
    Loaded symbol image file: atikmdag.sys
    Image path: atikmdag.sys
    Image name: atikmdag.sys
    Timestamp:        Tue Jun 03 04:48:31 2008 (4844BF0F)
    CheckSum:         00394B68
    ImageSize:        00513000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

Search

Search

Vista notebook, 0x9F, "Too many Irp stacks to be believed (>30)!!"

satrow

Moderator, BSOD Kernel Dump Senior Analyst

mgrzeg

BSOD Kernel Dump Senior Analyst

satrow

Moderator, BSOD Kernel Dump Senior Analyst

Tekno Venus

Senior Administrator, Developer

cluberti

Senior Member

jcgriff2

Co-Founder/Admin
BSOD Expert (Ret.)
Microsoft MVP (Ret.)
PRIMARILY RETIRED

satrow

Moderator, BSOD Kernel Dump Senior Analyst

Corrine

Administrator,
Microsoft MVP,
Security Analyst

tom982

Emeritus

satrow

Moderator, BSOD Kernel Dump Senior Analyst

x BlueRobot

Administrator

Vista notebook, 0x9F, "Too many Irp stacks to be believed (>30)!!"

Moderator, BSOD Kernel Dump Senior Analyst

BSOD Kernel Dump Senior Analyst

Moderator, BSOD Kernel Dump Senior Analyst

Senior Administrator, Developer

Senior Member

Co-Founder/AdminBSOD Expert (Ret.)Microsoft MVP (Ret.)PRIMARILY RETIRED

Moderator, BSOD Kernel Dump Senior Analyst

Administrator, Microsoft MVP, Security Analyst

Emeritus

Moderator, BSOD Kernel Dump Senior Analyst

Administrator

Co-Founder/Admin
BSOD Expert (Ret.)
Microsoft MVP (Ret.)
PRIMARILY RETIRED

Administrator,
Microsoft MVP,
Security Analyst