Hello analysts & debuggers! I am going to be creating a # of information threads in this here forum as time goes on. I blog about quite a bit enough, so I figured why not port most of it over to Sysnative, and go more in-depth, etc, if possible.
Right, so we have a thread (
here) for 0x9F's having the 1st parameter = 0x3. In its simplest definition, this indicates a/the device object has been blocking an IRP for too long a time. It's important to note that in terms of debugging & analyzing BSOD's on online communities, and essentially the internet in general, you will really only run into two types of the 0x9F bug check.
1. 1st parameter = 0x3, which implies a device object has been blocking an IRP for too long a time.
-- In the case of #1, in most if not all cases, dumping with
!analyze -v doesn't provide us with the driver that actually caused the hold up. For example, you may see Windows system drivers such as
usbhub.sys, etc, being labeled as the fault. This is not correct, and you have to go ahead and then run
!irp 4th parameter address here.
Let's say the 4th parameter =
1234567891234567, you'd run
!irp 1234567891234567.
That's the bare basics of it, and
usually all it takes when the 1st parameter = 0x3. I will not go further in-depth as far as 0x3 goes, because there's already quite a bit of that discussed in the thread I linked above.
-- 0x3 for 1st parameter is different on Windows 2000, which instead implies a/the device driver did not correctly set the IRP as "pending" or complete the IRP. I listed this for knowledge related purposes, and because you never know when you'll be stuck on an island with a computer, and the OS is Windows 2k and you need to fix its 0x9F problem first : )
2. 1st parameter = 0x4, which implies that a power IRP has failed to synchronize with the PnP Manager. This is what we're
really interested in here.
As with most bug checks, we have 4 parameters, whereas only 3 of the 4 are useful (to my knowledge):
1st parameter = 0x4 (see above).
2nd parameter = This is where the time-out value is displayed, specifically in seconds.
-- In regards to the 2nd parameter, something interesting I've noticed is the fact that the 2nd parameter is always
258 seconds, at least every 0x9F-0x4 I've seen thus far. If this is truly the case, my guess is the system is set to automatically bugcheck if the time-out reaches 258 seconds. I'll need to do more research on this.
3rd parameter = The address of the thread currently holding onto the Plug-and-Play (PnP) lock.
4th parameter = ???? AFAIK, this is a
RESERVED address. Not relevant to us, to my knowledge.
With all of that explained and out of the way, let's get to some debugging! I'm using a random 0x9F I've solved as the example. The full thread can be found
here.
Code:
0: kd> .bugcheck
Bugcheck code 0000009F
Arguments 00000000`00000004 00000000`00000258 fffffa80`070b5b50 fffff800`00b9c3d0
Here we can see:
1st = 0x4
2nd = 258 (timeout in seconds)
Code:
0: kd> knL
# Child-SP RetAddr Call Site
00 fffff800`00b9c498 fffff800`032feeb6 nt!KeBugCheckEx
01 fffff800`00b9c4a0 fffff800`034ab2fc nt!PnpBugcheckPowerTimeout+0x76
02 fffff800`00b9c500 fffff800`03278ac6 nt!PopBuildDeviceNotifyListWatchdog+0x1c
03 fffff800`00b9c530 fffff800`03277dc6 nt!KiProcessTimerDpcTable+0x66
04 fffff800`00b9c5a0 fffff800`0327899e nt!KiProcessExpiredTimerList+0xc6
05 fffff800`00b9cbf0 fffff800`03278137 nt!KiTimerExpiration+0x1be
06 fffff800`00b9cc90 fffff800`0327519a nt!KiRetireDpcList+0x277
07 fffff800`00b9cd40 00000000`00000000 nt!KiIdleLoop+0x5a
Not really too much info, just a few PnP routines that call into the bugcheck eventually. No driver in the stack, etc. So, we have no informative stack, and we have no 4th parameter IRP address to run !irp on. What now? Well, let's learn about
!locks.
Essentially,
!locks dumps information about kernel ERESOURCE locks.
I won't go into detail about ERESOURCES because it's pretty hefty, but here's a fantastic article -
Introduction to ERESOURCE Routines. You'll want to understand the basics of ERESOURCE, at least I recommend it.
Let's run !locks to see what's going on:
Code:
0: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks..
Resource @ nt!IopDeviceTreeLock (0xfffff80002cd2ce0) Shared 1 owning threads <--- This is the ERESOURCE.
Threads: fffffa800357c040-01<*> <--- We only have one thread here, which appears to be the owner.
KD: Scanning for held locks.
Resource @ nt!PiEngineLock (0xfffff80002cd2be0) Exclusively owned <--- Again, this is the ERESOURCE itself.
Contention Count = 12 <--- The amount of contention for the specific object.
NumberOfExclusiveWaiters = 1 <--- Self-explanatory, it's talking about the above 'nt!IopDeviceTreeLock'.
Threads: fffffa800357c040-01<*> <-- Again, only one owner thread here.
Threads Waiting On Exclusive Access:
fffffa800357fb50 <--- This is a thread that's waiting on exlusive access.
Rather than explaining all of the above, I thought outlining it like that would be easier. I have colored the 'Threads' addresses in red because it's important to note that both of those resource thread addresses match our bugcheck's 3rd parameter.
Great, so now that we have a thread address, let's take a look:
Code:
0: kd> !thread fffffa800357c040
THREAD fffffa800357c040 Cid 0004.0040 Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Non-Alertable
fffff880031c43a0 NotificationEvent
IRP List:
fffffa8005a06bb0: (0006,01f0) Flags: 00000000 Mdl: 00000000
Not impersonating
DeviceMap fffff8a000008ca0
Owning Process fffffa8003566040 Image: System
Attached Process N/A Image: N/A
Wait Start TickCount 584115 Ticks: 38461 (0:00:09:59.995)
Context Switch Count 25688 IdealProcessor: 1 NoStackSwap
UserTime 00:00:00.000
KernelTime 00:00:02.106
Win32 Start Address nt!ExpWorkerThread (0xfffff80002ad8150)
Stack Init fffff880031c4db0 Current fffff880031c4090
Base fffff880031c5000 Limit fffff880031bf000 Call 0
Priority 15 BasePriority 12 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP RetAddr : Args to Child : Call Site
fffff880`031c40d0 fffff800`02ac45f2 : fffffa80`0357c040 fffffa80`0357c040 00000000`00000000 00000000`0000000c : nt!KiSwapContext+0x7a
fffff880`031c4210 fffff800`02ad599f : 00000000`00000000 fffff880`01500992 00000000`00000000 fffff880`00000000 : nt!KiCommitThreadWait+0x1d2
fffff880`031c42a0 fffff880`01531d83 : ffffffff`fffe7900 fffffa80`00000000 00000000`00000000 fffffa80`03797800 : nt!KeWaitForSingleObject+0x19f <--- Current thread is put into a wait-state.
fffff880`031c4340 fffff880`0150f0af : fffffa80`037978c0 fffff880`01558110 fffffa80`037978c0 00000000`00000001 : ndis!ndisPauseFilter+0x203
fffff880`031c43d0 fffff880`015ae8ef : fffffa80`036b7cd0 00000000`00000000 00000000`00000008 fffffa80`0600ff00 : ndis! ?? ::FNODOBFM::`string'+0x30ed
fffff880`031c4400 fffff880`015afc2b : 00000000`00000001 fffffa80`036b01a0 fffffa80`036b7cd0 fffffa80`036b0050 : ndis!ndisCloseMiniportBindings+0x11f
fffff880`031c4510 fffff880`01546e6a : fffffa80`036b01a0 fffffa80`036b01a0 fffffa80`05a06bb0 fffff880`014f4300 : ndis!ndisPnPRemoveDevice+0x25b
fffff880`031c46b0 fffff880`015a35b2 : 00000000`00000000 fffffa80`036b01a0 fffffa80`05a06bb0 00000000`00000002 : ndis!ndisPnPRemoveDeviceEx+0xca
fffff880`031c4710 fffff800`02d3b121 : fffff8a0`09146650 fffffa80`036b0050 00000000`c00000bb 00000000`00000000 : ndis! ?? ::LNCPHCLB::`string'+0x7363
fffff880`031c47b0 fffff800`02ebb3a1 : fffffa80`03694580 00000000`00000000 fffffa80`0376c900 00000000`00000801 : nt!IopSynchronousCall+0xe1
fffff880`031c4820 fffff800`02bd1063 : fffff8a0`0ac87d80 fffff8a0`0ac87d80 00000000`00000018 00000000`00000000 : nt!IopRemoveDevice+0x101
fffff880`031c48e0 fffff800`02ebaef4 : fffffa80`0376c900 00000000`00000000 00000000`00000002 00000000`00000000 : nt!PnpRemoveLockedDeviceNode+0x1a3
fffff880`031c4930 fffff800`02ebb000 : 00000000`00000000 fffffa80`03694500 fffff8a0`09e5d9e0 fffff800`02cd2ae0 : nt!PnpDeleteLockedDeviceNode+0x44
fffff880`031c4960 fffff800`02ebb0f9 : fffffa80`04cc2c02 fffffa80`04cc2c70 00000000`00000001 00000000`00000000 : nt!PnpDeleteLockedDeviceNodes+0xa0
fffff880`031c49d0 fffff800`02ebb271 : fffffa80`04cc2c70 00000000`00000000 fffffa80`04cc2c70 00000000`00000001 : nt!PnpDelayedRemoveWorker+0x79
fffff880`031c4a20 fffff800`02bd129a : 00000000`00000000 fffffa80`05d03400 00000000`0000000a 00000000`00000000 : nt!PnpChainDereferenceComplete+0x131
fffff880`031c4a60 fffff800`02f4c2a0 : 00000000`00000000 fffffa80`0376c900 fffff8a0`02eed6a0 00000000`00000001 : nt!PnpIsChainDereferenced+0xda
fffff880`031c4ae0 fffff800`02f4c53c : fffff880`031c4cb8 00000000`00000000 fffff8a0`09163000 fffffa80`00000000 : nt!PnpProcessQueryRemoveAndEject+0xff0
fffff880`031c4c20 fffff800`02e3573e : 00000000`00000000 fffffa80`047d9a20 fffff8a0`02eed6a0 00000000`00000001 : nt!PnpProcessTargetDeviceEvent+0x4c
fffff880`031c4c50 fffff800`02ad8261 : fffff800`02d39f88 fffff8a0`02eed6a0 fffff800`02c742d8 00000000`00000000 : nt! ?? ::NNGAKEGL::`string'+0x54d9b
fffff880`031c4cb0 fffff800`02d6b2ea : 00000000`00000000 fffffa80`0357c040 00000000`00000080 fffffa80`03566040 : nt!ExpWorkerThread+0x111
fffff880`031c4d40 fffff800`02abf8e6 : fffff880`02fd3180 fffffa80`0357c040 fffff880`02fddfc0 ffffffff`e6fffffe : nt!PspSystemThreadStartup+0x5a
fffff880`031c4d80 00000000`00000000 : fffff880`031c5000 fffff880`031bf000 fffff880`031c4030 00000000`00000000 : nt!KxStartSystemThread+0x16
Now here's where we have some more info, and a much better stack. We can see a few
ndis.sys routines being called (Network Driver Interface Specification driver). The Network Driver Interface Specification (NDIS) is an application programming interface (API) for network interface cards (NICs). The NDIS forms the Logical Link Control (LLC) sublayer, which is the upper sublayer of the OSI data link layer (layer 2). Therefore, the NDIS acts as the interface between the Media Access Control (MAC) sublayer, which is the lower sublayer of the data link layer, and the network layer (layer 3).
The NDIS is a library of functions often referred to as a "wrapper" that hides the underlying complexity of the NIC hardware and serves as a standard interface for level 3 network protocol drivers and hardware level MAC drivers. Another common LLC is the Open Data-Link Interface (ODI).
-- I outlined
ndis!ndisPauseFilter from the call stack in red, because here in this specific case, we can also run
!ndiskd to find the object that is in a pausing state.
Let's look into the IRP list address that I've outlined in green:
Code:
0: kd> !irp fffffa8005a06bb0 7
Irp is active with 2 stacks 2 is current (= 0xfffffa8005a06cc8)
No Mdl: No System Buffer: Thread fffffa800357c040: Irp stack trace.
Flags = 00000000
ThreadListEntry.Flink = fffffa800357c430
ThreadListEntry.Blink = fffffa800357c430
IoStatus.Status = c00000bb
IoStatus.Information = 00000000
RequestorMode = 00000000
Cancel = 00
CancelIrql = 0
ApcEnvironment = 00
UserIosb = fffff880031c47e0
UserEvent = fffff880031c47f0
Overlay.AsynchronousParameters.UserApcRoutine = 00000000
Overlay.AsynchronousParameters.UserApcContext = 00000000
Overlay.AllocationSize = 00000000 - 00000000
CancelRoutine = 00000000
UserBuffer = 00000000
&Tail.Overlay.DeviceQueueEntry = fffffa8005a06c28
Tail.Overlay.Thread = fffffa800357c040
Tail.Overlay.AuxiliaryBuffer = 00000000
Tail.Overlay.ListEntry.Flink = 00000000
Tail.Overlay.ListEntry.Blink = 00000000
Tail.Overlay.CurrentStackLocation = fffffa8005a06cc8
Tail.Overlay.OriginalFileObject = 00000000
Tail.Apc = 00000000
Tail.CompletionKey = 00000000
cmd flg cl Device File Completion-Context
[ 0, 0] 0 0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
>[ 1b, 2] 0 0 fffffa80036b0050 00000000 00000000-00000000
*** ERROR: Module load completed but symbols could not be loaded for ew_jucdcecm.sys
\Driver\huawei_cdcecm
Args: 00000000 00000000 00000000 00000000
ew_jucdcecm.sys = HUAWEI Mobile Connect driver (Bus Enumerate Device). This is why we were seeing so many network related routines in the stack above.
Let's now go ahead and run !devstack on the hightlighted address for ew_jucdcecm.sys (green):
Code:
0: kd> !devstack fffffa80036b0050
!DevObj !DrvObj !DevExt ObjectName
> fffffa80036b0050 \Driver\huawei_cdcecmfffffa80036b01a0 NDMP19
fffffa8003694580 \Driver\huawei_enumeratorfffffa8003694390 00000086
!DevNode fffffa800376c900 :
DeviceInst is "usbcdcncm\Vid_12D1&Subclass_02&Prot_16\7&2fa1189a&0&0001_00"
ServiceName is "huawei_cdcecm"
In this case,
!devstack was used, which displays/dumps a formatted view of the device stack associated with a device object.
!devobj is used to display/dump information regarding a DEVICE_OBJECT structure. To get the address of a specific device object, you'll want to use
!drvobj.
I instructed the user to do two things:
1. Update the driver itself -
Support - Downloads - Huawei Consumer BG
2. ESET may be causing NETBIOS conflicts, which is in turn causing crashes. With this said, please remove and replace ESET with MSE for temporary troubleshooting purposes:
ESET removal -
How do I uninstall or reinstall ESET Smart Security/ESET NOD32 Antivirus? - ESET Knowledgebase
MSE -
Microsoft Security Essentials - Microsoft Windows
One (or both) solved it.
So, hopefully that helps anyone and/or is informative enough! I will add more to this when I read more into a few different things, but I am happy with it for the time being.
