How to Debug a Driver Power State Failure (Bugcheck 0x9f) when 1st param = 0x3

jcgriff2 · Mar 19, 2019

Info

Bugcheck 0x9f (P1, P2, P3, P4)

0x9f=DRIVER_POWER_STATE_FAILURE and indicates that a driver is in an inconsistent or invalid power state.

P1, P2, P3, P4 = the Parameters (numbers) inside the parenthesis after the bugcheck code. Every bugcheck has 4 Parameters enclosed by parenthesis following it.

P1 = In this tutorial, the value = 0x3
P2 = memory address of the physical device object (PDO) of the stack
P3 = memory address of the functional device object (FDO) of the stack. In Windows 7 and later, nt!TRIAGE_9F_POWER
P4 = memory address of the blocked IRP (an I/O Request Packet)

Cause - A device object has been blocking an IRP for too long a time.

If you come upon a dump with bugcheck and parms = 0x9f (0x3,,,fffffa80'0ab61bd0)¹ (the memory address in P4 is variable - no 2 dumps or systems will contain the exact same memory address). NOTE: there is no P2 or P3 listed in the parameters of the bugcheck - I simply listed the commas with no values between them where these items would normally be)

For 0x9f bugcheck dumps, first, you look at P1. In this case it is 0x3, which tells you immediately that the Windbg command !irp can be run, which hopefully will reveal the name of a 3rd party driver (non-Microsoft driver). Microsoft drivers are considered to be sacrosanct and therefore are ruled out as the culprit in a BSOD.

If after Windows Updates are installed by hundreds-of-millions of systems on one night, or even spread over a few nights, if there is a rogue driver by Microsoft that got through, Microsoft's error reporting arm, WERCON, would immediately begin sending crash data back to Microsoft (including dump files) and Microsoft would know very quickly that one of its own modules is responsible. It does happen from time-to-time, but it is also a very rare event. In the event that Microsoft inadvertently sent out a bad driver, they would know it within 1-3 days; pull the affected module from future Windows Update runs; fix the module in question, re-release it to the public via Windows Updates and/or set up a download page for those users who wish to do it immediately and do it themselves.

The structure of the Windbg command (entered toward the bottom of the Windbg screen, to the right of kd>), is:

Rich (BB code):

 !irp fffffa80'0ab61bd0

The memory address next to !irp comes from P4)

¹ The 2 commas between P1 and P4 represent Parameter 2, 3 (P2, P3). They may or may not be filled with a number or memory address. In a 0x9f (0x3,,,) dump, we do not particularly care about the contents of P2 or P3. We look at P1 to see if it is a 0x3; if it is, then we issue the !irp Windbg command with the memory address found in P4.

Sometimes it is not even necessary to issue the !irp command as a "Warning" statements about missing symbols will give you a clue.

Code:

fffff800`00b9c8f8 fffff880`0638e288Unable to load image \SystemRoot\system32\DRIVERS\igdkmd64.sys, Win32 error 0n2
*** WARNING: Unable to verify timestamp for igdkmd64.sys
*** ERROR: Module load completed but symbols could not be loaded for igdkmd64.sys
igdkmd64+0x2c2288

The answer is igdkmd64.sys - the Intel graphics driver -- the driver needs to be updated, which likely will be very difficult to find as the driver not only contains video, but all chipset drivers. OP should be told to check the system manufacturer's support site (OEM) as well as Intel.com support for an updated driver. The driver(s) from Intel may not work as many manufacturers (OEMs) customize drivers they obtain from Intel or other company supplying drivers.

0x9F: DRIVER_POWER_STATE_FAILURE

Here is the IRP command in action. As you can see, the bugcheck on the dump was -

Rich (BB code):

BugCheck 9F, {3, ffffe58f5fa6d060, ffffe2000f6298d0, ffffe58f67630bf0}

Rich (BB code):

The driver named on the line with the greater than symbol > is nvlddmkm.sys - which I happen to know is a NVIDIA video driver and because it is a 3rd party (non-Microsoft), we give it our full attention as it is likely our culprit here.

I recommended that the OP update the NVIDIA driver because it was from 2016 and the thread occurred 1 March 2019. So the driver was probably about 3 years old. In some cases it is necessary to have the OP roll the driver back.

Unfortunately, the OP never replied back, so we don't know the outcome.

Here is additional literature on 0x9f bugcheck BSODs - the first two were written in-house by our Sysnative BSOD Experts.

LITERATURE

Regards. . .

jcgriff2

Warning

Not all 0x9f (0x3,,,) BSODs are this easy as often a Microsoft driver will be listed when you run the !irp command. This often means that we're dealing with unknown hardware failure and you must instruct the OP to run hardware diagnostic tests starting with RAM.

It is also a very good idea to have the OP run Driver Verifier first as Driver Verifier may flag a rogue 3rd party driver. If Driver Verifier flags a Microsoft driver, then unknown hardware failure is the cause and you should start the OP running the hardware diagnostic tests.

philc43 · Mar 20, 2019

My feedback...

I think it would be good to be clear about the aims of what these articles are for. I see them as guides for how to debug a particular bugcheck with worked examples. Since we know that there are differences between the WinDBG versions and also the new WinDBG Preview App it would be good to provide the mini-dump file(s) so that students can perform their own analysis and confirm the results.

Formatting of the articles will be important and it would be good to stick with a common style and use code boxes or other styles to keep it visually clear and attractive. Screenshots can be useful too like you have shown.

Example:

1: kd> !analyze -v
***

*
Bugcheck Analysis *
*

***

DRIVER_POWER_STATE_FAILURE (9f)
A driver has failed to complete a power IRP within a specific time.
Arguments:
Arg1: 0000000000000003, A device object has been blocking an Irp for too long a time
Arg2: ffffe58f5fa6d060, Physical Device Object of the stack
Arg3: ffffe2000f6298d0, nt!TRIAGE_9F_POWER on Win7 and higher, otherwise the Functional Device Object of the stack
Arg4: ffffe58f67630bf0, The blocked IRP

1: kd> !irp ffffe58f67630bf0
Irp is active with 6 stacks 5 is current (= 0xffffe58f67630de0)
No Mdl: No System Buffer: Thread 00000000: Irp stack trace. Pending has been returned
cmd flg cl Device File Completion-Context
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[IRP_MJ_POWER(16), IRP_MN_WAIT_WAKE(0)]
0 0 ffffe58f5fa6d060 00000000 fffff8047545d870-ffffe2000f61b890
\Driver\pci dxgkrnl!DpiFdoPowerCompletionRoutine
Args: 00000000 00000000 00000000 00000000
>[IRP_MJ_POWER(16), IRP_MN_SET_POWER(2)]
0 e1 ffffe58f65c36030 00000000 fffff80460f86930-ffffe58f68d0fc80 Success Error Cancel pending
\Driver\nvlddmkm nt!PopRequestCompletion
Args: 00000000 00000001 00000001 00000000
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-ffffe58f68d0fc80

Args: 00000000 00000000 00000000 00000000

Notice I have highlighted the > which is a useful indicator provided by the !irp command output to indicate the point where the problem is occurring.

Note: At the moment I am not too happy about the way that you can't cut and paste from WinDBG and preserve the formatting but that may be me not being used to the editing capabilities of this site.

MrPepka · Mar 20, 2019

I would also be in favor of providing full memory dumps for students . Why? Take for example the error code 0x9F with the first parameter 0x3.
After reviewing the blocked IRP you can come across what kind of driver caused the error right? But what if the blocked IRP has a Microsoft driver or driver with blocked IRP, however, is not guilty (we exchanged it for another one, and also exchanged the equipment for another,
but this other hardware has the same driver and there are still problems)? Then in this case you have to look at the driver branch with the extension drvobj. The problem is that it is only available in a full dump of memory (as most information). Of course, it will not always be necessary, but if students have something useful, why not?

jcgriff2 · Mar 20, 2019

MrPepka said:
But what if the blocked IRP has a Microsoft driver or driver with blocked IRP, however, is not guilty (we exchanged it for another one,

What do you mean by "exchanged it for another one" - when referring to the driver?

MrPepka said:
and also exchanged the equipment for another,

If a hardware change occurred, we would need a new set of dumps, would we not? I mean that all prior dumps are pretty useless at this point, in my opinion.

I am not a hardware expert by any means, but if during the course of working with an OP I learned that OP replaced RAM or other part, I would then consider all pre-hardware change dumps to be of no value because you don't know what affect the hardware change had on the BSODs. If this scenario actually were to happen, I would only work with new dumps generated after the hardware change.

MrPepka · Mar 21, 2019

jcgriff2 said:
What do you mean by "exchanged it for another one" - when referring to the driver?

I mean change version of driver

jcgriff2 said:
If a hardware change occurred, we would need a new set of dumps, would we not? I mean that all prior dumps are pretty useless at this point, in my opinion.

Yes, but if you change the NVIDIA graphics card to a different NVIDIA graphics card, the driver stays the same

jcgriff2 · Mar 21, 2019

OK... I'll buy that one.

@philc43 - I never knew that the > meant that is where the problem is occurring.

Feel free to quote mine; insert your changes and delete my stuff if you wish to do so.

jcgriff2 · Mar 27, 2019

I do understand the reasons for wanting full kernel memory dumps and we certainly can get to that at some point.

I think initially that we should stick with minidumps as we have literally many thousands of threads out there from which to choose for our new students and have a lot of ground to cover just to teach them the basics.

The other problem or issue that I see with full kernel dumps is that if we're dealing with unknown hardware failure, full kernel dumps will lead us down the wrong path in every direction.

jcgriff2 · Mar 27, 2019

@philc43 - I added a new Windbg output highlighting on the > sign and the NVIDIA driver only.

I also added note boxes at the beginning and at the end of the post.

Thoughts?

philc43 · Apr 1, 2019

Looking good now! I think the title could be changed. Something like: How to Debug a Driver Power State Failure (Bugcheck 0x9f) when 1st param = 3

jcgriff2 · Apr 1, 2019

Title changed.

jcgriff2 · Apr 4, 2019

Here is another 0x9f (0x3..._) naming NVIDIA, but the driver is brand new.

I advised OP to swap out his video card if Desktop PC.

Recurring BSOD - Windows 10 version 1809 - Windows Crashes and Blue Screen of Death (BSOD) Help and Support

Search

Search

How to Debug a Driver Power State Failure (Bugcheck 0x9f) when 1st param = 0x3

jcgriff2

Co-Founder / Admin
BSOD Instructor/Expert
Microsoft MVP (Ret.)

philc43

BSOD Forum Moderator, BSOD Academy Instructor, BSOD Kernel Dump Expert

Attachments

MrPepka

Sysnative Staff, BSOD Kernel Dump Senior Analyst

jcgriff2

Co-Founder / Admin
BSOD Instructor/Expert
Microsoft MVP (Ret.)

MrPepka

Sysnative Staff, BSOD Kernel Dump Senior Analyst

jcgriff2

Co-Founder / Admin
BSOD Instructor/Expert
Microsoft MVP (Ret.)

jcgriff2

Co-Founder / Admin
BSOD Instructor/Expert
Microsoft MVP (Ret.)

jcgriff2

Co-Founder / Admin
BSOD Instructor/Expert
Microsoft MVP (Ret.)

philc43

BSOD Forum Moderator, BSOD Academy Instructor, BSOD Kernel Dump Expert

jcgriff2

Co-Founder / Admin
BSOD Instructor/Expert
Microsoft MVP (Ret.)

jcgriff2

Co-Founder / Admin
BSOD Instructor/Expert
Microsoft MVP (Ret.)

How to Debug a Driver Power State Failure (Bugcheck 0x9f) when 1st param = 0x3

Co-Founder / AdminBSOD Instructor/ExpertMicrosoft MVP (Ret.)

BSOD Forum Moderator, BSOD Academy Instructor, BSOD Kernel Dump Expert

Attachments

Sysnative Staff, BSOD Kernel Dump Senior Analyst

Co-Founder / AdminBSOD Instructor/ExpertMicrosoft MVP (Ret.)

Sysnative Staff, BSOD Kernel Dump Senior Analyst

Co-Founder / AdminBSOD Instructor/ExpertMicrosoft MVP (Ret.)

Co-Founder / AdminBSOD Instructor/ExpertMicrosoft MVP (Ret.)

Co-Founder / AdminBSOD Instructor/ExpertMicrosoft MVP (Ret.)

BSOD Forum Moderator, BSOD Academy Instructor, BSOD Kernel Dump Expert

Co-Founder / AdminBSOD Instructor/ExpertMicrosoft MVP (Ret.)

Co-Founder / AdminBSOD Instructor/ExpertMicrosoft MVP (Ret.)

Co-Founder / Admin
BSOD Instructor/Expert
Microsoft MVP (Ret.)

Co-Founder / Admin
BSOD Instructor/Expert
Microsoft MVP (Ret.)

Co-Founder / Admin
BSOD Instructor/Expert
Microsoft MVP (Ret.)

Co-Founder / Admin
BSOD Instructor/Expert
Microsoft MVP (Ret.)

Co-Founder / Admin
BSOD Instructor/Expert
Microsoft MVP (Ret.)

Co-Founder / Admin
BSOD Instructor/Expert
Microsoft MVP (Ret.)

Co-Founder / Admin
BSOD Instructor/Expert
Microsoft MVP (Ret.)