[Win8.1] Kernel Thread Priority Floor Violation

blueelvis

BSOD Kernel Dump Senior Analyst
Joined
Apr 14, 2014
Posts
970
Location
India
Hi people,

On working on a case, while debugging I got this as the bugcheck code which has been introduced lately in the latest Windows 8.1 according to the Carrona DRT. Here is the analysis of the Dump file :-

6: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************


KERNEL_THREAD_PRIORITY_FLOOR_VIOLATION (157)
An illegal operation was attempted on the priority floor of a particular
thread.
Arguments:
Arg1: ffffe000d6aa2880, The address of the thread
Arg2: 0000000000000009, The target priority value
Arg3: 0000000000000002, The priority counter for the target priority underflowed
Arg4: 0000000000000000, Reserved


Debugging Details:
------------------




CUSTOMER_CRASH_COUNT: 1


DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT


BUGCHECK_STR: 0x157


PROCESS_NAME: chrome.exe


CURRENT_IRQL: 2


ANALYSIS_VERSION: 6.3.9600.16384 (debuggers(dbg).130821-1623) amd64fre


LAST_CONTROL_TRANSFER: from fffff801f25f66cd to fffff801f25dbfa0


STACK_TEXT:
ffffd000`21bd3858 fffff801`f25f66cd : 00000000`00000157 ffffe000`d6aa2880 00000000`00000009 00000000`00000002 : nt!KeBugCheckEx
ffffd000`21bd3860 fffff801`f24c4cb5 : 00000000`00000100 ffffd000`21bd3980 ffffe000`d6aa2880 00000000`00000002 : nt! ?? ::FNODOBFM::`string'+0xa21d
ffffd000`21bd38a0 fffff801`f24ecdaa : ffffe000`d6aa2ba0 00000000`00000000 00000000`00000000 ffffd000`21bd3980 : nt!KiAbThreadUnboostCpuPriority+0x39
ffffd000`21bd38d0 fffff801`f24eca45 : ffffd000`21c47943 00000000`00000000 ffffe000`d6a9e000 fffff801`00000000 : nt!KeAbEntryFree+0x66
ffffd000`21bd3900 fffff801`f24efa09 : ffffe000`0000004f ffffe000`d6aa2880 ffffe000`d6a39758 ffffd000`00000040 : nt!ExfAcquirePushLockExclusiveEx+0x1b5
ffffd000`21bd39c0 fffff801`f25e622f : 00000000`00000001 00000000`0341fdac 00000000`00000001 ffffd000`21bd3b00 : nt!MmAccessFault+0x7e9
ffffd000`21bd3b00 00000000`5fa356a7 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiPageFault+0x12f
00000000`0341f744 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x5fa356a7




STACK_COMMAND: kb


FOLLOWUP_IP:
nt! ?? ::FNODOBFM::`string'+a21d
fffff801`f25f66cd cc int 3


SYMBOL_STACK_INDEX: 1


SYMBOL_NAME: nt! ?? ::FNODOBFM::`string'+a21d


FOLLOWUP_NAME: MachineOwner


MODULE_NAME: nt


IMAGE_NAME: ntkrnlmp.exe


DEBUG_FLR_IMAGE_TIMESTAMP: 53388e13


IMAGE_VERSION: 6.3.9600.17085


BUCKET_ID_FUNC_OFFSET: a21d


FAILURE_BUCKET_ID: 0x157_nt!_??_::FNODOBFM::_string_


BUCKET_ID: 0x157_nt!_??_::FNODOBFM::_string_


ANALYSIS_SOURCE: KM


FAILURE_ID_HASH_STRING: km:0x157_nt!_??_::fnodobfm::_string_


FAILURE_ID_HASH: {68a86235-3c52-e42e-e778-31dc90dd5954}


Followup: MachineOwner
---------

A simple google search also did not reveal anything except the links to the Carrona DRT where these are stated as added in Windows 8.1 . So could someone shed some light over this and how should I proceed to debug this? Dumps along with relevant info have been attached.


View attachment Lachanadan.zip
 
re: Kernel Thread Priority Floor Violation - Windows 8.1

As noted, this bug check has essentially no information whatsoever, so it's up to any pre-existing knowledge to attempt to understand what's going on here. I will do my best to share what I know!

First of all, the big thing to note is to my knowledge, a client/user system/OS will never experience this bug check, and only server-based OS' 8+ do. To confirm this:

Code:
6: kd> vertarget
Windows 8 Kernel Version 9600 MP (8 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 9600.17085.amd64fre.winblue_gdr.140330-1035
Machine Name:
Kernel base = 0xfffff801`f2488000 PsLoadedModuleList = 0xfffff801`f27522d0
Debug session time: Mon Jun  2 11:17:52.139 2014 (UTC - 4:00)
System Uptime: 0 days 5:26:21.799

This system is a TS, and/or Terminal System. It's not a client/user system.

Right, so now that we know that, let's take a look at the bug check itself.

KERNEL_THREAD_PRIORITY_FLOOR_VIOLATION (157)

An illegal operation was attempted on the priority floor of a particular thread.

BugCheck 157, {ffffe000d6aa2880, 9, 2, 0}

1st parameter - The address of the thread.

2nd parameter - The target priority value (?).

3rd parameter - The priority counter for the target priority underflowed (?).

4th parameter - Reserved.

As I am unsure as to what the 2nd/3rd parameters are given there's no documentation on them, we can only work with the 1st.

Code:
6: kd> !thread ffffe000d6aa2880
GetPointerFromAddress: unable to read from fffff801f27dc000
THREAD ffffe000d6aa2880  Cid 0e04.139c  Teb: 00000000fe5e1000 Win32Thread: 0000000000000000 RUNNING on processor 6
Not impersonating
GetUlongFromAddress: unable to read from fffff801f2728300
Owning Process            ffffe000d6a39280       Image:         chrome.exe
Attached Process          N/A            Image:         N/A
fffff78000000000: Unable to get shared data
Wait Start TickCount      1253235      
Context Switch Count      2108197        IdealProcessor: 6             
ReadMemory error: Cannot get nt!KeMaximumIncrement value.
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address 0x000000005f38cd40
Stack Init ffffd00021bd3c90 Current ffffd00021bd35d0
Base ffffd00021bd4000 Limit ffffd00021bce000 Call 0
Priority 9 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           : Args to Child                                                           : Call Site
ffffd000`21bd3858 fffff801`f25f66cd : 00000000`00000157 ffffe000`d6aa2880 00000000`00000009 00000000`00000002 : nt!KeBugCheckEx
ffffd000`21bd3860 fffff801`f24c4cb5 : 00000000`00000100 ffffd000`21bd3980 ffffe000`d6aa2880 00000000`00000002 : nt! ?? ::FNODOBFM::`string'+0xa21d
ffffd000`21bd38a0 fffff801`f24ecdaa : ffffe000`d6aa2ba0 00000000`00000000 00000000`00000000 ffffd000`21bd3980 : [COLOR=#4b0082]nt!KiAbThreadUnboostCpuPriority+0x39[/COLOR]
ffffd000`21bd38d0 fffff801`f24eca45 : ffffd000`21c47943 00000000`00000000 ffffe000`d6a9e000 fffff801`00000000 : nt!KeAbEntryFree+0x66
ffffd000`21bd3900 fffff801`f24efa09 : ffffe000`0000004f ffffe000`d6aa2880 ffffe000`d6a39758 ffffd000`00000040 : [COLOR=#0000cd]nt!ExfAcquirePushLockExclusiveEx+0x1b5[/COLOR]
ffffd000`21bd39c0 fffff801`f25e622f : 00000000`00000001 00000000`0341fdac 00000000`00000001 ffffd000`21bd3b00 : [COLOR=#ff0000]nt!MmAccessFault+0x7e9[/COLOR]
ffffd000`21bd3b00 00000000`5fa356a7 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiPageFault+0x12f (TrapFrame @ ffffd000`21bd3b00)
00000000`0341f744 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x5fa356a7

nt!MmAccessFault+0x7e9
calls into nt!ExfAcquirePushLockExclusiveEx+0x1b5. With this said, we likely have a deadlock occurring, but why? We can see a nt!KiAbThreadUnboostCpuPriority+0x39 call, which we'll now discuss.

Every thread has has what is called a dynamic priority. This is essentially the priority the scheduler uses to determine which thread to execute. Initially, a thread's dynamic priority is the same as its base priority. The system can boost and lower the dynamic priority, to ensure that it is responsive and that no threads are starved for processor time. The system does not boost the priority of threads with a base priority level between 16 and 31. Only threads with a base priority between 0 and 15 receive dynamic priority boosts.

Some drivers create their own driver and/or device-dedicated system threads and set their thread's base priority to the lowest real-time priority value. Other highest-level drivers, particularly file system drivers, use system worker threads with a base priority that is usually set to the highest variable priority value. The kernel schedules a thread with the lowest real-time priority to run ahead of every thread with a variable priority, which includes almost every user-mode thread in the system.

Way more information here if interested - Scheduling Priorities (Windows)

What was the illegal instruction exactly? Not entirely sure, really. Not much debugging and/or curiousity digging to be done with a minidump. If I had to take a guess, I would say we had an issue with priority inversion, in which two entirely different threads were scheduled. The way to properly keep things running smooth if this occurs is the low priority threads will run long enough to exit the critical section, and the high-priority thread can enter the critical section. If the low-priority thread does not get enough CPU time to exit the critical section the first time, it will get another chance during the next round of scheduling.

Either that, or the driver that is causing the suspected lock is buggy.



Perusing the loaded modules list, here are the drivers I suspect may be causing a problem:

1. Kaspersky 100% more than anything. I have seen many issues with Kaspersky and Windows 8/8.1.

2. AsIO.sys (Asus PC Probe, other Asus bloatware).

Regards,

Patrick
 
re: Kernel Thread Priority Floor Violation - Windows 8.1

Oh, also, I forgot!

Given we have a thread address, that also means we have a raw stack to dump:

Code:
Base [COLOR=#ff0000]ffffd00021bd4000 [/COLOR]Limit [COLOR=#0000cd]ffffd00021bce000[/COLOR]

Code:
ffffd000`21bd31b0  fffff801`f2665ae8 [COLOR=#ff0000]nt!KiBugCheckProgress[/COLOR]
ffffd000`21bd31d0  00000000`0000002f
ffffd000`21bd31d8  fffff801`f241dff2 [COLOR=#000080]hal!HalRequestSoftwareInterrupt+0xd3[/COLOR]
ffffd000`21bd31e0  00000000`00140001
ffffd000`21bd31f0  00000000`0000002f
ffffd000`21bd31f8  fffff801`f241dff2 [COLOR=#000080]hal!HalRequestSoftwareInterrupt+0xd3[/COLOR]
ffffd000`21bd3200  ffffd000`21bd3218
ffffd000`21bd3220  00000000`00000009
ffffd000`21bd3228  fffff801`f24c68f6 [COLOR=#4b0082]nt!KiSetPriorityThread+0x4e[/COLOR]
ffffd000`21bd3230  ffffe000`d6aa2880
ffffd000`21bd3240  ffffe000`d6aa2800
ffffd000`21bd3248  fffff801`f24dff32 [COLOR=#006400]nt!KiAbProcessContextSwitch+0x2f2[/COLOR]

Let's discuss!

nt!KiAbProcessContextSwitch+0x2f2 -

The scheduler maintains a queue of executable threads for each priority level. These are known as ready threads. When a processor becomes available, the system performs a context switch. The steps in a context switch are:



  1. Save the context of the thread that just finished executing.
  2. Place the thread that just finished executing at the end of the queue for its priority.
  3. Find the highest priority queue that contains ready threads.
  4. Remove the thread at the head of the queue, load its context, and execute it.

We can see we then call the nt!KiSetPriorityThread+0x4e routine which sets the run-time priority of a driver-created thread. This confirms my earlier mentioned suspicion of a driver causing a possible deadlock.

We then see we get a software interrupt - hal!HalRequestSoftwareInterrupt+0xd3, but then we bug check.

So, as mentioned above, I would say one of two drivers is misbehaving.

Regards,

Patrick
 
Re: Kernel Thread Priority Floor Violation - Windows 8.1

Hi Patrick!

Sorry for my late reply as I was out of town for the past 4 days.

I have notified the original user of this and awaiting his reply. He found the Asus PC Probe uninstaller somewhere on his system, ran it as system admin but nothing happened and the drivers from Asus were still present there.

If this is still not resolved, I will try getting a Memory dump file. By the way how big it is generally?

According to you, the thread stack contains HAL being interacted so can I think that the driver is interacting with the hardware and ASIO.sys might have a more chance of the culprit than the Kaspersky?

Could you please explain priority inversion again as I am confused with that <_<


As always thanks for your support ^_^
 
Re: Kernel Thread Priority Floor Violation - Windows 8.1

No, as Asus PC Probe/Suite is OS > BIOS software. It's a direct link between the two. It's also why you see HAL calls, because it's monitoring voltages, fan speeds, and they are all modifiable via the OS.

Kernel-dumps differ in size greatly. It all depends on how much memory was in use by the kernel at the time of the crash. I've seen it as big as ~1GB and as small as a few hundred MBs.

Here's a good article on priority inversion - Priority Inversion

Regards,

Patrick
 
Re: Kernel Thread Priority Floor Violation - Windows 8.1

Ah, got it!

Will ask for a Memory dump if the problem still persists.
 
Back
Top