BSOD secondary processor clock interrupt - Windows 7 x64

DotFooz · Mar 1, 2014

Hello. So lately I've been having the exact same BSOD occur over and over again.
I've noticed that it occurs a lot during gaming, but it may have happened before
that as well. The screen locks up and the sound freezes. After about 5-10 seconds
it will switch to the blue screen. Any ideas on what's causing this?

Specs:

Code:

OS - Windows 7 64-bit SP1
Original OS - windows 7 64-bit SP1
The OS came pre-installed on the system
Age of system  - ~4 years
Age of the OS - ~2 years, i switched hard drives and cloned it over
CPU - Intel Core i7 2600K @ 3.40GHz
Video card - NVIDIA GeForce GTX 770
Motherboard - ASUSTeK SABERTOOTH P67 (LGA1155)
Power Supply  - Corsair TX-850
Laptop or Desktop? Desktop

EDIT: forgot to add attachments. Added them now.

Patrick · Mar 1, 2014

Hi,

There are no .DMP files located in your jcgriff2 output folder. As this is an 0x101 bug check, we'll need a Kernel-Dump. Can you please check C:\Windows for a MEMORY.DMP? If it's listed there, please upload that to a 3rd party website such as Onedrive, Mediafire, Dropbox, etc. If not, let's configure your system to generate Kernel-Dumps - Creating a Kernel-Mode Dump File (Windows Debuggers)

Regards,

Patrick

DotFooz · Mar 1, 2014

Patrick said:
Hi,

There are no .DMP files located in your jcgriff2 output folder. As this is an 0x101 bug check, we'll need a Kernel-Dump. Can you please check C:\Windows for a MEMORY.DMP? If it's listed there, please upload that to a 3rd party website such as Onedrive, Mediafire, Dropbox, etc. If not, let's configure your system to generate Kernel-Dumps - Creating a Kernel-Mode Dump File (Windows Debuggers)

Regards,

Patrick

Memory.DMP files were not being created it turns out. but they are now. I read of the force crash with keys, but would there be a way to trigger the real error that keeps crashing my computer? Or is a keyboard crash good enough?

Patrick · Mar 1, 2014

Don't keyboard crash, just wait. If you set up the forced method of crashing via the registry, we'll get a 0xDEADDEAD bugcheck and because you forced it, it won't tell us anything regarding what's actually causing your crash. It will just say that you crashed the system manually, essentially.

Regards,

Patrick

DotFooz · Mar 16, 2014

Another crash finally occurred. It seems to only happen when playing games on my computer. Here's the Memory.DMP for you.
https://www.dropbox.com/s/c50byfbzqkhd1qz/MEMORY.DMP

Patrick · Mar 16, 2014

CLOCK_WATCHDOG_TIMEOUT (101)

This indicates that an expected clock interrupt on a secondary processor, in a multi-processor system, was not received within the allocated interval.

So there's the basic definition of this particular bug check. Let's get into the debugging now.

Code:

BugCheck 101, {[COLOR=#ff0000][I][U][B]19[/B][/U][/I][/COLOR], 0, [COLOR=#ff0000][I][U][B]fffff880036a5180[/B][/U][/I][/COLOR], [COLOR=#ff0000][I][U][B]5[/B][/U][/I][/COLOR]}

^^ 19 clock ticks in regards to the timeout.

fffff880036a5180 is the PRCB address of the hung processor, let's keep this address in mind.

Code:

0: kd> !prcb 5
PRCB for Processor 5 at [COLOR=#ff0000]fffff880036a5180[/COLOR]:
Current IRQL -- 0
Threads--  Current fffffa8013055b50 Next fffffa80123fe8b0 Idle fffff880036b00c0
Processor Index 5 Number (0, 5) GroupSetMember 20
Interrupt Count -- 041ea28a
Times -- Dpc    0000006b Interrupt 000001b0 
         Kernel 001a2603 User      00024425

For reference, I did not do !prcb 0 through 5. That would have been very tedious. Instead, you can use !running -it. The "i" argument causes it to display idle processors too, and "t" displays the stack trace for the thread running on each processor. If we run that extension, it shows the is an 8 core box.

Hint: At times, the 4th parameter of the bug check will show you the responsible processor. For example, in your *101 here, it was correct as the 4th parameter was 5.

Hint #2: You can also generally tell the amount of cores on the box by checking the bugcheck_string - BUGCHECK_STR: CLOCK_WATCHDOG_TIMEOUT_8_PROC

As this matches the 3rd parameter of the bug check, processor #5 is the responsible processor. Now with the information we have here thus far, we know that processor #5 reached 19 clock ticks without responding, therefore the system crashed. Before we go further, what is a clock tick? A clock interrupt is a form of interrupt which involves counting the the cycles of the processor core, which is running a clock on the processors to keep them all in sync. A clock interrupt is handed out to all processors and then they must report in, and when one doesn't report in, you then crash.

Let's now look at the stacks of the different processors to see what the threads were involved in:

We can use knL and go through a grueling method of obtaining the trap frame, but we don't like having to put in more work, so let's use kv instead on Processor 0:

Code:

0: kd> kv
Child-SP          RetAddr           : Args to Child                                                           : Call Site
fffff880`009a9628 fffff800`050d4a4a : 00000000`00000101 00000000`00000019 00000000`00000000 fffff880`036a5180 : nt!KeBugCheckEx
fffff880`009a9630 fffff800`050876f7 : 00000000`00000000 fffff800`00000005 00000000`00002710 fffff880`009a9628 : nt! ?? ::FNODOBFM::`string'+0x4e3e
fffff880`009a96c0 fffff800`055f7895 : fffff800`0561d460 fffff880`009a9870 fffff800`0561d460 00000000`00000000 : nt!KeUpdateSystemTime+0x377
fffff880`009a97c0 fffff800`0507a113 : 00000000`763d1536 00000000`00000000 00000000`00000000 00000000`00000000 : hal!HalpHpetClockInterrupt+0x8d
fffff880`009a97f0 fffff800`050b34bd : 00000000`00000000 00000000`00000000 00000000`00000000 fffffa80`0c5ea618 : nt!KiInterruptDispatchNoLock+0x163 (TrapFrame @ [COLOR=#ff0000]fffff880`009a97f0[/COLOR])
fffff880`009a9980 fffff800`05082a1c : 00000000`00000000 fffff880`009a9ab8 00000000`00000000 00000000`00000000 : nt!KxFlushEntireTb+0xcd
fffff880`009a99c0 fffff800`050406e9 : 00000000`00000008 00000000`0000007f fffffa80`0027cde0 00000000`00000080 : nt!KeFlushMultipleRangeTb+0x28c
fffff880`009a9a90 fffff800`05040f37 : ffffffff`ffffffff 00000000`0000007f 00000000`00000000 fffffa80`0c5e96f8 : nt!MiZeroPageChain+0x14e
fffff880`009a9ad0 fffff800`0531a2ea : fffffa80`0c7a3040 00000000`00000080 fffffa80`0c7a26f0 fffff800`0506e8d9 : nt!MmZeroPageThread+0x83a
fffff880`009a9c00 fffff800`0506e8e6 : fffff800`051f8e80 fffffa80`0c7a3040 fffff800`05206cc0 00000000`00000000 : nt!PspSystemThreadStartup+0x5a
fffff880`009a9c40 00000000`00000000 : fffff880`009aa000 fffff880`009a4000 fffff880`009a9870 00000000`00000000 : nt!KxStartSystemThread+0x16

Code:

0: kd> .trap fffff880`009a97f0
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=00000000031ed2b9 rbx=0000000000000000 rcx=fffff880009a9ab8
rdx=00000000000000fa rsi=0000000000000000 rdi=0000000000000000
[COLOR=#4b0082]rip=fffff800050b34bd[/COLOR] rsp=fffff880009a9980 rbp=0000000000000008
 r8=fffff880009a9ac0  r9=0000000000000001 r10=0000000000000001
r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl nz na po nc
[COLOR=#ff0000]nt!KxFlushEntireTb+0xcd[/COLOR]:
[COLOR=#006400]fffff800`050b34bd[/COLOR] a801            test    al,1

Code:

0: kd> u @rip
nt!KxFlushEntireTb+0xcd:
fffff800`050b34bd a801            test    al,1
fffff800`050b34bf 75e6            [COLOR=#ff0000]jne [/COLOR]    [COLOR=#4b0082]nt!KxFlushEntireTb+0xb7[/COLOR] (fffff800`050b34a7)
fffff800`050b34c1 f00fba2db6e6140000 lock bts dword ptr [nt!KiTbFlushTimeStamp (fffff800`05201b80)],0
fffff800`050b34ca 0f834bffffff    jae     [COLOR=#4b0082]nt!KxFlushEntireTb+0x2b[/COLOR] (fffff800`050b341b)
fffff800`050b34d0 ebd5            [COLOR=#ff0000]jmp[/COLOR]     n[COLOR=#4b0082]t!KxFlushEntireTb+0xb7[/COLOR] (fffff800`050b34a7)
fffff800`050b34d2 90              nop
fffff800`050b34d3 90              nop
fffff800`050b34d4 90              nop

^^ Disassembling the first few instructions reveals a jump (jmp) that is back up in the nt!KxFlushEntireTb function.

So, what's the summary so far? Processor #0 was the thread that created the bugcheck itself, and must have been interrupted by a clock interrupt in order to trigger the CLOCK_WATCHDOG_TIMEOUT bug check.

Let's take a look into Processor #1's call stack like we did Processor #0:

Code:

1: kd> kv
Child-SP          RetAddr           : Args to Child                                                           : Call Site
fffff880`09b35d20 fffff800`0509d251 : 00000000`00000000 00000000`00000005 00000000`00000001 fffffa80`185ce970 : nt!KeFlushMultipleRangeTb+0x260
fffff880`09b35df0 fffff800`051aeb5e : 00000000`00000005 fffff880`09b35f40 fffff6fd`400be480 fffffa80`185ce970 : nt!MiFlushTbAsNeeded+0x1d1
fffff880`09b35f00 fffff800`0509d9b0 : 00000000`00005000 fffff800`0520d580 00000000`00000000 fffff800`051ae514 : nt!MiAllocatePoolPages+0x4de
fffff880`09b36040 fffff800`051b243e : 00000000`00000000 00000000`00000000 fffffa80`00000000 00000000`00005000 : nt!ExpAllocateBigPool+0xb0
fffff880`09b36130 fffff880`019d9beb : fffff880`019e6964 00000000`00000000 00000000`6e4d7641 fffff8a0`00000000 : nt!ExAllocatePoolWithTag+0x82e
fffff880`09b36220 fffff880`019d9fb9 : 00000000`00000000 00000000`00000014 00000000`00000001 00000000`00000001 : aswMonFlt+0x1beb
fffff880`09b36270 fffff880`019f377a : 00000000`00000001 fffff880`09b36458 fffff8a0`16345be0 fffff880`00000001 : [COLOR=#ff0000]aswMonFlt+0x1fb9[/COLOR]
fffff880`09b362d0 fffff880`01004288 : 00000000`00000000 fffff880`09b36458 fffffa80`185ceda0 00000000`00000000 : [COLOR=#ff0000]aswMonFlt+0x1b77a[/COLOR]
fffff880`09b36410 fffff880`01002d1b : fffffa80`11755030 fffffa80`17b63bd0 fffffa80`132fe010 fffffa80`132fe230 : fltmgr!FltpPerformPostCallbacks+0x368
fffff880`09b364e0 fffff880`010222b9 : fffffa80`185ce970 fffffa80`11761010 fffffa80`185ce900 fffffa80`11762450 : fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x39b
fffff880`09b36570 fffff800`0537d43c : 00000000`00000005 fffffa80`17a1caa8 fffffa80`0d05ea90 00000000`00000000 : fltmgr!FltpCreate+0x2a9
fffff880`09b36620 fffff800`05378db8 : fffffa80`10cc4cd0 fffff800`00000000 fffffa80`17a1c8f0 00000000`00000001 : nt!IopParseDevice+0x14d3
fffff880`09b36780 fffff800`05379fd6 : 00000000`00000000 fffffa80`17a1c8f0 00000000`c0000225 fffffa80`0c7b58e0 : nt!ObpLookupObjectName+0x588
fffff880`09b36870 fffff800`0537b8dc : 00000000`00000000 00000000`00000000 fffffa80`1900ba01 fffffa80`14ce30c0 : nt!ObOpenObjectByName+0x306
fffff880`09b36940 fffff800`05386ed4 : 00000000`0020eee8 fffff8a0`80100080 00000000`0020ef38 00000000`0020eef8 : nt!IopCreateFile+0x2bc
fffff880`09b369e0 fffff800`0507ce53 : 00000000`00000000 000007fe`f0122808 00000000`00000016 fffff880`09b36ae0 : nt!NtCreateFile+0x78
fffff880`09b36a70 00000000`77a0180a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`09b36ae0)
00000000`0020edb8 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x77a0180a

^^ avast! isn't playing nice (to no surprise).

Let's check Processor #2:

Code:

2: kd> kv
Child-SP          RetAddr           : Args to Child                                                           : Call Site
fffff880`0bbb8c90 fffff800`050b23cd : 00000000`00000000 00000000`00000001 ffffffff`ffffffff fffff800`051b1d0e : nt!KxFlushEntireTb+0xcd
fffff880`0bbb8cd0 fffff800`050d69b0 : 00000000`00000001 fffff680`00038b80 00000000`00000001 fffffa80`124164a8 : nt!KeFlushTb+0x119
fffff880`0bbb8d50 fffff800`0508a89d : fffffa80`0a2c1b30 fffff880`0bbb8e00 fffffa80`177abd40 00000000`00000000 : nt! ?? ::FNODOBFM::`string'+0xada2
fffff880`0bbb8d90 fffff800`0507bcee : 00000000`00000000 00000000`07170000 00000000`53646100 00000000`00000000 : nt!MmAccessFault+0xa7d
fffff880`0bbb8ef0 fffff800`0535de04 : fffff880`0bbb9b60 00000000`00000000 fffffa80`0d2e8070 fffff880`0bbb90d8 : nt!KiPageFault+0x16e (TrapFrame @ fffff880`0bbb8ef0)
fffff880`0bbb9080 fffff880`052f9aa9 : 00000000`00000001 00000000`000005f0 00000000`0951f0c0 00000000`00000000 : nt!NtQueryObject+0xa4
fffff880`0bbb9190 fffff880`053b83c3 : 00000000`00000000 00000000`82ac8004 00000000`0951f0c0 00000000`0000008c : aswSnx+0x2caa9
fffff880`0bbb9200 fffff880`052d07bc : 00000000`00c6b80a 00000000`00000000 fffffa80`177b8740 fffffa80`178c0128 : [COLOR=#ff0000]aswSnx+0xeb3c3[/COLOR]
fffff880`0bbb9880 fffff800`0539a3a7 : fffffa80`00000000 fffffa80`145c9c50 fffffa80`145c9c50 fffffa80`178c0010 : [COLOR=#ff0000]aswSnx+0x37bc[/COLOR]
fffff880`0bbb98d0 fffff800`0539ac06 : fffff880`0bbb9a01 00000000`00000000 00000000`00000001 00000000`00000000 : nt!IopXxxControlFile+0x607
fffff880`0bbb9a00 fffff800`0507ce53 : fffffa80`134b0b50 fffff880`0bbb9b60 fffff880`0bbb9b60 00000000`00000000 : nt!NtDeviceIoControlFile+0x56
fffff880`0bbb9a70 00000000`77a0132a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`0bbb9ae0)
00000000`0951e718 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x77a0132a

^^ More avast! not playing nice.

Let's check Processor #3:

Code:

3: kd> kv
Child-SP          RetAddr           : Args to Child                                                           : Call Site
fffff880`03041680 fffff800`05388d4f : 00000000`00000000 fffff880`03041b60 00000000`00000000 00001f80`00000200 : [COLOR=#ff0000]nt!KeFlushProcessWriteBuffers+0x6b[/COLOR]
fffff880`030416f0 fffff800`053893ad : 00000000`003fed10 fffff800`05373cce 00000000`00000000 fffff800`0507cd19 : nt!ExpQuerySystemInformation+0x13af
fffff880`03041aa0 fffff800`0507ce53 : 00000000`00000000 fffff880`03041b60 ffffffff`fffe7960 000007fe`fad80b48 : nt!NtQuerySystemInformation+0x4d
fffff880`03041ae0 00000000`77a0161a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`03041ae0)
00000000`011cf3e8 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x77a0161a

^^ On Proc #3, the KeFlushProcessWriteBuffers function was called. This function generates an interprocessor interrupt (IPI) to all processors that are part of the current process affinity. It guarantees the visibility of write operations performed on one processor to the other processors.

Essentially, it flushes the write queue of each processor that is running a thread of the current process.

Let's check Processor #4:

Code:

4: kd> kv
Child-SP          RetAddr           : Args to Child                                                           : Call Site
fffff880`0c46e880 fffff800`050b23cd : 00000000`00000000 00000000`00000001 00000000`000007d3 00000000`00000001 : [COLOR=#ff0000]nt!KxFlushEntireTb+0x79[/COLOR]
fffff880`0c46e8c0 fffff800`050d69b0 : 00000000`00000002 fffff680`0005fc00 00000000`00000001 00000000`754e2450 : nt!KeFlushTb+0x119
fffff880`0c46e940 fffff800`0508a89d : fffffa80`0acf1a00 fffff880`0c46ea00 00000000`0409e310 00000000`00000000 : nt! ?? ::FNODOBFM::`string'+0xada2
fffff880`0c46e980 fffff800`0507bcee : 00000000`00000001 00000000`0bf80000 00000000`199e7401 00000000`00000040 : nt!MmAccessFault+0xa7d
fffff880`0c46eae0 00000000`5bb95c37 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiPageFault+0x16e (TrapFrame @ fffff880`0c46eae0)
00000000`037ce578 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x5bb95c37

^^ Similiar to Proc #1.

Let's now take a look at the problematic processor (#5):

Code:

5: kd> kv
Child-SP          RetAddr           : Args to Child                                                           : Call Site
fffff880`02eff418 fffff800`055f4ae7 : fffffa80`14cff150 fffff800`051b130d fffff800`0520b940 fffff800`050822e0 : hal!HalpLegacyApicReadGenericReg+0xd
fffff880`02eff420 fffff800`0506f7f3 : fffffa80`13055b50 fffffa80`177e8908 fffffa80`14cff290 00000000`00000140 : hal!HalRequestSoftwareInterrupt+0x58
fffff880`02eff450 fffff800`0505e6a4 : fffffa80`13055b50 00000000`00000001 00000000`00000000 fffffa80`177e8908 : nt!KiInsertQueueApc+0x243
fffff880`02eff480 fffff800`05092afc : fffffa80`177e8890 00000000`00000000 00000000`00000000 00000000`00000082 : nt!KeInsertQueueApc+0x80
fffff880`02eff4e0 fffff800`050705f7 : 00000000`00000000 00000000`00000000 3dcc2d58`20707200 00000000`00000000 : nt!IopCompleteRequest+0xb8c
fffff880`02eff5b0 fffff800`050737fd : fffffa80`13055b50 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiDeliverApc+0x1c7
fffff880`02eff630 fffff800`050800ea : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiCommitThreadWait+0x3dd
fffff880`02eff6c0 fffff960`0015a6a0 : fffff900`00000002 fffffa80`12328820 fffff900`00000001 fffff880`0000000d : nt!KeWaitForMultipleObjects+0x272
fffff880`02eff980 fffff960`0015b663 : 00000000`00000000 fffff900`c3eaf010 fffff960`003a77c0 00000000`00000004 : win32k!xxxMsgWaitForMultipleObjects+0x108
fffff880`02effa00 fffff960`00115084 : fffffa80`00000001 fffffa80`0000000c fffffa80`13055b50 fffff6fc`40017630 : win32k!xxxDesktopThread+0x253
fffff880`02effa80 fffff960`0019619a : fffffa80`00000001 fffff960`003a77c0 00000000`00000020 00000000`00000000 : win32k!xxxCreateSystemThreads+0x64
fffff880`02effab0 fffff800`0507ce53 : fffffa80`13055b50 00000000`00000004 000007ff`fffaa000 00000000`00000000 : win32k!NtUserCallNoParam+0x36
fffff880`02effae0 000007fe`fd771eea : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`02effae0)
00000000`0417fd78 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x000007fe`fd771eea

Interesting stack...

We have an NT user call, which appears to be to create a system thread, which then creates a desktop thread. Afterwards, we appear to have to wait and the commit for the wait is called. We then have a need to deliver an APC (asynchronous procedure call - will explain below), which is then completed, and it's inserted into a queue. Directly afterwards however, the Hardware Abstraction Layer requests a software interrupt.

As I said, APC's:

An asynchronous procedure call (APC) is a function that executes asynchronously in the context of a particular thread. When an APC is queued to a thread, the system issues a software interrupt. The next time the thread is scheduled, it will run the APC function. An APC generated by the system is called a kernel-mode APC. An APC generated by an application is called a user-mode APC. A thread must be in an alertable state to run a user-mode APC.

To spare some post-space and time, I'll just go ahead and say that Proc's #6 and #7 weren't too interesting, only in that they had some pool allocation routines going on in the stack, and virtual memory addresses were being freed and deleted.

Overall, avast! does not appear to be playing nice, and it's very evident as to why. I took a look at your loaded modules list, and I see avast! + AVG Secure Search + COMODO. All of this = lots and lots of interceptor conflicts, causing kernel corruption.

1. Remove and replace avast! with Microsoft Security Essentials for temporary troubleshooting purposes:

avast! removal - avast! Uninstall Utility | Download aswClear for avast! Removal

MSE - Microsoft Security Essentials - Microsoft Windows

2. Remove AVG Secure Search.

3. Remove COMODO.

Also, I note TrueCrypt. In my time as an analyst, from what I've seen, 3rd party antiviruses don't seem to play too well with encrypted drives. Just some food for thought.

Regards,

Patrick

sergmat · Mar 18, 2014

This looks interesting:

Code:

5: kd> !ipi
IPI State for Processor 0
    TargetCount          0  PacketBarrier        0  IpiFrozen     0 [Running]

IPI State for Processor 1
    As a sender, awaiting IPI completion from processor(s) [COLOR=#FF0000]5[/COLOR].
    TargetCount          1  PacketBarrier        1  IpiFrozen     2 [Frozen]

IPI State for Processor 2
    TargetCount          0  PacketBarrier        0  IpiFrozen     2 [Frozen]

IPI State for Processor 3
    As a sender, awaiting IPI completion from processor(s) [COLOR=#FF0000]5[/COLOR].
    TargetCount          1  PacketBarrier        1  IpiFrozen     2 [Frozen]

IPI State for Processor 4
    As a sender, awaiting IPI completion from processor(s) [COLOR=#FF0000]5[/COLOR].
    TargetCount          1  PacketBarrier        1  IpiFrozen     2 [Frozen]

IPI State for Processor [COLOR=#FF0000]5[/COLOR]
    As a receiver, unhandled requests are pending from processor(s)[COLOR=#0000FF] 1, 3, 4, 7.[/COLOR]
    TargetCount          0  PacketBarrier        0  IpiFrozen     2 [Frozen]
    From processor 1, active request of type: flush multiple range
        Flush Count 1  Flush List fffff88009b35e28  (dp fffff88009b35e28 l1)
    From processor 3, active request of type: packet ready
        WorkerRoutine fffff8000506ee10 (nt!FsRtlpNopStackOverflowRoutine)
        Parameter[0] 0  Parameter[1] 0  Parameter[2] 0
    From processor 4, active request of type: flush all
    From processor 7, active request of type: flush multiple range
        Flush Count 1  Flush List fffff880091958e8  (dp fffff880091958e8 l1)
IPI State for Processor 6
    TargetCount          0  PacketBarrier        0  IpiFrozen     2 [Frozen]

IPI State for Processor 7
    As a sender, awaiting IPI completion from processor(s) [COLOR=#FF0000]5[/COLOR].
    TargetCount          1  PacketBarrier        1  IpiFrozen     2 [Frozen]

It seems that on CPU #5 interrupts are disabled, INTERRUPTS_ENABLED flag not set.

Code:

0: kd> !eflags
D:\Documents\BugCheck101.DMP
BIT_1_RESERVED
PARITY_FLAG
ZERO_FLAG
INTERRUPTS_ENABLED
0: kd> ~1s
1: kd> !eflags
D:\Documents\BugCheck101.DMP
BIT_1_RESERVED
INTERRUPTS_ENABLED
1: kd> ~2s
2: kd> !eflags
D:\Documents\BugCheck101.DMP
BIT_1_RESERVED
INTERRUPTS_ENABLED
2: kd> ~3s
3: kd> !eflags
D:\Documents\BugCheck101.DMP
BIT_1_RESERVED
INTERRUPTS_ENABLED
3: kd> ~4s
4: kd> !eflags
D:\Documents\BugCheck101.DMP
BIT_1_RESERVED
INTERRUPTS_ENABLED
4: kd> ~5s
[COLOR=#FF0000]5[/COLOR]: kd> !eflags
D:\Documents\BugCheck101.DMP
BIT_1_RESERVED
IDENTIFICATION         
5: kd> ~6s
6: kd> !eflags
D:\Documents\BugCheck101.DMP
CARRY_FLAG
BIT_1_RESERVED
INTERRUPTS_ENABLED
6: kd> ~7s
7: kd> !eflags
D:\Documents\BugCheck101.DMP
BIT_1_RESERVED
INTERRUPTS_ENABLED

Patrick · Mar 18, 2014

Great analysis, Sergei!

That is very interesting, as it appears 1, 3, 4, and 7 are all waiting for inter-processor interrupts from #5. It's also said here from the !ipi output you pasted:

Code:

As a receiver, unhandled requests are pending from processor(s)[COLOR=#0000FF] 1, 3, 4, 7.[/COLOR]

Given this is the case, I imagine due to seeing mention of the nt!KxFlushEntireTb function, which requires the use of an IPI to clear the TLB cache, we never actually had a cache clear, and that's what 1, 3, 4, and 7 were indefinitely waiting on. It's also likely why we have the jump up into nt!KxFlushEntireTb, and then the bugcheck. This all to me appears to have been caused by avast! conflicting with the other security software installed, but we'll have to wait on the user to report back to us before we ever really know.

Regards,

Patrick

DotFooz · Apr 19, 2014

Hello again guys. Getting the crashes again. uninstalled all the antivirus too. Here's the new MEMORY.DMP
https://www.dropbox.com/s/c50byfbzqkhd1qz/MEMORY.DMP

Patrick · Apr 19, 2014

Hi,

Unfortunately, it's a severely corrupt dump. We'll need to wait for another crash and another kernel-dump. For the record, when dumps get so corrupt like this, it's from what I've seen at least as my time as an analyst a fault with the CPU in some shape or form. Overheating, faulty cache, etc.

Regards,

Patrick

DotFooz · Apr 19, 2014

I don't think I let that previous one fully dump. However I did for this one. Here's another thing. These crashes only seem to occur when I'm playing games. Could that mean anything?
Here's the dump: https://www.dropbox.com/s/c50byfbzqkhd1qz/MEMORY.DMP

Patrick · Apr 19, 2014

That's definitely why! If the dumping process doesn't complete gracefully, it'll be corrupt.

Code:

0: kd> kv
Child-SP          RetAddr           : Args to Child                                                           : Call Site
fffff880`009a9628 fffff800`052d4a4a : 00000000`00000101 00000000`00000019 00000000`00000000 fffff880`009b3180 : nt!KeBugCheckEx
fffff880`009a9630 fffff800`052876f7 : 00000000`00000000 fffff800`00000004 00000000`00002711 00000000`00000000 : nt! ?? ::FNODOBFM::`string'+0x4e3e
fffff880`009a96c0 fffff800`057f7895 : fffff800`0581d460 fffff880`009a9870 fffff800`0581d460 00000000`00000000 : nt!KeUpdateSystemTime+0x377
fffff880`009a97c0 fffff800`0527a113 : 00000000`5587bd8b fffff800`053f8e80 fffff800`053f8e80 00000000`00000080 : hal!HalpHpetClockInterrupt+0x8d
fffff880`009a97f0 fffff800`052b3473 : fffff800`053f8e80 00000000`00000001 00000000`00000000 00000000`00000000 : nt!KiInterruptDispatchNoLock+0x163 ([COLOR=#ff0000]TrapFrame @ fffff880`009a97f0[/COLOR])
fffff880`009a9980 fffff800`05282a1c : 00000000`00000000 fffff880`009a9ab8 00000000`41b5c703 00000000`00000000 : nt!KxFlushEntireTb+0x83
fffff880`009a99c0 fffff800`052406e9 : 00000000`00000080 00000000`0000007f fffffa80`005c57d0 00000000`00000080 : nt!KeFlushMultipleRangeTb+0x28c
fffff880`009a9a90 fffff800`05240f37 : 00000000`0001ec00 00000000`0000007f 00000000`00000000 00000000`00000000 : nt!MiZeroPageChain+0x14e
fffff880`009a9ad0 fffff800`0551a2ea : fffffa80`0c7a3450 00000000`00000080 fffffa80`0c7a39e0 fffff800`0526e8d9 : nt!MmZeroPageThread+0x83a
fffff880`009a9c00 fffff800`0526e8e6 : fffff800`053f8e80 fffffa80`0c7a3450 fffff800`05406cc0 38c48348`fffbe43c : nt!PspSystemThreadStartup+0x5a
fffff880`009a9c40 00000000`00000000 : fffff880`009aa000 fffff880`009a4000 fffff880`009a9770 00000000`00000000 : nt!KxStartSystemThread+0x16

Code:

0: kd> .trap fffff880`009a97f0
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000000001 rbx=0000000000000000 rcx=00000000000406f8
rdx=00000000000c00e1 rsi=0000000000000000 rdi=0000000000000000
[COLOR=#ff0000]rip=fffff800052b3473[/COLOR] rsp=fffff880009a9980 rbp=0000000000000080
 r8=00000000000000e1  r9=0000000000000001 r10=0000000000000000
r11=fffff88003789180 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl nz na pe nc
[COLOR=#ff0000]nt!KxFlushEntireTb+0x83[/COLOR]:
fffff800`052b3473 ffc3            inc     ebx

Code:

0: kd> u @rip
nt!KxFlushEntireTb+0x83:
fffff800`052b3473 ffc3            inc     ebx
fffff800`052b3475 851d4d1f2000    test    dword ptr [nt!HvlLongSpinCountMask (fffff800`054b53c8)],ebx
fffff800`052b347b 0f8479e1f5ff    je      nt! ?? ::FNODOBFM::`string'+0x7437 (fffff800`052115fa)
fffff800`052b3481 f390            pause
fffff800`052b3483 ebe4            [COLOR=#ff0000]jmp[/COLOR]    [COLOR=#ff0000] nt!KxFlushEntireTb+0x79[/COLOR] ([COLOR=#4b0082]fffff800`052b3469[/COLOR])
[COLOR=#006400]fffff800`052b3485[/COLOR] f08305f3e6140001 lock add dword ptr [nt!KiTbFlushTimeStamp (fffff800`05401b80)],1
fffff800`052b348d 400fb6c6        movzx   eax,sil
fffff800`052b3491 440f22c0        mov     cr8,rax

Code:

0: kd> u fffff800`052b3469 fffff800`052b3485
nt!KxFlushEntireTb+0x79:
fffff800`052b3469 8b8780200000    mov     eax,dword ptr [rdi+2080h]
fffff800`052b346f 85c0            test    eax,eax [COLOR=#ff8c00]<--- Checking if value is non-zero.[/COLOR]
fffff800`052b3471 7412            [COLOR=#ff0000]je[/COLOR]      [COLOR=#4b0082]nt!KxFlushEntireTb+0x95[/COLOR] (fffff800`052b3485) [COLOR=#ff8c00]<--- Looks like it takes the jump (jmp) to stay in a loop.[/COLOR]
fffff800`052b3473 ffc3            inc     ebx
fffff800`052b3475 851d4d1f2000    test    dword ptr [nt!HvlLongSpinCountMask (fffff800`054b53c8)],ebx
fffff800`052b347b 0f8479e1f5ff    je      nt! ?? ::FNODOBFM::`string'+0x7437 (fffff800`052115fa)
fffff800`052b3481 f390            [COLOR=#000080]pause[/COLOR] [COLOR=#ff8c00]<--- Executing a pause (a CPU delay), and doing this in a loop waiting for a release.[/COLOR]
fffff800`052b3483 ebe4            [COLOR=#ff0000]jmp[/COLOR]     [COLOR=#4b0082]nt!KxFlushEntireTb+0x79 (fffff800`052b3469)[/COLOR]
fffff800`052b3485 f08305f3e6140001 lock add dword ptr [nt!KiTbFlushTimeStamp (fffff800`05401b80)],1

Okay, so what's causing the loop?

Let's switch to Processor 4 (the problematic processor):

Code:

4: kd> kv
Child-SP          RetAddr           : Args to Child                                                           : Call Site
fffff880`0af79118 fffff800`057f4ae7 : fffffa80`13d1b5e0 fffffa80`0c7cec58 fffff800`05423288 00000000`00000000 : hal!HalpLegacyApicReadGenericReg+0xd
fffff880`0af79120 fffff800`05273e7c : 00000000`00000000 fffffa80`0c7cec58 fffffa80`0c7cec58 fffffa80`0c7ceb50 : hal!HalRequestSoftwareInterrupt+0x58
fffff880`0af79150 fffff800`05289454 : fffffa80`00000000 fffffa80`13671a80 fffffa80`13671a00 fffffa80`00000000 : nt!KiInsertQueue+0x27c
fffff880`0af791d0 fffff880`01b407c1 : fffffa80`13671a70 00000000`00000005 00000000`00000000 00000000`00000005 : nt!ExQueueWorkItem+0x44
fffff880`0af79210 fffff880`01abb193 : fffffa80`119651a0 fffffa80`11a9a630 fffffa80`11a9a630 fffffa80`11a9a630 : ndis!ndisQueueRequestWorkItem+0x161
fffff880`0af79270 fffff880`01b77495 : fffff880`00000201 fffffa80`13d1b5e0 00000000`00000000 00000000`000007ff : ndis!ndisQueueRequestOnTop+0x3a3
fffff880`0af79310 fffff880`01b77388 : fffff880`0af793a8 fffff880`0af79630 00000000`00000008 fffffa80`11a9fb30 : ndis!ndisLegacyRequest+0xf5
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for VBoxNetFlt.sys - 
fffff880`0af79340 fffff880`0670ce11 : fffff880`0af79630 00000000`00000008 fffff880`0af793d8 fffff880`0af793f0 : ndis!NdisRequest+0x18
fffff880`0af79370 fffff880`0670cece : 00000000`00020103 00000000`000000b0 fffffa80`11a9fb30 fffffa80`1196d1a0 : [COLOR=#ff0000]VBoxNetFlt+0x4e11[/COLOR]
fffff880`0af793a0 fffff880`01b4f53d : fffffa80`10e11c70 fffffa80`00000000 fffff880`0af79640 00000000`00000000 : [COLOR=#ff0000]VBoxNetFlt+0x4ece[/COLOR]
fffff880`0af793d0 fffff880`01b46c14 : fffffa80`1196d1a0 fffff880`0af79688 fffff880`0af79688 00000000`00000000 : ndis!ndisMOidRequestToRequest+0x26d
fffff880`0af79440 fffff880`01abb0c5 : 00000000`00000000 00000000`00000000 fffffa80`1196d1a0 fffff880`0af79600 : ndis! ?? ::DKGKHJNI::`string'+0x324d
fffff880`0af794f0 fffff880`01abe3f3 : fffffa80`13727f00 fffff880`0af79640 00000000`00000000 00000000`00000000 : ndis!ndisQueueRequestOnTop+0x2d5
fffff880`0af79590 fffff880`01ad861c : fffff880`00000200 fffffa80`13727f10 fffff880`01b1b110 00000000`00000000 : ndis!ndisQuerySetMiniportEx+0x143
fffff880`0af79600 fffff880`01abb758 : fffffa80`11b61010 00000000`c0000001 fffffa80`11b61010 00000000`c0000001 : ndis! ?? ::FNODOBFM::`string'+0xa249
fffff880`0af79770 fffff880`01abbcf9 : fffffa80`13cd6600 fffff880`01b1b110 fffff8a0`03d6c000 fffffa80`1196d1a0 : ndis!ndisFQueueRequestOnNext+0x138
fffff880`0af797e0 fffff880`03325231 : fffffa80`10c6e360 00000000`00020106 fffffa80`10c6e360 00000000`00000000 : ndis!NdisFOidRequest+0xc9
fffff880`0af798c0 fffff880`01b40582 : fffffa80`13727f10 fffffa80`11b61010 00000000`00020106 00000000`00000000 : nm3!NetmonOidRequest+0x109
fffff880`0af798f0 fffff880`01abb84d : 00000000`00000000 fffffa80`11b61010 00000000`00000000 00000000`00000001 : ndis!ndisFDoOidRequest+0x222
fffff880`0af799c0 fffff880`01abbcf9 : fffffa80`11b61001 fffff880`01b1b110 00000000`00000000 fffffa80`11b61010 : ndis!ndisFQueueRequestOnNext+0x22d
fffff880`0af79a30 fffff880`032ff11b : fffffa80`1289acf0 00000000`00020106 fffffa80`1289acf0 fffffa80`11abb4e0 : ndis!NdisFOidRequest+0xc9
fffff880`0af79b10 fffff880`01b40582 : fffffa80`10c6e360 fffffa80`11abc4e0 00000000`00000000 fffffa80`11abc4e0 : pacer!PcFilterRequest+0x5b
fffff880`0af79b40 fffff880`01abb84d : 00000000`00000000 fffffa80`11abc4e0 00000000`00000000 00000000`00000001 : ndis!ndisFDoOidRequest+0x222
fffff880`0af79c10 fffff880`01abbcf9 : fffffa80`11abc501 fffff880`01b1b110 00000000`00020200 fffffa80`11abc4e0 : ndis!ndisFQueueRequestOnNext+0x22d
fffff880`0af79c80 fffff880`032f6625 : fffff880`0af7a060 00000000`0002020a fffff880`0af7a060 fffffa80`11abd490 : ndis!NdisFOidRequest+0xc9
fffff880`0af79d60 fffff880`01b40582 : fffffa80`1289acf0 fffffa80`11abfc80 00000000`00000000 fffffa80`11abfc80 : wfplwf!FilterOidRequest+0x61
fffff880`0af79d90 fffff880`01abb009 : fffffa80`00000000 fffffa80`11abfc80 fffffa80`1196d1a0 00000000`00000001 : ndis!ndisFDoOidRequest+0x222
fffff880`0af79e60 fffff880`01abe3f3 : fffffa80`1196d101 fffff880`0af7a060 fffffa80`11abfc80 fffffa80`11950801 : ndis!ndisQueueRequestOnTop+0x219
fffff880`0af79f00 fffff880`01abe7a7 : fffffa80`13e62000 fffffa80`1196d1a0 fffffa80`11abfc80 fffffa80`13e62090 : ndis!ndisQuerySetMiniportEx+0x143
fffff880`0af79f70 fffff880`01abd019 : 0072006f`00770074 006f0051`002d006b 00610050`00200053 fffff880`0af7a3e8 : ndis!ndisIfGetMiniportStatistics+0x2a8
fffff880`0af7a1a0 fffff880`01abd417 : fffffa80`11815ed8 00000000`0000ffff 00060000`0d000000 fffff880`01abdea9 : ndis!ndisIfQueryObject+0x389
fffff880`0af7a330 fffff880`01abc8bd : fffffa80`0d64c870 fffffa80`11ac04f0 fffff880`0af7a410 00060000`00000000 : ndis!ndisNsiGetInterfaceRodInformation+0x207
fffff880`0af7a3d0 fffff880`01a03f16 : fffff880`01b1ca50 fffffa80`13e61001 00000000`00000001 fffffa80`115f8500 : ndis!ndisNsiGetAllInterfaceInformation+0x42e
fffff880`0af7a490 fffff880`0595fe29 : fffffa80`13e61000 fffff8a0`00000070 fffffa80`12b923c0 00000000`0028f5c0 : NETIO!NsiEnumerateObjectsAllParametersEx+0x6df
fffff880`0af7a670 fffff880`059618e8 : fffffa80`12b923c0 fffffa80`12b922f0 00000000`00000000 fffffa80`12b92328 : nsiproxy!NsippEnumerateObjectsAllParameters+0x305
fffff880`0af7a860 fffff880`059619db : fffffa80`117157f0 00000000`00000000 00000000`00000001 00000000`00000003 : nsiproxy!NsippDispatchDeviceControl+0x70
fffff880`0af7a8a0 fffff800`0559a3a7 : fffffa80`11e55bc0 fffffa80`11e55bc0 fffffa80`12b92408 fffffa80`12b922f0 : nsiproxy!NsippDispatch+0x4b
fffff880`0af7a8d0 fffff800`0559ac06 : 00000000`0028f440 00000000`0000089c 00000000`00000000 00000000`00000000 : nt!IopXxxControlFile+0x607
fffff880`0af7aa00 fffff800`0527ce53 : fffffa80`11db4b50 00000000`0028f428 fffff880`0af7aa88 00000000`00000001 : nt!NtDeviceIoControlFile+0x56
fffff880`0af7aa70 00000000`77c6132a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`0af7aae0)
00000000`0028f4b8 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x77c6132a

^^ Looks like VirtualBox's network driver isn't playing nice.

Having a look through your modules list shows a bunch of different VM's (Sandboxie, VMware, VirtualBox, etc). Why do you have so many? It could very well be that regardless of whether or not one is running at a time, given the different VM's have installed their own network drivers, etc, they may be conflicting. With that said, uninstall all VM's minus one, and stick to that one. FWIW, I would recommend not sticking to VirtualBox, and using one of the others full-time if possible.

Regards,

Patrick

DotFooz · Apr 19, 2014

Well Sandboxie was used for security means to test applications to see if they were infected. VirtualBox was a free alternative to running Virtual machines until I saved the money to buy VMware Workstation. Anyways I'll report back soon with any crash reports if they occur. Thanks for the help so far!

Patrick · Apr 19, 2014

Understood, thanks for the information. I look forward to your update.

Regards,

Patrick

Tekno Venus · Apr 19, 2014

Patrick said:
It could very well be that regardless of whether or not one is running at a time, given the different VM's have installed their own network drivers, etc, they may be conflicting. With that said, uninstall all VM's minus one, and stick to that one. FWIW, I would recommend not sticking to VirtualBox, and using one of the others full-time if possible.

Hi Patrick,

Sorry to jump in here. May I ask why you don't recommend sticking to VirtualBox? I use VM's regularly for sourcing files for Windows Update threads, and a combination of VMWare Player, VirtualBox and sometimes Hyper-V, although I cannot use VMWare/VirtualBox when Hyper-V is turned on due to issues.

Have you had bad experiences with VirtualBox? It's always worked fine for me.

I have both VMware and VirtualBox network adapters configured:

and have no issues.

I'm not questioning your debugging, just wondering!

@DotFooz - Have you tried VMWare player? It is built upon VMWare workstation, but has less features. May be a good alternative depending on your needs?

Thanks,
Stephen

Patrick · Apr 19, 2014

I have no issues whatsoever with VBox, I should have changed my wording. What I meant to say was, it may be a good idea to remove VBox (given it's mentioned throughout the stack) + all VM's but one, and stick to that for awhile. If no crashes, presume that there was a VM conflict. At that point, you could go ahead and reinstall VBox if you'd like, however, you should then uninstall whatever other VM you are using.

Regards,

Patrick

Tekno Venus · Apr 19, 2014

Understood - thanks Patrick! :)

Patrick · Apr 19, 2014

Not a problem, thanks for actually bringing that up! I didn't want to make it seem as if I had a secret hatred for Virtual Box :grin1:

DotFooz · Apr 26, 2014

Trouble in paradise once more! I think it might be Comodo Firewall. A relative installed it onto my computer for some reason. I uninstalled it, but it started crashing after it was installed the second time.
Link to MEMORY.DMP: https://www.dropbox.com/s/c50byfbzqkhd1qz/MEMORY.DMP

Patrick · Apr 26, 2014

Yes, please remove COMODO and then keep me updated. I would have recommended its removal as soon as I saw it, but alas, I missed it.

Regards,

Patrick

BSOD secondary processor clock interrupt - Windows 7 x64

Member

Sysnative Staff

Member

Sysnative Staff

Member

Sysnative Staff

New member

Sysnative Staff

Member

Sysnative Staff

Member

Sysnative Staff

Member

Sysnative Staff

Senior Administrator, Developer

Sysnative Staff

Senior Administrator, Developer

Sysnative Staff

Member

Sysnative Staff