BSOD Win 7 64 - Clock Interrupt 2nd processor

MiloBloom · Dec 13, 2014

Good Evening

I started getting BSOD errors the other day and am looking for any help or direction. Here is the mini dump
Windows 7 Kernel Version 7601 (Service Pack 1) MP (8 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 7601.18409.amd64fre.win7sp1_gdr.140303-2144
Machine Name:
Kernel base = 0xfffff800`02e4d000 PsLoadedModuleList = 0xfffff800`03090890

CLOCK_WATCHDOG_TIMEOUT (101)
An expected clock interrupt was not received on a secondary processor in an
MP system within the allocated interval. This indicates that the specified
processor is hung and not processing interrupts.
Arguments:
Arg1: 0000000000000019, Clock interrupt time out interval in nominal clock ticks.
Arg2: 0000000000000000, 0.
Arg3: fffff880031e6180, The PRCB address of the hung processor.
Arg4: 0000000000000004, 0.

Debugging Details:
------------------

BUGCHECK_STR: CLOCK_WATCHDOG_TIMEOUT_8_PROC

CUSTOMER_CRASH_COUNT: 1

DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT

PROCESS_NAME: svchost.exe

CURRENT_IRQL: d

ANALYSIS_VERSION: 6.3.9600.17298 (debuggers(dbg).141024-1500) amd64fre

STACK_TEXT:
fffff880`0bc10118 fffff800`02f19a4a : 00000000`00000101 00000000`00000019 00000000`00000000 fffff880`031e6180 : nt!KeBugCheckEx
fffff880`0bc10120 fffff800`02ecc6f7 : 00000000`00000000 fffff800`00000004 00000000`00002710 00000000`00000000 : nt! ?? ::FNODOBFM::`string'+0x4e3e
fffff880`0bc101b0 fffff800`02e0e895 : fffff800`02e34460 fffff880`0bc10360 fffff800`02e34460 fffff800`00000000 : nt!KeUpdateSystemTime+0x377
fffff880`0bc102b0 fffff800`02ebf113 : fffff880`0bc10360 fffff880`0bc10301 fffffa80`00000001 00000000`00000000 : hal!HalpHpetClockInterrupt+0x8d
fffff880`0bc102e0 fffff800`02ee230a : 00000000`00000000 fffff8a0`00024010 00000000`00000001 fffff800`03125e1e : nt!KiInterruptDispatchNoLock+0x163
fffff880`0bc10470 fffff800`02ee4c98 : 00000000`00000008 fffff880`0bc105c0 fffff8a0`1025e000 00000000`00000001 : nt!MiFlushTbAsNeeded+0x28a
fffff880`0bc10580 fffff800`02ff3f86 : 00000000`00008000 fffffa80`0699a000 00000000`00000009 fffff800`02ec5a8a : nt!MiAllocatePagedPoolPages+0x4cc
fffff880`0bc106a0 fffff800`02ee29b0 : 00000000`00008000 fffffa80`0699a000 00000000`00000009 20206553`02eb85f2 : nt!MiAllocatePoolPages+0x906
fffff880`0bc107e0 fffff800`02ff743e : 00000000`00000000 fffff880`09043220 00000000`00000000 00000000`00008000 : nt!ExpAllocateBigPool+0xb0
fffff880`0bc108d0 fffff800`02ed5f56 : 00000000`00000000 00000000`00000009 fffff8a0`05178060 fffff800`031aec5f : nt!ExAllocatePoolWithTag+0x82e
fffff880`0bc109c0 fffff800`0312ef86 : 00000000`00000000 00000000`00008000 00000000`00000000 00000000`00000001 : nt!ExAllocatePoolWithQuotaTag+0x56
fffff880`0bc10a10 fffff800`03186b94 : fffff8a0`0f3fde00 fffff800`00008000 fffff880`0bc10b01 fffff800`03390da0 : nt!PiControlGetInterfaceDeviceList+0x92
fffff880`0bc10a90 fffff800`02ec1e53 : fffffa80`0a499b50 00000000`0059ebc0 fffff880`0bc10b60 00000000`0059ec48 : nt!NtPlugPlayControl+0x100
fffff880`0bc10ae0 00000000`77b5230a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
00000000`0059eb88 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x77b5230a

STACK_COMMAND: kb

SYMBOL_NAME: ANALYSIS_INCONCLUSIVE

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: Unknown_Module

IMAGE_NAME: Unknown_Image

DEBUG_FLR_IMAGE_TIMESTAMP: 0

IMAGE_VERSION:

FAILURE_BUCKET_ID: X64_CLOCK_WATCHDOG_TIMEOUT_8_PROC_ANALYSIS_INCONCLUSIVE

BUCKET_ID: X64_CLOCK_WATCHDOG_TIMEOUT_8_PROC_ANALYSIS_INCONCLUSIVE

ANALYSIS_SOURCE: KM

FAILURE_ID_HASH_STRING: km:x64_clock_watchdog_timeout_8_proc_analysis_inconclusive

FAILURE_ID_HASH: {b37a9ee0-e89a-8796-d6e1-c509c7ba4f05}

I have the latest drivers and bios loaded, I have run memory scan, checked for heat issues, reloaded defaults in the bios.

Here is a link to the full memory dump if it will help
https://onedrive.live.com/?cid=59465301639b73a7&id=59465301639B73A7!106

Any help will be hugely appreciated

thanks in advance

Jared · Dec 14, 2014

A 0x101 clock interrupt indicates one or more processors failed to complete the clock interrupt within the given time interval.

Code:

CLOCK_WATCHDOG_TIMEOUT (101)
An expected clock interrupt was not received on a secondary processor in an
MP system within the allocated interval. This indicates that the specified
processor is hung and not processing interrupts.
Arguments:
Arg1: 0000000000000019, Clock interrupt time out interval in nominal clock ticks.
Arg2: 0000000000000000, 0.
Arg3: fffff880031e6180, The PRCB address of the hung processor.
Arg4: 0000000000000004, 0.

Debugging Details:
------------------


BUGCHECK_STR:  CLOCK_WATCHDOG_TIMEOUT_8_PROC

DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT

PROCESS_NAME:  svchost.exe

CURRENT_IRQL:  d

ANALYSIS_VERSION: 6.3.9600.17029 (debuggers(dbg).140219-1702) amd64fre

STACK_TEXT:  
fffff880`0bc10118 fffff800`02f19a4a : 00000000`00000101 00000000`00000019 00000000`00000000 fffff880`031e6180 : nt!KeBugCheckEx
fffff880`0bc10120 fffff800`02ecc6f7 : 00000000`00000000 fffff800`00000004 00000000`00002710 00000000`00000000 : nt! ?? ::FNODOBFM::`string'+0x4e3e
fffff880`0bc101b0 fffff800`02e0e895 : fffff800`02e34460 fffff880`0bc10360 fffff800`02e34460 fffff800`00000000 : nt!KeUpdateSystemTime+0x377
fffff880`0bc102b0 fffff800`02ebf113 : fffff880`0bc10360 fffff880`0bc10301 fffffa80`00000001 00000000`00000000 : hal!HalpHpetClockInterrupt+0x8d
fffff880`0bc102e0 fffff800`02ee230a : 00000000`00000000 fffff8a0`00024010 00000000`00000001 fffff800`03125e1e : nt!KiInterruptDispatchNoLock+0x163
fffff880`0bc10470 fffff800`02ee4c98 : 00000000`00000008 fffff880`0bc105c0 fffff8a0`1025e000 00000000`00000001 : nt!MiFlushTbAsNeeded+0x28a
fffff880`0bc10580 fffff800`02ff3f86 : 00000000`00008000 fffffa80`0699a000 00000000`00000009 fffff800`02ec5a8a : nt!MiAllocatePagedPoolPages+0x4cc
fffff880`0bc106a0 fffff800`02ee29b0 : 00000000`00008000 fffffa80`0699a000 00000000`00000009 20206553`02eb85f2 : nt!MiAllocatePoolPages+0x906
fffff880`0bc107e0 fffff800`02ff743e : 00000000`00000000 fffff880`09043220 00000000`00000000 00000000`00008000 : nt!ExpAllocateBigPool+0xb0
fffff880`0bc108d0 fffff800`02ed5f56 : 00000000`00000000 00000000`00000009 fffff8a0`05178060 fffff800`031aec5f : nt!ExAllocatePoolWithTag+0x82e
fffff880`0bc109c0 fffff800`0312ef86 : 00000000`00000000 00000000`00008000 00000000`00000000 00000000`00000001 : nt!ExAllocatePoolWithQuotaTag+0x56
fffff880`0bc10a10 fffff800`03186b94 : fffff8a0`0f3fde00 fffff800`00008000 fffff880`0bc10b01 fffff800`03390da0 : nt!PiControlGetInterfaceDeviceList+0x92
fffff880`0bc10a90 fffff800`02ec1e53 : fffffa80`0a499b50 00000000`0059ebc0 fffff880`0bc10b60 00000000`0059ec48 : nt!NtPlugPlayControl+0x100
fffff880`0bc10ae0 00000000`77b5230a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
00000000`0059eb88 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x77b5230a

So, we're allocating pages for a process, we suddenly have to flush the translation lookaside buffers as we hit a page fault, it would seem it happened somewhere along the 1st processor's execution. To flush the TLB, all processors have to respond; to do this, a clock interrupt is required to synchronise all processors, so we send an IPI to update the system time for all the processors, and begin the TLB flushing.
Alright, we know what happened with the first processor, but why did we crash? Lets take a look.

Code:

0: kd> !running -it

System Processors:  (00000000000000ff)
  Idle Processors:  (0000000000000000) (0000000000000000) (0000000000000000) (0000000000000000)

       Prcbs             Current         (pri) Next            (pri) Idle
  0    fffff8000303de80  fffffa800a499b50 ( 8)                       fffff8000304bcc0  ................

Child-SP          RetAddr           Call Site
fffff880`0bc10118 fffff800`02f19a4a nt!KeBugCheckEx
fffff880`0bc10120 fffff800`02ecc6f7 nt! ?? ::FNODOBFM::`string'+0x4e3e
fffff880`0bc101b0 fffff800`02e0e895 nt!KeUpdateSystemTime+0x377
fffff880`0bc102b0 fffff800`02ebf113 hal!HalpHpetClockInterrupt+0x8d
fffff880`0bc102e0 fffff800`02ee230a nt!KiInterruptDispatchNoLock+0x163
fffff880`0bc10470 fffff800`02ee4c98 nt!MiFlushTbAsNeeded+0x28a
fffff880`0bc10580 fffff800`02ff3f86 nt!MiAllocatePagedPoolPages+0x4cc
fffff880`0bc106a0 fffff800`02ee29b0 nt!MiAllocatePoolPages+0x906
fffff880`0bc107e0 fffff800`02ff743e nt!ExpAllocateBigPool+0xb0
fffff880`0bc108d0 fffff800`02ed5f56 nt!ExAllocatePoolWithTag+0x82e
fffff880`0bc109c0 fffff800`0312ef86 nt!ExAllocatePoolWithQuotaTag+0x56
fffff880`0bc10a10 fffff800`03186b94 nt!PiControlGetInterfaceDeviceList+0x92
fffff880`0bc10a90 fffff800`02ec1e53 nt!NtPlugPlayControl+0x100
fffff880`0bc10ae0 00000000`77b5230a nt!KiSystemServiceCopyEnd+0x13
00000000`0059eb88 00000000`00000000 0x77b5230a

Lets look at the actual instruction being executed.

Code:

0: kd> u @rip
nt!MiFlushTbAsNeeded+0x28a:
fffff800`02ee230a 72e4            jb      nt!MiFlushTbAsNeeded+0x270 (fffff800`02ee22f0)
fffff800`02ee230c e945ffffff      jmp     nt!MiFlushTbAsNeeded+0x1d6 (fffff800`02ee2256)
fffff800`02ee2311 90              nop
fffff800`02ee2312 90              nop
fffff800`02ee2313 90              nop
fffff800`02ee2314 90              nop
fffff800`02ee2315 90              nop
fffff800`02ee2316 90              nop

So we're trying to flush the TLB, we keep jumping trying to execute the instruction but it's not happening, why?

Code:

0: kd> u fffff800`02ee22f0 fffff800`02ee230c
nt!MiFlushTbAsNeeded+0x270:
fffff800`02ee22f0 41ffc6          inc     r14d
fffff800`02ee22f3 448535ce802100  test    dword ptr [nt!HvlLongSpinCountMask (fffff800`030fa3c8)],r14d
fffff800`02ee22fa 0f840d980300    je      nt! ?? ::FNODOBFM::`string'+0xaefd (fffff800`02f1bb0d)
fffff800`02ee2300 f390            pause
fffff800`02ee2302 0fba257648160000 bt      dword ptr [nt!KiTbFlushTimeStamp (fffff800`03046b80)],0
fffff800`02ee230a 72e4            jb      nt!MiFlushTbAsNeeded+0x270 (fffff800`02ee22f0)
fffff800`02ee230c e945ffffff      jmp     nt!MiFlushTbAsNeeded+0x1d6 (fffff800`02ee2256)

We're executing a CPU delay, it seems to be looping, trying to flush the TLB for all processors, it doesn't work so it keeps trying until we crash.

Code:

  1    fffff880009c0180  fffffa80098eeb50 (27) fffffa8006e04770 (30) fffff880009cb0c0  ................

Child-SP          RetAddr           Call Site
fffff880`0a070680 fffff800`031ce80f nt!KeFlushProcessWriteBuffers+0x6b
fffff880`0a0706f0 fffff800`031cee6d nt!ExpQuerySystemInformation+0x13af
fffff880`0a070aa0 fffff800`02ec1e53 nt!NtQuerySystemInformation+0x4d
fffff880`0a070ae0 00000000`77b5161a nt!KiSystemServiceCopyEnd+0x13
00000000`00f1f4f8 00000000`00000000 0x77b5161a

  2    fffff88003142180  fffffa8006a35040 (16)                       fffff8800314d0c0  ................

Child-SP          RetAddr           Call Site
fffff880`03732790 fffff800`02ec7a1c nt!KxFlushEntireTb+0x93
fffff880`037327d0 fffff800`02ef8963 nt!KeFlushMultipleRangeTb+0x28c
fffff880`037328a0 fffff800`02ec7c34 nt!MiReplenishBitMap+0x468
fffff880`037329b0 fffff800`02ead46f nt!MiEmptyPteBins+0x10d
fffff880`03732a00 fffff800`02eacad8 nt!MiAdjustPteBins+0xbf
fffff880`03732a40 fffff800`02eacfb3 nt!MmWorkingSetManager+0x40
fffff880`03732a90 fffff800`0315e73a nt!KeBalanceSetManager+0x1c3
fffff880`03732c00 fffff800`02eb38e6 nt!PspSystemThreadStartup+0x5a
fffff880`03732c40 00000000`00000000 nt!KxStartSystemThread+0x16

//It seems to be that the TLB needs to be flushed, most likely due to alterations in the page table that is associated with multiple processors. As a result, the entire TLB needs to be flushed to remove the invalid pages from all the TLBs of each processor.

  3    fffff880031b4180  fffffa8006a7ab50 (13) fffffa8009c79060 (15) fffff880031bf0c0  ................

Child-SP          RetAddr           Call Site
fffff880`0354d170 fffff800`02f0695c nt!KeFlushMultipleRangeTb+0x258
fffff880`0354d240 fffff800`02f03751 nt!MmSetAddressRangeModified+0x2b0
fffff880`0354d340 fffff880`015181b0 nt!CcFlushCache+0x561
fffff880`0354d440 fffff880`0151b54e Ntfs!NtfsCheckpointVolume+0xbe1
fffff880`0354d850 fffff880`01519e12 Ntfs!NtfsCheckpointAllVolumesWorker+0x4e
fffff880`0354d8a0 fffff880`0151c02e Ntfs!NtfsForEachVcb+0x172
fffff880`0354d940 fffff800`02ecc261 Ntfs!NtfsCheckpointAllVolumes+0xbe
fffff880`0354db70 fffff800`0315e73a nt!ExpWorkerThread+0x111
fffff880`0354dc00 fffff800`02eb38e6 nt!PspSystemThreadStartup+0x5a
fffff880`0354dc40 00000000`00000000 nt!KxStartSystemThread+0x16

//I believe flushing multiple range tb means flushing multiple different TLBs among different processors, which is what we're trying to do.

  4    fffff880031e6180  fffffa8009f37060 ( 8) fffffa8008c30360 (22) fffff880031f10c0  ................

Child-SP          RetAddr           Call Site
00000000`00000000 00000000`00000000 0x0 //Null call stack and registers.

  5    fffff880032a5180  fffffa800979c060 ( 1) fffffa8006ef0060 (22) fffff880032b00c0  ................

Child-SP          RetAddr           Call Site
00000000`00000000 00000000`00000000 0x0 //Null call stack and registers.

  6    fffff88003317180  fffffa800ab30b50 ( 1)                       fffff880033220c0  ................

Child-SP          RetAddr           Call Site
00000000`040ce5f0 00000000`00000000 0x00000001`41b39bb2

  7    fffff88003389180  fffffa80074e0b50 ( 1)                       fffff880033940c0  ................

Child-SP          RetAddr           Call Site
00000000`041ce620 00000000`00000000 0x00000001`41bdf13c

So now we've looked at the callstack from each processor, we can see what's going on. The cause is related to processor #4 and #5, but why? What's going on?
Let's look at the Inter-Processor Interrupts active.

Code:

0: kd> !ipi
IPI State for Processor 0
    TargetCount          0  PacketBarrier        0  IpiFrozen     0 [Running] //Active and ready


IPI State for Processor 1

    As a sender, awaiting IPI completion from processor(s) 4, 5.

    TargetCount          2  PacketBarrier        1  IpiFrozen     2 [Frozen] //Idle, waiting for IPI completion



IPI State for Processor 2

    As a sender, awaiting IPI completion from processor(s) 4, 5.

    TargetCount          2  PacketBarrier        1  IpiFrozen     2 [Frozen] //Idle, waiting for IPI completion



IPI State for Processor 3

    As a sender, awaiting IPI completion from processor(s) 4, 5.

    TargetCount          2  PacketBarrier        1  IpiFrozen     2 [Frozen] //Idle, waiting for IPI completion


IPI State for Processor 4

    As a receiver, unhandled requests are pending from processor(s) 1, 2, 3.

    TargetCount          0  PacketBarrier        0  IpiFrozen     5 [Target Freeze]

    From processor 1, active request of type: packet ready
        WorkerRoutine fffff80002eb3e10 (nt!xHalReportIdleStateUsage) //IPI from Processor #1 asking to report idle
        Parameter[0] 0  Parameter[1] 0  Parameter[2] 0
    From processor 2, active request of type: flush all //IPI from Processor #2 to flush its TLB
    From processor 3, active request of type: flush multiple range //IPI from Processor #3 to flush its TLB
        Flush Count 1  Flush List fffff8800354d268  (dp fffff8800354d268 l1)

IPI State for Processor 5

    As a receiver, unhandled requests are pending from processor(s) 1, 2, 3.

    TargetCount          0  PacketBarrier        0  IpiFrozen     5 [Target Freeze]

    From processor 1, active request of type: packet ready
        WorkerRoutine fffff80002eb3e10 (nt!xHalReportIdleStateUsage) //IPI from Processor #1 asking to report idle
        Parameter[0] 0  Parameter[1] 0  Parameter[2] 0
    From processor 2, active request of type: flush all //IPI from Processor #2 to flush its TLB
    From processor 3, active request of type: flush multiple range //IPI from Processor #3 to flush its TLB
        Flush Count 1  Flush List fffff8800354d268  (dp fffff8800354d268 l1)

IPI State for Processor 6
    TargetCount          0  PacketBarrier        0  IpiFrozen     2 [Frozen] //Idle


IPI State for Processor 7
    TargetCount          0  PacketBarrier        0  IpiFrozen     2 [Frozen] //Idle

With all of this in mind, I'm almost certain you've got yourself a bad CPU, two processor's are null with nothing in them, failing to respond to 3 IPIs from different processors.
Is your CPU still under warranty?
You don't appear to be overclocking, so that's another sign that it's failing.
Is your CPU overheating?

Jared · Dec 14, 2014

We could try some software alterations first, as they might be the cause.
I notice a lot of routines ointhe 4th processor's raw stack, related to your Nvidia graphics driver and Norton.
Can you try updating/rolling back your display driver.
Also, remove Norton 360 and replace it with MSE, see if the problems still persist, if they do upload the new Kernel memory dump and I'll take a look.

Norton removal tool: https://support.norton.com/sp/en/uk/home/current/solutions/kb20080710133834EN_EndUserProfile_en_us

MSE: Microsoft Security Essentials - Microsoft Windows

MiloBloom · Dec 14, 2014

Wow thanks you guys are really fast. I removed all MS system updates from 12/9 and have not had a crash since. Probably a coincidence but who knows. My nvida drivers have remained the same since 11/18 I could uninstall and reinstall if it happens again. As far as norton goes well its always a possibility that one of the updates did something but i havent seen the cpu load or memory usage jump. Ill keep my eye on it. I am really apprehensive about running a system with MSE only on it. Even MS own internal documentation recommends you run it in conjunction with a 3rd party app. If I crash again I will definitely try it though. I recorded the CPU temp for 30 mins last night while gaming and it ranged from 50-63c which appears to be ok for this chip but I am not sure. Here is my question, this is a new system, around 2 months old. If I have a flaky cpu I want to know asap because I might be able to get an rma. So I am guessing some kind of stress test would be the right thing to do. Any suggestion for a program and how to do it?

Again the fast response and analysis is just incredible, thanks

Jared · Dec 14, 2014

MSE on it's own isn't advnaced protection but it's enough if you know what you're doing. I recommend it mainly for testing purposes to see if the problems still persist without Norton.
The actual usage doesn't matter, although it does usually happen more with a higher usage, it doesn't necessarily mean it will.
60C+ is high for a CPU, even under load, I'd try to find a way to bring those temps down, have yoiu dusted the components? Is there enough air flow? Have you replaced the thermal paste with a better one? Is the heat sink good enough for the CPU?
I don't recommend any stress tests for CPUs, they're aren't reliable, if your CPU is under warranty then I would seriously try to RMA as the evidence in the dump file shows the CPU is the culprit.

If you get any other crashes, post th elink to the Kernel memory dump and I'll check it. It should help us narrow it down especially if the results are consistent.

Search

Search

BSOD Win 7 64 - Clock Interrupt 2nd processor

MiloBloom

New member

Jared

Sysnative Staff, BSOD Kernel Dump Expert

Jared

Sysnative Staff, BSOD Kernel Dump Expert

MiloBloom

New member

Jared

Sysnative Staff, BSOD Kernel Dump Expert