Thanks!
CLOCK_WATCHDOG_TIMEOUT (101)
This indicates that an expected clock interrupt on a secondary processor, in a multi-processor system, was not received within the allocated interval.
Code:
BugCheck 101, {19, 0, fffff880009ef180, 1}
^^ 19 clock ticks in regards to the timeout.
fffff880009ef180 is the PRCB address of the hung processor, let's keep this address in mind.
Code:
0: kd> !prcb 1
PRCB for Processor 1 at fffff880009ef180:
Current IRQL -- 0
Threads-- Current fffffa80116c4770 Next 0000000000000000 Idle fffff880009fa0c0
Processor Index 1 Number (0, 1) GroupSetMember 2
Interrupt Count -- 02239c56
Times -- Dpc 00000022 Interrupt 00000296
Kernel 000a41ed User 0001416e
As this matches the 3rd parameter of the bug check, processor #1 is the responsible processor. Now with the information we have here thus far, we know that processor #1 reached 19 clock ticks without responding, therefore the system crashed. Before we go further, what is a clock tick? A clock interrupt is a form of interrupt which involves counting the the cycles of the processor core, which is running a clock on the processors to keep them all in sync. A clock interrupt is handed out to all processors and then they must report in, and when one doesn't report in, you then crash.
Code:
0: kd> kv
Child-SP RetAddr : Args to Child : Call Site
fffff800`00b9c0c8 fffff800`0451ca4a : 00000000`00000101 00000000`00000019 00000000`00000000 fffff880`009ef180 : nt!KeBugCheckEx
fffff800`00b9c0d0 fffff800`044cf6f7 : 0000057f`00000000 fffff800`00000001 00000000`00002710 fffff880`00f36cec : nt! ?? ::FNODOBFM::`string'+0x4e3e
fffff800`00b9c160 fffff800`04411895 : fffff800`04437460 fffff800`00b9c310 fffff800`04437460 fffffa80`00000000 : nt!KeUpdateSystemTime+0x377
fffff800`00b9c260 fffff800`044c2113 : fffff880`03890180 fffff800`00b9c301 fffffa80`0c12ef00 fffffa80`14b22b50 : hal!HalpHpetClockInterrupt+0x8d
fffff800`00b9c290 fffff800`044cb5ae : fffffa80`0d245270 fffffa80`0d471c50 fffff800`00b9c460 fffffa80`0cfda190 : nt!KiInterruptDispatchNoLock+0x163 (TrapFrame @ fffff800`00b9c290)
fffff800`00b9c420 fffff800`044d0787 : fffffa80`15412c10 fffffa80`14b22c58 fffffa80`14b22c58 fffffa80`1541d100 : nt!KiDeferredReadyThread+0x6a7
fffff800`00b9c4a0 fffff800`044d05de : 0000001b`68142479 fffff800`00b9cb18 00000000`000b8374 fffff800`04644108 : nt!KiProcessExpiredTimerList+0x157
fffff800`00b9caf0 fffff800`044d03c7 : 00000009`17658fc8 00000009`000b8374 00000009`17658fee 00000000`00000074 : nt!KiTimerExpiration+0x1be
fffff800`00b9cb90 fffff800`044bd8ca : fffff800`04640e80 fffff800`0464ecc0 00000000`00000000 fffff880`00000000 : nt!KiRetireDpcList+0x277
fffff800`00b9cc40 00000000`00000000 : fffff800`00b9d000 fffff800`00b97000 fffff800`00b9cc00 00000000`00000000 : nt!KiIdleLoop+0x5a
Code:
0: kd> u @rip
nt!KiDeferredReadyThread+0x6a7:
fffff800`044cb5ae 498b4530 mov rax,qword ptr [r13+30h]
fffff800`044cb5b2 4885c0 test rax,rax
fffff800`044cb5b5 75e5 jne nt!KiDeferredReadyThread+0x695 (fffff800`044cb59c)
fffff800`044cb5b7 f0490fba6d3000 lock bts qword ptr [r13+30h],0
fffff800`044cb5be 72dc jb nt!KiDeferredReadyThread+0x695 (fffff800`044cb59c)
fffff800`044cb5c0 4c8d15394af8ff lea r10,[nt!KiSelectNextThread <PERF> (nt+0x0) (fffff800`04450000)]
fffff800`044cb5c7 e97dfbffff jmp nt!KiDeferredReadyThread+0x239 (fffff800`044cb149)
fffff800`044cb5cc 440fb6477b movzx r8d,byte ptr [rdi+7Bh]
Code:
0: kd> u fffff800`044cb5b5 fffff800`044cb5cc
nt!KiDeferredReadyThread+0x6ae:
fffff800`044cb5b5 75e5 jne nt!KiDeferredReadyThread+0x695 (fffff800`044cb59c)
fffff800`044cb5b7 f0490fba6d3000 lock bts qword ptr [r13+30h],0
fffff800`044cb5be 72dc jb nt!KiDeferredReadyThread+0x695 (fffff800`044cb59c)
fffff800`044cb5c0 4c8d15394af8ff lea r10,[nt!KiSelectNextThread <PERF> (nt+0x0) (fffff800`04450000)]
fffff800`044cb5c7 e97dfbffff jmp nt!KiDeferredReadyThread+0x239 (fffff800`044cb149)
fffff800`044cb5cc 440fb6477b movzx r8d,byte ptr [rdi+7Bh]
It looks like
nt!KiDeferredReadyThread+0x695 is in a tight loop waiting for a lock to likely release. In fact, save a lot of posting space, all of the cores
except #1 are the same. Example:
Code:
2: kd> kv
Child-SP RetAddr : Args to Child : Call Site
fffff880`03794790 fffff800`044d0787 : fffffa80`1502f120 fffffa80`1502f168 fffffa80`1502f168 fffffa80`13848100 : nt!KiDeferredReadyThread+0x6ab
fffff880`03794810 fffff800`044d05de : 0000001b`68142479 fffff880`03794e88 00000000`000b8374 fffff880`03768408 : nt!KiProcessExpiredTimerList+0x157
fffff880`03794e60 fffff800`044d03c7 : fffff880`037651c3 fffffa80`000b8374 fffffa80`150f8bc0 00000000`00000074 : nt!KiTimerExpiration+0x1be
fffff880`03794f00 fffff800`044c8d15 : 00000000`00000000 fffffa80`1513e060 00000000`00000000 fffff800`045e6a90 : nt!KiRetireDpcList+0x277
fffff880`03794fb0 fffff800`044c8b2c : fffff880`03765180 fffff800`044d11b8 00000000`bb046778 00000000`fff35000 : nt!KxRetireDpcList+0x5 (TrapFrame @ fffff880`03794e70)
fffff880`0e5d7aa0 fffff800`04510b53 : fffff800`044d0f4c fffff800`044d0fb8 00000000`0eaf0034 fffff880`0e5d7b60 : nt!KiDispatchInterruptContinue
fffff880`0e5d7ad0 fffff800`044d0fb8 : 00000000`0eaf0034 fffff880`0e5d7b60 00000000`00000018 00000000`00000080 : nt!KiDpcInterruptBypass+0x13
fffff880`0e5d7ae0 00000000`b7543cbb : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSecondaryClockInterrupt+0x1a8 (TrapFrame @ fffff880`0e5d7ae0)
00000000`0e260a64 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0xb7543cbb
On processor #2, we can see we have a timer expiration and then hang on
nt!KiDeferredReadyThread+0x6ab.
From here we can go ahead and run !timer to view all of the DPCs that were pending:
We had 572 DPCs pending,
very bad!
Code:
145 fffff80004676140 P 6f7dad9c 0000001b [ 7/ 6/2014 18:04:46.682] nt!PopCheckForIdleness (DPC @ fffff80004676100)
146 fffffa800c3008b0 685c29e9 0000001b [ 7/ 6/2014 18:04:34.719] aswNdisFlt (DPC @ fffffa800c3008f0)+b830
fffff88004c85cb8 P 685c5ac0 0000001b [ 7/ 6/2014 18:04:34.720] tunnel!TunnelTimeoutRoutine (DPC @ fffff88004c85cf8)
fffff88004b17760 685c5ac0 0000001b [ 7/ 6/2014 18:04:34.720] afd!AfdTimerWheelHandler (DPC @ fffff88004b17720)
149 fffffa800d4b0278 431748ae 00000131 [ 7/ 8/2014 03:13:32.282] ndis!ndisMWakeUpDpcX (DPC @ fffffa800d4b02b8)
fffffa800d4b6278 431748ae 00000131 [ 7/ 8/2014 03:13:32.282] ndis!ndisMWakeUpDpcX (DPC @ fffffa800d4b62b8)
fffffa800d4bc278 431748ae 00000131 [ 7/ 8/2014 03:13:32.282] ndis!ndisMWakeUpDpcX (DPC @ fffffa800d4bc2b8)
fffffa800d4c7278 431748ae 00000131 [ 7/ 8/2014 03:13:32.282] ndis!ndisMWakeUpDpcX (DPC @ fffffa800d4c72b8)
152 fffffa8013c061f0 686b1d20 0000001b [ 7/ 6/2014 18:04:34.817] afd!AfdTimeoutPoll (DPC @ fffffa8013c061b0)
154 fffffa80154aac10 686f8a03 0000001b [ 7/ 6/2014 18:04:34.846] thread fffffa80154aab50
fffffa800e340120 85004f01 0000001b [ 7/ 6/2014 18:05:22.771] thread fffffa800e340060
155 fffffa800bb515c0 P 6872d5f8 0000001b [ 7/ 6/2014 18:04:34.867] ndis!ndisMTimerObjectDpc (DPC @ fffffa800bb51600)
157 fffffa800b6b37f0 6f9b4e54 0000001b [ 7/ 6/2014 18:04:46.877] thread fffffa800b6b3730
158 fffff880019d6500 0639e6a3 0000002c [ 7/ 6/2014 20:03:31.522] cng!seedFileDpcRoutine (DPC @ fffff880019d64c0)
161 fffffa8015a75120 68800501 0000001b [ 7/ 6/2014 18:04:34.954] thread fffffa8015a75060
fffffa800d4ca278 6fa44d23 0000001b [ 7/ 6/2014 18:04:46.935] ndis!ndisMWakeUpDpcX (DPC @ fffffa800d4ca2b8)
fffffa8015161120 76c7ea60 0000001b [ 7/ 6/2014 18:04:58.913] thread fffffa8015161060
162 fffffa800cf60260 P 6d44c4ba 0000001b [ 7/ 6/2014 18:04:42.954] aswSP (DPC @ fffffa800cf60220)+30280
nt!PopCheckForIdleness fires in 100 second intervals to check whether or not a DPC has been completed. If a DPC has not been completed in a set amount of time, the bug check is called. This is a particularly difficult 0x101, probably one of the more difficult ones I've analyzed. It requires a lot of knowledge regarding thread context switching, etc. It appears though in this case that avast! had a role in blocking a network packet, therefore not allowing the network subsystem to do its job.
With that said, I dove into the modules list and noted COMODO backup + avast! which is no good. You have two active security softwares running, certainly causing a potential amount of network conflicts.
1. Remove and replace avast!
and COMODO with Microsoft Security Essentials for temporary troubleshooting purposes as they are causing conflicts:
avast! removal - avast! Uninstall Utility | Download aswClear for avast! Removal
MSE - Microsoft Security Essentials - Microsoft Windows
2. AppleCharger.sys is listed and loaded which is the GIGABYTE On/Off Charge driver. See here for details -
GIGABYTE ON/OFF Charge
Very troublesome software, so please uninstall ASAP!
3.
Code:
0: kd> lmvm UltraMonUtility
start end module name
fffff880`0d4f9000 fffff880`0d502000 UltraMonUtility (deferred)
Image path: \??\C:\Program Files (x86)\Common Files\Realtime Soft\UltraMonMirrorDrv\x64\UltraMonUtility.sys
Image name: UltraMonUtility.sys
Timestamp: Thu Nov 13 20:10:30 2008
UltraMon's multi monitor drivers are
very old (2008) and often cause many problems. Please uninstall.
4. wdcsam64.sys is listed and loaded which is the Western Digital SES (SCSI Enclosure Services) driver. Please remove this software ASAP as it's very troublesome and is also not necessary to the functionality of your system.
Regards,
Patrick