Okay, let's go through this... *101's are fairly difficult so bear with me.
As usual, it's a
CLOCK_WATCHDOG_TIMEOUT (101) bugcheck.
BugCheck 101, {
31, 0,
fffff88002fd7180, 3}
31 clock ticks in regards to the timeout.
fffff88002fd7180 is the PRCB address of the hung processor, let's keep this address in mind.
Running a !prcb on processor 0:
Code:
0: kd> !prcb 0
PRCB for Processor 0 at fffff80003643e80:
Current IRQL -- 13
Threads-- Current fffffa800c5cab50 Next 0000000000000000 Idle fffff80003651cc0
Processor Index 0 Number (0, 0) GroupSetMember 1
Interrupt Count -- 1e52881e
Times -- Dpc 0001a981 Interrupt 0000e408
Kernel 00524e0a User 0010e2e0
No match for address, let's try processor 1 this time:
Code:
0: kd> !prcb 1
PRCB for Processor 1 at fffff880009eb180:
Current IRQL -- 0
Threads-- Current fffffa800c2a0060 Next fffffa8009b7d060 Idle fffff880009f5fc0
Processor Index 1 Number (0, 1) GroupSetMember 2
Interrupt Count -- 1acd8bf2
Times -- Dpc 000000c3 Interrupt 00000640
Kernel 0050afcc User 0012810b
Nope, no match either. I'll spare you the space in the post and tell you that processor #3 is the one we're looking for :+)
Code:
0: kd> !prcb 3
PRCB for Processor 3 at fffff88002fd7180:
Current IRQL -- 0
Threads-- Current fffff88002fe1fc0 Next fffffa800b289640 Idle fffff88002fe1fc0
Processor Index 3 Number (0, 3) GroupSetMember 8
Interrupt Count -- 18eaff17
Times -- Dpc 000000ee Interrupt 000006b3
Kernel 004d01ad User 00162c9c
As this matches the 3rd parameter of the bugcheck, processor #3 is the responsible processor. Now with the information we have here thus far, we know that processor #3 reached 31 clock ticks without responding, therefore the system BSOD'd. Before we go further, what is a clock tick? A clock_interrupt is a form of interrupt which involves counting the the cycles of the processor core, which is running a clock on the processors to keep them all in sync. A clock interrupt is handed out to all processors and then they must report in, and when one doesn't report in, you then crash.
Now how and why did this take place? Let's check the IRQL of each one of the processors before the system crash:
Code:
0: kd> !irql 0
Debugger saved IRQL for processor 0x0 -- 13
0: kd> !irql 1
Debugger saved IRQL for processor 0x1 -- 0 (LOW_LEVEL)
0: kd> !irql 2
Debugger saved IRQL for processor 0x2 -- 0 (LOW_LEVEL)
0: kd> !irql 3
Debugger saved IRQL for processor 0x3 -- 0 (LOW_LEVEL)
The IRQL of the processor 1 is 13, which is CLOCK for x64 processors. The rest are at 0.
Let's see the call stacks of the processors:
Processor 0:
Code:
0: kd> kv
Child-SP RetAddr : Args to Child : Call Site
fffff880`0d3d1328 fffff800`0351fa3a : 00000000`00000101 00000000`00000031 00000000`00000000 fffff880`02fd7180 : nt!KeBugCheckEx
fffff880`0d3d1330 fffff800`034d26f7 : 00000000`00000000 fffff800`00000003 00000000`00002710 fffff880`0d3d1450 : nt! ?? ::FNODOBFM::`string'+0x4e3e
fffff880`0d3d13c0 fffff800`03414895 : fffff800`0343a460 fffff880`0d3d1570 fffff800`0343a460 00000000`00000000 : nt!KeUpdateSystemTime+0x377
fffff880`0d3d14c0 fffff800`034c50d3 : fffff800`03643e80 00000000`00000001 00000000`00000001 fffff800`03453000 : hal!HalpHpetClockInterrupt+0x8d
fffff880`0d3d14f0 fffff800`0349d8e7 : fffff6fc`0001bea0 fffff800`034e5c8f 00000000`00000000 74a00000`037d4860 : nt!KiInterruptDispatchNoLock+0x163 (TrapFrame @ fffff880`0d3d14f0)
fffff880`0d3d1680 fffff800`037d566f : 00000000`00000000 fffff880`0d3d1b60 00000000`00000000 fffff800`034f86c4 : nt!KeFlushProcessWriteBuffers+0x63
fffff880`0d3d16f0 fffff800`037d5ccd : 00000000`06ae0a90 fffff800`037c05ee 00000000`00000000 00000000`00010246 : nt!ExpQuerySystemInformation+0x13af
fffff880`0d3d1aa0 fffff800`034c7e13 : 00000000`00000000 fffff880`0d3d1b60 ffffffff`fffe7960 000007fe`fbfc0b48 : nt!NtQuerySystemInformation+0x4d
fffff880`0d3d1ae0 00000000`777a161a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`0d3d1ae0)
00000000`0461f028 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x777a161a
Processor 1:
Code:
1: kd> kv
Child-SP RetAddr : Args to Child : Call Site
fffff880`0cd33320 fffff800`03577c26 : 00000000`00000000 fffff880`009eb180 fffff980`37980d10 fffff800`034f383b : nt!KiIpiSendRequestEx+0xb2
fffff880`0cd33360 fffff800`035d0a69 : 00000000`00000001 fffff880`0cd33420 fffff6fc`c01bcc00 fffff980`37980d10 : nt!KeFlushSingleTb+0x126
fffff880`0cd333e0 fffff800`035fb93b : fffff800`03453000 00000000`6c624e54 00000000`097cb740 00000000`00000100 : nt!MmFreeSpecialPool+0x349
fffff880`0cd33530 fffff880`01635232 : fffff980`37980d30 fffff980`37980d30 fffff980`0149ac80 fffff980`0149aa00 : nt!ExDeferredFreePool+0xf33
fffff880`0cd335e0 fffff880`0186756a : 00000000`00000000 fffff980`37980e60 00000000`00000000 00000000`ea94000f : ndis!NdisFreeNetBufferList+0x112
fffff880`0cd33610 fffff880`01727872 : fffffa80`082f5bc8 00000000`00000001 00000000`00000001 fffffa80`08688800 : tcpip!TcpSendDatagramsComplete+0x9a
fffff880`0cd33650 fffff880`01885e6f : 00000000`00000000 00000000`00000001 fffff880`0cd33700 00000000`00000000 : NETIO!NetioDereferenceNetBufferListChain+0x132
fffff880`0cd33720 fffff880`01861d75 : fffff980`37980f00 00000000`00000000 00000000`00000000 00000000`00000000 : tcpip!TcpFlushDelay+0x13f
fffff880`0cd33800 fffff880`0183e2ba : fffffa80`084f3880 fffffa80`0000ea0c fffff880`000020fe fffff880`0cd320fe : tcpip!TcpPreValidatedReceive+0x3e5
fffff880`0cd338d0 fffff800`034d4878 : fffff880`0196e9a0 fffff800`034f3a06 00000000`00000000 fffff880`0cd33ae0 : tcpip!IppLoopbackTransmit+0x11a
fffff880`0cd33980 fffff880`0183ebcf : fffff880`0188e4b4 fffff880`0000000c fffff880`0cd33b02 00000000`00000000 : nt!KeExpandKernelStackAndCalloutEx+0xd8
fffff880`0cd33a60 fffff880`01860357 : fffff980`378f8d30 00000000`00000004 fffff880`0cd33f01 fffffa80`0b9e6010 : tcpip!IppLoopbackEnqueue+0x22f
fffff880`0cd33b10 fffff880`01866a85 : fffffa80`084f3820 00000000`00000001 fffffa80`00000001 00000000`00000080 : tcpip!IpNlpFastSendDatagram+0xb77
fffff880`0cd33ec0 fffff880`01867640 : fffff880`00000000 fffffa80`06d324d8 00000000`00000000 00000000`00004800 : tcpip!TcpTcbSend+0x495
fffff880`0cd34140 fffff880`01866398 : 00000000`00000000 000000ec`1711a9e9 fffff880`0cd34360 fffff880`0cd34470 : tcpip!TcpEnqueueTcbSendOlmNotifySendComplete+0xa0
fffff880`0cd34170 fffff880`0186655b : fffffa80`08d69000 00000000`fffffffe 00000000`00004800 fffff800`036d6b10 : tcpip!TcpEnqueueTcbSend+0x258
fffff880`0cd34220 fffff800`034d4878 : 00000000`00000001 fffff880`00000000 00000000`00000404 00000000`0cd34ae0 : tcpip!TcpTlConnectionSendCalloutRoutine+0x1b
fffff880`0cd34250 fffff880`0186741a : fffff880`01866540 00000000`00000000 00000000`00000000 fffff880`02d47601 : nt!KeExpandKernelStackAndCalloutEx+0xd8
fffff880`0cd34330 fffff880`02d61b2b : fffffa80`083dbcf0 fffff880`0cd34b60 00000000`0000003c fffffa80`0c2303b0 : tcpip!TcpTlConnectionSend+0x7a
fffff880`0cd343a0 fffff880`02d46469 : fffffa80`094b0000 fffffa80`094b0000 00000000`00000000 00000000`00000000 : afd!AfdFastConnectionSend+0x38b
fffff880`0cd34560 fffff800`037e6a33 : 00000000`0000003c fffffa80`0bb7ba10 00000000`00b7f5ec fffff880`00000001 : afd!AfdFastIoDeviceControl+0x459
fffff880`0cd348d0 fffff800`037e7526 : 00000000`00000000 00000000`00000114 00000000`00000001 00000000`00000000 : nt!IopXxxControlFile+0x373
fffff880`0cd34a00 fffff800`034c7e13 : 00000000`73b82450 00000000`001feb30 00000000`001ffd20 fffff880`0cd34ae0 : nt!NtDeviceIoControlFile+0x56
fffff880`0cd34a70 00000000`73b82e09 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`0cd34ae0)
00000000`001feab8 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x73b82e09
Processor 2:
Code:
2: kd> kv
Child-SP RetAddr : Args to Child : Call Site
fffff880`02748cc0 fffff800`03529188 : 0000007f`fffffff8 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KeFlushMultipleRangeTb+0x266
fffff880`02748d90 fffff800`035d060d : fffff800`03703b00 fffff880`02748fa0 fffff8a0`00000000 fffff8a0`00000006 : nt! ?? ::FNODOBFM::`string'+0x204ce
fffff880`02748f80 fffff800`035d1bd7 : fffffa80`09b91060 00000000`00000000 00000000`00000000 00000000`00000008 : nt!MiEmptyWorkingSet+0x27d
fffff880`02749030 fffff800`0396d371 : 00000000`00000000 fffff980`0149aa02 fffff980`0149ad00 fffff880`02d470e2 : nt!MiTrimAllSystemPagableMemory+0x218
fffff880`02749090 fffff800`0396d4cf : fffff980`0149aa02 fffffa80`06c90578 fffffa80`06c90578 fffff6fc`40016a38 : nt!MmVerifierTrimMemory+0xf1
fffff880`027490c0 fffff800`0396dc24 : fffff980`0149aa08 fffff980`0149aa00 00000000`000001f1 00000000`00000000 : nt!ViKeRaiseIrqlSanityChecks+0xcf
fffff880`02749100 fffff880`01635206 : 00000000`00000000 fffff980`378f8d30 fffff980`0149ad00 fffff980`378f8d30 : nt!VerifierKeAcquireSpinLockRaiseToDpc+0x54
fffff880`02749160 fffff880`0186718f : fffff980`378f8f70 fffff980`378f8d30 fffff980`378f8e60 fffff8a0`12fda0b0 : ndis!NdisFreeNetBufferList+0xe6
fffff880`02749190 fffff880`01727872 : fffffa80`07a56d70 fffff880`02749650 00000000`00000001 00000000`00000001 : tcpip!TcpTcbSendDatagramsComplete+0x12f
fffff880`027491f0 fffff880`0187c731 : 00000000`00000038 00000000`00000000 fffff880`02749238 00000000`00000000 : NETIO!NetioDereferenceNetBufferListChain+0x132
fffff880`027492c0 fffff880`02d1d44d : fffff880`02749650 fffff980`378f8d30 fffff880`02749b60 fffffa80`0b1387b0 : tcpip!TcpTlProviderReleaseIndicationList+0x81
fffff880`027492f0 fffff880`02d60a01 : 00000000`02e7fd70 fffff880`02749650 fffff880`02749650 00000000`00000000 : afd!AfdTLReleaseIndications+0x2d
fffff880`02749340 fffff880`02d467bb : fffff880`02749560 fffff880`02749560 fffffa80`00000000 00000000`00000000 : afd!AfdFastConnectionReceive+0x8d1
fffff880`02749560 fffff800`037e6a33 : 00000000`00000038 fffffa80`0885f090 00000000`02e7fc58 fffffa80`0861eb01 : afd!AfdFastIoDeviceControl+0x7ab
fffff880`027498d0 fffff800`037e7526 : fffff880`02749ab8 00000000`000204c8 00000000`00000000 00000000`00000000 : nt!IopXxxControlFile+0x373
fffff880`02749a00 fffff800`034c7e13 : fffff880`02749b60 fffffa80`098fe530 fffff880`02749ab8 fffff8a0`0e1b4001 : nt!NtDeviceIoControlFile+0x56
fffff880`02749a70 00000000`777a132a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`02749ae0)
00000000`02e7fba8 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x777a132a
Processor 3
Code:
3: kd> kv
Child-SP RetAddr : Args to Child : Call Site
00000000`00000000 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x0
From the above, we can see that processor 0 was the processor that performed the bugcheck. In processors 1-2 we can see various network related routines and such being called, tcpip... netio, etc.
We can also see that processor 3 fails to do anything at all. It's a 0x0, as John mentioned above earlier. *101's are very difficult to debug and analyze, and I am tongue in cheek as I type this, but at this point I very well believe this is a processor issue (if not motherboard). This is two times now we've seen the culprit processor 0x0. Do you happen to have a wireless adapter of some sort or something connected? If so, I would recommend directly connecting to the onboard NIC if possible to circumvent possible adapter issues (just throwing that out there given the network routines being called).
Regards,
Patrick