Windows Server 2012 Essentials BSOD

dbone1026 · Jan 21, 2015

View attachment 10590 View attachment Perfmon.zip

I have been having an issue with my DIY Home Server for a while now. It will just randomly enter into a BSOD, the typical message is about Special Pool Detected Memory Corrupt (I have gotten other BSODs as well, but this one is about 95% of the cases). Once I restart the server everything is fine. I may get this BSOD every day, or I may got weeks without getting. Any help would be greatly appreciated as it gets frustrating when the wife or I sit down to watch a movie/TV show only to find the server is in BSOD. Any other information I can provide I would be glad to. There is approx 40TB+ worth of hard drives connected to the RAID controllers. Everything is set as JBOD using Stablebit DrivePool.

O/S: Windows Server 2012 Essentials
x64
Full Retail Version
Age of System (hardware): Almost 2 years
Age of OS Installation: Almost 2 years (no reinstallations done)

CPU: Intel Core i5-3570K Ivy Bridge
Video Card: None
RAID Card: 2 x HighPoint RocketRAID 2720SGL 8-Port PCI Express 2.0
Motherboard: Gigabyte G1.Sniper 3 LGA 1155 Intel Z77
Power Supply: Corsair Gold AX850 850W ATX12V v2.31
Memory: G.SKILL Ripjaws X Series 16GB (8x2) 240Pin DDR3 SDRAM DDR3 1600

System Manufacturer: DYI build

Case: Lian Li PC-D8000

Jared · Jan 21, 2015

Code:

1: kd> .bugcheck
Bugcheck code 000000C1
Arguments fffff980`1e140fd0 fffff980`1e140b4a 00000000`00594030 00000000`00000032

1: kd> kn
 # Child-SP          RetAddr           Call Site
00 fffff880`1a790408 fffff802`c4dfc67a nt!KeBugCheckEx //BSOD
01 fffff880`1a790410 fffff802`c4dfbba4 nt!MiCheckSpecialPoolSlop+0x8a //Check special pool slop bytes
02 fffff880`1a790450 fffff802`c4ee17e5 nt!MmFreeSpecialPool+0x178 //Free special pool allocation
03 fffff880`1a790580 fffff802`c52ad587 nt!ExFreePool+0x6d8 //Kernel, Free pool
04 fffff880`1a790660 fffff880`0133edbb nt!VerifierExFreePoolWithTag+0x47 //Free pool with tag
05 fffff880`1a790690 fffff880`0133e37c NETIO!RemoveFilterFromFilterHashtable+0x7b //Remove network filter
06 fffff880`1a7906c0 fffff880`0133d27f NETIO!DeleteFilterFromIndex+0x12c //Remove filter from index
07 fffff880`1a790730 fffff880`0133e9fe NETIO!KfdCommitTransaction+0x123 //Initiate transaction
08 fffff880`1a7907a0 fffff880`01c482c4 NETIO!IoctlKfdCommitTransaction+0xee
09 fffff880`1a790800 fffff802`c52acd66 tcpip!KfdDispatchDevCtl+0xf8
0a fffff880`1a790840 fffff802`c50a4127 nt!IovCallDriver+0x3e6 //Call driver
0b fffff880`1a790890 fffff802`c50b92f6 nt!IopXxxControlFile+0x7e5 //I/O control file
0c fffff880`1a790a20 fffff802`c4cc8553 nt!NtDeviceIoControlFile+0x56 //Device I/O
0d fffff880`1a790a90 000007f8`e8ea2c1a nt!KiSystemServiceCopyEnd+0x13 //System service ends
0e 000000d5`bf24f318 00000000`00000000 0x000007f8`e8ea2c1a //User mode address

..trimmed...
fffff980`1e140f80  59595959 59595959 59595959 59595959  YYYYYYYYYYYYYYYY
fffff980`1e140f90  59595959 59595959 59595959 59595959  YYYYYYYYYYYYYYYY
fffff980`1e140fa0  59595959 59595959 59595959 59595959  YYYYYYYYYYYYYYYY
fffff980`1e140fb0  59595959 59595959 59595959 59595959  YYYYYYYYYYYYYYYY
fffff980`1e140fc0  59595959 59595959 59595959 59595959  YYYYYYYYYYYYYYYY
fffff980`1e140fd0  1e5c2fd0 fffff980 0d55c6c0 fffffa80  ./\.......U.....
fffff980`1e140fe0  16c20df7 63fb5b0d 00000001 0000002c  .....[.c....,...
fffff980`1e140ff0  00010293 00000000 1e13efd0 fffff980  ................

Bytes within the same page have been corrupt, most likely caused by a network driver not cleaning up.

The dump files are all related to the same kind of corruption within the same page, different addresses though.

We will need a better dump if we are to examine this corruption.

Go the Start
Right click My Computer
Select Properties
Click Advanced system settings
Click on the Advanced tab
Select Settings under Startup and Recovery
Then under Write debugging information select Complete memory dump.

Once a dump is created go to:

Code:

C:\Windows\memory.dmp

Copy the file to the desktop, zip it up and upload it to a file sharing site like Onedrive. After the upload is done post the download link in your next reply.

dbone1026 · Jan 21, 2015

Thanks, set up so I will post the dump file after my server BSODs.

When I first built the server it was a nightmare getting the NIC drivers installed as it wouldn't even recognize my NIC. I believe the issue was in part due to a lack of support from Intel since Server 2012 Essentials was new. IIRC I followed something similar to this to get thinigs working properly:

Enable the Intel 82579V NIC in Windows Server 2012 - www.ivobeerens.nl

Hopefully I will have a new dump file for you soon.

dbone1026 · Jan 24, 2015

OK, system BSOD last night, here is the complete memory dump:

MEMORY

dbone1026 · Feb 14, 2015

Hi - was wondering if anyone had any thoughts on best option to correct my issues. The BSODs have been picking up in frequency of late, debating whether or not I just rebuild my server from scratch at this point ...

Jared · Feb 14, 2015

Could you upload a new complete memory dump? Preferably zipped and to a different site such as Onedrive, Dropbox or Google drive. Mediafire has very poor download speeds.

dbone1026 · Feb 15, 2015

Jared said:
Could you upload a new complete memory dump? Preferably zipped and to a different site such as Onedrive, Dropbox or Google drive. Mediafire has very poor download speeds.

Sorry about that, see link below on my google drive zipped:

https://drive.google.com/file/d/0BxtYAJ7xBZGgbUZpR2RIcXZqLXM/view?usp=sharing

Jared · Feb 15, 2015

Code:

3: kd> .bugcheck
Bugcheck code 000000C1
Arguments fffff980`b4d1ae70 fffff980`b4d1a2da 00000000`00474190 00000000`00000032

//Special pool corruption

3: kd> kn
 # Child-SP          RetAddr           Call Site
00 fffff880`07d85108 fffff802`8c00582a nt!KeBugCheckEx //Bugcheck, corruption was found
01 fffff880`07d85110 fffff802`8c004d54 nt!MiCheckSpecialPoolSlop+0x8a //Check the slop bytes for corruption
02 fffff880`07d85150 fffff802`8c0e97e9 nt!MmFreeSpecialPool+0x178 //Memory manager free special pool function
03 fffff880`07d85280 fffff802`8c4b5577 nt!ExFreePool+0x6d8 //Kernel support routine to free the pool
04 fffff880`07d85360 fffff880`01b6942d nt!VerifierExFreePoolWithTag+0x47 //Free pool verifier hooked function
05 fffff880`07d85390 fffff880`01b77944 covefs+0x342d
06 fffff880`07d853c0 fffff880`01b779ad covefs+0x11944
07 fffff880`07d853f0 fffff880`01b6b951 covefs+0x119ad
08 fffff880`07d85440 fffff880`01b6a600 covefs+0x5951
09 fffff880`07d85520 fffff880`01b6a4ab covefs+0x4600
0a fffff880`07d855a0 fffff880`01b6b21e covefs+0x44ab
0b fffff880`07d85610 fffff880`01b6a6ee covefs+0x521e
0c fffff880`07d85780 fffff880`01b6a4ab covefs+0x46ee
0d fffff880`07d85800 fffff880`01b6b21e covefs+0x44ab
0e fffff880`07d85870 fffff880`01b6a6ee covefs+0x521e
0f fffff880`07d859e0 fffff880`01b6a4ab covefs+0x46ee
10 fffff880`07d85a60 fffff880`01b6b21e covefs+0x44ab
11 fffff880`07d85ad0 fffff880`01b6a6ee covefs+0x521e
12 fffff880`07d85c40 fffff880`01b6a4ab covefs+0x46ee
13 fffff880`07d85cc0 fffff880`01b6b21e covefs+0x44ab
14 fffff880`07d85d30 fffff880`01b6a6ee covefs+0x521e
15 fffff880`07d85ea0 fffff880`01b6a4ab covefs+0x46ee
16 fffff880`07d85f20 fffff880`01b6b21e covefs+0x44ab
17 fffff880`07d85f90 fffff880`01b6a6ee covefs+0x521e
18 fffff880`07d86100 fffff880`01b6a4ab covefs+0x46ee
19 fffff880`07d86180 fffff880`01b6b21e covefs+0x44ab
1a fffff880`07d861f0 fffff880`01b6a6ee covefs+0x521e
1b fffff880`07d86360 fffff880`01b6a4ab covefs+0x46ee
1c fffff880`07d863e0 fffff880`01b6b21e covefs+0x44ab
1d fffff880`07d86450 fffff880`01b6a6ee covefs+0x521e
1e fffff880`07d865c0 fffff880`01b6a4ab covefs+0x46ee
1f fffff880`07d86640 fffff880`01b6b21e covefs+0x44ab
20 fffff880`07d866b0 fffff880`01b6a6ee covefs+0x521e
21 fffff880`07d86820 fffff880`01b6a4ab covefs+0x46ee
22 fffff880`07d868a0 fffff880`01b6b21e covefs+0x44ab
23 fffff880`07d86910 fffff880`01b6ba77 covefs+0x521e
24 fffff880`07d86a80 fffff880`01b6bd67 covefs+0x5a77
25 fffff880`07d86af0 fffff880`01bb1aae covefs+0x5d67
26 fffff880`07d86b90 fffff802`8bea4f69 covefs+0x4baae //Covefs internal functions
27 fffff880`07d86c10 fffff802`8bf592f6 nt!PspSystemThreadStartup+0x59 //Start system thread
28 fffff880`07d86c60 00000000`00000000 nt!KiStartSystemThread+0x16 //Start system thread

3: kd> !address fffff980b4d1ae70
Mapping user range ...
Mapping system range ...
Mapping non addressable range ...
Mapping page tables...
Mapping hyperspace...
Mapping HAL reserved range...
Mapping User Probe Area...
Mapping system shared page...
Mapping system cache working set...
Mapping loader mappings...
Mapping system PTEs...
Mapping system paged pool...
Mapping session space...
Mapping dynamic system space...
Mapping PFN database...
Mapping non paged pool...
Mapping VAD regions...
Mapping module regions...
Mapping process, thread, and stack regions...
Mapping system cache regions...


Usage:                  
Base Address:           fffff980`b4600000
End Address:            fffff980`b5a00000
Region Size:            00000000`01400000
VA Type:                SystemDynamicSpace //Dynamic kernel system space
VAD Address:            0x2c003400360072
Commit Charge:          0x10000013a
Protection:             0x7feea5cff88 []
Memory Usage:           Private
No Change:              yes
More info:              !vad 0xfffff980b4600000

//Corruption to page

fffff980`b4d1ae30  47474747 47474747 47474747 47474747  GGGGGGGGGGGGGGGG
fffff980`b4d1ae40  47474747 47474747 47474747 47474747  GGGGGGGGGGGGGGGG
fffff980`b4d1ae50  47474747 47474747 47474747 47474747  GGGGGGGGGGGGGGGG
fffff980`b4d1ae60  47474747 47474747 47474747 47474747  GGGGGGGGGGGGGGGG
fffff980`b4d1ae70  07d85360 fffff880 5cafce70 fffff980  `S......p..\....
fffff980`b4d1ae80  00aa00aa 00000000 e56faf50 fffff980  ........P.o.....
fffff980`b4d1ae90  00aa00aa 00000000 e56faf50 fffff980  ........P.o.....
fffff980`b4d1aea0  008a008a 00000000 5ab40f70 fffff980  ........p..Z....

//Full data block for known allocation

fffff980`b4d1ae70  07d85360 fffff880 5cafce70 fffff980  `S......p..\....
fffff980`b4d1ae80  00aa00aa 00000000 e56faf50 fffff980  ........P.o.....
fffff980`b4d1ae90  00aa00aa 00000000 e56faf50 fffff980  ........P.o.....
fffff980`b4d1aea0  008a008a 00000000 5ab40f70 fffff980  ........p..Z....
fffff980`b4d1aeb0  00000001 00000000 008a002e 00000000  ................
fffff980`b4d1aec0  5ab40f70 fffff980 00000000 00000000  p..Z............
fffff980`b4d1aed0  07d854f0 fffff880 5cafced0 fffff980  .T.........\....
fffff980`b4d1aee0  00000000 00000000 00000000 00000000  ................
fffff980`b4d1aef0  00000000 00000000 008a002e 00000000  ................
fffff980`b4d1af00  5ab40f70 fffff980 008a008a 00000000  p..Z............
fffff980`b4d1af10  5ab40f70 fffff980 01340134 00000000  p..Z....4.4.....
fffff980`b4d1af20  b4bb4ec0 fffff980 00000001 00000000  .N..............
fffff980`b4d1af30  01340134 00000000 b4bb4ec0 fffff980  4.4......N......
fffff980`b4d1af40  00000000 00000000 3a3795ff 00000000  ..........7:....
fffff980`b4d1af50  08f51510 00000000 08f51510 00000000  ................
fffff980`b4d1af60  00000008 00000200 00000007 00000004  ................
fffff980`b4d1af70  00000002 00000000 00000000 00000000  ................
fffff980`b4d1af80  00000000 00000000 00000000 00000000  ................
fffff980`b4d1af90  00000000 00000000 00000000 00000000  ................
fffff980`b4d1afa0  00000000 00000000 00000000 00000000  ................
fffff980`b4d1afb0  00000000 00000000 00000000 00000000  ................
fffff980`b4d1afc0  00000000 00000000 00000000 00000000  ................
fffff980`b4d1afd0  00000000 00000000 00000000 00000000  ................
fffff980`b4d1afe0  00000000 00000000 0a606d70 fffff980  ........pm`.....
fffff980`b4d1aff0  00000000 00000000 00000000 00000000  ................
fffff980`b4d1b000  ???????? ???????? ???????? ????????  ????????????????

3: kd> lmvm covefs
start             end                 module name
fffff880`01b66000 fffff880`01bcd000   covefs     (no symbols)           
    Loaded symbol image file: covefs.sys
    Image path: \??\C:\Windows\system32\drivers\covefs.sys
    Image name: covefs.sys
    Timestamp:        Sat Jun 07 19:34:27 2014 (53935B33)
    CheckSum:         0006C805
    ImageSize:        00067000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

Well, we know the Kernel covefs driver didn't clean up it's allocations and caused corruption on a page. The pool is corrupt to the point where we cannot check who owned the page prior to the corruption. It's no longer a valid page, thus writing over all information.
Covefs is a Windows Driver Development Kit Kernel driver, I haven't used the kit myself, I have been meaning to give it a try.
I suggest you remove the package that this driver comes with, or the entire kit, whichever option is the most feasible for your machine. You could try an update as it dates back to June 2014, if that doesn't stop the crashes then remove the software.

x BlueRobot · Feb 15, 2015

I keep meaning to use !address more often, and the page where is the slop bytes reside is simply a junk page to test if a driver is writing into different page buffers.

Jared · Feb 15, 2015

I thought slop bytes are written in a pattern that matches the allocation header, and when the allocation is freed using ExFreePoolWithTag, verifier checks the slop bytes to see if the pattern matches, if not then the driver within that pool almost certainly caused the corruption. AFAIK it's used as a backup if the guard patch doesn't catch any buffer overruns.

x BlueRobot · Feb 15, 2015

That's what I meant, it's not a 'real' page.

dbone1026 · Feb 15, 2015

Jared said:
Well, we know the Kernel covefs driver didn't clean up it's allocations and caused corruption on a page. The pool is corrupt to the point where we cannot check who owned the page prior to the corruption. It's no longer a valid page, thus writing over all information.
Covefs is a Windows Driver Development Kit Kernel driver, I haven't used the kit myself, I have been meaning to give it a try.
I suggest you remove the package that this driver comes with, or the entire kit, whichever option is the most feasible for your machine. You could try an update as it dates back to June 2014, if that doesn't stop the crashes then remove the software.

Thanks. I believe Covefs is the Stablebit Drivepool software I use to do software pooling on my server. I am going to reach out to the Drivepool developers if true so hopefully we can determine a resolution. Greatly appreciate the help here as I have been dealing with this issue for over a year now.I will post back once I hear from the developers.

Search

Search

Windows Server 2012 Essentials BSOD

dbone1026

Member

Jared

Sysnative Staff, BSOD Kernel Dump Expert

dbone1026

Member

dbone1026

Member

dbone1026

Member

Jared

Sysnative Staff, BSOD Kernel Dump Expert

dbone1026

Member

Jared

Sysnative Staff, BSOD Kernel Dump Expert

x BlueRobot

Administrator

Jared

Sysnative Staff, BSOD Kernel Dump Expert

x BlueRobot

Administrator

dbone1026

Member