BSOD Win2k8

JamieG · Nov 11, 2014

Hi - wonder if anyone can help with this BSoD issue we've had. I appreciate any help.

This is a Windows 2008 R2 box hosted on a vm 5.5 platform.

I've attached your FileCollection.zip.

Results of the mini dump
Could not read faulting driver name
READ_ADDRESS: GetPointerFromAddress: unable to read from fffff80001afd100
fffff68000000080
FAULTING_IP:
nt!MiRemoveWorkingSetPages+388
fffff800`0189e7a0 498b4500 mov rax,qword ptr [r13]
MM_INTERNAL_CODE: 5
CUSTOMER_CRASH_COUNT: 1
DEFAULT_BUCKET_ID: DRIVER_FAULT_SERVER_MINIDUMP
BUGCHECK_STR: 0x50
PROCESS_NAME: cabarc.exe
CURRENT_IRQL: 0
TRAP_FRAME: fffff88007b33700 -- (.trap 0xfffff88007b33700)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=fffff70001080488 rbx=0000000000000000 rcx=0000000000010009
rdx=0000098000000000 rsi=0000000000000000 rdi=0000000000000000
rip=fffff8000189e7a0 rsp=fffff88007b33890 rbp=0000000000000005
r8=0000000000000000 r9=fffff70001080000 r10=0000007ffffffff8
r11=0000000000000080 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na pe nc
nt!MiRemoveWorkingSetPages+0x388:
fffff800`0189e7a0 498b4500 mov rax,qword ptr [r13] ds:8028:00000000`00000000=????????????????
Resetting default scope
LAST_CONTROL_TRANSFER: from fffff800018748af to fffff800018cdfc0
STACK_TEXT:
fffff880`07b33598 fffff800`018748af : 00000000`00000050 fffff680`00000080 00000000`00000000 fffff880`07b33700 : nt!KeBugCheckEx
fffff880`07b335a0 fffff800`018cc0ee : 00000000`00000000 fffff680`00000080 fffff8a0`00040000 fffff700`010804b0 : nt! ?? ::FNODOBFM::`string'+0x437c1
fffff880`07b33700 fffff800`0189e7a0 : 00000000`00000001 00000000`00000000 fffff6fb`7dc00000 00000000`00000014 : nt!KiPageFault+0x16e
fffff880`07b33890 fffff800`0189f0d2 : fffffa80`00000001 00000000`00000080 fffff700`01080000 fffff700`01080488 : nt!MiRemoveWorkingSetPages+0x388
fffff880`07b33910 fffff800`01b9f65a : fffff8a0`0bd5e060 00000000`00000001 00000000`00000000 fffffa80`137e27b0 : nt!MmCleanProcessAddressSpace+0x4ca
fffff880`07b33960 fffff800`01b80f48 : 00000000`c0000005 00000000`00000001 00000000`7efdb000 00000000`00000000 : nt!PspExitThread+0x56a
fffff880`07b33a60 fffff800`018cd253 : fffffa80`1413a220 00000000`c0000005 fffffa80`137e27b0 00000000`7efdf000 : nt!NtTerminateProcess+0x138
fffff880`07b33ae0 00000000`772d15da : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
00000000`0008f758 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x772d15da

STACK_COMMAND: kb
FOLLOWUP_IP:
nt!MiRemoveWorkingSetPages+388
fffff800`0189e7a0 498b4500 mov rax,qword ptr [r13]
SYMBOL_STACK_INDEX: 3
SYMBOL_NAME: nt!MiRemoveWorkingSetPages+388
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: nt
DEBUG_FLR_IMAGE_TIMESTAMP: 503f82be
IMAGE_NAME: memory_corruption
FAILURE_BUCKET_ID: X64_0x50_nt!MiRemoveWorkingSetPages+388
BUCKET_ID: X64_0x50_nt!MiRemoveWorkingSetPages+388
Followup: MachineOwner
---------

Results of the memory dmp
Microsoft (R) Windows Debugger Version 6.12.0002.633 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [C:\Memory dumps\MEMORY.DMP]
Kernel Summary Dump File: Only kernel address space is available
Symbol search path is: .sympath SRV*f:\localsymbols*http://msdl.microsoft.com/download/symbols;SRV*C:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 7 Kernel Version 7601 (Service Pack 1) MP (8 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Built by: 7601.17944.amd64fre.win7sp1_gdr.120830-0333
Machine Name:
Kernel base = 0xfffff800`0184f000 PsLoadedModuleList = 0xfffff800`01a93670
Debug session time: Sun Nov 9 19:52:09.175 2014 (UTC + 0:00)
System Uptime: 32 days 0:30:10.572
Loading Kernel Symbols
...............................................................
...............................................Page 3e67c4 not present in the dump file. Type ".hh dbgerr004" for details
.................
...........
Loading User Symbols
PEB is paged out (Peb.Ldr = 00000000`7efdf018). Type ".hh dbgerr001" for details
Loading unloaded module list
.....
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
Use !analyze -v to get detailed debugging information.
BugCheck 50, {fffff68000000080, 0, fffff8000189e7a0, 5}
Page 3e67c4 not present in the dump file. Type ".hh dbgerr004" for details
Probably caused by : memory_corruption ( nt!MiRemoveWorkingSetPages+388 )
Followup: MachineOwner
---------
5: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
PAGE_FAULT_IN_NONPAGED_AREA (50)
Invalid system memory was referenced. This cannot be protected by try-except,
it must be protected by a Probe. Typically the address is just plain bad or it
is pointing at freed memory.
Arguments:
Arg1: fffff68000000080, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffff8000189e7a0, If non-zero, the instruction address which referenced the bad memory
address.
Arg4: 0000000000000005, (reserved)
Debugging Details:
------------------
Page 3e67c4 not present in the dump file. Type ".hh dbgerr004" for details
READ_ADDRESS: fffff68000000080
FAULTING_IP:
nt!MiRemoveWorkingSetPages+388
fffff800`0189e7a0 498b4500 mov rax,qword ptr [r13]
MM_INTERNAL_CODE: 5
DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT
BUGCHECK_STR: 0x50
PROCESS_NAME: cabarc.exe
CURRENT_IRQL: 0
TRAP_FRAME: fffff88007b33700 -- (.trap 0xfffff88007b33700)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=fffff70001080488 rbx=0000000000000000 rcx=0000000000010009
rdx=0000098000000000 rsi=0000000000000000 rdi=0000000000000000
rip=fffff8000189e7a0 rsp=fffff88007b33890 rbp=0000000000000005
r8=0000000000000000 r9=fffff70001080000 r10=0000007ffffffff8
r11=0000000000000080 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na pe nc
nt!MiRemoveWorkingSetPages+0x388:
fffff800`0189e7a0 498b4500 mov rax,qword ptr [r13] ds:8028:0000=????????????????
Resetting default scope
LAST_CONTROL_TRANSFER: from fffff800018748af to fffff800018cdfc0
STACK_TEXT:
fffff880`07b33598 fffff800`018748af : 00000000`00000050 fffff680`00000080 00000000`00000000 fffff880`07b33700 : nt!KeBugCheckEx
fffff880`07b335a0 fffff800`018cc0ee : 00000000`00000000 fffff680`00000080 fffff8a0`00040000 fffff700`010804b0 : nt! ?? ::FNODOBFM::`string'+0x437c1
fffff880`07b33700 fffff800`0189e7a0 : 00000000`00000001 00000000`00000000 fffff6fb`7dc00000 00000000`00000014 : nt!KiPageFault+0x16e
fffff880`07b33890 fffff800`0189f0d2 : fffffa80`00000001 00000000`00000080 fffff700`01080000 fffff700`01080488 : nt!MiRemoveWorkingSetPages+0x388
fffff880`07b33910 fffff800`01b9f65a : fffff8a0`0bd5e060 00000000`00000001 00000000`00000000 fffffa80`137e27b0 : nt!MmCleanProcessAddressSpace+0x4ca
fffff880`07b33960 fffff800`01b80f48 : 00000000`c0000005 00000000`00000001 00000000`7efdb000 00000000`00000000 : nt!PspExitThread+0x56a
fffff880`07b33a60 fffff800`018cd253 : fffffa80`1413a220 00000000`c0000005 fffffa80`137e27b0 00000000`7efdf000 : nt!NtTerminateProcess+0x138
fffff880`07b33ae0 00000000`772d15da : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
00000000`0008f758 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x772d15da

STACK_COMMAND: kb
FOLLOWUP_IP:
nt!MiRemoveWorkingSetPages+388
fffff800`0189e7a0 498b4500 mov rax,qword ptr [r13]
SYMBOL_STACK_INDEX: 3
SYMBOL_NAME: nt!MiRemoveWorkingSetPages+388
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: nt
DEBUG_FLR_IMAGE_TIMESTAMP: 503f82be
IMAGE_NAME: memory_corruption
FAILURE_BUCKET_ID: X64_0x50_nt!MiRemoveWorkingSetPages+388
BUCKET_ID: X64_0x50_nt!MiRemoveWorkingSetPages+388
Followup: MachineOwner
---------

x BlueRobot · Nov 11, 2014

Code:

[COLOR=#ff0000]BugCheck 50[/COLOR], {[COLOR=#008000]fffff68000000080[/COLOR], 0, fffff8000189e7a0, 5}

Probably caused by : [COLOR=#ff0000]memory_corruption[/COLOR] ( nt!MiRemoveWorkingSetPages+388 )

It seems like a freed address may have been referenced.

Code:

5: kd> [COLOR=#008000]knL[/COLOR]
 # Child-SP          RetAddr           Call Site
00 fffff880`07b33598 fffff800`018748af nt!KeBugCheckEx
01 fffff880`07b335a0 fffff800`018cc0ee nt! ?? ::FNODOBFM::`string'+0x437c1
02 fffff880`07b33700 fffff800`0189e7a0 nt!KiPageFault+0x16e
03 fffff880`07b33890 fffff800`0189f0d2 nt!MiRemoveWorkingSetPages+0x388
04 fffff880`07b33910 fffff800`01b9f65a nt!MmCleanProcessAddressSpace+0x4ca
05 fffff880`07b33960 fffff800`01b80f48 nt!PspExitThread+0x56a
06 fffff880`07b33a60 fffff800`018cd253 nt!NtTerminateProcess+0x138
07 fffff880`07b33ae0 00000000`772d15da nt!KiSystemServiceCopyEnd+0x13
08 00000000`0008f758 00000000`00000000 0x772d15da

Code:

5: kd> [COLOR=#008000]!pte fffff68000000080[/COLOR]
                                           VA 0000000000010000
PXE at FFFFF6FB7DBED000    PPE at FFFFF6FB7DA00000    PDE at FFFFF6FB40000000    PTE at FFFFF68000000080
contains 0000000000000000
[COLOR=#ff0000]not valid[/COLOR]

Since the PTE isn't valid, it would explain why a page fault occured at the the second frame.

Code:

5: kd> [COLOR=#008000].frame /r 2[/COLOR]
02 fffff880`07b33700 fffff800`0189e7a0 [COLOR=#0000ff]nt!KiPageFault+0x16e[/COLOR]
rax=fffff6fb7dbed000 rbx=fffff700010804b0 rcx=0000000000000050
[COLOR=#ff0000]rdx=fffff68000000080[/COLOR] rsi=fffff70001080508 rdi=0000000000000001
rip=fffff800018cc0ee rsp=fffff88007b33700 rbp=fffff88007b33780
 r8=0000000000000000  r9=fffff88007b33700 r10=0000000000000000
r11=0000000000000000 r12=0000000000000028 r13=fffff68000000080
r14=fffffa801413a5b8 r15=0000000000000010
iopl=0         nv up ei pl zr na po nc
cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
nt!KiPageFault+0x16e:
fffff800`018cc0ee 85c0            [COLOR=#ff0000]test    [/COLOR]eax,eax

Code:

5: kd> [COLOR=#008000]r @cr0[/COLOR]
cr0=000000008005003[COLOR=#ff0000]1[/COLOR]

According to some Intel documentation, if a page fault happens when using the test (Logical AND) instruction between two registers, then a exception will be raised.

Looking at the other dump file, during the cleanup for a termination of the process (see stack), the savonaccess.sys driver was found:

Code:

6: kd> [COLOR=#008000]lmvm savonaccess[/COLOR]

start             end                 module name
fffff880`0188f000 fffff880`018c3000   savonaccess T (no symbols)           
    Loaded symbol image file: savonaccess.sys
    Image path: \SystemRoot\system32\DRIVERS\savonaccess.sys
    Image name: savonaccess.sys
    Timestamp:        [COLOR=#ff0000]Mon Jun 10 09:39:25 2013[/COLOR] (51B590BD)
    CheckSum:         00036250
    ImageSize:        00034000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

The driver belongs to your Sophos AV software, and is most likely conflicting with another program or driver on the system; a common issue with most AV programs.

Have you attempted to temporarily disable the program?

[TD="class: small"][/TD]

JamieG · Nov 12, 2014

Thanks for that excellent write up - after reading quite a few other threads regarding BSoD I was suspecting the AV software may be causing this but really wanted confirmation.

Thanks again for that excellent knowledge :)

JamieG · Nov 12, 2014

I've got this machine excluded from the policy that turns on Sophos On Access scanning and will see what gives from here.

Thanks again.

x BlueRobot · Nov 13, 2014

Thanks for the update, let's see how it goes.

BSOD Win2k8

JamieG

New member

x BlueRobot

Administrator

JamieG

New member

JamieG

New member

x BlueRobot

Administrator