BSOD Woes

edallen · Apr 4, 2023

Good morning everyone,

I was recently handed a PC to take a look at to help resolve a BSOD issue. More specifically, there were 2 BSOD messages I see. The first BSOD message on boot was PAGE_FAULT_IN_NONPAGED_AREA which resulted in a restart. At restart, I would see either that same message or the BSOD IRQL_NOT_LESS_OR_EQUAL. After several auto-restarts and attempts to repair, I find that I can navigate the options menus and boot to Safe Mode. In Safe Mode, the system is stable, meaning no BSODs occur. So I ran the tool provided to gather the information from the log files. That detail is attached.

I did swap out memory from another same system in an attempt to resolve the PAGE_FAULT_IN_NONPAGED_AREA. Did not resolve the issue.

The following was run in Safe Mode/Boot from USB:

Ran Passmark MemTest86 (boot from USB), no issues found

Ran chkdsk /f /r (Safe Mode) - no issues found

Ran SFC /scannow (Safe Mode) - found some issues that were resolved. Re-ran and result was OK.

After running these tests, rebooted. Am able to logon however after about 2 minutes, BSOD PAGE_FAULT_IN_NONPAGED_AREA pops up again. Allowed the system to attempt to repair itself only to be presented with the options to repair, etc. Selected to enter Safe Mode (#5) and rebooted the system. The following information was gathered/ran in Safe Mode:

Per the instructions, here are the other details:

System Manufacturer: Lenovo ThinkCentre
Desktop PC
Model:M710e SFF (10UR001LUS)
OS: Windows 10 Pro (10.0.19045 Build 19045)
x64
Original Installed OS: unknown- a C:\Windows10Upgrade directory does exist though and the app Windows 10 Update Assistant is installed.
Original or OEM version: unknown - I assume it is OEM.
Age of system: based on Wikipedia, it appears this model was manufactured in 2019 making it 4 years old.
Age of Installation: The C:\Windows10Upgrade directory has a timestamp of 9/2019.
Attempted re-install OS: No, not yet......
CPU: Intel(R) Core(TM) i5-7400 @ 3.0 GHz.
RAM: 8 GB (2x4GB - Channel 1A-DIMM1 and Channel 1B-DIMM2) - All slots occupied
RAM Brand: Ramaxel (ran wmic memorychip get devicelocator, manufacturer at an admin command prompt)
RAM Model: RMUA5120ME86H9F - 2666
Video Card: On board Intel(R) HD Graphics 630 (PCI\VEN_8086&DEV_5912&SUBSYS_313C17AA&REV_04\3&111583659&1&10)
Motherboard: Lenovo Product 313C, Ver SDK0J40697 WIN 3305217874673
Power Supply: AC Bel 180W PS, Lenovo P/N SP50H29553
Driver Verifier: disabled
Security software: Security Manager AV Defender
Are you using proxy, vpn, ipfilters or similar software: yes, N-ABLE
Are you using Disk Image tools: No
Are you currently under/overclocking: No

Meanwhile, back at the ranch.....

Ran Maylwarebytes Free edition. See attached output file
Ran HDD Diagnostics. Both short and long self tests passed.
Already mentioned running PassMark MemTest86 which passed
I ran Driver Verifier however was using a different set of options. Will re-run following your instructions. Reset Driver Verifier (turned off). I rebooted the system. BSODs showed up again hence the reason I am submitting this request for assistance.

Finally, rebooted again. Logon prompt screen appears. Again, after about 2 minutes, BSOD message SYSTEM_SERVICE_EXCEPTION
System reboots - logon prompt screen appears. Logged on with AD Admin account credentials. After about 5-10 minutes, get the BSOD message KMODE_EXCEPTION_NOT_HANDLED
System reboots - logon prompt screen appears. Logged on with AD Admin account credentials. After about 5 minutes, get the BSOD message SYSTEM_SERVICE_EXCPETION.
System reboots - attempted to logon with AD Admin account credentials. System crashed - no BSOD.
System reboots - attempted to logon with AD credentials. System crashed with BSOD message AGE_FAULT_IN_NONPAGED_AREA with a notation clipsp.sys
System reboots - presented with the Recovery option screen. Selected Restart PC.
System reboots - BSOD screen presented SYSTEM_SERVICE_EXCEPTION noting ntfs.sys
System reboots - logged in with AD Admin credentials immediately followed by BSOD SYSTEM SERVICE EXCEPTION noting ntfs.sys
System reboots - presented with the Recovery option screen again. Opted to hold off on any other actions.

Thank you all kindly.

Best,

Doug

ubuysa · Apr 5, 2023

From these dumps your problem would appear to be BitDefender. In three of the five dumps BitDefender drivers are on the call stack (vlflt.sys and atc.sys). The last dump (040423-4921-01.dmp) has numerous calls to the rt640x64.sys driver, this is a Realtek network driver and it's old - though this may be the most recent version available...

Code:

2: kd> lmDvmrt640x64
Browse full module list
start             end                 module name
fffff806`23800000 fffff806`23918000   rt640x64   (deferred)             
    Image path: \SystemRoot\System32\drivers\rt640x64.sys
    Image name: rt640x64.sys
    Browse all global symbols  functions  data
    Timestamp:        Thu Mar  7 09:53:18 2019 (5C80CDEE)
    CheckSum:         00124C7D
    ImageSize:        00118000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4
    Information from resource tables:

However, even though no BitDefender drivers are on the call stack this could stioll be a BitDefender generated BSOD.

The first dump (040423-3375-01.dmp) is inconclusive. There are no BitDefender drivers on the call stack but there are calls to ntfs.sys, the Windows filesystem driver, which would indicate a file access operation - which again touches BitDefender.

I would uninstall BitDefender in the first instance. If it still BSODs then post back and I'll get you to run Driver Verifier with some specific driver selections.

edallen · Apr 6, 2023

Good morning,

Thank you for this valuable feedback.

First, I booted into Safe Mode and searched for vlflt.sys and atc.sys on the C: partition. As indicated, both files (details) are associated with BitDefender however Bit Defender is not installed on this system. After a bit of searching, it looks like the those two driver files are associated with the N-able Security Manager AV Defender. Unfortunately, I cannot uninstall from within Safe Mode. So, I renamed the vlflt.sys and atc.sys to vlflt.sys.bak and atc.sys.bak in the system32\drivers folder..... nothing to lose here. I also disabled all related N-able services with the exception of the N-able Technologies Endpoint Protected Service (Access is denied).

With regard to the Realtek PCIe GbE Family Controller driver, I first renamed the driver by adding the .bak extension to it. Then decided to see if Lenovo had an t driver. The 10.34.xxx version installed was the most up to date from Lenovo. Was able to download directly from Realtek v 10.63.1014.2022x and install.

Rebooted from Safe Mode to normal boot and logged on with AD Admin credentials.

First BSOD screen that popped up was for SYSTEM SERVICE EXCEPTION followed by reboot
Second BSOD screen that popped up was for KERNEL SECURITY CHECK FAILURE followed by reboot
Third BSOD screen that popped up was for SECURITY CHECK FAILURE followed by reboot to Recovery screen.

Booted to Safe Mode and renamed the vlflt.sys.bak and atc.sys.bak by removing the .bak extension. Also re-enabled the AV services noted above to go back to where I started today with the exception of installing the updated Realtek LAN driver.

Rebooted.

BSOD after logon screen comes up = KERNEL SECURITY CHECK FAILURE. System rebooted.
BSOD after logon screen comes up = SYSTEM SERVICE EXCEPTION (ntfs.sys). System rebooted to Recovery option
BSOD popped up when Recovery option is up. PAGE FAULT IN NONPAGED AREA. System rebooted to logon screen.
BSOD after logon screen = PAGE FAULT IN NONPAGED AREA (farflt.sys). System rebooted to logon screen.
BSOD after logon screen = SYSTEM SERVICE EXCEPTION. System rebooted to Recovery option.

Re-ran the SysnativeBSODCollectionApp from within Safe Mode. See attached.

Thanx!

In the interim, I'll see if there are manual uninstall procedures for the AV product installed since it's only uninstall-able outside of Safe Mode.

ubuysa · Apr 6, 2023

That's unusual, I did check those two drivers and it seemed pretty clear that they were part of BitDefender. It seems they are also used in the N-Able product that you have.

I think those two drivers are the cause of all the dumps you just uploaded. Most of them show NTFS filesystem accesses, several show calls to Malwarebytes, and one shows calls to vlflt.sys. It's never wise to run two real-time antivirus products and I think the N-Able product is causing Malwarebytes to crash. I don't think it's wise to try any more troubleshooting until that N-Able product (and those two drivers) is uninstalled.

One dump is an 0x124 WHEA_UNCORRECTABLE_ERROR for a machine check exception. These are usually hardware related and are most often cause by overly aggressive overclocking or undervolting. If you are overclocking or undervolting anything (CPU, RAM, GPU) please revert to clock voltages and frequencies.

edallen · Apr 6, 2023

No worries. The computer isn't overclocked nor undervolted. This is a catch-22 situation where the computer is not up long enough to remove/repair/reinstall the application. Looks like Lenovo's recovery (OS reinstall) is in order.

Thanx for all the feedback. Very helpful.

Best,

Doug

x BlueRobot · Apr 6, 2023

You could just remove the drivers/services of the application from the registry within Safe Mode.

FRST Registry Search
1. Please download Farbar Recovery Scan Tool and save it to your Desktop.
Note: You need to run the 64-bit Version so please ensure you download that one.
2. Run FRST64 by Right-Clicking on the file and choosing Run as administrator.
3. Copy and paste vlflt; atc into the Search box and click the Search Registry button.
4. When the scan is complete a notepad window will open with the results. Please attach this to your next reply. It is saved on your desktop named SearchReg.txt.

edallen · Apr 6, 2023

OK. See attached results file.

Best,

Doug

x BlueRobot · Apr 7, 2023

Thanks, please create a backup of your system if you haven't done so already and then delete the service using the following commands:

Code:

sc stop vlflt
sc delete vlflt

edallen · Apr 10, 2023

Good morning all,

OK. Stopped the vlflt service in Safe Mode, then deleted it - Success
Restarted the PC (normal) and logged on as AD Admin
BSOD appeared SYSTEM SERVICE EXCEPTION.
BSOD on reboot appeared SYSTEM SERVICE EXCEPTION.
Computer rebooted to Recovery option.
Rebooted to Safe Mode. While running the BSOD Collection App, got a BSOD - PAGE FAULT IN NONPAGED AREA.
System rebooted (normal) to logon screen. BSOD screen popped up - SYSTEM SERVICE EXCEPTION.
System rebooted again to logon screen. Logged on with AD Admin creds. BSOD screen popped up - IRQL NOT LESS OR EQUAL
System reboot (normal) to logon screen. Logged on with AD Admin creds. Initiated BSOD Collection App. App ran successfully followed by BSOD - PAGE FAULT IN NONPAGED AREA.
System rebooted (normal) to logon screen. BSOD - PAGE FAULT IN NONPAGED AREA.
System rebooted (normal) to logon screen. Logged on with AD Admin creds. BSOD screen popped up - KMODE EXCEPTION NOT HANDLED (ntfs.sys failed).
System rebooted to Recovery option.
Rebooted to Safe Mode. Re-ran BSOD Collection App. This time the app ran to completion and was able to grab the file (attached) before any crash could occur.

Thanx!

D

ubuysa · Apr 10, 2023

Just from your description this feels like hardware to me. In your System log there are a multitude of hardware warning messages, all the same...

Code:

Event[158]:
  Log Name: System
  Source: Microsoft-Windows-WHEA-Logger
  Date: 2023-04-10T07:24:48.9260000Z
  Event ID: 19
  Task: N/A
  Level: Warning
  Opcode: Info
  Keyword: N/A
  User: S-1-5-19
  User Name: NT AUTHORITY\LOCAL SERVICE
  Computer: HSI008.harfordsystems.com
  Description:
A corrected hardware error has occurred.

Reported by component: Processor Core
Error Source: Unknown Error Source
Error Type: Internal parity error
Processor APIC ID: 4

The details view of this entry contains further information.

Although these are just 'warning' messages, and the problem was apparently corrected, I don't think you should ignore these.

In one of the dumps the bugcheck happens whilst manipulating page tables, in another the bugcheck happens when manipulating an object in memory, two dumps seems to bugcheck when accessing registry keys in memory, and the last is related to an NTFS file access. I suspect this last one may be related to this System log error, which also occurs numerous times...

Code:

Event[159]:
  Log Name: System
  Source: Ntfs
  Date: 2023-04-10T07:24:45.7640000Z
  Event ID: 55
  Task: N/A
  Level: Error
  Opcode: Info
  Keyword: N/A
  User: S-1-5-18
  User Name: NT AUTHORITY\SYSTEM
  Computer: HSI008.harfordsystems.com
  Description:
A corruption was discovered in the file system structure on volume C:.

A corruption was found in a file system index structure.  The file reference number is 0x4f7000000001683.  The name of the file is "\$WinREAgent\Scratch".  The corrupted index attribute is ":$I30:$INDEX_ROOT".

I wonder however, whether this filesystem corruption may have been caused by whatever hardware issue is also occurring?

Since the hardware warning messages talk about parity errors I would suspect RAM first. I know you've tested the RAM with Memtest86, but no RAM tester is foolproof, so I would suggest you run on one RAM stick at a time and see whether it BSODs on just on stick.

It's also important to run a chkdsk /f on the C: drive to see whether that filesystem corruption can be corrected.

edallen · Apr 10, 2023

OK. Since I was already in Safe Mode, ran chkdsk c: /f. Since a reboot was required, I didn't see anything pop up after it ran. Of course, system crashed with PAGE FAULT IN NONPAGED AREA at the logon screen. Powered the unit off manually to remove RAM sticks.

Removed one of the two RAM sticks. Rebooted, logged in as AD Admin and got the BSOD = PAGE FAULT IN NONPAGED AREA.
Powered off the PC manually again.
Removed the second RAM stick and inserted the first removed into the same system board slot it was removed from.
Booted the computer up and logged in as AD Admin. System is stable so far.

Rebooted using Win10 media to run chkdsk /f on C: from the USB drive so that I can see the results (first run and second run detail attached.
Continued on to Win 10. Logged on as AD Admin. System is stable. Will leave the PC up and wait to see if a BSOD occurs again.

As suggested, looks like bad RAM..... but time will tell.

edallen · Apr 10, 2023

System crashed (BSOD) after about 20 minutes with IRQL NOT LESS OR EQUAL.
There have been other reboots without BSOD messages. No doubt the system is more stable after removal of the one RAM stick so there has been improvement in behavior.

Letting the system run for a while unless you'd like to see the Sysnative app output again.

Thanx!

D

edallen · Apr 10, 2023

BTW.... I have a couple more RAM modules for this system. Swapped out the one in DIMM1 with a spare unit and booted the system. Will see what happens...

Thanx!

D

Search

Search

BSOD Woes

edallen

Contributor

Attachments

ubuysa

Sysnative Staff
BSOD Kernel Dump Senior Analyst
Contributor

edallen

Contributor

Attachments

ubuysa

Sysnative Staff
BSOD Kernel Dump Senior Analyst
Contributor

edallen

Contributor

x BlueRobot

Administrator

edallen

Contributor

Attachments

x BlueRobot

Administrator

edallen

Contributor

Attachments

ubuysa

Sysnative Staff
BSOD Kernel Dump Senior Analyst
Contributor

edallen

Contributor

Attachments

edallen

Contributor

edallen

Contributor

BSOD Woes

Contributor

Attachments

Sysnative Staff BSOD Kernel Dump Senior AnalystContributor

Contributor

Attachments

Sysnative Staff BSOD Kernel Dump Senior AnalystContributor

Contributor

Administrator

Contributor

Attachments

Administrator

Contributor

Attachments

Sysnative Staff BSOD Kernel Dump Senior AnalystContributor

Contributor

Attachments

Contributor

Contributor

Sysnative Staff
BSOD Kernel Dump Senior Analyst
Contributor

Sysnative Staff
BSOD Kernel Dump Senior Analyst
Contributor

Sysnative Staff
BSOD Kernel Dump Senior Analyst
Contributor