[SOLVED] DRIVER_IRQL_NOT_LESS_OR_EQUAL (NETIO.SYS) on different machines. FIX: I removed Safetica (data loss prevention) until they solve the issue.

P

PiotrIr

Member
Joined
Sep 11, 2014
Posts
9
Hi,

I just wonder if somebody could help me with analysing memory.dmp file. Recently we moved around 80 machines to Office 365 and Azure Active Directory. Mostly they are working fine, however 8 of them blue screen every few days (or more often). There is no pattern in terms of hardware, as we have 5 years old HP ZBook, few HP EliteBook 840 G3, few new XPS (PC, laptops different models). Analyse of memory.dmp file shows, that all of them are crashing due to DRIVER_IRQL_NOT_LESS_OR_EQUAL and NETIO.SYS which is not very helpful. All machines are rebuilt with Windows 10 Pro image and latest drivers.

Common things to all (crashing and not crashing) machines in this company:

  • All are Azure AD joined
  • Intune and policies set
  • Office 365 and OneDrive installed
  • ESET Endpoint Security as antivirus
  • Duo Security as 2FA
  • Safetica as DLP (only monitoring)
  • All laptops are encrypted, however some with BitLocker other with ESET EE
We are trying to resolve this issue for few weeks with no success do I would be very grateful if somebody could help

Thank you
 

Attachments

Hello PiotrIr,

I can see you have been a member for some time but this is your first post! Welcome!

I've looked at your two most recent crash dumps provided in the FileCollection zip and can see reference to a Rivet Networks Killer Network driver which seems to be the cause for the crash.

Please try and look into this driver - it may be getting loaded by mistake as there is also another driver listed in the msinfo
kfecosvc KfeCoSvc c:\windows\system32\drivers\rivetnetworks\killer\kfeco10x64.sys





Code: 
12: kd> .load pde;!dpx
=========================================================================================
PDE v11.3 - Copyright 2017 Andrew Richards
=========================================================================================
Start memory scan : 0xfffff580712b55a8 ($csp)
End memory scan : 0xfffff580712b8000 (Kernel Stack Base)

               rsp : 0xfffff580712b55a8 : 0xfffff80152407769 : nt!KiBugCheckDispatch+0x69
               r11 : 0xfffff580712b56e8 : 0xfffff80152403a69 : nt!KiPageFault+0x469
0xfffff580712b55a8 : 0xfffff80152407769 : nt!KiBugCheckDispatch+0x69
0xfffff580712b55d0 : 0xfffff80157dcd7f4 : NETIO!StreamInvokeCalloutAndNormalizeAction+0x5c
Unable to load image \SystemRoot\system32\DRIVERS\epfwwfp.sys, Win32 error 0n2
*** WARNING: Unable to verify timestamp for epfwwfp.sys
0xfffff580712b56e8 : 0xfffff80152403a69 : nt!KiPageFault+0x469
Unable to load image \SystemRoot\system32\drivers\netmonitor_wfp.sys, Win32 error 0n2
*** WARNING: Unable to verify timestamp for netmonitor_wfp.sys
0xfffff580712b5858 : 0xfffff80157dcd7f4 : NETIO!StreamInvokeCalloutAndNormalizeAction+0x5c
0xfffff580712b58b0 : 0xfffff80157e3a000 : NETIO!WPP_GLOBAL_Control
0xfffff580712b58b8 : 0xfffff80157dcdc52 : NETIO!StreamUnpublishClassifyData+0xda
0xfffff580712b58e8 : 0xfffff80157e1bf08 : NETIO!StreamDataBlock+0x158
0xfffff580712b5948 : 0xfffff80157dcd508 : NETIO!StreamProcessCallout+0x3fc
0xfffff580712b5a78 : 0xfffff80157dccb86 : NETIO!ProcessCallout+0x706
0xfffff580712b5b60 : 0xfffff80157e3a590 : NETIO!handleTableLock
0xfffff580712b5bf8 : 0xfffff80157dc95eb : NETIO!ArbitrateAndEnforce+0x71b
0xfffff580712b5d58 : 0xfffff80157dc818a : NETIO!KfdClassify+0x37a
0xfffff580712b60e8 : 0xfffff801522ec5ca : nt!HalPerformEndOfInterrupt+0x1a
0xfffff580712b6148 : 0xfffff80157dc789b : NETIO!StreamClassify+0x28b
0xfffff580712b62e8 : 0xfffff80157dc74ba : NETIO!StreamCommonInspect+0x282
0xfffff580712b62f8 : 0xfffff8015224caf2 : nt!ExFreeHeapPool+0x362
0xfffff580712b66a8 : 0xfffff80157e3a000 : NETIO!WPP_GLOBAL_Control
0xfffff580712b66c8 : 0xfffff80157dc6dcd : NETIO!WfpStreamInspectReceive+0x18d
0xfffff580712b66d8 : 0xfffff80157e63d24 : NDIS!NdisAllocateNetBufferAndNetBufferList+0x214
0xfffff580712b66e0 : 0xfffff80157dc26d0 : NETIO!NetioCompleteCopyNetBufferListChain
0xfffff580712b6758 : 0xfffff8015931eca0 : tcpip!InetInspectReceive+0x80
0xfffff580712b6768 : 0xfffff8015931ee00 : tcpip!TcpCovetNetBufferList+0xf8
0xfffff580712b6798 : 0xfffff8015931ee39 : tcpip!TcpCovetNetBufferList+0x131
0xfffff580712b6808 : 0xfffff8015931ebb8 : tcpip!TcpInspectReceive+0x9c
0xfffff580712b6988 : 0xfffff80159345ede : tcpip!WfpTlShimInspectRecvTcpDatagram+0x2de
0xfffff580712b6ba8 : 0xfffff80159302a52 : tcpip!IpNlpFastContinueSendDatagrams+0x502
0xfffff580712b6bf8 : 0xfffff801592e77a1 : tcpip!TcpTcbExtractDatagram+0xd1
0xfffff580712b6c80 : 0xfffff801594d1938 : tcpip!AddressFamilyInformation
0xfffff580712b6ce8 : 0xfffff801592e54ee : tcpip!TcpTcbReceive+0x30e
0xfffff580712b6cf8 : 0xfffff801593039e4 : tcpip!IppSendDatagramsCommon+0x4c4
0xfffff580712b6e48 : 0xffffcd0c14f740c0 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b6e50 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b6e78 : 0xfffff801593024a9 : tcpip!IpNlpFastSendDatagram+0x349
0xfffff580712b6e90 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b6ea8 : 0xfffff80157e63ee3 : NDIS!NdisAllocateNetBufferList+0xc3
0xfffff580712b6eb0 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b6ef8 : 0xfffff80159308897 : tcpip!ShimIpPacketInV4+0x93
0xfffff580712b6f38 : 0xfffff801592e470f : tcpip!TcpMatchReceive+0x51f
[COLOR=rgb(184, 49, 47)]*** WARNING: Unable to verify timestamp for KfeCo10X64.sys[/COLOR]
0xfffff580712b70a0 : 0xfffff80158e07a90 : fwpkclnt!FwppInjectComplete
0xfffff580712b70d8 : 0xfffff801529b1019 : nt!ExFreePool+0x9
0xfffff580712b7148 : 0xfffff801592e0d4e : tcpip!TcpValidateReceive+0xee
0xfffff580712b7158 : 0xfffff801594cec00 : tcpip!Fl6tpProviderState+0x6f0
0xfffff580712b7168 : 0xfffff80157dc5b64 : NETIO!NetioDereferenceNetBufferListChain+0x104
0xfffff580712b71e8 : 0xfffff801592e3798 : tcpip!TcpReceive+0x358
0xfffff580712b71f8 : 0xfffff801594cec00 : tcpip!Fl6tpProviderState+0x6f0
0xfffff580712b7208 : 0xfffff801594c9598 : tcpip!Ipv4Global+0x368
0xfffff580712b7218 : 0xfffff8015942bb01 : tcpip!McTemplateK0qbr0qbr2qqqqqqqqqqqqqqqq_EtwWriteTransfer+0xe9
0xfffff580712b7230 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b72a0 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b72a8 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b72b8 : 0xffffcd0c14f740c0 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b72d0 : 0xfffff801594c9598 : tcpip!Ipv4Global+0x368
0xfffff580712b72d8 : 0xfffff80159349142 : tcpip!TcpNlClientReceiveDatagrams+0x22
0xfffff580712b7318 : 0xfffff8015930a241 : tcpip!IppProcessDeliverList+0xc1
0xfffff580712b7348 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b73a8 : 0xfffff801522083eb : nt!KeQueryCurrentStackInformationEx+0x8b
0xfffff580712b73d0 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b73f8 : 0xfffff80159306b8b : tcpip!IppReceiveHeaderBatch+0x21b
0xfffff580712b7408 : 0xfffff8015235460e : nt!KiExpandKernelStackAndCalloutSwitchStack+0x11e
0xfffff580712b74e0 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b74f8 : 0xfffff8015930610f : tcpip!IppFlcReceivePacketsCore+0x32f
0xfffff580712b7500 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b7548 : 0xfffff801594c9230 : tcpip!Ipv4Global
0xfffff580712b75c8 : 0xfffff801592f2e86 : tcpip!RtlAcquireReadLock+0x42
0xfffff580712b7618 : 0xfffff80159430152 : tcpip!IppInspectInjectReceiveEx+0x172
0xfffff580712b7658 : 0xfffff80158e086a0 : fwpkclnt!FwppInjectionStackCallout
0xfffff580712b7668 : 0xfffff8015942ffd4 : tcpip!IppInspectInjectReceive+0x24
0xfffff580712b76c8 : 0xfffff80158e087b6 : fwpkclnt!FwppInjectionStackCallout+0x116
0xfffff580712b76d0 : 0xfffff80158e086a0 : fwpkclnt!FwppInjectionStackCallout
0xfffff580712b76e8 : 0xfffff8015235460e : nt!KiExpandKernelStackAndCalloutSwitchStack+0x11e
0xfffff580712b7738 : 0xfffff80158e086a0 : fwpkclnt!FwppInjectionStackCallout
0xfffff580712b7758 : 0xfffff80152354488 : nt!KeExpandKernelStackAndCalloutInternal+0x78
0xfffff580712b77c8 : 0xfffff801523543fd : nt!KeExpandKernelStackAndCalloutEx+0x1d
0xfffff580712b77d0 : 0xfffff80158e086a0 : fwpkclnt!FwppInjectionStackCallout
0xfffff580712b77f8 : 0xfffff80158e07fd7 : fwpkclnt!FwppInjectPrologue+0x13b
0xfffff580712b7808 : 0xfffff80158e0a2b4 : fwpkclnt!NetioExpandKernelStackAndCallout+0x58
0xfffff580712b7848 : 0xfffff80158e09ea4 : fwpkclnt!FwpsInjectTransportReceiveAsync0+0x304
0xfffff580712b7b08 : 0xfffff80152317e25 : nt!PspSystemThreadStartup+0x55
0xfffff580712b7b58 : 0xfffff801523fcdd8 : nt!KiStartSystemThread+0x28
0xfffff580712b7b70 : 0xfffff80152317dd0 : nt!PspSystemThreadStartup
 
Last edited:
Many thanks for your reply and welcome.

Is any thing in memory.dmp which wold point to something else than NIC driver? I have 8 different machines with different NIC models (Intel as well), on some I've already updated drivers - and this didn't help...
 
Without seeing the crash dumps from all 8 machines it would be difficult to give an opinion. If any of the other 8 do not have the Rivet Killer Network device I would be interested in seeing their crash dumps.
 
There are tcpip.sys and fwpkclnt.sys, both microsoft drivers.
Fwpkclnt.sys could have something to do with security, hence firewall/antivirus.
You could try to uninstall eset altogether in this machine to see if it resolves the issue.
 
Last edited:
It's not do with the NIC directly, it's related to one of the network filter drivers, which usually end up causing issues since the level of privilege they have within the kernel.

Rich (BB code): 
12: kd> .trap 0xfffff580712b56f0
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000000000 rbx=0000000000000000 rcx=fffff580712b59f8
rdx=ffffcd0c332eb7c0 rsi=0000000000000000 rdi=0000000000000000
rip=fffff80157dcd7f4 rsp=fffff580712b5880 rbp=fffff580712b58f9
 r8=ffffcd0c332eb7c0  r9=0000000000000045 r10=ffffcd0c347693f0
r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl zr na po nc
NETIO!StreamInvokeCalloutAndNormalizeAction+0x5c:
fffff801`57dcd7f4 488b4808        mov     rcx,qword ptr [rax+8] ds:00000000`00000008=????????????????

The crash was caused because a driver used a pointer which contains a garbage address and hence couldn't be resolved by the page fault handler. I would be suspicous of ESET due to the issues I've seen it cause in the past. It would be worthwhile seeing the other crash dumps though.
 
Thank you for all your replies. I will organise other dump file from machine which doesn't have killer NIC - they show the same reason - DRIVER_IRQL_NOT_LESS_OR_EQUAL and NETIO.SYS. We are on latest version of ESET and we are using it on around 500 machines so this is why I didn't suspect this.
I'm attaching crash dump from another XPS with killer.
 

Attachments

PiotrIr said:
Thank you for all your replies. I will organise other dump file from machine which doesn't have killer NIC - they show the same reason - DRIVER_IRQL_NOT_LESS_OR_EQUAL and NETIO.SYS. We are on latest version of ESET and we are using it on around 500 machines so this is why I didn't suspect this.
I'm attaching crash dump from another XPS with killer.
Click to expand...

5 minidumps for this machine. They say BugCheck 116, {DifferentAddressForAllDumps, DifferentAddressForAllDumps, 0, d}
Probably caused by : nvlddmkm.sys.

Resolution:
  • You may need to install the latest updates for your display driver, so that it properly supports the TDR process.
  • Hardware issues that impact the ability of the video card to operate properly, including:
    • Over-clocked components, such as the motherboard
    • Incorrect component compatibility and settings (especially memory configuration and timings)
    • Insufficient system cooling
    • Insufficient system power
    • Defective parts (memory modules, motherboards, etc.)
  • Visual effects, or too many programs running in the background may be slowing your PC down so that the video card can not respond as necessary.
Given that the driver is already updated, you can try the NVIDIA Studio Driver (instead of game ready).
If that won't work:
- load bios (optimized) defaults
- run memtest (see hardware tutorials).
 
Last edited:
Hmm, as all 8 laptops are on similar set (O365, Intune, Safetica, ESET, Duo) I was suspecting the same reason for all of them, especially memory dump points to the same - DRIVER_IRQL_NOT_LESS_OR_EQUAL and NETIO.SYS.
 
PiotrIr said:
This is crash dump from laptop which doesn't have the Killer NIC.
I just wander - is i possible to list network filter drivers which may be responsible for this issue?
Click to expand...
This set of logfiles for the laptop without the Killer driver does not show any BSOD crash dumps.

What I can see that is common to both laptop type whether Killer or Intel is the driver called:
netmonitor_wfp.sys Thu Jul 2 11:14:53 2020

This was implicated in the crash analysis I did for you in my first post. I could not determine where this driver came from, perhaps you will know?
 
If you're able could you please check the following directory on one of the problematic machines:

Code: 
%systemroot%\MEMORY.DMP

The file will need to be zipped and then uploaded to a file sharing site such as DropBox or OneDrive.

I want try and see if I find what netmonitor_wfp.sys belongs to.
 
Thank you! I'm really grateful for your help.

Yes, netmonitor_wfp belongs to Safetca and this may cause BOSD (it has in past). However is past I got clear pointing in memory.dmp which showed this. I can log a call with their support but somehow need to proof this is their issue.
Is it possible to list filters attached to network adapter? Do they have also order which matters? I would like to have something to compare with laptops which actually work...

Dump file from EliteBook below:
MEMORY.zip
 
Thanks for providing me with the dump file, I'll have a look into it and see what I can find.
 
There were 3 different software (multiple drivers) and 1 hardware driver seen in the memory crash dump file.





Netwtw06.sys Intel Dual Band Wireless-AC 8265 wifi driver Download Windows® 10 Wi-Fi Drivers for Intel® Wireless Adapters



npcap.sys
Insecure.Org - Nmap Free Security Scanner, Tools & Hacking resources



em042_64.dll - em042_64
epfw.sys
epfwwfp.sys
em008k_64.dl
dlpfde.sys

ESET has support for BSOD troubleshooting:
Contact ESET Technical Support | ESET
[KB7686] Blue Screen (BSOD) in ESET Endpoint Encryption



netmonitor_wfp.sys
Safetica | Data Loss Prevention
 
Rich (BB code): 
ffff868e`b07942f8  fffff804`6b3d4b29 nt!KiBugCheckDispatch+0x69
ffff868e`b0794300  00000000`0000000a
ffff868e`b0794308  00000000`00000008
ffff868e`b0794310  00000000`00000002
ffff868e`b0794318  00000000`00000000
ffff868e`b0794320  fffff804`6f06bdb2 NETIO!StreamInvokeCalloutAndNormalizeAction+0x62
[...]
ffff868e`b07943e8  ffff868e`b0794630
ffff868e`b07943f0  ffff0584`a0635c23
ffff868e`b07943f8  fffff804`73a1b39b epfwwfp+0xb39b
ffff868e`b0794400  ffffcd02`978892b0
ffff868e`b0794408  ffff868e`b0794750
ffff868e`b0794410  ffffcd02`978892b0
ffff868e`b0794418  ffff868e`b0794a70
ffff868e`b0794420  ffff868e`b0794a00
ffff868e`b0794428  ffff868e`b0794728
ffff868e`b0794430  ffffcd02`94ccc4b0
ffff868e`b0794438  fffff804`6b3d0e69 nt!KiPageFault+0x469 << Crash because of the illegal page fault
[...]
ffff868e`b07944a8  fffff804`673c4450 netmonitor_wfp+0x14450 << Suspect this driver
[...]
ffff868e`b0794528  fffff804`767a1b41 Netwtw06+0x1b1b41 << Intel Wi-Fi driver
ffff868e`b0794530  ffff868e`b0794ef8
ffff868e`b0794538  fffff804`73a13257 epfwwfp+0x3257 << ESET firewall driver
ffff868e`b0794540  ffff868e`b07946d0
ffff868e`b0794548  fffff804`766cd879 Netwtw06+0xdd879
ffff868e`b0794550  00000000`00000000
ffff868e`b0794558  ffff868e`b0794728
ffff868e`b0794560  00000000`00000000
ffff868e`b0794568  ffff868e`b0794a70
ffff868e`b0794570  fffff804`6f0d7000 NETIO!WPP_GLOBAL_Control
ffff868e`b0794578  fffff804`673b148f netmonitor_wfp+0x148f
[...]
ffff868e`b07945a8  fffff804`6f06bdb2 NETIO!StreamInvokeCalloutAndNormalizeAction+0x62

The reason I'm suspicious of the Safetca driver is because it's the last third-party callout driver which was present at the time of the crash.

Rich (BB code): 
3: kd> knL
 # Child-SP          RetAddr           Call Site
00 ffff868e`b07942f8 fffff804`6b3d4b29 nt!KeBugCheckEx
01 ffff868e`b0794300 fffff804`6b3d0e69 nt!KiBugCheckDispatch+0x69
02 ffff868e`b0794440 fffff804`6f06bdb2 nt!KiPageFault+0x469
03 ffff868e`b07945d0 fffff804`6f06b879 NETIO!StreamInvokeCalloutAndNormalizeAction+0x62
04 ffff868e`b07946a0 fffff804`6f06aed9 NETIO!StreamProcessCallout+0x2c9
05 ffff868e`b07947e0 fffff804`6f06d1da NETIO!ProcessCallout+0x5a9
06 ffff868e`b0794960 fffff804`6f069b91 NETIO!ArbitrateAndEnforce+0xc3a
07 ffff868e`b0794af0 fffff804`6f06933a NETIO!KfdClassify+0x5a1
08 ffff868e`b0794f00 fffff804`6f068f63 NETIO!StreamClassify+0x26a
09 ffff868e`b0795090 fffff804`6f068bdc NETIO!StreamCommonInspect+0x293
0a ffff868e`b0795470 fffff804`6f5ed0b8 NETIO!WfpStreamInspectReceive+0x19
[...]

Essentially, what is happening is the NIC receives a TCP packet which is then passed through the network stack and then passed to the filter engine which then executes the filter callout drivers. In this case, it's the ESET firewall and the Safetica monitoring program, which will inspect the network data byte stream and do what they've been written to do.

More information is available here - Windows Filtering Platform Architecture Overview - Windows drivers

Here's a list of the third-party callout filter drivers which are possible culprits:

Rich (BB code): 
3: kd> r $t0=ffffcd02`85935000;.for ( r $t1=0; @$t1 < 400; r $t1=@$t1+1 ) {dps @$t0+2*@$ptrsize L2; r $t0=@$t0+50;}
[...]
ffffcd02`8593b0a8  fffff804`736f8d40 epfw+0x8d40 << ESET
[...]
ffffcd02`8593b0f0  fffff804`73a13ce0 epfwwfp+0x3ce0 << ESET Firewall Driver
[...]
ffffcd02`8593b4b0  fffff804`673b1460 netmonitor_wfp+0x1460 << Safetca Driver
[...]
 

Has Sysnative Forums helped you? Please consider donating to help us support the site!

Back
Top