storport.sys Latency

[bashrah]

Hi everyone i cant help myself anymore.... first i get latency issues with ntoskrnl.sys wich get fixed by the latest windows update and now its storport.sys. i replaced my mainboard already because i thought its broken.

My system: MSI X370 GAMING PRO CARBON (MS-7A32) Performance Results - UserBenchmark


i registred in this forum because i found this post : [SOLVED] - Very high DPC Latency by storport.sys and ndis.sys and i hope you guys can help me with analysing the xperf file


like TomasD wrote i used xperf:


xperf -on PROC_THREAD+LOADER+PROFILE+INTERRUPT+DPC+DRIVERS+POWER+IDLE_STATES -stackwalk Profile -BufferSize 1024 -MinBuffers 256 -MaxBuffers 256 -MaxFile 256 -FileMode Circular

-

xperf -stop -d C:\CPU.etl

at the stop command i get a error, but a file called kernel.etl maybe its the same?!

kernel.etl

 

[cwsink]

Hi bashrah - welcome to Sysnative

All of the DPCs are running for an acceptable amount of time from what I can see. What symptoms are you experiencing that suggest there's a DPC latency problem? Were you experiencing latency issues when you made the trace? What are you using that is telling you storport.sys is having problems?

 

[bashrah]

hi i have sound stutters and my fps is going down to 1 for less than a second. When im in idle it happens only every hour and when i play games every 20 to 40 minutes.
yes i had issues while making the trace and i stop the trace close after it happens.

i use LatencyMon.
Driver with highest DPC routine execution time: storport.sys
Driver with highest DPC total execution time: dxgkrnl.sys <- yesterday it was also Wdf01000.sys

i have the newest Realtek audio, intel network, graphics and chipset driver.

 

[cwsink]

The trace doesn't show anything even close to what LatencyMon is showing. The highest reading for a storport.sys DPC in the trace is 0.069200ms compared to 100ms in the LatencyMon stats. 100ms is very high and typically indicates a problem reading/writing to a drive. Have you checked the health of your drives and also for firmware updates? Are you using the latest UEFI/BIOS for your motherboard? Have you updated to the latest AMD Chipset drivers?

 

[bashrah]

yes i have latest chipset driver and bios
the 3 SSDs and the seagate hdd have the newest firmware and i test them with seatools, samsung magican and WD SSD dashboard
i throw out all my old hdds and let latencymon check again and the problem is still there.
i tried xperf again but i get this error:
xperf: error: NT Kernel Logger: Cannot create a file when that file already exists

but still i have a new kernel.etl file.
2kernel.etl

 

[cwsink]

I get the impression LatencyMon can interfere with xperf traces so please uninstall it, reboot, and continue with xperf for now. I'd also delete kernel.etl from the root of your system drive. It is supposed to just be a temporary file used by xperf. I'd suggest using the attached script to capture another trace as it will keep the trace below 300MB no matter how long you run it (it creates a circular buffer in memory rather than continuously writing to a file). It's possible writing the trace to disk during the capture is interfering with the disk I/O performance readings. The kernel.etl file will likely reappear when the script is run but don't delete it; at least not until it's finished merging out the trace as explained below.

Extract the .cmd file from the zip to your Desktop, right-click that script icon and select "Run as administrator" and a command prompt window should open. Press a key to start the trace and leave the command windows open running in the background. Use the system until you experience the issue you're having and switch back to the command prompt window, press a key, and the the trace should be merged out to a file on your Desktop named "trace.etl". It's about a 30 to 60 second circular buffer so you should have plenty of time to switch back to the command prompt window after reproducing the issue. Please zip up the trace.etl file and make it available for download. It will likely be too big to attach so you might need to use a cloud drive or file sharing service.

 

[bashrah]

Trace.zip

 

[cwsink]

Okay, so I see a spike where a number of disk I/O related DPCs are each running for almost 10ms (easily long enough to cause audio glitches.) They look to be storport.sys DPCs defined by RaidpAdapterTimerDpcRoutine but the function which is actually taking a long time within those DPCs is one defined by amd_sata.sys. I believe this is a driver that gets installed with the AMD Chipset installer. Have you installed the latest for your motherboard from here? If not, please do so and test the system afterwards.

edit: I'd also install the AMD RAID installer from that same page. That could be what actually has the amd_sata.sys driver.

 

[bashrah]

thank you!!
yes
i installed this chipset drivers but i never touched anything that have something to do with RAID.
when i try to install the amd raid driver it shows me:

edit: i found some guide to manually install the driver but i dont see "AHCI... Raid...." at my storage controllers. devicemanager.png

 

[cwsink]

It may not apply to your system configuration. I wouldn't try to force it. Can you clarify if you meant you had already installed the chipset drivers I linked above prior to your last trace or you have since installed those drivers?

edit: FYI, other than that spike the trace looks extremely good as far as DPC processing.

 

[bashrah]

yes i installed the x370 64bit chipset driver before the trace.

i read a bit about the amd_sata.sys and problems with the raid driver installation.
some people say something about that the AMD Sata Controller didnt get updated since 2015
and the microsoft version shows 2006 but have the newest version number 10.0.18362.1
so i deactivated the amd sata controller and use now the one from MS maybe that solve the problem?
i let latencyMon run and check

 

[cwsink]

Thank you for clarifying it for me. The version number of the Microsoft driver is the same as the build version of Windows 10 1903 so I think they simply use the same 2006 driver but give it a new version number for whatever reason. It probably makes it easier internally at Microsoft, I suspect. I'll post some screenshots of what I'm seeing hoping more eyes might be able to spot something I'm missing.

 

[cwsink]

It's my understanding that audio playback always happens on core 0 so I've filtered out the other cores to reduce clutter. I've also zoomed in on the spike which is the long part above 6 percent in the middle of the graph. The blue rows in the table correspond to the blue region in the graph and I've sorted it by Fragment Enter Time. The table extends beyond the bottom of the screen but it basically shows storport.sys and HDAudBus.sys DPCs using core 0 for a little over 100ms total. Storport.sys is grabbing and keeping core 0 for as long as 9ms at a time, though, and that will definitely cause audible pops/glitches. I think this is a good example of why an xperf trace is more useful than LatencyMon at trying to find the root cause of such problems. It seems like the DPCs generated by those 2 drivers are preventing anything else from getting CPU time on core 0 as well - which is a long time to not be getting video or network related DPCs.

I'm still looking into it but if anyone else has thoughts or suggestions please join in.



edit: To clarify, the storport.sys DPC is taking just over 100ms to complete (it's a single DPC from that driver that is taking that long to be completed) but it's getting pre-empted by multiple HDAudBus.sys DPCs over that time period.

 

[cwsink]

Actually, those aren't HDAudBus.sys DPCs. They are HDAudBus interrupts which is why they are able to pre-empt the storport.sys DPCs. So, I think it's correct to say that something is going wrong during drive I/O which is causing some storport.sys DPCs to take a very long time.

Those storport.sys DPCs are calling a amd_sata.sys driver function which itself is calling a hal.dll function. I think it's safe to assume the storport.sys driver and hal.dll binary are not the culprit which leaves amd_sata.sys or an undiagnosed hardware problem as the more likely cause.

 

[bashrah]

i let latencymon run and the problem is gone there are still some high latencys but no pops/glitches thank you very much for your help!

 

[cwsink]

Thank you for the update. One of the things LatencyMon doesn't seem to take into account is that Windows 10 takes better advantage of multiple cores for processing DPCs and ISRs. I often see it report somewhat high DPC processing time for dxgkrnl.sys and ndis.sys but my guess is those particular DPCs are not being processed on core 0 so they wouldn't interfere with audio playback.

I'm not sure what has changed on your system that would have fixed the issue. Do you know if something got updated? I was under the impression what we'd tried was redundant or wasn't installing.

 

[bashrah]

i just deactivated the AMD SATA Controller
so i guess the system is now using the other one SATA AHCI- Controller

-
The AMD SATA controller component is not compatible with and will not be offered nor installed by the chipset driver package on AMD desktop systems running Window® 10 or using AMD Socket AM4.....

.....Socket AM4, and Socket TR4 chipsets running Windows® 10 should use the Microsoft provided Standard SATA AHCI Controller driver that comes installed with Windows® by default.

https://www.amd.com/en/support/kb/faq/pa-250
-

maybe i get it from one of this automatic driver install. programms like "Driver booster" or "Driver Genius"

 

[cwsink]

Thank you for the info and nice find with the support link. :) This might be yet another data point in support of my distrust of 3rd party driver update programs. Personally, I only use Windows Update or a device manufacturer's product support site for drivers.

Please do let us know if the problems return.