Million pagefault latency and high DPC/ISR

Erethism

Member
Joined
Jun 24, 2016
Posts
9
For a long time I've been trying to figure out why I get audio pops. I realised they go away after a reboot, but that is not a permanent fix. A couple weeks ago I learned about latency and I've been reading and testing out different solutions for some time without finding out what causes it. I have been overclocking my system and thought that was the reason, but after resetting everything to stock I still have issues. Worst part is that these incredible pagefaults didn't happen a week ago.

Having reached a dead end I now turn to you for help.

I have followed this thread and uploaded two CPU logs as TomasD describes. You can find them here.

Dropbox - CPU3.zip
Dropbox - CPU4.zip

The one called CPU3 contains a log from yesterday not long after a reboot. I ran Witcher 3 a couple minutes and some youtube. CPU4 is a log from today with a one day uptime. Here I run CorsairLink4 (since it creates lots of pagefaults) and browse Chrome a bit.

I've been using LatencyMon until now to check for culprits. Nvidia and DirectX always show some latency, but I believe they are within acceptable levels and aren't the problem. I think perhaps it is some network issue (don't have wireless card, only use wired) or some of the Windows services, but I can't tell for sure.

Here are some photos from LatencyMon.

1.370.JPG This one is with an OC running a 6600K at 4.6 Ghz and 1.37 V

1.380.JPG Upped voltage to 1.38, didn't help.

NO OC.JPG This one is stock everything.

pagefault.JPGpossible culprits.JPGPID.JPG These are from today if they give a clue. One day uptime, stock everything.

My system is a 6600K running Windows 10 with an Asus 170 PRO GAMING motherboard and an Intel 750 SSD (400 GB) plus ZOTAC GTX770. I have 32 GB of 2666 Kingston HyperX, but I experimented also with 16 GB in the latest tests.

Thanks for your help.
 
Hi Erethism ... and welcome to the forums ...

I'm on a production Linux system at the moment, so I won't be able to look at your traces until I get back to my office. I'll take a look at them, though, as soon as I can (likely tomorrow afternoon)... Oh - and I'd recommend keeping all 32 GB of your system memory installed. It is always a good idea when facing unexplained system behavior to run diagnostics first, before moving on software solutions (since software cannot fix broken hardware). Once things pass diagnostics, move on to the software troubleshooting.

You mention that the hard pagefault counts ramped up just recently? Over the last seven to ten days? Since a faulty programs (one with a memory leak, in particular) can cause this effect, see if you can make a little list of all the software changes in that same period of time. Added programs/apps/updates - things like that. Then use your favorite seach engine and see if any system misbehavior is linked to any of your recent software additions.

Background activity can also cause high pagefault counts - see if you can have a look during those times that your system is experiencing the audio/video stuttering ... In Task Manager, System Monitor, Process Explorer, Process Monitor (whichever tool you are most comfortable with) and see if anything pops out as a high-resource hog. Updates (windows, java, apple, vendor updates [PC vendor updaters, printer vendor updaters, etc]), messaging, VOIP (skype, etc.), Windows Media Player Network sharing service, Indexing (check Indexing Optons in Control Panel), Superfetch, Windows Search, etc ..... (even the Photos apps in Win10, which tends to want to search your photo collection from time to time -- something to do with "albums")... Basically, look at anything using an unhappily large amount of resources. Many such processes you might not need at all. And for some, you can limit their activities (or simply suspend them during high usage times).

Below is a list of things I've been collecting for these issues. I suppose one I should mention first: some users whose computers had no trouble at all running a previous version of Windows find that reverting back to that version solves their problem. For some machines, it seems that the Windows 10 drivers just aren't mature enough to run without issues. Below is the rest of the list:
_______________
Some things to try for limiting audio/video glitches due to high latency, interrupt, and hard pagefault counts:


SSDs (for systems with an SSD along with a "spinning" HDD or two)
Since SSDs are dropping in price and growing in size, lately I've been recommending that the worst-offending audio-pop suffering programs/games be installed onto an SSD. Windows doesn't take up all that much room, and neither do a few games (you can still keep "saved games" and other non-demanding software installed to a larger "spinning" hard drive, along with your photo, music, and video collections).


SATA ports (this tip applies to desktops)
Some users who haven't seen any improvement by means of updating drivers have had success by altering their SATA port usage. A recent thread here concerns a system where the audio stuttering was helped by unplugging a DVD SATA drive (...and our tech Cluberti was nice enough to help explain why) ... The last few posts are where you will see mention of the SATA ports (and the nature of USB polling & the effect on latencies) .... Here's a link to that thread:
https://www.sysnative.com/forums/wi...efaults-fresh-install-usbport-sys-others.html


Hard Pagefaults
For high hard pagefaults, it can help to:
1) Limit startup processes that you don't need
2) Limit Windows-Updates file-sharing (in Settings > Updates and security > Windows Update > Advanced Options > Choose how updates are delivered > turn off "updates from more than one place" - that will limit unnecessary background network activity)
3) Limit the amount of "telemetry" (data sent to Microsoft ... choose the "Basic" option in Settings > Privacy > Feedback & Diagnostics ... and you can try "Never" as your "Feedback Frequency" while testing your latency issues)
4) Limit any Notifications that you don't need or want (Settings > System > Notifications & actions ... (for example, you don't need "Show me tips about Windows", nor many other notifications)
5) Turn off "Live" tiles on your Start Menu (to limit unnecessary background updating of any information you don't want or need)
6) Uninstall any Windows "Universal" apps that you don't want or need (Settings > System > Apps & features -- click the app you wish to uninstall) ... and for those that you don't want, but can't uninstall (some of the built-in apps) - you can still keep them from starting unwanted services (in Control Panel > Administrative Tools > Services) or startup modules (in Task Manager > Startup tab)
7) Visit the Windows Store, and make sure that all your "Universal" apps are up-to-date (updating in the background can cause high memory usage and generate hard pagefaults)
8) Test using a large set Virtual Memory range (see the "Virtual Memory" item below)


Drivers
Nvidia: Especially for those with Nvidia video cards: try downloading and installing the latest driver. Install it using the "custom" option, and the "clean install" option. You can also see if you want or need all of the background Nvidia services that come with the latest drivers, many as part of the "Nvidia GeForce Experience" - which you can choose to install or not when you choose the "custom" install method. Here's a guide that has some details on the background services installed along with the Nvidia drivers, it might help you decide if you can do without some of them:
Disable NVIDIA Streamer Service and other NVIDIA processes - gHacks Tech News

ASMedia: For some systems, removing the ASMedia USB 3.1 driver and replacing it with a generic driver helps.


Antivirus
Some users find that certain modules of their antivirus program can have a rough affect on their audio quality. For some, it is especially noticeable when using the third-party firewall. You can try reverting to using just the Windows firewall, or to turn off a feature in the third-party firewall that blocks port scans (since most routers/modems/gateways have built-in port scanning blocking, it doesn't necessarily lower your defenses, since your hardware provides the service). In a recent thread here at SysNative, a system improved after uninstalling the premium (not-free) Avast Internet Security package, and installing a version of Kaspersky antivirus. I actually like Avast quite a lot ... I've used it for years with no problems - but I use the free version (which doesn't have the firewall).


Virtual Memory & a large set range
Since some computers experience the audio/video glitching during gaming, and Latency Monitor graphs often point to high hard pagefaults as a contributing factor (as in the pagefault tips above), I like to recommend testing things with the Virtual Memory settings for your computer manually set to a generous range (generally, with the lowest value the amount of your total installed physical system memory, and the highest value twice that number). This is an overly large range = but it can prove useful for testing. [after testing, the upper range could be lowered to one and a half times the physical memory total]... The reason for the large set range is that some games are infamous for memory leaks, and the large range might give you a better chance of surviving the memory trouble caused by the game with the memory leak.
_______________

Cheers
 
Hey, Gary

Thanks for your reply and tips. I'll see if I can go through this step by step.

I have been suspecting one of my DIMMs to be faulty and therefore I've been removing and testing them separately. So far it doesn't seem to have made a difference and I only tried it two days ago (after the massive pagefault). I will restore full system memory for future troubleshooting.

I need to mention that the particular problems of latency and sound dropouts have been an issue for a long time. It would always appear over time as I rarely shut down my computer. Only when I started using LatencyMon I could see the connection.

Illustration.JPG Here you can see its behavior over time (apart from the pagefaults). The pagefault is now defaulting at 2600 at launch of the monitor test and soon spikes to 8000 and climbs over time.

Yes, the pagefaults have suddenly increased about a week ago as I was testing out different voltages. At first I thought I had to find the optimal voltage for my OC, but later it turned out not to matter. For the sake of troubleshooting I am running everything at stock until I find the solution. There haven't been any programs installed and no particular program shows up in LatencyMon to grab my attention. I have also limited the amount of programs I have installed (removed any excess) and tried to disable a few non-essential services, and at startup there are very few apps running. No games are running and yet after a fresh reboot I will see the pagefaults spike. After light usage (i.e. Chrome) I will start to see the ISR and DPC creep up.

When I look at background services who trigger pagefaults the result of which one is the worst may change. I attached one picture in the previous post that showed some PID of possible culprits. When I look in Services to see what they contain I am completely at a loss for what to disable or what I can disable safely. Nothing in my task manager seems to take extra resources either. These problems appear even though my system has a healthy amount of spare resources.

As for the list you brought:
SSD: I only have my Intel 750 installed.
SATA: I don't use SATA currently. I have two 850 EVO SSDs I will install next week when they arrive. Could extra disk space help?
Pagefaults: I will test limiting my programs even more and see how it does. I have already shut down (in desperation) most of unessential Windows updates and notifications.
Drivers: Did a custom reinstall and removing the Nvidia audio part since I heard that might be it. I also removed all of ASUS's pre-installed software. That had some positive effect, but that was before the massive pagefaults. ASMedia USB 3.1 is uninstalled as a result of this. Wdf01000.sys has sometimes shown to be an issue and I have been looking at USB devices to see if the latency comes from there. I think it calmed down after I removed the ASUS software.
Antivirus: I only use Windows one. No other software installed.
Virtual memory: I have tried turning it off completely, setting it to several GB and letting it run automatically. No immediate success.

If you have tips for diagnostics I should run or need me to provide anything specific, please let me know.

Thanks for your help so far!
 
Small update:

All the RAM is back in. I disabled the following startup items and even unplugged my LAN-cable in case access to a network might be the trigger.
Razer Comms, Razer Synapse, Virtual CloneDrive. The top four I didn't touch and the last four were already disabled.

no programs.jpg As you can see, with even no programs running and no internet access, the pagefault spiked much harder and faster than the usual 2600-8000 from a clean reboot. DSP and ISR stayed low for a short while, but went back up to the 700s soon after (which is acceptable). The other screenshots from LatencyMon shows which processes that had any pagefaults immediately after launch.

There has been an updated driver with regards to the new GTX 10xx launch, so I will try updating to that one now (seeing as NVIDIAs kernel jumps to the top). I will also upgrade to a GTX 1080 next week. Maybe that will help?

Edit: Plugging in the ethernet cable and opening up Chrome and GeForce Experience gave me this result. I have not rebooted yet. Could it be LAN-related?
Capture.JPG
 
Hi again

Thanks for the latest charts and graphs. I still haven't had a chance to fire up the Windows 10 PC and have a look with WPA at your first few traces, but I'm hoping for some time later tonight (still on Linux again, too much work to get done!)....

Your DPC, ISR, interrupts ... all these have dropped quite a lot from your latest efforts. The stubborn holdout is the pagefaults. Just out of curiousity, has the audio/video performance been better? (... since I'm guessing the pagefault issues might come and go). The odd part about the pagefaults ... if you look at the chart listing the pagefaults by process, the numbers for each process don't look bad. How interesting is that!

Keep testing the network for a bit - both with and without the LAN/WAN cards in use or disabled. Fairly early on during the Insider Preview testing period (before Windows 10 was officially released) there were performance issues on some systems linked to the ndis.sys file ... users with Intel network cards seemed to suffer more than average. So your suspicion that network hardware/drivers are involved is certainly a possibility. Early on, some got relief by reverting to an earlier driver rather than a newer driver. My preference is to try the most recent Windows 10 compliant driver first, and only try older drivers if performance is truly awful. Another Intel driver linked to performance issues during the testing period was the Intel Management Engine component. Some found relief from that, too, by reverting to older drivers (or disabling the IME altogether if it wasn't needed). Since a year has passed since Windows 10's official release, I'd expect the driver situation to be at least better than a year ago.

The 1080 looks to be a beast. It will probably give your power supply a bestial workout :)

I haven't heard much about the Nvidia drivers behavior in a system with a 1080 installed ... yet. When I'm back on my Windows 10 machine, I can take a quick look while the WPA traces are loading in (those can take a few minutes on my modest Windows 10 machines).

Should be interesting.
Cheers
 
Hi, Gary

No worries. No need to rush it, I'm just happy someone takes time to help me out :)

Audio dropouts usually occur only after long periods of uptime (> 1 week), though I can force it to happen sooner if I deliberately try to stress the system. In fact, if LatencyMon has been up for a day or more and the measurements are getting high, I can usually get a half second/one second audio "tearing" (loud and scary sound, tbh!) simply by stopping and restarting the test in LatencyMon. Audio issues seems to be related to the DPC and ISR when they get high enough over time. It was also the original problem I have been looking to solve. Now it's the pagefaults that make me scratch my head.

I have been reading up on other forums, especially with regards to GPUs. One user had similar experiences and found out his card had poor power management. A replacement card fixed it. Apparently Nvidia drivers have had issues like this for a long time. I will report back once I have tested my system with the new 1080.

Intel drivers (networking and storage) is also one of the major suspects at the moment, seeing as several people have had issues with it online. Sadly I don't have a PCI Ethernet adapter to test with and I don't know where to begin with testing older drivers. All my Intel drivers should be up to date as per the Intel Driver Update Utility. I also have two computers on a network and use one as a media server, so I'm not sure if I can disable IMEI.
Let's say the pagefiles is due to some error with the Intel 750 drive. Would the addition of the Samsung SSDs help with reducing pagefile errors? They won't be used as boot drive(s), though.

I have no wireless card installed, but I will check what happens if I disable the LAN adapter altogether.

Thanks again for your help!
 
Small update:

I installed both Asus Ramcache and Ramdisk. I made a disk of 5 GB and set it as my pagefile. Rebooted. Allocated another 6 GB as usable for Ramcache. Now my LatencyMon shows these results at launch.
ramcaheramdisk.JPG

Usage of the machine will randomly spike it like before, but seemingly not as aggressive.
asd.JPG This happened as I uploaded the screenshot above.

Surely this must be a clue? My SSDs will arrive tomorrow along with my GPU. I will give an update then.
 
Bad news. Installing the SSDs and the 1080 did not improve my situation. In fact, it got worse. As you can see in LatencyMon, both DirectX and the Nvidia driver have gone up.

1080.JPG

I also captured a new CPU log which includes browsing, Youtube, music streaming and some Witcher 3 gameplay already after several hours of uptime. Hopefully some clues will be apparent in the log.

Dropbox - CPU.zip

I'm getting desperate.

Thanks for all the help so far.
 
Hi again

Three days go by fast, nowadays! .... (never a dull moment)

Since you have all the memory back in, I wonder how it tests with a overnight run of MemTest86 - using the latest version (updated for the latest chipsets and DDR4 memory) ... it can usually run from a bootable CD or USB. You should be OK using the free edition ... MemTest86 - Offical Site of the x86 Memory Testing Tool

I imagine the pagefile should be OK on any of the SSDs ... but since the Intel 750 is so roomy, see what happens if you leave the pagefile there and try the set range - 16384 MB to 24586 MB .... just to see if it's a "memory leak" issue. Having the one drive seems like it might cut down on the come and go traffic over the system buses? (I mentioned the SSDs, simply because their performance is so far beyond a 5400 rpm HDD).

There was a system a while back, where the owner found that their wireless mouse and keyboard would slow their system down over time .... they found that by unplugging the wireless transmitter briefly, and then replugging it in - the sluggishness disappeared. If you've a wireless keyboard/mouse set: test that just to see.

I just downloaded your third CPU trace, and I'm planning on loading them all in soon. (would have done this already, if there were just a few more copies of myself somewhere) ...

Let's hope the traces will talk!
 
Hey, Gary

Sorry for the late update. I only got around to running Memtest today. It took longer than expected, so I stopped after 14 hours. This test was also while running on an overclocked system (4.6 GHz CPU and 3200 MHz RAM). No errors so far. Pagefaults still up there.

Got any clues from the CPU dumps? Any other tests I could run?

View attachment 21582View attachment 21583View attachment 21584

Latency caused by DirectX and Nvidia kernel doesn't really bother me as I feel they are not responsible for any issues (audio pops). I have the feeling it is another source to the problem.

Again, appreciate the assistance on the matter :)
 
Hi again


I took a look at the traces a while back, and nothing jumped out and said "I'm the culprit!". ... But I'll check them out again, since its been a while.

Cumulative effect?
Does the trouble still have that same symptom of getting worse over time? (You mentioned that it got better after a reboot). If yes, I'm still guessing a program or driver might have a "memory leak" sort of issue....

Not sure what is in your latest attachments... I wasn't able to open them. Attachments can be tricky to attach, and there are limits to their sizes, file types, and such. (I think there's a guide to all that somewhere here in the forums... but I can't remember just where at the moment)....

Memory test: 14 hours with no errors is a pretty good sign the memory is in good shape :)

Nvidia seems to release new drivers every other week sometimes. I get dizzy trying to keep up.... You can always check to see if yours are the latest available (not that newer is always better, but it's a chance worth taking)... I like to recommend installing the latest using the "custom" install option, and also the "clean install" option (this will uninstall your older version first). I also have heard that it can help to trim the Nvidia modules that you might not want or need (which you can do using the "custom" install options).... Here's a link to that guide again, in case you find new Nvidia drivers to try & want a memory refresher for the steps:
Disable NVIDIA Streamer Service and other NVIDIA processes - gHacks Tech News

We'll keep at it ...
 
Hello, Gary

The attachments were my fault. I was using a timed out page to send the message in, though it appeared like the attachments went through as usual. Here they are again. All they show is the fact that no errors are present yet the pagefault still spikes.

Capture.JPGIMG_3828.jpgIMG_3829.jpg

Thanks for the tips regarding the Nvidia drivers. I've already done a clean install from before where I removed the audio driver etc. that I don't need. In fact, I was on the same driver when I swapped cards. You can see on my pictures in this thread that it went from about 700 to 4000 latency once I put in the new GTX 1080, so definitely something to do with Nvidia's drivers.

I guess next I will have to badger Nvidia/Intel about their drivers.

My conclusion so far is that regardless of the results in LatencyMon (and the fact it looks scary), I don't really notice a performance hit or audio pops in daily use. At least not with the frequent restarts. Unless there are other tests you need me to perform, I plan on letting my computer run for a week or longer (until I hear audio pops/have microstutter) and try to capture it in another CPU dump. Maybe that would point to the process that causes it.

Thanks again for all your help on the matter, Gary!
 
Hi again


I loaded the three cpu traces again, and looked for clues in all the graphs and charts as well. The constant factors for ISR and DPC counts are two processes: one which you can't change (the Windows kernel - ntoskrnl.exe) and one that Nvidia updates quite a lot - the main module in its graphics drivers = nvlddmkm.sys. While these two are the source of most of your ISR and DPC counts, most of the time those counts don't look high enough to matter.

In the first two traces, if we remove the counts due to the Windows kernel, the Nvidia driver accounts for 70% in both the ISR/DPC counts. What's a bit stranger is that in the last trace, the Nvidia driver jumps to over 85% of the ISR/DPC counts (with the Windows kernel counts removed). So - well - if the ISR/DPC levels ever jump up high enough to matter, the Nvidia driver actually does rather jump up and say "look at me!" ....

But I'm still guessing that the high hard pagefaults are continuing to cause trouble, along with spiking cpu usage. The clues for that show up in the chart lists that you sorted by high pagefaults .... some high usage points to processes running inside of the Service Host groups of processes (svchost.exe). These collections of processes can add up to some serious resource usage - and it can be hard to pin down the culprit --- if you don't know where to look ....

One tool that might help you determine what process/service running inside that resource-hogging svchost.exe is causing trouble is Process Explorer - from Microsoft's SysInternals utilities. At such times that you notice audio/video lagging, you can fire up Process Explorer, select the svchost.exe host that has the high resource usage, and Process Explorer will expand to show the resource usage for each process/service running inside that host. You can then do a bit of research as to your options for limiting the resource usage for that process/service [might be something, in fact, that you can completely do without].
Process Tools from Microsoft/SysInternals:
Process Explorer (to provide real-time views of what's running)
Process Explorer
Process Monitor (to track usage over a certain period of time)
Process Monitor
Autoruns (to enable/disable processes that run at system startup ... and often continue indefinitely)
Autoruns for Windows


The offending processes can sometimes be traced to automatic updaters of all sorts, along with apps that require large collections of data to be indexed. When such indexing ruins audio/video performance, it can help to modify/limit the schedule for such processes (so they don't "stop the music").

I noticed that you have two processes with a fair amount of resource usage that are linked to something called the "Intel System Usage Report".... I'm still looking into this one. And there is a background process called "TimeBroker" that I'm also checking on. I haven't run into these two before, so I'll have to try a bit more research. I'll let you know if I find anything that might help for your system.

That's the news for now -- I'll keep checking back when I can.
 

Has Sysnative Forums helped you? Please consider donating to help us support the site!

Back
Top