Hi faust ... and welcome to the forums ...
Just to see if the results change a little, see what results you get with the newer version of Latency Monitor (the versions from 6.50 or newer are a bit more compatible with Windows 10 .... I believe you currently are using version 6.00). Also, can you see if you can produce a chart that lists the processes by high pagefault counts, as well as the one for the high latencies? (You'd then be posting three different things - the overall graph, a latency chart of processes, and a pagefault chart of processes). Let's see if it still points to the "same suspects".
Since SSDs are dropping in price and growing in size, I've lately been recommending that the worst-offending audio-pop suffering programs/games be installed onto the SSD. Windows doesn't take up all that much room, and neither do a few games (you can still keep "saved games" and other non-demanding software installed to a larger "spinning" hard drive, along with your photo, music, and video collections).
Some users who haven't seen any improvement by means of updating drivers have had success by altering their SATA port usage. A recent thread here concerns a system where the audio stuttering was helped by unplugging a DVD SATA drive (...and our tech Cluberti was nice enough to help explain why) ... The last few posts are where you will see mention of the SATA ports (and the nature of USB polling & the effect on latencies) .... Here's a link to that thread:
https://www.sysnative.com/forums/wi...efaults-fresh-install-usbport-sys-others.html
For the hard pagefaults, it can help to:
1) Limit startup processes that you don't need
2) Limit Windows-Updates file-sharing (in Settings > Updates and security > Windows Update > Advanced Options > Choose how updates are delivered > turn off "updates from more than one place" - that will limit unnecessary background network activity)
3) Limit the amount of "telemetry" (data sent to Microsoft ... choose the "Basic" option in Settings > Privacy > Feedback & Diagnostics ... and you can try "Never" as your "Feedback Frequency" while testing your latency issues)
4) Limit any Notifications that you don't need or want (Settings > System > Notifications & actions
5) Turn off "Live" tiles on your Start Menu (to limit unnecessary background updating of any information you don't want or need)
6) Uninstall any Windows "Universal" apps that you don't want or need (Settings > System > Apps & features -- click the app you wish to uninstall) ... and for those that you don't want, but can't uninstall (some of the built-in apps) - you can still keep them from starting unwanted services (in Control Panel > Administrative Tools > Services) or startup modules (in Task Manager > Startup tab)
Drivers
You mentioned that you've updated most drivers. Did you already get the latest 368.39 Nvidia driver? If yes, did you install it using the "custom" option, and the "clean install" option? If not, try that.... you might also see if your want or need all of the background Nvidia services that come with the latest drivers, many as part of the "Nvidia GeForce Experience" - which you can choose to install or not when you choose the "custom" install method. Here's a guide that has some details on the background services installed along with the Nvidia drivers, it might help you decide if you can do without some of them:
Disable NVIDIA Streamer Service and other NVIDIA processes - gHacks Tech News
Antivirus
Some users find that certain modules of their antivirus program can have a rough affect on their audio quality. For some, it is especially noticeable when using the third-party firewall. You can try reverting to using just the Windows firewall, or to turn off a feature in the third-party firewall that blocks port scans (since most routers/modems/gateways have built-in port scanning blocking, it doesn't necessarily lower your defenses, since your hardware provides the service).
See if any of this helps.