I seem to have fixed this by completely uninstalling the device via Device Manager, then reinstalling it using the same utility I originally used to update the driver. Maybe it was a busted driver?
I'll leave LatencyMon running, but I'm not having any of the issues I was previously having.
EDIT: Within 5 minutes of me posting this the issues started back up again. They're less frequent now, but still very existent.