1. #1

    [Server2012R2 Hyper-V] Two Server2008 boxes just hang at random.

    We have a 2012 R2 Hyper-V server that houses a handful of systems. Two of those systems are Windows 2008 servers (We're stuck with them for now.) Both of the Windows 2008 servers hang whenever they feel like it, for no good reason. No blue screen, the systems are pingable, but control-alt-delete does nothing at the login screens and we can not browse to their shared drives. We have to just reset them.

    Hangs are inconsistent in regards to how long the system will run. Usually we'll get a few weeks out of them, sometimes months, other times days. This morning the system hung and we reset it. An hour later, hung again. Never been in the hour realm before.

    I forced a blue screen on that particular box this morning to see if I could get anything out of it, but I'm unable to make heads or tails. Would anyone be willing to look at the mini-dump and see if they see anything? I realize the mini-dump isn't optimal (Or maybe completely useless.) Unfortunately, I have to assume the memory.dmp is corrupt as it won't let me view it, copy it to Dropbox, zip it, anything.

    SysnativeFileCollectionApp.zip


    • Ad Bot

      advertising
      Beep.

        
       

  2. #2
    Moderator
    BSOD Kernel Dump Analyst
    Windows Update Senior Analyst
    softwaremaniac's Avatar
    Join Date
    Oct 2014
    Location
    Croatia
    Age
    23
    Posts
    12,170
    • specs System Specs
      • Motherboard:
        ASUS MAXIMUS ROG HERO X
      • CPU:
        Intel Core i7-8700K 3.7GHz
      • Memory:
        Crucial 2x8GB DDR4 2666 MHz
      • Graphics:
        Gigabyte GTX 1080 G1 Gaming 8 GB
      • Sound Card:
        Asus Xonar DSX
      • Hard Drives:
        WD Caviar Black 1TB SATA III 7200rpm, WD Caviar Black 6TB SATA III 7200rpm
      • Disk Drives:
        Samsung 960 Evo 256GB NVME PCIe
      • Power Supply:
        Corsair HX 750W 80+ Platinum
      • Case:
        Fractal Design Define R6
      • Cooling:
        Noctua NH-D14
      • Display:
        Philips Brilliance BDM4065UC 4K 3840x2160
      • Operating System:
        Windows 10 Pro 1803 x64

    Re: Two server 2008 boxes under Hyper-V just hang at random

    Dump is indicative of a HW failure. Please start hardware testing: Initial Hardware Diagnostics

  3. #3

    Re: Two server 2008 boxes under Hyper-V just hang at random

    HW failure? On a virtual guest? If they were all having problems, sure, but we've got another dozen machines that run just fine on that physical box.

  4. #4
    Moderator
    BSOD Kernel Dump Analyst
    Windows Update Senior Analyst
    softwaremaniac's Avatar
    Join Date
    Oct 2014
    Location
    Croatia
    Age
    23
    Posts
    12,170
    • specs System Specs
      • Motherboard:
        ASUS MAXIMUS ROG HERO X
      • CPU:
        Intel Core i7-8700K 3.7GHz
      • Memory:
        Crucial 2x8GB DDR4 2666 MHz
      • Graphics:
        Gigabyte GTX 1080 G1 Gaming 8 GB
      • Sound Card:
        Asus Xonar DSX
      • Hard Drives:
        WD Caviar Black 1TB SATA III 7200rpm, WD Caviar Black 6TB SATA III 7200rpm
      • Disk Drives:
        Samsung 960 Evo 256GB NVME PCIe
      • Power Supply:
        Corsair HX 750W 80+ Platinum
      • Case:
        Fractal Design Define R6
      • Cooling:
        Noctua NH-D14
      • Display:
        Philips Brilliance BDM4065UC 4K 3840x2160
      • Operating System:
        Windows 10 Pro 1803 x64

    Re: Two server 2008 boxes under Hyper-V just hang at random

    Well, you have to assign a certain amount of resources to a VM, it could be that the two problematic VMs are on bad blocks. I'd still run tests, btw, if you run sfc on the two VMs what are the results?

  5. #5

    Join Date
    Dec 2014
    Location
    Germany
    Posts
    179

    Re: Two server 2008 boxes under Hyper-V just hang at random

    Funny,
    Mini103117-01
    Symantec Tamper Protection caused an NMI.
    Underlying Host Filesystem or whatever.

    Inspect the Logs (Host) and configuration of Symantec Tamper Protection.

    honestly, mini minidumps for Server^^ with almost nothing in it.
    >>
    Mini Kernel Dump File: Only registers and stack trace are available


  6. #6
    jcgriff2's Avatar
    Join Date
    Feb 2012
    Location
    New Jersey Shore
    Posts
    16,424
    • specs System Specs
      • Manufacturer:
        HP
      • Model Number:
        HP ENVY TouchSmart 17-j130us Notebook - E8A04UA
      • Motherboard:
        HP Insyde 720265-501 6050A2549501-MB-A02
      • CPU:
        Intel Core i7-4700MQ Processor with Turbo Boost up to 3.4GHz.
      • Memory:
        12GB DDR3L SDRAM (2 DIMM)
      • Graphics:
        Intel HD graphics 4600 with up to 1792MB total graphics memory
      • Sound Card:
        Beats Audio quad speakers and two subwoofers
      • Hard Drives:
        1TB 5400RPM hard drive with HP ProtectSmart Hard Drive Protection
      • Disk Drives:
        Hitachi 500 GB SSD; 7 TB USB External
      • Power Supply:
        90w
      • Case:
        Laptop
      • Display:
        17.3-inch diagonal HD+ BrightView LED-backlit touchscreen display (1600 x 900)
      • Operating System:
        Windows 8.1

    Re: Two server 2008 boxes under Hyper-V just hang at random

    Is your RAM matched?

    1 stick is shown as 4,160,749,568 bytes; the other 134,217,728 bytes.

    The bugcheck was a very rare 0x80. I've never seen one before and I've been doing this for just over 10 years now.

    Code:
    NMI_HARDWARE_FAILURE (80)
    This is typically due to a hardware malfunction.  
    The hardware supplier should be called.
    Bug Check 0x80: NMI_HARDWARE_FAILURE
    The NMI_HARDWARE_FAILURE bug check has a value of 0x00000080. This bug check indicates that a hardware malfunction has occurred.

    Parameters

    None

    Cause
    A variety of hardware malfunctions can cause the NMI_HARDWARE_FAILURE bug check. The exact cause is difficult to determine.

    Resolution
    Remove any hardware or drivers that have been recently installed. Make sure that all memory modules are of the same type.
    Of course BSOD dumps are really only useful for finding software (driver) problems. That was Windbg's original intent.

    Sometimes, Windbg may point us in a certain general hardware direction, but the dumps are incapable of telling us the exact piece of failed hardware. That's why you often see the probable cause listed as "unknown hardware failure".

    Regards. . .

    jcgriff2
    MichaelB says thanks for this.

  7. #7

    Join Date
    Dec 2014
    Location
    Germany
    Posts
    179

    Re: Two server 2008 boxes under Hyper-V just hang at random

    Congrats John,

    indeed while smbios does not show the amount of memory at all, its cut off
    but
    Code:
    [System Configuration Options (Type 12) - Length 5 - Handle 004eh]
    [32Bit Memory Error Information (Type 18) - Length 23 - Handle 0050h]
    [Physical Memory Array (Type 16) - Length 15 - Handle 0051h]
      Location                      02h - Unknown
      Use                           03h - System Memory
      Memory Error Correction       03h - None
      Maximum Capacity              [Unknown]
      Memory Error Inf Handle       0050h
      Number of Memory Devices      64
    [Memory Array Mapped Address (Type 19) - Length 15 - Handle 0052h]
    This Error is mapped to actual
    Code:
    0: kd> !handle 0050h
    GetPointerFromAddress: unable to read from fffff80005a85000
    
    PROCESS fffffa83045c8040
        SessionId: none  Cid: 0880    Peb: 7fffffd6000  ParentCid: 0304
        DirBase: c1613000  ObjectTable: fffff880127b2840  HandleCount: <Data Not Accessible>
        Image: LTSVC.exe
    in other dumps it looks similar, errorhandle memory points to failing dll's / modules

  8. #8

    Re: Two server 2008 boxes under Hyper-V just hang at random

    Ahhh, I think I know why I have everyone thrown for a loop.

    As I said in my original post, I FORCED a blue screen to see if I could get any information. To do this in Hyper-V 2012 R2, you can generate an NMI call - "Debug-VM -Name "VM Name" -InjectNonMaskableInterrupt -ComputerName Hostname "

    So that's why you guys are probably seeing stuff you've NEVER seen before! :)

  9. #9

    Join Date
    Dec 2014
    Location
    Germany
    Posts
    179

    Re: Two server 2008 boxes under Hyper-V just hang at random

    mh, i dont think being caught in a loop

    Code:
    Mini062714-01.dmp
    BugCheck 50, {fffffa8306041000, 0, fffffa8303842ed0, 0}
    Debug session time: Fri Jun 27 08:02:04.890 2014 (UTC + 1:00) // when it happened
    
    System Configuration Options (Type 12) - Length 5 - Handle 004eh]
    [32Bit Memory Error Information (Type 18) - Length 23 - Handle 0050h]
    [Physical Memory Array (Type 16) - Length 15 - Handle 0051h]
      Location                      02h - Unknown
      Use                           03h - System Memory
      Memory Error Correction       03h - None
      Maximum Capacity              [Unknown]
      Memory Error Inf Handle       0050h
      Number of Memory Devices      64
    PROCESS fffffa8304cfe7f0
        SessionId: none  Cid: 1844    Peb: 7efdf000  ParentCid: 1108
        DirBase: 100da1000  ObjectTable: fffff8801dfab580  HandleCount: <Data Not Accessible>
        Image: Lua.exe 
    
    Mini070714-01.dmp
    BugCheck 50, {fffffa83060f9000, 0, fffffa8306ac0ed0, 0}
    Debug session time: Mon Jul  7 11:46:18.430 2014 (UTC + 1:00) // when it happened
    
    [System Configuration Options (Type 12) - Length 5 - Handle 004eh]
    [32Bit Memory Error Information (Type 18) - Length 23 - Handle 0050h]
    [Physical Memory Array (Type 16) - Length 15 - Handle 0051h]
      Location                      02h - Unknown
      Use                           03h - System Memory
      Memory Error Correction       03h - None
      Maximum Capacity              [Unknown]
      Memory Error Inf Handle       0050h
      Number of Memory Devices      64
    
    PROCESS fffffa830388a170
        SessionId: none  Cid: 181c    Peb: 7efdf000  ParentCid: 100c
        DirBase: 9b1a4000  ObjectTable: fffff8800e8e7ab0  HandleCount: <Data Not Accessible>
        Image: Lua.exe
    resolves to: Kaseya Scripting Component

    only the mentioned 0x80 NMI was forced and yes that is known at least by me
    and visible by the time stamp.

    But always the same handle shown as error as mentioned before.

    btw: a server and no ECC-Memory?
    as we only can see the Guests..........

    Usual if there is no problem with memory there is no error handle for it.

  10. #10

    Re: Two server 2008 boxes under Hyper-V just hang at random

    We got rid of Kaseya a long time ago, so not worried about those dumps from 2014.

    The server does have ECC memory in it - is it because this is a virtual guest that something like that isn't exposed?

  11. #11

    Join Date
    Dec 2014
    Location
    Germany
    Posts
    179

    Re: Two server 2008 boxes under Hyper-V just hang at random

    those dumps where included and for reference examined too.

    > The server does have ECC memory in it
    Then inspect your Host.

  12. #12

    Re: Two server 2008 boxes under Hyper-V just hang at random

    PatD,

    I was curious if you ever resolved this problem? Funny I think you are the only person I've ever seen use Debug-Vm instead of livekd.
    Taking a dump of a VM running on Hyper-V vimalsdesk

    Curious if I undertand the problem correctly.

    Windows Server 2012 R2 Hyper-V Host with 2 Guest OSE's 2008 SP2 x64. The Guests after a period of time become unresponsive and require being force off from hyper-v manager to recover from the hang.
    How long does it take for the VM's to become hung?
    This issue effects only the guest operating systems?
    At the time of the hang?
    * Can you connect to admin share of the VM?
    * Can you ping the VM?

    Obviously start with ensuring everything is up to date, that is HOST and the GUEST Operating systems and Integration services. As well as drivers & firmware on the host. You should also ensure that C-STATES/C1E is disabled on the host.
    Then disable (uninstall or unload the filter) any 3rd party filter drivers, AV and Backup software you can see this via fltmc.

    Finally if you still repro your problem after all of the above is done then you look to grab a dump and it should probably be a compelte dump however kernel at a minimum.
    And if you've never analyzed a hang dump you can see a genral overview of basics here
    Red alert! My Server is hung – what do I do? – Ntdebugging Blog

  13. #13

    Re: Two server 2008 boxes under Hyper-V just hang at random

    I had issues with livekd, so I gave up until I heard about debug-vm, but sounds like that didn't really help either.

    No, never resolved. We have to monitor and bounce the machine when issues come up.

    VMs hang at random. I've had them go weeks, I have them go hours. When they hang, I can ping the machines, but can not connect to any shares.

    This issue only affects the two 2008 boxes - all other guests on the host are fine. The machines do not hang at the same time. All power saving is turned off, all machines (Guest and host) are up-to-date for Microsoft updates, drivers, integration services. AV was uninstalled some time ago for testing purposes but made no difference.

Similar Threads

  1. [Server2008 x64] Not Installing Updates
    By ZeroEffect in forum Windows Update
    Replies: 11
    Last Post: 11-14-2017, 09:30 PM
  2. [Server2012R2] WU failure Error 800F0831
    By choder in forum Windows Update
    Replies: 1
    Last Post: 11-06-2017, 02:11 PM
  3. [SOLVED] [Server2012R2 DataCenter] Code 800F0831 - KB2973351
    By montenos in forum Windows Update
    Replies: 2
    Last Post: 10-17-2017, 03:51 PM
  4. Windows 8.1 Random Hang
    By DanID in forum Windows 8 | Windows RT
    Replies: 1
    Last Post: 03-18-2017, 01:47 PM
  5. [SOLVED] Server2012R2 No changes to Roles/Features possible
    By myreflex in forum Windows Update
    Replies: 7
    Last Post: 03-04-2016, 01:50 PM

Log in

Log in