[Server2012R2 Hyper-V] Two Server2008 boxes just hang at random.

P

PatD

Well-known member
Joined
Dec 26, 2015
Posts
125
We have a 2012 R2 Hyper-V server that houses a handful of systems. Two of those systems are Windows 2008 servers (We're stuck with them for now.) Both of the Windows 2008 servers hang whenever they feel like it, for no good reason. No blue screen, the systems are pingable, but control-alt-delete does nothing at the login screens and we can not browse to their shared drives. We have to just reset them.

Hangs are inconsistent in regards to how long the system will run. Usually we'll get a few weeks out of them, sometimes months, other times days. This morning the system hung and we reset it. An hour later, hung again. Never been in the hour realm before.

I forced a blue screen on that particular box this morning to see if I could get anything out of it, but I'm unable to make heads or tails. Would anyone be willing to look at the mini-dump and see if they see anything? I realize the mini-dump isn't optimal (Or maybe completely useless.) Unfortunately, I have to assume the memory.dmp is corrupt as it won't let me view it, copy it to Dropbox, zip it, anything.

View attachment SysnativeFileCollectionApp.zip
 
Re: Two server 2008 boxes under Hyper-V just hang at random

HW failure? On a virtual guest? If they were all having problems, sure, but we've got another dozen machines that run just fine on that physical box.
 
Re: Two server 2008 boxes under Hyper-V just hang at random

Well, you have to assign a certain amount of resources to a VM, it could be that the two problematic VMs are on bad blocks. I'd still run tests, btw, if you run sfc on the two VMs what are the results?
 
Re: Two server 2008 boxes under Hyper-V just hang at random

Funny,
Mini103117-01
Symantec Tamper Protection caused an NMI.
Underlying Host Filesystem or whatever.

Inspect the Logs (Host) and configuration of Symantec Tamper Protection.

honestly, mini minidumps for Server^^ with almost nothing in it.
>>
Mini Kernel Dump File: Only registers and stack trace are available

 
Re: Two server 2008 boxes under Hyper-V just hang at random

Is your RAM matched?

1 stick is shown as 4,160,749,568 bytes; the other 134,217,728 bytes.

The bugcheck was a very rare 0x80. I've never seen one before and I've been doing this for just over 10 years now.

Code: 
NMI_HARDWARE_FAILURE (80)
This is typically due to a hardware malfunction.  
The hardware supplier should be called.

Bug Check 0x80: NMI_HARDWARE_FAILURE
The NMI_HARDWARE_FAILURE bug check has a value of 0x00000080. This bug check indicates that a hardware malfunction has occurred.

Parameters
None

Cause
A variety of hardware malfunctions can cause the NMI_HARDWARE_FAILURE bug check. The exact cause is difficult to determine.

Resolution
Remove any hardware or drivers that have been recently installed. Make sure that all memory modules are of the same type.
Click to expand...

Of course BSOD dumps are really only useful for finding software (driver) problems. That was Windbg's original intent.

Sometimes, Windbg may point us in a certain general hardware direction, but the dumps are incapable of telling us the exact piece of failed hardware. That's why you often see the probable cause listed as "unknown hardware failure".

Regards. . .

jcgriff2
 
Re: Two server 2008 boxes under Hyper-V just hang at random

Congrats John,

indeed while smbios does not show the amount of memory at all, its cut off
but
Code: 
[COLOR=#1e1e1e][System Configuration Options (Type 12) - Length 5 - Handle 004eh][/COLOR]
[COLOR=#1e1e1e][32Bit Memory Error Information (Type 18) - Length 23 - Handle [/COLOR][COLOR=#ff0000]0050h[/COLOR][COLOR=#1e1e1e]][/COLOR]
[COLOR=#1e1e1e][Physical Memory Array (Type 16) - Length 15 - Handle 0051h][/COLOR]
[COLOR=#1e1e1e]  Location                      02h - Unknown[/COLOR]
[COLOR=#1e1e1e]  Use                           03h - System Memory[/COLOR]
[COLOR=#1e1e1e]  Memory Error Correction       03h - None[/COLOR]
[COLOR=#1e1e1e]  Maximum Capacity              [Unknown][/COLOR]
[COLOR=#1e1e1e]  Memory Error Inf Handle       0050h[/COLOR]
[COLOR=#1e1e1e]  Number of Memory Devices      64[/COLOR]
[COLOR=#1e1e1e][Memory Array Mapped Address (Type 19) - Length 15 - Handle 0052h][/COLOR]

This Error is mapped to actual
Code: 
[COLOR=#1e1e1e]0: kd> !handle 0050h[/COLOR]
[COLOR=#1e1e1e]GetPointerFromAddress: unable to read from fffff80005a85000[/COLOR]

[COLOR=#1e1e1e]PROCESS fffffa83045c8040[/COLOR]
[COLOR=#1e1e1e]    SessionId: none [/COLOR][COLOR=#1e1e1e] Cid: 0880    Peb: [/COLOR][U][COLOR=#0066cc]7fffffd6000[/COLOR][/U][COLOR=#1e1e1e]  ParentCid: [/COLOR][U][COLOR=#0066cc]0304[/COLOR][/U]
[COLOR=#1e1e1e]    DirBase: c1613000  ObjectTable: fffff880127b2840  HandleCount: <Data Not Accessible>[/COLOR]
[COLOR=#1e1e1e]    Image: [/COLOR][COLOR=#ff0000]LTSVC.exe[/COLOR][COLOR=#1e1e1e][/COLOR]

in other dumps it looks similar, errorhandle memory points to failing dll's / modules :thumbsup2:
 
Re: Two server 2008 boxes under Hyper-V just hang at random

Ahhh, I think I know why I have everyone thrown for a loop.

As I said in my original post, I FORCED a blue screen to see if I could get any information. To do this in Hyper-V 2012 R2, you can generate an NMI call - "Debug-VM -Name "VM Name" -InjectNonMaskableInterrupt -ComputerName Hostname "

So that's why you guys are probably seeing stuff you've NEVER seen before! :)
 
Re: Two server 2008 boxes under Hyper-V just hang at random

mh, i dont think being caught in a loop :wave:

Code: 
[B]Mini062714-01.dmp[/B]
BugCheck 50, {fffffa8306041000, 0, fffffa8303842ed0, 0}
Debug session time: [B]Fri Jun 27 08:02:04.890 2014[/B] (UTC + 1:00) // when it happened

System Configuration Options (Type 12) - Length 5 - Handle 004eh]
[32Bit Memory Error Information (Type 18) - Length 23 - Handle [COLOR=#ff0000]0050h[/COLOR]]
[Physical Memory Array (Type 16) - Length 15 - Handle 0051h]
  Location                      02h - Unknown
  Use                           03h - System Memory
  Memory Error Correction       03h - None
  Maximum Capacity              [Unknown]
  Memory Error Inf Handle       0050h
  Number of Memory Devices      64
PROCESS fffffa8304cfe7f0
    SessionId: none  Cid: 1844    Peb: 7efdf000  ParentCid: 1108
    DirBase: 100da1000  ObjectTable: fffff8801dfab580  HandleCount: <Data Not Accessible>
    Image: [B]Lua.exe[/B] 

[B]Mini070714-01.dmp[/B]
BugCheck 50, {fffffa83060f9000, 0, fffffa8306ac0ed0, 0}
Debug session time: [B]Mon Jul  7 11:46:18.430 2014[/B] (UTC + 1:00) // when it happened

[System Configuration Options (Type 12) - Length 5 - Handle 004eh]
[32Bit Memory Error Information (Type 18) - Length 23 - Handle [COLOR=#ff0000]0050h[/COLOR]]
[Physical Memory Array (Type 16) - Length 15 - Handle 0051h]
  Location                      02h - Unknown
  Use                           03h - System Memory
  Memory Error Correction       03h - None
  Maximum Capacity              [Unknown]
  Memory Error Inf Handle       0050h
  Number of Memory Devices      64

PROCESS fffffa830388a170
    SessionId: none  Cid: 181c    Peb: 7efdf000  ParentCid: 100c
    DirBase: 9b1a4000  ObjectTable: fffff8800e8e7ab0  HandleCount: <Data Not Accessible>
    Image: [B]Lua.exe[/B]

resolves to: Kaseya Scripting Component

only the mentioned 0x80 NMI was forced and yes that is known at least by me
and visible by the time stamp.

But always the same handle shown as error as mentioned before.

btw: a server and no ECC-Memory?
as we only can see the Guests..........

Usual if there is no problem with memory there is no error handle for it.
 
Re: Two server 2008 boxes under Hyper-V just hang at random

We got rid of Kaseya a long time ago, so not worried about those dumps from 2014.

The server does have ECC memory in it - is it because this is a virtual guest that something like that isn't exposed?
 
Re: Two server 2008 boxes under Hyper-V just hang at random

those dumps where included and for reference examined too.

> The server does have ECC memory in it
Then inspect your Host.
 
Re: Two server 2008 boxes under Hyper-V just hang at random

PatD,

I was curious if you ever resolved this problem? Funny I think you are the only person I've ever seen use Debug-Vm instead of livekd.
Taking a dump of a VM running on Hyper-V – vimalsdesk

Curious if I undertand the problem correctly.

Windows Server 2012 R2 Hyper-V Host with 2 Guest OSE's 2008 SP2 x64. The Guests after a period of time become unresponsive and require being force off from hyper-v manager to recover from the hang.
How long does it take for the VM's to become hung?
This issue effects only the guest operating systems?
At the time of the hang?
* Can you connect to admin share of the VM?
* Can you ping the VM?

Obviously start with ensuring everything is up to date, that is HOST and the GUEST Operating systems and Integration services. As well as drivers & firmware on the host. You should also ensure that C-STATES/C1E is disabled on the host.
Then disable (uninstall or unload the filter) any 3rd party filter drivers, AV and Backup software you can see this via fltmc.

Finally if you still repro your problem after all of the above is done then you look to grab a dump and it should probably be a compelte dump however kernel at a minimum.
And if you've never analyzed a hang dump you can see a genral overview of basics here
Red alert! My Server is hung – what do I do? – Ntdebugging Blog
 
Re: Two server 2008 boxes under Hyper-V just hang at random

I had issues with livekd, so I gave up until I heard about debug-vm, but sounds like that didn't really help either.

No, never resolved. We have to monitor and bounce the machine when issues come up.

VMs hang at random. I've had them go weeks, I have them go hours. When they hang, I can ping the machines, but can not connect to any shares.

This issue only affects the two 2008 boxes - all other guests on the host are fine. The machines do not hang at the same time. All power saving is turned off, all machines (Guest and host) are up-to-date for Microsoft updates, drivers, integration services. AV was uninstalled some time ago for testing purposes but made no difference.
 

Has Sysnative Forums helped you? Please consider donating to help us support the site!

Back
Top