1. #1

    Rootkit Debugging (runtime2 postmortem) - SwishDbgExt, SysecLabs script, etc.

    Today we're going to be doing some rootkit debugging, specifically regarding runtime2, with a bit of a twist! I have a ton of rootkit debugging posts coming in the next few weeks, as I've decided to break them up rather than throwing them together in one giant mess of a post.

    I've shown various scenarios in which I've debugged a rootkit before (0x7A, etc), but this time we're going to use various extensions to help us, other methods, and overall go a lot more in-depth. The postmortem runtime2 rootkit KMD that will be used in this post was generated by our beloved niemiro, so a big thanks to him! He aimed to make it a good example of some things a rootkit/malware developer can do to make things not as obvious when you resort to methods such as hooking the SSDT, which is rather old and very detectable these days.

    Code:
    CRITICAL_OBJECT_TERMINATION (f4)  
    A process or thread crucial to system operation has unexpectedly exited or been  
    terminated.  
    Several processes and threads are necessary for the operation of the  
    system; when they are terminated (for any reason), the system can no  
    longer function.  
    Arguments:  
    Arg1: 00000003, Process  
    Arg2: 86664d90, Terminating object  
    Arg3: 86664edc, Process image file name  
    Arg4: 819e91f0, Explanatory message (ascii)
    Right, so here's our bug check. Most if not all of these 'older' rootkits will use Direct Kernel Object Manipulation (DKOM) to hook low-level routines/functions within the System Service Dispatch Table (SSDT). When this is occurring, assuming the developer of the rootkit didn't do a very good job in writing their rootkit, a lot can go wrong when disabling write protection, carelessly swapping memory, and inserting hooks. Malware scans from many AV programs can also cause crashes when they detect that the SSDT is hooked under certain circumstances. The rootkit can also intentionally call a bug check if written this way when it has detected a scan has initiated.

    However, if the driver is well written, you may not crash at all. What do you do if you're suspicious of a rootkit infection/hooking, yet a bug check isn't occurring naturally due to proper programming of the rootkit? This is when in some cases (like in this specific example here), you may need to take matters into your own hands and either:



    • Force a bug check.
    • Live kernel debugging session.
    • Run an ARK (Anti-Rootkit) tool. This is no fun... we want to have fun and learn : )



    The first option is much less effective than the second, and that's due to the fact that the rootkit may not be doing any obvious hooking of the SSDT, etc, at the time of you forcing the crash. If you're investigating during a live session, it's much more effective and there are a lot more commands you can use. However, for learning purposes, I will show both.

    1st argument - The value in the 2nd argument (0x3)implies it was a process as opposed to a thread that unexpectedly terminated. If it was a thread however, it would have instead been 0x6.

    2nd argument - The value in the 2nd argument (86664d90) is a pointer to the _EPROCESS object that unexpectedly terminated. We can dump it using !object, which I will show below:

    Code:
    kd> !object 86664d90  
    Object: 86664d90 Type: (841537e8) Process  
      ObjectHeader: 86664d78 (old version)  
      HandleCount: 4 PointerCount: 127

    3rd argument - The value in the 3rd argument (86664edc) is the process image file name. Essentially, it's the process name that unexpectedly terminated. We can dump it by using dc, which I will show below:

    Code:
    kd> dc 86664edc  
    86664edc 73727363 78652e73 00000065 00000000 csrss.exe.......  
    86664eec 00000000 00000000 00000000 8605a278 ............x...  
    86664efc 869356d8 00000000 00000000 0000000a .V..............  
    86664f0c 8c04d631 00000000 00000000 7ffdc000 1...............  
    86664f1c 00000000 00000099 00000000 00000000 ................  
    86664f2c 00000000 000005c6 00000000 0003cc50 ............P...  
    86664f3c 00000000 00000000 00000000 00006d77 ............wm..  
    86664f4c 00000000 00000000 00000162 00000000 ........b.......
    The process that unexpectedly terminated was csrss.exe, which is the Client/Server Runtime Subsystem.

    For some extra food for thought, we can get a lot of the information above manually if we'd like to. If we take the 2nd argument (_EPROCESS object) and run the following:

    Code:
    dt _EPROCESS 86664d90 imageFileName
    dt will display the type, which will show us the offset, etc. More specifically it displays information about a local variable, global variable or data type:

    Code:
    kd> dt _EPROCESS 86664d90 imageFileName  
    nt!_EPROCESS  
      +0x14c ImageFileName : [16] "csrss.exe"
    If we take the _EPROCESS object and add it to the offset (+0x14c), we get the following:

    Code:
    kd> ? 86664d90+0x14c  
    Evaluate expression: -2040115492 = 86664edc
    Look familiar? It's our 3rd argument, the process image name (csrss.exe).

    4th argument - The value in the 3rd argument (819e91f0) is the explanatory message regarding the reason for the bug check, specifically in ASCII. To dump this, we'll need to use the dc command as we used earlier on the 3rd argument:

    Code:
    kd> dc 819e91f0  
    819e91f0 6d726554 74616e69 20676e69 74697263 Terminating crit  
    819e9200 6c616369 6f727020 73736563 25783020 ical process 0x%
    As we can see, this is reiterating the 1st argument.



    So far all we know is that this bug check occurred because csrss.exe, a critical Windows process, unexpectedly terminated. Why? If you weren't able to tell whilst reading through by now, we purposely terminated it via Task Manager to force the bug check.With that said, for this post we're choosing the first method (forcing a bug check).

    Pretending we ourselves didn't purposely force the bug check for a moment, what would we at this point do if this was a crash dump we were looking at from a system that isn't ours, and/or from our doing? Among many things, one of the first few things I do in 'not so obvious' 0xF4's is I check the summary and stats regarding virtual memory on the system at the time of the crash:

    Code:
    kd> !vm  
    *** Virtual Memory Usage ***  
         Physical Memory:   917370 (  3669480 Kb)  
         Page File: \??\C:\pagefile.sys  
          Current:  3976680 Kb Free Space:  3976676 Kb  
          Minimum:  3976680 Kb Maximum:   4193280 Kb  
         Available Pages:   769568 (  3078272 Kb)  
         ResAvail Pages:    869504 (  3478016 Kb)  
         Locked IO Pages:      0 (     0 Kb)  
         Free System PTEs:   348283 (  1393132 Kb)  
         Modified Pages:    11884 (   47536 Kb)  
         Modified PF Pages:   11830 (   47320 Kb)  
         NonPagedPool Usage:   8265 (   33060 Kb)  
         NonPagedPool Max:   522998 (  2091992 Kb)  
         PagedPool 0 Usage:   5501 (   22004 Kb)  
         PagedPool 1 Usage:   10013 (   40052 Kb)  
         PagedPool 2 Usage:    623 (   2492 Kb)  
         PagedPool 3 Usage:    631 (   2524 Kb)  
         PagedPool 4 Usage:    726 (   2904 Kb)  
         PagedPool Usage:    17494 (   69976 Kb)  
         PagedPool Maximum:  523264 (  2093056 Kb)  
         Session Commit:     3758 (   15032 Kb)  
         Shared Commit:     9951 (   39804 Kb)  
         Special Pool:       0 (     0 Kb)  
         Shared Process:     1260 (   5040 Kb)  
         Pages For MDLs:      2 (     8 Kb)  
         PagedPool Commit:   17550 (   70200 Kb)  
         Driver Commit:     2335 (   9340 Kb)  
         Committed pages:   115256 (  461024 Kb)  
         Commit limit:    1882649 (  7530596 Kb)
    We can see right away that we don't have insufficient non-paged pool, which is a pretty popular cause of most 0xF4's as it at that point cannot handle I/O operations, etc. It's generally due to buggy drivers causing pool related memory leaks, etc.

    With the above said, assuming we are looking at a crash dump that wasn't ours, we can almost entirely rule out this specific 0xF4 being caused by a buggy driver (MSFT or 3rd party). To be extra sure, double-check the modules list and see if there's anything that jumps out as problematic.



    At this point I would start becoming suspicious of a rootkit, as I would not right away suggest the hard disk (whether HDD or SSD) is the immediate problem given we'd probably see obvious NT_STATUS codes for that. For example, possibly 0xc0000006. Either that, or an entirely different bug check (perhaps 0x7A). With that said, now we get to have some fun!

    There are many ways to go about detecting a rootkit hooking the SSDT, and we will discuss extensions/scripts for the moment:

    1st Method - SwishDbgExt

    The first method we will be using in this postmortem debugging example is the wonderful SwishDbgExt, which was developed/created by a friend of mine (Matt Suiche). I've made various contributions to the help file considering the love I have gathered for this extension. It makes a lot of our lives as debuggers much easier.

    Once you have the extension loaded, we're going to be using the !ms_ssdt command. This command displays the System Service Dispatch Table, which is extremely helpful in the investigation of suspected rootkit hooks through using what is known as Direct Kernel Object Manipulation (DKOM).

    -- Chopping some of the SSDT output as it's fairly large, let's skip to what's important:

    Code:
    |-------|--------------------|--------------------------------------------------------|---------|--------|  
    | Index | Address      | Name                          | Patched | Hooked |  
    |-------|--------------------|--------------------------------------------------------|---------|--------|  
    *** ERROR: Module load completed but symbols could not be loaded for Gjglly.sys  
    |  126 | 0xFFFFFFFF91F054C2 | Gjglly                         |     |    |  
    |  127 | 0xFFFFFFFF81A34F80 | nt!NtDeviceIoControlFile                |     |    |  
    |  128 | 0xFFFFFFFF81949B44 | nt!NtDisplayString                   |     |    |  
    |  129 | 0xFFFFFFFF81A2117F | nt!NtDuplicateObject                  |     |    |  
    |  130 | 0xFFFFFFFF81A18134 | nt!NtDuplicateToken                  |     |    |  
    |  131 | 0xFFFFFFFF81AB14E8 | nt!NtEnumerateBootEntries               |     |    |  
    |  132 | 0xFFFFFFFF81AB278A | nt!NtEnumerateDriverEntries              |     |    |  
    |  133 | 0xFFFFFFFF91F04FFA | Gjglly                         |     |    |  
    |  134 | 0xFFFFFFFF81AB10B7 | nt!NtEnumerateSystemEnvironmentValuesEx        |     |    |  
    |  135 | 0xFFFFFFFF81A9F073 | nt!NtEnumerateTransactionObject            |     |    |  
    |  136 | 0xFFFFFFFF91F051B6 | Gjglly                         |     |    |  
    |  137 | 0xFFFFFFFF81A802D5 | nt!NtExtendSection                   |     |    |  
    |  138 | 0xFFFFFFFF819A113A | nt!NtFilterToken                    |     |    |  
    |  139 | 0xFFFFFFFF819B39FC | nt!NtFindAtom                     |     |    |  
    |  140 | 0xFFFFFFFF819DCA86 | nt!NtFlushBuffersFile                 |     |    |  
    |  141 | 0xFFFFFFFF819AD0F6 | nt!NtFlushInstructionCache               |     |    |  
    |  142 | 0xFFFFFFFF819781EB | nt!NtFlushKey                     |     |    |  
    |  143 | 0xFFFFFFFF818B11C1 | nt!NtFlushProcessWriteBuffers             |     |    |  
    |  144 | 0xFFFFFFFF819C175B | nt!NtFlushVirtualMemory                |     |    |  
    |  145 | 0xFFFFFFFF81A82D64 | nt!NtFlushWriteBuffer                 |     | Yes  |
    We can see a module (Gjglly.sys), and nt!NtFlushWriteBuffer is hooked. Let's not jump to conclusions just yet as this could be a completely legitimate hook.

    First of all, what's Gjglly.sys? This is a driver in relation to AntiDebugLIB.

    An advanced software encryption tool.

    AntiDebugLIB is a useful tool that was designed in order to assist software developers protect their applications against advanced reverse engineering and software cracking. It offers a powerful advanced license control which allow developers to distribute trial versions of their applications securely.
    This is a legitimate driver, but we can also at the same time not jump to conclusions on it being safe as said above (hint: it's not - I will discuss later).

    2nd Method - Script

    The second method we will be using in this postmortem debugging example is an older script developed by Lionel d'Hauenens of Laboskopia. This script will only work with the x86 WinDbg client. Regardless, as we know, it's always best to debug a crash dump in the client based off of the system's architecture. Given the system that generated this crash dump was 32-bit, we'll be debugging it with the x86 WinDbg client.

    This script is incredibly helpful in not only checking the SSDT for hooking, but also detecting what is known as a Shadow SSDT hook. You can find a great article on Shadow SSDT hooking from my very good friend Harry - Shadow SSDT Hooking with Windbg.

    Nevertheless, if you don't understand French instructions, to install the script, simply drag and drop the script folder into your x86 Debuggers folder.

    Once it's installed, type the following command into WinDbg with your postmortem rootkit crash dump:

    Code:
    kd> $$><script\@@init_cmd.wdbg
    So long as you installed the script properly, you should see the following:

    Code:
    SysecLabs Windbg Script : Ok :)  
    ('al' for display all commands)
    Once you see that which has verified the script is loaded properly, type the following command:

    Code:
    !!display_system_call
    Here's a few excerpts:

    Code:
    007D  0003  OK  nt!NtDeleteObjectAuditAlarm (81a46500)  
    007E  0002 HOOK-> *** ERROR: Module load completed but symbols could not be loaded for Gjglly.sys  
    007F  000A  OK  nt!NtDeviceIoControlFile (81a34f80)  
    0080  0001  OK  nt!NtDisplayString (81949b44)  
    0081  0007  OK  nt!NtDuplicateObject (81a2117f)  
    0082  0006  OK  nt!NtDuplicateToken (81a18134)  
    0083  0002  OK  nt!NtEnumerateBootEntries (81ab14e8)  
    0084  0002  OK  nt!NtEnumerateDriverEntries (81ab278a)  
    0085  0006 HOOK-> Gjglly+0x1ffa (91f04ffa)  
    0086  0003  OK  nt!NtEnumerateSystemEnvironmentValuesEx (81ab10b7)  
    0087  0005  OK  nt!NtEnumerateTransactionObject (81a9f073)  
    0088  0006 HOOK-> Gjglly+0x21b6 (91f051b6)
    Code:
    00BC  0003  OK  nt!NtOpenJobObject (81a912df)  
    00BD  0003 HOOK-> Gjglly+0x1f4c (91f04f4c)  
    00BE  0004  OK  nt!NtOpenKeyTransacted (819727e1)
    Code:
    0143  0001  OK  nt!NtSetUuidSeed (8195977b)  
    0144  0006 HOOK-> Gjglly+0x2372 (91f05372)  
    0145  0005  OK  nt!NtSetVolumeInformationFile (81a6c5de)
    Here's Gjglly.sys again, and we can see it's hooking. At this point, we would be suspicious enough to run an ARK tool such as GMER. Of course, if we ran GMER at this point, it would show that the system is in fact infected with runtime2, and Gjglly.sys is our rootkit driver. Normally, by itself and by default, the runtime2 driver is runtime2.sys, not Gjglly.sys. In this specific scenario, niemiro used a different loader to inject the driver than the original, and renamed runtime2.sys to Gjglly.sys (a legitimate module name). Although GMER if run would still say the system is infected with a rootkit, and it would label Gjglly.sys as the rootkit driver, it's done not to trick various ARK tools, but the user.

    If you've ever been infected with runtime2, you know it additionally drops startdrv.exe in the Temp directory of the current Windows install. startdrv.exe is the part that infuriates those who become infected with runtime2, because it's the part that immediately makes you suspicious of an infection as it's a trojan dropper. This is the specific part of the rootkit that will cause slowness, garbage popups, etc.

    In this specific case, niemiro intentionally corrupted startdrv.exe which stopped it from executing entirely. You may be saying to yourself, what's the point of a rootkit's protection driver with no execution of payload, etc, by its dropped trojan? Well, there really is no point (unless your goal was to use something else that's malicious and not as obvious as a trojan dropper)! This was just an example to show how you can better hide a rootkit (protection driver, really) if you wanted to. Although ARK tools would still pick it up, it's not anywhere near as obvious to the user.

    Thanks for reading, and I will get around to doing a live debugging of runtime2 as soon as I can.



    References

    Hunting rootkits with Windbg.
    Some notes about System Service Dispatch Table hook.
    Last edited by Patrick; 09-07-2014 at 03:52 AM.
    Jared, satrow and x BlueRobot say thanks for this.


    • Ad Bot

      advertising
      Beep.

        
       

  2. #2
    Jared's Avatar
    Join Date
    Feb 2014
    Age
    20
    Posts
    1,568
    • specs System Specs
      • Manufacturer:
        Custom
      • Motherboard:
        ASUS Maximus VII Ranger
      • CPU:
        i7 4790K @ 4.4GHz
      • Memory:
        Corsair Vengeance 16GB 1866MHz
      • Graphics:
        MSI Gaming 4G GTX 980
      • Sound Card:
        Creative Soundblaster ZxR
      • Hard Drives:
        Samsung 850 SSD 250GB
      • Disk Drives:
        Western Digital Black Caviar 2TB
      • Power Supply:
        Corsair RM650 Modular 650 Watts
      • Case:
        Fractal Design Define R5 Window
      • Cooling:
        Corsair H100i GTX
      • Display:
        Dell U2515H 25inch 2560x1440 + LG Flatron M2262D 22inch 1920x1080
      • Operating System:
        Windows 10 Professional x64

    Re: Rootkit Debugging (runtime2 postmortem) - SwishDbgExt, SysecLabs script, etc.

    Excellent work Patrick!

    Among many things, one of the first few things I do in 'not so obvious' 0xF4's is I check the summary and stats regarding virtual memory on the system at the time of the crash:
    I thought I was the only one!
    The first thing I check is that, especially if there isn't a disk error code.

  3. #3
    x BlueRobot's Avatar
    Join Date
    May 2013
    Location
    Minkowski Space
    Posts
    1,590

    Re: Rootkit Debugging (runtime2 postmortem) - SwishDbgExt, SysecLabs script, etc.

    You can detect IRP Hooking with !ms_drivers /scan. Speedfan is an example of a program which uses legitimate hooking.
    Patrick and Jared say thanks for this.
    Machines Can Think

    I am currently studying again, and therefore may not be available very often.


  4. #4
    Jared's Avatar
    Join Date
    Feb 2014
    Age
    20
    Posts
    1,568
    • specs System Specs
      • Manufacturer:
        Custom
      • Motherboard:
        ASUS Maximus VII Ranger
      • CPU:
        i7 4790K @ 4.4GHz
      • Memory:
        Corsair Vengeance 16GB 1866MHz
      • Graphics:
        MSI Gaming 4G GTX 980
      • Sound Card:
        Creative Soundblaster ZxR
      • Hard Drives:
        Samsung 850 SSD 250GB
      • Disk Drives:
        Western Digital Black Caviar 2TB
      • Power Supply:
        Corsair RM650 Modular 650 Watts
      • Case:
        Fractal Design Define R5 Window
      • Cooling:
        Corsair H100i GTX
      • Display:
        Dell U2515H 25inch 2560x1440 + LG Flatron M2262D 22inch 1920x1080
      • Operating System:
        Windows 10 Professional x64

    Re: Rootkit Debugging (runtime2 postmortem) - SwishDbgExt, SysecLabs script, etc.

    Excellent work Harry. Thanks for the info.

  5. #5
    Forum Moderator, BSOD
    BSOD Kernel Dump Expert
    Contributor
    blueelvis's Avatar
    Join Date
    Apr 2014
    Location
    India
    Age
    22
    Posts
    969
    • specs System Specs
      • Manufacturer:
        Toshiba
      • CPU:
        Intel Core i5 @ 2.4 GHz 2nd Generation
      • Memory:
        8 GB @ 1600MHz Dual Channel B)
      • Graphics:
        Intel HD 3000 B)
      • Hard Drives:
        Hitachi 1TB 7200 RPM &amp; WD 500 GB
      • Cooling:
        There is some fan inside but it keeps whirring &lt;_&lt;
      • Display:
        1366x768
      • Operating System:
        Windows 8.1 Embedded Industry Pro

    Re: Rootkit Debugging (runtime2 postmortem) - SwishDbgExt, SysecLabs script, etc.

    Thanks for the article Patrick!

    Really helpful and detailed. Now, awaiting your next post on this ^_^
    Ever wanted to learn to debug BSODs? PM me now!

    Feel free to PM me in case I haven't replied within 48 Hours ^_^. Anything else? Still feel free to PM me :thumbsup2:

  6. #6
    x BlueRobot's Avatar
    Join Date
    May 2013
    Location
    Minkowski Space
    Posts
    1,590

    Re: Rootkit Debugging (runtime2 postmortem) - SwishDbgExt, SysecLabs script, etc.

    The script doesn't seem to load for me?

    Code:
    0: kd> $$><script\@@init_cmd.wdbg
    Command file execution failed, Win32 error 0n2
        "The system cannot find the file specified."
    Machines Can Think

    I am currently studying again, and therefore may not be available very often.


  7. #7

    Re: Rootkit Debugging (runtime2 postmortem) - SwishDbgExt, SysecLabs script, etc.

    Are you running the script/dump in the x86 debugger? It won't work in the x64 debugger.
    Jared says thanks for this.

  8. #8
    x BlueRobot's Avatar
    Join Date
    May 2013
    Location
    Minkowski Space
    Posts
    1,590

    Re: Rootkit Debugging (runtime2 postmortem) - SwishDbgExt, SysecLabs script, etc.

    I was using a x86 Debugger, I always use the x86 Debugger since I'm using a x86 system.
    Machines Can Think

    I am currently studying again, and therefore may not be available very often.


  9. #9
    x BlueRobot's Avatar
    Join Date
    May 2013
    Location
    Minkowski Space
    Posts
    1,590

    Re: Rootkit Debugging (runtime2 postmortem) - SwishDbgExt, SysecLabs script, etc.

    !ms_idt detects hooking and patching within the IDT too.

    !ms_idt = IDT Hooking
    !ms_drivers /scan = IRP Hooking
    !ms_ssdt = SSDT Hooking
    Jared and blueelvis say thanks for this.
    Machines Can Think

    I am currently studying again, and therefore may not be available very often.


Similar Threads

  1. Rootkit Debugging - SYSTEM_SERVICE_EXCEPTION (3b), KERNEL_DATA_INPAGE_ERROR (7a)
    By Patrick in forum BSOD Kernel Dump Analysis Debugging Information
    Replies: 5
    Last Post: 11-06-2014, 02:19 PM
  2. SwishDbgExt
    By Corrine in forum BSOD Kernel Dump Analysis Debugging Information
    Replies: 1
    Last Post: 07-25-2014, 09:44 PM
  3. Did I miss anything with this rootkit infestation?
    By LilBambi in forum Security Arena
    Replies: 4
    Last Post: 09-08-2013, 01:51 AM
  4. New TDL4 rootkit successfully hiding from AV
    By JMH in forum Security News
    Replies: 0
    Last Post: 10-10-2012, 01:34 AM

Log in

Log in