SysnativeBSODApps - additional check 'drivers found in stack'

M

mgrzeg

BSOD Kernel Dump Senior Analyst
Joined
Jul 26, 2012
Posts
131
Location
Warsaw, Poland
Hello,
what do you think about additional command (along with others):
(x86)->dps @esp-100 @esp+200
(x64)->dps @rsp-200 @rsp+400
which could be helpful in finding drivers in the stack (they may be not visible in the !analysis -v)? I think this is especially helpful for some upperfilters (from AV's).
(The name came from nirsoft's BlueScreenView)

m.g.
 
Could you supply us with a few dump files for testing purposes?

I selected dumps at random and did not find any added benefit. From the sole attached dump file:

Read More:

Stack:
Read More:

Output from kd> dps @rsp-200 @rsp+400
Read More:
 

Attachments

1. !analyze -v

Read More:

2. Stack

Read More:

3. dps @rsp @rsp+200

Read More:

m.g.
 

Attachments

I can see SYMEVENT64x86+0x1cb0e showing in the dps results.

However, NIS removal would have been at the top of my list, especially given BugCheck 0xfe, even though NIS not on stack.

3rd party driver list - Symantec/Norton:
Read More:
Any idea about the symbol errors?
Read More:


i.e., could the SYM errors have resulted in NIS not appearing on the stack?
 
I could create an area to list extra kd commands for users to manually add the ones they want. The output files would be user1.txt and user2.txt for {1} and {2} respectively.

i.e.

[TABLE="class: grid, width: 500, align: center"]
[TR]
[TD]kd Commands[/TD]
[/TR]
[TR]
[TD]{1} dps @esp-100 @esp+200
{2} dps @rsp-200 @rsp+400[/TD]
[/TR]
[/TABLE]


In answer to the symbols errors encountered, I have found that is common for usb related crashes.
 
writhziden said:
In answer to the symbols errors encountered, I have found that is common for usb related crashes.
Click to expand...

I honestly have not had many symbol errors related to USB crashes, unless a hotfix was applied.

Even so, they should be on MSDL site at some point.
 
writhziden said:
I could create an area to list extra kd commands for users to manually add the ones they want. The output files would be user1.txt and user2.txt for {1} and {2} respectively.
Click to expand...
That would be great! :)

m.g.
 
The additional kd command spot would be great!
Then we can rerun all or just one of the dumps with the additional info

Can you code for more than 2 commands? (not that I'd ever use them)
How about a section in the Help file for some suggested additional commands (and maybe a link to a Sysnative topic on helpful commands).
I started a helpful commands page but never got around to doing anything with it: http://www.carrona.org/debugcmd.html Feel free to use anything from it if you'd like.
 
The {1} and {2} were just an example. I can allow for as many commands as you want to list. :)

I can also add to the help file a list of helpful commands and/or a link to a Sysnative thread or html file with those commands. I could also include your page in the help file as a link.

I'm personally not familiar with too many of the commands. I probably only know 5-10 of them at the most, and only a few off the top of my head. I've only been doing analysis for about 9 months, so my experience with the kernel debugger is quite limited. Thankfully I can make up for those limitations with my knowledge of Windows that continues to grow as I learn from everyone here and on the web.


mgrzeg said:
Consider also checkbox 'run only kd commands provided in user section' :)

m.g.
Click to expand...
Will do.
 
I didn't know about this. There are plenty of ways of getting the driver out, but I didn't know this one. I have used Child-SP of nt!KeBugCheckEx before when analysing dump files, but had never made the connection that this is stored in rsp. Using your dump as the example:

Code: 
2: kd>[B] kv[/B]
Child-SP          RetAddr           : Args to Child                                                           : Call Site
[B][COLOR=#ff0000]fffff880`033c4c18[/COLOR][/B] fffff880`04a55a60 : 00000000`000000fe 00000000`00000008 00000000`00000006 00000000`00000005 : nt!KeBugCheckEx
fffff880`033c4c20 fffff800`03385f33 : fffffa80`04eb4050 00000000`00000001 ffffffff`dc3a58a0 fffff800`0322c658 : usbhub!UsbhHubProcessChangeWorker+0xec
fffff880`033c4c80 fffff800`03099a21 : fffff800`0322c600 fffff800`03385f01 fffffa80`03b65600 fffffa80`03b65680 : nt!IopProcessWorkItem+0x23
fffff880`033c4cb0 fffff800`0332ccce : 00000000`00000000 fffffa80`03b65680 00000000`00000080 fffffa80`03b4f840 : nt!ExpWorkerThread+0x111
fffff880`033c4d40 fffff800`03080fe6 : fffff880`031d3180 fffffa80`03b65680 fffff880`031ddfc0 ffffffff`fffff7fb : nt!PspSystemThreadStartup+0x5a
fffff880`033c4d80 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x16

2: kd> [B]r[/B]
rax=0000000000000004 rbx=0000000000000005 rcx=00000000000000fe
rdx=0000000000000008 rsi=fffffa8004eb4050 rdi=fffffa8004ed2000
rip=fffff8000308f640 [COLOR=#ff0000][B]rsp=fffff880033c4c18 [/B][/COLOR]rbp=fffff8000322c658
 r8=0000000000000006  r9=0000000000000005 r10=fffff8000300f000
r11=fffffa8005ac4050 r12=fffffa8006ff93b0 r13=0000000000000001
r14=0000000000000000 r15=0000000000000001
iopl=0         nv up ei ng nz na pe nc
cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00000282
nt!KeBugCheckEx:
fffff800`0308f640 48894c2408      mov     qword ptr [rsp+8],rcx ss:0018:fffff880`033c4c20=00000000000000fe

This is something I have never noticed before, so thank you. It makes automation practical.

Ususally at this point I would use Child-SP of nt!KeBugCheckEx address in a dps command like so:

Code: 
2: kd> [B]dps[/B] [B][COLOR=#ff0000]fffff880`033c4c18[/COLOR][/B] [B]L20[/B]
fffff880`033c4c18  fffff880`04a55a60 usbhub!UsbhHubProcessChangeWorker+0xec
fffff880`033c4c20  00000000`000000fe
fffff880`033c4c28  00000000`00000008
fffff880`033c4c30  00000000`00000006
fffff880`033c4c38  00000000`00000005
fffff880`033c4c40  fffffa80`04ed2000
fffff880`033c4c48  fffff880`02f10b0e [COLOR=#008000][B]SYMEVENT64x86[/B][/COLOR]+0x1cb0e
fffff880`033c4c50  00000000`00000000
fffff880`033c4c58  00000000`00000001
fffff880`033c4c60  fffffa80`03b64150
fffff880`033c4c68  fffffa80`03b64150
fffff880`033c4c70  fffffa80`03b65680
fffff880`033c4c78  fffff800`03385f33 nt!IopProcessWorkItem+0x23
fffff880`033c4c80  fffffa80`04eb4050
fffff880`033c4c88  00000000`00000001
fffff880`033c4c90  ffffffff`dc3a58a0
fffff880`033c4c98  fffff800`0322c658 nt!ExWorkerQueue+0x58
fffff880`033c4ca0  fffff800`03385f10 nt!IopProcessWorkItem
fffff880`033c4ca8  fffff800`03099a21 nt!ExpWorkerThread+0x111
fffff880`033c4cb0  fffff800`0322c600 nt!ExWorkerQueue
fffff880`033c4cb8  fffff800`03385f01 nt!ObpWaitForMultipleObjects+0x4cb
fffff880`033c4cc0  fffffa80`03b65600
fffff880`033c4cc8  fffffa80`03b65680
fffff880`033c4cd0  fffff880`033c4ce0
fffff880`033c4cd8  ffffffff`00000001
fffff880`033c4ce0  fffffa80`03b64150
fffff880`033c4ce8  ddfffcdf`ffd778b1
fffff880`033c4cf0  00000000`00000000
fffff880`033c4cf8  ef7b7ffa`ff3ee36e
fffff880`033c4d00  ffffeb0a`9e7db819
fffff880`033c4d08  effffcfd`f7fffaf9
fffff880`033c4d10  fffff880`031d7f40

Alternatively, raw stack also works:

Code: 
2: kd> [B]!thread[/B]
GetPointerFromAddress: unable to read from fffff800032c0000
THREAD fffffa8003b65680  Cid 0004.0040  Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 2
Not impersonating
GetUlongFromAddress: unable to read from fffff800031fdba4
Owning Process            fffffa8003b4f840       Image:         System
Attached Process          N/A            Image:         N/A
fffff78000000000: Unable to get shared data
Wait Start TickCount      136406       
Context Switch Count      7272           IdealProcessor: 1  NoStackSwap
ReadMemory error: Cannot get nt!KeMaximumIncrement value.
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address nt!ExpWorkerThread (0xfffff80003099910)
Stack Init [B][COLOR=#800080]fffff880033c4db0[/COLOR][/B] Current[B] [COLOR=#0000ff]fffff880033c4970[/COLOR]
[/B]Base fffff880033c5000 Limit fffff880033bf000 Call 0
Priority 12 BasePriority 12 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           : Args to Child                                                           : Call Site
fffff880`033c4c18 fffff880`04a55a60 : 00000000`000000fe 00000000`00000008 00000000`00000006 00000000`00000005 : nt!KeBugCheckEx
fffff880`033c4c20 fffff800`03385f33 : fffffa80`04eb4050 00000000`00000001 ffffffff`dc3a58a0 fffff800`0322c658 : usbhub!UsbhHubProcessChangeWorker+0xec
fffff880`033c4c80 fffff800`03099a21 : fffff800`0322c600 fffff800`03385f01 fffffa80`03b65600 fffffa80`03b65680 : nt!IopProcessWorkItem+0x23
fffff880`033c4cb0 fffff800`0332ccce : 00000000`00000000 fffffa80`03b65680 00000000`00000080 fffffa80`03b4f840 : nt!ExpWorkerThread+0x111
fffff880`033c4d40 fffff800`03080fe6 : fffff880`031d3180 fffffa80`03b65680 fffff880`031ddfc0 ffffffff`fffff7fb : nt!PspSystemThreadStartup+0x5a
fffff880`033c4d80 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x16

2: kd> [B]dps [COLOR=#0000ff]fffff880033c4970[/COLOR] [COLOR=#800080]fffff880033c4db0[/COLOR][/B]
fffff880`033c4970  ????????`????????
fffff880`033c4978  ????????`????????
fffff880`033c4980  ????????`????????
fffff880`033c4988  ????????`????????
fffff880`033c4990  ????????`????????
fffff880`033c4998  ????????`????????
fffff880`033c49a0  ????????`????????
fffff880`033c49a8  ????????`????????
fffff880`033c49b0  ????????`????????
fffff880`033c49b8  ????????`????????
fffff880`033c49c0  ????????`????????
fffff880`033c49c8  ????????`????????
fffff880`033c49d0  ????????`????????
fffff880`033c49d8  ????????`????????
fffff880`033c49e0  ????????`????????
fffff880`033c49e8  ????????`????????
fffff880`033c49f0  ????????`????????
fffff880`033c49f8  ????????`????????
fffff880`033c4a00  ????????`????????
fffff880`033c4a08  ????????`????????
fffff880`033c4a10  ????????`????????
fffff880`033c4a18  ????????`????????
fffff880`033c4a20  ????????`????????
fffff880`033c4a28  ????????`????????
fffff880`033c4a30  ????????`????????
fffff880`033c4a38  ????????`????????
fffff880`033c4a40  ????????`????????
fffff880`033c4a48  ????????`????????
fffff880`033c4a50  ????????`????????
fffff880`033c4a58  ????????`????????
fffff880`033c4a60  ????????`????????
fffff880`033c4a68  ????????`????????
fffff880`033c4a70  ????????`????????
fffff880`033c4a78  ????????`????????
fffff880`033c4a80  ????????`????????
fffff880`033c4a88  ????????`????????
fffff880`033c4a90  ????????`????????
fffff880`033c4a98  ????????`????????
fffff880`033c4aa0  ????????`????????
fffff880`033c4aa8  ????????`????????
fffff880`033c4ab0  ????????`????????
fffff880`033c4ab8  ????????`????????
fffff880`033c4ac0  ????????`????????
fffff880`033c4ac8  ????????`????????
fffff880`033c4ad0  ????????`????????
fffff880`033c4ad8  ????????`????????
fffff880`033c4ae0  ????????`????????
fffff880`033c4ae8  ????????`????????
fffff880`033c4af0  ????????`????????
fffff880`033c4af8  ????????`????????
fffff880`033c4b00  ????????`????????
fffff880`033c4b08  ????????`????????
fffff880`033c4b10  ????????`????????
fffff880`033c4b18  ????????`????????
fffff880`033c4b20  ????????`????????
fffff880`033c4b28  ????????`????????
fffff880`033c4b30  ????????`????????
fffff880`033c4b38  ????????`????????
fffff880`033c4b40  ????????`????????
fffff880`033c4b48  ????????`????????
fffff880`033c4b50  ????????`????????
fffff880`033c4b58  ????????`????????
fffff880`033c4b60  ????????`????????
fffff880`033c4b68  ????????`????????
fffff880`033c4b70  ????????`????????
fffff880`033c4b78  ????????`????????
fffff880`033c4b80  ????????`????????
fffff880`033c4b88  ????????`????????
fffff880`033c4b90  ????????`????????
fffff880`033c4b98  ????????`????????
fffff880`033c4ba0  ????????`????????
fffff880`033c4ba8  ????????`????????
fffff880`033c4bb0  ????????`????????
fffff880`033c4bb8  ????????`????????
fffff880`033c4bc0  ????????`????????
fffff880`033c4bc8  ????????`????????
fffff880`033c4bd0  ????????`????????
fffff880`033c4bd8  ????????`????????
fffff880`033c4be0  ????????`????????
fffff880`033c4be8  ????????`????????
fffff880`033c4bf0  ????????`????????
fffff880`033c4bf8  ????????`????????
fffff880`033c4c00  ????????`????????
fffff880`033c4c08  ????????`????????
fffff880`033c4c10  ????????`????????
fffff880`033c4c18  fffff880`04a55a60 usbhub!UsbhHubProcessChangeWorker+0xec
fffff880`033c4c20  00000000`000000fe
fffff880`033c4c28  00000000`00000008
fffff880`033c4c30  00000000`00000006
fffff880`033c4c38  00000000`00000005
fffff880`033c4c40  fffffa80`04ed2000
fffff880`033c4c48  fffff880`02f10b0e [B][COLOR=#008000]SYMEVENT64x86[/COLOR][/B]+0x1cb0e
fffff880`033c4c50  00000000`00000000
fffff880`033c4c58  00000000`00000001
fffff880`033c4c60  fffffa80`03b64150
fffff880`033c4c68  fffffa80`03b64150
fffff880`033c4c70  fffffa80`03b65680
fffff880`033c4c78  fffff800`03385f33 nt!IopProcessWorkItem+0x23
fffff880`033c4c80  fffffa80`04eb4050
fffff880`033c4c88  00000000`00000001
fffff880`033c4c90  ffffffff`dc3a58a0
fffff880`033c4c98  fffff800`0322c658 nt!ExWorkerQueue+0x58
fffff880`033c4ca0  fffff800`03385f10 nt!IopProcessWorkItem
fffff880`033c4ca8  fffff800`03099a21 nt!ExpWorkerThread+0x111
fffff880`033c4cb0  fffff800`0322c600 nt!ExWorkerQueue
fffff880`033c4cb8  fffff800`03385f01 nt!ObpWaitForMultipleObjects+0x4cb
fffff880`033c4cc0  fffffa80`03b65600
fffff880`033c4cc8  fffffa80`03b65680
fffff880`033c4cd0  fffff880`033c4ce0
fffff880`033c4cd8  ffffffff`00000001
fffff880`033c4ce0  fffffa80`03b64150
fffff880`033c4ce8  ddfffcdf`ffd778b1
fffff880`033c4cf0  00000000`00000000
fffff880`033c4cf8  ef7b7ffa`ff3ee36e
fffff880`033c4d00  ffffeb0a`9e7db819
fffff880`033c4d08  effffcfd`f7fffaf9
fffff880`033c4d10  fffff880`031d7f40
fffff880`033c4d18  00000000`00000000
fffff880`033c4d20  fffff800`03099910 nt!ExpWorkerThread
fffff880`033c4d28  00000000`00000001
fffff880`033c4d30  00000000`00000001
fffff880`033c4d38  fffff800`0332ccce nt!PspSystemThreadStartup+0x5a
fffff880`033c4d40  00000000`00000000
fffff880`033c4d48  fffffa80`03b65680
fffff880`033c4d50  00000000`00000080
fffff880`033c4d58  fffffa80`03b4f840
fffff880`033c4d60  fffffa80`03b4f840
fffff880`033c4d68  00000000`00000000
fffff880`033c4d70  00000000`00000000
fffff880`033c4d78  fffff800`03080fe6 nt!KiStartSystemThread+0x16
fffff880`033c4d80  fffff880`031d3180
fffff880`033c4d88  fffffa80`03b65680
fffff880`033c4d90  fffff880`031ddfc0
fffff880`033c4d98  ffffffff`fffff7fb
fffff880`033c4da0  00000000`00000000
fffff880`033c4da8  00000000`00000000
fffff880`033c4db0  ????????`????????

Of course, this is all going around in circles looking at the same data in different ways. Indeed, the memory regions are the same:

Code: 
rsp:        fffff880033c4c18 
rsp-200:    fffff880033c4770
rps+400:    fffff880033c4d70
Raw stack:  fffff880033c4970 - fffff880033c4db0

It is an interesting new angle which can lead to automation.

To the rest of you, we have already been using this technique under a different name.

If you have ever used the raw stack to find a culprit:

https://www.sysnative.com/forums/showthread.php/444-The-Raw-Truth-Using-Raw-Stacks
http://www.dumpanalysis.org/blog/index.php/raw-stack-analysis-scripts/

(not about raw stacks, but x64 stack reconstruction in general):
http://blogs.msdn.com/b/ntdebugging/archive/2010/05/12/x64-manual-stack-reconstruction-and-stack-walking.aspx
!cmkd.stacks: http://www.codemachine.com/tool_cmkd.html

then you have employed the non-automated version of this. This might give you some idea of where this is and isn't effective. But yes, an interesting prospect to automate, so thanks again, mgrzeg :)

Richard
 
Last edited:
Great! :) Now it's possible to use own scripts! I've tried to use my sample script:
Code: 
$$ Fibo.wds - calculate the n-th Fibonnacci
.block
{
 .if (${$arg1} > 1)
 {
  r $t0 = 0
  r $t1 = 1
  r $t3 = ${$arg1}
  .while (@$t3-1)
   {
    r $t2 = @$t1
    r $t1 = @$t1 + @$t0
    r $t0 = @$t2
 r $t3 = @$t3 - 1
   }
 }
 .printf "Fibonacci(%p) = %p\n", ${$arg1}, @$t1
}

and it works :)
{1} .bugcheck
{2} $$>a<C:\Dumps\Fibo.wds 5
->user1.txt, user2.txt

Of course it opens new possibilities (like .shell -ci "command" program.exe), so thanks once again! :)

m.g.
 
You're welcome. Glad it meets your needs. :D

I think it will help others who do not need all the info that the default settings use, or if they want additional info from other commands/scripts like you did.
 
If you want to grab the whole stack range specified for that thread, be it both unused portion (to check for potential stack trashing) and used, you'll want to nab the Base and Limit values provided by the output from the !thread command.

I personally would like to have some sort of !rawstack extension for Windbg that just grabs current thread context (or a thread address you specify, like !thread) and dumps dps using the Base and Limit values automatically. Doing it manually for a lot of crashdumps can get tedious quick. Bonus points if you can give the extension an argument (like -c) to have it print out using dc instead of dps (helps find text strings trashing the stack).

Other than that, excellent work fellas. Having this added would help a lot.
 
I still don't understand the symbol errors for NT & HAL.

Read More:
 
Vir Gnarus said:
If you want to grab the whole stack range specified for that thread, be it both unused portion (to check for potential stack trashing) and used, you'll want to nab the Base and Limit values provided by the output from the !thread command.

I personally would like to have some sort of !rawstack extension for Windbg that just grabs current thread context (or a thread address you specify, like !thread) and dumps dps using the Base and Limit values automatically. Doing it manually for a lot of crashdumps can get tedious quick. Bonus points if you can give the extension an argument (like -c) to have it print out using dc instead of dps (helps find text strings trashing the stack).

Other than that, excellent work fellas. Having this added would help a lot.
Click to expand...

What about a way to type dps <base> <limit> and have the apps determine the base and limit values by running !thread automatically? In other words, if you input into the User kd Commands exactly as shown here:

[table="width: 500, class: grid"]
[tr]
[td]User kd Commands[/td]
[/tr]
[tr]
[td]dps <base> <limit>[/td]
[/tr]
[/table]

The apps would then determine that you want base and limit from the <base> and <limit> components of the line. The apps could then run !thread, parse out the base and limit info, and replace <base> with the base memory reference and <limit> with the limit memory reference. Or if you want it in reverse:

[table="width: 500, class: grid"]
[tr]
[td]User kd Commands[/td]
[/tr]
[tr]
[td]dps <limit> <base>[/td]
[/tr]
[/table]
 
@JC:

Remember that different hardware calls for different environment changes to Windows and therefore the kernel dlls used will change to compensate. Notice that the symbols load just fine for one particular HAL file and one particular NT kernel file, but the other variations for each failed. That's most likely because that particular dump file was from an environment using those two specific files, and the rest were rejected because they weren't necessary. A quick google search came up with this nice list of the different HALs available. It's outdated, but the premise behind the use of each one is still the same.

@writhziden:

That could work with making some sort of switch to tell it to print out current thread's stack, like you mentioned, but that would mean changing the dps command, which I don't know if that's even possible, since dps is a command and not an extension. It'd be easier to just make a new extension and use that.

I'm not sure it would work with the values reversed, since limit always comes before base, due to how stacks work. DPS command only accepts a start and end value for a range, respectively. You'll get an error if you enter a higher memory address before a lower one.
 
Vir Gnarus said:
If you want to grab the whole stack range specified for that thread, be it both unused portion (to check for potential stack trashing) and used, you'll want to nab the Base and Limit values provided by the output from the !thread command.

I personally would like to have some sort of !rawstack extension for Windbg that just grabs current thread context (or a thread address you specify, like!thread) and dumps dps using the Base and Limit values automatically. Doing it manually for a lot of crashdumps can get tedious quick. Bonus points if you can give the extension an argument (like -c) to have it print out using dc instead of dps (helps find text strings trashing the stack).

Other than that, excellent work fellas. Having this added would help a lot.
Click to expand...

I have looked into this before, but hit some problems. Firstly, I haven't yet discovered a way to capture the output of any extension in dbgeng. It is easy enough to call extensions, but the output prints straight to windbg/kd, and it is hard to progmatically catch. I have the code for capturing command prompt data floating around in my head, and it potentially needs a similar thing here, but it is really complicated.

Secondly, GetCurrentThreadStackLimits does not seem to have an equivalent in dbgeng, nor a version accepting a thread handle (which would hopefully also accept an imaginary handle)

Finally, I don't know where in memory or registers current thread stack limits are stored for manual memory reading.

I just need to figure one of these out. I am still very new to dbgeng, and finding it hard :(
 
Back
Top