Debugging Stop 0xCA - DDDriver.sys & dddriver64Dcsa.sys

[x BlueRobot]

Here's another case of a Stop 0xCA caused by drivers. DDDriver.sys and dddriver64Dcsa.sys are both part of the Dell Diagnostics software which is notorious for causing Stop 0x18 bugchecks. This particular variant of Stop 0xCA is very similar in nature to a Stop 0x18, however, the subtle difference is that the device node is still present within the device tree while the reference count for the object is 0. You may be wondering how this is possible and that's because there is two systems at play here: the PnP Manager which manages the device tree and its device nodes, and the Object Manager which is responsible for managing objects; the device object in our case.

PNP_DETECTED_FATAL_ERROR (ca)
PnP encountered a severe error, either as a result of a problem in a driver or
a problem in PnP itself.  The first argument describes the nature of the
problem, the second argument is the address of the PDO.  The other arguments
vary depending on argument 1.
Arguments:
Arg1: 0000000000000005, PDO freed while still linked in devnode tree.
    The object manager reference count on a PDO dropped to zero
    while the devnode was still linked in the tree.  This usually
    indicates that the driver is not adding a reference when
    returning the PDO in a query IRP.
Arg2: ffffce8f334fc060, PDO.
Arg3: 0000000000000000
Arg4: 0000000000000000

The bugcheck description informs us that the a physical device object was freed while it was still present in the device tree. As we can see, it was freed because the reference count managed by the Object Manager had dropped to 0. The most important part is the last sentence which actually describes the bug. The driver hasn't been handling the IRP_MN_QUERY_DEVICE_RELATIONS request correctly.

Here's a small excerpt from the documentation on the aforementioned IRP:

A driver must reference the PDO of any device that it reports in this IRP (ObReferenceObject). The PnP manager removes the reference when appropriate.

Let's examine the device object which was being removed using the !devobj command with the second parameter:

3: kd> !devobj ffffce8f334fc060
Device object (ffffce8f334fc060) is for:
 Cannot read info offset from nt!ObpInfoMaskToOffset
 \Driver\PnpManager DriverObject ffffce8f335fad70
Current Irp 00000000 RefCount 0 Type 00000004 Flags 00001040
SecurityDescriptor ffff8288d32057a0 DevExt ffffce8f334fc1b0 DevObjExt ffffce8f334fc1b8 DevNode ffffce8f334fc250
ExtensionFlags (0000000000) 
Characteristics (0x00000180)  FILE_AUTOGENERATED_DEVICE_NAME, FILE_DEVICE_SECURE_OPEN
AttachedDevice (Upper) ffffce8f3b527050 \Driver\DDDriver
Device queue is not busy.

As mentioned in the first parameter, we can see that the reference count for the physical device object has reached 0, however, the device node is still present within the device tree. If we examine the device node itself using the !devnode command, then we can find what the device node is associated to.

3: kd> !devnode ffffce8f334fc250 
DevNode 0xffffce8f334fc250 for PDO 0xffffce8f334fc060
  Parent 0xffffce8f335fb010   Sibling 0xffffce8f334c8590   Child 0000000000  
  InstancePath is "ROOT\SYSTEM\0001"
  ServiceName is "DDDriver"
  State = DeviceNodeStarted (0x308)
  Previous State = DeviceNodeEnumerateCompletion (0x30d)
  StateHistory[09] = DeviceNodeEnumerateCompletion (0x30d)
  StateHistory[08] = DeviceNodeEnumeratePending (0x30c)
  StateHistory[07] = DeviceNodeStarted (0x308)
  StateHistory[06] = DeviceNodeStartPostWork (0x307)
  StateHistory[05] = DeviceNodeStartCompletion (0x306)
  StateHistory[04] = DeviceNodeStartPending (0x305)
  StateHistory[03] = DeviceNodeResourcesAssigned (0x304)
  StateHistory[02] = DeviceNodeDriversAdded (0x303)
  StateHistory[01] = DeviceNodeInitialized (0x302)
  StateHistory[00] = DeviceNodeUninitialized (0x301)
  StateHistory[19] = Unknown State (0x0)
  StateHistory[18] = Unknown State (0x0)
  StateHistory[17] = Unknown State (0x0)
  StateHistory[16] = Unknown State (0x0)
  StateHistory[15] = Unknown State (0x0)
  StateHistory[14] = Unknown State (0x0)
  StateHistory[13] = Unknown State (0x0)
  StateHistory[12] = Unknown State (0x0)
  StateHistory[11] = Unknown State (0x0)
  StateHistory[10] = Unknown State (0x0)
  Flags (0x6c000131)  DNF_MADEUP, DNF_ENUMERATED,
                      DNF_IDS_QUERIED, DNF_NO_RESOURCE_REQUIRED,
                      DNF_NO_LOWER_DEVICE_FILTERS, DNF_NO_LOWER_CLASS_FILTERS,
                      DNF_NO_UPPER_DEVICE_FILTERS, DNF_NO_UPPER_CLASS_FILTERS

Ah, looks like it's associated to Dell Diagnostics software and to find further evidence to support our claim and to find the driver which caused the crash, we can dump the associated device stack using the !devstack command and the address of the device object shown in the third parameter.

3: kd> !devstack 0xffffce8f334fc060
  !DevObj           !DrvObj            !DevExt           ObjectName
  ffffce8f3b527050  \Driver\DDDriver   ffffce8f3b5271a0  InfoMask field not found for _OBJECT_HEADER at ffffce8f3b527020

  ffffce8f334fc5d0  \Driver\PnpManager ffffce8f334fc720  Cannot read info offset from nt!ObpInfoMaskToOffset

!DevNode ffffce8f334c8590 :
  DeviceInst is "<???>"
  ServiceName is "DDDriver"

As shown above, the problematic driver is DDDriver.sys and the recommended action is for the user to completely remove all Dell software from their system. Remember, while the PnP Manager is responsible for completing the IRP, any other drivers in the device stack are able to amend the device relations request has it goes down or up the device stack. Here's another excerpt from the documentation:

If there are one or more bus filter drivers in the device stack, such drivers might handle the IRP on its way down to the bus driver and/or on the IRP's way back up the device stack (if there are IoCompletion routines). According to the PnP IRP rules, such a driver can add PDOs to the IRP on its way down the stack and/or modify the relations list on the IRP's way back up the stack (in IoCompletion routines).

So, we know who caused the crash and how it was caused, but let's have a look at the call stack for completeness sake.

3: kd> knL
 # Child-SP          RetAddr           Call Site
00 ffffdf00`0e4e7408 fffff804`4460cd64 nt!KeBugCheckEx
01 ffffdf00`0e4e7410 fffff804`444a58de nt!IopDestroyDeviceNode+0x167440 << Attempt to release memory for the device node and then crash
02 ffffdf00`0e4e7450 fffff804`44473440 nt!IopDeleteDevice+0x1e
03 ffffdf00`0e4e7480 fffff804`44008357 nt!ObpRemoveObjectRoutine+0x80 << Call the Remove object method
04 ffffdf00`0e4e74e0 fffff804`44793db5 nt!ObfDereferenceObjectWithTag+0xc7 << Deference the object and set it to 0
05 ffffdf00`0e4e7520 fffff804`44084f1e nt!PopSystemIrpCompletion+0xc5 << Power IRP is completed
06 ffffdf00`0e4e75a0 fffff804`44084de7 nt!IopfCompleteRequest+0x11e
07 ffffdf00`0e4e7690 fffff804`44188801 nt!IofCompleteRequest+0x17
08 ffffdf00`0e4e76c0 fffff804`44195c5f nt!IopPowerDispatch+0x51 << Send out a IRP_MJ_POWER request
09 ffffdf00`0e4e76f0 fffff804`4408f70d nt!IopPoHandleIrp+0x3b
0a ffffdf00`0e4e7720 fffff804`5e102736 nt!IofCallDriver+0x6d
0b ffffdf00`0e4e7760 ffffce8f`446d1ab8 dddriver64Dcsa+0x2736 << Sibling driver for DDDriver.sys; both are part of Dell Diagnostics
0c ffffdf00`0e4e7768 ffffce8f`3b5271f0 0xffffce8f`446d1ab8
0d ffffdf00`0e4e7770 ffffce8f`42feb710 0xffffce8f`3b5271f0
0e ffffdf00`0e4e7778 fffff804`447b1149 0xffffce8f`42feb710
0f ffffdf00`0e4e7780 00000000`00000000 nt!ExFreePool+0x9

I haven't been able to get as much as information from the call stack as I would like since I'm dealing with a Minidump. Having said that, we can get a good idea of what has happened. The power manager sends out a power request to change the power state of a device, this request is successfully completed and presumably the device was being powered down hence the reason why the device reference count had been decremented to 0. The Object Manager then calls the Remove object method associated to the object - remember each object type will have a slightly different implementation - which then releases any non-paged pool associated to the device object and subsequently the device node. However, the device node hasn't formerly removed from the device tree itself, and therefore the system crashes with the Stop 0xCA bugcheck.

References:

IRP_MN_QUERY_DEVICE_RELATIONS - Windows drivers