MY SOLUTION:
So, I finally got the server fixed up, and while it was round about, it is working. The following was my solution, and naturally may be very different from yours....
TL;DR:
- Observed previous updates were not listed as superseded.
- Uninstalled Cumulative update 3630 and recent servicing stack. This broke hal.dll and SAM registry.
- Extracted hal.dll from Slipstreamed 3630 ISO and recovered SAM from regbackup.
- Manually applied 3630 again. This got windows booting up again but resulted in corrupted / unserviceable image.
- Cleared the unserviceable entry in HKLM registry.
- dism /scanhealth, /restorehealth, SFCFix, and sfc /scannow fix all corruptions.
- Windows is now apparently not activated (although it says it is). Manually reactivate.
- Can now install updates.
The long, want to read:
The Good: The /cleanup-image commands all returned good and SFCFix wasn't finding anything.
The Bad: Only sfc /scannow was finding problems --warnings mostly and updates still weren't installing.
Thus: I began poking around again in the DISM logs. Naturally there were warnings about certain files, but there was one warning that kept cropping up that bothered me: "Warning DISM DISM OS Provider: PID=7620 TID=8368 Unable to set the DLL search path to the servicing stack folder. C:\Windows may not point to a valid Windows folder. - CDISMOSServiceManager::Final_OnConnect"
Now, this warning, I've read, is often benign. Nevertheless, it got me wondering if I had a problem with one of the servicing stacks that was installed. To that end, I began uninstalling not only the servicing stack, but also the Cumulative updates, back to the 3630 release of Server 2016 (April 2020). Once I reached this point, I tried the update to 3930 (Sept. 2020) again. Naturally, it failed. Not only did it fail, but trying to install the cumulative updates I had just uninstalled also failed.
Enumerating of the installed packages via the command
dism /online /get-packages /format:table showed an oddity. Of all the packages I still had installed, only one was listed as superseded, despite having updates installed that should supersede others. How strange....
I attempted to clean up all the packages that were superseded and no longer needed via the command:
dism /onlone /cleanup-image /spsuperseded. This command errored out indicating that there were no uninstallation packages available for the current service pack updates... which should be WRONG. I never did that kind of clean up and always keep the SP uninstall packages (you never know, right?)
Naturally, I tried to uninstall if via dism. It failed. Then I tried uninstalling it through the windows update manager (or security update, update history, remove packages).
That "WORKED." When the system rebooted, it failed to come back up, indicating that the HAL.DLL was corrupt or missing. Off to the Windows Recovery Environment we go....
Windows Recovery Environment
In the recovery environment, using the Slipstreamed ISO of 3630 I generated, I extracted the hal.dll to replace the "corrupted" one. Naturally this isn't an exact replacement, but it is usually enough to get windows going to redetect the other hardware present. Once replaced, windows failed to boot again with the dreaded "thread exception" error. This is often indicative of a corrupted SAM registry.
C:\windows\system32\config\regbackup to the rescue!
This got me to the point of windows attempting to load, but then dropping me to the "failed to load windows" screen and asking me if I wanted to attempt a repair. In the Windows recovery environment again, I verified the integrity of the of the offline image:
dism /image:c:\ /cleanup-image /scanhealth. It indicated a good image (obviously not working correctly). So now I attmpted to reinstall the 3630 cumulative update.
BAM! Windows back up and running. Loaded fine. (For the note takers: Uninstalling the April Cumulative update borked the hal.dll and SAM registry. Or maybe it was a corrupted install to begin with...).
Windows take 2:
Back in windows, I attempted to install the 3930 update via dism. It failed indicating that the image was corrupt and unserviceable!
At this point I "bit my thumb" at Windows and cleared the unserviceable registry entry:
HLKM/Software/Microsoft/Windows/CurrentVersion/ComponentBasedServicing/Unserviceable.
Running /scanhealth again indicated a corrupted windows image but was repairable. Slipstreamed ISO to the rescue AGAIN:
dism /mount-image /imagefile:d:\scratch\slipstream\Server2016_3630\sources\install.wim /index:2 /mountdir:d:\scratch\mount-3630
dism /online /cleanup-image /source:d:\scratch\mount-3630\windows /restorehealth /limitaccess
The operation complete successfully.
Run /scanhealth and /checkhealth again. Both indicate a GOOD image!
Run SFCFix. It indicates some corruption is detected but finishes correctly indicating successful repairs (although the log files don't show any repairs).
Run sfc /scannow.
IT RETURNS CLEAN!
I run everything one more time and it all checks out okay!
Excitedly, I attempt to install the update again at the command line via dism /add-package.
IT FAILS with error: 0x8007007B! But this is a new error, so I'm not completely deflated.
Oddly, a quick search indicates this error is due to a problem with windows activation, indicating that it ISN'T activated. Check the activation status under Properties of My Computer shows that windows THINKS it still IS activated.
Nevertheless, this OS activates against a KMS server. So, I manually activate it again at the command prompt.
I attempt to install the September 3930 Cumulative update:
SUCCESS!
So, I'm not entirely sure what the final problem was. Perhaps some intrepid soul will comb through the logs and figure it out. I went over them many times trying to track it down, but without success. Ultimately, this is what got the system up and running and updating again.
For the record, I was ready to go to an inplace 2019 upgrade weeks ago. But there is software on this server that I don't want to reinstall and that may not run correctly on 2019 (according to the vendor). So, a repair was in order....
THE END!
