Server 2016 updates install failure

Tech · Jan 14, 2022

I found that one of our servers has been trying and failing to install the same 2 updates every day since August 2020. They are:
2020-07 Servicing Stack Update for Windows Server 2016 for x64-based Systems (KB4565912) - Error 0x8000ffff
2020-08 Cumulative Update for Windows Server 2016 for x64-based Systems (KB4571694) - Error 0x8000ffff
I manually downloaded the September Servicing Stack Update and tried to install it and after a long time it reported that the update had failed. I tried running sfc /scannow and it reported "There is a system repair pending which requires reboot to complete. Restart Windows and run sfc again."
I rebooted the server and retried and got the same error. I found that "C:\Windows\WinSxS\pending.xml" is 175KB and that there are dozens of items under each of these registry keys:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\SessionsPending\
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\PackagesPending\
I tried running SFCFix and it gave several errors some of which weren't included in it's text log, one I think was a DISM error 0x8000ffff

Tech · Jan 14, 2022

I tried running DISM.exe /Online /Cleanup-image /Scanhealth
Result: Error: 0x8000ffff
DISM failed. No operation was performed.
For more information, review the log file at C:\Windows\Logs\DISM\dism.log

Looked through it at one point it referred to C:\Windows\Logs\CBS\CBS.log for more info. In there I found:

2022-01-14 16:58:29, Info CSI 00000003 StoreCorruptionRepair transaction begun. WcpVersion: [l:41]'10.0.14393.3744 (rs1_release.200514-1751)'.
2022-01-14 16:58:29, Info CSI 00000004@2022/1/14:22:58:29.936 Starting corruption detection (InnerFlags=5)
2022-01-14 16:58:29, Info CBS FLOW: Entering stage: CheckCsi
2022-01-14 16:59:03, Info CBS Winlogon: TiCoreOnCreateSession has been called
2022-01-14 17:11:27, Info CSI 00000005 Hashes for file member \SystemRoot\WinSxS\amd64_windows-defender-management-powershell_31bf3856ad364e35_10.0.14393.0_none_1137c685b804c9a3\MSFT_MpThreatCatalog.cdxml do not match actual file [l:26]'MSFT_MpThreatCatalog.cdxml' :
Found: {l:32 /4u949dHRavEuL72J4xGvZu6BcLmHw6wM3h0Q83oD0Q=} Expected: {l:32 EG1FlULD7H3Zt+VF10pKmPCtdMfbahMzz82RaOKjwHY=}
2022-01-14 17:11:27, Info CSI 00000006 Hashes for file member \SystemRoot\WinSxS\amd64_windows-defender-management-powershell_31bf3856ad364e35_10.0.14393.0_none_1137c685b804c9a3\Defender.psd1 do not match actual file [l:13]'Defender.psd1' :
Found: {l:32 sJYRL7pOOrzeh61t/IQ8fi10+0aRJaAEBOEvvl8FTss=} Expected: {l:32 SsIaLRmRVHiEjS8RMoQAygj7Gsj8861DFPUlawqGoH0=}
2022-01-14 17:11:28, Info CSI 00000007 Hashes for file member \SystemRoot\WinSxS\amd64_windows-defender-management-powershell_31bf3856ad364e35_10.0.14393.0_none_1137c685b804c9a3\MSFT_MpWDOScan.cdxml do not match actual file [l:20]'MSFT_MpWDOScan.cdxml' :
Found: {l:32 Bk2MYuHWE1JKtvQU1JBrp32O78S6K646x5azMysWgew=} Expected: {l:32 gN3soJxshc3kBr2TUsJMtspYqxcGIVlxPwWRxG+nuf0=}
2022-01-14 17:11:28, Info CSI 00000008 Hashes for file member \SystemRoot\WinSxS\amd64_windows-defender-management-powershell_31bf3856ad364e35_10.0.14393.0_none_1137c685b804c9a3\MSFT_MpSignature.cdxml do not match actual file [l:22]'MSFT_MpSignature.cdxml' :
Found: {l:32 lWqGnQBu8rR34UaDW+/qnp2mryYFBjieec6KZpkG8LM=} Expected: {l:32 27hKBIml3+7LToUEEqq7cucl5fiTGdnhWKi6qb5kcNo=}
2022-01-14 17:11:28, Info CSI 00000009 Hashes for file member \SystemRoot\WinSxS\amd64_windows-defender-management-powershell_31bf3856ad364e35_10.0.14393.0_none_1137c685b804c9a3\MSFT_MpThreat.cdxml do not match actual file [l:19]'MSFT_MpThreat.cdxml' :
Found: {l:32 3S3s28bsA1fdvZUq08KcCgX8c2i62zd2SLuPCWTu3uw=} Expected: {l:32 rllSNoMqOITSqj+RNFSsb/THNfVegZx/ORyUtlMoH5I=}
2022-01-14 17:11:28, Info CSI 0000000a Hashes for file member \SystemRoot\WinSxS\amd64_windows-defender-management-powershell_31bf3856ad364e35_10.0.14393.0_none_1137c685b804c9a3\MSFT_MpComputerStatus.cdxml do not match actual file [l:27]'MSFT_MpComputerStatus.cdxml' :
Found: {l:32 y/ymyxS1qE9b0wy8mwXwCPSRblSjOlE/zt+L/5AAL10=} Expected: {l:32 JQpYuexsMn59Flzsvbrgsf3KIDzTQtuhJhZSe0ekRIY=}
2022-01-14 17:11:28, Info CSI 0000000b Hashes for file member \SystemRoot\WinSxS\amd64_windows-defender-management-powershell_31bf3856ad364e35_10.0.14393.0_none_1137c685b804c9a3\MSFT_MpScan.cdxml do not match actual file [l:17]'MSFT_MpScan.cdxml' :
Found: {l:32 UqFDoLy5VMfsizUETvkoQFXHXn4v4xbLqhINZlwGsac=} Expected: {l:32 onYwVmaE7nmvQaxo3HC72//JaLJImqFr4ND6Dg7M8Mg=}
2022-01-14 17:11:28, Info CSI 0000000c Hashes for file member \SystemRoot\WinSxS\amd64_windows-defender-management-powershell_31bf3856ad364e35_10.0.14393.0_none_1137c685b804c9a3\MSFT_MpThreatDetection.cdxml do not match actual file [l:28]'MSFT_MpThreatDetection.cdxml' :
Found: {l:32 PAgqVc0OLYNXZLtVw1KSJ3s8OnqQlUMSOpt1vd9+8g8=} Expected: {l:32 oZvwAOuVDd2EOMMFYsD6bzwISEMvVaIZ7z15swKqbPE=}
2022-01-14 17:11:28, Info CSI 0000000d Hashes for file member \SystemRoot\WinSxS\amd64_windows-defender-management-powershell_31bf3856ad364e35_10.0.14393.0_none_1137c685b804c9a3\MSFT_MpPreference.cdxml do not match actual file [l:23]'MSFT_MpPreference.cdxml' :
Found: {l:32 3VV5hrUQdPPkt9XAkjkDvxC6kZoSNWBfXLqKSbBe8H0=} Expected: {l:32 zbiTyZoXoGxGmd+AOLbOJqmYHoTwkMkcKkoXw+Uz/ng=}
2022-01-14 17:11:29, Warning CBS Current tick count lower than last tick count. [HRESULT = 0x8007000d - ERROR_INVALID_DATA]
2022-01-14 17:11:29, Info CSI 0000000e@2022/1/14:23:11:29.922 Corruption detection complete. numCorruptions = 9, Disp = 1.
2022-01-14 17:11:29, Info CBS Repr: CSI meta data corruption found, will commit repair transaction if repair is asked.
2022-01-14 17:11:29, Info CSI 0000000f Transaction merge required, do-not-merge flag passed.
2022-01-14 17:11:29, Info CSI 00000010@2022/1/14:23:11:29.933 CSI Transaction @0x26ceb7dede0 initialized for deployment engine {d16d444c-56d8-11d5-882d-0080c847b195} with flags 00000000 and client id 'TI5.30935450_300963947:1/'

2022-01-14 17:11:29, Info CSI 00000011 Pending transaction content must be resolved, cannot begin another transaction yet
2022-01-14 17:11:29, Error CSI 00000012 (F) E_UNEXPECTED #5203700# from Windows::COM::CComponentStore::InternalTransact(...)[gle=0x8000ffff]
2022-01-14 17:11:29, Error CSI 00000013@2022/1/14:23:11:29.934 (F) onecore\base\wcp\componentstore\com\store.cpp(4765): Error E_UNEXPECTED originated in function Windows::COM::CComponentStore::InternalTransact expression: ((HRESULT)0x8000FFFFL) [gle=0x80004005]

So I don't know if this is the root or only cause but it appears there are corrupted defender files in the Component Store causing problems.

Tech · Jan 28, 2022

Bump...
Does anyone have information on fixing a messed-up Component Based Servicing/Component Store?

philc43 · Jan 31, 2022

Hello Tech,

Yes, we can provide help. The hash errors you have highlighted need fixing. I have provided a fix below which includes these files all with the correct hash keys.

Warning: this fix is specific to the user in this thread. No one else should follow these instructions as it may cause more harm than good. If you are after assistance, please start a thread of your own.

Download SFCFix.exe (by niemiro) and save this to your Desktop.
Download the file below, SFCFix.zip, and save this to your Desktop. Ensure that this file is named SFCFix.zip - do not rename it.
Save any open documents and close all open windows.
On your Desktop, you should see two files: SFCFix.exe and SFCFix.zip.
Drag the file SFCFix.zip onto the file SFCFix.exe and release it.
SFCFix will now process the script.
Upon completion, a file should be created on your Desktop: SFCFix.txt.
Please Attach this file into your next post for me to check.

Afterwards you can try a DISM /Online /Cleanup-image /RestoreHealth and if there are further errors please provide the new set of CBS Logs.

Tech · Jan 31, 2022

I ran the script with SFCFix and it completed successfully. I was then able to run "DISM.exe /Online /Cleanup-image /Scanhealth" and it completed successfully with "No component store corruption detected."
The server is in active use so I won't be able to try applying the updates again or anything else that would require a reboot until Thursday evening.
Thanks,

philc43 · Feb 1, 2022

That looks promising but there are still a number of errors showing in the logs, however, I think they are mainly related to the pending updates which you are aware of. Please do a reboot when you are able to and see what happens. If the updates fail after a reboot then please upload a new set of logs.

Tech · Feb 3, 2022

I rebooted the server around 18:10, then tried installing the recent SSU kb5005698 - after 10-15 minutes or so it reported that installation failed. Tried running sfc /scannow - it reported "There is a system repair pending which requires reboot to complete. Restart Windows and run sfc again." Rebooted around 18:40, sfc reported the same results. Ran a check for updates and after a while it returned an error 0x8024500C.

Tech · Feb 3, 2022

and here's the pending.xml and migration.xml

philc43 · Feb 4, 2022

Hello Tech,

I'd like to have a look at your COMPONENTS hive.

Retrieve Components Hive
1. Navigate to C:\Windows\System32\Config and locate the COMPONENTS file.
2. Please copy this file to your desktop.
Note: If you receive an error that this file is in-use, simply reboot your computer and try again.
3. Right-click on this file on your desktop and select Send To...Compressed (zipped) folder. This will create a file named COMPONENTS.ZIP on your desktop.
4. The file will likely be too large to upload here so please upload to a file sharing service. Examples of services to upload to are Dropbox or OneDrive or SendSpace and then just provide the link in your reply.

Tech · Feb 4, 2022

It's at COMPONENTS.7z (13.86MB) - SendSpace.com

philc43 · Feb 5, 2022

Thanks Tech.

I'd like to remove the following two values from your Components hive and see if it makes a difference.
PendingXmlIdentifier
ExecutionState

The following procedure will do this:
Warning: this fix is specific to the user in this thread. No one else should follow these instructions as it may cause more harm than good. If you are after assistance, please start a thread of your own.

If needed, download SFCFix.exe (by niemiro) and save this to your Desktop.
Download the attached file, SFCFixScript.txt, and save this to your Desktop. Ensure that this file is named SFCFixScript.txt - do not rename it.
Save any open documents and close all open windows.
On your Desktop, you should see two files: SFCFix.exe and SFCFixScript.txt.
Drag the file SFCFixScript.txt onto the file SFCFix.exe and release it.
SFCFix will now process the script.
Upon completion, a log should be created on your Desktop: SFCFix.txt.
Please attach this file into your next post for me to check.

Tech · Feb 7, 2022

I used Disk2VHD to create a vhdx copy of the server's C: and created a VM clone from it so I can test and reboot as needed. I ran your sfcfix script, sfc still reported the pending repair error, I then rebooted and the system rebooted several times and when it came back up sfc ran and reported that it didn't find any integrity violations. Windows Updates then said 4 updates were available: Office update KB5002138, SSU update KB5005698, Security update KB4535680, and Flash Removal KB4577586, it appeared to be hung at "Downloading updates 38%" for about 30 minutes, then it finally continued downloading and installed the updates and asked for a reboot. After the reboot Windows Updates said 2 updates were available: SSU update KB5005698 and Security update KB4535680, I clicked install and they installed without requiring a reboot. I clicked check for updates and after a long time it said there was one update 2022-01 Cumulative KB5010790 that it started downloading and then seemed to hang at 9%, I let it sit for an hour and there is no change.
CBS log is at CBS.7z (2.10MB) - SendSpace.com

philc43 · Feb 7, 2022

It looks as if the system is almost fully recovered but ran into a problem with the last update.

Please run DISM /Online /Cleanup-image /RestoreHealth in an elevated Command Prompt and then post the new CBS.log created.

Tech · Feb 7, 2022

The restore operation completed successfully.
The operation completed successfully.

philc43 · Feb 7, 2022

Please reboot and check again for updates. If it fails post the CBS.log again.

Tech · Feb 7, 2022

Seems to be stuck at Downloading Updates 9% again.

philc43 · Feb 8, 2022

There appears to be a missing component on your VM which is preventing this latest update:

Code:

2022-02-07 10:41:22, Info                  CBS    Appl: detect Parent, Package: Package_4038_for_KB5010790~31bf3856ad364e35~amd64~~10.0.1.2, Parent: Microsoft-Windows-ServerStandardEdition~31bf3856ad364e35~amd64~~10.0.14393.0, Disposition = Detect, VersionComp: EQ, BuildComp: EQ, RevisionComp: EQ, Exist: present
2022-02-07 10:41:22, Info                  CBS    Appl: detectParent (exact match): Parent: Microsoft-Windows-ServerStandardEdition~31bf3856ad364e35~amd64~~10.0.14393.0, parent state: Installed
2022-02-07 10:41:22, Info                  CBS    Appl: Evaluating package applicability for package Package_4038_for_KB5010790~31bf3856ad364e35~amd64~~10.0.1.2, applicable state: Installed Invalid
2022-02-07 10:41:22, Info                  CBS    Unexpected: Source State: Absent, Target State: Installed Invalid

I want to check another section of your registry. Please export the Component Based Servicing hive by following the instructions below, make sure you save the export as a Registry Hive file:

Click on the Start button and in the search box, type regedit
When you see regedit on the list, right-click on it and select Run as administrator.
When regedit opens, using the left pane, navigate to the following registry key and select it by clicking on it once.

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing
Once selected, click File > Export....
Change the Save as type: to Registry Hive Files (.).
Name this file ComponentBasedServicing (with no file extension) and save it to your Desktop.
Right-click on the saved file and choose Send To -> Compressed (zipped) Folder.
Attach the .ZIP file to your next post.
If the file is too large to upload here, upload to Dropbox or OneDrive or SendSpace and just provide the link here.

Tech · Feb 8, 2022

File is here ComponentBasedServicing.7z (1.52MB) - SendSpace.com
Should I go through the deleting SoftwareDistribution process?

philc43 · Feb 8, 2022

Should I go through the deleting SoftwareDistribution process?

No need at the moment. Please wait for further intructions after I have had a look through the file.

philc43 · Feb 8, 2022

Have you tried downloading the update from the Microsoft Catalogue and installing it that way?

http://download.windowsupdate.com/c..._df441589ce8556dc94d051df30be306266b25828.msu

Server 2016 updates install failure

Active member

Attachments

Active member

Active member

BSOD Forum Moderator, BSOD Academy Instructor, BSOD Kernel Dump Expert

Attachments

Active member

Attachments

BSOD Forum Moderator, BSOD Academy Instructor, BSOD Kernel Dump Expert

Active member

Attachments

Active member

Attachments

BSOD Forum Moderator, BSOD Academy Instructor, BSOD Kernel Dump Expert

Active member

BSOD Forum Moderator, BSOD Academy Instructor, BSOD Kernel Dump Expert

Attachments

Active member

BSOD Forum Moderator, BSOD Academy Instructor, BSOD Kernel Dump Expert

Active member

Attachments

BSOD Forum Moderator, BSOD Academy Instructor, BSOD Kernel Dump Expert

Active member

Attachments

BSOD Forum Moderator, BSOD Academy Instructor, BSOD Kernel Dump Expert

Active member

BSOD Forum Moderator, BSOD Academy Instructor, BSOD Kernel Dump Expert

BSOD Forum Moderator, BSOD Academy Instructor, BSOD Kernel Dump Expert