[SOLVED] Server Core 2019 - Unable to install Windows Updates

[CheesyBean]

Windows Version: Windows Server 2019 Datacenter x64 (10.0.17763.5820)

I have a Server Core 2019 server that has just stopped installing Windows Updates. The last one was May 2024 (KB5037765) but it wouldn't install the out of band update (KB5039705) or other security updates since. It will install others though, like ssu, .net.

Ran DISM /Online /Cleanup-Image /RestoreHealth a few times and there are no corruptions anymore.

2024-07-11 11:42:47, Info                  CBS    Last Successful Step: CSI store detection completes.
2024-07-11 11:42:47, Info                  CBS    Total Detected Corruption:    0
2024-07-11 11:42:47, Info                  CBS        CBS Manifest Corruption:    0
2024-07-11 11:42:47, Info                  CBS        CBS Metadata Corruption:    0
2024-07-11 11:42:47, Info                  CBS        CSI Manifest Corruption:    0
2024-07-11 11:42:47, Info                  CBS        CSI Metadata Corruption:    0
2024-07-11 11:42:47, Info                  CBS        CSI Payload Corruption:    0
2024-07-11 11:42:47, Info                  CBS    Total Repaired Corruption:    0
2024-07-11 11:42:47, Info                  CBS        CBS Manifest Repaired:    0
2024-07-11 11:42:47, Info                  CBS        CSI Manifest Repaired:    0
2024-07-11 11:42:47, Info                  CBS        CSI Payload Repaired:    0
2024-07-11 11:42:47, Info                  CBS        CSI Store Metadata refreshed:    True
2024-07-11 11:42:47, Info                  CBS   
2024-07-11 11:42:47, Info                  CBS    Total Operation Time: 482 seconds.
2024-07-11 11:42:47, Info                  CBS    Ensure CBS corruption flag is clear
2024-07-11 11:42:47, Info                  CBS    Ensure WCP corruption flag is clear
2024-07-11 11:42:47, Info                  CBS    All CSI corruption was fixed, ensure CorruptionDetectedDuringAcr is clear
2024-07-11 11:42:47, Info                  CBS    Failed to clear CorruptionDetectedDuringAcr store corrupt flag (slow mode trigger). [HRESULT = 0x80070002 - ERROR_FILE_NOT_FOUND]
2024-07-11 11:42:47, Info                  CBS    CheckSur: hrStatus: 0x0 [S_OK], download Result: 0x0 [S_OK]
2024-07-11 11:42:47, Info                  CBS    Count of times corruption detected: 3
2024-07-11 11:42:47, Info                  CBS    Seconds between initial corruption detections: -1
2024-07-11 11:42:47, Info                  CBS    Seconds between corruption and repair: -1
2024-07-11 11:42:48, Info                  CBS    Reboot mark cleared

If i run sfc /scannow, it always fails at 38% (both before and after a clean DISM /restorehealth report).

2024-07-11 12:17:20, Error                 CSI    00001594@2024/7/11:11:17:20.083 (F) onecore\base\wcp\componentstore\deltastore.cpp(3065): Error STATUS_INVALID_PARAMETER originated in function ComponentStore::CRawStoreLayout::DecompressFile expression: Parameter check failed
[gle=0x80004005]

SFCfix.exe found no issues.

AutoAnalysis::
SUMMARY: No corruptions were detected.
AutoAnalysis:: directive completed successfully.

Successfully processed all directives.
SFCFix version 3.0.2.1 by niemiro has completed.

ComponentScanner found 1 warning.

==== Warnings ====

== f! Mark Count Mismatch ==
amd64_microsoft-windows-d..mc-domainsandtrusts_31bf3856ad364e35_10.0.17763.1697_none_2336f78601fc35e0 has 3 f! marks, expected 1

Attached CBS and DISM logs, and Components hive

 

[Maxstar]

Hi and welcome to Sysnative,

Step 1. Download

SFCFix and save it to your desktop.

Warning: This fix was written specifically for this system. Do not run this fix on another system.

• Save any work you have open, and close all programs.
• Download the attachment SFCFix.zip and save it to your desktop.
• Drag the SFCFix.zip file over the SFCFix.exe executable and release it.

• SFCFix will launch, let it complete.
• Once done, a file will appear on your desktop, called SFCFix.txt.
• Post the logfile (SFCFix.txt) as attachment in your next reply.

Step 2. Reboot the server and run

ComponentsScanner again.

• Right-click ComponentsScanner.exe and select "Run as administrator", click Yes on the UAC (User Account Control) prompt which appears.
• Follow the on-screen instructions.
• Once complete, a report will be saved to your desktop called ComponentsScanner.txt.
• Post the logfile ComponentsScanner.txt as attachment into your next reply.

 

[CheesyBean]

Thanks. Is there a way to do it manually. This is a server core edition and .zip files aren't registered. I tried in at a admin cmd prompt sfcfix.exe sfcfix.zip and it failed.

Using .zip script file at SFCFix.zip [1]
Failed to extract zip archive. Failed to unzip in location 2 with error code 0x1.

 

[Maxstar]

Hi,

Yes, but you'll need to extract the attached ZIP-file on another system

Step 1. Open an elevated command prompt and run the following commands:

icacls "%systemroot%\WinSxS\amd64_microsoft-windows-d..mc-domainsandtrusts_31bf3856ad364e35_10.0.17763.1697_none_2336f78601fc35e0" /save "%userprofile%\desktop\perms.acl" /t
takeown /f "%systemroot%\WinSxS\amd64_microsoft-windows-d..mc-domainsandtrusts_31bf3856ad364e35_10.0.17763.1697_none_2336f78601fc35e0" /r
icacls "%systemroot%\WinSxS\amd64_microsoft-windows-d..mc-domainsandtrusts_31bf3856ad364e35_10.0.17763.1697_none_2336f78601fc35e0" /grant administrators:(F) /t

Copy the WinSxS component directory from Files.zip to "%systemroot%\WinSxS" and run the following commands.

icacls "%systemroot%\WinSxS\amd64_microsoft-windows-d..mc-domainsandtrusts_31bf3856ad364e35_10.0.17763.1697_none_2336f78601fc35e0" /setowner "NT SERVICE\TrustedInstaller" /t
icacls "%systemroot%\WinSxS" /restore "%userprofile%\desktop\perms.acl" /t

Step 2. Please do the following for the *.manifest file:

icacls "%systemroot%\WinSxS\Manifests\amd64_microsoft-windows-d..mc-domainsandtrusts_31bf3856ad364e35_10.0.17763.1697_none_2336f78601fc35e0.manifest" /save "%userprofile%\desktop\perms.acl" /t
takeown /f "%systemroot%\WinSxS\Manifests\amd64_microsoft-windows-d..mc-domainsandtrusts_31bf3856ad364e35_10.0.17763.1697_none_2336f78601fc35e0.manifest"
icacls "%systemroot%\WinSxS\Manifests\amd64_microsoft-windows-d..mc-domainsandtrusts_31bf3856ad364e35_10.0.17763.1697_none_2336f78601fc35e0.manifest" /grant administrators:(F) /t

Copy the *manifest file from Files.zip to "%systemroot%\WinSxS\Manifests" and run the following commands.

icacls "%systemroot%\WinSxS\Manifests\amd64_microsoft-windows-d..mc-domainsandtrusts_31bf3856ad364e35_10.0.17763.1697_none_2336f78601fc35e0.manifest" /setowner "NT SERVICE\TrustedInstaller" /t
icacls "%systemroot%\WinSxS\Manifests" /restore "%userprofile%\desktop\perms.acl" /t

Step 3. Do the the following to import the registry file.
Run the following command in an elevated prompt to load the COMPONENTS hive into the registry:

reg load HKLM\COMPONENTS C:\WINDOWS\SYSTEM32\CONFIG\COMPONENTS

Then import Regfix.reg into the registry.

Let me know the result of these instructions.

 

[CheesyBean]

Hi.

Thanks for that. Only 1 file didn't process.
When running:

icacls "%systemroot%\WinSxS\amd64_microsoft-windows-d..mc-domainsandtrusts_31bf3856ad364e35_10.0.17763.1697_none_2336f78601fc35e0" /setowner "NT SERVICE\TrustedInstaller" /t

I get this:

processed file: C:\Windows\WinSxS\amd64_microsoft-windows-d..mc-domainsandtrusts_31bf3856ad364e35_10.0.17763.1697_none_2336f78601fc35e0
processed file: C:\Windows\WinSxS\amd64_microsoft-windows-d..mc-domainsandtrusts_31bf3856ad364e35_10.0.17763.1697_none_2336f78601fc35e0\Active Directory Domains and Trusts.lnk
C:\Windows\WinSxS\amd64_microsoft-windows-d..mc-domainsandtrusts_31bf3856ad364e35_10.0.17763.1697_none_2336f78601fc35e0\domadmin.dll: Access is denied.
Successfully processed 2 files; Failed processing 1 files

but otherwise, everything else processed fine.

 

[Maxstar]

Hi,

Great, please run the following command as well in an elevated prompt.

certutil -hashfile C:\Windows\WinSxS\amd64_microsoft-windows-d..mc-domainsandtrusts_31bf3856ad364e35_10.0.17763.1697_none_2336f78601fc35e0\domadmin.dll SHA256

The result should be:

fcedb4d4ab2cffa2a0c85da55f1964fec3f8a600a310e10a8741eafb7a9141e9
CertUtil: -hashfile command completed successfully.

 

[CheesyBean]

The result should be:

fcedb4d4ab2cffa2a0c85da55f1964fec3f8a600a310e10a8741eafb7a9141e9
CertUtil: -hashfile command completed successfully.

Yes, that's exactly what I got

 

[Maxstar]

Perfect, please run the ComponentsScannner again and post the result.

 

[CheesyBean]

all good and no warnings

Hive scanned: %windir%\System32\config\COMPONENTS
Number of keys: 162167
Number of values: 384275

==== Critical Errors ====
None

==== Corrupt Key Names ====
None

==== Corrupt Value Names ====
None

==== Corrupt Value Data Type ====
None

==== Corrupt Value Data ====
None

==== Repair Log ====
No possible repairs

==== Warnings ====
None

 

[Maxstar]

Great, please run the System File Checker again. If it fails attach the new CBS logs.

SFC /Scannow

 

[CheesyBean]

Still fails at 38%. CBS.log attached.

 

[Maxstar]

Let's run SFC again with Process Monitor running. You can run this tool from the command line as well.

Capture Process Monitor Trace
1. Download and run Process Monitor. Leave this running while you perform the next steps.
2. Run the System File Checker just like you have in the past.
3. Stop Process Monitor a minute after it fails. You can simply do this by clicking the capture icon (CTRL +E) on the toolbar as shown below.

4. Select the File menu...Save... and save the file to your desktop. This is likely the default location. The name (unless changed) will be LogFile.PML. This is fine.
5. Zip up the LogFile.PML and upload it to WeTransfer - Send Large Files & Share Photos Online - Up to 2GB Free and provide the link.
6. Attach also the latest CBS for the time stamps.

 

[CheesyBean]

I can see Procmon in task manager but it doesn't show a gui. It shows the initial licence agreement then nothing.

 

[Maxstar]

The logfile (*.PML-file) should be saved in the same location?

Edit:
when SFC failed you can use the following command to save the PML trace.

/SaveAs <path>

 

[CheesyBean]

Managed to get it working with procmon.exe /accepteula /backingfile <path>

Here's the download link: <removed>
it is password protected, I'll DM you the pw. Thanks.

 

[Maxstar]

Hi,

Here's the next fix.

Step 1. Open an elevated command prompt and run the following commands:

icacls "%systemroot%\WinSxS\amd64_microsoft-windows-i..raries-servercommon_31bf3856ad364e35_10.0.17763.4252_none_f728389c41d204a5" /save "%userprofile%\desktop\perms.acl" /t
takeown /f "%systemroot%\WinSxS\amd64_microsoft-windows-i..raries-servercommon_31bf3856ad364e35_10.0.17763.4252_none_f728389c41d204a5" /r
icacls "%systemroot%\WinSxS\amd64_microsoft-windows-i..raries-servercommon_31bf3856ad364e35_10.0.17763.4252_none_f728389c41d204a5" /grant administrators:(F) /t

Copy the WinSxS component directory from Files2.zip to "%systemroot%\WinSxS" and run the following commands.

icacls "%systemroot%\WinSxS\amd64_microsoft-windows-i..raries-servercommon_31bf3856ad364e35_10.0.17763.4252_none_f728389c41d204a5" /setowner "NT SERVICE\TrustedInstaller" /t
icacls "%systemroot%\WinSxS" /restore "%userprofile%\desktop\perms.acl" /t

Step 2. Do the the following to import the registry file.
Run the following command in an elevated prompt to load the COMPONENTS hive into the registry:

reg load HKLM\COMPONENTS C:\WINDOWS\SYSTEM32\CONFIG\COMPONENTS

Then import Regfix.reg into the registry.

Step 3. Reboot the server and run the System File Checker again with Process Monitor running.
Attach the latest CBS log and the PML trace to your next post/PM.

 

[CheesyBean]

SFC ran fine. just in the process of obtaining the logs and will zip then transfer it shortly.

Beginning system scan.  This process will take some time.

Beginning verification phase of system scan.
Verification 100% complete.

Windows Resource Protection did not find any integrity violations.

 

[Maxstar]

Great, glad the SFC Scan completed without any issues, for now I don't need any new logs.

I would first try to attempt to update, and when it fails provide the latest CBS logs.

 

[CheesyBean]

The update install! huzzah. The server is still messed up but it's compliant now so it buys us some time to migrate to a newer one. Thank you for your help.

 

[Maxstar]

You're welcome. Glad I could help get this server up to date for now, even there are still other issues! But as mentioned it will give you some time to migrate everything to a new server. So, good luck with the migration...