Update on Microsoft’s Symbol Server
"Can you please make available the symbols for the latest public non-beta Creators Update ntkrnlmp on the MS Symbol server? Having a real hard time debugging issues w/out them."
"We specifically require the symbols for ntoskrnl.exe (ntkrnlmp.pdb) in order to debug a critical deadlock in the file system drivers we are using. Would it be possible to get these new symbols ASAP?"
We've received a lot of emails like this in the past 6 months or so. Developers trying to do the right thing and develop on the latest Windows previews were having problems because our symbol pipeline couldn't keep up with how quickly Windows was releasing.
When the Windows Insider program started releasing builds on a cadence quicker than once a month, we realized we really need to re-work how our symbol processing works in order to make sure that developers on the fast ring would be able to properly debug their drivers and applications. We started down this long road many months back, and as of today, we're moving the back-end behind
https://msdl.microsoft.com/download/symbols/ to an Azure-based symbol store that is backed by a more robust publishing pipeline than the one we've had in the past.
Here's a bit about the journey it took to get here and some of the inner workings of our engineering system. Note that most of this is Windows focused, but relevant to all Microsoft published symbols (Windows is just the vast majority of what we get feedback around).
Where are we coming from?
The symbol server that's in use today has been around for 10+ years, back when it was brought online, Microsoft shipped all its products every few years in cardboard or plastic boxes, and the uploading of symbols was something that someone had to do a few times a year at most. As time has gone on, the actual hardware powering the symbol server has of course changed and grown with the scale of Microsoft and Windows, and we've automated a lot of the actual publishing of symbols, but the services that process the files and make them available publicly have largely stayed the same.
As the release model for big releases and patches for Windows evolved from XP all the way to 8.1, there weren't any major issues with symbols. A patch would come out on a Tuesday, symbols would be available that Thursday or so, and everything would be fine until the next month. Big releases were typically years apart with longer runways to get public symbols in the hands of customers. Back when this was all being setup, the processes that were put into place to publish symbols took somewhere between one and three days and no one really noticed. The actual time it took for symbols to be available depended a lot on when during the day the request was made, how many other requests had been made recently, and normal maintenance tasks for all the servers and services involved.
With the monthly patch cadence, symbols tended to go through that delay before most people even had the update installed, let alone started debugging crashes on it. Then Windows 10 and the Windows Insider Program came around…
What changed?
If you go look through the Windows Insider Blog for late 2016 or early 2017, you'll notice a pretty clear trend. The Insider Program really likes releasing Windows builds whenever possible. With fast ring releases going out every few days, there were builds that were out of date before symbols were even available. Developers, rightfully so, started clamoring about how difficult it was to develop and debug on Insider builds due to missing symbols.
A short delay on symbols being available relative to a monthly release was minor, a short delay when builds are coming out weekly is a workflow blocking issue.
So is it fixed?
Not entirely but we’ve hit a point where we wanted to start talking about the progress more publicly. We've removed the largest impacting factor on the delay. We took measurements with some recent releases and found that the new back-end has symbols available right along-side the build they're related to. There are still some areas in our pipeline that can cause delays which we are already working on, and we'll be closely monitoring what we have completed and any feedback we get from developers as the system evolves.
When is the change happening?
We're incrementally moving traffic over from the old back-end to the new one starting today. We don't anticipate any issues with the transition, we're doing it in a way that you really wouldn't notice unless you read this blog post, but if you hit any problems that you think could be related, feel free to comment here or email
WinDbgFB@Microsoft.com
