Microsoft's ongoing patching mishaps: Who's in charge of this mess, anyway?

Microsoft ships replacement patch KB 2993651 with two known bugs

By InfoWorld Tech Watch

Created 2014-08-28 04:41AM

Even by Microsoft standards, this month's botched Black Tuesday Windows 7/8/8.1 MS14-045 patch hit a new low. The original patch (KB 2982791) is now officially "expired" and a completely different patch (KB 2993651) offered in its stead; there are barely documented revelations of new problems with old patches; patches that have disappeared; a "strong" recommendation to manually uninstall a patch that went out via Automatic Update for several days; and an infuriating official explanation that raises serious doubts about Microsoft's ability to support Windows 9's expected rapid update pace.
I've been covering (and suffering) Microsoft's patching mishaps for more than a decade, and I have just one question: Who the hell is in charge of this mess?
As of early this morning, one Windows 8 user was reporting black screens [1] with the new patch, KB 2993651. Answers Forum posters pacman10, JohnBurgessUK, and chadlan can't get Windows Update to check for new updates after installing KB 2993651 (although rseiler reports all's well). It's too early to tell for sure, but there may be more problems with the new patch.
It all harkens back to the Blue Screen Stop 0x050 error (in Windows 7) and the black screen (in Windows 8/8.1) attributed totwo bad kernel-mode driver updates [2] that went down the Automatic Update chute on Black Tuesday, Aug. 12. Two days later, a Windows customer and denizen of the Microsoft Answers forum found a manual workaround [3] that let people with bricked machines get back up and working. Microsoft finally pulled four bad patches -- KB 2982791, KB 2970228, KB 2975719, and KB 2975331 -- on Friday night. As I documented at the time [4], it took Microsoft more than three days to acknowledge the problem publicly and another day to pull the patches.
It looks like those four bad patches turned belly-up when they encountered OpenType fonts with links in the \Fonts folder. That's not a typical situation, but it's perfectly valid. Microsoft employee Kurt Phillips, posting on the Answers Forum [5] main thread, put it this way:
One thing to keep in perspective here - the actual numbers we get through telemetry (clearly not exhaustive, but definitely representative) are that the failures are only happening in ~0.01 percent of the overall population. So, about 1 in 10000 machines are crashing. We have an obligation to fix that, and we will because we take that obligation very seriously... Just wanted to clear up some of the hyperbole - Microsoft isn't crumbling, all of our testers weren't fired, etc. 99.99 percent success is pretty good in most jobs in this world, but clearly we need to strive for higher.
Of course, Phillips is right. Brushing aside the question of how Microsoft gathers telemetry on bricked machines, 0.01 percent of the 1.5 billion Windows users (25 percent of whom are on XP and aren't affected) is a small percentage but a large number.
Yesterday, apparently without any warning, Microsoft re-released MS14-045, changing the KB number(s) associated with the patch. In Windows 7/8/8.1, KB 2993651 is now offered in place of the old KB 2982791, which no longer exists. (Strikingly, the Knowledge Base article for KB 2982791 [6] hasn't been updated and doesn't reflect the demise of the now-disavowed patch.)
This advice appears at the very end of the lengthy MS14-045 article [7]:
Why was this bulletin revised on August 27, 2014? What happened to the original 2982791 security update? To address known issues with security update 2982791, Microsoft rereleased MS14-045 to replace the 2982791 update with the 2993651 update for all supported releases of Microsoft Windows. Microsoft expired update 2982791 on August 15, 2014. All customers should apply the 2993651 update, which replaces the expired 2982791 update. Microsoft strongly recommends that customers who have not uninstalled the 2982791 update do so prior to applying the 2993651 update.
I already successfully installed the original 2982791 security update and am not experiencing any difficulties. Should I apply the replacement update (2993651) released on August 27, 2014? Yes. All customers should apply the 2993651 update, which replaces the expired 2982791 update. Customers do not need to uninstall the expired 2982791 update before applying the 2993651 update; however, Microsoft strongly recommends it. Customers who do not remove the expired update will retain a listing for 2982791 under installed updates in Control Panel.
I uninstalled the original 2982791 security update. Should I apply the August 27, 2014 rereleased update (2993651)? 
Yes. To be protected from CVE-2014-0318 and CVE-2014-1819, all customers should apply the rereleased update (2993651), which replaces the expired 2982791 update.
It's amazing to me that Microsoft recommends -- in a dusty corner of an obscure document -- that people who were bitten by KB 2982791 need to go in and manually uninstall it. We're talking about a bad patch that Microsoft spread through the Auto Update mechanism for almost four days. Why the new patch, KB 2993651, doesn't uninstall the bad old patch absolutely blows my beleaguered mind. I don't know for sure and Microsoft isn't saying, but my guess is that if you (or your dearly sainted Aunt Mable) leave KB 2982791 installed, and at some point in the future you happen to install an OpenType font with a link in the \Fonts folder, your machine will blue screen (or black screen) when you next reboot. Try explaining that to Aunt Mable -- or your CEO.
This shiny new replacement patch, KB 2993651, actually ships with two known bugs. The new KB 2993651 article [8]explains the first bug like this:
Known issue 1
After you install this security update, fonts that are installed in a location other than the default fonts directory ( percentwindir percent\fonts\) cannot be changed when they are loaded into any active session. Attempts to change, replace, or delete these fonts are blocked, and a "File in use" message is displayed.
I'd call that weird but not overwhelming. But the second bug not only affects this new patch, it also comes along with old patches. Specifically:
Known issue 2
After you install this update, the z-order of the windows is changed. (The z-order calls the SetWindowPos function together with the HWND_TOP parameter.) Therefore, the windows of certain applications may become invisible or may be incorrectly displayed behind other windows.
Status 
We are currently working on a resolution for this issue.
Notes
This issue also occurs after you install the following updates:
2965768 Stop error 0x3B when an application changes the z-order of a window in Windows 7 SP1 and Windows Server 2008 R2 SP1
2970228 Update to support the new currency symbol for the Russian ruble in Windows
2973201 MS14-039: Description of the security update for Windows on-screen keyboard: July 8, 2014
2975719 August 2014 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2
Todd Cassell at the Bentley Technical Support Group has more details on this newly announced z-order bug [9]:
The issue was first introduced in KB2965768 on 5 Feb 2014. In that version win32k.sys makes changes to the z-order of the windows. That z-order calls the SetWindowPos function) . KB2973201 (8 Jul 2014) and KB2982791 (12 Aug 2014) also contained these changes to win32k.sys. Updates that affect the behavior of the toolboxes as well are: KB2970228 and KB2975719. The issue may also be in other security or service updates released since 5 Feb 2014. The last well working version of win32k.sys (Windows\system32) is: 6.1.7601.22665 from 23 Apr 2014.
Due to dependencies with .dll and other files, just replacing win32k.sys is NOT possible and will result in system crashes. Currently the only solution is to uninstall KB2965768, KB2970228, KB2973201, KB2975719, KB2982791 and KB2993651.as that will bring win32k.sys back as the working version.
If I may cut through the alphabet soup for a second, that means this newly released patch, KB 2993651, and another active patch -- KB 2973201 [10], the July Black Tuesday patch for the Windows on-screen keyboard -- have known bugs. Microsoft just released the new, known-to-be-faulty patch and continues to offer the older bad patch knowing that both have bugs wherein "windows of certain applications may become invisible or may be incorrectly displayed behind other windows." There's no indication of when the z-order problem will be fixed, how many machines are affected, or what the source of the problem might be. If you experience problems with dancing windows, you have to manually uninstall six patches to restore win32k.sys to working order.
Other than the newly announced z-order bug, I still don't have any idea what's happening with this month's other three pulled patches: KB 2970228 [11], which adds the ruble to valid currencies, KB article last updated Aug. 20; KB 2975719 [12], the vestiges of what we once laughingly called "Windows 8.1 Update 2," [13] KB article last updated Aug. 22, at revision number 11; and KB 2975331 [14], a giant Windows 8/8.1/RT patch rollup, KB article last updated Aug. 22, revision number 8, not in the z-order bug list. If you find any of those missing bad boys, have them drop me a line, OK?
As a historical note, every single vestige of Windows 8.1 Update 2 has now been pulled, as best I can tell. All that happy talk about rapid "Update Tuesday" deployment of Windows updates has succumbed to the hard-core realities of patching Windows, which is a bear. [15]
Against that background, Tracey Pretorius, director of Microsoft's Trustworthy Computing effort, posted a blog [16] on the Microsoft Security Response Center yesterday, explaining why MS14-045 was released. This is such an astounding piece of ... I don't know what to call it ... that I'd like to take it apart, piece by piece. It starts:
Every month for many years, we've released a number of updates focused on the continuous improvement of customers' experiences with our technology. Historically, these updates happened at different times during the month, with the security-specific ones occurring on the second Tuesday of each month. Recently, to further streamline, we decided to include more of our non-security updates together with our security updates and begin the global release to customers on the second Tuesday of each month.
That's a distinctly revisionist retelling of the migration to a "fourth Tuesday" patching cycle and Microsoft's apparent attempt to return to the old method, wherein all of the in-band non-emergency patches come out on the second Tuesday. I gave a rather different telling of the history [17] earlier this month.
Pretorius continues:
This month we had our first roll out with additional non-security updates. We then began working on a plan to rerelease the affected updates.
That's simply not true. Microsoft has been issuing non-security updates on Black Tuesday (the second Tuesday) for many years. As I said last week in my blog about how Microsoft's new approach to Windows updates is all marketing sizzle no steak [17], "From where I sit, this month's Patch Tuesday feature improvements weren't as interesting as last month's -- or the month before, or the month before that, for that matter."
In August Microsoft not only released a bumper crop of Black Tuesday patches, it also had a first Tuesday patch (KB 2973544 [18]), that was botched and re-issued on the fourth Tuesday. Take a look at the official master list [19] and I guarantee your scrolling finger will grow weary. First rollout with additional non-security updates? Hardly.
Back to the Pretorius blog:
A small number of customers experienced problems with a few of the updates. As soon as we became aware of some problems, we began a review and then immediately pulled the problematic updates, making these unavailable to download.
I guess a Microsoft-estimated 0.01 percent of 1 billion = 100,000 customers is "small" by Microsoft standards, but I personally heard from dozens of folks who bricked their systems, and read forum posts from dozens more. It took Microsoft almost four days to pull the problematic updates. At this point, exactly none of the original "Windows 8.1 Update 2" improvements are available for download; they've all been yanked. There's a newly identified series of z-order bugs that affects five patches, two of which are still shipping with the bug. And there are Gates-knows-how-many millions of Automatic Update lemmings out there with a time bomb of an old security patch sitting around ticking, and obscure instructions telling users they are supposed to manually remove it.
We're supposed to trust this company with an accelerated rate of patches, a concentrated "Update Tuesday [20]," in the new Windows Threshold scheme of things?
Tell me again. Who the hell is in charge of this mess?
This story, "Microsoft ships replacement patch KB 2993651 with two known bugs [21]," was originally published atInfoWorld.com [22]. Get the first word on what the important tech news really means with the InfoWorld Tech Watch blog [23]. For the latest developments in business technology news, follow InfoWorld.com on Twitter [24].

---

Source URL (retrieved on 2014-09-04 11:02PM): http://www.infoworld.com/t/microsoft-windows/microsoft-ships-replacement-patch-kb-2993651-two-known-bugs-249342

Links:
[1] http://answers.microsoft.com/en-us/windows/forum/windows_7-windows_update/blue-screen-stop-0x50-after-applying-update/6da4d264-02d8-458e-89e2-a78fe68766fd?page=56
[2] http://www.infoworld.com/t/microsoft-windows/blue-screen-stop-0x050-error-reported-systems-installing-kb2976897-kb2982791-and-kb2970228-248363
[3] http://www.infoworld.com/t/microsoft-windows/users-find-fix-botched-kb-2982791-and-kb-2970228-windows-update-248476
[4] http://www.infoworld.com/t/microsoft-windows/microsoft-yanks-botched-black-tuesday-patches-kb-2982791-kb-2970228-kb-2975719-and-kb-2975331-248582
[5] http://answers.microsoft.com/en-us/windows/forum/windows_7-windows_update/blue-screen-stop-0x50-after-applying-update/6da4d264-02d8-458e-89e2-a78fe68766fd?page=49
[6] http://support.microsoft.com/kb/2982791
[7] https://technet.microsoft.com/library/security/ms14-045
[8] http://support.microsoft.com/kb/2993651
[9] http://communities.bentley.com/products/microstation/w/microstation__wiki/15905.toolboxes-disappear.aspx
[10] http://support.microsoft.com/kb/2973201
[11] http://support.microsoft.com/kb/2970228
[12] http://support.microsoft.com/kb/2975719
[13] http://www.infoworld.com/t/microsoft-windows/its-official-windows-81-update-2-dud-247773
[14] http://support.microsoft.com/kb/2975331
[15] http://www.infoworld.com/t/microsoft-windows/kb-2919355-you-cant-patch-the-desktop-phone-240878
[16] http://blogs.technet.com/b/msrc/archive/2014/08/27/security-bulletin-ms14-045-rereleased.aspx
[17] http://www.infoworld.com/t/microsoft-windows/microsofts-new-approach-windows-updates-all-marketing-sizzle-no-steak-248573
[18] http://support.microsoft.com/kb/2973544
[19] http://support.microsoft.com/kb/894199
[20] http://blogs.windows.com/bloggingwindows/2014/08/05/august-updates-for-windows-8-1-and-windows-server-2012-r2/
[21] http://www.infoworld.com/t/microsoft-windows/microsofts-ongoing-patching-mishaps-whos-in-charge-of-mess-anyway-249342?source=footer
[22] http://www.infoworld.com/?source=footer
[23] http://www.infoworld.com/blogs/infoworld-tech-watch?source=footer
[24] http://twitter.com/infoworld

http://www.infoworld.com/t/microsoft-windows/microsoft-ships-replacement-patch-kb-2993651-two-known-bugs-249342