In the past I've written a great deal about the steps to take before implementing a patch -- how to prepare for the event, how to log any changes that are made and so on. But what about afterwards? You're probably going to spend more time dealing with what happens after a given patch is applied, so it only makes sense to talk about that as well.
Here are some basic tenets to keep in mind after applying patches -- especially after Microsoft's once-a-month "Patch Tuesday" -- culled from many different people's experiences with administering both remote servers and local workstations.
Be mindful of changes in general system behavior. If something is clearly wrong, it'll tend to announce itself. That said, this is a little easier to do when you are solely responsible for the system in question -- for instance, a server. If you deal with multiple workstations, stay close to one of the patched machines, if you can, and if anyone reports bizarre behaviors, you can investigate them and then try and duplicate them on your own "pet" machine.
This particular theme is going to be echoed throughout this article in different forms, since there are so many ways it can manifest. For instance, if a patch silently changes permissions on objects and breaks undocumented access to that object, that may be by design, so just be mindful of it. Also, if it's a behavior that only an admin (rather than an end user) tends to notice, pay extra attention to it; you may be seeing something other people would miss.
Inspect the system logs. Compare error logs both before and after a patch, and see if anything new or unusual jumps out at you. Sometimes it might not be anything, or it might be coincidental, but error logs are one of the best places to go to for concrete information about something not working correctly. This is especially important if what is going wrong has no other outward symptoms yet, except for a logged error. (Also keep in mind that some errors may simply be false alarms and have no real connection to anything; sometimes it can be hard to tell the difference.)
Check compatibility on any potentially affected applications. If there's a chance a given patch might affect the way a program works, test it before and after the fact. For instance, test a patch to a middleware component by making sure no database connections are suddenly throwing errors. In addition, make sure other non-trivial database operations that take place in your environment -- and that require middleware (like retrieving large amounts of data or opening many connections at once -- also get tested whenever possible.
Be mindful of the effects of "stealth" updates. Sometimes when you patch one thing, five other things get changed in synchrony -- often without you knowing about it. This, in turn, can cause unpredictable behaviors if you didn't prepare for them ahead of time. Microsoft updates are notorious for slipstreaming in updates to .DLLs that sometimes don't seem related to the fix at hand. But the Knowledge Base article for a given hotfix will almost always have a manifest of all the updated components.
Read the news. If word breaks about a certain patch being problematic, don't put yourself in the position of finding out on your own by accident -- or, worse, from one of the people you're supposed to be helping! Stay abreast of news about existing hotfixes and patches, as well as what's to be released; if something that has been released turns out to have unwanted cross-interactions that only turn up later, it's better to know about them before you wind up being a test case.
There's an example of this from back in April 2006, when Microsoft released a patch collection for Windows, MS06-015, which caused about as many problems as it solved. The problems were, in time, untangled, but someone who simply installed the patch and moved on with their day might not have realized the scope of the trouble, or how to address it.
Prepare for feedback. Get ready to answer questions from your users about what's been updated. Some of them might be a lot savvier than you think about vulnerabilities and what's been patched (and what hasn't). Some may have what sounds like an innocent question that turns out to be a symptom of a problem in disguise. Either way, keep your ears open; you never know what they might come to you with.
Stubbornly refuse to panic. If something does go wrong after a patch goes out, keep your cool. Don't let the prospect of undoing something on a dozen or a hundred workstations freak you out. All Microsoft patches have multiple rollback methods and can be reverted by a script or by hand if you have to.
About the author: Serdar Yegulalp is editor of the Windows Power Users Newsletter. Check it out for the latest advice and musings on the world of Windows network administrators -- and please share your thoughts as well!
