Software patching is a necessity for every business and yet a decade after this idea became an established fact it still looks and feels more like a risky black art than a predictable form of engineering. There are a number of complex technical problems in play here but a new analysis from security firm Tripwire has come up with a single phrase it thinks sums up the condition as a whole: patch fatigue.
Security firms like to ambulance-chase problems because there are always plenty of problems to chase but 'patch fatigue' doesn't feel like the usual hyperbole. Amidst survey numbers drawn from talking to nearly 500 US-based IT professionals, it identifies a number of pressure points that seem to be getting worse rather than better.
The business world has never had so many patches from so many vendors, many of whom never used to update anything unless their software minions has written a new version of a product. It should be the ideal world and yet it has become very heavy going. The contribution this state of affairs has made to software insecurity is impossible to calculate but it's clear that without some new thinking, something will eventually topple over.
First, some figures that underline the scale of what lies behind the term 'patching' itself. Based on the Common Vulnerabilities and Exposures (CVE) database, 2015 saw 6,000 new vulnerability additions across a wide range of products, vendors and open source projects. Most organisations above a certain size will be affected by several hundreds of those, between one and three per day.
According to Tripwire, almost half of the IT admins it asked admitted they were now struggling to keep up with patching, with 76 percent owning up to being confused about which patch should be applied to which system. The connection between this and the negatives outcomes such as data breaches is hard to assess but the importance of software flaws in documented attacks is there for all to see.
Issue 1 - patching has become too time-consuming
Tripwire's assumed desktop image would have required 188 security patches during 2015 with these having to be applied across anything from a few hundred to 5,000 systems in an organisation. It sounds like an obvious problem but patching has risen dramatically over the last decade as the scale of software vulnerability and exploits exploded. If applying some of these patches requires a dreaded reboot, the process of applying them in a server environment can quickly become onerous and lead to downtime.
Issue 2 - some vendors are easier to patch than others
The easiest to patch were said to be Google's Chrome, Red Hat Linux, WordPress while the toughies were Oracle Database, Oracle Java, and Cisco IOS. Interestingly Microsoft Windows and VMware vCenter divided opinion, making it on to both lists. Some people find them a pain, others don't. Overall, however, Microsoft was seen favourable, Oracle definitely wasn't. Cisco is seen as too complex.
This doesn't necessarily have anything to do with the number of security patches. RHEL issued 2,859 in 2015, Microsoft Windows 2,804, while Oracle only issued 276 for its products plus 116 for Java.
Issue 3 - Structured patching is good but critical fixes are paramount
Microsoft kicked this idea off in 2003 after XP started to buckle under sustained pressure from cybercriminals, since when most other vendors have followed suit with a monthly cycle. Cisco's is still quarterly while Apple's is intermittent. About a third of organisations would still prefer to get critical security patches out of band when they are available rather than wait until the next stop on the patching cycle. Overall, the industry has done well but the different cycles is not always helpful.
Issue 4 - Windows 10 branching has added to confusion
Microsoft's shift towards Current Branch (CB), Current Business Branch (CBB), and the Long-Term Servicing Branch (LTSB)patching for Windows 10 was inevitable given the different needs of enterprises using this software but has proved confusing. Each has its own versioning and rules - the firm even released an update buried within a general OS update - promoting 41 percent of respondents to suggest that this had made life more difficult.
"Windows 10 is a very different OS than we've seen from Microsoft in the past and the structure has thrown people for a loop right now. I don't know if we're going to see improvements of not or if people will become more accustomed to it," commented Tripwire's Tyler Reguly.
Issue 5 - bulletin quality matters
The quality of bulletins, which tells admins what to patch, how to patch it and which flaws this will resolve, are now a major factor in current patch fatigue to the extent that some now describe them as much a hindrance than a help. Vendors vary in the quality of their bulletins with Microsoft generally rated as good for providing step-by-step patching instructions and Oracle once again called out by a few respondents for the workload required to understand what is being communicated.
"There is too much click-through, 100 links on the landing page. That is a lot to go through," comments Reguly of Oracle's typical bulletin design. "It is very clear that there are some vendors who could improve their bulleting process. The one that stands out is Oracle."
Reguly argues for an industry consortium to address the bulletin issue by defining how these should be designed.
Issue 6 - Java and Flash need sorting out
A simple win when it comes to patching is simply to remove potentially vulnerable software, particular desktop components that might not even be needed. This is called reducing the 'attack surface'. Top candidates for this are, not surprisingly, Oracle's Java and Adobe's Flash, neither of which is seen as essential despite still having a strange grip on many organisations. The peak year for Java vulnerabilities was 2013, since when the number has dropped
Admins suffering 'patch fatigue: Lesson 7 - patch management v vulnerability management
A subtle distinction can be drawn between patch management (the application of a software patch across an enterprise) and vulnerability management (which also checks for any steps that must be applied after the patch has been applied). According to Tripwire, only 43 percent were able to spot this distinction, which also effects products that are no longer supported but which might still contain vulnerabilities. Patch management won't notice this.
"A lot of people assume that they are overlapping tools that do the same job. There is a big difference between resolving a vulnerability and applying a patch," says Reguly.
In a perfect world, machines would patch themselves but for now that looks a way off. In only ten years patching has gone from the act of a conscientious employee to an everyday occurrence built into the job roles of admins across the globe. The next stage is to make it easier to do - and understand.