How often do you apply security patches on Linux?

Regular patching is essential for maintaining a secure environment, but there is no one-size fits all approach for keeping your Linux estate safe. So how do you balance the frequency of updates with operational stability?  There are strategies for enabling security patching automations in a compliant and safe way, for even the most restrictive and regulated environments. Understanding Canonical’s release schedules for software updates and knowing security patching coverage windows are essential pieces of information when defining a security patching strategy. I recently hosted a live webinar and security Q&A whereI explained how to minimise your patching cadence, or minimise the time unpatched vulnerabilities can be exploited. In this article, I’ll summarise my main points from that webinar and explain the most vital considerations for updates scheduling. 

Security patching for the Linux kernel

There are two types of kernels available in Ubuntu, and there are two ways these kernels can be packaged. The two types are the general availability (GA) kernel and the variant kernel. The two package types are debian packages, and snap packages. The GA kernel is the kernel version included on day 1 of an Ubuntu LTS release. Every Ubuntu LTS release receives a point release update every February and August, and typically there are 5 point releases. Ubuntu Server defaults to staying on the GA kernel for the lifetime of the Ubuntu Pro coverage for that release. Ubuntu Desktop defaults to upgrading the kernel from the second point release onwards, to a later version of the upstream kernel, referred to as the hardware enablement (HWE) kernel.

Security coverage for the GA kernel extends for the lifetime of the Ubuntu Pro coverage for that release. Security coverage for the HWE kernel extends for the lifetime of the HWE kernel, which is 6 months, plus another 3 months. This additional 3 months of security coverage beyond the end of life of the HWE kernel gives users a window to perform an upgrade to the next HWE kernel.

Is a reboot required to apply a security patch?

When the kernel package is updated, the Ubuntu instance must be rebooted to load the patched kernel into memory. When the general availability kernel is installed as a snap, updates to this snap package will reboot the device. When the general availability kernel is installed as a deb package, a reboot is not automatically performed, but it must be performed to apply the security patch.

There are some other packages in Ubuntu that also require a reboot, when they are updated. Any security updates to glibc, libc, CPU microcode, and the grub bootloader, will require a reboot to effect. Software which runs as a service will require those services to restart when security patches are applied, examples of such software includes ssh, web servers, and others. Other software which doesn’t run as a service, and runs on demand, doesn’t require a system reboot or service restart to take effect.

The Livepatch service will apply high and critical security patches to the running kernel, in memory. It will not upgrade the installed kernel package, so rebooting the machine and clearing its memory would result in removal of the Livepatch applied security patch. Livepatch provides 13 months of security patches to a GA kernel, and 9 months of security patches to an HWE kernel. After these 13- or 9-month windows, the kernel package must be upgraded and the Ubuntu instance must be rebooted, for continued security coverage through Livepatch.

3 approaches for security patching

Canonical provides an array of tools and services, ranging from Livepatch, Landscape, Snaps, and command line utilities such as unattended-upgrade. These tools and services can be used together, or selectively, and they provide security patching automation capabilities in Ubuntu. You have flexibility with using these tools to achieve a variety of different security patching goals on desktop, server, and IoT devices. Given the assumption that nobody wants to run software that isn’t being secured or supported by its maintainers, you may prefer one of the following security patching approaches:

1. Delay security patching for as long as possible, to achieve maximum procrastination.
2. Perform security patching with the least frequency, but on a predefined regularly recurring schedule.
3. Minimise the window of the time systems can be exploited by a vulnerability, by reducing the time between the release of a security patch, and its installation.

Regardless of which approach is used, it’s worth noting that unscheduled security maintenance windows may be required, to patch security vulnerabilities in glibc, libc, or CPU microcode.

Security patching for procrastinators

With Livepatch enabled, Ubuntu LTS instances that stay on the GA kernel need to be upgraded and rebooted every 13 months. When running on an HWE kernel, the first upgrade and reboot must happen after 13 months, in May. Another upgrade and reboot must happen after 6 months, in November. After another upgrade and reboot activity in the subsequent May and November, the HWE kernel will be promoted to the next GA kernel. The GA kernel will need an upgrade and reboot every 13 months.

Many months may pass between upgrades and reboots, and so when procrastinating in this way, medium and below kernel vulnerabilities will remain unpatched. This approach maximises the amount of time medium and below kernel vulnerabilities remain unpatched.

Security patching on a recurring schedule

If a GA kernel is being used, an annual patching cadence can work. Taking the GA kernel’s 13-month Livepatch security coverage window into account, installing security patches and rebooting every May should keep the kernel and other packages on the machine in secure shape.

Assuming the HWE kernel is being used, an annual patching cadence cannot be used to apply security patches in the same month year after year. This approach would result in a lapse in security patching coverage for the kernel, for a period of time. Excluding the third year of a Ubuntu LTS release, when the fourth point release is published, it is possible to apply security patches once a year when using the HWE kernel.

A bi-annual security patching cadence every May and September would provide peace of mind, regardless of your choice of kernel. This security patching cadence takes Canonical’s release schedule and kernel security coverage windows into account. There is no lapse in security coverage when scheduling security maintenance windows bi-annually, every May and September.

Security patching for the smallest exploit footprint

More frequent security maintenance windows are obviously better. A very common schedule is monthly: there is no chance of running a stale kernel when upgrading and rebooting monthly. Weekly security patching and reboot cadences are recommended, and a daily security patching regimen is applauded. Canonical’s security patches are published as they are made available, it is recommended to take advantage of these security patches as they are released. Running your workloads in high availability, enabling security patching automations, and rolling out phased upgrades and reboots for groups of machines on a daily basis, provides the strongest security posture.

Best practices for enabling security patching automations

The best practices for scheduling security patching automations webinar answers all the big questions:

• What patching automation options are available?
• Where do security patches come from, and where can they be distributed from?
• How are security patches distributed, and how should they be applied? How does Canonical’s rolling kernel strategy extend the security coverage window, for certain kernels?
• When should security maintenance events be scheduled?