Mastering GCP Virtual Machines – Part 6 (Patching)
In Part 5 we take a look at some key networking tools that help you secure and troubleshoot our environment.
Now in this part we will see how to use the VM Manager in Google Cloud to orchestrate patching for our VM instances. Finally this is Google’s counterpart to Microsoft’s Azure Update Manager.
VM Manager is a suite of tools that you can use to manage operating systems for large virtual machine (VM) fleets running Windows and Linux on Compute Engine.
VM Manager helps drive efficiency through automation and reduces the operational burden of maintaining these VM fleets.
The following services are available as part of the VM Manager suite:
- Patch: Use this service to apply on-demand and scheduled patches. You can also use Patch for patch compliance reporting in your environment.
- OS inventory management: Use this service to collect and review operating system information.
- OS policies: Use this service to install, remove, and auto-update software packages.
About mastering Azure Update Manager you can read my following post.
Patch (GCP’s Counterpart to Azure’s Update Manager)
Use Patch to apply operating system patches across a set of Compute Engine VM instances (VMs). Long running VMs require periodic system updates to protect against defects and vulnerabilities.
The Patch feature has two main components:
- Patch compliance reporting, which provides insights on the patch status of your VM instances across Windows and Linux distributions. Along with the insights, you can also view recommendations for your VM instances.
- Patch deployment, which automates the operating system and software patch update process. A patch deployment schedules patch jobs. A patch job runs across VM instances and applies patches.
We also need to enable the VM Manager to view OS info, we can enable it either for the current project or just for individual VMs.
We can enable the VM Manager directly under current project -> Compute Engine -> VM Manager -> Patch -> Dashboard.
VM Manager is available at no charge for up to 100 VMs per account. See pricing details
We can use the the Patch dashboard for monitoring and reporting of patch jobs and compliance status.
A few minutes later the dashboard will show all 4 VM instances running Ubuntu OS in my project.
To use the Patch feature, we must set up the OS Config API and install the OS Config agent.
The OS Config service enables patch management in our environment while the OS Config agent uses the update mechanism for each operating system to apply patches.
Updates are pulled from the package repositories (otherwise called the distribution source package) or a local repository for the operating system, finally the same as in Microsoft’s Azure Update Manager.
We can automatically enable VM Manager for all new VMs in our organization, folder, or project by using the Require OS Config organization policy.
I am not using an organization in Google Cloud but the config is enabled by default on all new projects.
enable-osconfig=TRUE is included in the project metadata for all new projects.
More about here https://cloud.google.com/compute/vm-manager/docs/setup.
About how to set up an organization in Google Cloud, you can read my following post.
Create a Patch Job
We can now create a new patch job under VM Manager -> Patch to configure either an on-demand or scheduled task that applies operating system updates to our selected VM instances.
It tells the OS Config agent on each VM to run the native updater (such as apt, yum, zypper, or Windows Update), installs available patches, and reports the results back. Patch Jobs can run immediately or at a specified time, making them ideal for ad-hoc or planned maintenance.
We can patch either individual VMs, all VMs in a zone, region or project and all VMs in a Managed Instance Group (MIG).
An instance group is a collection of virtual machine (VM) instances that you can manage as a single entity.
Azure’s equivalent of a GCP MIG is a Virtual Machine Scale Set (VMSS).
Compute Engine offers two kinds of VM instance groups, managed and unmanaged.
Managed instance groups (MIGs) let you operate apps on multiple identical VMs. You can make your workloads scalable and highly available by taking advantage of automated MIG services, including: autoscaling, autohealing, regional (multiple zone) deployment, and automatic updating.
Unmanaged instance groups let you load balance across a fleet of VMs that you manage yourself.
More about here https://cloud.google.com/compute/docs/instance-groups.
By default, a standard system patch will be performed on all targeted VMs. You can override the default settings here.
More about here https://cloud.google.com/compute/vm-manager/docs/patch?#whats-in-a-patch.
If a patch job is still running beyond this duration window below, it usually will be cancelled although some patch jobs cannot be stopped at a late stage.
A 60-minute window often works well, especially for Linux machines. However, there is wide variance among OSes, machine types (patching takes more time on less powerful machines), software packages, non-patch activity on the VM, and new technical complications. As a result, some trial and error may be necessary, or you can set a very long window (for example, 2880 minutes).
The zone rollout settings let you specify how many zones are patched in parallel and how many VMs within each zone are patched at the same time.
This way, you can stagger updates across your managed instance groups or fleets of VMs to balance patch speed with service availability.
For the patch reboot options I will use the default settings. The agent will decide if a reboot is necessary by checking well known signals on each OS.
Finally we can click Deploy.
Under Compute Engine -> VM Manager -> Patch -> Job schedules we will find our newly created job schedule.
We can also pause it here.
Active running patch installation jobs we will see in the Jobs Run tab as shown below.
When clicking on a running job above, we will see its details like shown below.
We will also get detailed logs about the running patch installation when switching above to the Logs tab.
OS inventory management
Google Cloud’s VM Manager (OS Config) has a concept similar to Azure Update Manager’s assessment, but it’s framed a bit differently.
Instead of a standalone “assessment job,” GCP uses OS inventory and patch compliance reporting. The OS Config agent collects package and patch state from VMs and reports it back to the service.
Patch jobs in GCP can also include a dry-run mode (--dry-run flag in gcloud), which simulates the patch and tells you what would be updated, without actually applying it.
Use OS inventory management to collect and view operating system details for your virtual machine (VM) instances. These operating system details include information such as hostname, operating system, and kernel version.
You can also get information about installed OS packages, available OS package updates, Windows applications and OS vulnerabilities.
More about you will find here https://cloud.google.com/compute/vm-manager/docs/os-inventory/os-inventory-management.
Troubleshooting
Integrity validation failed” or “Boot integrity check failed – policy violation”.
After system updates that modify boot components, such as kernel or bootloader changes, the VM may fail to boot because its measured values no longer match the stored integrity policy.
In this case, the serial console output typically shows messages like
“Integrity validation failed” or “Boot integrity check failed – policy violation”.
Integrity monitoring lets you monitor and verify the runtime boot integrity of your shielded VM instances using Cloud Monitoring. Requires vTPM to be enabled.
The initial integrity policy baseline is derived from the implicitly trusted boot image when the instance is created.
Updating the baseline updates the integrity policy baseline using the current instance configuration. The VM instance must be running when you update the baseline.
You should update the baseline after any planned boot-specific changes in the instance configuration, like kernel updates or kernel driver installation, as these will cause integrity validation failures.
If you have an unexpected integrity validation failure, you should investigate the reason for the failure and be prepared to stop the instance if necessary.
You must have the setShieldedInstanceIntegrityPolicy permission to be able to update the integrity policy baseline.
Use the following procedure to update the integrity policy baseline.
Update the VM instance’s integrity policy baseline by using the compute instances update command with the
--shielded-learn-integrity-policyflag.Source: https://cloud.google.com/compute/shielded-vm/docs/integrity-monitoring
This command updates the VM’s integrity policy with the new trusted boot measurements, allowing it to boot normally again.
$ gcloud compute instances update <instance name> --zone=<zone> --shielded-learn-integrity-policy $ gcloud compute instances update instance-ubuntu-vm01 --zone=europe-west3-a --shielded-learn-integrity-policy
In Part 7 we will see how to protect and backup VM instances by using the Google Cloud Backup and DR service and Management Console (appliance).
Links
About VM Manager
https://cloud.google.com/compute/docs/vm-managerAbout Patch
https://cloud.google.com/compute/vm-manager/docs/patchInstance groups
https://cloud.google.com/compute/docs/instance-groups
