Scaling and Load Balancing on Google Cloud

The case

One VM, and the Father's Day campaign took the store down. Four VMs, and one of them stopped at 10 a.m.

During Father's Day week, Isaac put the coffees on sale on the home page. Orders came in within minutes, the only VM ran out of memory, and the site was down for forty minutes.

The following week, he set up four identical VMs. At 10 a.m. on a Tuesday, one of them stopped responding. He only found out when a customer complained.

This post builds on Virtual Machines: from the dedicated server to the cloud and Compute Engine: virtual machines on Google Cloud. We already know how to create a VM; what's missing is making several of them work as one.

The problem

A single VM is a single point of failure

A single VM has two limits. When traffic rises, it can't keep up, and the team has to change it by hand. When it stops, the store stops with it.

Adding more VMs solves the capacity, but it raises another question: who decides which VM serves each customer, and who notices that one of them is sick?

The solution

A group of identical VMs

A managed instance group (MIG) is a set of VMs created from the same instance template. The template holds the recipe: machine type, image, disk and startup script. When the group needs one more VM, it is born identical to the others.

With the group, you describe the result, not each machine. For example: "I want between two and ten VMs for the site, and more whenever the average CPU goes above 70%." Google Cloud adds or removes VMs to meet that. This is autoscaling.

Recovery

The group replaces the VM that got sick

The group can check each VM with an application health check: a periodic request to a page or a port of the application. If the answer doesn't come as expected several times in a row, the group deletes the VM and creates another one from the template. This process is autohealing.

There is an important detail: a new VM takes time to start, and a check run too early would raise a false alarm. That's why autohealing has an initial delay, during which failures are ignored while the machine is still booting.

As for scope, the group can be zonal (all the VMs in a single zone) or regional (VMs spread across several zones in a region). If a whole zone goes down, the regional group keeps going with the VMs in the other zones.

Update

Changing the version without taking the store down

When the site gets a new version, it makes no sense to stop the group to replace all the VMs at once. The group offers two ways to update:

Rolling update: the VMs are replaced in batches. You set how many can be out at the same time, and the group waits for each batch to become healthy before moving on.
Canary update: a small part of the VMs gets the new version, while the rest keeps the old one. If the behavior is good, the whole group moves to the new version. If not, you only need to roll back the part that changed.

To ask for a percentage of VMs in the canary, the group needs at least ten instances. With a smaller group, you choose a fixed number of VMs.

Who receives the request

The load balancer

The group takes care of the VMs, but someone has to receive the customer's request and hand it to a healthy VM. That role belongs to the load balancer. On Google Cloud, the global balancer has a single IP address (anycast), which receives requests from around the world and forwards them to the backends with the best path.

A backend service groups the VM groups that receive the traffic. Each backend has its own health check. When a VM stops answering the check, the balancer stops sending requests to it. It does not delete the VM: the group does that.

Don't mix them up

Balancer health check × group autohealing

Balancer health check

“Where does the request go?”

Example: The VM doesn't respond, and the balancer sends the requests to the others.
Focus: Takes traffic away from the sick VM, without deleting it.

Group autohealing

“Is the sick VM replaced?”

Example: The VM doesn't respond to several checks in a row, and the group creates another one in its place.
Focus: Recreates the VM from the instance template.

In practice, the two work together: the balancer protects the customer while the group solves the problem.

In practice

Which one to choose

Store site on a peak day

Managed group with autoscaling and a balancer in front. The capacity follows the orders.

New version of the site

Canary update: part of the VMs gets the new version, and the team watches before releasing the rest.

Store that can't go offline

Regional group, spread across several zones, with an application health check for autohealing.

One VM for a quick test

No group. A group for a single VM is complication with no gain.

Key terms

Five terms to remember

Tap a card to see the definition.

Quick test

Three questions, exam style

One of the site's VMs returns errors, but the group still considers it healthy. What should happen for it to be replaced automatically?

The store must stay online if an entire zone becomes unavailable. Which kind of group should you choose?

You want to test the new version of the site on only part of the VMs before releasing it to the whole group. Which update should you use?

Scaling and load balancing show up in any cloud: groups of VMs with autoscaling and balancers exist on AWS, Azure and Oracle Cloud, under other names. The idea is the same: the machine that fails leaves the route, and the capacity follows the demand.