The case
The fix from Wednesday that nobody saw. The error started in the deploy, and only showed up on Friday.
Isaac's store published a new version of the checkout on Wednesday afternoon, by hand, with no automated tests. Over the following days, one in every twenty purchases failed at payment. Nobody noticed, because no alert existed. Only on Friday, with complaints arriving on WhatsApp, did the team find the problem.
This post builds on Google Cloud: the starting point, which introduces Google Cloud's services. It also depends on Kubernetes and GKE and Serverless and Cloud Run, because that is where the software runs once it is delivered.
The problem
Delivering by hand means delivering in the dark
When each deploy depends on a manual step, the team makes a different mistake every time. And when nobody measures the system in production, the first to know about the problem is the customer.
Automated delivery
From code to production, with no manual step
Automated delivery (CI/CD) follows a fixed path: the code is tested, packaged and taken to production by the same process, every time.
Build (Cloud Build)
Runs the tests and generates the artifact, such as a container image, from the code.
Artifact storage (Artifact Registry)
Stores the generated images and packages, with versions and cleanup policies, so the same artifact reaches every environment.
Continuous delivery (Cloud Deploy)
Takes the same version from the test environment to production, with approval when the store needs it.
In Isaac's case, a pipeline would have run the payment tests before the deploy, and the version would only reach production after they passed.
Observability
Knowing what happens before the customer complains
Observability is the ability to understand the state of the system from the data it emits. On Google Cloud, the base is Cloud Monitoring (metrics, dashboards and alerts), Cloud Logging (logs) and Cloud Trace (the path of each request between services).
SRE
Reliability measured, not promised
The practice of SRE (Site Reliability Engineering) treats reliability as a measured target. The team defines an SLO (service level objective), a measurable objective, such as "99.9% of purchases completed without error in the month". How much the system can fail without breaking that objective is the error budget.
While there is budget left, the team can release new features. When it runs out, the focus returns to stability. This way, the decision to release or to slow down comes from a number, not from a debate.
APIs
Exposing services with control
An API is how a system offers services to other systems. A gateway sits at the entrance and receives the calls before they reach the service: it checks who is calling, limits the volume and records each use.
| Service | What it is for | When it makes sense |
|---|---|---|
| API Gateway | Managed gateway for serverless services | A simple API, on top of functions or Cloud Run |
| Apigee | Full API management | Many APIs, with usage plans, a developer portal and analytics |
At the store, the API that checks inventory can be exposed to delivery partners. The gateway limits each partner to a number of calls per minute, and the record shows who used what.
Key terms
Five terms to remember
Tap a card to see the definition.
Quick test
Three questions, exam style
01
The store's pipeline builds a container image for each version. Which service stores these images so they reach every environment unchanged?
02
The team wants to be alerted when the rate of purchases with errors goes above 1% for five minutes. Which service does that?
03
At the end of the month, the checkout's error budget is already used up. Which SRE practice should the team follow?
In any cloud exam, the operations question is usually the same: how does software reach the customer, and how does the team know it is working. On Google Cloud, the answer goes through automated delivery, monitoring and SRE targets.