Latency

The case

The site is online. But in Tokyo, it's slow.

Isaac's store runs in São Paulo and is fast for people who buy from Brazil. He started selling coffee to Japan, and there every click seems to take forever. The server is the same. What changed was the distance.

This post builds on the concepts of Cloud Fundamentals and sets the stage for Regions and Zones.

The concept

What latency is

Latency is the time it takes for data to go from one point to another. In practice, what matters most is the round-trip time (RTT): the request leaves the browser, reaches the server, and the response comes back.

It's like sending a question by mail. It doesn't matter how long the letter is: if the recipient lives on the other side of the world, the answer takes a long time to arrive. Writing faster doesn't solve it; the distance is still there.

Latency is measured in milliseconds (ms). It sounds like little, but a page usually makes dozens of round trips before it appears in full, and the milliseconds add up.

The limit

Blame physics

Data travels as light inside optical fiber, at about 200,000 km/s, two-thirds of the speed of light in a vacuum. It's fast, but not instantaneous: every 1,000 km costs 5 ms one way, or 10 ms round trip.

From São Paulo, the theoretical minimum round trip to each city.

From São Paulo to	Distance	Minimum round trip (round-trip)
Rio de Janeiro	361 km	4 ms
Virginia (USA)	7.651 km	77 ms
London	9.496 km	95 ms
Tokyo	18.534 km	185 ms

No code optimization beats physics. To reduce that part of the latency, there is only one way: shorten the distance.

Don't mix them up

Latency × bandwidth

Both are often called "internet speed", but they measure different things.

Latency (latency)

“How long does the data take to arrive?”

Example: A click in Tokyo waits for the response from a server in São Paulo.
Focus: Distance and response time, in milliseconds.

Bandwidth (bandwidth)

“How much data passes at the same time?”

Example: Downloading a 4 GB video over a 100 Mbps connection.
Focus: Volume per second, in Mbps or Gbps.

Think of a highway. The bandwidth is the number of lanes: how many cars pass at the same time. The latency is the time of the trip from one end to the other. Widening the road lets more cars through, but it doesn't shorten the trip.

In practice

How to measure

The simplest way is the ping command, which measures the round-trip time to an address:

text

$ ping exemplo.com
64 bytes from 93.184.215.14: icmp_seq=0 ttl=56 time=118 ms

In real systems, nobody looks at a single measurement. The time varies with every request, and the average is misleading: a few very slow requests hide behind many fast ones. That's why percentiles are used:

p50 (median)

Half of the requests were faster than this value. It's the typical experience.

p95

95% of the requests were faster. It shows the common bad cases.

p99

Only 1% was slower. In millions of requests, that 1% is still a lot of people.

The solution

How to reduce it

If distance is the problem, the solution is to bring the content closer to who uses it, and to make fewer trips.

Closest region (closest region)

Run the application in the region closest to most of the users. That is the subject of Regions and Zones.

CDN

A network of servers spread around the world keeps copies of static content (images, videos, scripts) close to each user.

Cache

Store responses that were already computed, so that the work and the trip don't have to be repeated for every request.

Fewer round trips (fewer round trips)

Bundle several requests into one. Each round trip avoided is latency saved.

CDNs work with points of presence (edge locations) in dozens of cities. Someone in Tokyo gets the store's images from a server in Tokyo, not from São Paulo.

A CDN spreads copies of the content across points like these.

The providers offer their own CDNs or partners' CDNs. Some examples: Amazon CloudFront on AWS, Azure Front Door on Azure and Cloud CDN on Google Cloud.

Key terms

Six terms to remember

Tap a card to see the definition.

Quick test

Three questions, exam style

A company doubled its bandwidth, but users on the other side of the world still see the site as slow. What explains it?

A video site has users on several continents. Which service reduces the latency to deliver the videos?

An API's average response time is 80 ms, but some users complain about slowness. Which metric reveals the problem?

Latency shows up in every cloud certification: AWS, Azure, Google Cloud or Oracle Cloud. It is what justifies regions close to the users, CDNs and caches. The names of the services change; the physics is the same.