Where the last 40 ms of a request actually go
We traced a slow checkout from the browser to the database and back. Most of the time was not where we expected.
A checkout that felt slow to customers in Sydney looked fine on every dashboard we had. The median was quick. The p99 was not, and the p99 is where people decide whether a site is broken.
We followed one request end to end, from the tap on the Pay button to the confirmation on screen. The function itself took 3 ms. The rest went to three places nobody had looked: a database read that crossed an ocean to the primary, a retry that waited out a full timeout before trying again, and a TLS handshake that was set up fresh on every call instead of being reused.
None of these showed up in the median, because most requests came from cities close to the primary. They only appeared for the users furthest away, which is exactly the group a global product is supposed to serve.
We moved the read to the nearest replica, shortened the retry so it fired after 50 ms instead of two seconds, and kept connections warm between requests. The p99 from Sydney went from 410 ms to 38 ms. These are demo figures for this template; the method is the part worth copying. Trace the slow tail, not the average, and fix the distance first.
We now run the same trace every week from six cities, and any request whose p99 doubles opens a ticket before a customer notices. It costs one scheduled function and a few minutes of reading.