The ID Gapless-Latency tradeoff

You can get unique identifiers at scale either by Coordinating or by Centralizing ID creation. But you can't get gapless sequences without having to wait it.

We don't pay enough attention to IDs.

In fact we only care about them when they exhaust:

At 7:21am CST, we first got alerted that we had run out of ID numbers on an important tracking table in the database. This was because the column in database was configured as an integer rather than a big integer. The integer runs out of numbers at 2147483647. The big integer can grow until 9223372036854775807.

— Update on Basecamp 3 being stuck in read-only

We don't think enough about them in our day to day. We're only aware of IDs when we require things like gapless sequences, or uniqueness.

IDs can do so much more than that.

But first, the basics. IDs are crucial to differentiate entities in our domain by something other than their values. That's because databases use predicate logic, and it would be impossible to do that unless there's a column that's guaranteed to be unique.

The way most databases enforce this guarantee is by having the primary key be a serial or auto-incremental type. This type assigns a monotonically¹ increasing integer to each row. Most common database engines also guarantee that this sequence is gapless, which is pretty convenient for things like invoices.

However, these databases can only make guarantees within their own instances. And that's when things get tricky.

Successful payment systems scale. And while I agree that scalability is overrated, payment systems should be designed to be scalable from the get-go.

That's because light isn't fast enough. Not across global distances. Even small startups now serve customers from every corner of the globe; placing all your data in one place is fragile. The time it takes to serve a payment request from Auckland with a server in us-east-1 (North Virginia), plus the time it takes to reach the card issuer (back in Auckland, most likely) is dangerously close to timeout limits.

This approach makes legit payments from distant places get rejected.

We solve this latency problem by sharding our databases: placing smaller buckets of data in different places across the Earth that will serve requests locally.

This is what companies like Stripe or Shopify do: they move data around to achieve lower latency, bu also better load and faster analytics.

And it would be impossible if they didn't have a mechanism to handle IDs in a distributed manner.

I'm Alvaro Duran, and this is The Payments Engineer Playbook. It is the only newsletter in the world dedicated exclusively to the builders of money software.

In today's article, we're going to cover how to start thinking about IDs that scale globally. Here's what this article will cover:

• What are Ticket Servers

• How Snowflake's IDs are created

• What Einstein's relativity theory has to do with all this

Enough intro, let's dive in.