Ledger Transactions Don't Have to Be Atomic
Up-in-the-airness, and looking at ledgers through the lens of Sagas
Financial transactions aren’t database transactions.
Sure, they are the inspiration for database transactions. Turing Award-winner Jim Gray defined the standard measure of database transaction processing in terms of debits and credits. They also follow a layman’s intuition that, when money leaves an account, it immediately arrives at another.
But that’s not what usually happens.
When you make a transfer from your bank account, those funds almost always take some time to arrive. It may be a few seconds, or a business day, or a whole month (if your bank is settling payments made to you in Brazil1).
What happens during that time is treasury. Whoever is holding the funds in transit can use them as a (very) short-term collateral. It can deposit them and earn some interest, it can use them to improve their cash-flow, it can do as they please, as long as the funds arrive where they’re supposed to, when they’re supposed to.
That’s not how database transactions work at all.
Nevertheless, modeling ledger transactions as atomic units can work fine. You wrap debits and credits, make sure they’re double-entry safe, and work under the assumption that all those entries are all pending, or all posted. And if something goes wrong, you archive those entries, knowing that you don’t have to do anything else. It’s clean, safe, and predictable.
You may very well do that; Stripe doesn’t.
Stripe processes billions of events per day through a system they call Ledger. And in Ledger, they don’t assume that both sides of a money movement arrive together. After all, they don’t have to come from the same service, or in the same order. Each event is its own thing, with its own timestamp, source system, and moment of arrival; the only thing they share is an identifier.
That’s one limitation we already saw in stock markets: the risk of quickness versus completeness.
This risk is central to stock market systems. In order to scale, multiple matching engines are running in parallel, processing massive amounts of buy and sell orders with low latency. As soon as two orders are crossed (a buy order matches a sell order for a given price), there’s an execution, and that information gets disseminated quickly and fairly.
But what if something bad happens? What if regulators step in and ask “what happened here first”?
Like, two buy orders for the same price get sent at the same time by two traders, but one gets crossed and the other doesn’t, or does at a higher price.
How can you be sure that you were fair to both orders? Which one is “technically first”?
That, weirdly enough, is an ambiguous question. Because you haven’t clarified which notion of time is being used to sort the orders.
It’s not that transactions in a ledger can’t be atomic; they can. It’s that you don’t need them to.
And the real question is: what are you missing out when you force transactions to be atomic?
I’m Alvaro Duran, and this is The Payments Engineer Playbook, the only newsletter on Earth tailor-made for engineers of money software. Every week, more than 2,000 subscribers from companies like Stripe, Coinbase and Modern Treasury get a dive deep on how to build software that moves money around. Not to pass interviews, but to do their job exceptionally well.
When money is on the line, stakes are sky high and the margin for error is razor thin.
In The Payments Engineer Playbook, we investigate the technology that transfers money. All to help you become a smarter, more skillful and more successful payments engineer. And we do that by cutting off one sliver of it and extracting insights from it.
Here’s what you can expect in today’s article:
Why computer science trained us to think of transactions as atomic, and why that assumption breaks down in distributed ledgers
How Stripe’s Ledger posts individual entries instead of atomic transactions, and gets away with it
The trade-offs between atomic and non-atomic models, and when to pick which
Enough intro, let’s dive in.
