Nobody Got Fired for Uber's $8 Million Ledger Mistake?
LedgerStore became a case study for DynamoDB, and system design publications keep praising it to this day. But the design was abandoned 3 years after it went live.
Uber has rewritten its ledger systems five times in the last ten years. And at least one of those rewrites, if not all, could have been avoided.
That’s because the root of each generation of money software at Uber was driven from bad incentives. Each started with a brand new proposal, approved as the definitive solution; in time, a fatal flaw was surfaced; and finally, a new proposal came along to replace it.
Every rewrite was someone’s promotion project.
At least one of them could’ve been avoided: the one where Uber moved to DynamoDB. In 2017, Uber launched their new payment platform on it, and the critical factor that everyone involved seemed to miss was that DynamoDB is a consumption-priced database.
You pay for every read, and every write.
With each trip generating multiple ledger entries, and Uber as a whole processing 15 million trips per day, it didn’t matter that DynamoDB was great because of high throughput at global scale. The proverbial bean counter should’ve stopped this madness from happening.
Within 2 years, the cost became prohibitely expensive:
At Uber’s scale, DynamoDB became expensive. Hence, we started keeping only 12 weeks of data (i.e., hot data) in DynamoDB and started using Uber’s blobstore, TerraBlob, for older data (i.e., cold data). TerraBlob is similar to AWS S3. For a long-term solution, we wanted to use LSG.
— Migrating a Trillion Entries of Uber’s Ledger Data from DynamoDB to LedgerStore
A redesign that gets replaced 2 years later is a catastrophe.
And yet, history remembers Uber’s ledger on top of DynamoDB as a masterpiece. As late as 2024, ByteByteGo has an article praising it.
And that’s what concerns me. Uber’s design was a failure, but nobody seems to remember it that way.
That ends today.
I’m Alvaro Duran, and this is The Payments Engineer Playbook, the only newsletter on Earth tailor-made for engineers of money software. Every week, more than 2,000 subscribers from companies like Stripe, Coinbase and Modern Treasury get a dive deep on how to build software that moves money around. Not to pass interviews, but to do their job exceptionally well.
When money is on the line, stakes are sky high and the margin for error is razor thin.
In The Payments Engineer Playbook, we investigate the technology that transfers money. All to help you become a smarter, more skillful and more successful payments engineer. And we do that by cutting off one sliver of it and extracting insights from it.
Here’s what you can expect in today’s article:
Why DynamoDB works for payments but breaks when you use it as a ledger
The napkin math that would have saved Uber millions of dollars
And one shocking conclusion from all of this
Enough intro, let’s dive in.
