The Most Important Book In Payments Is a Data Systems Book
Challenging the belief that good software design isolates business concerns on the domain layer.
There’s nothing like a technical advantage your competitors don’t understand.
A few years ago, I interviewed at Capchase. I had the chance to talk to Ezequiel Cura, their SVP of Engineering.
He never asked me any algorithm questions. He didn’t ask me to solve a system design either.
What he did was tell me that he was looking for someone willing to spend a few years studying a single book.
That book was Designing Data Intensive Applications.
In the end, I didn’t join Capchase. But I bought the book anyway. And after reading it, I realized what Ezequiel was after.
Designing Data Intensive Applications is a blueprint for what payments systems are going to be.
Most engineers building payments bounce between conversations on databases and conversations on distributed systems. It’s as if the two actual hard things in computer science are migrations and race conditions.
And I think I know why.
I got a hint of that a few weeks ago, while doing research on Caitie McCaffrey’s distributed sagas. Here’s what I wrote back then:
At the end of the paper, there’s an intriguing sentence that caught the attention of McCaffrey: “We believe that a saga [as a distributed system] can be implemented with relatively little effort”.
It was as if the authors had left the most valuable part of the paper as an exercise to the reader.
A challenge.
That was the insight that McCaffrey needed. Sagas were the missing step between the single relational database and the distributed system that could make Halo 4 possible.
To create a system that felt like a relational database but was distributed, McCaffrey had to design a data system that preserved some of its guarantees, but didn’t need to be built on a single machine. That’s why she got inspired to build a distributed system after reading a database paper.
And that got me thinking.
When I wrote how Uber tests in production, I argued that creating a staging environment to test services you can’t control is pointless.
Payment systems work like the Sagas that McCaffrey described, but with a twist. In payments, some services are beyond our reach.
And that is the real challenge. That’s the biggest reason why Uber can’t test Google Pay in staging. It depends on Google.
When customers key in their credit card numbers, they trust that you’re going to take good care of that data. But your payment system depends on services that are built at other companies. Companies that sometimes seem as if they’re run by monkeys.
That is the paradox of payment systems: to be correct when you depend on an infrastructure that isn’t dependable.
The central theme of Designing Data Intensive Applications.
I’m Alvaro Duran, and this is The Payments Engineer Playbook. Scroll for five minutes on Youtube and you’ll find tons of tutorials that show you how to pass software design interviews that use payment systems. But there’s not much that teaches you how to build this critical software for real users and real money.
The reason I know this is because I’ve built and maintained payment systems for almost ten years. And I’ve been able to see all types of interesting conversations about what works and what doesn't behind closed doors.
These conversations are what inspired this newsletter.
In The Payments Engineer Playbook, we investigate the technology that transfers money. All to help you become a smarter, more skillful and more successful payments engineer. And we do that by cutting off one sliver of it and extract tactics from it.
One of the biggest pushbacks I’ve got on this newsletter was about its core topic. Why do we need to separate payments engineers from the rest?
Some people, even veterans, have dismissed that payments systems have specific trade-offs. Good architects, they might say, can split any system into a domain-aware “service” layer and a general-purpose “infrastructure” layer.
They believe that people who need to know about the specifics of the domain haven’t made that split correctly.
I think they’ve got a point. Good payment systems demand a layer that contains all the business logic, and another that stores, processes and communicates the outcomes of that logic.
But after all these years, I’ve realized that while the domain specifies the software that implements the business logic, it also informs the trade-offs that the infrastructure layer must engage in to make the system viable.
It’s not that payments have to follow the regulations properly. They must also guarantee the data integrity, and provide reasonable levels of availability under strong consistency.
Think about it: why would Stripe announce that their database supported 99.999% uptime? Because it makes a business difference.
In fact, an important factor in Shopify’s success has been the design of an infrastructure that is not only scalable, but also very elastic to peaks like Black Friday:
According to Bart de Water, a Shopify pod is a mini-version of the whole Shopify application, with all the data for a few of its merchants. This architecture follows what de Water calls Tenant Isolation Principle: each shop in the platform doesn’t know about any other shop.
That is a version of the Separate Schema data architecture, but at scale.
That’s why I believe Designing Data Intensive Applications should be part of your library. There’s no way you can build a competitive payment system without being familiar with the specific challenges of data systems.
Payments demand complete ownership of the tech stack.
That is the technical advantage some engineers refuse to understand. The domain is explicit in the business logic. But it is implicit in the data systems that support it.
This has been The Payments Engineer Playbook. I’ll see you next week.
PS: I have something important that I want to share with you.
August has been the best month ever for this newsletter. I’m a bit hesitant to share numbers, but enough said it’s been bananas.
There’s a lot of people who want to learn more about payment systems. And I have lots of ideas. It’s only chance that there are companies out there who have shared enough information about them. It makes crafting a compelling story much, much easier.
Over the next few weeks, I’ll share stories from companies like Dropbox, Linkedin and Netflix. I can’t wait to tell you all about it.
If that sounds interesting, you can help me by doing one of two things.
The first one is to share this article with a colleague at work, one that could benefit from reading The Payments Engineer Playbook. You can tell them what you like about it. You can share this article with them.
The second thing you can do is to give a like to this article on LinkedIn. You know how the social media algorithms work: they show to others what people like you have found interesting. A 👍 is all it takes.
And if someone you respect shared this article with you, do me a favor and subscribe. Every week I feel I’m getting better at this. That means that my best articles on how to build payment systems are probably yet to be written.
You can only find out if you subscribe to The Payments Engineer Playbook. I’ll see you around.