Relational Databases Aren't Paying Off In Payments
Choosing Relational Databases for Payments was the result of a historical accident. Should the payment industry move on to better alternatives?
Welcome to Money In Transit, the newsletter for startup founders who find themselves dragged into payments technology. I’m Alvaro Duran.
Since I presented Working in Units at EuroPython last year, I’ve been having scary thoughts about the possibility that relational databases aren’t a good fit for payment applications. And I wanted to marinate those thoughts into something clear and explicit, so that they could be open to criticism. Which, honestly, it’s something even scarier.
Consider sharing this post with someone who has recently complained about making migration changes to their payment application.
The payment industry made a mistake when it converged on relational databases.
I became uncomfortable with this type of database about 2 years ago, when I had to fix a bug on a payment application I was working on. This application had implemented transactions everywhere. Transactions are a database mechanism, particular of the relational model, that treats a set of changes as a single unit, so that you can undo them when things go wrong.
But, boy, undoing things is the last thing you want in payments. You can’t have holes in your audit trail!
So, here’s an industry where the systems that store data blindly follow other industries’ “best practices”. Like using transactions, or thinking of your data as if it were laid out on a spreadsheet. But, sometimes, engineers find themselves making horrible mistakes because of that.
It would be OK if this is a choice that was made early on deliberately. But it isn’t.
Storing data in the format of the spreadsheet has a very long history.
Relational databases have their roots in the data processing that mainframes were already doing for banks in the 1960s. Bankers needed a system where they could enter payment transactions easily, and they could get invoices, payrolls and reports automatically.
These systems were an instant hit. And, despite the fact that there’s been many competing approaches over the years, and computers were being used for more purposes as they became more powerful, relational databases have generalized amazingly well. Competitors came and went in hype waves that never lasted. The relational model came on top every single time.
As a result, a lot of payment applications today are built on top of relational databases. It seems like a solved problem already. Something you choose without thinking too much about it.
A piece of boring technology.
An Obstacle Not Worth The Hassle
Relational databases are great because they enforce a bunch of guarantees over your data.
It rejects inserts that try to add text where numbers are expected. Same with references to nonexistent rows on other tables. Those get rejected too.
But none of these guarantees have anything to do with business concepts. Relational databases deal in terms of computer concepts—business needs are out of their scope. If you’re not careful, relational databases can accept double payments. They can open the door to race conditions that could even bankrupt your company.
Constraints are helpful, but only when they’re also meaningful to the business. But using relational databases means accepting that connecting the guarantees it provides with what the end product can benefit from is left as an exercise to the developer. While also requiring a lot of operational work to have them.
Have you considered the possibility that relational databases might not be worth it?
Engineers Should Not Decide What Is Correctness
What kind of payment applications could we build without the restrictions that relational databases impose?
This is not a theoretical exercise. Even today, thousands of payment applications are limping along, maintained by exhausted developers who could improve these applications faster if they weren’t forced to live with these restrictions.
They put up with this nightmare because they believe that relational databases ensure the quality of the data. That these restrictions stabilize the ground on which they can build, operate and expand their payment applications, and without them they would be building on top of thin ice. But every developer I know has lived through situations where the data was plain wrong, despite all the checks and balances.
Only the business can determine correctness. Whatever guarantees relational databases can give, they are analogous to compiler errors. Good to know, but not enough.
I much rather have few guarantees that mean something than a bunch of constraints that are not directly connected with the product’s goals.
Moore’s Law and Data
The history of data and databases is analogous to the history of compute power and Moore’s Law.
More powerful machines have led to growing ambitions. Since the 1960s, when relational databases first appeared, the payment industry has witnessed explosive technological acceleration.
I’m skeptical that payment applications that use relational databases can adapt to what’s coming. For instance, most of them still assume that credit cards are the predominant payment method online. But a Cambrian explosion of new methods (iDEAL, UPI, PIX, and a long etc) has changed everything. The industry’s reaction has been slow, and full of compromises. Most applications have built support for these new methods on top of abstractions built for cards.
Those abstractions are unsuitable, but that is as fast as they can go. Hamstrung by early decisions, this is all that they can do.
Are relational databases really paying off in payments?