The Turbulent Tango Of Payment Providers
Unraveling the Technological Challenges in Making Money Software
Welcome to Money in Transit, a newsletter for merchants for whom payments, whether they like it or not, is critical to their success.
A revised version of this post will appear in my upcoming book The Databases of Money: Designing Payment Applications. If the topic is of interest to you, consider subscribing to be notified when the book is out.
How often do you pay online with credit cards? A lot? Maybe even always?
The credit card schemes, Visa, Mastercard and a few others, have slowly but certainly got a handle on online merchants and their customers. But this is by no means final—other payment systems have competed and still compete with cards in the now ubiquitous domain of online payments.
Apple and Google keep offering new and easier ways to “just pay”, even embedding credit offerings.
Banks are leveraging their distribution advantages to bypass the schemes, with products such as Venmo and Zelle or local services like Bizum, iDeal or Sofort.
Countries as a whole are increasingly becoming key players in this space, bringing forth national rails such as UPI in India, PIX in Brazil, and the upcoming central bank digital currencies.
There are also countless of startups out there building payment methods on top of the ones mentioned—e.g., Marqueta’s partnerships with banks— or alongside them—e.g., wallets like Gojek in Indonesia, or Paypal in the US.
And, of course, crypto companies that are building their own currencies, or using Bitcoin as a platform for their own wallets, like Osmo.
That is an massive density of links, and I haven’t even covered some odd cases, like Klarna’s Buy Now Pay Later, a hybrid between wallet and bank partnership. Nor did I cover those that will appear in the future, or those that were hyped, then sold and eventually written off.
Engineers building payment systems are only mildly acquainted with the ecosystem. They may understand how card payments work, and are capable of building a system that can deal with some use cases.
But the ecosystem is hectic, and the only thing we know for sure is that more things are coming. How can you build payment systems in such circumstances?
Separating The General From The Particular
The really hard and unpleasant conversation about payment systems is the one about moving at pace while avoiding the constraints of your own legacy code. If payment systems were tennis, your goal would be to minimize unforced errors.
That means emphasizing correctness, reversibility and maintainability, even over things like performance or scale. The kind of resilience that is described as cannot fail.
As a result of this, payment systems are made to be understood by people beyond engineering. Making money software were code involves dealing with data structures or memory management is, hence, a mistake. It should require business knowledge, not computer science degrees.
And most importantly, it should be designed in such a way that ensures that if any third party provider is experiencing problems, it wouldn’t affect the system’s ability to keep the money coming.
Payment’s SNAFU
In some form or another, all payment systems deal with third party providers that connect with the scheme, the acquirer, the issuers or the fraud provider services.
Payments combine two defining characteristics that give it its explosive nature. On the one hand, they are conceptually a utility: people expect them to work, all the time1. But on the other hand, this expectation has been ongoing for so long that multiple layers of metaphorical mud and rusty software have accumulated. For that reason, providers are frequently malfunctioning, or simply unable to provide services. Only last month, Square experienced an outage that lasted a whole day, bringing down payments for millions of merchants globally.
Payment systems must anticipate and mitigate potential outages. That is, even the smallest payment systems must deal with at least a few providers, just in case one of them, or even some of them, are down.
Shea’s Law
It takes two to tango. Integrating with a provider is like a trickiest dance, more like meshing than leg entwining, more stressful than dramatic. It’s a full embrace, a leap of faith, estilo milonguero: inward focused.
Dancing tango milonguero with your software provider is fine, as long as you are willing to dance only with them. The problem, though, is twofold: not only are you building a system tailored to the specifications of your provider, you’re also relinquishing control of the relationship, as you’re Just Another Customer for the provider, while for you, they are Everything.
A few companies called gateways have addressed this issue by brokering your technological relationship with your provider, giving you a generic API to connect to and dealing with the specifics of each provider themselves. Still, the problem remains: they are in control, and screwing up a design occurs primarily at the interfaces, which means that you’re at the mercy of their engineering abilities.
Truly valuable payment systems are built to facilitate the integration and decommission of third party providers with minimal impact to the rest of the system.
Distributed Databases
Another, more subtle problem occurs when we address the problem of what data to own. The very moment engineers start storing information on payments, they eventually realize that it’s not fully theirs. Data belongs, in the end, to the systems that produce it, which are most often those very providers whose control you don’t have.
This impacts business intelligence in ways non-engineers are faster than engineers to catch up. Even if you manage to isolate your technology from that of your provider, you still must rely on them as a source of truth for any problem with reporting or analytics. You can duplicate that data internally, and perform business intelligence with it, but it takes engineering work, and you’re always double checking yourself against the provider’s solution.
Engineering Payments means Optimization
In the end, engineers that develop payment systems are up against a few challenges. Many of them tackle the problem woefully unprepared for what’s coming.
But engineering the system correctly opens up a universe of possibilities. Having close contact to the way payments are handled means that the data you are privy too is much more contextualized than that of your providers.
You simply understand your customers better. Which is another way of saying that you are in a better position to infer fraudulent payments in that corner of the Internet that is yours.
Except in countries like the Philippines, where payments, even online, are only open on a 9 to 5 basis, or Malaysia, which switches off everyday at midnight for an hour of back-end shenanigans.