From Stock Markets to Ledgers, part II: Fault-tolerance

Paxos is Not-Invented-Here consensus. That's why Raft is succeeding.

There’s a reason why there aren’t many newsletters focused on engineering money software: because people think they’re pointless.

On the one hand, you’ve got the payment experts. They know all about what make issuing banks tick. They get excited about Visa’s VAMP program or Account Updater. They do A/B tests, know the data, and can speak about it.

On the other hand, you’ve got the software engineers. They talk BDD, DDD, TDD. They secretly hope for a rewrite in Rust, have strong opinions about using AI, and publish them on their own blogs in articles that start with unless you’ve been living under a rock.

But these two worlds never talk to each other. Like, ever. Because they agree that you can completely separate the two worlds, domain and engineering. Payment experts would say that the implementation is either what they expect, or wrong. And engineers would say that they can build anything, for any domain whatsoever, following standard patterns.

I can only suspect that this is unlikely in other domains. But I can confidently say that in the world that builds money software, believing that you can separate domain expertise from engineering practices is blatantly wrong.

I can actually prove it to you.

Over the last ten years, there’s been an explosion of distributed systems. Back in the day, you had to buy and rack your own servers, and operate them yourself. There were managed databases like Oracle sitting on top of Windows operating systems, served with Apache HTTP software.

At some point, however, people realized that, even if hardware was keeping up the pace of the Internet, it was much more convenient to deploy distributed systems instead of a single monolith for everything.

Many very important workloads can fit on one such machine.

Or could, if scale was all we cared about.

Scale, and scalability, is only a small part of the overall reason distributed systems are interesting. Other practical reasons include: Availability, Durability, Utilization, Latency, Specialization, Isolation, and Changes.

— Marc Brooker, Not Just Scale

Availability is the most obvious of them all: single machine systems can only improve availability by diminishing the chances of the machine going down, or reducing the time to go back up when it does. A distributed system just needs to keep one machine up, even if all the rest are down.

Why Payments Engineers Should Avoid State Machines

The problem with highly available systems, as it is often the case, is consistency.

The most basic thing you can do to maintain “good enough” consistency in a distributed system is to have a Primary + Secondary setup. One machine is up, and the other sits idle. If something happens to the primary, there’s a failover to the secondary machine, which temporarily sustains traffic until the Primary is back up.

However, there are two fundamental problems with the Primary + Secondary setup. First, you have effectively one machine running. You may have zero downtime deployments, and small availability issues are somewhat mitigated, but traffic concentrates on one machine at a time. And second, you can’t have more than 2 machines per setup, and the second one is sitting there, wasting time and resources.

That said, we’ve had better configurations for distributed systems already available since the late 1980s. Liskov’s Viewstamped replication and Lamport’s Paxos improve on the Primary + Secondary setup by allowing all machines within a distributed system to figure out consistency in a collaborative way. That’s why they’re called consensus algorithms.

Stock markets had nevertheless strongly resisted phasing out systems configured with Primary + Secondary setups in favor of consensus.

Even though it’s known to be flawed.

The pushback doesn’t come from the payment experts. In fact, the lawyers are scared shitless about Primary + Secondary, mostly because this configuration is very prone to manual mistakes at the worst possible time. If your system servers retail customers, and your primary machine fails, you’re running a very real legal risk if a customer doesn’t get a fair price. What if your secondary fails too, or starts degrading because of the load?

NoDB: Processing Payments Without a Database

But the pushback doesn’t come from the engineers either. Consensus algorithms are proven to be fault-tolerant. Like math-proven. That’s very desirable for engineers, who wouldn’t hesitate to adopt a configuration that’s proven to be without any bugs.

The pushback is implicit: consensus algorithms are too difficult to implement correctly.

This week, we’re going to look at fault-tolerance in stock market systems from the point of view of fidelity.

In this article, we’ll see:

• Why correctness has diminishing returns

• What makes fidelity a more desirable property

• Why Raft is taking over the stock market industry

• What consequences does it have for ledger systems

I’m so excited about this, so let’s go right in.