From Stock Markets To Ledgers, Part IV: Queues

AMQP failed, but companies building ledgers pretend that it didn't

AMQP was born with the goal of solving business messaging once and for all.

Messages are atomic pieces of data. When you send messages, the recipient either gets them completely, or they don’t. What’s useful about this is that messages can be queued, which releases a lot of pressure on the receiver about the way these messages are processed. Queues also decouple the sending and the receiving of the messages, which is particularly helpful for systems that communicate a lot, but belong to different companies.

Banking systems are almost always designed this way, and that’s why the Advanced Message Queueing Protocol project was started by a bank consortium.

Being this a series on stock exchange systems, you shouldn’t be surprised to learn that middlewares have historically been the main performance bottleneck. Processing north of 500K events per second and executing timely makes it impossible to use TCP because it comes with built-in safety mechanisms that affect throughput and latency.

[A] large part of the uncertainty and the variation that comes from TCP are these self-learning algorithms that are very concerned about network health. When we would work with the Linux kernel maintainers and have questions about variability that we saw, then they would say, “Well, you shouldn’t be using TCP. If you care about latency, you shouldn’t be using TCP. TCP makes trade offs all the time, for network health, and so on and so forth. And for the Internet that is absolutely necessary. And if you really have these super tight requirements, and you really want them to get there fast, and you have a controlled network with very little packet loss, and very few layers between participants, you should be using UDP.”

— Brian Nigito, Multicast and the Markets

Over time, banks have managed to create open technical standards to do business. You may have heard of FIX (Financial Information Exchange), which is used regularly in trading, but it’s so complex that you often need some tool to generate the message for you. Same with FAST (FIX Adapted for Streaming), FpML (Financial products Markup Language) and SWIFT¹: these are terrible protocols to work with, and the only reason people put up with them is because there are no widespread alternatives.

Which begs the question: why not? Well, banks have quite a bad reputation when it comes to technology. The whole reason fintech is a thing is because, for the longest time, banks have considered technology a “non-core competency”. Tech is for nerds. Engineers are the water boys of the football team.

And then, all of a sudden, they weren’t.

It’s 2003. John O’Hara, then Distinguished Engineer at JP Morgan, has had enough. His job is to build front- and back-office processing systems, and it seems to him that banks live in “integration Groundhog Day”.

Every discussion on making things better ended in “the chosen middleware makes it really expensive”.

The issue is, of course, interoperability. Or the lack thereof. It’s the same issue that Modern Treasury’s CEO raised in Bad Software and Expensive Mistakes when he talked about that time Citigroup “accidentally” wired $900 million and then couldn’t get them back:

Building on these interfaces takes investment of time and resources, because bank cores are challenging to integrate with and understand. Some banks use Direct Transmission over SFTP while others provide SOAP APIs. Timings vary. There are different, competing, data exchange formats for things like reporting on errors, and the configurations can vary from banks on identical cores.

When pressed for resources, teams cut corners. It’s hard to justify investing in wonderful design and robust systems for internal processes, so much of what’s out there is half-built, inflexible, and non-intuitive, and rarely gets any attention from the design team.

The result of poor software is workarounds.

— Dimitri Dadiomov, Bad Software and Expensive Mistakes

Having waited 7 years for someone to do something about it, O’Hara took the matter into his own hands. Initially, as a JP Morgan-only initiative, then through a contracting company called iMatix.

Two years later, in order to increase adoption, JP Morgan approached all the right technology companies (Cisco, Red Hat, Microsoft) and the right banks (Bank of America, Goldman, Credit Suisse) and created a working group that included people from 23 different companies.

And quickly, things started to decline.

In 2008, the CEO of iMatix published What is wrong with AMQP (and how to fix it), highlighting the problems derived from too many cooks in the kitchen:

The specification seems to be out of control, getting more and more complex without a clear plan or architecture. AMQP products don’t interoperate except in some limited cases. Core protocol aspects are unstable, and more ambitious functionality not happening. The Working Group is getting tired, and many of the original developers have burnt out.

Today, most of my team at iMatix has written off AMQP as a “failure”.

— Pieter Hintjens, What is wrong with AMQP (and how to fix it)

By then, iMatix had “soft abandoned” the AMQP project, and had started working on a better alternative. One that turned all the assumptions embedded in AMQP upside down.

They called it ZeroMQ.

---

A Newsletter For The Engineers That Keep Money Moving

Designing payment systems for interviews is easy. Designing them for millions of transactions is not.

After nearly a decade building and maintaining large-scale money software, I’ve seen what works (and what doesn’t) about software that moves money around. In The Payments Engineer Playbook, I share one in-depth article every Wednesday with breakdowns of how money actually moves.

If you’re an engineer or founder who needs to understand money software in depth, join close to 2,000 subscribers from companies like Shopify, Modern Treasury, Coinbase or Flywire to learn how real payment engineers plan, scale, and build money software.

---

This series is called From Stock Markets to Ledgers. We’re looking at the technology that powers NASDAQ and NYSE in order to build the software that moves money around. Faster, more performant, more correct.

This week, we’re looking at ZeroMQ, a protocol for message communication that, like Raft, looks at the problem at hand from first principles. In this article, we’re going to look at:

• Why ZeroMQ is a library

• The tradeoff between throughput and latency

• And a big takeaway to build ledgers with better throughput

Enough intro, let’s dive in.