Assume that Manual Work in a Ledger Is Impossible
You can make adjustments to your books, but you can't have inconsistent reads.
A ledger that needs humans to work properly can only have as much impact as your headcount allows.
But most ledgers out there never really eliminate the need for human intervention. They hum along most of the time, but closing the books becomes a very human intensive activity, dreaded by accountants in every organization on Earth.
Call me an idealist, but I expected something else when I heard that software would eat the world. I was hoping for a world where we didn’t have to make manual adjustments to data.
A world where manual work isn’t just avoided, but genuinely impossible.
I’m Alvaro Duran, and this is The Payments Engineer Playbook. You’re already subscribed to free newsletters that “teach” you how to get a job as a software engineer.
But you don’t want to get a job; you already have one. What you want is to learn how to be great at your job. Especially as a payments engineer, where stakes are sky-high, and the margin for error is razor-thin.
In The Payments Engineer Playbook, we investigate the technology that transfers money. All to help you become a smarter, more skillful, and more successful payments engineer. And we do that by cutting off one sliver of it and extracting tactics from it.
This article is the first in a two-part series on ledger versioning. Engineers are often confused by the fact that there’s a versioning for reads and versioning for writes in a ledger system; it’s time we clarify this once and for all.
This week, we’re looking at the read side of ledger versioning.
Why a ledger needs to version its reads
How to do it
And when you might get away with not doing it
Enough intro, let’s dive in.
How to get your CFO in jail
The article I quote the most, by a long margin, is the one that differentitates between Recording and Authorizing ledgers.
The idea is simple: some ledgers are about consolidation, and others are about gatekeeping. There are ledgers designed to stitch together all of a company’s financial data, a sort of materialized view of its finances; and there are ledgers designed to approve or reject incoming transaction requests.
The former are Recording, and the latter are Authorizing.
Versioning the reads has to do with the Recording mode of a ledger, and that’s because recording ledgers are designed to admit backdated entries. After all, the point of data consolidation is that the data has already been produced elsewhere: a Recording ledger is built under the assumption that the data received is correct, but unaccounted for in the reports that will be produced later.
This creates a problem downstream: when you can add backdated entries, entries that are effective at a time prior to the one they were inserted into the ledger, whoever checked some account’s balance at some point in time will get a different result afterwards.
In other words: your company gets audited. Some external people check your financial statements, where it says that your company had a loss last month. But they now go check into your system, and it now says that you didn’t have a loss, that you actually made a tiny amount of profit. And it’s all legal, by the way: the reason for this mismatch is that you had some extra revenue recognized on the very last day of the month, and was cleared into the ledger the next day.
Without a way to differentiate, at scale, which entries were added after the report was produced, you’re in deep financial trouble. And, depending on the scale of the difference, your CFO may be facing jail.
Which is why you need to version your reads: only then, the entries that were added into the ledger after a particular read was made to produce the report can be identified at scale, and the issue becomes a footnote, and not a subpoena.
Balances of Future Pasts
Recording ledgers are eventually consistent to the sources of truth of their data, which is what allows for high throughput reads.
So in order to move money in a way that’s internally consistent, you need to make sure that all reads produce the same result, regardless of any backdated changes.
It is not enough to provide the timestamp from which to query all previous effective entries. Clients also need to provide the version of the account they’re querying, so that entries added in the future that are also effective won’t change the result of that query.
Accounts, therefore, have a version field, an integer that gets incremented everytime there’s a new Entry associated with it. But the entries themselves also have a version, which match the version that the Account had the moment they got added.
With these, Account balance calculations are uniquely identified by the effective date, and the version of its entries.
Make The Past Reproducible Again
The only reason not to implement Account versioning is that you’re not scared by inconsistent reads.
There are two ways this can happen: corrections are rare, or having inaccurate reports aren’t that big of a deal at the level at which you’re building. Which isn’t an engineer decision, but a product one.
To put it another way: there is no such thing as The Perfect Ledger for Everyone.
TigerBeetle isn’t a failure. On the contrary, it’s the signal that ledger technology isn’t done yet. There’s still so much to do. And that we haven’t nailed down the ultimate financial database yet.
Perhaps we never will.
The moment people depend on your ledger to explain what the company knew at a particular point in time, versioned reads stop being optional.
Because a trustworthy ledger shouldn’t merely tell you what is true now.
It should also be able to prove what was true then.
This was it for this week in The Payments Engineer Playbook. I’ll see you next week with the second part of this series.
P.S. I help fintech companies diagnose ledger problems before they become irreversible. If you feel that a second pair of eyes could be useful, send an email to alvaro@alvaroduran.com, and I’ll be in touch.