Caching In the Past Is a Rare Event

How domain knowledge turns caching into an easy problem

I didn’t plan on writing this one this week.

In fact, I have another article already written on fraud management that I’m sure you’re going to like. It’s 80 percent done, with a few tweaks here and there in order to be publishable. Something that would take me half an hour or so to pull out.

That was the plan last Sunday. On Monday, I’ve already decided to put it off till next week, because I had become obsessed with something else.

With what? With a nice little trick to improve the performance of ledgers.

But before that, I need to tell you what the problem is all about.

You see, back a couple of weeks ago, in A Tale of Two Ledgers, I made the distinction between ledgers that look more like data warehouses, where the system is simply collecting data from an operation that’s happened somewhere else, and ledgers that look more like bouncers, where the system is actively approving or rejecting operations as they’re being attempted.

These two modes are called, respectively, Recording and Authorizing.

A Tale of Two Ledgers

Unlike the Authorizing mode, where journal entries are always added in the present time, the Recording mode is meant to allow other systems to add entries that have already happened, and must expose in its API a timestamp that allows users to specify when that journal entry should appear in the history of the ledger.

Modern Treasury, for example, calls that attribute effective_at.

effective_at allows clients to modify historical balances, so that they reflect the balances as they were in an external system. Using effective_at, we can support querying historical balances on accounts.

Typically, the effective_at timestamp sent by the client will be close to the created_at timestamp set by the ledger. The difference between effective_at and created_at is the delay between getting information from the external system into the ledger—usually on the order of seconds or minutes.

— Matt McNierney, from Modern Treasury’s How To Scale a Ledger Part IV

This, while absolutely needed, presents a huge performance problem.

If you’ve dealt with ledger systems, you’ve probably noticed how difficult it is to make them fast. That’s because the most common read operation (getting an Account’s balance) scales with the number of journal entries.

As time passes, and new entries are added into the ledger, it becomes slower and slower to calculate the Account’s balance.

I used to manage my finances using a paper notebook, and every now and then I calculated the current balance for all my accounts and jot that down, like a snapshot. That’s actually how most ledger systems operate: they have some sort of cache where they store the Account’s current balance, so that further calculations become easier without resorting to taking into account all the history of the journal.

That is, unless you can add entries retroactively.

I’m Alvaro Duran, and this is The Payments Engineer Playbook. We’re talking ledgers, and caching strategies, the sweetest spot where payments knowledge and technical insight collide.

Let me be clear: the Recording mode is great if you want to consolidate all of your financial data into a single “source of truth” that’s highly available and accurate. But the possibility of adding missing entries that are effective since a previous time invalidates the classic approach of keeping a sequence of snapshots that let you calculate Account balances faster than “check the numbers from the beginning of times”.

Modern Treasury’s How To Scale a Ledger series, again, already addresses this problem: