Enterprise Python
Software That Lives Long and Prosper 🖖
Editor’s note: This email is best enjoyed online (lots of images and email limits don’t go well).
Every year, I produce a big presentation exploring the impact of technology in business. For 2024, “Enterprise Python: Software That Lives Long and Prosper”.
In this week’s post, I share a loose transcript of this presentation.
If you’re new here, welcome! This is Money in Transit, the newsletter that dives deep into the technology to move money around. I’m Alvaro Duran.
I post weekly articles on the technical challenges of building at speed money software that is correct and performant.
Sounds impossible? Then better subscribe to find out how to do it.
Today, I’m talking about the kind of software that never gets the spotlight, and yet has gone through one of the most radical changes in the industry.
The history of enterprise software also has a deep connection to my own career, and I bet it has a connection to yours, too. Even if you don’t see it that way yet.
Let me explain.
5 years ago, I was working for a small enterprise software company in Barcelona. This company developed products for tax advisors, and made almost all of its revenue from a car registration product.
Covid-19 hit us dreadfully. It became an existential threat to the company. Nobody was driving, so nobody needed to register their car.
Thankfully, the team I was working on had developed a prototype for an electronic signature product, software that lets you sign documents online. Customers had been very slow to adopt it, because they didn’t see the need to do online what could be done face to face.
But when we were all at home, this product was the only thing they wanted from us!
It took us a week to go from a prototype to serving real users. And we did it, of course, in Python.
I bet that most of you had a similar experience during Covid. A neglected product suddenly becomes strategic for the company you were working for. And it has to go live, however crude, because customers needed it “yesterday”.
Did you have to go through that too?
Those hectic days were, for many companies, their first real exposure to a new way of building software. One that emphasizes speed over certainty, and iteration over perfection.
Covid-19 changed the world. It also changed how enterprises build software.
My name is Alvaro Duran. I’m the Tech Ambassador for Backend at Kiwi.com, the leading global travel tech company headquartered in the Czech Republic. And I admit that I engineer enterprise software. Specifically payments.
What happens right after you click Pay is my responsibility.
What do I even mean by enterprise software? I mean electronic signatures. I mean payments and travel reservations.
But I mean more than that.
There are three characteristics that enterprise software has in common.
First, enterprise software often has a lot of persistent data. It is what makes these systems valuable.
Second, this data is accessed and manipulated concurrently by many users. That’s how the data is being produced: through the actions of its users.
And third, this software never gets built in isolation, but rather it’s integrated with other services. Enterprise software is meant to be a piece of a greater puzzle, piping data from one service to another.
Data, concurrency and integrations. These are what give enterprise software its reputation.
Now ask yourself this: isn’t it like the software you build? With nerve-racking data migrations, elusive race conditions, and unnecessarily difficult APIs?
Admit to yourself: you build enterprise software.
Some people think that Python is not a language for “real” software engineering. Back in the 90s, Java established its supremacy with two powerful features: you could write a Java program once and run it everywhere, and you could count on Oracle to support the language.
Banks, for example, are the quintessential enterprise company. Many banks treat technology as a non-core competency, and for a long time they outsourced IT to specialized firms.
They were happy to run licensed software, because that established clear lines of blame, and allowed them to extend the life of their hardware.
Banks chose Java because Java is the “professionals” language.
But new banks are ditching Java for good. In the UK, Monzo is building its tech infrastructure in Go, and in Brazil, Nubank has embraced Clojure.
Go and Clojure aren’t bad choices to build enterprise software. In fact, at Kiwi.com we use Go to manage the transportation data we need to come up with the itineraries we sell to our customers.
But for everything else, we speak Python.
I learned how to build enterprise software by reading books written in Java, like Domain Driven Design, Working Effectively With Legacy Code, and Clean Code.
So imagine my surprise when, in 2018, a tech company called Dropbox IPOed at an $8 billion valuation, having built most of its software in Python.
Data, check.
Millions of concurrent users, check.
Integrations with annoying operating systems’ APIs? Check.
Dropbox was the first enterprise company I know that was built in Python. It confused the hell out of me.
What was very interesting to me was that Dropbox had been initially backed by a VC firm you probably know: Y Combinator, the VC started by Paul Graham.
PG had written an essay in 2004 called The Python Paradox, in which he said that you could get smarter programmers to work on a Python project than you could to work on a Java project, because in the end, languages created by people who cared about programming will win.
Some people have dismissed this essay because it feels like PG is arguing for his particular taste. And I agree with that sentiment: choosing a language because it’s trendy is an unsound business decision.
A trend is always a trap. Out-trending the competition is exhausting and fragile.
In fact, if trends were all that mattered, then nowadays Rust would be prevalent, given that it’s been the most admired language for years.
But I think that Paul Graham meant something else.
My thesis is this: just like Covid has shaken the enterprises’ attitude towards working remotely, other global trends have shaken the way enterprises build software.
What are these trends? I’ve identified at least four.
First, this is a world where computers are faster than ever. AWS now offers instances with dozens of Terabytes of memory, close to 900 virtual CPUs and 200 Gbps of network bandwidth.
Hardware that powerful makes efficient software not as relevant as it used to be.
We can trade some of that efficiency for something more valuable. That’s why the second trend is the move towards Service Oriented Architecture, pioneered in the enterprise by Amazon, and now effectively standard practice. SOA gives us high availability, safer changes and specialized teams. And in an architecture where most of the response time is due to network latency, performance is no longer a competitive advantage.
Third, scaling out hardware, rather than building an efficient machine, would be prohibitively expensive with licensed software. That’s why many enterprises have leaned into the engineers’ preference for open source software.
It works, it’s great, and it’s free.
And fourth, none of this mattered if the only enterprises out there were the ones that had already built what they needed in Java. However, today, companies that help people have better lives with software find themselves with billions of users as potential customers, competing against the Java powerhouses.
That’s because our ways of living have become dominated by computers. We live in a Tech Society.
Here’s a theme that runs across all these four trends: speed. In the Java days, big ate the slow.
Now, it is the fast that eats the slow.
Have you ever heard of the word disruption? Most people believe it’s just another word for “new”.
But it isn’t. Disruption is a term coined by Clayton Christensen, the author of The Innovator’s Dilemma. A disruptive technology is one that wins not by being superior in a static world, but by being already adapted to a world we’ve shifted into.
No disruptive technology gets taken seriously in the beginning, but they end up winning anyway.
This works best with an example. Where were you when the iPhone came out in 2007?
Steve Ballmer, then CEO of Microsoft, was in front of a journalist, who was asking him what he made of that new Apple device.
He laughed at it.
That is the most expensive phone in the world, and it doesn’t appeal to business customers because it doesn’t have a keyboard, which makes it not a very good email machine.
— Steve Ballmer
Ballmer has received a lot of backlash for this comment, but I think he was making a perfectly reasonable remark.
He was actually quoting his best customers:
Business customers need email machines.
Email machines must have keyboards.
Only business customers are willing to pay for expensive phones.
Expensive phones, therefore, must have keyboards.
We’ve all seen technology used by a billion users that no longer exists, like Nokia brick phones. Disruption implies that “quality” is a shifting term; it means different things at different points in time.
That’s what kills billion-users technology: the interplay between technological acceleration and the competitive business landscape.
New technology changes users’ needs and priorities. Gaps in the market get addressed by entrepreneurs who shuffle tech around and see what sticks.
Python is a case study for disruption. Dismissed as a hobbyist toy, it took over when enterprise software development shifted.
Python now dominates because its strengths have become indispensable, and its weaknesses have become irrelevant.
I want to talk about two features of the language that made the difference.
The first feature is duck typing. Now, some of you may think that not having types is the absence of a feature, but I disagree.
Adding a feature always comes at the expense of something else.
Performance and correctness are two reasons why people believe statically typed languages are superior. By being explicit about types, the compiler can make the program run faster, and enforce a set of guardrails on the code you can write, preventing certain bugs.
Performance and Correctness are the two benefits of statically typed code.
But what do types force you to give up?
Speed of development.
A programming language that forces you to use types is by definition one that slows you down.
Is what you get worth what you’re paying for? Are Performance and Correctness always more desirable than Speed of development?
No.
Even for enterprise software, what you need early on is Speed of development. That is the language’s most desirable feature. Performance and Correctness are amazing things to have, but only once we have validated that there is a business need that can be satisfied with that software.
So many times, we focus on correctness from the very beginning. We are even encouraged to, because managers understand that high quality requires time.
But as the deadline looms in, you cut corners. You start using undefined types like Any or Optional, eroding the benefits you sought early on. In the end, it looks nothing like what you envisioned.
Duck typing and Static typing delineate the two phases every enterprise software product must go through.
Building the right thing, and building the thing right.
What happens when you remove the need for types in a programming language? The program becomes slower. You may introduce errors. But you’re faster, and therefore you’re less afraid to throw it away, and start over.
Start better. Start from a better place.
Many companies are building great products using statically typed languages. But there’s an opportunity to emulate Picasso and build many prototypes, looking for the simplest, most elegant, most valuable option.
The opportunity to spend less time on what could go wrong, and spend more time on what did; surfing the next wave playfully, rather than overthinking your stance and balance.
That opportunity is not available to companies building enterprise software with statically typed languages.
But it is in Python.
These are two valid ways of writing the same function in Python. Duck typed on the left, statically typed on the right.
With tools like mypy, you can add types to your software product, incrementally.
It’s no coincidence that this tool was created when its inventor was working at Dropbox.
The second feature of Python I want to talk about is not really a feature.
It’s you.
One of the consequences of our Tech Society is that now everyone understands the value of software engineers. But for a long time, hiring was difficult; there weren’t enough people who were into the arcane world of programming.
When software engineering was done in Java, even the juniors were accomplished programmers.
As you know, Python was invented as a successor to ABC, a programming language intended to make beginners more comfortable with code. It is undeniably easy to get started with Python.
If “hello world” examples are some kind of proof of that, Python’s is the simplest:
print(“Hello, world”)
As a result, Python is by far the most popular language to teach to vernacular developers.
You might know them for another name: “non-professional” programmers. But that is misleading—they are professionals, just at something else. They create software in their own domain, like biology, physics or finance.
For vernacular developers, software is a means to an end, not an end in itself.
Before learning Python, vernacular developers do what they can with tools like R, Fortran, or MATLAB, or even spreadsheets. But once their needs are more complex or require a scale beyond what these tools can provide, they always turn to Python.
Here’s a fun fact: vernacular developers outnumber software engineers by at least 100 to 1, according to a study.
There are A LOT of them.
Why is that relevant? Because future enterprise products will most likely come from teams of vernacular developers.
Let me put it like this. Let’s say you’re an executive for a large organization. You’re meeting a couple of scientists that have developed an AI prototype. In Python, of course. These people aren’t engineers. Their expertise is precisely in the problem they’ve solved with that prototype.
In your opinion, which of these two options has more chances of success?
Another team takes that prototype, and “translates” it into something written in a “serious” programming language.
This team of vernacular developers takes the prototype live, iterates aggressively with their customers’ feedback, and hires some engineers to help them productize it.
I haven’t seen any research on this, but I would bet my money on option 2.
Vernacular developers have the domain knowledge to build the enterprise software of the future, and they no longer need a “serious” programming language to do it.
They are the most innovative minds available. With Python, the tools to build are no longer an obstacle.
When I worked at EdgeTier, that’s exactly what happened. Its founders developed an AI tool to detect anomalies in the messages sent to client support teams.
Once they validated the need, EdgeTier founders didn’t rewrite the prototype in Java; they productized it, all in Python.
Duck typing and the rise of the vernacular developer are two ways in which Python has challenged a long-held assumption. That only “serious” languages can be used to build enterprise software.
The only question that remains to be answered is: how?
What are the standards for building enterprise software in Python?
The answer is that there really aren’t. Not yet.
There are no standards on how to build Enterprise software using Python.
What do exist are the standards for building in Java retrofitted into Python. That’s the phase that disruptive technology always goes through in the early days. It’s called skeuomorphism.
It happened, for example, when the TV was invented. The first news programs on television were just someone repeating on camera what they had just broadcasted on radio.
They were radio programs with images.
The same is happening now in enterprise software. Java still exerts a powerful influence in the enterprise world, and many of the patterns used by Java have been ported, successfully, into Python.
The book Architecture Patterns with Python is exactly that. A great stepping stone for building enterprise software in Python.
But only a stepping stone.
Like radio programs on TV, Java enterprise patterns in Python can work. But we can do so much more than that.
What does “more” look like? I’m not sure.
As my generation, who is more proficient in Python, starts reaching Staff plus roles in big organizations, we’ll see the emergence of standards for building enterprise software specific to our language.
Early this year, I wanted to figure it out myself. So, I built an open source library to build payments, the kind of enterprise software I know well.
It’s called acquiring.
A few relevant design ideas that I used to build this library are:
Framework and Database agnosticism: This library can work with any framework you use (Django, Flask, FastAPI) and any database engine (The Django ORM, SQLAlchemy, PonyORM, etc).
That’s helpful for enterprises who want to decommission certain technology they’re using (like Oracle Database or SQL Server) but don’t want to rewrite the whole service from scratch.Structural Duck Typing with Protocols: Incremental typing is good; separating type hints from their implementation is better.
Immutable Dataclasses: Immutability helps engineers follow the flow of data more easily, because certain operations on the data encapsulated in classes is prohibited (or at least heavily discouraged).
Event-driven communication: Rather than updating the state of classes, I think that keeping an immutable log of events is much more effective. The auditability benefits are worth sacrificing a bit of read load and more storage size.
If these ideas are appealing to you, hell if you liked this talk, it would mean the world to me if you gave the project a star on GitHub.
Enjoy the rest of your day.