Software Complexity Is Killing Us

Software Complexity Is Killing Us

Since the dawn of time (before software, there was only darkness), there has been one constant: businesses want to build software cheaper and faster.

It is certainly an understandable and laudable goal – especially if you’ve spent any time around software developers. It is a goal that every engineer should support wholeheartedly, and we should always strive to create things as efficiently as possible, given the constraints of our situation.

However, the truth is we often don’t. It’s not intentional, but over time, we get waylaid by unforeseen complexities in building software and train ourselves to seek out edge cases, analysis gaps, all of the hidden repercussions that can result from a single bullet point of requirements.

We get enthralled by the maelstrom of complexity and the mental puzzle of engineering elegant solutions: Another layer of abstraction! DRY it up! Separate the concerns! Composition over inheritance! This too is understandable, but in the process, we often lose sight of the business problems being solved and forget that managing complexity is the second most important responsibility of software developers.

So how did we get here?

Software has become easier…in certain ways.

Over the last few decades, our industry has been very successful at reducing the amount of custom code it takes to write most software.

Much of this reduction has been accomplished by making programming languages more expressive. Languages such as Python, Ruby, or JavaScript can take as little as one third as much code as C in order to implement similar functionality. C gave us similar advantages over writing in assembler. Looking forward to the future, it is unlikely that language design will give us the same kinds of improvements we have seen over the last few decades.

But reducing the amount of code it takes to build software involves many other avenues that don’t require making languages more expressive. By far the biggest gain we have made in this over the last two decades is open source software (OSS). Without individuals and companies pouring money into software that they give freely to the community, much of what we build today wouldn’t be possible without an order of magnitude more cost and effort.

These projects have allowed us to tackle problems by standing on the shoulders of giants, leveraging tools to allow us to focus more of our energy on actually solving business problems, rather than spending time building infrastructure.

That said, businesses are complex. Ridiculously complex and only getting moreso. OSS is great for producing frameworks and tools that we can use to build systems on top of, but for the most part, OSS has to tackle problems shared by a large number of people in order to gain traction. Because of that, most open source projects have to either be relatively generic or be in a very popular niche. Therefore, most of these tools are great platforms on which to build out systems, but at the end of the day, we are still left to build all of the business logic and interfaces in our increasingly complex and demanding systems.

So what we are left with is a stack that looks something like this (for a web application)…

<Our Code>
<Web Framework>
<Web Server>
<Data Stores>
<Operating System>

That “Our Code” part ends up being enormously complex, since it mirrors the business and its processes. If we have custom business logic, and custom processes, then we are left to build the interfaces, workflow, and logic that make up our applications. Sure, we can try to find different ways of recording that logic (remember business rules engines?), but at the end of the day, no one else is going to write the business logic for your business. There really doesn’t seem to be a way around that… at least not until the robots come and save us all from having to do any work.

Don’t like code, well how about Low-Code?

So if we have to develop the interfaces, workflow, and logic that make up our applications, then it sounds like we are stuck, right? To a certain extent, yes, but we have a few options.

To most developers, software equals code, but that isn’t reality. There are many ways to build software, and one of those ways is through using visual tools. Before the web, visual development and RAD tools had a much bigger place in the market. Tools like PowerBuilder, Visual Foxpro, Delphi, VB, and Access all had visual design capabilities that allowed developers to create interfaces without typing out any code.

These tools spanned the spectrum in terms of the amount of code you needed to write, but in general, you designed your app visually and then ended up writing a ton of code to implement the logic of your app. In many cases you still ended up programmatically manipulating the interface, since interfaces built using these tools often ended up being very static. However, for a huge class of applications, these tools allowed enormous productivity gains over the alternatives, mostly at the cost of flexibility.

The prevalence of these tools might have waned since the web took over, but companies’ desire for them has not, especially since the inexorable march of software demand continues. The latest trend that is blowing across the industry is “low code” systems. Low code development tools are a modern term put on the latest generation of drag and drop software development tools. The biggest difference between these tools and their brethren from years past is that they are now mostly web (and mobile) based and are often hosted platforms in the cloud.

And many companies are jumping all over these platforms. Vendors like Salesforce (App Cloud), Outsystems, Mendix, or Kony are promising the ability to create applications many times faster than “traditional” application development. While many of their claims are probably hyperbole, there likely is a bit of truth to them as well. For all of the downsides of depending on platforms like these, they probably do result in certain types of applications being built faster than traditional enterprise projects using .NET or Java.

So, what is the problem?

Well, a few things. First is that experienced developers often hate these tools. Most Serious Developers™ like to write Real Software™ with Real Code™. I know that might sound like I’m pandering to a bunch of whiney babies (and maybe I am a bit), but if the core value you deliver is technology, it is rarely a good idea to adopt tools that your best developers don’t want to work with.

Second is that folks like me look at these walled platforms and say “nope, not building my application in there.” That is a legitimate concern and the one that bothers me the most.

If you built an application a decade ago with PHP, then that application might be showing its age, but it could still be humming along right now just fine. The language and ecosystem are open source, and maintained by the community. You’ll need to keep your application up to date, but you won’t have to worry about a vendor deciding it isn’t worth their time to support you anymore.

…folks like me look at these walled platforms and say “nope, not building my application in there.” That is a legitimate concern and the one that bothers me the most.

If you picked a vendor 10 years ago who had a locked down platform, then you might be forced into a rewrite if they shut down or change their tooling too much (remember Parse?). Or even worse, your system gets stuck on a platforms that freezes and no longer serves your needs.

There are many reasons to be wary of these types of platforms, but for many businesses, the allure of creating software with less effort is just too much to pass up. The complexity of software continues on, and software engineers unfortunately aren’t doing ourselves any favors here.

What needs to change?

There are productive platforms out there, that allow us to build Real Software™ with Real Code™, but unfortunately our industry right now is far too worried with following the lead of the big tech giants to realize that sometimes their tools don’t add a lot of value to our projects.

I can’t tell you the number of times I’ve had a developer tell me that building something as a single page application (SPA) adds no overhead versus just rendering HTML. I’ve heard developers say that every application should be written on top of a NoSQL datastore, and that relational databases are dead. I’ve heard developers question why every application isn’t written using CQRS and Event Sourcing.

It is that kind of thought process and default overhead that is leading companies to conclude that software development is just too expensive. You might say, “But event sourcing is so elegant! Having a SPA on top of microservices is so clean!” Sure, it can be, but not when you’re the person writing all ten microservices. It is that kind of additional complexity that is often so unnecessary.

We, as an industry, need to find ways to simplify the process of building software, without ignoring the legitimate complexities of businesses. We need to admit that not every application out there needs the same level of interface sophistication and operational scalability as Gmail. There is a whole world of apps out there that need well thought-out interfaces, complicated logic, solid architectures, smooth workflows, etc…. but don’t need microservices or AI or chatbots or NoSQL or Redux or Kafka or Containers or whatever the tool dujour is.

A lot of developers right now seem to be so obsessed with the technical wizardry of it all that they can’t step back and ask themselves if any of this is really needed.

It is like the person on MasterChef who comes in and sells themselves as the molecular gastronomist. They separate ingredients into their constituent parts, use scientific methods of pairing flavors, and then apply copious amounts of CO2 and liquid nitrogen to produce the most creative foods you’ve ever seen. And then they get kicked off after an episode or two because they forget the core tenet of most cooking, that food needs to taste good. They seem genuinely surprised that no one liked their fermented fennel and mango-essence pearls served over cod with anchovy foam.

Our obsession with flexibility, composability, and cleverness is causing us a lot of pain and pushing companies away from the platforms and tools that we love. I’m not saying those tools I listed above don’t add value somewhere; they arose in response to real pain points, albeit typically problems encountered by large companies operating systems at enormous scale.

What I’m saying is that we need to head back in the direction of simplicity and start actually creating things in a simpler way, instead of just constantly talking about simplicity. Maybe we can lean on more integrated tech stacks to provide out of the box patterns and tools to allow software developers to create software more efficiently.

…we are going to push more and more businesses into the arms of “low code” platforms and other tools that promise to reduce the cost of software by dumbing it down and removing the parts that brought us to it in the first place.

We need to stop pretending that our 20th line-of-business application is some unique tapestry that needs to be carefully hand-sewn.

Staying Focused on Simplicity

After writing that, I can already hear a million developers sharpening their pitchforks, but I believe that if we keep pushing in the direction of wanting to write everything, configure everything, compose everything, use the same stack for every scale of problem, then we are going to push more and more businesses into the arms of “low code” platforms and other tools that promise to reduce the cost of software by dumbing it down and removing the parts that brought us to it in the first place.

Our answer to the growing complexity of doing business cannot be adding complexity to the development process – no matter how elegant it may seem.

We must find ways to manage complexity by simplifying the development process. Because even though managing complexity is our second most important responsibility, we must always remember the most important responsibility of software developers: delivering value through working software.

More Insights

View All