Since the dawn of time (before software, there was only darkness), there has been one constant: businesses want to build software cheaper and faster.

It is certainly an understandable and laudable goal – especially if you’ve spent any time around software developers. It is a goal that every engineer should support wholeheartedly, and we should always strive to create things as efficiently as possible, given the constraints of our situation.

However, the truth is we often don’t. It’s not intentional, but over time, we get waylaid by unforeseen complexities in building software and train ourselves to seek out edge cases, analysis gaps, all of the hidden repercussions that can result from a single bullet point of requirements.

We get enthralled by the maelstrom of complexity and the mental puzzle of engineering elegant solutions: Another layer of abstraction! DRY it up! Separate the concerns! Composition over inheritance! This too is understandable, but in the process, we often lose sight of the business problems being solved and forget that managing complexity is the second most important responsibility of software developers.

So how did we get here?

Software has become easier…in certain ways.

Over the last few decades, our industry has been very successful at reducing the amount of custom code it takes to write most software.

Much of this reduction has been accomplished by making programming languages more expressive. Languages such as Python, Ruby, or JavaScript can take as little as one third as much code as C in order to implement similar functionality. C gave us similar advantages over writing in assembler. Looking forward to the future, it is unlikely that language design will give us the same kinds of improvements we have seen over the last few decades.

But reducing the amount of code it takes to build software involves many other avenues that don’t require making languages more expressive. By far the biggest gain we have made in this over the last two decades is open source software (OSS). Without individuals and companies pouring money into software that they give freely to the community, much of what we build today wouldn’t be possible without an order of magnitude more cost and effort.

These projects have allowed us to tackle problems by standing on the shoulders of giants, leveraging tools to allow us to focus more of our energy on actually solving business problems, rather than spending time building infrastructure.

That said, businesses are complex. Ridiculously complex and only getting moreso. OSS is great for producing frameworks and tools that we can use to build systems on top of, but for the most part, OSS has to tackle problems shared by a large number of people in order to gain traction. Because of that, most open source projects have to either be relatively generic or be in a very popular niche. Therefore, most of these tools are great platforms on which to build out systems, but at the end of the day, we are still left to build all of the business logic and interfaces in our increasingly complex and demanding systems.

So what we are left with is a stack that looks something like this (for a web application)…

<Our Code>
<Libraries>
<Web Framework>
<Web Server>
<Data Stores>
<Operating System>

That “Our Code” part ends up being enormously complex, since it mirrors the business and its processes. If we have custom business logic, and custom processes, then we are left to build the interfaces, workflow, and logic that make up our applications. Sure, we can try to find different ways of recording that logic (remember business rules engines?), but at the end of the day, no one else is going to write the business logic for your business. There really doesn’t seem to be a way around that… at least not until the robots come and save us all from having to do any work.

Don’t like code, well how about Low-Code?

So if we have to develop the interfaces, workflow, and logic that make up our applications, then it sounds like we are stuck, right? To a certain extent, yes, but we have a few options.

To most developers, software equals code, but that isn’t reality. There are many ways to build software, and one of those ways is through using visual tools. Before the web, visual development and RAD tools had a much bigger place in the market. Tools like PowerBuilder, Visual Foxpro, Delphi, VB, and Access all had visual design capabilities that allowed developers to create interfaces without typing out any code.

These tools spanned the spectrum in terms of the amount of code you needed to write, but in general, you designed your app visually and then ended up writing a ton of code to implement the logic of your app. In many cases you still ended up programmatically manipulating the interface, since interfaces built using these tools often ended up being very static. However, for a huge class of applications, these tools allowed enormous productivity gains over the alternatives, mostly at the cost of flexibility.

The prevalence of these tools might have waned since the web took over, but companies’ desire for them has not, especially since the inexorable march of software demand continues. The latest trend that is blowing across the industry is “low code” systems. Low code development tools are a modern term put on the latest generation of drag and drop software development tools. The biggest difference between these tools and their brethren from years past is that they are now mostly web (and mobile) based and are often hosted platforms in the cloud.

And many companies are jumping all over these platforms. Vendors like Salesforce (App Cloud), Outsystems, Mendix, or Kony are promising the ability to create applications many times faster than “traditional” application development. While many of their claims are probably hyperbole, there likely is a bit of truth to them as well. For all of the downsides of depending on platforms like these, they probably do result in certain types of applications being built faster than traditional enterprise projects using .NET or Java.

So, what is the problem?

Well, a few things. First is that experienced developers often hate these tools. Most Serious Developers™ like to write Real Software™ with Real Code™. I know that might sound like I’m pandering to a bunch of whiney babies (and maybe I am a bit), but if the core value you deliver is technology, it is rarely a good idea to adopt tools that your best developers don’t want to work with.

Second is that folks like me look at these walled platforms and say “nope, not building my application in there.” That is a legitimate concern and the one that bothers me the most.

If you built an application a decade ago with PHP, then that application might be showing its age, but it could still be humming along right now just fine. The language and ecosystem are open source, and maintained by the community. You’ll need to keep your application up to date, but you won’t have to worry about a vendor deciding it isn’t worth their time to support you anymore.

…folks like me look at these walled platforms and say “nope, not building my application in there.” That is a legitimate concern and the one that bothers me the most.

If you picked a vendor 10 years ago who had a locked down platform, then you might be forced into a rewrite if they shut down or change their tooling too much (remember Parse?). Or even worse, your system gets stuck on a platforms that freezes and no longer serves your needs.

There are many reasons to be wary of these types of platforms, but for many businesses, the allure of creating software with less effort is just too much to pass up. The complexity of software continues on, and software engineers unfortunately aren’t doing ourselves any favors here.

What needs to change?

There are productive platforms out there, that allow us to build Real Software™ with Real Code™, but unfortunately our industry right now is far too worried with following the lead of the big tech giants to realize that sometimes their tools don’t add a lot of value to our projects.

I can’t tell you the number of times I’ve had a developer tell me that building something as a single page application (SPA) adds no overhead versus just rendering HTML. I’ve heard developers say that every application should be written on top of a NoSQL datastore, and that relational databases are dead. I’ve heard developers question why every application isn’t written using CQRS and Event Sourcing.

It is that kind of thought process and default overhead that is leading companies to conclude that software development is just too expensive. You might say, “But event sourcing is so elegant! Having a SPA on top of microservices is so clean!” Sure, it can be, but not when you’re the person writing all ten microservices. It is that kind of additional complexity that is often so unnecessary.

We, as an industry, need to find ways to simplify the process of building software, without ignoring the legitimate complexities of businesses. We need to admit that not every application out there needs the same level of interface sophistication and operational scalability as Gmail. There is a whole world of apps out there that need well thought-out interfaces, complicated logic, solid architectures, smooth workflows, etc…. but don’t need microservices or AI or chatbots or NoSQL or Redux or Kafka or Containers or whatever the tool dujour is.

A lot of developers right now seem to be so obsessed with the technical wizardry of it all that they can’t step back and ask themselves if any of this is really needed.

It is like the person on MasterChef who comes in and sells themselves as the molecular gastronomist. They separate ingredients into their constituent parts, use scientific methods of pairing flavors, and then apply copious amounts of CO2 and liquid nitrogen to produce the most creative foods you’ve ever seen. And then they get kicked off after an episode or two because they forget the core tenet of most cooking, that food needs to taste good. They seem genuinely surprised that no one liked their fermented fennel and mango-essence pearls served over cod with anchovy foam.

Our obsession with flexibility, composability, and cleverness is causing us a lot of pain and pushing companies away from the platforms and tools that we love. I’m not saying those tools I listed above don’t add value somewhere; they arose in response to real pain points, albeit typically problems encountered by large companies operating systems at enormous scale.

What I’m saying is that we need to head back in the direction of simplicity and start actually creating things in a simpler way, instead of just constantly talking about simplicity. Maybe we can lean on more integrated tech stacks to provide out of the box patterns and tools to allow software developers to create software more efficiently.

…we are going to push more and more businesses into the arms of “low code” platforms and other tools that promise to reduce the cost of software by dumbing it down and removing the parts that brought us to it in the first place.

We need to stop pretending that our 20th line-of-business application is some unique tapestry that needs to be carefully hand-sewn.

Staying Focused on Simplicity

After writing that, I can already hear a million developers sharpening their pitchforks, but I believe that if we keep pushing in the direction of wanting to write everything, configure everything, compose everything, use the same stack for every scale of problem, then we are going to push more and more businesses into the arms of “low code” platforms and other tools that promise to reduce the cost of software by dumbing it down and removing the parts that brought us to it in the first place.

Our answer to the growing complexity of doing business cannot be adding complexity to the development process – no matter how elegant it may seem.

We must find ways to manage complexity by simplifying the development process. Because even though managing complexity is our second most important responsibility, we must always remember the most important responsibility of software developers: delivering value through working software.

51 Comments

Joshua Wedekind

I agree with 99.87% of what you just wrote. But unless all of your developers are using the exact same OS and libraries locally that are installed on your production server, then you really ought to be using containers. Because who wants to debug library differences on production?

By the way, if anyone is interested in trying out an incredibly productive web framework, I recommend checking out Django. Need an API to go with it? Look at the Django Rest Framework.

Reply
L J Laubenheimer

Not everything belongs in a container. Sure, if the only deployment framework you know is containers, then everything looks like a stateless web application. (If the only tool you have is a hammer, then everything looks like a nail.)

Reply
Engin

I agree with the complexity/simplicity aspect of the post but I cannot say the same thing for OOS/Vendor platforms comparison. In the past 17 years of my experience, Microsoft has always provided me great support where the framework has reached its limits but I cannot say the same thing with PHP. I feel a lack of responsibility and documentation with every open source project I felt excited to use, it’s almost like why do you need the documentation when you can see the code itself.

Reply
Nitesh

I think people pick on a few open source projects that they don’t like and then paint all OSS with the same brush, which is wrong. There are many OSS projects supported by large organizations that are successful, and then smaller ones that aren’t, which are also successful. If you’re using .NET, then that’s open source, supported by a large organization. Then you’re also probably using a Newtonsoft Json library, which is open source, not supported by a large organization.

Angular, React, Vue and several other OSS javascript frameworks have fantastic documentation and support from the community.

Reply
Tim Ritzer

Ever have an app built on Silverlight? I did. And paid heavily for MS killing it after only a few years.

No company is a saint in this regard.

Reply
Kris Tuttle

I think improvement is close at hand. It stems from more software being callable and able to integrate with your custom business logic via API. So even though I have to write *some* custom business logic it’s small because I can focus on the really unique parts. Because I can access systems via API there is plenty that I don’t have to build.

If API access keeps proliferating and “moving up the stack” it will enable much purer focus on coding what’s “special” about infrastructure and leveraging other software to do everything else.

Reply
Tim

I have been a big advocate for this type of “return to simplicity” movement for a long time. I don’t use the most fancy tech at work and manage to still do some pretty fancy things but in a straight forward and simplistic way. I think part of the problem is also the lack of creativity in problem solving. Someone rather download and setup a react component for a simple widget that probably could have been done with straight JavaScript and a few creative ideas… and probably at 80% the size.

Reply
Seb V

“Someone rather download and setup a react component for a simple widget that probably could have been done with straight JavaScript and a few creative ideas…”

A good chunk of my energy is spent agonizing on whether writing custom code or going third-party with a distinct possibility of bloat.

I worked in the past with people suffering of the Not Invented Here syndrom at a terminal degree, and I watched in despair as awesome open source frameworks emerged while we were fumbling about, clumsily reinventing every wheels.

On the other hand, I bang my head on my desk everyday because of the insanity that has become of the web frontend stack.

In short, nothing is absolute. The best compromise between simplicity and efficiency is the Holy Grail of any software project, and I’m still wandering around Britannia.

Reply
jc

OSS and low barrier to entry technologies like javascript has produced alot of good free software, but has had a terrible effect on the overall complexity front. Its too easy to roll your own solution for every little thing, and publish to NPM, so everyone does.

I happen to think JS is probably the most important technology in the world, but the framework explosion is a sick joke that taints the whole community. It happened, I believe, because the browser vendors never sufficiently grappled with the inherent problems of porting what was born as hypertext markup into a full blown programming environment.

Web components is perhaps the best hope. Maybe React since its got the popularity, but would have to be made native in browser and made a real standard. Don’t care for the html/js mixing personally so I wont use React, but the fact this hasn’t been solved 20 years in, is evidence of a real problem.

Reply
Teemu

100% with you, simplicity rocks, complexity kills. Lets get back to the basics and keep our eye on the ball!

Reply
Dmitriy

You are calling for simplicity but you didn’t tell how.
But technologies move in the opposite direction.
Look at modern c++, Java, C#, at all those new constructs

Reply
Derek Hunter

Me and my programming team are still using C libraries we wrote 30 years ago. Those libraries have survived multiple operating systems and hardware platforms. Simplicity comes from reducing dependencies. The majority of our code has one set of dependencies looking down to the operating system and the hardware, and another set looking up to the user interface. These principles apply whether we are deploying to an Arduino or a virtual cloud instance.

I had to smile at one of the early comments above. No disrespect Joshua, but where do you think the “Django Rest Framework” will be in 30 years? Probably with me – singing in programming heaven 🙂

Reply
David Mcbride

The thing that causes the complexity is the fact that business wants something for nothing, and devalues the software development principles. So often, complexity is added because business wants to impose the solutions platform thinking it is making better choices to speed up development when it does not possess the acumen to make that decision, and so often the business refuses to follow proven life cycle and allow software to be developed in a way which is software oriented, not car cleaning oriented. Business is adding the complexity and then moaning about everything we do to try and working with what they want and how they want it. Get back to basics, use a properly joined up cross functional team and put software developers at one of the heads of this as was done with IT many years ago and watch the industry change. Carry on as we are, and in 10 years we will still be where we are.

Reply
L J Laubenheimer

Bingo.

Often the business owners get sold a cargo-cult solution tech stack by some “guru” and then mandate that *everything* be ported over and run through that stack, and that stack only. Then it’s d*mn the consequences and skilled hours that it takes to port perfectly functional application over to a screwy rube-goldberg tech stack, supposedly in the name of “efficiency”, when the efficiency in truth goes to hell because the application was never designed to work in that environment.

Reply
Doug Hancock

I’ve been writing software since 1969. I remember numerous occasions when the scribes and the Parisees of IT said “one day we won’t need programmers”. We have complicated software development to the point that nobody knows what is going on, especially when it doesn’t work. We ‘hide’ errors and faults from the user. And just when you get used to some technology or development tool, the fashion changes and business insists that you change to the latest, whether it is efficient or not.
I can honestly tell you, I have had enough! Good luck fellows you going to need it. I am becoming a carpenter, at least wood and a carpenter’s tools don’t change much.

Reply
temp

Lets talk from a developers perspective. A developer is more proficient in one particular tool rather than all the tools. So lets say a developer has experience in writing code using NoSQL. Then he can produce better code( scalable /efficient/abstraction). Now lets say he gets a next project which is simpler from db perspective and relational database can serve his purpose. THen how good code can he write considering that he do not have the same level of expertise in it.
The bottom line is with so much of variety available(programming language, database, tools, scripting languge), it is impossible for a single developer to expertise in all of them. Instead he can get expertise in a db/programmingLanguage/OS/tools. If there is developer who only has expertise in writing mico-services and containers, it would be difficult for him to write code using alternatives. And what about the pressure of learning something new that comes in the market. Lets face the fact: Our code is complex because we require complex logic. Unless the logic is simplified, there is no solution. And the logic cannot be simplified because we create more features every-day, new use cases every-day. THe only thing can be done is to identify the language/tool/db required for a particular application and develop expertise in it rather than be jack of all trade.

Reply
Brian Joseph Johns

Certainly the endless possibilities can overwhelm the imaginative and creative developer and coder who wants to code the solution while guaranteeing the code’s reusability for future projects. All of this while compromising the goals of the solution upon which they currently work by making the work horse functionality too generic.

While working with libraries, components and frameworks, we are building a solution simultaneously from the top down and the bottom up, hoping that the gui requirements meet the library’s, component’s or framework’s API interfaces in the same or similar aspects. Is the behavior of the end product shaped by the libraries that we use for a solution or is it the other way around? The difficulty and quantity of work and code is directly related to this in most modern applications.

Consider how many times we’ve had to change our approach to a given solution as a result of the way that the operating system and related APIs force us to code a certain way in order to use these services. I guess that is overcome by experience on the same platform and with said APIs.

Perhaps the challenge is related to keeping ourselves focused on not what we *could* do but what we *need* to do. Great article by the way.

Reply
Topher Hunt

My general rule is to minimize the vocabulary of different concepts that developers need to understand and work with, in order to maintain / work on a project. Every new language & technology introduced, brings in a slew of new concepts you need to be familiar with. And this extends to code style too: are your React components composed of dumb 101-level JS functions, or are they strewn with second-order functions that return functions that you then curry together in order to avoid a simple `.map()` anonymous function call? I’ve seen devs go wild in Ruby with inheritance and mixins and metaprogramming when a few dumb class methods would have served the same need. This isn’t a matter of different devs coming from different paradigms and having different perspectives on what code is “simple”; the latter is clearly, unambiguously more accessible, more readable, and quicker for other devs on the team to understand. (And often briefer to boot.)

My favorite quote of the year so far:

“We should use technologies that are as simple as possible, so that as many people as possible can use and extend them without needing to understand our advanced techniques. We should use advanced techniques only when we are not smart enough to figure out how to use more common techniques.”

https://matthewrocklin.com/blog//work/2018/01/27/write-dumb-code

Reply
Toby Farley

I feel like a fraud when I go to conferences because I write software for a relatively small organization so all my applications and their databases are hosted locally. It all works the way it’s suppose to but since I am not “in the cloud”, my development methodology is suspect. There is no real value for me to go NoSQL or use containers. There is a chauvinism among developers if you are not doing it “their way”.

Reply
Bear

There is a fundamental principle — perhaps a corollary to Occam’s Razor — that says “once a system’s complexity reaches a certain point, a new simpler system with key improvements will win out…” Basically, evolution happens.

– Was Linux better than UNIX or Windows? No, but Linux was simpler, extensible and free. So it got lots of attention. Now it is in everything. Currently, I’m working with uCLinux, which cd

– Why does C consistently stay in the top 3 programming languages, even after 40 years? It is simple, but complex things can be built from it. It has resisted efforts to improve it over the years by adding “features”. C++ on the other hand, not so much.

– Once something useful is created — let’s call it A — it is natural to add features to it to it to make it more useful. New features implies more complexity. If this is not managed well, A becomes overly complicated, and usually overly specialized. Eventually something new comes along — let’s call it B — with most of the useful features of A but with a redone-from-scratch implementation and interface. If you need A but could use B, you’d probably choose B for a variety of reasons — simpler implies less time and money is needed to make do something useful. This is the basic nature of things.

Reply
Steve Naidamast

As a senior software engineer who retired in 2014 after 42+ years in the corporate environments both as an employee and consultant, I cannot disagree with anything contended in this essay.

However, there are very serious factors in the profession that have substantially increased complexity in software development while at the same time placing more responsibility on individual developers and engineers.

On the one hand we have had the terrible, mass outsourcing of the later 1990s through even today, though some of this has been mitigated by circumstances. However, what this trend did was to break down the vital functional areas of development; one of the most critical being that of the system analyst who would work with business analysts to develop specifications for developers and who would be responsible for understanding the tasks at hand both from a business and technical perspective.

These areas were lost with the outsourcing trend providing the second factor, that of the subsequent movement to take on additional development responsibilities with the advent of Agile and now DevOps to meet ever tighter deadlines with fewer and fewer resources; both paradigms which will prove to be failures in the long term.

The third factor has been the rampant introduction of new tools for developers that have made development not only more complex but needlessly so. This has been highlighted by the move from ASP.NET WebForms to ASP.NET MVC in the Microsoft Community and the introduction of mobile technologies initiated by Apple Corporation starting in 2007; the latter which is complete crap in terms of serious development. With over 85% of all such mobile applications being devoted to pornography and entertainment no one can seriously suggest that such platforms are viable for serious development. And the sociological harm such platforms have caused is becoming not only frightening but immeasurable.

In reality, business requirements have never changed in the interim since all businesses run against standardized paradigms of profit, loss, expense, and investment, none of which will ever change no matter how anyone tries to place fancy veneer over them. Instead, the idea of changing business requirements has been massively hyped by vendors in order to promote new tools and paradigms for their own bottom lines.

Finally, the fourth major factor has been the active support of such trends by a good portion of the development communities; especially the younger generations, who have no long term experiences to compare current development environments to, simply so that they can work with the latest technologies. There is a lot of truth to the saying that “Wisdom is wasted on the old…” Such technologies hardly provide anything new just new ways of doing the same things computers have been doing since their commercial introduction in the 1960s.

The result of all this is that both business and development have combined together to create their own “perfect storm”, which is incessantly fueled by the hype of ever more increasing complexity in the future. What utter nonsense!

What has happened as a result is that societies are beginning to deteriorate increasingly from the overwhelming nature of such complexities while most of them find their way to the lowest common denominators in societies, the criminals, the business psychopaths, and the massive numbers of incompetent technical managers that run the show.

Really intelligent developers and software engineers should run from such evolutionary crap as it has proven that it is no longer about the creation of quality applications that are easy and simple to use but instead about what tools can be implemented to support the various competing agendas between developer, technical manager, and business person.

This article and others like it are beginning to surface in the profession all imply that many are trying to find a way to survive such a convoluted profession that offers very little in the way of true satisfaction any longer, if it ever did…

Reply
Paul Gehrman

I agree completely. I’ve been writing software for over twenty years and it has changed significantly over that time. Many more hipsters joining the programming ranks. They have to maintain an aura of coolness by bringing in the latest tools and trends, despite the underlying lack of utility of many of those trends. The other problems I see are letting inexperienced people architect solutions, how programming is taught in colleges (too much dogmatic OO), and authors trying to make a buck writing books that advocate over-architected solutions to problems.

Reply
Eric Peterson

<— The problem starts here

The reason the stack is complex is we are pretending that Our Code can be simplified by the use of web frameworks. But that is incorrect. Web frameworks add enormous complexity and vulnerabilities and attempt to hide it. The hiding only works for toy problems. In the long run the code we end up writing is unmaintainable glue. Even worse, some of our “code” is XML or other structure and relationship description languages. Those should be code, not data.

A much better approach for the long run:

|

Note the size of our code in the top line, it’s not an accident. Even though our code will be more “complex” in this architecture, when properly organized and by using libraries judiciously it can be robust, stable, relatively bug-free and easy to maintain. I would point out that tomcat is just one possible choice and has some framework characteristics, but not many.

Often the argument for using a web framework is “don’t reinvent the wheel”. On the contrary the wheel must be reinvented, in every project, to make progress over time. A simple example is user sessions. Tomcat provides a session cookie which is adequately safe when using HTTPS (which is not optional these days). But obviously a cookie is not enough to store everything we need to know about a user. So we create a users database table with a session column and other columns for things we need to know about our users.

But, some would argue, that’s what you use Redis for, because Redis scales. We need to scale that much? Really? And you use Redis with a Spring client, because… Or you should not write your own SQL, you should use hibernate. Because… writing SQL is hard? No, writing SQL is easy because it is extremely easy to test, and everyone should know how to do it. So you use Redis because it scales and you use another database with database wrapper because Redis won’t store the complex relational data you need and try to link them together. What a mess.

The answer is very simple. Write your own (simple) SQL wrapper code. Write your own java classes for use by GSON when you need to exchange data among tiers. Do not write XML of any sort or anything like it to define structures, configure code, etc. Write code instead. You are stuck with a small amount of XML (e.g. tomcat’s web.xml) and some env files but that’s a necessary evil and not hard to maintain.

Reply
Brian Young

I agree 100% KISS principle is what I try to live by. Keep it simple stupid. One thing that developers often forget is that no one, not one person/user, gives a crap what language, framework, stack and blah blah blah that you chose to write the application in. Does it work? Does it simplify or enhance a business process? The business doesn’t care as long as it solves the problem. So simple is the way to go for me and the business continues to love the application even if I wrote it in classic ASP. Ok that last part was a joke! 🙂

Reply
Basant Kumar

8 years back, I happened to work on a work-flow tool (which is internally built in Java). The tool had drag and drop components even for a “for loop”. So if the business requirement is known, just use the components to build the logic, deploy it and you are done.

Apparently, it sounds so simple but believe me it was complex, for the reason that for a simple “for loop”, a developer with Java background can simply write for(int x = 0; x < 10; x++), which is more simpler than using the component.

Being Simple is always good than being complex, after all the maintenance is much easier with less cost for simpler implementation.

Reply
Ben Lopatin

From my own perch, working on customers’ existing web apps, there are two things which I see lead to overly complex solutions:

1. Separation between developer and business problem/customer
2. A separation between responsibility for greenfield development and long term maintenance

The first isn’t so much about distance between the developer and the customer’s needs – what agile is intended to solve – but empathy with the root problem. The problem of getting overly engaged with the “mental puzzle of engineering elegant solutions” doesn’t go away but is at least ameliorated when developers share similar incentives as the customer, or at least empathize with *their* problem, instead of the developer’s own.

The second is more strictly related to incentives. Most of the fancy new frameworks – even some of the more mature ones – advertise their strength in *starting* projects. It’s easier to get things built, it’s easier to move quickly – and this is often at odds with what’s needed to sensibly and sanely maintain software well after launch. I’ve noticed more unjustified complexity in projects that were handed off by a dev team for initial build than in those built and maintained by the same team.

I’m not going to say these are *the* reasons for greater software complexity, but they’re two tractable reasons.

Reply
Justin Etheredge

Ben, good to hear from you! Hope you’re doing well. I’m planning on writing a follow-up post that has more concrete examples of why engineers do what they do, and I had intended to include some of it in this post but I already had a small novel going on. Your point about alignment of incentives is a big one for me, and something that I think about often. Thank you for the great feedback!

Reply
Ken Ambrose

[[why engineers do what they do]]
sorry, programmers are not engineers. only wishful (and self serving) thinking. engineering involves the application of mathematics and scientific methods and techniques to predict results. no one can do that in the area of programming.
we are not even close to engineering software- maybe in another 200 years?

Reply
Ken Ambrose

and if I may I would add that simplicity of design and implementation is actually the primary intellectual challenge and effort of programming.

Scientists know this. The more complex a theory and solution the greater the skepticism among peers that it is a optimal answer- and rightly so. And conversely the more elegant and simple a solution, the more provably correct.

When I worked at SLAC as a programmer, and explained to my managers that it took three times as long to design and develop a simple and elegant solution to a programming problem than for a quick and dirty design, they understood immediately. Because that is their world.

Try explaining that concept to an accounting manager. She/he will look at you like you have two heads.

Reply
Steve Franzkowiak

Nicely stated and I totally agree! Been developing applications for 30 plus years using the latest and greatest tool sets, languages, and methodologies. My experience has been when we’ve focused on simplify an application’s complexity we ended up writing better code that was understandable, resilient, less costly to support, and provided greater value to our client. Some development principles are always true regardless of the technology!

Reply
Bob G.

Writing code that is intelligible, amenable to change and that accurately models the problem domain is an inherently complex activity.

Albert Einstein said that things should be as simple as possible but not simpler.

Reply
Ted Warring

Excellent essay, and it is encouraging that more software engineers and thought leaders are starting to understand the cost of complexity.

We have been trapped in a hype cycle for patterns and architectures for many years now. Each new approach is lauded as the answer to the ills and evils of the last, with simplistic examples given at conferences that show how whiz-bang it is, and all the cool devs of course will look down on you if you stick to the yucky old ways of doing things.

Every new approach falls short of promise when scaled and is followed by added complexity in compensation for the lost benefits that went along with the costs of the previous approach. I have come to believe that the majority of us are unable to realistically evaluate the true ROI of frameworks, patterns, and architectures. But once they get mention in the magic quadrant by Gartner all the CIOs and consultants will be demanding its use, so it is not just engineers at fault.

The rate of failure and/or cost overrun in large software projects is not sustainable, but everyone is afraid to point out that the emperor has no clothes when it comes to the latest “correct” way to do things.

My thoughts on the subject are much aligned with yours: http://onverus.com/index.php/just-enough-complexity/

Reply
Travis

I’ve been having an increasingly difficult time making this clear to my junior developers. I’ve seen a lot of ideas come and go in my 20 years. You know what wins every time? Easy to understand code and concepts. Not the “hey let’s configure every aspect of this module”. Get data, process it, put it somewhere. Do not over complicate from the start. Complicate when you cannot work inside the simple framework. Then realize that your complicated solution isn’t even solving the problem.

This has been 90% of my career’s experience. Solve the problems you have today. Ignore all the other bullshit until it’s actually a problem worth solving.

Reply
Bill Melendez

Wow. Lots of comments. I’m new in current programming languages. Use to do Basic –I mean real old basic. I have seen a big problem with programmers –they do love their codes and making a simple one liner into a complex subprogram. So I comb the internet and garner one liners and simple code and am making my programs that way. I see a lot of request for help from beginners. And it is hard understanding the jargon and complex codes that are offered as solutions to these folks when the one liners would do. I wonder why we can’t just call red “red” and blue “blue”?

Reply
Alistair Gill

As a non-SW person, my customers have made a lot of money with machines controlled using my software. It is important to recognise that tools are important. Having started life with a slide-rule and Log-Tables, through calculators to advanced algorithms running in FPGAs and, yes, Excel macros, it would be clearly absurd to suggest that modern methods should not be employed. It’s a similar story in the world of PCB layouts and mechanical CAD. The important unit of measure for me is, does the new tool simplify what I am doing so that I can stand on the back of giants?

Reply
Matt Thompsett

Having worked with OutSystems Low Code for the best part of two years now, I can assure there is no hyperbole associated with the OutSystems Platform. It does precisely what it says ‘on the tin’, robust enterprise applications built for mobile (any platform) and web 10-15 times faster than hand coding. The platform’s capabilities are simply extraordinary, especially as DevOps is ‘managed’ within the platform, enabling deployment with a single click. Whatever you might think right now, there is a wave of change sweeping the tech world. The biggest misconception of low code is that it sits firmly at the ‘citizen developer’ end of the spectrum, wrong! The power of technology is shifting into the hands of users, there will be a place for specialist coding but that’s it. You’re right in that tech should be simple, not simple in its form but simple in its application…

Reply
Gates VP

> First is that experienced developers often hate these tools. Most Serious Developers™ like to write Real Software™ with Real Code™

Sure, but most businesses are also doing it this way in order to cut costs. And that typically includes pay. So if you’re a developer who wants to get paid well, then that whole “Real Software” and “Read Code” spiel actually makes a lot sense.

These tools can provide real business value at a significantly lower cost. But a big chunk of that “lower cost” is the fact that developers are getting paid less to work with “not Real Code”. The pushback is very real, because we’re talking about developer livelihoods here.

I have worked on and delivered systems in Access ’98. I respect businesses that run on Excel. I’ve done consulting for small businesses and government. At the end of the day, those businesses are not going to pay me top dollar to implement their systems. Google will do that.

So can you blame people for wanting to learn the tools that Google is using?

Reply
Ali Tahbaz

You’re right on target, at least as far as business software consulting is concerned (and not pure pushing the envelope research). Businesses that pay our salaries in one way or another benefit from ROI on the time they pay for. The quickest, longest lasting, lowest TCO solution is what helps them succeed in their industry. Those of us who help that happen repeatedly become trusted business collaborators and can rest assured we’ve delivered maximum value. Of course not everything can be simple and there is no one size fits all. I’ve found providing multiple options on different stacks really helps the best choice present itself for each situation. Thanks for sharing where your head’s at.

Reply
MikeyB

This is so true. I recently wrote two very sophisticated projects – one synchronized multithreaded, one distributed fault-tolerant WAN app. The key is to keep it simple or it will not be maintained and survive. MS may not always be the best tools, but they will always be there. Make sure all NuGet and external dependencies are well documented.

Reply
Nelson Chamberlain

Well said and right on target. And I believe the reason why code complexity continues to be an issue is because programming languages lack “simplifying abstractions”. For example, cars use three simplifying abstractions: keys for user authentication, pedal on the floor to stop and start the car, and a steering wheel to point the car. Users who master these three simplifications can (mostly) safely commute from Point A to Point B at insanely high speeds inches away from other vehicles traveling at high speeds.
Unfortunately, writing code has no equivalent simplifying abstractions that will remove complexity from the developer’s domain/responsibility/control. Sure, methods and functions and API’s try to reduce complexity but all they really accomplish is broaden the scope of what we need to know without deepening our knowledge of its functionality; they defer our exposure to the complexity of the code.
Instead of using a programming language to write descriptive narratives of what the computer should do, useful simplifying abstractions could be based on training a dog (or other animal of your choice) or exploring a maze or making an engineering drawing or building a structure using the equivalent of digital Legos(TM). These simplifying abstractions would require two basic characteristics: (1) specifying program behavior by modifying properties instead of changing code, and (2) ability to “get under the hood” and make modifications to handle the edge-cases, for example (using the car analogy), special situations like driving at high altitude or extended driving at high speeds or extended periods of idling.
The point about simplifying abstractions is that we humans should do what we do best (define/visualize what the app should do) and computers should do what they do best (take care of the details with absolute precision and infinite patience). Without adequate simplifying abstractions, we are stuck dealing with expanding complexity with increasingly inadequate tools.

Reply
Tim Dietrich

Thanks for writing this post and sharing your thoughts on the increasing complexity of software development. Based on all of the comments that you’ve received, it seems that a lot of us share your observations and concerns.

You wrote that “our industry right now is far too worried with following the lead of the big tech giants to realize that sometimes their tools don’t add a lot of value to our projects.” There’s a growing trend to over-engineer solutions and prematurely address their scalability, and it seems to me that this is a significant contributor to the increased complexity that you described.

You also wrote, “Maybe we can lean on more integrated tech stacks to provide out of the box patterns and tools to allow software developers to create software more efficiently.” That perfectly describes my situation. I think one of the keys to success and longevity in doing software development is finding development tools that are simple, dependable, flexible, and perhaps most importantly, that you enjoy using. My current stack consists of Aurora (Amazon’s version of MySQL and PostgreSQL) and a little-known programming environment called Xojo (which I consider to be my “secret weapon”). It couldn’t be any more simple than that. I’m finding that with it I get more done and in far less time.

I also found Steve Naidamast’s comment about the additional responsibilities that developers are taking on to be interesting. He wrote that the “movement to take on additional development responsibilities with the advent of Agile and now DevOps to meet ever tighter deadlines with fewer and fewer resources; both paradigms which will prove to be failures in the long term.” I see this movement as an attempt to merge the responsibilities that have traditionally been assigned to distinct development and operations teams. I think it’s too early to tell if this is a successful IT strategy, but as Steve said, I suspect that in many cases it won’t be.

Thanks again for the post. I’m a developer based in Richmond, and I hope our paths cross at some point.

Reply
Randy Kelly

Great article Justin. The 20+ years that I have been in this industry, I’ve heard and seen so many religious arguments for one technology versus the other (anyone remember the introduction of Java?). While I do have my own ideas about what works best, ultimately the technology is situational to a certain extent. It just depends on what you are trying to achieve and the customer’s needs is ALWAYS the most important. Simplification is really the goal as I think most people would agree. Controlling costs should always be at the top of your list when building systems. As well as, balancing that happy medium with the customer of cost versus complexity. While they may want software for complex processes, this often opens a dialogue for process improvement or modification in an effort to reduce software development costs. Once a plan is put in place, then frameworks can be discussed to address the requirements. It’s important to note that in the consideration of frameworks, who will have to maintain it in the future? You or the customer? Again, yet another point to come to an understanding about with your customers because you may need to pick technologies that play to the strength of the their own teams.

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *