Since the dawn of time (before software, there was only darkness), there has been one constant: businesses want to build software cheaper and faster.
It is certainly an understandable and laudable goal – especially if you’ve spent any time around software developers. It is a goal that every engineer should support wholeheartedly, and we should always strive to create things as efficiently as possible, given the constraints of our situation.
However, the truth is we often don’t. It’s not intentional, but over time, we get waylaid by unforeseen complexities in building software and train ourselves to seek out edge cases, analysis gaps, all of the hidden repercussions that can result from a single bullet point of requirements.
We get enthralled by the maelstrom of complexity and the mental puzzle of engineering elegant solutions: Another layer of abstraction! DRY it up! Separate the concerns! Composition over inheritance! This too is understandable, but in the process, we often lose sight of the business problems being solved and forget that managing complexity is the second most important responsibility of software developers.
So how did we get here?
Software has become easier…in certain ways.
Over the last few decades, our industry has been very successful at reducing the amount of custom code it takes to write most software.
But reducing the amount of code it takes to build software involves many other avenues that don’t require making languages more expressive. By far the biggest gain we have made in this over the last two decades is open source software (OSS). Without individuals and companies pouring money into software that they give freely to the community, much of what we build today wouldn’t be possible without an order of magnitude more cost and effort.
These projects have allowed us to tackle problems by standing on the shoulders of giants, leveraging tools to allow us to focus more of our energy on actually solving business problems, rather than spending time building infrastructure.
That said, businesses are complex. Ridiculously complex and only getting moreso. OSS is great for producing frameworks and tools that we can use to build systems on top of, but for the most part, OSS has to tackle problems shared by a large number of people in order to gain traction. Because of that, most open source projects have to either be relatively generic or be in a very popular niche. Therefore, most of these tools are great platforms on which to build out systems, but at the end of the day, we are still left to build all of the business logic and interfaces in our increasingly complex and demanding systems.
So what we are left with is a stack that looks something like this (for a web application)…
That “Our Code” part ends up being enormously complex, since it mirrors the business and its processes. If we have custom business logic, and custom processes, then we are left to build the interfaces, workflow, and logic that make up our applications. Sure, we can try to find different ways of recording that logic (remember business rules engines?), but at the end of the day, no one else is going to write the business logic for your business. There really doesn’t seem to be a way around that… at least not until the robots come and save us all from having to do any work.
Don’t like code, well how about Low-Code?
So if we have to develop the interfaces, workflow, and logic that make up our applications, then it sounds like we are stuck, right? To a certain extent, yes, but we have a few options.
To most developers, software equals code, but that isn’t reality. There are many ways to build software, and one of those ways is through using visual tools. Before the web, visual development and RAD tools had a much bigger place in the market. Tools like PowerBuilder, Visual Foxpro, Delphi, VB, and Access all had visual design capabilities that allowed developers to create interfaces without typing out any code.
These tools spanned the spectrum in terms of the amount of code you needed to write, but in general, you designed your app visually and then ended up writing a ton of code to implement the logic of your app. In many cases you still ended up programmatically manipulating the interface, since interfaces built using these tools often ended up being very static. However, for a huge class of applications, these tools allowed enormous productivity gains over the alternatives, mostly at the cost of flexibility.
The prevalence of these tools might have waned since the web took over, but companies’ desire for them has not, especially since the inexorable march of software demand continues. The latest trend that is blowing across the industry is “low code” systems. Low code development tools are a modern term put on the latest generation of drag and drop software development tools. The biggest difference between these tools and their brethren from years past is that they are now mostly web (and mobile) based and are often hosted platforms in the cloud.
And many companies are jumping all over these platforms. Vendors like Salesforce (App Cloud), Outsystems, Mendix, or Kony are promising the ability to create applications many times faster than “traditional” application development. While many of their claims are probably hyperbole, there likely is a bit of truth to them as well. For all of the downsides of depending on platforms like these, they probably do result in certain types of applications being built faster than traditional enterprise projects using .NET or Java.
So, what is the problem?
Well, a few things. First is that experienced developers often hate these tools. Most Serious Developers™ like to write Real Software™ with Real Code™. I know that might sound like I’m pandering to a bunch of whiney babies (and maybe I am a bit), but if the core value you deliver is technology, it is rarely a good idea to adopt tools that your best developers don’t want to work with.
Second is that folks like me look at these walled platforms and say “nope, not building my application in there.” That is a legitimate concern and the one that bothers me the most.
If you built an application a decade ago with PHP, then that application might be showing its age, but it could still be humming along right now just fine. The language and ecosystem are open source, and maintained by the community. You’ll need to keep your application up to date, but you won’t have to worry about a vendor deciding it isn’t worth their time to support you anymore.
…folks like me look at these walled platforms and say “nope, not building my application in there.” That is a legitimate concern and the one that bothers me the most.
If you picked a vendor 10 years ago who had a locked down platform, then you might be forced into a rewrite if they shut down or change their tooling too much (remember Parse?). Or even worse, your system gets stuck on a platforms that freezes and no longer serves your needs.
There are many reasons to be wary of these types of platforms, but for many businesses, the allure of creating software with less effort is just too much to pass up. The complexity of software continues on, and software engineers unfortunately aren’t doing ourselves any favors here.
What needs to change?
There are productive platforms out there, that allow us to build Real Software™ with Real Code™, but unfortunately our industry right now is far too worried with following the lead of the big tech giants to realize that sometimes their tools don’t add a lot of value to our projects.
I can’t tell you the number of times I’ve had a developer tell me that building something as a single page application (SPA) adds no overhead versus just rendering HTML. I’ve heard developers say that every application should be written on top of a NoSQL datastore, and that relational databases are dead. I’ve heard developers question why every application isn’t written using CQRS and Event Sourcing.
It is that kind of thought process and default overhead that is leading companies to conclude that software development is just too expensive. You might say, “But event sourcing is so elegant! Having a SPA on top of microservices is so clean!” Sure, it can be, but not when you’re the person writing all ten microservices. It is that kind of additional complexity that is often so unnecessary.
We, as an industry, need to find ways to simplify the process of building software, without ignoring the legitimate complexities of businesses. We need to admit that not every application out there needs the same level of interface sophistication and operational scalability as Gmail. There is a whole world of apps out there that need well thought-out interfaces, complicated logic, solid architectures, smooth workflows, etc…. but don’t need microservices or AI or chatbots or NoSQL or Redux or Kafka or Containers or whatever the tool dujour is.
A lot of developers right now seem to be so obsessed with the technical wizardry of it all that they can’t step back and ask themselves if any of this is really needed.
It is like the person on MasterChef who comes in and sells themselves as the molecular gastronomist. They separate ingredients into their constituent parts, use scientific methods of pairing flavors, and then apply copious amounts of CO2 and liquid nitrogen to produce the most creative foods you’ve ever seen. And then they get kicked off after an episode or two because they forget the core tenet of most cooking, that food needs to taste good. They seem genuinely surprised that no one liked their fermented fennel and mango-essence pearls served over cod with anchovy foam.
Our obsession with flexibility, composability, and cleverness is causing us a lot of pain and pushing companies away from the platforms and tools that we love. I’m not saying those tools I listed above don’t add value somewhere; they arose in response to real pain points, albeit typically problems encountered by large companies operating systems at enormous scale.
What I’m saying is that we need to head back in the direction of simplicity and start actually creating things in a simpler way, instead of just constantly talking about simplicity. Maybe we can lean on more integrated tech stacks to provide out of the box patterns and tools to allow software developers to create software more efficiently.
…we are going to push more and more businesses into the arms of “low code” platforms and other tools that promise to reduce the cost of software by dumbing it down and removing the parts that brought us to it in the first place.
We need to stop pretending that our 20th line-of-business application is some unique tapestry that needs to be carefully hand-sewn.
Staying Focused on Simplicity
After writing that, I can already hear a million developers sharpening their pitchforks, but I believe that if we keep pushing in the direction of wanting to write everything, configure everything, compose everything, use the same stack for every scale of problem, then we are going to push more and more businesses into the arms of “low code” platforms and other tools that promise to reduce the cost of software by dumbing it down and removing the parts that brought us to it in the first place.
Our answer to the growing complexity of doing business cannot be adding complexity to the development process – no matter how elegant it may seem.
We must find ways to manage complexity by simplifying the development process. Because even though managing complexity is our second most important responsibility, we must always remember the most important responsibility of software developers: delivering value through working software.
Loved the article? Hated it? Didn’t even read it?
We’d love to hear from you.
True that, could not agree more
Dido, Great Article
I agree with 99.87% of what you just wrote. But unless all of your developers are using the exact same OS and libraries locally that are installed on your production server, then you really ought to be using containers. Because who wants to debug library differences on production?
By the way, if anyone is interested in trying out an incredibly productive web framework, I recommend checking out Django. Need an API to go with it? Look at the Django Rest Framework.
Not everything belongs in a container. Sure, if the only deployment framework you know is containers, then everything looks like a stateless web application. (If the only tool you have is a hammer, then everything looks like a nail.)
You are what he complained about.
Containers have “overhead” and don’t correspond to servers, physical or virtual.
Docker schmocker. I avoid it like the plague. My favorite was the lazy IT guy who tried to hand one of our EDL engineers a dockerized FPGA toolchain that normally wants it’s own 32gb ram to compile HDL etc…
I’ll wholeheartedly suggest Ansible or similar in lieu of Kubercrap as well.
Infrastructure as code? Absolutely. There are many ways of arriving at the
Django? Maybe a decade ago for a low-volume web app, but there’s no reason to use such a dog-slow platform in a world in which Go, Crystal, and other typed compiled “built for the web” languages power servers with less bottlenecks and no dynamic language runtime errors. I’d say that Node directly sort of competes here and wins out, popularity wise, if not performance wise. (I’m not a big fan of having to reboot servers nor memory leaks nor gargantuan memory requirements for running a single-threaded event-loop that spits out strings, but YMMV)
Well, even the original author of Node (Ryan Dahl) left the Node project in favor of Go. So I wouldn’t invest too much time in Node anymore… In fact, Ryan started a new project to fix the ‘design mistakes’ of Node, it’s called Deno.
> there’s no reason to use such a dog-slow platform in a world in which Go
Sounds like like you never read this article. Why do you think the fact a language is interpreted would be a deal breaker to deliver value? Who cares?
On the opposite, having full featured frameworks with batteries included like Django or Rails are an actual way to reduce development complexity, not reinvent the wheel, or not spending weeks to build your own custom framework in Node which still has no equivalent.
I’m not saying node is always bad and Django is always great, I’m just saying you don’t chose your tech stack in a an clever way.
I agree with the complexity/simplicity aspect of the post but I cannot say the same thing for OOS/Vendor platforms comparison. In the past 17 years of my experience, Microsoft has always provided me great support where the framework has reached its limits but I cannot say the same thing with PHP. I feel a lack of responsibility and documentation with every open source project I felt excited to use, it’s almost like why do you need the documentation when you can see the code itself.
I think people pick on a few open source projects that they don’t like and then paint all OSS with the same brush, which is wrong. There are many OSS projects supported by large organizations that are successful, and then smaller ones that aren’t, which are also successful. If you’re using .NET, then that’s open source, supported by a large organization. Then you’re also probably using a Newtonsoft Json library, which is open source, not supported by a large organization.
Ever have an app built on Silverlight? I did. And paid heavily for MS killing it after only a few years.
No company is a saint in this regard.
I believe well written code doesn’t need documentation. It explains itself.
That’s such a stale old phrase that needs to be dead and buried. Forever. Good code *requires* commenting, especially in business domains when it’s important to know not HOW the code works, but WHY, i.e., for what business purpose the code is doing this or that.
Seriously. Stop parroting that stupid phrase.
I think improvement is close at hand. It stems from more software being callable and able to integrate with your custom business logic via API. So even though I have to write *some* custom business logic it’s small because I can focus on the really unique parts. Because I can access systems via API there is plenty that I don’t have to build.
If API access keeps proliferating and “moving up the stack” it will enable much purer focus on coding what’s “special” about infrastructure and leveraging other software to do everything else.
A good chunk of my energy is spent agonizing on whether writing custom code or going third-party with a distinct possibility of bloat.
I worked in the past with people suffering of the Not Invented Here syndrom at a terminal degree, and I watched in despair as awesome open source frameworks emerged while we were fumbling about, clumsily reinventing every wheels.
On the other hand, I bang my head on my desk everyday because of the insanity that has become of the web frontend stack.
In short, nothing is absolute. The best compromise between simplicity and efficiency is the Holy Grail of any software project, and I’m still wandering around Britannia.
I happen to think JS is probably the most important technology in the world, but the framework explosion is a sick joke that taints the whole community. It happened, I believe, because the browser vendors never sufficiently grappled with the inherent problems of porting what was born as hypertext markup into a full blown programming environment.
Web components is perhaps the best hope. Maybe React since its got the popularity, but would have to be made native in browser and made a real standard. Don’t care for the html/js mixing personally so I wont use React, but the fact this hasn’t been solved 20 years in, is evidence of a real problem.
100% with you, simplicity rocks, complexity kills. Lets get back to the basics and keep our eye on the ball!
You are calling for simplicity but you didn’t tell how.
But technologies move in the opposite direction.
Look at modern c++, Java, C#, at all those new constructs
The issue is that we all love complexity, even when we say otherwise:
Me and my programming team are still using C libraries we wrote 30 years ago. Those libraries have survived multiple operating systems and hardware platforms. Simplicity comes from reducing dependencies. The majority of our code has one set of dependencies looking down to the operating system and the hardware, and another set looking up to the user interface. These principles apply whether we are deploying to an Arduino or a virtual cloud instance.
I had to smile at one of the early comments above. No disrespect Joshua, but where do you think the “Django Rest Framework” will be in 30 years? Probably with me – singing in programming heaven 🙂
The thing that causes the complexity is the fact that business wants something for nothing, and devalues the software development principles. So often, complexity is added because business wants to impose the solutions platform thinking it is making better choices to speed up development when it does not possess the acumen to make that decision, and so often the business refuses to follow proven life cycle and allow software to be developed in a way which is software oriented, not car cleaning oriented. Business is adding the complexity and then moaning about everything we do to try and working with what they want and how they want it. Get back to basics, use a properly joined up cross functional team and put software developers at one of the heads of this as was done with IT many years ago and watch the industry change. Carry on as we are, and in 10 years we will still be where we are.
Often the business owners get sold a cargo-cult solution tech stack by some “guru” and then mandate that *everything* be ported over and run through that stack, and that stack only. Then it’s d*mn the consequences and skilled hours that it takes to port perfectly functional application over to a screwy rube-goldberg tech stack, supposedly in the name of “efficiency”, when the efficiency in truth goes to hell because the application was never designed to work in that environment.
I’ve been writing software since 1969. I remember numerous occasions when the scribes and the Parisees of IT said “one day we won’t need programmers”. We have complicated software development to the point that nobody knows what is going on, especially when it doesn’t work. We ‘hide’ errors and faults from the user. And just when you get used to some technology or development tool, the fashion changes and business insists that you change to the latest, whether it is efficient or not.
I can honestly tell you, I have had enough! Good luck fellows you going to need it. I am becoming a carpenter, at least wood and a carpenter’s tools don’t change much.
Lets talk from a developers perspective. A developer is more proficient in one particular tool rather than all the tools. So lets say a developer has experience in writing code using NoSQL. Then he can produce better code( scalable /efficient/abstraction). Now lets say he gets a next project which is simpler from db perspective and relational database can serve his purpose. THen how good code can he write considering that he do not have the same level of expertise in it.
The bottom line is with so much of variety available(programming language, database, tools, scripting languge), it is impossible for a single developer to expertise in all of them. Instead he can get expertise in a db/programmingLanguage/OS/tools. If there is developer who only has expertise in writing mico-services and containers, it would be difficult for him to write code using alternatives. And what about the pressure of learning something new that comes in the market. Lets face the fact: Our code is complex because we require complex logic. Unless the logic is simplified, there is no solution. And the logic cannot be simplified because we create more features every-day, new use cases every-day. THe only thing can be done is to identify the language/tool/db required for a particular application and develop expertise in it rather than be jack of all trade.
You’re wrong. If you’ve got less than 10,000 node_modules folders then you’re not a good developer.
Certainly the endless possibilities can overwhelm the imaginative and creative developer and coder who wants to code the solution while guaranteeing the code’s reusability for future projects. All of this while compromising the goals of the solution upon which they currently work by making the work horse functionality too generic.
While working with libraries, components and frameworks, we are building a solution simultaneously from the top down and the bottom up, hoping that the gui requirements meet the library’s, component’s or framework’s API interfaces in the same or similar aspects. Is the behavior of the end product shaped by the libraries that we use for a solution or is it the other way around? The difficulty and quantity of work and code is directly related to this in most modern applications.
Consider how many times we’ve had to change our approach to a given solution as a result of the way that the operating system and related APIs force us to code a certain way in order to use these services. I guess that is overcome by experience on the same platform and with said APIs.
Perhaps the challenge is related to keeping ourselves focused on not what we *could* do but what we *need* to do. Great article by the way.
My general rule is to minimize the vocabulary of different concepts that developers need to understand and work with, in order to maintain / work on a project. Every new language & technology introduced, brings in a slew of new concepts you need to be familiar with. And this extends to code style too: are your React components composed of dumb 101-level JS functions, or are they strewn with second-order functions that return functions that you then curry together in order to avoid a simple `.map()` anonymous function call? I’ve seen devs go wild in Ruby with inheritance and mixins and metaprogramming when a few dumb class methods would have served the same need. This isn’t a matter of different devs coming from different paradigms and having different perspectives on what code is “simple”; the latter is clearly, unambiguously more accessible, more readable, and quicker for other devs on the team to understand. (And often briefer to boot.)
My favorite quote of the year so far:
“We should use technologies that are as simple as possible, so that as many people as possible can use and extend them without needing to understand our advanced techniques. We should use advanced techniques only when we are not smart enough to figure out how to use more common techniques.”
I feel like a fraud when I go to conferences because I write software for a relatively small organization so all my applications and their databases are hosted locally. It all works the way it’s suppose to but since I am not “in the cloud”, my development methodology is suspect. There is no real value for me to go NoSQL or use containers. There is a chauvinism among developers if you are not doing it “their way”.
There is a fundamental principle — perhaps a corollary to Occam’s Razor — that says “once a system’s complexity reaches a certain point, a new simpler system with key improvements will win out…” Basically, evolution happens.
– Was Linux better than UNIX or Windows? No, but Linux was simpler, extensible and free. So it got lots of attention. Now it is in everything. Currently, I’m working with uCLinux, which cd
– Why does C consistently stay in the top 3 programming languages, even after 40 years? It is simple, but complex things can be built from it. It has resisted efforts to improve it over the years by adding “features”. C++ on the other hand, not so much.
– Once something useful is created — let’s call it A — it is natural to add features to it to it to make it more useful. New features implies more complexity. If this is not managed well, A becomes overly complicated, and usually overly specialized. Eventually something new comes along — let’s call it B — with most of the useful features of A but with a redone-from-scratch implementation and interface. If you need A but could use B, you’d probably choose B for a variety of reasons — simpler implies less time and money is needed to make do something useful. This is the basic nature of things.
As a senior software engineer who retired in 2014 after 42+ years in the corporate environments both as an employee and consultant, I cannot disagree with anything contended in this essay.
However, there are very serious factors in the profession that have substantially increased complexity in software development while at the same time placing more responsibility on individual developers and engineers.
On the one hand we have had the terrible, mass outsourcing of the later 1990s through even today, though some of this has been mitigated by circumstances. However, what this trend did was to break down the vital functional areas of development; one of the most critical being that of the system analyst who would work with business analysts to develop specifications for developers and who would be responsible for understanding the tasks at hand both from a business and technical perspective.
These areas were lost with the outsourcing trend providing the second factor, that of the subsequent movement to take on additional development responsibilities with the advent of Agile and now DevOps to meet ever tighter deadlines with fewer and fewer resources; both paradigms which will prove to be failures in the long term.
The third factor has been the rampant introduction of new tools for developers that have made development not only more complex but needlessly so. This has been highlighted by the move from ASP.NET WebForms to ASP.NET MVC in the Microsoft Community and the introduction of mobile technologies initiated by Apple Corporation starting in 2007; the latter which is complete crap in terms of serious development. With over 85% of all such mobile applications being devoted to pornography and entertainment no one can seriously suggest that such platforms are viable for serious development. And the sociological harm such platforms have caused is becoming not only frightening but immeasurable.
In reality, business requirements have never changed in the interim since all businesses run against standardized paradigms of profit, loss, expense, and investment, none of which will ever change no matter how anyone tries to place fancy veneer over them. Instead, the idea of changing business requirements has been massively hyped by vendors in order to promote new tools and paradigms for their own bottom lines.
Finally, the fourth major factor has been the active support of such trends by a good portion of the development communities; especially the younger generations, who have no long term experiences to compare current development environments to, simply so that they can work with the latest technologies. There is a lot of truth to the saying that “Wisdom is wasted on the old…” Such technologies hardly provide anything new just new ways of doing the same things computers have been doing since their commercial introduction in the 1960s.
The result of all this is that both business and development have combined together to create their own “perfect storm”, which is incessantly fueled by the hype of ever more increasing complexity in the future. What utter nonsense!
What has happened as a result is that societies are beginning to deteriorate increasingly from the overwhelming nature of such complexities while most of them find their way to the lowest common denominators in societies, the criminals, the business psychopaths, and the massive numbers of incompetent technical managers that run the show.
Really intelligent developers and software engineers should run from such evolutionary crap as it has proven that it is no longer about the creation of quality applications that are easy and simple to use but instead about what tools can be implemented to support the various competing agendas between developer, technical manager, and business person.
This article and others like it are beginning to surface in the profession all imply that many are trying to find a way to survive such a convoluted profession that offers very little in the way of true satisfaction any longer, if it ever did…
“With over 85% of all such mobile applications being devoted to pornography and entertainment no one can seriously suggest that such platforms are viable for serious development.”
$60 billion is not serious?!?
IBM would beg to differ.
I agree completely. I’ve been writing software for over twenty years and it has changed significantly over that time. Many more hipsters joining the programming ranks. They have to maintain an aura of coolness by bringing in the latest tools and trends, despite the underlying lack of utility of many of those trends. The other problems I see are letting inexperienced people architect solutions, how programming is taught in colleges (too much dogmatic OO), and authors trying to make a buck writing books that advocate over-architected solutions to problems.
<— The problem starts here
The reason the stack is complex is we are pretending that Our Code can be simplified by the use of web frameworks. But that is incorrect. Web frameworks add enormous complexity and vulnerabilities and attempt to hide it. The hiding only works for toy problems. In the long run the code we end up writing is unmaintainable glue. Even worse, some of our “code” is XML or other structure and relationship description languages. Those should be code, not data.
A much better approach for the long run:
Note the size of our code in the top line, it’s not an accident. Even though our code will be more “complex” in this architecture, when properly organized and by using libraries judiciously it can be robust, stable, relatively bug-free and easy to maintain. I would point out that tomcat is just one possible choice and has some framework characteristics, but not many.
Often the argument for using a web framework is “don’t reinvent the wheel”. On the contrary the wheel must be reinvented, in every project, to make progress over time. A simple example is user sessions. Tomcat provides a session cookie which is adequately safe when using HTTPS (which is not optional these days). But obviously a cookie is not enough to store everything we need to know about a user. So we create a users database table with a session column and other columns for things we need to know about our users.
But, some would argue, that’s what you use Redis for, because Redis scales. We need to scale that much? Really? And you use Redis with a Spring client, because… Or you should not write your own SQL, you should use hibernate. Because… writing SQL is hard? No, writing SQL is easy because it is extremely easy to test, and everyone should know how to do it. So you use Redis because it scales and you use another database with database wrapper because Redis won’t store the complex relational data you need and try to link them together. What a mess.
The answer is very simple. Write your own (simple) SQL wrapper code. Write your own java classes for use by GSON when you need to exchange data among tiers. Do not write XML of any sort or anything like it to define structures, configure code, etc. Write code instead. You are stuck with a small amount of XML (e.g. tomcat’s web.xml) and some env files but that’s a necessary evil and not hard to maintain.
Oops my diagrams were interpreted as XML and stripped.
I agree 100% KISS principle is what I try to live by. Keep it simple stupid. One thing that developers often forget is that no one, not one person/user, gives a crap what language, framework, stack and blah blah blah that you chose to write the application in. Does it work? Does it simplify or enhance a business process? The business doesn’t care as long as it solves the problem. So simple is the way to go for me and the business continues to love the application even if I wrote it in classic ASP. Ok that last part was a joke! 🙂
8 years back, I happened to work on a work-flow tool (which is internally built in Java). The tool had drag and drop components even for a “for loop”. So if the business requirement is known, just use the components to build the logic, deploy it and you are done.
Apparently, it sounds so simple but believe me it was complex, for the reason that for a simple “for loop”, a developer with Java background can simply write for(int x = 0; x < 10; x++), which is more simpler than using the component.
Being Simple is always good than being complex, after all the maintenance is much easier with less cost for simpler implementation.
From my own perch, working on customers’ existing web apps, there are two things which I see lead to overly complex solutions:
1. Separation between developer and business problem/customer
2. A separation between responsibility for greenfield development and long term maintenance
The first isn’t so much about distance between the developer and the customer’s needs – what agile is intended to solve – but empathy with the root problem. The problem of getting overly engaged with the “mental puzzle of engineering elegant solutions” doesn’t go away but is at least ameliorated when developers share similar incentives as the customer, or at least empathize with *their* problem, instead of the developer’s own.
The second is more strictly related to incentives. Most of the fancy new frameworks – even some of the more mature ones – advertise their strength in *starting* projects. It’s easier to get things built, it’s easier to move quickly – and this is often at odds with what’s needed to sensibly and sanely maintain software well after launch. I’ve noticed more unjustified complexity in projects that were handed off by a dev team for initial build than in those built and maintained by the same team.
I’m not going to say these are *the* reasons for greater software complexity, but they’re two tractable reasons.
Ben, good to hear from you! Hope you’re doing well. I’m planning on writing a follow-up post that has more concrete examples of why engineers do what they do, and I had intended to include some of it in this post but I already had a small novel going on. Your point about alignment of incentives is a big one for me, and something that I think about often. Thank you for the great feedback!
[[why engineers do what they do]]
sorry, programmers are not engineers. only wishful (and self serving) thinking. engineering involves the application of mathematics and scientific methods and techniques to predict results. no one can do that in the area of programming.
we are not even close to engineering software- maybe in another 200 years?
and if I may I would add that simplicity of design and implementation is actually the primary intellectual challenge and effort of programming.
Scientists know this. The more complex a theory and solution the greater the skepticism among peers that it is a optimal answer- and rightly so. And conversely the more elegant and simple a solution, the more provably correct.
When I worked at SLAC as a programmer, and explained to my managers that it took three times as long to design and develop a simple and elegant solution to a programming problem than for a quick and dirty design, they understood immediately. Because that is their world.
Try explaining that concept to an accounting manager. She/he will look at you like you have two heads.
Nicely stated and I totally agree! Been developing applications for 30 plus years using the latest and greatest tool sets, languages, and methodologies. My experience has been when we’ve focused on simplify an application’s complexity we ended up writing better code that was understandable, resilient, less costly to support, and provided greater value to our client. Some development principles are always true regardless of the technology!
Writing code that is intelligible, amenable to change and that accurately models the problem domain is an inherently complex activity.
Albert Einstein said that things should be as simple as possible but not simpler.
Excellent essay, and it is encouraging that more software engineers and thought leaders are starting to understand the cost of complexity.
We have been trapped in a hype cycle for patterns and architectures for many years now. Each new approach is lauded as the answer to the ills and evils of the last, with simplistic examples given at conferences that show how whiz-bang it is, and all the cool devs of course will look down on you if you stick to the yucky old ways of doing things.
Every new approach falls short of promise when scaled and is followed by added complexity in compensation for the lost benefits that went along with the costs of the previous approach. I have come to believe that the majority of us are unable to realistically evaluate the true ROI of frameworks, patterns, and architectures. But once they get mention in the magic quadrant by Gartner all the CIOs and consultants will be demanding its use, so it is not just engineers at fault.
The rate of failure and/or cost overrun in large software projects is not sustainable, but everyone is afraid to point out that the emperor has no clothes when it comes to the latest “correct” way to do things.
My thoughts on the subject are much aligned with yours: http://onverus.com/index.php/just-enough-complexity/
I’ve been having an increasingly difficult time making this clear to my junior developers. I’ve seen a lot of ideas come and go in my 20 years. You know what wins every time? Easy to understand code and concepts. Not the “hey let’s configure every aspect of this module”. Get data, process it, put it somewhere. Do not over complicate from the start. Complicate when you cannot work inside the simple framework. Then realize that your complicated solution isn’t even solving the problem.
This has been 90% of my career’s experience. Solve the problems you have today. Ignore all the other bullshit until it’s actually a problem worth solving.
Wow. Lots of comments. I’m new in current programming languages. Use to do Basic –I mean real old basic. I have seen a big problem with programmers –they do love their codes and making a simple one liner into a complex subprogram. So I comb the internet and garner one liners and simple code and am making my programs that way. I see a lot of request for help from beginners. And it is hard understanding the jargon and complex codes that are offered as solutions to these folks when the one liners would do. I wonder why we can’t just call red “red” and blue “blue”?
As a non-SW person, my customers have made a lot of money with machines controlled using my software. It is important to recognise that tools are important. Having started life with a slide-rule and Log-Tables, through calculators to advanced algorithms running in FPGAs and, yes, Excel macros, it would be clearly absurd to suggest that modern methods should not be employed. It’s a similar story in the world of PCB layouts and mechanical CAD. The important unit of measure for me is, does the new tool simplify what I am doing so that I can stand on the back of giants?
My team found eventsourcing really simple, easy to learn, well thought out. Documentation is very good. Library code is really simple easy to read. It’s Python, so not for everybody I guess. We do domain driven design, and use event storming, so it’s a great fit for us. I first heard about it from this nice podcast: https://www.podcastinit.com/event-sourcing-with-john-bywater-episode-131/
Having worked with OutSystems Low Code for the best part of two years now, I can assure there is no hyperbole associated with the OutSystems Platform. It does precisely what it says ‘on the tin’, robust enterprise applications built for mobile (any platform) and web 10-15 times faster than hand coding. The platform’s capabilities are simply extraordinary, especially as DevOps is ‘managed’ within the platform, enabling deployment with a single click. Whatever you might think right now, there is a wave of change sweeping the tech world. The biggest misconception of low code is that it sits firmly at the ‘citizen developer’ end of the spectrum, wrong! The power of technology is shifting into the hands of users, there will be a place for specialist coding but that’s it. You’re right in that tech should be simple, not simple in its form but simple in its application…
> First is that experienced developers often hate these tools. Most Serious Developers™ like to write Real Software™ with Real Code™
Sure, but most businesses are also doing it this way in order to cut costs. And that typically includes pay. So if you’re a developer who wants to get paid well, then that whole “Real Software” and “Read Code” spiel actually makes a lot sense.
These tools can provide real business value at a significantly lower cost. But a big chunk of that “lower cost” is the fact that developers are getting paid less to work with “not Real Code”. The pushback is very real, because we’re talking about developer livelihoods here.
I have worked on and delivered systems in Access ’98. I respect businesses that run on Excel. I’ve done consulting for small businesses and government. At the end of the day, those businesses are not going to pay me top dollar to implement their systems. Google will do that.
So can you blame people for wanting to learn the tools that Google is using?
You’re right on target, at least as far as business software consulting is concerned (and not pure pushing the envelope research). Businesses that pay our salaries in one way or another benefit from ROI on the time they pay for. The quickest, longest lasting, lowest TCO solution is what helps them succeed in their industry. Those of us who help that happen repeatedly become trusted business collaborators and can rest assured we’ve delivered maximum value. Of course not everything can be simple and there is no one size fits all. I’ve found providing multiple options on different stacks really helps the best choice present itself for each situation. Thanks for sharing where your head’s at.
This is so true. I recently wrote two very sophisticated projects – one synchronized multithreaded, one distributed fault-tolerant WAN app. The key is to keep it simple or it will not be maintained and survive. MS may not always be the best tools, but they will always be there. Make sure all NuGet and external dependencies are well documented.
Well said and right on target. And I believe the reason why code complexity continues to be an issue is because programming languages lack “simplifying abstractions”. For example, cars use three simplifying abstractions: keys for user authentication, pedal on the floor to stop and start the car, and a steering wheel to point the car. Users who master these three simplifications can (mostly) safely commute from Point A to Point B at insanely high speeds inches away from other vehicles traveling at high speeds.
Unfortunately, writing code has no equivalent simplifying abstractions that will remove complexity from the developer’s domain/responsibility/control. Sure, methods and functions and API’s try to reduce complexity but all they really accomplish is broaden the scope of what we need to know without deepening our knowledge of its functionality; they defer our exposure to the complexity of the code.
Instead of using a programming language to write descriptive narratives of what the computer should do, useful simplifying abstractions could be based on training a dog (or other animal of your choice) or exploring a maze or making an engineering drawing or building a structure using the equivalent of digital Legos(TM). These simplifying abstractions would require two basic characteristics: (1) specifying program behavior by modifying properties instead of changing code, and (2) ability to “get under the hood” and make modifications to handle the edge-cases, for example (using the car analogy), special situations like driving at high altitude or extended driving at high speeds or extended periods of idling.
The point about simplifying abstractions is that we humans should do what we do best (define/visualize what the app should do) and computers should do what they do best (take care of the details with absolute precision and infinite patience). Without adequate simplifying abstractions, we are stuck dealing with expanding complexity with increasingly inadequate tools.
Thanks for writing this post and sharing your thoughts on the increasing complexity of software development. Based on all of the comments that you’ve received, it seems that a lot of us share your observations and concerns.
You wrote that “our industry right now is far too worried with following the lead of the big tech giants to realize that sometimes their tools don’t add a lot of value to our projects.” There’s a growing trend to over-engineer solutions and prematurely address their scalability, and it seems to me that this is a significant contributor to the increased complexity that you described.
You also wrote, “Maybe we can lean on more integrated tech stacks to provide out of the box patterns and tools to allow software developers to create software more efficiently.” That perfectly describes my situation. I think one of the keys to success and longevity in doing software development is finding development tools that are simple, dependable, flexible, and perhaps most importantly, that you enjoy using. My current stack consists of Aurora (Amazon’s version of MySQL and PostgreSQL) and a little-known programming environment called Xojo (which I consider to be my “secret weapon”). It couldn’t be any more simple than that. I’m finding that with it I get more done and in far less time.
I also found Steve Naidamast’s comment about the additional responsibilities that developers are taking on to be interesting. He wrote that the “movement to take on additional development responsibilities with the advent of Agile and now DevOps to meet ever tighter deadlines with fewer and fewer resources; both paradigms which will prove to be failures in the long term.” I see this movement as an attempt to merge the responsibilities that have traditionally been assigned to distinct development and operations teams. I think it’s too early to tell if this is a successful IT strategy, but as Steve said, I suspect that in many cases it won’t be.
Thanks again for the post. I’m a developer based in Richmond, and I hope our paths cross at some point.
Great article Justin. The 20+ years that I have been in this industry, I’ve heard and seen so many religious arguments for one technology versus the other (anyone remember the introduction of Java?). While I do have my own ideas about what works best, ultimately the technology is situational to a certain extent. It just depends on what you are trying to achieve and the customer’s needs is ALWAYS the most important. Simplification is really the goal as I think most people would agree. Controlling costs should always be at the top of your list when building systems. As well as, balancing that happy medium with the customer of cost versus complexity. While they may want software for complex processes, this often opens a dialogue for process improvement or modification in an effort to reduce software development costs. Once a plan is put in place, then frameworks can be discussed to address the requirements. It’s important to note that in the consideration of frameworks, who will have to maintain it in the future? You or the customer? Again, yet another point to come to an understanding about with your customers because you may need to pick technologies that play to the strength of the their own teams.
Most new coders do have a formal education – not so 15 years ago. People have longer and formal IT educations now.
Go back 30 years ago – most was high educated but often not IT-related (eg any kind of engineer).
15 years ago – still many came into the IT business by being employed in a cooperate that needed more resources in the IT department. Needless to say they didn’t try to make complex code (which is often good). Programs that was utterly crap – was often a step forward anyway – like having a running web-shop earlier than competition.
DRY, KISS, YAGNI and SOC are probably the most beneficial and important principles in software engineering to master. Part of that mastery is knowing when not to go too far as well. You can oversimplify and overcomplicate but it’s probably easier to fix oversimplifications with some exceptions (missing data because you stored the result of a + b which you imagined was simpler rather than a + b but later a requirement comes in and you need a independently). To master them isn’t purely a matter of study. Practice makes perfect. Simplest way, make your thing work then after that work out how to simplify it as much as possible.
You start the bar a bit low at DRY. DRY is one of the most fundamentally important principles in producing good software. It tends to be measurable or easily demonstrable and while like anything people can take it too far a high level of DRY is nearly always good and you have to take it to quite an extreme for it to be too much compared to other things. Gains are usually very quick and obvious with DRY and not so commonly debatable. You have a handful of thresholds and exceptions that should be easy to both conform to and demonstrate the pros and cons of precisely. As you raise the bar things become increasingly easier to adhere to problematically or inappropriately. When and how to DRY tends to be obvious and easy to learn to do automatically with little overhead in terms of delivery especially when well practiced. As I said there are exceptions but it’s not really that ambiguous or complex. There’s a particular concern with DRY. It’s so easy to copy and paste creating ten times the code. You can inflate things enormously very easily, changing one or two things and creating a spot the difference nightmare. You really need to adhere to some basic level of DRY otherwise there’s no limit as to how much code you can have. DRY also helps with your architecture. It’ll naturally reveal shared dependencies doing half the work for you in terms of organization. It’s not just about preventing inflation.
SOC is very important but easier than DRY to do unnecessarily or without gain. People can get it wrong not only in respect to what to separate but how. Is it always good to have presentation distant from logic it’s intimately dependent on? Do you separate by file, class, method, etc? For small programs unlikely to expand is the separating needed?
Talking about composition versus inheritance you’re moving much more into the zone of does it really matter. Wait for inheritance to become a problem. Wait for composition to become a problem.
You’re starting right at the top of the food chain. This isn’t what’s happening with overcomplexity. If your complaint consists of DRY and SOC then I don’t think you’ve seen the worst of it. Your argument might be better placed many many years ago when potentially excessive standards concerned more bare minimum basic principles which generally if not applied idiotically (naively, religiously, excessively) tend to deliver worth with very little extra effort and tend not to be excessive. At least for any code endeavor that’s not going to be tiny. Your starting position isn’t quite what I might call high investment, high standards or high maintenance unless people are being overly zealous like a grammar nazi. Those are just normal standards. Same as consistency. If you don’t stick to these standards to an appropriate degree you can expect to see productivity dive as well as quality in respect to correctly functioning software over a relatively short time span (even within a year).
Over time what I’ve the bar go up and up for programming on a LAMP stack in terms of minimum complexity needed regardless of the actual requirements of an endeavor. None of these things are strictly bad but if you do then for the sake of doing them and even worse entirely by the book in the most strict sense then what you have is ten to a hundred times the complexity needed. The worst thing is that I quickly switched from Java to PHP to get away from this madness.
Today you need to do: DDD, TDD, BDD, anything with DD in it, all the “best practices” (anything anyone claims is a best practice from whatever soapbox they’re given), all the standards, power tools (ultra IDEs, etc), design patterns (all of them all the time), SOLID (all the time), OOP (all the time even though PHP is multi disciplinary), Agile (both XP and Scrum at the same time), TDD, Unit Tests (maximizing coverage), integration testing, REST (I guess that’s progress over SOAP same as JSON replacing XML), UI testing, full integration of tools for the full SDLC pipeline, Event Sourcing, Command Bus, CQRS, Queues, Microservices, Cloud, DTOs, Virtual Services, NoSQL, distributed everything, in memory databases, docker, template languages even though PHP is one, huge frontend and backend frameworks, ORM, functional programming, etc.
The list never ends. Many of these are useful in specific niche circumstances and when not over done. I’ve used a good many of these and taken it upon myself to apply them but when I’ve had problems where they’ve been relevant and usually one at a time. What I tend to find is I look at this list and think well that’s daunting, all of that. If I’m a good programmer and even I find having to do all of that a bit much then you have to ask what about the rest. What you’ll find is that they’re not coping with all this stuff. They’re just learning to repeat it by rote. They’re not solving problems they’re just making code. You don’t have to understand these things and apply them appropriately or well, you just have to do it.
Apart from OOP you’ll not see programming in that list. Though even when it comes to OOP you see people spending more time orienting objects than programming.
There are a number of drives for this. Part of it’s the Java effect where a lot of people have been trained in that but end up going into something else. Much of it is the illusion that it’ll make things simpler. A lot of training wheels stuff like TDD or perceived productivity enhancers. Also the illusion that it’ll contain bad programmers and force then to write good code. Another notion that it’ll future proof the code (which is practically premature optimization). Often cases people are just copying big names in technology because they somehow think using the same technology stack will lead to magical success. This is well known as Cargo Cult. Rosetta code is another sought after benefit attempting to bridge the gap between developer and requirements or code and the business. Many of these things are the titles of CS courses or tech presentations which tends to tell you these people aren’t really avid programmers but avid students.
All laudable goals but the lofty the promise the less likely it’ll be fulfilled. People have been creating successful applications without all the works and fundamentally the difference is quality of programmer not so much quality of tools and methodology (which often turn out to be far from perfect).
In nearly all cases you’ll end up getting the opposite. You have your programmers spread out. I call this anything but the code. It’s almost malicious. Finding excuse to do anything else other than the code that needs to be done to meet the requirements. It puts a huge mountain of work in front of even the most basic programming task. You’re really onto something when you point out that there’s stuff that should be secondary to the immediate business needs. You can’t always sacrifice the future for the present but you should be still spending a large portion of your time on present concerns and not staring into a crystal ball. Even experienced developers have significant limits to their clairvoyance.
Increment slowly. Don’t add all the things then have to remove things. Solve problems as they arise. If you’re going to solve a problem preemptively then make sure you have the experience to really be certain about how that problem is going to play out and that the solution will fit.
None of these things come with a magical ROI. In some cases the thinking behind that couldn’t be more wrong. As in that adding more and more will make things more maintainable when minimalism is one of the most fundamental principles of maintainability. Anyone with a bit of common sense would know that as a rule of thumb more layers of indirection, more objects, more lines of code, more to maintain. Adding something as complex as a million lines of code framework or toolkit to your project might not make it simpler. Adding complexity to gain simplicity is never a sure thing. Good programmers will instead apply things on a case by case basis.
Sometimes these things are promoted on the basis that if everyone uses the same thing then it’ll make it easier to have some kind of widespread compatibility with developers that only have to learn one thing or already learnt it while ignoring that you’re adding something new to learn in the first place. Your solution is your problem.
Subjective and highly tentative gains are regularly presented as well as claims that these are proven. Unlike a lot of people only reading blogs I’ve read one of two of those research papers and they’re not saying or proving much of anything. They tend to apply to specific cases, are purely anecdotal. It’s hard to rule out conflict of interest.
Maintainability is drilled in at university. I remember that but spotting the problem isn’t as easy as solving it. You ask someone why something is maintainable or easier to read and when you get to the bottom of it you’ll find it’s because someone else said that. It’s not tested, measured or understood. An education only goes so far in making up for a lack of experience and practice.
Quite often people don’t understand the basics behind these solutions. You need to test your code and make sure it actually works. There are many ways to make sure your code is testable. It doesn’t have to be one way. Divide and conquer is a common principle for making code maintainable but you don’t do that down to the individual line and there are a number of strategies to compartmentalize. Breaking things up in the wrong place can make things larger rather than smaller. Start from the top down. Divide by two, then if need be divide by two again. Don’t divide by a thousand straight away. Let things grow and when they’re actually starting to get too big then break them up.
I’ve seen first hand some of the mess this makes. A template language is used in the belief it’ll stop bad developers putting logic in views that doesn’t belong there and that it’ll provide security. Now you have ten problems. A bad developer will always find some way to break out of the cage or abuse their tools. You now have to learn two languages. Your pipeline is now much more complex. You template language likely wont always get it right with its best guess escaping and will try to escape for everything rather than what’s needed by the context (some people seem to think context based escaping going by extension solved everything). Developers that can’t write secure code in the first place can only be helped by tools up to a point and if anything it’ll deprive them the learning opportunity. Your template language is a crippled language often missing even basic functionality. There’s no rule stating that a view always needs to be simple.
A darker side to this is that once you get over all of these learning curves most of them really offer little benefit. It’s a kind of gnosticism. It’s not programmer friendly but anti programmer.
I have to raise a point of contention with you on the toolkits. Some experienced developers will object because they don’t want to learn knew things if what they’ve learnt works. Some however have the experience to know these things aren’t delivering on their promises. Quite a few things even when they do they add other costs that far exceed the benefits of saving a few lines of code. I’ve been saying for a decade now that libraries tend to achieve a better outcome and people are finally starting to warm up to this idea as they’ll gaining experience with their frameworks and toolkits. It’s not perfect but it’s a start.
I whole heartedly agree as well. I remember being a PowerBuilder programmer back in the Mid 90’s up until around 2005, right when the shift to the web became more prominent. At first I embraced HTML, CSS, etc and thought it was the coolest thing until Flash came along and rocked my world. Between then and now there has been so many new technologies, probably well over 100+, all related to the web and mobile. Each year it gets more and more complicated too the point I am seriously missing the good old days when life was grand, simple, fun and you could get things working really fast especially having a visual interface, that I always considered my painters canvas. Currently, things are way too crazy: I’m doing api design and development using ASP.NET Core, REST XML/JSON, Swagger (YAML), using Dapper as an ORM, AutoMapper, NLog, AWS Toolkit, Amazon API Gateway, Models, DTOs, maybe POCOs, a Dependency Injection Framework, Fluent Validation, Repository Patterns, blah, blah, blah, all all of this is just the integration services piece (middleware) that doesn’t include the front end involving angular, etc…. omg we are living in a crazy age LOL. I can’t keep up with all this stuff. Some parts are truly fun, don’t get me wrong, but this is developer brain overload.
The problem with the software industry is it has a short-term focus on the wishes of developers and completely ignores the long-term needs of maintainers. Most toolkits and frameworks have very short lives compared to those of the languages they support, and this leaves us with unmaintainable products in the long term – and often in the short term too.
The thing that never changes is language. If you can describe in unambiguous English what your code should do, then it can be read by any intelligent person and you have a story that will live for decades. It’s just a matter of coding as close to that story as possible. I am convinced this is why SQL is so effective and long-lasting. It’s the story that matters; readability ensures maintainability. If you have a component that really is complex, give it a sensible API and package it as a library module so it doesn’t interfere with the story. It’s the way human languages work.
Just my 5¢.
Mind sharing this language of yours?
Leave a comment