This post was migrated from Justin’s personal blog, 'Codethinked.com.' Views, opinions, and colorful expressions should be taken in context, and do not necessarily represent those of Simple Thread (and were written under the influence of dangerous levels of caffeination).

I hate these posts. I hate them because I feel like I am drawing attention to another post which made me cringe a little bit. But I think that reacting to these posts is good, because having a healthy debate about topics is good, especially if you can keep from devolving into mud slinging and attacks. The post I am talking about is this one, which is titled “It’s OK Not to Write Unit Tests”. I was surprised to see that it is actually from March of this year, and maybe I have seen it before, maybe not, but this time I felt the need to respond to it.

One thing that I want to say first is that I want to keep this calm and mellow. I used to react very calmly in retort to this kind of post, but over time as I wrote more and more on my blog, I found myself starting to become the “everything is black and white and I’m going to tell you that you’re wrong with my blaring megaphone” kind of writer that I often loathe when I read. Sure, it might not make the best link-bait in the world, but maybe I’ll feel a bit better after I write it. Oh, and the blood pressure might stay down a little bit. Having said all that, here we go…

Bad Experience Does Not Equal Bad Practice

The first thing I want to point out, which at first glace will seem blindingly obvious, is that a bad experience with a particular tool or practice does not mean it is necessarily bad. It could be bad, but correlation is not causation and context is everything. The experience we had was bad, but it is often hard to put all of the blame on the tool or practice. For example, what if I went and tried to use Scrum on one of my projects and the entire project failed miserably. Does that mean that Scrum is a bad framework for building applications? Of course not, many people find Scrum very very effective. And if it works for a lot of other people, then sometimes the best thing to say is “let me see how these other people are doing it”, before you decide that the tool doesn’t work for you. You might find out that you were wrong, or you might confirm your beliefs, the important thing is to have an open mind.

Misapplication

We have all heard the saying “when all you have is a hammer, everything looks like a nail”. It is one of my favorite sayings because people don’t spend enough time looking outside of their own boxes in order to discover that the world is full of screws and nuts (pun intended). Not every screw can be banged in with a hammer, but at the same time, you can’t wield a screwdriver like a hammer and expect positive results. In order to be successful with any tool or practice, you need to research in order to figure out how the tool works. Or as almost any developer on IRC will tell you, RTFM.

If we are misapplying tools and techniques, then we are going to have bad experiences with them. And because I believe in testing a system at a low level, and testing a system at a high level are both important, I am going to outline a few steps that I think need to be taken toward your testing enlightenment.

First Step: Admit You’re Not As Smart As You Think You Are

Unit testing, like many other practices that the development community has adopted, are here for one simple reason. We, individually and collectively, are not that smart. We like to think we are smart, but as any Douglas Adams fan knows, the dolphins are the ones who really have it figured out. We write unit tests because all of the pieces that are involved in building large and complex applications just can’t fit in our tiny little heads. You’ve heard of the magical number 7 right? Sure, people will argue it is a wives tale, but the point is the same, we can only hold a finite number of items in our head. And for most of us, that number is not very big.

In order to alleviate the “tiny head” problem, we need to break things down. Breaking things down won’t help us if we don’t validate as we go along that each part is working. Think about it, what if we were going to build an engine. Wouldn’t it be easier to individually design, build, and test 150 different parts that make the whole before we assemble them? Surely it is easy to make an alternator and install that into the engine as a subsystem, rather than individually bring in every part that makes it up.

That doesn’t mean we aren’t going to fire up the engine after we build it to make sure the whole thing works, but if we didn’t have each piece designed right, it would be a heck of a lot harder to figure out that the fuel injectors weren’t spitting out enough fuel if we hadn’t already tested that independently as part of their design.

The author makes the point of saying:

Like, let’s say I was writing a SHA-1 hash implementation. That’s a lot of code. And I wouldn’t write little tests all the way down.  But I would have a few tests validating that it works at a very high level, absolutely.

Okay, so writing an implementation of SHA-1 is not exactly a common occurrence. And quite frankly, a SHA-1 hash algorithm is actually not that much code. It is very complex, but it is not a huge amount of code. To say that you are going to go through that algorithm without writing any tests as you go tells me that you are going to spend a huge amount of time in the debugger. Or you are a genius. Either way, why not formalize all of the intermediate testing that you are going to do anyways so that when someone finds a bug in your implementation, they don’t have to retrace your steps.

Second Step: Admit You Need Training Wheels, Or At Least Some Crutches

The author of the article makes the statement that unit tests are akin to training wheels. Something that you should do when you aren’t very good, and then get rid of when you become an expert. I can’t wrap my head around this analogy for a few reasons. First, unit tests are not testing your ability as a developer. They are testing the correctness of your code. Writing correct code is about being a domain expert, writing good code is about being a good developer. The code you are writing probably involves significant business logic that you did not create, in a field that you are probably not an expert in, and is mixed with lots of code that you did not write. And, in the future, will be modified by people which you will probably never even meet. And those people will probably be idiots (just sayin).

Let’s just make it easier for everyone and assume that you and I are both going to break your code, so put some harnesses in place that will tell me that. And no, I don’t want to know that I broke your code from a test that is 10,000 lines above where I made my modification, I want to know at a much lower level than that. So I don’t waste a ton of time in the debugger stepping through line by line.

Third Step: Address The Problem Not The Symptom

Brittle tests. Ugh. I’ve heard this argument so many times. “Unit tests are brittle, unit tests are brittle.” Brittle tests are a symptom. Go grab a few books on writing effective unit tests. There are many available. All of them have strategies for mitigating this. Most of the time these strategies don’t involve complaining about brittle tests, they usually involve teaching people how to write tests which don’t depend on external resources and which operate on public interfaces and not internal implementation details… which leads me to the next step.

Fourth Step: Understand That Encapsulation Is Your Friend

One of the practices that the author complains about is the process of making private methods public in order to test them. Again, we are looking at the symptoms. First, the only tool or entity that I have ever seen advocate this is Microsoft and MSTest. The tool that MSTest provides which exposes private methods via proxies for unit testing was one of the biggest mistakes they made with their testing toolset. Most unit testing experts will tell you that once you start digging into the internal implementations of classes you are, by definition, creating brittle tests. A private method is a big flag that says “DON’T CALL THIS METHOD EXTERNALLY, EVEN USING REFLECTION.” If you decide to do so, you are doing it at your own risk, knowing that you are creating a dependency on something that may change.

Encapsulation is one of those fundamental concepts that you should go out of your way to not violate. The sooner you learn that you should only be testing the inputs, outputs,and public state on your classes (which you should work to minimize), the happier that you’ll be. If you see something in your class that you need to really exercise fully and you can’t get to it because it is wrapped up in a sea of private methods, it sounds like it is time for some refactoring. Pull it out into a class so that you can exercise the complex behavior in isolation. That is the idea after all.

Fifth Step: Admit That You Don’t Know Statistics (AKA: Combinatorics Are Not Your Friend)

If you are writing tests at the system level then you aren’t thoroughly exercising your system. Simple as that. Have you ever sad down and thought about the insane number of interactions within the applications that you are working on? If you haven’t, take a moment and ponder it. Software projects are often giant spider webs of logic and interaction where the deeper you get into the web, the harder it is to understand how things at the top affect it. If you have ever seen a game of Plinko then you’ll know exactly what I am talking about. You may be able to predict where the chip will go for one or two levels down, but past that point there are simply too many possibilities and too much randomness. Testing “in the small” on classes that are at the lower levels of your system will allow you to have more confidence that you have actually exercised these pieces.

Sixth Step: Realize That Isolation Is The Only Way To Have Control

We all want control, right? Resorting to only doing “whole system” or “acceptance” testing makes it difficult to test anything in isolation. But why would you want to test in isolation? This is one of the most frequent questions I receive when I talk about mocking to people. And the best answer is that it allows you to control the execution environment. How many times have you seen code where there aren’t any tests that cover an error case because there is no good way of forcing an error at test time? Or how many times have you not tested something thoroughly because you couldn’t call out to a web service or read from a file during a test? These are common problems. If you don’t have a strategy for testing pieces of your system in isolation then you can’t thoroughly test your system.

Seventh Step: Understand That It Isn’t Just Integration Tests or Unit Tests

Many people, when they argue against unit tests, are merely saying that testing at the level of units is often just too small of a piece of code to be worth it. To a certain degree, I agree with this. Writing tons of tests that test itty bitty pieces of logic will probably not get you too much value in the end, and will probably end up being an anchor around your neck. But many of those same people seem to think that if you aren’t writing tests at the unit level, then the only other option is to write tests at the system level. Argh!

The more you work on a system, the better of a feeling you will get about what level most of your testing needs to occur at. For me, that level of testing is mostly at the class level and frequently at the sub-system level. Sure, you also want whole system tests, but as I said earlier, you shouldn’t lean on those too heavily.

Don’t Take My Word For It

In the immortal words of LeVar Burton, “you don’t have to take my word for it”. Go out there and grab a few books like xUnit Test Patterns or Pragmatic Unit Testing. Try to write tests the way that many of the experts out there would prescribe, and see if it doesn’t make your life easier. I know that the author of the referenced article said that he didn’t think find grained tests caught regressions, but I have never worked on a system which had a good suite of lower level tests that weren’t constantly catching unexpected regressions. I just don’t know what to think about that. And that is why I say it is so important to try it out for yourself, I have worked on well tested projects and in the end they ended up higher quality, better designed, and more maintainable because of the tests, not in spite of them.

16 Comments

Richard Lowe

I essentially agree with your points. I will say that I have empathy for the "it’s okay NOT…" dude simply because, as you say:

> If we are misapplying tools and techniques, then we are going to have bad experiences with them.

It’s pretty much given that testing will be misapplied many times as one learns to do it. I, myself, don’t feel particularly capable at it yet, though I understand its importance and am trying to test my code.

It’s always been a mystery to me why more pundits (is that a good word?) in favor of testing on all levels don’t make more appeals to the idea of testing as a kind of naive science… Science works because it’s empirical and rigorous enough to make positive headway. Science and the philosophical justification of it essentially prove our case for testing in order to have knowledge of what is or is not more likely (i.e. how likely it is our software will work).

I kinda wish we’d start leaning on our cousins in scientific fields more for justifying our own testing.

Reply
Nick Berardi

I think the 7th step, that you laid out, is the real key.

I have run across too many people that make unit testing look very time consuming and ultimately not worth the level of effort to beginners by testing every single get and set property on their classes, even if the get and set was created by the framework.

public int ID { get; set; }

The point of unit testing shouldn’t be code coverage for code coverage sake, it should be about developing consistently working high quality software.

When ever I start teaching somebody about unit testing, I start out by telling them not to test their code, but test the desired behavior of their code.

There is a big difference there, if you test your code, you usually fall in to the habit of just making tests that work with in your understanding of your own code. Which may catch small problems beyond compile issues. But rarely I have found that they catch the big issues at the system level.

But if you test the desired behavior of the code, you may not get 100% code coverage, but you can be sure your system meets the requirements and you are interacting with your code in a way that the user is expected to interact with the code. I have found a lot more, conceptual issues, using the behavior testing model, than I have with the straight code testing model. Because for most of the cases all the code worked as it was designed, it is only when you started mixing different pieces of code that things really started to show where the bugs where.

So that is my take, I usually test from the behavior level, and if time and clients dictate I take it down to the code level getting done the highest priority stuff first. But at least if the behavior level is the last thing I do, I know that any changes in the system will behave correctly as long as the unit tests continue to pass.

Reply
Al Tenhundfeld

Dammit Justin, you compelled me to read the original post, and now I’m depressed.

Reply
John Farrell

I don’t see anything substantial in this post. Its mostly arm waving and waxing poetically about human nature.

The author of the blog post your refuting brings up a lot of good points and you respond with "learn more" and book links.

Reply
Aaron Comyn

I had much the same reaction that you did when reading the original article (that is to say "[i]ummmm no[/i]").

One of the points the original author made that made me scratch my head was the idea that unit testing does not help with design. In my experience tests provide two major benefits to design:
[u]
1) It encourages looser coupling: [/u]

it’s always easier to test functionality that has fewer dependencies. If your code is hard to test it’s usually a sign that your objects are either too tightly bound to one another, or there is functionality that you should probably refactor out to it’s own object. Unit testing provides a safety harness for the refactoring, and an impetus for loose coupling and IoC.

[u]2) Cleaner and more logical interfaces and APIs:[/u]

When you unit test you are adding another client to the objects in your system. This encourages re-usability and generalized functions. The needs of test code are often very similar to the needs of your systems’ code, and the very act of using your objects more frequently will help smooth out your interface as it has been used more. I can’t count the number of times I’ve needed some new functionality in a system only to be a happily surprised when the method or constructor I was looking for had already been created to simplify testing.

Additionally, writing the interface before you write the functionality lets you focus on your code as other developers will. Logical, clean, consistant interfaces are extremely valuable in large systems and drastically simplify maintenance work. Writing code interface first isn’t strictly a technique which *requires* unit testing, but unit tests will give you a nice harness to improve the interface at a later point.

——-

All-in-all the author of the original post seems to have looked at some bad and overly dogmatic practices and habits applied with unit testing and come to the conclusion that unit testing is not helpful…

A the end of the day the level of testing that is appropriate and useful is going to depend on the kind of system you are working on, and where the probable points of failure are going to be. I think we can all agree, however, with the authors reminder that we should fearlessly analyze all our practices for value, and not go along with trends just for the sake of going along with them.

Reply
Todd

[b]Step 8 – Admit that your application will change[/b]
Face it, application change, evolve, grow. The best way to insulate yourself from changes that break an application is to have good tests that validate what you [i]expect to happen[/i]. This allows you to make your changes with confidence.

Reply
noob

1. The original author DID NOT say you SHOULD NOT write unit test. You’re article title DID NOT contradict him.

2. Think about what you are testing. If you are testing units instead of interface, then you’re wasting the client’s time.

3. Unit tests increase maintenance. When business change, you need to be able to refactor, recode without having to change every single dumb unit test.

4. Unit tests is a feel good exercise for the developer, not the client. As a client, I don’t give a shit about your coverage. All i care about is that your product does what you say it does. Verification and validation. If you are wasting my money testing things that only happens on rare situation, you’re fired. In the real world, there are acceptable quality.

5. Test services, test utils, test sub-systems at the interface level. If you have low confidence if your code would work without unit tests, work on your confidence and write better code.

6. Learn to use a debugger.

Reply
freddy

@noob

give this post another read, u somehow got the feeling he cares about coverage, when he clearly stated that the idea is not just to test code for coverage’s sake, but instead that u want to focus on what the code needs to do … in other words, just as u said in point 4. verification and validation.

Oh, and btw, if you missed to test one of those things that only happened in a rare situation but had such an impact that made me loose tons of money then u are the one fired 😉

Reply
whatever

This whole conversation would unbelievably frustrating if I cared. Of course there are circumstances in which unit testing is less valuable or less pragmatic.

But people like @noob completely miss the point. Unit testing [i]done right[/i] makes software development faster and greatly improves the quality. That means [b]greater ROI[/b] for my clients.

This is how I imagine our conversation would be if we were all surgeons.
Me: Hand-washing is a fundamental requirement for professional surgery.
Noob: I don’t have time to hand-wash. I’m trying to save lives.
Me: ? I think hand-washing [i]helps[/i] save lives.
Noob: But who has time to fetch a pail of water and bring it to a boil before EVERY surgery?
Me: What? You’re doing it wrong; here are some ways to do it smarter…
Noob: Stop talking. I’ve tried it; it’s pointless. Hand-washing is a luxury for doctors who don’t live under real deadlines. If you don’t think your hands are clean, stop living such a dirty life.

Reply
Chris Vann

I totally get the purpose of unit testing, and I find it invaluable in certain circumstances, but most people who preach and practice unit tests are developing in an open source or an open time table environment.

Very few people I’ve met that actually develop on tight time lines in an enterprise environment have the time to complete unit tests for every function they design, nor do they have the political clout to justify the extra expense to upper management.

I think it’s important for unit testing advocates to step back and examine their own perspective and realize that hey, it’s not the only one.

Reply
daniel

i really think it was an april fools joke. the post was dated march 31. so maybe just a joke posted a day early?

Reply
DevMentia

Interesting points. I can’t agree with you on the dolphins though. They just jumped ship (planet) at the last minute. The true geniuses were the mice.

Reply
Gary Woodfine

I agree totally with the sentiments in the post. A side point to this, is the code link to the SHA1 implementation is a classic example of code that was written without being tested. The code does not conform to any conventions, there are plenty of assumptions. The code could do with modularisation. All this could easily be achieved by following a succinct unit test strategy. It would have caused the author of the code to break the algorithm down into small workable units.

I suppose it all comes down to a matter of style. I have worked with many developers who don’t write unit tests, and they have a tendency of running the entire application just to get to the piece they want to test, and spend alot of time in the debugger. Funnily enough most of these guys are C++ types !

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *