Posts Tagged ‘testing’

Testing re-Revisited

Monday, November 26, 2012

Last week I went to SCNA: first time, but good conference overall. I may post separately about it, but no promises.

Anyway, Michael Feathers gave a talk titled Testing Revisited in the afternoon of the first day. The basic message of the talk is that we shouldn’t become hostage of our tests: they are just a means to an end and at every step we should be able to evaluate the value that they provide and even (GASP!) delete them when they are not needed anymore. In principle, this is a valid message. When you’re new to something you get to exaggerate and stick to a new principle or to a newly learned technique a bit too much. It may also be a matter of staying in the comfort zone. In fact, every team that successfully transitions to automated testing may be afraid of losing confidence in its code base by reducing the test suite.

Feathers motivations are mainly fed by the idea that he has met several teams reporting to him that at a certain point they felt that their test base was slowing them down. That is to say that the developers had a clear impression that without tests, or with less tests, they would have moved faster. This is already controversial, but there are some elements presented by Feathers which are interesting nonetheless:

  • First of all, as a team we should always be able to evaluate the actual utility that every test brings to the table. Tests that used to serve a purpose but not anymore should be deleted from the suite. An example could be a regression test originally written to reproduce a bug at  a big scale, slow and clumsy => if the bug cannot happen anymore by design, then there’s no reason to keep that test in the suite.
  • Secondly we should be able to reason about the risk involved in removing any test, especially before scaling a build system that cannot accommodate an infinite growth. More importantly, by reasoning about the risk and the utility of every test we can decide to rewrite them, maybe in a lighter way.

Let’s get to the point of this blog post then. Feathers said something very specific, which was: “maybe we rely too much on automated tests”.

I think that this is a dangerous and ambiguous message. In my experience, the only way to be slowed down by tests is always, absolutely always, no exceptions, because of the code base that is under test, not because of the tests themselves.

What follows is what I tweeted the day of the conference:

thinkingbox
While I appreciate some elements of @mfeathers talk I disagree about the sentence: “maybe we rely too much on automated tests” #scna
12-11-09 5:45 PM
thinkingbox
Teams who deliver good code and have good practices are the ones writing more tests and suffering less from them. /cc @mfeathers #scna
12-11-09 5:48 PM
thinkingbox
Teams who suffer from legacy code, with devs not up to the par & suffering from tests are the ones who need them the most. @mfeathers #scna
12-11-09 5:48 PM
thinkingbox
Going after tests seem to me going after the symptoms, not the cause, hence perpetuating the problem. @mfeathers #scna
12-11-09 5:48 PM

And then the day after:

thinkingbox
@yehoram @mfeathers I think that the main problem is sending a somewhat mixed message to teams which don’t have consolidated practices yet
12-11-10 9:33 AM
thinkingbox
@yehoram @mfeathers While it’s true that we should always evaluate the risk which is covered by tests and ultimately their actual utility
12-11-10 9:34 AM
thinkingbox
@yehoram @mfeathers Team which work better are the ones having the least amount of problems with their tests.
12-11-10 9:35 AM

What I’m saying here is that in my experience the teams that are truly slowed down by their tests are in reality slowed by the bad quality of the code under test, not the tests themselves. In practice, I’ve seen cumbersome and brittle tests due to the complexity and intricate dependencies of the code under test. This can be noticed more easily with legacy code.

Some teams struggle, but they are able to navigate through these problems and radically change their systems. Some other teams don’t seem to be able to take the bull by its horns, sometimes also because they focus too much on the tests and not enough on the cause of the pain: their code.

So in the end, while I appreciate Feathers point of view and overall his talk, I think that in particular this aspect can be misunderstood and taken as an excuse to test less, so that teams won’t be slowed down. Unfortunately this is just an illusion and when it backfires it hurts.

Advertisements

Our opposable thumb makes us all professional mechanics, doesn’t it?!?

Thursday, October 29, 2009
Usage of duct tape

Duct tape on Apollo 17, courtesy of Nasa

After watching the video of Uncle Bob’s Clean Code presentation at NDC 2009 last Friday, I’ve thought a bit about the paradox in which our profession lives, especially in light of the recent debate around testing ignited collaterally once again by Joel Spolsky’s blog post. This  paradox is interesting, because it is multifaceted.

Facet 1: most of the companies prefer to deal with many cheap, sloppy programmers instead of investing in a few very good programmers.

Facet 2: despite all the knowledge about software development is easily available a) on the internet b) in multiple ways and c) at a very convenient price (if not free), most of the developers don’t learn much after the university and don’t take any pride in exercising their profession.

Facet 1 may be a side effect of the whole “IT Doesn’t Matter” phenomenon: if investing in IT doesn’t give you anymore a competitive advantage, then maybe it’s not worth investing in the best developers you can find, but just in “developers”. While I generally agree with Carr, especially with regards to “simple” tasks like Office Automation or building corporate websites, there are still huge areas of inefficiency where IT can play a significant role. Maybe it will not give you a significant advantage over your competitors, but it can still save you a lot of money. I personally know a few examples in the logistics domain, but unfortunately I cannot disclose any details because of existing NDAs. Let’s just say that we are talking about many millions of dollars of savings because of problems that in theory have been solved already at the beginning of the nineties. IT wouldn’t matter, if only it was applied and used consistently!

I have also experienced personally the 10x productivity factor: I have seen sloppy programmers being 10x slower than good programmers (not exceptional programmers, just good ones), whose salary wasn’t even nearly 10x bigger than the former ones (let’s be optimistic, let’s talk about 30-40% more). So, it still seems kind of dumb to resolve to hire mediocre people, especially when you are a medium-large company: you certainly have bigger capacity to absorb inefficiencies, but those numbers are starting to be so big, that unbelievable amounts of money go down the drain just because of this. Having said that, there are not nearly enough good programmers in the world and it seems that HR and head hunters have just given up. Sidenote: I happen to know many good programmers and all of them are having a successful career, so it seems that there is always a need for good programmers, at least in this continent. Good programmers, people who care, please don’t give up!

Is there a cause-effect relationship between 1 and 2? I don’t think so. It seems simply that there is a bit of pressure to level software development discipline. I’m sure that this is not the intent of Joel Spolsky, quite the contrary, but singing an ode to the duct tape programmer, even when the programmer is a good one, doesn’t seem the smartest idea to me. Put duct tape in the hands of an exceptional programmer and he’ll do great things. Put duct tape in the hands of a sloppy programmer and he’ll do an awful job. Have you ever seen a participant to Canada’s Worst Handyman assembling a shelf?

For some reasons, everybody thinks he/she can program, even without the appropriate knowledge or background. After some practice in Excel, everybody can certainly write VBA macros (and do a terrible job anyway), but it’s like saying that because everybody can use a wrench then we are all professional mechanics. I guess it’s all about perception, so a lot of people oversimplify the complexity behind implementing and maintaining a code base for any significantly complicated business problem. I’m not saying that every software programmer needs to be as good as Ward Cunningham, but in my mind there is a clear trend towards embracing more and more people in the industry with less and less capable minds. Even worse than that: a lot of programmers seem simply to not care at all.

What is wrong about letting more people get into software development? Generally speaking, nothing – assuming that the complexity at hand has been reduced over the years, thanks to astonishing advancements in tools and technologies. To some degrees this is true: the things that we can do today would have been virtually impossible twenty years ago, but during the same period of time the industry has just raised the bar, tackling more and more complicated problems. The revolution that started with personal computing isn’t just done yet.

Hence, we need smart people, willing to learn new things every day, constantly looking for ways to improve their work. It’s like having your legs solidly planted in today’s problems, but your eyes are always looking at tomorrow.

Duct tape is an exceptional tool when you need to fix something quickly and for whatever reason you don’t have access to the appropriate tool, knowledge, etc. Maybe sometimes you leave duct tape in place for years (hell, my plastic pipe that conduct warm and humid air from the dryer outside of my house sticks into its place thanks to duct tape), but this is generally not a good solution, capable of lasting for a long time. Sooner or later somebody will have to come and clean the mess that the duct tape expert has left. Two, three times a year I have to replace the duct tape because it simply wears out over time: it’s annoying and I would have fixed it already if I wasn’t renting my house.

There may be specific situations where this is tolerable, but not in much broader terms. And yet, there seems to be a tendency to underestimate the effects over time of such short-sighted decisions in our profession. Is it a tolerance to mediocrity? Are we all getting dumber?

Think about it this way: would you accept to have a sloppy pediatrician taking care of your child’s health? Would you go into the surgery room with a mediocre surgeon? Would you put your life on the line in a judicial trial with an incompetent lawyer? Would you trust an ignorant plumber to do the plumbing of your house? Etc., the examples are endless. We are not talking about NASA-like situations – these are more or less very common problems.

The point is that you would NOT. And yet, many managers and entrepreneurs, even when they are technical, do with software development what they would not do in their personal lives. Is it because it’s not their personal wealth or beloved ones at stake?

I’ve always found that the best managers and entrepreneurs are the ones which know the difference between cost and investment. There is no return from a cost, but there better be from investments. If you are on the upper side of the food chain, which return do you expect from the developers working for you? How can you maximize it? But if you are on the lower side: how can you provide value to your company? How can you be always on top of your profession?

If it’s for test then it has to be good

Wednesday, March 25, 2009

You are dealing with some old code: it’s not too bad, you have definitely seen much worse. There’s a class within a home-grown framework that doesn’t allow you to test some application code. You spend some time understanding how to change things to make it more testable. After a while you come up with the idea of adding a method to that class solely for testing purposes – it’s not a fantastic thing, but the alternatives are frightening and this will make a lot of code much more testable. Because you are a good person you want to make it very clear that that code is intended to be used for testing only, not in real life:

public void setMickeyMouseForTest(Class cheeseClass,
                                  String cheeseName, Mouse mouse) {
    cheeseClass = resolveAssociatedClass(cheeseClass);

    if (getLocalFactory().getCheeseName(cheeseClass) != null)
        throw new RuntimeException("Use standard setMickeyMouse instead");

    mouses.put(cheeseName, mouse);
}

This method is similar to the real thing:

public void setMickeyMouse(Class cheeseClass, Mouse mouse);

Method setMickeyMouse determines cheeseName by calling getLocalFactory().getCheeseName(cheeseClass), while method setMickeyMouseForTest assumes that cheeseName is not set (even stronger: if it is not null it throws an exception) => instead, it uses the parameter provided in the call.
In the end, the difference between the two lies in the way a certain condition is handled in the code: the method written for testing purposes is capable of dealing with an exceptional condition differently. You’ve created that method exactly to simplify the amount of setup required for tests, so that it doesn’t matter if getLocalFactory().getCheeseName(cheeseClass) is not capable of providing a cheeseName.

So, the code is committed, lots of tests are now possible, the team celebrates your genius.

Some weeks later you’re pair programming with a colleague. You’re still dealing with the same framework, but this time it’s a different corner-case. After lots of mumbling, back and forth, debugging sessions, you superficially look at the signature of that method, obviously without remembering what you have done a month before. You just vaguely remember that it was capable of dealing in a better way with a corner-case. And the method is called “…ForTest” so it has to be good, in the end we are testing indeed.

It doesn’t work. An exception is thrown. The exception hints to “Use standard setMickeyMouse instead…“. Oh gosh, let’s take a look: ops, I know what it is, we’ve forgotten to set cheeseName for cheeseClass in the local factory. Let’s do that.

<type> <type> <type> <type> <type> <type> <type> <type> <type> <type> <type> <type> <type> <type> <type> <type>

Damn, it’s still not working. What is it happening?!?

<debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug>

It’s very strange.

getLocalFactory().getCheeseName(cheeseClass) is not null, so how can it possibly say that it is null instead?!?

<more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug>

Oh my gosh! The exception is actually thrown WHEN getLocalFactory().getCheeseName(cheeseClass) is not null, not viceversa!

What happened is that you and your colleague just wasted 20′ because you misread “!= null” (reading “== null” instead). What does that tell you?

It told me a few things:

  • Managing null values is always evil and error prone. At the very least, checking for a null disrupts the business logic and it makes the code less readable, ultimately more likely to break.
  • It’s too easy to misread a check based on != vs ==. Not only that: the fact that an exception is being thrown if the expression evaluates to true hints that the check is to verify if the value is null, because that is what you would typically do when you are adding a guard to your code. Which brings me to the next point.
  • Moreover, the intent of the code is not clear. It’s not enough to call a method “somethingMeaningfulForTest“, that is the suffix “ForTest” is misleading if the method is not truly applicable for every test. This is even reinforced by the fact that the exception message is not enough communicative.
  • As usual, test code has to follow the same standard of quality as application code, which means readability, expressiveness, simplicity, lack of duplication. This is particularly important for accessory code that is written to support tests.

So, in the end, how should that method be called? What about:

public void setMickeyMouseForTestWhenThereIsNoAvailableCheeseName(
    Class cheeseClass, String cheeseName, Mouse mouse);

The underlying precondition is that method getLocalFactory().getCheeseName(cheeseClass) returns null for this code to execute properly. This should be made clearer with a checked exception being thrown by the method, or, even better, by a custom annotation that marks the method with the defined precondition. In Java land this doesn’t go very much beyond documentation purposes for a variety of reasons:

  • First of all there are limitations on the attributes that an Annotation can exhibit. Specifically, only String, Class and primitive types are allowed, not even object version of scalar type like Integer can be used.
  • Second and foremost, annotations can be queried to provide metadata about code, but they don’t really affect code execution. An annotation is useful only to the degree it is recognized by the runner executing some code. There is in the end no native and effective support for Design By Contract in Java and there will probably never be. A useful result can be achieved by combining together AOP with Annotations, but I have mixed feelings about it => in the end I always have the impression that the mumbo jumbo typical of the supporting infrastructure it’s there just to circumvent a limitation that should be inherently addressed by the language itself.

Or shouldn’t it?

Using functors to handle test connections

Thursday, October 9, 2008

A few days ago I was working on some functional tests that have the following structure:

      public void testSomething() throws Exception {
        // 1. istantiate the system
        // 2. interact with it through the UI
        connectDB();
        // 3. assert the state of the system by examining the DB
        relinquishDB();       

        // 4. interact a bit more through the UI
        connectDB();
        // 5. assert again the state of the system by examining the DB
        relinquishDB();        

        // etc.
    }

Some notes about them:

  • Tests are implemented with JUnit 3.8.2, but this is irrilevant to what will follow.
  • The application is a webapp implemented with a proprietary web framework.
  • Those tests are implemented running the servlet engine in process through ServletUnit.
  • The DB is interfaced through a proprietary framework and it is accessed behind a mock JDBC driver that works translating SQL statements in memory.
  • The state of the system is examined by fetching either an Entity or EntityList, abstract classes that represent respectively a single record or a collection of records in a table.
  • Methods connectDB() and relinquishDB() are used to manage the test connection, which needs to be different than the one used by the application for structural reasons.

All the tests that follow the above structure inherit from an abstract TestCase which implements its own setUp and tearDown, treated like template methods whose behaviour can be refined further (for example, choosing to install a different mock DB). connectDB() and relinquishDB() use connection parameters that are potentially established differently for every test, but in reality they are seamlessly copied over from test to test.

While this structure makes sense, I thought that repeating connectDB() and relinquishDB() several times in the test is extremely annoying. It would be nice if we can wrap some logic into a block that takes care of the repetition for us: calling connectDB() first, executing our code and then calling relinquishDB() in the end. If I was programming in ruby, that would have been extremely easy to implement with proper blocks, wouldn’t it be? Well, in java is not that difficult either, if you tolerate the syntactic noise of course. Let’s see how.

To start, we can define a Functor interface that can be used to implement a block without parameters:

public interface Functor<T> {
    T exec();
}

Now we need to wrap a bit of logic around it that takes care of connecting the DB and relinquishing it afterwards:

public class Fetcher {
    public static <T> T queryStateFor(Functor<T> f) {
        connectDB();
        T result = f.exec();
        relinquishDB();
        return result;
    }
    private static void connectDB() {
        // ...
    }
    private static void relinquishDB() {
        // ...
    }
}

The actual implementations of Fetcher.connectDB() and Fetcher.relinquishDB() are fairly trivial and the methods are in reality inlined: I’ve written them in a Compose Method style just to make them easier to understand and to trace back Fetcher structure to the test structure above.

The Fetcher can be used in the following way (assuming that it is imported statically in the test case):

    public void testSomething() throws Exception {
        // 1. istantiate the system
        // 2. interact with it through the UI
        final PurchaseOrder po = //...
        PurchaseOrderStatus status = queryStateFor(new Functor<PurchaseOrderStatus>() {
            public PurchaseOrderStatus exec() {
                return po.getLastStateChangeStatus();
            }
        });

        // etc.
    }

As you can see, this refactoring is not about saving keystrokes.

A developer was asking me which kind of advantage this style achieves over the other. In general, I think that functors are a powerful tool, often underestimated in statically typed languages like java. But these are the reasons why I think they are particularly relevant in the context of queries and tests:

  1. Avoid repeating connectDB(); … relinquishDB(); without the introduction of methods that would have to be rewritten for every test.
  2. Ensure that queries for checking purposes correctly connect to the proper DB and relinquish it in the end.
  3. Make the test more readable, without hiding relevant details (connecting and relinquishing the DB are irrelevant details to the test, only the query matters in the eyes of the reader / maintainer).
  4. In general, make the test less dependant upon lower level details whose relevance is marginal.
  5. Using a functor the query can be executed as a generified method, hence it definitely scores better in terms of type safety.

I’ve also thought about any disadvantage:

  1. The syntax looks a bit odd for the java world, but it’s not entirely a stranger.
  2. Current functor implementation can handle only one result per call, but it can certainly be extended and in general it doesn’t hurt significantly calling several functors in a row.

And you? What do you think about this way of handling a test connection to the DB?

PS: While I think that this style provides a better alternative to repeat calls to connectDB() and relinquishDB() all over the code, there’s actually a simpler way to solve the problem ==> the tests can be provided with their own connection that has nothing to deal with the connection handled by application code running in the servlet engine (credits to Mark for recognizing it). Nevertheless, functors are a powerful tool typically neglected in the java world and they definitely deserve more attention in situations like this one.