Testing re-Revisited

Last week I went to SCNA: first time, but good conference overall. I may post separately about it, but no promises.

Anyway, Michael Feathers gave a talk titled Testing Revisited in the afternoon of the first day. The basic message of the talk is that we shouldn’t become hostage of our tests: they are just a means to an end and at every step we should be able to evaluate the value that they provide and even (GASP!) delete them when they are not needed anymore. In principle, this is a valid message. When you’re new to something you get to exaggerate and stick to a new principle or to a newly learned technique a bit too much. It may also be a matter of staying in the comfort zone. In fact, every team that successfully transitions to automated testing may be afraid of losing confidence in its code base by reducing the test suite.

Feathers motivations are mainly fed by the idea that he has met several teams reporting to him that at a certain point they felt that their test base was slowing them down. That is to say that the developers had a clear impression that without tests, or with less tests, they would have moved faster. This is already controversial, but there are some elements presented by Feathers which are interesting nonetheless:

• First of all, as a team we should always be able to evaluate the actual utility that every test brings to the table. Tests that used to serve a purpose but not anymore should be deleted from the suite. An example could be a regression test originally written to reproduce a bug at  a big scale, slow and clumsy => if the bug cannot happen anymore by design, then there’s no reason to keep that test in the suite.
• Secondly we should be able to reason about the risk involved in removing any test, especially before scaling a build system that cannot accommodate an infinite growth. More importantly, by reasoning about the risk and the utility of every test we can decide to rewrite them, maybe in a lighter way.

Let’s get to the point of this blog post then. Feathers said something very specific, which was: “maybe we rely too much on automated tests”.

I think that this is a dangerous and ambiguous message. In my experience, the only way to be slowed down by tests is always, absolutely always, no exceptions, because of the code base that is under test, not because of the tests themselves.

What follows is what I tweeted the day of the conference:

thinkingbox While I appreciate some elements of @mfeathers talk I disagree about the sentence: “maybe we rely too much on automated tests” #scna 12-11-09 5:45 PM

thinkingbox Teams who deliver good code and have good practices are the ones writing more tests and suffering less from them. /cc @mfeathers #scna 12-11-09 5:48 PM

thinkingbox Teams who suffer from legacy code, with devs not up to the par & suffering from tests are the ones who need them the most. @mfeathers #scna 12-11-09 5:48 PM

thinkingbox Going after tests seem to me going after the symptoms, not the cause, hence perpetuating the problem. @mfeathers #scna 12-11-09 5:48 PM

And then the day after:

thinkingbox @yehoram @mfeathers I think that the main problem is sending a somewhat mixed message to teams which don’t have consolidated practices yet 12-11-10 9:33 AM

thinkingbox @yehoram @mfeathers While it’s true that we should always evaluate the risk which is covered by tests and ultimately their actual utility 12-11-10 9:34 AM

thinkingbox @yehoram @mfeathers Team which work better are the ones having the least amount of problems with their tests. 12-11-10 9:35 AM

What I’m saying here is that in my experience the teams that are truly slowed down by their tests are in reality slowed by the bad quality of the code under test, not the tests themselves. In practice, I’ve seen cumbersome and brittle tests due to the complexity and intricate dependencies of the code under test. This can be noticed more easily with legacy code.

Some teams struggle, but they are able to navigate through these problems and radically change their systems. Some other teams don’t seem to be able to take the bull by its horns, sometimes also because they focus too much on the tests and not enough on the cause of the pain: their code.

So in the end, while I appreciate Feathers point of view and overall his talk, I think that in particular this aspect can be misunderstood and taken as an excuse to test less, so that teams won’t be slowed down. Unfortunately this is just an illusion and when it backfires it hurts.

Information dissonance and TDD

A colleague from Italy asked me for my opinion about this piece from TechCrunch, You’ve Got To Admit It’s Getting Better. This is a post related to the now infamous rant from Ryan Dahl of Node.js fame about the status of software in general, and how different layers of code and components integrate in particular. Rather than an harmonious dance of well balanced and fine tuned libraries, Dahl is right to point out that quite often we work with layers and layers of complexity and leaky abstractions which end to expose, most of all, an annoying dissonant impedance mismatch. While I partly agree with Dahl, I have to admit that also Jon Evans of TechCrunch has a valid point: it’s getting better, not enough probably, but better.

What has caught my attention more is something else though. I’m pretty sure that my friend asked me about this because of some unkind words for TDD. So let me get the record straight:

• First of all, it’s just wrong to say that TDD is “the notion that a development team’s automated tests are even more important than the actual software they write, and should be written first” (Jon Evans himself) => no TDD practitioner in his right mind would describe TDD in that way. In particular, nobody would consider the tests more important than the actual software. So, yes, we TDDers think that testing is important, because for us testing is first of all a design technique and tests are a nice byproduct of this technique, one that allows you to evolve your code without breaking existing behaviour in an uncontrolled way. But none of us would try to say that tests are more important, please! If you write about something that is clearly not your forte, at the very least you should not make such a gross mistake.
• I don’t know where I’ve lived in the last 10 years, but TDD has never “seemed almost sacrosanct” (Jon Evans again). I’d argue that still only a minority of developers actually test their code in these days. TDD practitioners then are a minority of a minority. So, please, spare me the drama about the industry changing its mind about TDD.
• With regards to the two linked posts, I have to confess that I have no idea who are the authors. That doesn’t necessarily mean anything per se, just that I don’t have any prejudice about the authors. I’ve never read anything from them before.
• Pieter Hintjens seems to recommend that rather than testing your code, you should throw it out in the open and have people (users) testing it for you. It seems that his main argument is that, even if you think you know how to test your code, you don’t know if your code is valid in the hands of users. Apart from the fact that it doesn’t seem a very professional approach, this is mostly nonsense to me: testing and validation are two different things. Sure, you need your software to do something useful for your users, otherwise it can be correct but it would be of no purpose at all. So, assuming that your software solves the right problem, how do you know that it solves it correctly? Testing to the rescue. In the end the author even admits that he uses TDD for his APIs…
  Having said that, remember that TDD is first of all about design and evolution. So I think that this point is totally wrong.
• Peter Sargeant rants about dogmas and Agile Testing Experts recommending to always write your tests first. The term “Agile Testing Expert” is funny enough already: there are no such people, there is no agile testing, thank God at least there is one aspect of XP that the Agile crowd has not been able to jeopardize with their brand (oh, they tried, but they also failed miserably – some people cover TDD, BDD, ATDD and other practices together under the umbrella of Agile Testing, but it seems quite fictitious to me and it has never been accepted in the community in that way). So, if you talk about Agile Testing Experts I’m already suspicious. Once again, this post seems to miss the point of TDD: it’s about design and evolution, testing is a collateral, an important one but still a collateral. Do I always write my code following TDD? No, I don’t. I rarely use TDD if I’m spiking or prototyping. I never use TDD for one off pieces of code that won’t survive to see the light of a new day. I very rarely use TDD for pure UI code, especially when it’s about HTML and views and layout and CSS… (you get what I mean I guess). And I never use TDD when I don’t know what I’m doing, when I’m exploring.
  Anyway, Peter Sargeant’s blog is in the end about promoting testing automation, so his rant is more about breaking away from dogmas and using your head to judge when to use a tool, so it doesn’t seem to me to be such a big deal.

In conclusion, yes, things may get a little better and no, I don’t think that there is a whole movement against mainstreamed (!!) TDD in the software industry.

Craftsmanship is not art and we care like craftsmen do

Some rights reserved, by Felix E. Guerrero

I’ve read Dan North post Programming is not a craft and I have to say that I have mixed feelings about it.

I’ve always been against recognizing beauty in software in general and in code in particular. I don’t think that coding is an art and I don’t think that any programmer can be characterized as an artist. Code has no aim beyond solving a problem and providing some utility to a user (where a user can be another system, an organization, etc., not necessarily the end user alone). Good code is code that solves a problem AND it’s easy to change, highly cohesive and decoupled, without duplications, etc. Bad code is code that doesn’t express those qualities. So it would be a stretch to talk about good code in terms of beauty, IMO.

And I agree with a lot more from that post. For example, I like and agree with the analogy of the samurai. And I do actually believe that: “It takes a real expert – a real craftsman – to see the elegant simplicity buried away inside the mess we call enterprise software, for instance, and tease it out.”

Having said that, I also disagree with his characterization of craftsmanship, especially in artistic terms. I think that our work, the way I see it, the way I do it every day, has a lot in common with what master craftsmen used to do in Italy during the Renaissance. And a lot of it clearly wasn’t art at all, it’s just good work well done.

Back in 2002, I had a lot of troubles trying to communicate to friends and family my passion about writing code. Passion is not even the right word, “care” is more appropriate. I do care a lot about the code that gets written. And for me, the idea that code must solve a problem, is a given. Speculative coding is like onanism: you get tired of it very soon. So, taking for granted that code must solve a problem, why do I care about how the problem is solved?

Literature to the rescue. In that period I was reading a book from Erri De Luca, callend Montedidio (by the way, it has been translated in English, I recommend it). A passage just drilled in my brain:

Mast’Errico mi ha messo a stendere il turapori e a carteggiare. Alliscio le ante di un armadio per vestiti. […] Oggi ha provato la chiusura della prima anta e quella ha combaciato così bene che ha fatto il rumore del soffio, l’aria se n’è scappata da dentro. Mi ha fatto mettere la faccia vicina al battente, ho sentito una carezza d’aria. […] Poi mast’Errico l’ha smontato e l’ha coperto, è un’opera importante, aggiusta tutta un’annata. I cassetti sono di faggio, gli incastri a coda di rondine, una soddisfazione vederglieli uscire da sotto le mani. Controlla gli squadri molte volte, ingrassa le guide, sfila e riinfila i cassetti senza rumore, come la lenza in mare, dice, che sale e scende muta in mano a lui. Mast’Errì, dico, siete un fenomeno, un ebanista pescatore.

A rough translation in English would be:

Mast’Errico put me to spread the filler and sanding. I smooth the doors of a wardrobe. […] Today, he tried closing the first door and it matched so well that it made the sound of the breath, the air has escaped from inside. It made me put my face close to the door, I felt a caress of air. […] Mast’Errico then has dismounted it and covered it, it is an important work, it fits an entire year within. The drawers are made of beech, the joints are shaped like dovetails, such a pleasure watching him crafting them with his hands. He checks out the alignments many times, he greases the rails, and he pulls the drawers back and forth without noise, as the line at sea, he said, rising and falling silent in his hand. Mast’Errì, I say, you’re a phenomenon, a cabinetmaker fisherman.

So, this is my take at software craftsmanship almost 9 years ago, recorded in a message to the Italian XP mailing list on 12/3/2002, years before anybody even started talking about it. 
My message back then ended with the following:

A software engineer is a postmodern version of a craftsman. The death of a craftsman can be violent (due to lack of competence, negligence) or because of starvation (overengineering, gold plating). When a technique is overcome a skilled craftsman must reinvent itself or die.

PS: I’ve stopped trying to explain to people my work. In a social context I typically say: “I work with computers”. If I’m really in a good mood I may even say: “I write software”. That’s it. I get less frustrated this way. 

The Technical Debt Trap at Agile Tour Toronto 2010

After Fowler’s keynotes, this is the best session that I have attended. The main reason is that its speaker, Michael Norton, was very good: sharp, clear, concise, as a speaker should be.

The presentation was focused on the concept of Technical Debt, conceived by Ward Cunningham back in ’92 and touched by Fowler as well in his third part keynote.

Norton’s intents are to clarify what the metaphor means and which is the proper way to use it. The point is that it’s ok to cut corners in the short timeframe to gain an opportunity (for example, for a demo), but things have to be fixed up properly as soon as possible immediately afterwards. The power of the metaphor lies primarily in the concept of debt and the need of repaying it:

• Every corner that you cut for a specific purpose, for example in terms of code that you know should be done in a different, better way, is debt
• All the debt adds up very quickly and it slows you down
• If you don’t fix that code potentially you’ll end up in a situation where development slows down to a halt and the system needs to be rebuilt from scratch

We should always remember, however, that this is just a metaphor and we can’t stretch it beyond its intended use. It’s indeed a way to think about our own work, it doesn’t magically replace our work with its own laws because we are leaving in a delusion. Norton uses the term metaphorphosis for this kind of phenomenon. In Cunnigham words:

• “[Many] have explained the debt metaphor and confused it with the idea that you could write code poorly with the intention of doing a good job later.”
• “The ability to pay back debt […] depends upon you writing code that is clean enough to be able to refactor as you come to understand your problem.”
• “Dirty code is to technical debt as the pawn broker is to financial debt. Don’t think you are ever going to get your code back.”

Hence, the important things to remember are:

1. Debt is bad => if you can get away without it, just do it
2. Tech debt is a strategic tool => use it under specific circumstances (eg: I need to change this thing and deploy it in an hour, I’ll refactor it right after the deployment), not as an excuse to do sloppy work
   • Remember Fowler => only the quadrant Deliberate/Prudent is acceptable and only for very short timeframes
3. I’d say that 99.99% of your daily work should be debt free
4. Unintentional dirty code, missing tests, bad choices will happen anyway, that’s why Norton called it cruft => it’s physiological, but it’s not technical debt
   • In my team, for example, we do code reviews also to catch that
   • Intentional cruft is always a bad choice, something that you need to avoid and that when done shows a lack of professionalism
5. You have to deal with cruft every day/hour/minute, don’t let it accumulate, because big incremental fixes don’t work => the complexity keeps growing up at the same rate
6. Clean constantly & follow the Boy Scout Rule: Leave the campground cleaner than you found it (of course campground=code)
7. Monitor your tech debt, also by means of metrics like test coverage, procedural complexity (aka cyclomatic complexity, McCabe number) and cohesion/coupling

I’m not a huge fan of slideshare & viewing slides without attending a presentation, but FWIW Norton has uploaded his slides and they are actually good/helpful. As usual, here you can find my mind map as well.

This is the mind map that I have taken live during Norton's presentation

Fowler’s keynotes at Agile Tour Toronto 2010 (with mind maps!)

On October 20th I attended the AgileTour Toronto 2010 conference with a colleague. Well, calling it “conference” is a bit generous but I’m not unhappy about it. It was warm and cozy. I’ll explain more in details in some other posts.

For me, the big highlight of the day was Martin Fowler‘s keynote. Let me rephrase: Fowler gave 3 keynotes in one, which is nice because it’s not too boring and it flows smoothly.

The essence of Agile

In the first keynote Fowler wanted to clarify his view on Agile, talking about its essence. His motivation was to steer clear of a phenomenon that he calls Semantic Diffusion: when you start being successful at innovating and modulating a message, people try to copy you without deeply understanding what you’re doing, just by mimicking your work and distorting your ideas badly. There’s not a lot that we can do about it, with the exception of reminding people about the core principles at the root of what we are doing.

So he wanted to characterize Agile first of all by difference, talking about traditional plan driven methods. In a plan driven method, success is achieved when you do something according to the plan. This is not absolutely impossible per se, but it requires at the very least that the requirements are stable. The bad news is that outside of very limited and insignificant niches, there is NO such thing (and we all know it very well).

So Agile methods are characterized by adaptive planning, which means continuous planning and release early and often. These qualities together constitute an evolutionary design approach, which I’d prefer to define differently and at the software design level in terms of responsive design, but that’s fine, I can take this.

Fowler also briefly mentioned a quote from the Poppendiecks, which sounds more or less like this: being able to deal with a lot of changes in requirements is a competitive advantage. This is a fairly powerful tool in your hands, so use it wisely.

Another fundamental quality of Agile is its focus on people. We all know that after the industrial revolution the way we intend work has been shaped too much and for too long by Taylor and his stopwatch, the so-called system process. This has greatly slowed down the evolution of knowledge work and it has taken a long time and great minds like Deming and Drucker to correct these problems and show a different path. As Deming said, “A bad process would beat a good person every time”. So Agile methods are characterized by putting that good person at the core of the method and allow her/him to shine and do great things, rather than suffocating her/him with bureaucracy.

Because of this, Agile methods need to be adapted to the specificities of the people and their organization, while maintaining the other characteristics mentioned above (while this is fundamental, it is also another contributor to Semantic Diffusion, but again we can’t do much about it). So, Agile methods are characterized by self adaptivity. As Kent Beck says, “In software development, “perfect” is a verb, not an adjective”. This means that the process is never static, but it is continuously verified and improved, always striving for perfection (but of course never achieving it fully).

This is the mind map that I have taken live during first Fowler's keynote

Continuous integration & delivery

In the second keynote Fowler focused on continually adding value to a system.

He started by describing an experience so common if you have developed software for a living in the previous millennium: if you have worked in an environment that doesn’t adopt continuous integration then you know that a project can be easily doomed, staying in integration mode for a very long time.

The idea is simple:

• Individual developers work on their own code, that is they develop feature independently
• Towards the end of the project they put everything together and of course nothing works

The metaphor is having 2 teams digging a tunnel into a mountain from 2 opposite sides and failing to meet in the middle. I actually have seen something like this happening in big construction work in Italy. It’s scary.

Even when versioning systems became popular, people tended to do feature branching more than anything else:

• Every developer has his/her own branch, defined for the feature that he/she is developing
• Because integration is difficult and painful integrate less often
• Because integration happens less and less, when it actually happens is more difficult and painful

This brings to what Fowler calls big scary merges, where you merge together a lot of files, even through sophisticated tools that do that automatically for you and from a text perspective the code seems good, but what about the semantics? You need tests to tell you whether the merge was good or not. And you have a lot of red bars for a very long time, because no tool can automate a slight change in the semantics of the protocol between two classes, for example.

The amount of pain that a team gets grows exponentially with the amount of time that incurs between integrations. I mean, more pain = more time & more bugs = more costs & fewer opportunities to generate revenues.

In the Agile world, typically when something hurts you wanna do it more often. What does this mean? It means that when something is problematic, we want to make it simpler, less expensive, so rather than hiding our head in the sand and try to procrastinate the pain as much as possible, we prefer to tackle it right away and make it better before it’s too late.

So, we want to do continuous integration many times a day, which is exactly what happens at my main gig (my beautiful day job) and in many other companies. I’d say that continuous integration has become more or less mainstream these days, which is good. For example, yesterday we have integrated our code something like 30 times. Integrating means compiling, building all the artifacts, running various kind of checks and thousands of tests. All this is entirely automated.

A bad consequence of feature branching, by the way, is that it encourages avoiding code/design improvement (because it’s painful). So the code quietly rots, day after day, to the point where it becomes a big steaming pile of PUT HERE YOUR FAVOURITE EXPRESSION. Been there, done that, never want to do it anymore in my life.

En passant, Fowler recommended Paul Duvall book on continuous integration.

Towards the end of this part Fowler spoke about continuous delivery, which is a big topic. For the non technical people out there the basic idea is to stretch continuous integration into the actual deployment. How can you do that? By building a pipeline of machines where:

• The code is automatically compiled and unit tested => this should take less than 10′
• Artifacts are generated into a repository
• Artifacts are taken and deployed in a different environment where all functional/acceptance tests are executed in a timeframe that can be measured in hours for complicated systems
• The next stage is typically the one where performance tests are executed to ensure that the system is still executing in an acceptable way and again it can take hours, even more than the previous stage
• If everything is fine the code can be automatically deployed in production in a controlled way, where you do smoke tests to see that nothing is horribly broken

Fowler emphasized that while the last step can be typically controlled manually, and it may vary a lot from org to org (eg: you can’t do it automatically if you have to ask for permission to a committee in a bureaucratic organization), the infrastructure should be in place to do it automatically, with a simple click of a button. As Fowler said: “There should be no technical reasons impeding an automated deployment, while it’s perfectly reasonable that there are business reasons to batch many commits in a single deployment controlled manually”.

Of course Fowler recommended Jez Humble book.

This is the mind map that I have taken live during second Fowler's keynote

A sketch of continuous delivery pipeline depicted by Fowler

Effort, quality and feature development

This third part can be summarized with the following quote: We need to put less effort on quality to put more effort on feature development

At a first glance, this seems sacrilegious, but if you think about it, it doesn’t talk about results, it talks about effort. Hence, it’s actually good to achieve high quality without putting into it much effort, so that we can concentrate on devoting this effort to more labour intensive feature development. Or anyway let’s put effort where our labour is more likely to pay us back directly.

Fowler started by mocking Uncle Bob and his preaching attitude towards quality. It was actually very funny and I think that Fowler is right. Nevertheless, in all fairness Uncle Bob makes the point that all effort devoted to quality pays you back in terms of sustainability and efficiency over time. But certainly he deserved to be mocked, even though with a great deal of sympathy.

Fowler wondered whether quality is tradable. He makes the distinction between internal and external quality:

• Internal => It’s not visible, it’s about software design, cohesion and coupling, code, architecture, etc. It’s like the engine of the car, it’s a kind of magic, hidden under the hood, if it’s not in order all of a sudden the car will break and you’ll be left counting on your feet only
• External => It’s the classic concept of quality in terms of visible behaviour or features of a system, what customers and end users experience out of it

Fowler’s point, and I can’t agree more, is that while external quality is tradable, internal quality is not. Now, we should open a big parenthesis here, related to the differences between startups and sustainable development, the cost of missing opportunities, but for now bear with me. Let’s just say that in the long run, internal quality is overall not tradable. I think that we would all agree on that.

Fowler goes on with his Design Stamina Hypothesis, in order to explain what I just briefly synthesized. The graph from Fowler’s article says it all I suppose.

The idea is that beyond a certain critical mass, the only way to have your system growing linearly with your effort is to keep things in order in the house. Otherwise, development slows gradually.

Then Fowler talks a little bit about the Technical Debt metaphor, that I will explain more in details when talking about another session. Just think that every bit of cruft that cripples into the system is going to sediment and accumulate and you’ll have to pay it back like accrued interest on the principal. It’s a very powerful metaphor that explains well certain phenomenons happening on aging code bases.

Lastly Fowler put up a nice table that tries to classify the different ways in which a team can get in Technical Debt.

	Reckless	Prudent
Deliberate	no time for design	ship now and deal with consequences
Inadvertent	what is layering?	now we know how we should have done it

I think that he was being nice, because I would have used Incompetent instead of Inadvertent. Anyway, the only quadrant where eventually can be temporarily acceptable to be is Deliberate/Prudent, but you don’t wanna stay there for very long. Certainly never Reckless.

So, in the end, the conclusion is that to put more effort into feature development you have to put more continuous effort into internal quality, without compromising in the long run, or this will slow you down more and more and more. I don’t remember if Fowler explicitly said that but that was clearly the whole point.

This is the mind map that I have taken live during third Fowler's keynote

To recap, an enjoyable keynote, or series of keynotes if you prefer, delivered by Fowler in a very pleasant way. For me, they keynote alone was well worth attending the conference.

Oh, does your method include poaching eggs blindfold and with one hand tied behind your back?

Photo by Fyffe

Recently, Ron Jeffries blogged about Scrum and the Scrum Alliance in Scrum is OK. Let’s get to work! => while I agree with the spirit of his post I think that Ron is too indulgent with certain Scrummers and the Scrum Alliance in particular. In any case, this post is not about that.

This post is about something that Ron’s post made me remember. Specifically, the idea of inclusiveness. Some years ago, the illuminated CEO of a software company hired me, impressed by the sense of novelty that most of the things about XP were inspiring at that time (and they still do, to some extent). We were green, surely naive, but genuine and we put all ourselves into it.

As usual, people matter more than anything.  Some colleagues were surprisingly positive and cooperative. Some other a huge disappointment. To my surprise. As I’ve just said, I was green. There was a guy in particular, not so young anymore, who was bullying anybody around and who kept making pointless observations about the fact that XP wasn’t covering explicitly a specific situation that one can encounter under some not so common circumstances. My replies were always that, no, XP in fact wasn’t addressing that problem as stated, but by using some logic and referring XP’s values and principles I was always able to come up with some decent strategy. I tried to talk about adaptation and positive/negative feedback. We could have started from that, if that was a real discussion, a genuine attempt to discover something together. But of course it was not. The guy was just trying to put me down. The irony is that this was a person who couldn’t describe his work in any sensible way, and for which the most sophisticated admissible software method was just slam some code together and we’ll figure it out. Yeah, right.

I guess that this story teaches a few things:

• Don’t feed the trolls (have I mentioned already that I was green?) => this seems obvious but especially in an organization where you start sensing hostility, not only you have to pick your fights but also your genuine discussions
• There will always be somebody trying to find some holes in your proposals and ideas, some aspects that what you’re proposing doesn’t cover => dig a little bit, to check if you’re dealing with a troll or if behind some roughness there’s an inquiring mind, but don’t waste too much time

In the end, colleagues are a bit like customers, and there will always be somebody who’s not happy with what you do. It’s the price to pay for innovation, whose alternative is stagnation. Especially when you’re trying to change an organization, the percentage of unhappy people may be significant, sometimes the majority. That’s ok, it’s part of the game. As they say, change the organization, or change organization.

Guide to Test-Driven Development on ThinkCode.TV

It is done: a few minutes ago the English catalog of ThinkCode.TV has gone live! We are 2 days later on our original plan (oh boy, you have no idea of the amount of glitches, 3rd party service disruptions, personal tragedies, Murphy’s law manifestations… that we have incurred into in the last few weeks), but at this point it doesn’t matter anymore. What matters is that the catalog is out and we can finally get feedback outside of the Italian technical community.

While some videos are derived from our Italian catalog, translated and narrated by an English mother-tongue speaker (David B. Harris, who happens to be a colleague, a good friend and an outstanding author as well), there is some original content that is not available in the Italian catalog too. For example, Renzo Borgatti has updated his screencast about MacRuby and HotCocoa to the latest version and David, as hinted above, has started his course about FOSS Document Management tools.

We are proud to offer subtitles for every video. I personally know English-speaking geeks who like subtitles in any case, at the very least because they can be searched and they can provide an alternative way to browse a video. On the other hand, we are convinced that a lot of people who are not too comfortable with Enligh-only content will benefit from the possibility to read the subtitles to catch even the tiniest detail.

With regards to my stuff, the first lesson of the Guide to Test-Driven Development is available. I’m very excited to see what the reaction of the community will be. Only in the last few days we are starting to see some excellent screencasts popping here and there, but nothing yet to my knowledge that resembles a complete approach to TDD like this course. The Italian version of the guide is one of the bestsellers of the Italian catalog and I hope that it will go well in English too. There are currently 3 lessons available in Italian (with the 4th one coming very soon), so you can expect a steady flow of lessons.

There are a lot of other screencasts that I’d like to produce, touching mainly on topics like OOP, design, refactoring and testing. So many things to do, so little time. 🙂

Coming soon, apart from continuing on the courses that we have started, we will have a great introduction to HTTP, an excellent course on Smalltalk and an interesting screencast covering how to upgrade a webapp to upcoming Rails 3.0 release. So, enjoy, give us feedback, and stay tuned!

If you like Python or interested into it, please grab a “bite”

I’ve been extremely quiet in the last months, but for a good reason: together with some friends we have founded ThinkCode Labs Inc., a Canadian startup whose first project is ThinkCode.TV. Our mission is to deliver high quality technical screencasts, mostly about programming themes.

We have launched on the Italian market in November 2009 and it has been quite a success and a hell of a ride! Now we are on the verge to launch in the English speaking market as well. We will launch on April 19th.

I’m producing an 11 lessons course on TDD which is doing great with the Italian public. I’m touching on subjects like characterizing TDD’s microcycle, finding the next test, dependency injection & mock objects, responsive design, BDD and dealing with aspects like legacy code, handling time & databases. I can’t wait to see which reaction we will get with English speakers.

In the meantime, we are offering a free appetizer for anybody interested: a fantastic 19′ video about implementing a maze solver in Python. You can grab it here: please, let us know what you think about it and if you like it, by all means, spread the news.

Our opposable thumb makes us all professional mechanics, doesn’t it?!?

Duct tape on Apollo 17, courtesy of Nasa

After watching the video of Uncle Bob’s Clean Code presentation at NDC 2009 last Friday, I’ve thought a bit about the paradox in which our profession lives, especially in light of the recent debate around testing ignited collaterally once again by Joel Spolsky’s blog post. This  paradox is interesting, because it is multifaceted.

Facet 1: most of the companies prefer to deal with many cheap, sloppy programmers instead of investing in a few very good programmers.

Facet 2: despite all the knowledge about software development is easily available a) on the internet b) in multiple ways and c) at a very convenient price (if not free), most of the developers don’t learn much after the university and don’t take any pride in exercising their profession.

Facet 1 may be a side effect of the whole “IT Doesn’t Matter” phenomenon: if investing in IT doesn’t give you anymore a competitive advantage, then maybe it’s not worth investing in the best developers you can find, but just in “developers”. While I generally agree with Carr, especially with regards to “simple” tasks like Office Automation or building corporate websites, there are still huge areas of inefficiency where IT can play a significant role. Maybe it will not give you a significant advantage over your competitors, but it can still save you a lot of money. I personally know a few examples in the logistics domain, but unfortunately I cannot disclose any details because of existing NDAs. Let’s just say that we are talking about many millions of dollars of savings because of problems that in theory have been solved already at the beginning of the nineties. IT wouldn’t matter, if only it was applied and used consistently!

I have also experienced personally the 10x productivity factor: I have seen sloppy programmers being 10x slower than good programmers (not exceptional programmers, just good ones), whose salary wasn’t even nearly 10x bigger than the former ones (let’s be optimistic, let’s talk about 30-40% more). So, it still seems kind of dumb to resolve to hire mediocre people, especially when you are a medium-large company: you certainly have bigger capacity to absorb inefficiencies, but those numbers are starting to be so big, that unbelievable amounts of money go down the drain just because of this. Having said that, there are not nearly enough good programmers in the world and it seems that HR and head hunters have just given up. Sidenote: I happen to know many good programmers and all of them are having a successful career, so it seems that there is always a need for good programmers, at least in this continent. Good programmers, people who care, please don’t give up!

Is there a cause-effect relationship between 1 and 2? I don’t think so. It seems simply that there is a bit of pressure to level software development discipline. I’m sure that this is not the intent of Joel Spolsky, quite the contrary, but singing an ode to the duct tape programmer, even when the programmer is a good one, doesn’t seem the smartest idea to me. Put duct tape in the hands of an exceptional programmer and he’ll do great things. Put duct tape in the hands of a sloppy programmer and he’ll do an awful job. Have you ever seen a participant to Canada’s Worst Handyman assembling a shelf?

For some reasons, everybody thinks he/she can program, even without the appropriate knowledge or background. After some practice in Excel, everybody can certainly write VBA macros (and do a terrible job anyway), but it’s like saying that because everybody can use a wrench then we are all professional mechanics. I guess it’s all about perception, so a lot of people oversimplify the complexity behind implementing and maintaining a code base for any significantly complicated business problem. I’m not saying that every software programmer needs to be as good as Ward Cunningham, but in my mind there is a clear trend towards embracing more and more people in the industry with less and less capable minds. Even worse than that: a lot of programmers seem simply to not care at all.

What is wrong about letting more people get into software development? Generally speaking, nothing – assuming that the complexity at hand has been reduced over the years, thanks to astonishing advancements in tools and technologies. To some degrees this is true: the things that we can do today would have been virtually impossible twenty years ago, but during the same period of time the industry has just raised the bar, tackling more and more complicated problems. The revolution that started with personal computing isn’t just done yet.

Hence, we need smart people, willing to learn new things every day, constantly looking for ways to improve their work. It’s like having your legs solidly planted in today’s problems, but your eyes are always looking at tomorrow.

Duct tape is an exceptional tool when you need to fix something quickly and for whatever reason you don’t have access to the appropriate tool, knowledge, etc. Maybe sometimes you leave duct tape in place for years (hell, my plastic pipe that conduct warm and humid air from the dryer outside of my house sticks into its place thanks to duct tape), but this is generally not a good solution, capable of lasting for a long time. Sooner or later somebody will have to come and clean the mess that the duct tape expert has left. Two, three times a year I have to replace the duct tape because it simply wears out over time: it’s annoying and I would have fixed it already if I wasn’t renting my house.

There may be specific situations where this is tolerable, but not in much broader terms. And yet, there seems to be a tendency to underestimate the effects over time of such short-sighted decisions in our profession. Is it a tolerance to mediocrity? Are we all getting dumber?

Think about it this way: would you accept to have a sloppy pediatrician taking care of your child’s health? Would you go into the surgery room with a mediocre surgeon? Would you put your life on the line in a judicial trial with an incompetent lawyer? Would you trust an ignorant plumber to do the plumbing of your house? Etc., the examples are endless. We are not talking about NASA-like situations – these are more or less very common problems.

The point is that you would NOT. And yet, many managers and entrepreneurs, even when they are technical, do with software development what they would not do in their personal lives. Is it because it’s not their personal wealth or beloved ones at stake?

I’ve always found that the best managers and entrepreneurs are the ones which know the difference between cost and investment. There is no return from a cost, but there better be from investments. If you are on the upper side of the food chain, which return do you expect from the developers working for you? How can you maximize it? But if you are on the lower side: how can you provide value to your company? How can you be always on top of your profession?

Big in Japan

Last Wednesday I’ve attended one of Thoughtworks Quarterly Technology Briefings in Toronto, Ontario. The presentation was about an introduction to Lean concepts and how they can apply in IT world. So, concepts like avoid waste, practice continuous improvement, etc. Most of the Lean ideas come from Taiichi Ohno and William Deming between the 50’s and the 80’s.

I’ve put here a scan of the mind maps that I’ve drawn live during the presentation, followed by some of the key ideas highlighted there. Take the maps as they are, probably imprecise and rough, very personal (they are indeed related to my mind and my experience during the presentation), but I hope that they can communicate the essence of the presentation. I think I’ll definitely consider attending future events like this, it was pretty good (and breakfast was great! ).

Scan of mind maps on index cards

The event has been recorded by infoQ guys, so I expect to see its video live on infoq.com sooner or later.

In the end the presentation has been articulated in 3 parts:

1. First of all a bit of history about Lean, highlighting differences between mass-production approaches like Fordism (influenced by the infamous Taylor) and quality-oriented approaches like the Toyota Production System (Taiichi Ohno and William Deming are the main influences)
   • In a tayloristic world, workers are stupid and cheap, don’t care about things and be good only at follow instructions (if disciplined). Only the managers know better.
   • Deming’s approach is totally different, workers are empowered and involved in all aspects of production. I can see a lot of influence from Drucker as well: the modern worker outside of Mac jobs is definitely a knowledge worker even if he/she works in a manufacturing plant. I’d say that Drucker and Deming are part of the same ecosystem.
   • Of course Ford is doomed (is it?) and Toyota, while in trouble, seems to have a much better perspective and it’s expected to recover much faster once the economy starts to grow again.
     • The underlying, oversimplified message is: “This is all because of Lean”
2. Then they did an introduction of Lean concepts. Some key ideas:
   • Kaizen is the continuous improvement component, delivered in small increments, focusing on the worst problem first
   • Lean systems are Pull systems, that is systems in which inventory is minimized not only as a whole, but also locally ==> nothing is produced unless required
   • Wastes in all shapes and forms are continuously removed
   • There’s a high degree of autonomation (Jidoka), that is the capability to automatically detect defects such that the line can be stopped and the workers can work on solving the problem (permanently)
   • There are 7+1 types of wastes
     1. Overproduction
     2. Waiting
     3. Unnecessary Trasportation
     4. Over-Processing
     5. Excess Inventory
     6. Unnecessary Movement
     7. Defects
     8. Unused Employee Creativity
   • A system of problem solving techniques like “go & see for yourself”, “ask WHY 5 times”, etc.
   • Anecdote: how many times on average the line is stopped at Toyota? 27000 times a day ==> it may seem counterproductive, but the more the line is stopped, the better the quality, the greater the output (of course after an initial decrease in productivity ==> this should tell us something, shouldn’t it?!? )
3. The last part was focused on transposing these ideas in the IT/sw dev world. I think that in the XP community we have already embraced these ideas and we take them for granted, more or less, but it was interesting nevertheless to see them applied in practice, rooting them into Lean concepts:
   • For example, a ceremonious waterfall-ish process has shown a Cycle Time Efficiency of 53%, where CTE is equal to the Total Value-Added Time / Cumulative Elapsed Time
     • By reducing handoffs, separate validation phases and anticipating testing activities CTE has been brought up to 78%
   • In the sw dev world:
     • Overproduction = Extra Features
     • Over-Processing = Gold Plating
     • Excess Inventory = Partially Completed Work

Of course Lean concepts and Agile/XP go hand in hand. For example, what about the relationship between stopping the line and fixing the build when is broken? A continuous build system is an example of autonomation.

Lots of good quotes and book recommendations in the presentation. I’ve found an older version very close to the one that I have seen. It’s available here. Let’s see if Thoughtworks will publish the final version soon.