Posts Tagged ‘toronto’

The Technical Debt Trap at Agile Tour Toronto 2010

Monday, November 22, 2010

After Fowler’s keynotes, this is the best session that I have attended. The main reason is that its speaker, Michael Norton, was very good: sharp, clear, concise, as a speaker should be.

The presentation was focused on the concept of Technical Debt, conceived by Ward Cunningham back in ’92 and touched by Fowler as well in his third part keynote.

Norton’s intents are to clarify what the metaphor means and which is the proper way to use it. The point is that it’s ok to cut corners in the short timeframe to gain an opportunity (for example, for a demo), but things have to be fixed up properly as soon as possible immediately afterwards. The power of the metaphor lies primarily in the concept of debt and the need of repaying it:

  • Every corner that you cut for a specific purpose, for example in terms of code that you know should be done in a different, better way, is debt
  • All the debt adds up very quickly and it slows you down
  • If you don’t fix that code potentially you’ll end up in a situation where development slows down to a halt and the system needs to be rebuilt from scratch

We should always remember, however, that this is just a metaphor and we can’t stretch it beyond its intended use. It’s indeed a way to think about our own work, it doesn’t magically replace our work with its own laws because we are leaving in a delusion. Norton uses the term metaphorphosis for this kind of phenomenon. In Cunnigham words:

  • “[Many] have explained the debt metaphor and confused it with the idea that you could write code poorly with the intention of doing a good job later.”
  • “The ability to pay back debt [...] depends upon you writing code that is clean enough to be able to refactor as you come to understand your problem.”
  • Dirty code is to technical debt as the pawn broker is to financial debt. Don’t think you are ever going to get your code back.”

Hence, the important things to remember are:

  1. Debt is bad => if you can get away without it, just do it
  2. Tech debt is a strategic tool => use it under specific circumstances (eg: I need to change this thing and deploy it in an hour, I’ll refactor it right after the deployment), not as an excuse to do sloppy work
    • Remember Fowler => only the quadrant Deliberate/Prudent is acceptable and only for very short timeframes
  3. I’d say that 99.99% of your daily work should be debt free
  4. Unintentional dirty code, missing tests, bad choices will happen anyway, that’s why Norton called it cruft => it’s physiological, but it’s not technical debt
    • In my team, for example, we do code reviews also to catch that
    • Intentional cruft is always a bad choice, something that you need to avoid and that when done shows a lack of professionalism
  5. You have to deal with cruft every day/hour/minute, don’t let it accumulate, because big incremental fixes don’t work => the complexity keeps growing up at the same rate
  6. Clean constantly & follow the Boy Scout RuleLeave the campground cleaner than you found it (of course campground=code)
  7. Monitor your tech debt, also by means of metrics like test coverage, procedural complexity (aka cyclomatic complexity, McCabe number) and cohesion/coupling

I’m not a huge fan of slideshare & viewing slides without attending a presentation, but FWIW Norton has uploaded his slides and they are actually good/helpful. As usual, here you can find my mind map as well.

Mind map of Norton's presentation

This is the mind map that I have taken live during Norton's presentation

Fowler’s keynotes at Agile Tour Toronto 2010 (with mind maps!)

Thursday, November 18, 2010

On October 20th I attended the AgileTour Toronto 2010 conference with a colleague. Well, calling it “conference” is a bit generous but I’m not unhappy about it. It was warm and cozy. I’ll explain more in details in some other posts.

For me, the big highlight of the day was Martin Fowler‘s keynote. Let me rephrase: Fowler gave 3 keynotes in one, which is nice because it’s not too boring and it flows smoothly.

The essence of Agile

In the first keynote Fowler wanted to clarify his view on Agile, talking about its essence. His motivation was to steer clear of a phenomenon that he calls Semantic Diffusion: when you start being successful at innovating and modulating a message, people try to copy you without deeply understanding what you’re doing, just by mimicking your work and distorting your ideas badly. There’s not a lot that we can do about it, with the exception of reminding people about the core principles at the root of what we are doing.

So he wanted to characterize Agile first of all by difference, talking about traditional plan driven methods. In a plan driven method, success is achieved when you do something according to the plan. This is not absolutely impossible per se, but it requires at the very least that the requirements are stable. The bad news is that outside of very limited and insignificant niches, there is NO such thing (and we all know it very well).

So Agile methods are characterized by adaptive planning, which means continuous planning and release early and often. These qualities together constitute an evolutionary design approach, which I’d prefer to define differently and at the software design level in terms of responsive design, but that’s fine, I can take this.

Fowler also briefly mentioned a quote from the Poppendiecks, which sounds more or less like this: being able to deal with a lot of changes in requirements is a competitive advantage. This is a fairly powerful tool in your hands, so use it wisely.

Another fundamental quality of Agile is its focus on people. We all know that after the industrial revolution the way we intend work has been shaped too much and for too long by Taylor and his stopwatch, the so-called system process. This has greatly slowed down the evolution of knowledge work and it has taken a long time and great minds like Deming and Drucker to correct these problems and show a different path. As Deming said, “A bad process would beat a good person every time”. So Agile methods are characterized by putting that good person at the core of the method and allow her/him to shine and do great things, rather than suffocating her/him with bureaucracy.

Because of this, Agile methods need to be adapted to the specificities of the people and their organization, while maintaining the other characteristics mentioned above (while this is fundamental, it is also another contributor to Semantic Diffusion, but again we can’t do much about it). So, Agile methods are characterized by self adaptivity. As Kent Beck says, “In software development, “perfect” is a verb, not an adjective”. This means that the process is never static, but it is continuously verified and improved, always striving for perfection (but of course never achieving it fully).

Mind map of first Fowler's keynote

This is the mind map that I have taken live during first Fowler's keynote

Continuous integration & delivery

In the second keynote Fowler focused on continually adding value to a system.

He started by describing an experience so common if you have developed software for a living in the previous millennium: if you have worked in an environment that doesn’t adopt continuous integration then you know that a project can be easily doomed, staying in integration mode for a very long time.

The idea is simple:

  • Individual developers work on their own code, that is they develop feature independently
  • Towards the end of the project they put everything together and of course nothing works

The metaphor is having 2 teams digging a tunnel into a mountain from 2 opposite sides and failing to meet in the middle. I actually have seen something like this happening in big construction work in Italy. It’s scary.

Even when versioning systems became popular, people tended to do feature branching more than anything else:

  • Every developer has his/her own branch, defined for the feature that he/she is developing
  • Because integration is difficult and painful integrate less often
  • Because integration happens less and less, when it actually happens is more difficult and painful

This brings to what Fowler calls big scary merges, where you merge together a lot of files, even through sophisticated tools that do that automatically for you and from a text perspective the code seems good, but what about the semantics? You need tests to tell you whether the merge was good or not. And you have a lot of red bars for a very long time, because no tool can automate a slight change in the semantics of the protocol between two classes, for example.

The amount of pain that a team gets grows exponentially with the amount of time that incurs between integrations. I mean, more pain = more time & more bugs = more costs & fewer opportunities to generate revenues.

In the Agile world, typically when something hurts you wanna do it more often. What does this mean? It means that when something is problematic, we want to make it simpler, less expensive, so rather than hiding our head in the sand and try to procrastinate the pain as much as possible, we prefer to tackle it right away and make it better before it’s too late.

So, we want to do continuous integration many times a day, which is exactly what happens at my main gig (my beautiful day job) and in many other companies. I’d say that continuous integration has become more or less mainstream these days, which is good. For example, yesterday we have integrated our code something like 30 times. Integrating means compiling, building all the artifacts, running various kind of checks and thousands of tests. All this is entirely automated.

A bad consequence of feature branching, by the way, is that it encourages avoiding code/design improvement (because it’s painful). So the code quietly rots, day after day, to the point where it becomes a big steaming pile of PUT HERE YOUR FAVOURITE EXPRESSION. Been there, done that, never want to do it anymore in my life.

En passant, Fowler recommended Paul Duvall book on continuous integration.

Towards the end of this part Fowler spoke about continuous delivery, which is a big topic. For the non technical people out there the basic idea is to stretch continuous integration into the actual deployment. How can you do that? By building a pipeline of machines where:

  • The code is automatically compiled and unit tested => this should take less than 10′
  • Artifacts are generated into a repository
  • Artifacts are taken and deployed in a different environment where all functional/acceptance tests are executed in a timeframe that can be measured in hours for complicated systems
  • The next stage is typically the one where performance tests are executed to ensure that the system is still executing in an acceptable way and again it can take hours, even more than the previous stage
  • If everything is fine the code can be automatically deployed in production in a controlled way, where you do smoke tests to see that nothing is horribly broken

Fowler emphasized that while the last step can be typically controlled manually, and it may vary a lot from org to org (eg: you can’t do it automatically if you have to ask for permission to a committee in a bureaucratic organization), the infrastructure should be in place to do it automatically, with a simple click of a button. As Fowler said: “There should be no technical reasons impeding an automated deployment, while it’s perfectly reasonable that there are business reasons to batch many commits in a single deployment controlled manually”.

Of course Fowler recommended Jez Humble book.

Mind map of second Fowler's keynote

This is the mind map that I have taken live during second Fowler's keynote

Continuous delivery pipeline

A sketch of continuous delivery pipeline depicted by Fowler

Effort, quality and feature development

This third part can be summarized with the following quote: We need to put less effort on quality to put more effort on feature development

At a first glance, this seems sacrilegious, but if you think about it, it doesn’t talk about results, it talks about effort. Hence, it’s actually good to achieve high quality without putting into it much effort, so that we can concentrate on devoting this effort to more labour intensive feature development. Or anyway let’s put effort where our labour is more likely to pay us back directly.

Fowler started by mocking Uncle Bob and his preaching attitude towards quality. It was actually very funny and I think that Fowler is right. Nevertheless, in all fairness Uncle Bob makes the point that all effort devoted to quality pays you back in terms of sustainability and efficiency over time. But certainly he deserved to be mocked, even though with a great deal of sympathy.

Fowler wondered whether quality is tradable. He makes the distinction between internal and external quality:

  • Internal => It’s not visible, it’s about software design, cohesion and coupling, code, architecture, etc. It’s like the engine of the car, it’s a kind of magic, hidden under the hood, if it’s not in order all of a sudden the car will break and you’ll be left counting on your feet only
  • External => It’s the classic concept of quality in terms of visible behaviour or features of a system, what customers and end users experience out of it

Fowler’s point, and I can’t agree more, is that while external quality is tradable, internal quality is not. Now, we should open a big parenthesis here, related to the differences between startups and sustainable development, the cost of missing opportunities, but for now bear with me. Let’s just say that in the long run, internal quality is overall not tradable. I think that we would all agree on that.

Fowler goes on with his Design Stamina Hypothesis, in order to explain what I just briefly synthesized. The graph from Fowler’s article says it all I suppose.

The idea is that beyond a certain critical mass, the only way to have your system growing linearly with your effort is to keep things in order in the house. Otherwise, development slows gradually.

Then Fowler talks a little bit about the Technical Debt metaphor, that I will explain more in details when talking about another session. Just think that every bit of cruft that cripples into the system is going to sediment and accumulate and you’ll have to pay it back like accrued interest on the principal. It’s a very powerful metaphor that explains well certain phenomenons happening on aging code bases.

Lastly Fowler put up a nice table that tries to classify the different ways in which a team can get in Technical Debt.

Reckless Prudent
Deliberate no time for design ship now and deal with consequences
Inadvertent what is layering? now we know how we should have done it

I think that he was being nice, because I would have used Incompetent instead of Inadvertent. Anyway, the only quadrant where eventually can be temporarily acceptable to be is Deliberate/Prudent, but you don’t wanna stay there for very long. Certainly never Reckless.

So, in the end, the conclusion is that to put more effort into feature development you have to put more continuous effort into internal quality, without compromising in the long run, or this will slow you down more and more and more. I don’t remember if Fowler explicitly said that but that was clearly the whole point.

Mind map of third Fowler's keynote

This is the mind map that I have taken live during third Fowler's keynote

To recap, an enjoyable keynote, or series of keynotes if you prefer, delivered by Fowler in a very pleasant way. For me, they keynote alone was well worth attending the conference.

Are all hardcore developers in Canada disappeared?

Wednesday, December 2, 2009

A couple of days ago I was talking with my boss, who is the VP of development at my day job (which I will not disclose, while my night job is thinkcode.tv).
We were talking about accents. I’m the first non-native English speaker of the company and despite being fluent in English, of course my accent can still be recognized (even though people typically don’t think I’m Italian, which is kind of funny).

Anyway, he told me that none of the persons replying to the job post in Spring 2008 that brought to my employment was actually a native English speaker. I was in shock. The posting was kind of hardcore and a bit silly at the same time, so I understand that not everybody could have been attracted by it, especially if you want to work in a typical big enterprise setup. But the idea that there were no Canadian-born technologists in Toronto willing to even reply to it and being able to survive a simple resume screening is kind of disarming.

Where have all the technologists in the GTA gone? Where are the hardcore developers? Have they maybe transitioned to a managerial path and they have totally lost the technical side of the job? Natively English-speaking software architects, development managers, XP coaches… where are you? Are you all doing consulting?

Hard to say. Certainly one of the things which motivated my decision to leave Italy was the increasing difficulty to find interesting technical jobs that could afford to pay my admittedly good salary (for Italian standard). But here the situation is different: there are still a lot of attractive jobs from a technical perspective, paying good money and yet, no old Canadians interested in doing them?!?

I find it fascinating. Good for immigrants and new Canadians I suppose, but nevertheless wrong. And this happens despite the presence of great Engineering faculties like Waterloo and UofT, just to name the first two that I can think about.

So, once again, where are you?


Follow

Get every new post delivered to your Inbox.

Join 258 other followers