Are all hardcore developers in Canada disappeared?

Wednesday, December 2, 2009

A couple of days ago I was talking with my boss, who is the VP of development at my day job (which I will not disclose, while my night job is thinkcode.tv).
We were talking about accents. I’m the first non-native English speaker of the company and despite being fluent in English, of course my accent can still be recognized (even though people typically don’t think I’m Italian, which is kind of funny).

Anyway, he told me that none of the persons replying to the job post in Spring 2008 that brought to my employment was actually a native English speaker. I was in shock. The posting was kind of hardcore and a bit silly at the same time, so I understand that not everybody could have been attracted by it, especially if you want to work in a typical big enterprise setup. But the idea that there were no Canadian-born technologists in Toronto willing to even reply to it and being able to survive a simple resume screening is kind of disarming.

Where have all the technologists in the GTA gone? Where are the hardcore developers? Have they maybe transitioned to a managerial path and they have totally lost the technical side of the job? Natively English-speaking software architects, development managers, XP coaches… where are you? Are you all doing consulting?

Hard to say. Certainly one of the things which motivated my decision to leave Italy was the increasing difficulty to find interesting technical jobs that could afford to pay my admittedly good salary (for Italian standard). But here the situation is different: there are still a lot of attractive jobs from a technical perspective, paying good money and yet, no old Canadians interested in doing them?!?

I find it fascinating. Good for immigrants and new Canadians I suppose, but nevertheless wrong. And this happens despite the presence of great Engineering faculties like Waterloo and UofT, just to name the first two that I can think about.

So, once again, where are you?

Advertisements

replacer.rb and deleter.rb

Friday, October 30, 2009

From time to time I find myself needing to replace certain strings globally (eg: in a directory tree), or deleting lines based on a match. I find sed extremely annoying and I don’t manage to get around its syntax. So ruby comes in handy!

replacer.rb

exp_search = '<%=launchBean\.getCustomVersion\(\)%>'
exp_replace = '${xsession.info.launchBean.customVersion}'
directory = '**/'
file_mask = '*.jsp'

Dir.glob(directory+file_mask).each do |file|
  File.open(file, 'r+') do |f|
    content = f.read
    if /#{exp_search}/ =~ content
      s = content.gsub(/#{exp_search}/, exp_replace)
      f.rewind
      f.write s
      puts "#{file}: #{content.split(/#{exp_search}/).size-1} matches"
    end
  end
end

deleter.rb

exp_search = 'useBean.*launchBean'
directory = '**/'
file_mask = '*.jsp'

Dir.glob(directory+file_mask).each do |file|
  count = 0
  content = ""
  File.open(file, 'r+') do |f|
    while line = f.gets
      if /#{exp_search}/ =~ line
        count += 1
      else
        content = content << line
      end
    end
  end
  if count > 0
    File.open(file, 'w') do |f|
      f.write content
      puts "#{file}: #{count} lines deleted"
    end
  end
end

These commands are especially useful if you have to refactor a codebase that doesn’t provide direct support with refactoring tools. In this case, I was refactoring a few hundred JSPs of a legacy system.

Credits to RSF & Ruby One-Liners for most of the code & ideas. rcscript seems to be a bit too much for me at this time, I’m perfectly happy modifying replacer.rb and deleter.rb when I need it.

Our opposable thumb makes us all professional mechanics, doesn’t it?!?

Thursday, October 29, 2009
Usage of duct tape

Duct tape on Apollo 17, courtesy of Nasa

After watching the video of Uncle Bob’s Clean Code presentation at NDC 2009 last Friday, I’ve thought a bit about the paradox in which our profession lives, especially in light of the recent debate around testing ignited collaterally once again by Joel Spolsky’s blog post. This  paradox is interesting, because it is multifaceted.

Facet 1: most of the companies prefer to deal with many cheap, sloppy programmers instead of investing in a few very good programmers.

Facet 2: despite all the knowledge about software development is easily available a) on the internet b) in multiple ways and c) at a very convenient price (if not free), most of the developers don’t learn much after the university and don’t take any pride in exercising their profession.

Facet 1 may be a side effect of the whole “IT Doesn’t Matter” phenomenon: if investing in IT doesn’t give you anymore a competitive advantage, then maybe it’s not worth investing in the best developers you can find, but just in “developers”. While I generally agree with Carr, especially with regards to “simple” tasks like Office Automation or building corporate websites, there are still huge areas of inefficiency where IT can play a significant role. Maybe it will not give you a significant advantage over your competitors, but it can still save you a lot of money. I personally know a few examples in the logistics domain, but unfortunately I cannot disclose any details because of existing NDAs. Let’s just say that we are talking about many millions of dollars of savings because of problems that in theory have been solved already at the beginning of the nineties. IT wouldn’t matter, if only it was applied and used consistently!

I have also experienced personally the 10x productivity factor: I have seen sloppy programmers being 10x slower than good programmers (not exceptional programmers, just good ones), whose salary wasn’t even nearly 10x bigger than the former ones (let’s be optimistic, let’s talk about 30-40% more). So, it still seems kind of dumb to resolve to hire mediocre people, especially when you are a medium-large company: you certainly have bigger capacity to absorb inefficiencies, but those numbers are starting to be so big, that unbelievable amounts of money go down the drain just because of this. Having said that, there are not nearly enough good programmers in the world and it seems that HR and head hunters have just given up. Sidenote: I happen to know many good programmers and all of them are having a successful career, so it seems that there is always a need for good programmers, at least in this continent. Good programmers, people who care, please don’t give up!

Is there a cause-effect relationship between 1 and 2? I don’t think so. It seems simply that there is a bit of pressure to level software development discipline. I’m sure that this is not the intent of Joel Spolsky, quite the contrary, but singing an ode to the duct tape programmer, even when the programmer is a good one, doesn’t seem the smartest idea to me. Put duct tape in the hands of an exceptional programmer and he’ll do great things. Put duct tape in the hands of a sloppy programmer and he’ll do an awful job. Have you ever seen a participant to Canada’s Worst Handyman assembling a shelf?

For some reasons, everybody thinks he/she can program, even without the appropriate knowledge or background. After some practice in Excel, everybody can certainly write VBA macros (and do a terrible job anyway), but it’s like saying that because everybody can use a wrench then we are all professional mechanics. I guess it’s all about perception, so a lot of people oversimplify the complexity behind implementing and maintaining a code base for any significantly complicated business problem. I’m not saying that every software programmer needs to be as good as Ward Cunningham, but in my mind there is a clear trend towards embracing more and more people in the industry with less and less capable minds. Even worse than that: a lot of programmers seem simply to not care at all.

What is wrong about letting more people get into software development? Generally speaking, nothing – assuming that the complexity at hand has been reduced over the years, thanks to astonishing advancements in tools and technologies. To some degrees this is true: the things that we can do today would have been virtually impossible twenty years ago, but during the same period of time the industry has just raised the bar, tackling more and more complicated problems. The revolution that started with personal computing isn’t just done yet.

Hence, we need smart people, willing to learn new things every day, constantly looking for ways to improve their work. It’s like having your legs solidly planted in today’s problems, but your eyes are always looking at tomorrow.

Duct tape is an exceptional tool when you need to fix something quickly and for whatever reason you don’t have access to the appropriate tool, knowledge, etc. Maybe sometimes you leave duct tape in place for years (hell, my plastic pipe that conduct warm and humid air from the dryer outside of my house sticks into its place thanks to duct tape), but this is generally not a good solution, capable of lasting for a long time. Sooner or later somebody will have to come and clean the mess that the duct tape expert has left. Two, three times a year I have to replace the duct tape because it simply wears out over time: it’s annoying and I would have fixed it already if I wasn’t renting my house.

There may be specific situations where this is tolerable, but not in much broader terms. And yet, there seems to be a tendency to underestimate the effects over time of such short-sighted decisions in our profession. Is it a tolerance to mediocrity? Are we all getting dumber?

Think about it this way: would you accept to have a sloppy pediatrician taking care of your child’s health? Would you go into the surgery room with a mediocre surgeon? Would you put your life on the line in a judicial trial with an incompetent lawyer? Would you trust an ignorant plumber to do the plumbing of your house? Etc., the examples are endless. We are not talking about NASA-like situations – these are more or less very common problems.

The point is that you would NOT. And yet, many managers and entrepreneurs, even when they are technical, do with software development what they would not do in their personal lives. Is it because it’s not their personal wealth or beloved ones at stake?

I’ve always found that the best managers and entrepreneurs are the ones which know the difference between cost and investment. There is no return from a cost, but there better be from investments. If you are on the upper side of the food chain, which return do you expect from the developers working for you? How can you maximize it? But if you are on the lower side: how can you provide value to your company? How can you be always on top of your profession?

Running JRuby with Terracotta

Tuesday, October 27, 2009

Lately I have been doing some interesting work integrating Terracotta in our application. We use Terracotta in a very limited scenario: as a tool to implement asynchronous DB replication on PostgreSQL (unfortunately the existing tools out there are not suitable for the kind of usage that we do of PostgreSQL – anyways, I’ll talk more in details about this in further posts).
At a certain point in time we decided to drive some java code with JRuby, in order to process information like log files more efficiently. The java code needed to integrate with Terracotta, so we have had to modify jruby scripts in the following way (eg.: jruby_tc.bat on Windows):

@echo off
rem ---------------------------------------------------------------------------
rem jruby.bat - Start Script for the JRuby Interpreter
rem
rem for info on environment variables, see internal batch script _jrubyvars.bat

setlocal

rem Terracotta
set TC_INSTALL_DIR="C:\Program Files\terracotta\terracotta-3.0.1"
set TC_CONFIG_PATH="C:\whatever\tc-config.xml"
call %TC_INSTALL_DIR%\bin\dso-env.bat -q
set JAVA_OPTS=%JAVA_OPTS% %TC_JAVA_OPTS%

rem Sometimes, when jruby.bat is being invoked from another BAT file,
rem %~dp0 is incorrect and points to the current dir, not to JRuby's bin dir,
rem so we look on the PATH in such cases.
IF EXIST "%~dp0_jrubyvars.bat" (set FULL_PATH=%~dp0) ELSE (set FULL_PATH=%~dp$PATH:0)

call "%FULL_PATH%_jrubyvars.bat" %*

if %JRUBY_BAT_ERROR%==0 "%_STARTJAVA%" %_VM_OPTS% -Xbootclasspath/a:"%JRUBY_CP%" -classpath ^
   "%CP%;%CLASSPATH%" -Djruby.home="%JRUBY_HOME%" -Djruby.lib="%JRUBY_HOME%\lib" ^
   -Djruby.shell="cmd.exe" -Djruby.script=jruby.bat org.jruby.Main %JRUBY_OPTS% %_RUBY_OPTS%
set E=%ERRORLEVEL%

call "%FULL_PATH%_jrubycleanup"

rem 1. exit must be on the same line in order to see local %E% variable!
rem 2. we must use cmd /c in order for the exit code properly returned!
rem    See JRUBY-2094 for more details.
endlocal & cmd /d /c exit /b %E%

The relevant differences are in bold in the batch file above. I’ve called the batch jruby_tc.bat in order to differentiate it from usual jruby.bat. As you can see these changes are for Terracotta 3.0.1 but they should work just the same for Terracotta 3.1

Presentations, Tufte and the devil of marketing

Thursday, June 4, 2009

After having heard good things about Edward R. Tufte over the years, I’ve finally put my hands (and eyes) on one of his most famous essays: “The Cognitive Style of PowerPoint: Pitching Out Corrupts Within“.

The basic thesis behind the essay is that PowerPoint is a low bandwidth tool for technical presentations and it shouldn’t be used at all where data is more relevant than mere suggestion. While I don’t disagree with Tufte, I also believe that his ideas need to be put in context.

Tufte’s point is specifically aimed at technical presentations (emphasis mine, but maybe Tufte should have emphasized this a bit more). NASA’s example about the Columbia incident is exactly right to the point: when you need to present the result of a technical analysis so that others can make an informed decision, then I concur entirely that a confused and contradictory presentation poor in data and details is probably not the best choice. Creating a false sense of security is in particular one of the worst things that can be done, especially when dealing with rocket science (literally). The consequences can be catastrophic, to the point where people may die. Actually, they died, which is the saddest, incredible and absurd part of the story.

There are a lot of other situations though where distilling the essence of a system of thoughts can actually be quite effective or even preferable than to provide a deep technical analysis. Let me explain why.

I’ve found myself preparing and handling a few presentations in the last months about themes like the SOLID design principles, pair programming, javascript testing, Selenium, Hudson, Terracotta and others. I’ve experimented a few different formats:

  • Mind maps
  • Google Docs slides
  • Lightning talks with no slide at all, but a decent body of knowledge accumulated in a wiki (specifically, Confluence)

I would divide those presentations in 2 groups:

  1. On one side lie the coaching interventions, where some sort of surprise and dramatization can help a lot to get the message through. SOLID principles talks are definitely part of this category.
  2. On the other side, more technical presentation where the goal was to communicate the essence of certain technical ideas as quickly as possible, with enough reference to deepen the related body of knowledge in autonomy.

The former presentations work better if no details are handed over prior to the presentation itself. The surprise effect is quite important.
The latter presentations can benefit from having the audience reading some sort of a short technical report before. This way, the slideware material can be kept quite light, just a recap of salient points, and there will be more space for discussion during the presentation.

I think that I’m gonna try to prepare a brief report for presentations of the second type. This means of course a little more work on my side, but that is not necessarily the case. I remember having spent hours trying to distill the essence of concepts that would have been expressed more easily in prose. That effort would be gone. Moreover, I will require from the attendees to read the material prior to the presentation, whether it’s on Confluence, in code or any other medium.

I’m curious to see how it will go. Luckily, virtually none of my presentations can be characterized as pitches, so eventually there is space for experimentation without having to sacrifice your soul to the devil of marketing.

Read the rest of this entry »

Big in Japan

Monday, May 18, 2009

Last Wednesday I’ve attended one of Thoughtworks Quarterly Technology Briefings in Toronto, Ontario. The presentation was about an introduction to Lean concepts and how they can apply in IT world. So, concepts like avoid waste, practice continuous improvement, etc. Most of the Lean ideas come from Taiichi Ohno and William Deming between the 50’s and the 80’s.

I’ve put here a scan of the mind maps that I’ve drawn live during the presentation, followed by some of the key ideas highlighted there. Take the maps as they are, probably imprecise and rough, very personal (they are indeed related to my mind and my experience during the presentation), but I hope that they can communicate the essence of the presentation. I think I’ll definitely consider attending future events like this, it was pretty good (and breakfast was great! ).

Scan of mind maps on index cards

Scan of mind maps on index cards

The event has been recorded by infoQ guys, so I expect to see its video live on infoq.com sooner or later.

In the end the presentation has been articulated in 3 parts:

  1. First of all a bit of history about Lean, highlighting differences between mass-production approaches like Fordism (influenced by the infamous Taylor) and quality-oriented approaches like the Toyota Production System (Taiichi Ohno and William Deming are the main influences)
    • In a tayloristic world, workers are stupid and cheap, don’t care about things and be good only at follow instructions (if disciplined). Only the managers know better.
    • Deming’s approach is totally different, workers are empowered and involved in all aspects of production. I can see a lot of influence from Drucker as well: the modern worker outside of Mac jobs is definitely a knowledge worker even if he/she works in a manufacturing plant. I’d say that Drucker and Deming are part of the same ecosystem.
    • Of course Ford is doomed (is it?) and Toyota, while in trouble, seems to have a much better perspective and it’s expected to recover much faster once the economy starts to grow again.
      • The underlying, oversimplified message is: “This is all because of Lean”
  2. Then they did an introduction of Lean concepts. Some key ideas:
    • Kaizen is the continuous improvement component, delivered in small increments, focusing on the worst problem first
    • Lean systems are Pull systems, that is systems in which inventory is minimized not only as a whole, but also locally ==> nothing is produced unless required
    • Wastes in all shapes and forms are continuously removed
    • There’s a high degree of autonomation (Jidoka), that is the capability to automatically detect defects such that the line can be stopped and the workers can work on solving the problem (permanently)
    • There are 7+1 types of wastes
      1. Overproduction
      2. Waiting
      3. Unnecessary Trasportation
      4. Over-Processing
      5. Excess Inventory
      6. Unnecessary Movement
      7. Defects
      8. Unused Employee Creativity
    • A system of problem solving techniques like “go & see for yourself”, “ask WHY 5 times”, etc.
    • Anecdote: how many times on average the line is stopped at Toyota? 27000 times a day ==> it may seem counterproductive, but the more the line is stopped, the better the quality, the greater the output (of course after an initial decrease in productivity ==> this should tell us something, shouldn’t it?!? )
  3. The last part was focused on transposing these ideas in the IT/sw dev world. I think that in the XP community we have already embraced these ideas and we take them for granted, more or less, but it was interesting nevertheless to see them applied in practice, rooting them into Lean concepts:
    • For example, a ceremonious waterfall-ish process has shown a Cycle Time Efficiency of 53%, where CTE is equal to the Total Value-Added Time / Cumulative Elapsed Time
      • By reducing handoffs, separate validation phases and anticipating testing activities CTE has been brought up to 78%
    • In the sw dev world:
      • Overproduction = Extra Features
      • Over-Processing = Gold Plating
      • Excess Inventory = Partially Completed Work

Of course Lean concepts and Agile/XP go hand in hand. For example, what about the relationship between stopping the line and fixing the build when is broken? A continuous build system is an example of autonomation.

Lots of good quotes and book recommendations in the presentation. I’ve found an older version very close to the one that I have seen. It’s available here. Let’s see if Thoughtworks will publish the final version soon.

Read the rest of this entry »

Tweets from Agile 2008

Monday, April 13, 2009

For what is worth, I’ve decided to publish my live tweets from Agile 2008.
Bear in mind, it’s twitter, so it’s meant to be volatile, imprecise, etc. I’ve just posted it to give you an idea.

  • very nice keynote from James Surowiecki, author of “Wisdom of crowds” => http://tinyurl.com/6acndq
    10:05 AM Aug 5th, 2008
  • Compared to other previous European and American editions, the conference is massive!
    10:38 AM Aug 5th, 2008
  • Is following “Test-Driven AJAX” ==> http://tinyurl.com/5magmf (interesting so far, basic but interesting)
    11:05 AM Aug 5th, 2008
  • Speaker has released his own Mock framework for javascript code => http://tinyurl.com/6fpzrv
    11:07 AM Aug 5th, 2008
  • The presentation was good, actually better than my expectation => check it out at http://tinyurl.com/5aczsh
    12:09 PM Aug 5th, 2008
  • “Programming with stars” is a great and funny idea, but execution is really poor. Maybe they’ll get better in the next days
    12:58 PM Aug 5th, 2008
  • Actually the pair doing javascript with continuous testing (Lasse Koskela’s pair) was quite good, even uncle Bob was nice with them… 🙂
    1:21 PM Aug 5th, 2008
  • Has attended “ET by example”: interesting, but I’ll pass for now => http://tinyurl.com/5aym6e
    3:41 PM Aug 5th, 2008
  • Attending 2 experience reports about releasing and deployment (from salesforce and yahoo)
    4:08 PM Aug 5th, 2008
  • Impressive numbers from salesforce.com
    4:20 PM Aug 5th, 2008
  • 40+ scrum teams, 1.1 M users, 3 releases x year, 1 codeline for all
    4:20 PM Aug 5th, 2008
  • The release manager is the scrum master of the release sprint, in a high collaboration env with the teams
    4:31 PM Aug 5th, 2008
  • over 26,000 automated tests that run on every build
    4:32 PM Aug 5th, 2008
  • on top of that there are selenium tests as well
    4:37 PM Aug 5th, 2008
  • imagine breaking the build with 40+ teams involved!
    4:37 PM Aug 5th, 2008
  • Yahoo can do dynamic, automated, realtime re-deployment based on peak observation (neat!)
    5:10 PM Aug 5th, 2008
  • It’s all about heavy automation, speed, dashboards to manage everything and don’t let bad things slip through
    5:31 PM Aug 5th, 2008
  • 4-6 months to transition from a chaotic, uncontrolled env to a fully automated one? don’t buy it, too good to be true
    5:33 PM Aug 5th, 2008
  • Attending “Coaching agile teams” => http://tinyurl.com/5vhwkt
    8:35 AM Aug 6th, 2008
  • The exercise was nice: a laptop screen decorator. Roles: 2 players, 1 coach (only asking questions), 1 observer
    9:16 AM Aug 6th, 2008
  • Nice: the coach brings the outside world to the team and reminds about that
    9:48 AM Aug 6th, 2008
  • The outcome of the workshop will be available at the conference website
    10:01 AM Aug 6th, 2008
  • Attending “Tools, Frameworks and Libraries: the Devil’s work or Manna from Heaven?” (another workshop)
    10:35 AM Aug 6th, 2008
  • Done, not particularly impressed. Just 2 decent recommendations out of it: Hudson & TeamCity
    12:02 PM Aug 6th, 2008
  • 4 pairs left at “programming with the stars”
    12:48 PM Aug 6th, 2008
  • Continuous testing seems the flavour of this year…
    12:53 PM Aug 6th, 2008
  • Belshee & Chen refactored some Fit.Net code (continuously testing). Vote: 26
    12:58 PM Aug 6th, 2008
  • I liked Mike Hill rant about copy&paste: please don’t always start coding with C&P existing code, because that may lead to deep legacy code
    12:59 PM Aug 6th, 2008
  • Koskela & Jacobson have been awesome again. They have changed jsunit own tests, cont. testing at the bottom. Vote: 25, even uncle Bob gave
    81:08 PM Aug 6th, 2008
  • Rainsberger & Brackley did TDD from 0, focused on Eclipse tricks for novices. Vote: 23 (but it was good in terms of showing how to teach)
    1:26 PM Aug 6th, 2008
  • Feathers & Bache refactored some C++ code. Better than yesterday, but still… Vote: 2
    41:31 PM Aug 6th, 2008
  • Bache and Feathers were saved by the crowd… don’t really agree about it, but who cares!
    1:33 PM Aug 6th, 2008
  • Attending Clean code session (uncle Bob book just came out)
    2:05 PM Aug 6th, 2008
  • Clean code: separate construction from use
    2:05 PM Aug 6th, 2008
  • I like it: lazy initialization is bad (for testing, because it messes construction & use, breaks SRP…)
    2:09 PM Aug 6th, 2008
  • Scale up systems on demand: keep it malleable, avoid rigidity (like EJB 1 & 2)
    2:21 PM Aug 6th, 2008
  • It’s so relieving to hear somebody intelligent talking about EJBs in terms of crap (even though EJB 3 is ok and it’s known stuff now)
    2:24 PM Aug 6th, 2008
  • The issue of mixing clean domain code with security, transactions and persistence, but aspects can help (AspectJ or Spring/JBoss AOP)
    2:32 PM Aug 6th, 2008
  • 2 cool citations: BDUF doesn’t give you feedback, you are thinking in a vacuum; hard to throw away when you have invested so much into it
    2:45 PM Aug 6th, 2008
  • Grow the system in response to test pressure
    2:46 PM Aug 6th, 2008
  • Optimize decision making: at the last responsible moment, when the most recent info is available
    2:55 PM Aug 6th, 2008
  • Does the std meet the needs? Use standards wisely, when they add demonstrable value
    3:07 PM Aug 6th, 2008
  • Minimize the mental gap between requirements and code
    3:07 PM Aug 6th, 2008
  • Does the code read like the problem domain? ==> DSLs!
    3:13 PM Aug 6th, 2008
  • Having clean code. well modularized, is an enabler in making a decision at the least possible moment
    3:16 PM Aug 6th, 2008
  • Overall a great implementation. Last version of the slides available at http://tinyurl.com/6mr6qf
    3:17 PM Aug 6th, 2008
  • Attending “Executable User Stories with RSpec and BDD”
    4:06 PM Aug 6th, 2008
  • From CHAOS report: 56% of errors are introduced with requirements gathering, elicitation & specification
    4:07 PM Aug 6th, 2008
  • I don’t like it: BDD starts with a functional test, from the very outside in. This means that BDD resolves to STDD. Bleah…
    4:30 PM Aug 6th, 2008
  • Nice: getters are like horseshit. In a matter of few days a lot of classes will be attracted to it, like flies to horseshit
    4:32 PM Aug 6th, 2008
  • I was hoping to focus more on RSpec Spec, rather than Story. Not much interested in cucumber, a reimpl of Story made by the presenter
    4:57 PM Aug 6th, 2008
  • You can use RSpec::Story or Cucumber to drive Watir => http://wtr.rubyforge.org/ or Celerity => http://celerity.rubyforge.org/
    5:09 PM Aug 6th, 2008
  • Testing java code in RSpec (or Cucumber) by using JRuby is kind of neat
    5:11 PM Aug 6th, 2008
  • Memo for the presenter: close all porn images when you plug your laptop to the projector! 🙂 🙂
    5:25 PM Aug 6th, 2008
  • Overall I’m quite disappointed, but maybe my expectations were too high (and the abstract kind of mislead me)
    5:31 PM Aug 6th, 2008
  • Attending “Agile Architecture IS Possible – You First Have to Believe!” => http://tinyurl.com/5j6fdc
    8:31 AM Aug 7th, 2008
  • Whackamole bug: you fix something here and suddenly something else breaks in a totally different place
    8:42 AM Aug 7th, 2008
  • Rule: No tasks longer than 2 days
    8:45 AM Aug 7th, 2008
  • Offline rollout can be an effective way to gradually migrate a 24/7 live service without disruptions
    8:56 AM Aug 7th, 2008
  • @bbossola maybe: presenter was implying a temporal discontinuity between one fix and a totally unrelated bug… fragility in the large? 🙂
    8:59 AM Aug 7th, 2008
  • Great session, maybe too short (30′ because it’s an experience report)
    9:07 AM Aug 7th, 2008
  • Attending “Technical Debt – How not to ignore it” now => http://tinyurl.com/5bf8vw
    9:07 AM Aug 7th, 2008
  • The presenter is the guy who wrote the excellent “Scrum and XP from the Trenches” => http://tinyurl.com/6zzazx
    9:11 AM Aug 7th, 2008
  • tech debt detection: gut feeling (customers & developers), metrics (code, velocity)
    9:17 AM Aug 7th, 2008
  • Why? 1) short-term thinking (“we’ll fix it later”), 2) pressure (lack of slack & sustainable pace), 3) not talking about it (denial)
    9:18 AM Aug 7th, 2008
  • 4) lack of dev skills, 5) broken window syndrome
    9:19 AM Aug 7th, 2008
  • Prj Management practices like Parkinson’s Law are major causes for tech debt (this is me saying it, not the presenter… 🙂 )
    9:20 AM Aug 7th, 2008
  • Ah nice, another attendee mentions the Parkinson’s Law 🙂
    9:21 AM Aug 7th, 2008
  • The maximum theoretical velocity is diminishing over time
    9:23 AM Aug 7th, 2008
  • To avoid it (or mitigate it): 1) talk about it (with PO too), 2) pull scheduling (taking for granted if you do XP & Scrum right)
    9:27 AM Aug 7th, 2008
  • 3) TDD, 4) put no increased tech debt with the definition of done (me: why not automatic measures on McCabe => build fails if it incr …
    9:29 AM Aug 7th, 2008
  • Another attendee was talking about failing the build if code coverage diminishes => nice!
    9:30 AM Aug 7th, 2008
  • Preferred approach to fix it is incremental improvements => slow down and stop accumulating debt
    9:35 AM Aug 7th, 2008
  • You can’t have a PO working against fixing the tech debt. Major crashes & disasters typically help the biz understanding! 🙂
    9:39 AM Aug 7th, 2008
  • Me: 20% of resources allocated to fixing tech debt. It took months of problems to convince management it was a good idea
    9:40 AM Aug 7th, 2008
  • In order to fix the process retrospectives are the key: root cause analysis, 5 whys, etc.
    9:40 AM Aug 7th, 2008
  • test automation backlog: what tests we are missing and we want to do? prioritize them and maintain in a backlog
    9:43 AM Aug 7th, 2008
  • It’s a nice technique once you have tech debt, but at the same time it institutionalize one aspect of the debt => you need to get out …
    9:54 AM Aug 7th, 2008
  • all in all: tech debt is a powerful metaphor to use with non tech people, stop accumulating it no matter what, fix it incrementally…
    9:56 AM Aug 7th, 2008
  • …with a viable pace (avoid big bang approaches), fix the process (not only the product)
    9:57 AM Aug 7th, 2008
  • Very nice session in the end
    10:01 AM Aug 7th, 2008
  • Attending “A Hundred Days of Continuous Integration” => http://tinyurl.com/554tfc
    10:27 AM Aug 7th, 2008
  • It’s on the proceedings at page 289
    10:31 AM Aug 7th, 2008
  • it’s from M$, amazing! 😉
    10:32 AM Aug 7th, 2008
  • it’s the Service Factory: Modeling edition, a team of ~15 people between Redmond, Holland, Argentina, India…
    10:34 AM Aug 7th, 2008
  • Nice: a distributed env that spans across 18 hours TZs invalidates the concept of nightly build => you are left with continuous integ …
    10:37 AM Aug 7th, 2008
  • But some specific tasks can be executed at specific times (eg: static analysis with NDepend => plug here your java tool if you want)
    10:38 AM Aug 7th, 2008
  • 100 days, 4000 hours of dev time, 551 check ins, 515 builds, 69 build breaks
    10:39 AM Aug 7th, 2008
  • distribution of breaks: 18 compile errors, 19 unit tests, 4 server issues, 27 fxcop (static analysis issues)
    10:43 AM Aug 7th, 2008
  • Worst breaks to fix (up to 9 hours to fix) were related to server issues & unit tests failures (but small time during working hours)
    10:44 AM Aug 7th, 2008
  • on avg 45′ to fix the build
    10:45 AM Aug 7th, 2008
  • Model shows an est. saving of 40% using CI against ideal teams that don’t use CI but never break the build too (using more time to checkin)
    10:50 AM Aug 7th, 2008
  • I like the idea of additional builds for time zone issues (with distributed teams)
    10:51 AM Aug 7th, 2008
  • Lessons: 1) overspec your CI srv. 2) additional builds for TZs, 3) capture all the data, 4) real time analyssi, 5) warnings ARE errors
    10:52 AM Aug 7th, 2008
  • (one warning is a broken window) 🙂
    10:52 AM Aug 7th, 2008
  • 5) don’t check in and go home (especially on a distributed team), 6) you break it so you fix it
    10:53 AM Aug 7th, 2008
  • they weren’t doing any rollback on checkin, because it was hard (heavily dependent on chosen SCM of course, pre commit hooks are the key)
    10:57 AM Aug 7th, 2008
  • ~70% coverage combining UTs and ATs
    10:59 AM Aug 7th, 2008
  • nice session, now “Team pace – Keeping build times down” from TW! => http://tinyurl.com/6j2em7 (page 294 on proc)
    11:01 AM Aug 7th, 2008
  • 2 teams working in java, bubble & squeak – bubble was ~10 people, 4w iterations, build dealing with out & in-container testing + fitnesse
    11:04 AM Aug 7th, 2008
  • team squeak had 2′ build, 12 devs, 16 people, 1w iteration, colocated, customer frequently on site, smaller build with jar,war, set db, ut
    11:06 AM Aug 7th, 2008
  • (team bubble part of a bigger distributed team, ~100)
    11:06 AM Aug 7th, 2008
  • team bubble checked in much less, much slower build time
    11:07 AM Aug 7th, 2008
  • between 760 and 10 checkins per day!! over 9 months (team squeak, while bubble a few per day only)
    11:09 AM Aug 7th, 2008
  • checkin every 15-20 seconds is nuts!!! 🙂 interesting though, I got to give it a try, but my current CVS srv would die miserably 🙂
    11:10 AM Aug 7th, 2008
  • team bubble had a huge resistance to refactor, once they had 6 weeks with no good build!! very unstable builds
    11:11 AM Aug 7th, 2008
  • of course team squeak had a great willingness to refactor
    11:12 AM Aug 7th, 2008
  • team bubble couldn’t cope dealing with the build, so they have created a cross-functional team to deal with it
    11:13 AM Aug 7th, 2008
  • bubble tried parallel test running, websphere was a major problem (5-10′ just to rebuild the profile), a lot of infr problems
    11:14 AM Aug 7th, 2008
  • squeak rarely had build issues, and those that happened were typically fixed in 20-30′
    11:15 AM Aug 7th, 2008
  • typically the build weren’t fixed for both teams the first time => a new failure gives you more info to actually fix it
    11:16 AM Aug 7th, 2008
  • because of the broken builds pairs started sequential checkins
    11:17 AM Aug 7th, 2008
  • broken builds for bubble prevented to get into the “flow”, it was a source for distractions and inefficiencies
    11:19 AM Aug 7th, 2008
  • team squeak did some manual tests for javascript & browser related issues
    11:20 AM Aug 7th, 2008
  • bubble worked on the order of tests to anticipate likely failures, segmented tests, added tests not dependent on DB
    11:21 AM Aug 7th, 2008
  • squeak worked with a in-mem db that didn’t help much, using the db itself was almost quite the same, all tests in junit
    11:21 AM Aug 7th, 2008
  • squeak really relied on out of container tests, with extremely light servlets and all biz logic in POJOs
    11:22 AM Aug 7th, 2008
  • make the build a priority for every member of the team, small improvements for every iteration
    11:23 AM Aug 7th, 2008
  • keep the build simple, ideally a single build file for all environments
    11:24 AM Aug 7th, 2008
  • keep the number of tech as low as possible, focus on small technologies
    11:24 AM Aug 7th, 2008
  • build once, reuse the built artifact as many times as possible
    11:25 AM Aug 7th, 2008
  • minimise runtime dependencies, run the application standalone and use stub/mocks as much as possible
    11:25 AM Aug 7th, 2008
  • keep test & data in the code together => it helps refactoring! (nice)
    11:26 AM Aug 7th, 2008
  • Not too bad, now attending “Effective and pragmatic Test Driven Development” => http://tinyurl.com/6kx76v
    11:31 AM Aug 7th, 2008
  • Exp report from Valtech about applying TDD to a project in T-Mobile
    11:31 AM Aug 7th, 2008
  • Decision to use TDD came after project inception (with management awareness, agile-savvy and willingness to try new things)
    11:35 AM Aug 7th, 2008
  • Risks of using TDD was shared with the whole team and interactive sessions / presentations were held
    11:36 AM Aug 7th, 2008
  • MVC app, this team responsible for backend, very well defined integration point (contract) with frontend team
    11:37 AM Aug 7th, 2008
  • top down dev, starting from AT (bleah), deriving major services, implementations, their UTs and DAOs (and their UTs) from it
    11:40 AM Aug 7th, 2008
  • In retrospect they thought they have left too much freedom to less experienced developers
    11:41 AM Aug 7th, 2008
  • Some negative feedback not related to TDD, but to the fact that devs were more involved (and being accountable) with estimates, meetings,etc
    11:51 AM Aug 7th, 2008
  • But TDD helped in getting things better in terms of improved efficiency
    11:51 AM Aug 7th, 2008
  • Too much focus on TDD “dogmas” and TDD application, too much time spent on enhancing tests
    11:51 AM Aug 7th, 2008
  • Benefits: better code, more cohesive, top down approach with mocks helped with SRP (but yet, bleah), YAGNI (discourage speculation)
    11:51 AM Aug 7th, 2008
  • Stakeholders involvement payed back with increased confidence and a favourable comparison with previous experience (better defect rate)
    11:51 AM Aug 7th, 2008
  • This in turn caused support for refactoring, which brought code cut faster, shorted feedback cycle, and more confidence (etc)
    11:51 AM Aug 7th, 2008
  • conclusions: TDD is good, can be difficult to introduce, not much experience around, devs who embrace it got rewarded
    11:56 AM Aug 7th, 2008
  • made real differences in delivery
    11:57 AM Aug 7th, 2008
  • all in all, I’d yeeeesh, not a fantastic session 🙂
    12:01 PM Aug 7th, 2008
  • at programming with the stars the 3 remaining pairs will do the same, STDD something (eh eh, Joshua… 🙂 )
    12:47 PM Aug 7th, 2008
  • pairs: Lasse & Noah, Michael & Emily, Arlo & Andrew
    12:48 PM Aug 7th, 2008
  • for once, the judges have been judged and scored a remarkable total of 6!! (they were doing uncle Bob’s prime factors kata)
    1:01 PM Aug 7th, 2008
  • lasse & noah are using RSpec Story with a well defined scenario ready to go
    1:03 PM Aug 7th, 2008
  • they are working on a rails app, in the end
    1:04 PM Aug 7th, 2008
  • Using Rspec Story to drive Watir
    1:05 PM Aug 7th, 2008
  • jumped on javascript code (with continuous testing) to implement the missing function, but didn’t have time to complete it
    1:09 PM Aug 7th, 2008
  • uncle Bob didn’t like the test through the gui (because it was slow and he was kind of right) ==> 2
    11:11 PM Aug 7th, 2008
  • Michael & Emily are using a swing app, with junit UTs and AT with a recorder that drives the app and manages the text through text output
    1:15 PM Aug 7th, 2008
  • the app is TextTest => http://texttest.carmen.se/
    1:16 PM Aug 7th, 2008
  • they have involved the “customer” => nice!
    1:16 PM Aug 7th, 2008
  • much better this time: they have scored 26 (uncle Bob biased towards Michael Feathers?)
    1:20 PM Aug 7th, 2008
  • Arlo & Andrew pulled out a board with planned cards, a PO, a kind of lavalamp for build status, rspec Spec, cruisecontrol.rb
    1:23 PM Aug 7th, 2008
  • they are really driving the lamp in the story
    1:23 PM Aug 7th, 2008
  • kind of eating your own dog food
    1:24 PM Aug 7th, 2008
  • nice idea, but confused execution
    1:27 PM Aug 7th, 2008
  • total score: 1
    81:29 PM Aug 7th, 2008
  • finale: Michael & Emily vs Lasse & Noah (Arlo & Andrew are gone)
    1:31 PM Aug 7th, 2008
  • Attending “The Economics of Agile: The competitive advantage of developing commercial software with Agile methods”
    2:02 PM Aug 7th, 2008
  • Remarkably empty room (a big one too), I guess that an IBM VP doesn’t attract a lot of geeks, but the topic is interesting
    2:03 PM Aug 7th, 2008
  • http://tinyurl.com/6j2ozy
    2:04 PM Aug 7th, 2008
  • Each table can hold 10 people and there are 36 tables, but only ~35 people in the room (vs a capacity of 360!!)
    2:05 PM Aug 7th, 2008
  • IBM started a transformation in 2005 in order to control quality better, manage stakeholders satisfaction, and anticipate return/profit
    2:09 PM Aug 7th, 2008
  • from waterfall (still used for mainframe dev), through iterative (lacking discipline), to agility (with discipline, adaptability)
    2:12 PM Aug 7th, 2008
  • extremely distributed teams, compliancies, heterogeneous environment => the Poppoendiecks helped them
    2:14 PM Aug 7th, 2008
  • Nice: a project dashboard with “corporate revenue”!! 🙂
    2:15 PM Aug 7th, 2008
  • 1100 people in agile@IBM, 5000 trained, 220 projects, 78% of teams are using some elements of agile
    2:16 PM Aug 7th, 2008
  • schedules are more predictable and early (18 -> 12 months, some other teams 12 months to 9 months)
    2:17 PM Aug 7th, 2008
  • IBM’s customers seem to like it, especially in terms of increased confidence of what is delivered (they are involved early and often)
    2:18 PM Aug 7th, 2008
  • Metrics: B/C R = Benefit/Cost Ratio, ROI
    2:19 PM Aug 7th, 2008
  • E^2 value framework: efficiency + effectiveness
    2:20 PM Aug 7th, 2008
  • headcount is used primarily as a source of cost, which is scary, but at the same time they say that people are reused elsewhere
    2:22 PM Aug 7th, 2008
  • revenue generated per head I like it better
    2:23 PM Aug 7th, 2008
  • Ted Rivera is focusing on DB2 (600 people) => 60 M$ in terms of investment (rough figure, but it scares!)
    2:25 PM Aug 7th, 2008
  • In the past it was impossible for IBM to make changes in terms of plans until it was too late, but now it’s different
    2:28 PM Aug 7th, 2008
  • the assessment is a total of ~90.6M savings up to date because of agile adoption (prod gain at startup & tasks implementations, …
    2:31 PM Aug 7th, 2008
  • cost & waste avoidance too)
    2:31 PM Aug 7th, 2008
  • it’s all about defect reduction & time saved in terms of quality injection after the fact (IBM has tons of historical data to qualify that)
    2:33 PM Aug 7th, 2008
  • expected: 168 field tickets, 1056 trouble tickets – actual: 32 field tickets, 57 trouble tickets (with more customers)
    2:35 PM Aug 7th, 2008
  • what does that mean? roughly 2.576M$ in savings and much more customer loyalty
    2:36 PM Aug 7th, 2008
  • it’s nice to see an IBM tech guy with a huge impact in the org to talk intelligently about XP practices and Lean principles, like TDD
    2:41 PM Aug 7th, 2008
  • An internal agile survey showed an improvement in every area of agile adoption (testing, CI, scrums, etc.)
    2:50 PM Aug 7th, 2008
  • IBM’s commitment to agile is remarkable. Practices are treated in guidelines and the situation is constantly evolving
    3:01 PM Aug 7th, 2008
  • Nice (from the crowd): IBM tools are notoriously heavy and difficult to work with for agile teams (eg: websphere in CI), so?!?
    3:03 PM Aug 7th, 2008
  • Basically: what are you going to do about it?
    3:03 PM Aug 7th, 2008
  • The answer is not that satisfactory, just a generic promise of making it simpler and having more internal people validating products
    3:04 PM Aug 7th, 2008
  • Question about pair programming adoption. Answer: we treat practices as a menu. PairProg is used to one degree or another.
    3:07 PM Aug 7th, 2008
  • But not broadly: if PairProg is not used some other practices related to code reviews should kick in
    3:08 PM Aug 7th, 2008
  • Reuse is a key aspect of savings, but it’s not clear the connection with agile adoption
    3:09 PM Aug 7th, 2008
  • I don’t know, I suppose that given the context it was an interesting session, but I didn’t hear much new and I was hoping for more meat
    3:26 PM Aug 7th, 2008
  • Attending “Workshop on build/test grids and selective testing tools” => http://tinyurl.com/6ntzrv
    4:02 PM Aug 7th, 2008
  • 5 attendees, too bad
    4:05 PM Aug 7th, 2008
  • VersionOne lead tester looking for breaking the tests and distribute them (+3hours now)
    4:06 PM Aug 7th, 2008
  • One more attendee (a good one: Rick Mugridge? )
    4:07 PM Aug 7th, 2008
  • selective testing: a tool that can dynamically figure out which tests to run, a dynamically built suite, based on modified code
    4:07 PM Aug 7th, 2008
  • Presenter: the team was large, many many thousands of developers. WOW! I wonder what it is.
    4:18 PM Aug 7th, 2008
  • Breaking the build was quite an issue. Done some prioritization of tests in python, not satisfactory enough. Dependencies were big and messy
    4:20 PM Aug 7th, 2008
  • TW lead tech guy talks about Cruise, evolution of cruise control that will take into account also distr test, selective test, etc
    4:24 PM Aug 7th, 2008
  • googletester is a selective tester tool
    4:25 PM Aug 7th, 2008
  • sorry, google-testar => http://tinyurl.com/vuhf
    74:27 PM Aug 7th, 2008
  • the sourceforge page is even better => http://tinyurl.com/5lba8f
    4:30 PM Aug 7th, 2008
  • Mugridge: the complexity is finding the right dependencies in order to run only certain tests
    4:35 PM Aug 7th, 2008
  • Instrumentation seems the key and it’s not that difficult if you use a system like Spring on top of your classes
    4:35 PM Aug 7th, 2008
  • 12′ for the whole build cycle, not bad but he’d like faster
    4:40 PM Aug 7th, 2008
  • TW was working on VM cloning, tests split and collection of results
    4:45 PM Aug 7th, 2008
  • Build cycle based on stages (packaging, UTs, FTs, PerfTs, UATs depl stage, Prod depl) where every stage can be broken and distributed
    4:51 PM Aug 7th, 2008
  • Using VMs gives the advantage & flexibility of deciding how to distribute it: all the VMs can run in 1 box in reality
    4:52 PM Aug 7th, 2008
  • 2 stages of the same pipeline cannot start concurrently, but Pkg can start when FTs are running because of a new commit
    4:53 PM Aug 7th, 2008
  • Robert Norton is the TW guy
    4:54 PM Aug 7th, 2008
  • 10′ build time for mingle
    4:58 PM Aug 7th, 2008
  • Selective testing tools are good but very context dependent (and even the same project may change in its lifetime)
    5:27 PM Aug 7th, 2008
  • Careful with getting false confidence with selective testing (all the attendees agreed on it)
    5:28 PM Aug 7th, 2008
  • The concept of smoke tests is to provide early feedback (some end-to-end tests) and they be considered an appendix to UTs
    5:29 PM Aug 7th, 2008
  • workshop report to be submitted at http://tinyurl.com/5spxln
    5:38 PM Aug 7th, 2008
  • Nice session, in the end, mostly because of the experiences and thoughts from various participants
    5:45 PM Aug 7th, 2008
  • Just had a demo of mingle => http://tinyurl.com/2otxqj => it rocks!!!
    6:49 PM Aug 7th, 2008
  • mingle is a fantastic product, but a few things are missing to make it really irresistible: snapshot, better and more integration, eg: build
    6:51 PM Aug 7th, 2008
  • and a 1 click gantt generator for those customers that demands it (for reporting purposes)
    6:51 PM Aug 7th, 2008
  • Attending “Starting a Kanban System for Software Engineering with Value Stream Maps and Theory of Constraints”
    8:34 AM Aug 8th, 2008
  • http://tinyurl.com/55vglo
    8:35 AM Aug 8th, 2008
  • Kanban systems & agile => http://tinyurl.com/2vlv9
    58:41 AM Aug 8th, 2008
  • Assigning tasks early creates and artificial critical path => that’s why on task allocation too we want to use the last possible moment
    9:24 AM Aug 8th, 2008
  • Limit multitasking (or avoid it, if possible)
    9:24 AM Aug 8th, 2008
  • I don’t agree that reprioritizing the sprint backlog continuously is waste => it looks like an opportunity to create biz value to me
    9:31 AM Aug 8th, 2008
  • @pierg Sorry, I haven’t detailed the context: when a person is available to pick a new task => reprioritize (if needed) seems good to me
    9:52 AM Aug 8th, 2008
  • @pierg Worth to mention that the presenter was proposing to have a READY state between TODO and IN PROCESS
    9:53 AM Aug 8th, 2008
  • Of course, low availability resources decrease productivity => you need to add buffering in order to cope with that and keep high thr …
    9:55 AM Aug 8th, 2008
  • pull model, not push; global optimization, not local
    9:56 AM Aug 8th, 2008
  • the workflow should reflect the relative skills and strengths of actual team members
    9:57 AM Aug 8th, 2008
  • check presenter’s blog at http://tinyurl.com/6s9nwg
    9:58 AM Aug 8th, 2008
  • and participate to http://tinyurl.com/5t9fa2 (low volume)
    9:59 AM Aug 8th, 2008
  • I’m not sure I’ve enjoyed much this presentation, kind of messy and conceptually unfocused (but the subject is very interesting!)
    10:00 AM Aug 8th, 2008
  • In 30′ it’s time for the last keynote, from Alan Cooper (aka mr. VB) 🙂
    10:00 AM Aug 8th, 2008
  • Alan Cooper is reading, I don’t particularly like that
    10:37 AM Aug 8th, 2008
  • BUM: programmers are the smartest people in the human race! 🙂
    10:38 AM Aug 8th, 2008
  • Watching a programmer works is as revealing as watching oysters grow
    10:39 AM Aug 8th, 2008
  • Once again, programming is craft and all software is bespoke, custom
    10:40 AM Aug 8th, 2008
  • programming is not art => YEAH! FINALLY!!! Programs are practical, art is not, even though some code is beautiful
    10:41 AM Aug 8th, 2008
  • programming is extremely creative and design is every day job
    10:42 AM Aug 8th, 2008
  • Interfaces are the most critical part of sw. They are the place in between (sw and humans, sw and sw => API)
    10:44 AM Aug 8th, 2008
  • Mmm: too much emphasis on using existing libraries, etc. => even though after that he recognize you don’t want to choose the wrong tool
    10:46 AM Aug 8th, 2008
  • there is no silver bullet, but agile is different => it’s about process and people rather than tools and it’s inclusive of other people
    10:48 AM Aug 8th, 2008
  • sw is a multistage process, but waterfall is an obsolete one => waterfall doesn’t work, it produces improper crap rejected by users/c …
    10:50 AM Aug 8th, 2008
  • waterfall introduces mistrust, opacity, isolation, delay, failure of design
    10:51 AM Aug 8th, 2008
  • the boundaries of stages are fuzzy and overlap and recur continuously
    10:52 AM Aug 8th, 2008
  • a reaction to waterfall failure was a reduction to just coding (construction), but that fails too
    10:53 AM Aug 8th, 2008
  • four stages: 1) big idea, 2) design (meaning product use), 3) Engineering (about how we build), 4) construction, that is let’s build it
    10:54 AM Aug 8th, 2008
  • Disagree: programming time requires solitude
    10:59 AM Aug 8th, 2008
  • agile is a powerful tool for coping with incompetence
    11:01 AM Aug 8th, 2008
  • agile is a tool for coping with BDUF, unreasonable customers (confront with the options), incompetent designers, foolish managers
    11:03 AM Aug 8th, 2008
  • the big sin of dealing sw production as a manifacturing line, cmd & control, but programming is post-industrial
    11:04 AM Aug 8th, 2008
  • dealing with a knowledge worker is completely different and it’s based on competence, respect, education, quality of work
    11:05 AM Aug 8th, 2008
  • efficiency (reducing prices) vs effectiveness (increasing quality)
    11:06 AM Aug 8th, 2008
  • going fast in the wrong direction is worse than going slow in the right direction => no real advantage in reducing costs
    11:07 AM Aug 8th, 2008
  • because of the lack of understanding of sw, pressure is applied to reduce time, quality is neglected
    11:09 AM Aug 8th, 2008
  • no competent business person would choose speed over correctness if they have confidence in the outcome
    11:09 AM Aug 8th, 2008
  • challenge stupid deadlines, demand quality lines
    11:11 AM Aug 8th, 2008
  • agile brings design, engineering and construction together in an iterative fashion
    11:14 AM Aug 8th, 2008
  • agile brings the desire to create better product to the programmers, which is exactly what the interaction designers want too
    11:16 AM Aug 8th, 2008
  • why they are not in love with each other? interaction designers love to do those things that the programmers typically don’t like
    11:17 AM Aug 8th, 2008
  • creating user stories is the programmers weakest skill, but it’s the interaction designers strongest one
    11:19 AM Aug 8th, 2008
  • interaction designers can be very effective and helpful at dealing with distortions, cognitive illusions and communication issues with users
    11:25 AM Aug 8th, 2008
  • requirements are not design, requirements cannot be gathered like colored eggs on Easter’s morning 🙂
    11:27 AM Aug 8th, 2008
  • the first work product of interaction designers is a narrative restatement of the problem => this triggers validation, feedback, corr …
    11:29 AM Aug 8th, 2008
  • the interaction designers when coding starts is related to reconcile all the ideas that pop up with the interaction with customers/prgmers
    11:32 AM Aug 8th, 2008
  • some misconceptions about planning and change in requirements => we do plan and we do design, we just do it continuously
    11:34 AM Aug 8th, 2008
  • Brooks: Plan to throw one away
    11:37 AM Aug 8th, 2008
  • Cooper suggests to REALLY plan to throw the first time away ==> prototyping?
    11:37 AM Aug 8th, 2008
  • The way he intends it is a big bang prototype that solves all the problems and build all the necessary knowledge, so that production is easy
    11:39 AM Aug 8th, 2008
  • agile methods are the first to make room for interaction designers, whose purpose is to make software better for the user
    11:40 AM Aug 8th, 2008
  • seems like a happy marriage, a win-win (in Cooper’s mind)
    11:41 AM Aug 8th, 2008
  • slides and notes available at http://tinyurl.com/5cvop5
    11:42 AM Aug 8th, 2008
  • question: what about biz analyst? is it the same; answer: BA are about biz processes, but interaction designers are about human behaviour
    11:46 AM Aug 8th, 2008
  • a question challenges the approach of doing it first in an agile way, throw it away and doing it again in a non agile away
    11:49 AM Aug 8th, 2008
  • but he doesn’t clarify the second time how it should be built
    11:50 AM Aug 8th, 2008
  • Mike Hill (the brit one) and Steve Freeman are going to demonstrate a bit of TDD as stars… 🙂
    1:05 PM Aug 8th, 2008
  • of course they are all messed up, one doesn’t listen to the other, one talks at the phone, one holds onto the keyboard, both complains cont.
    1:08 PM Aug 8th, 2008
  • test name: RobotLightbulbManagerServiceTest 🙂 😀
    1:09 PM Aug 8th, 2008
  • joshua killed them and asked them to go away! 🙂
    1:10 PM Aug 8th, 2008
  • now it’s Michael Feathers and Emily Bache (for real)
    1:11 PM Aug 8th, 2008
  • argh, python code
    1:11 PM Aug 8th, 2008
  • refactoring a poker engine written in python
    1:12 PM Aug 8th, 2008
  • got themselves a bit into trouble, the rhythm is slow, wrote some code, being bashed by the audience on YAGNI, still haven’t executed tests
    1:18 PM Aug 8th, 2008
  • still red bar
    1:18 PM Aug 8th, 2008
  • nope, time is up, no green bar, amazingly bad
    1:20 PM Aug 8th, 2008
  • judges: 5+5+6=16
    1:23 PM Aug 8th, 2008
  • a “Deloitte & Touche” consultant is going to collect audience scores 😀
    1:25 PM Aug 8th, 2008
  • Lasse and Noah now, refactoring slow tests, because of a movie file being read
    1:25 PM Aug 8th, 2008
  • actually is slow code and they want to decouple sync logic from file reading logic
    1:26 PM Aug 8th, 2008
  • using a stub for testing the sync reader
    1:27 PM Aug 8th, 2008
  • red bar, green bar
    1:27 PM Aug 8th, 2008
  • introducing a virtual clock to simulate lagging, nice
    1:28 PM Aug 8th, 2008
  • mixing a mock object for the TimeSource and the stub for file processing
    1:29 PM Aug 8th, 2008
  • I mean, they are going fast, but they are not refactoring any test code (and it would need it)
    1:32 PM Aug 8th, 2008
  • they have played the movie missing frames in the end, nice
    1:34 PM Aug 8th, 2008
  • uncle Bob gave a 9! total: 2
    11:36 PM Aug 8th, 2008
  • now the pairs have to type 2 sentences at the keyboard
    1:38 PM Aug 8th, 2008
  • unfortunately it’s an inverted one 🙂
    1:39 PM Aug 8th, 2008
  • -10 points because of overrun and errors for Lasse and Noah
    1:43 PM Aug 8th, 2008
  • Michael and Emily got a +12
    1:46 PM Aug 8th, 2008
  • the consultant is calculating the score for both pairs
    1:47 PM Aug 8th, 2008
  • 38 grand total for lasse & noah
    1:47 PM Aug 8th, 2008
  • 35 grand total for Michael and Emilty
    1:48 PM Aug 8th, 2008
  • hence, Lasse and Noah won!
    1:48 PM Aug 8th, 2008
  • prizes: beer, cutting plate, 2 books WOW!
    1:50 PM Aug 8th, 2008

Unfortunately this year I can’t attend Agile 2009 in Chicago, but if you have an opportunity to go there I can only recommend it. Please let me know if you are going to tweet live from it.

If it’s for test then it has to be good

Wednesday, March 25, 2009

You are dealing with some old code: it’s not too bad, you have definitely seen much worse. There’s a class within a home-grown framework that doesn’t allow you to test some application code. You spend some time understanding how to change things to make it more testable. After a while you come up with the idea of adding a method to that class solely for testing purposes – it’s not a fantastic thing, but the alternatives are frightening and this will make a lot of code much more testable. Because you are a good person you want to make it very clear that that code is intended to be used for testing only, not in real life:

public void setMickeyMouseForTest(Class cheeseClass,
                                  String cheeseName, Mouse mouse) {
    cheeseClass = resolveAssociatedClass(cheeseClass);

    if (getLocalFactory().getCheeseName(cheeseClass) != null)
        throw new RuntimeException("Use standard setMickeyMouse instead");

    mouses.put(cheeseName, mouse);
}

This method is similar to the real thing:

public void setMickeyMouse(Class cheeseClass, Mouse mouse);

Method setMickeyMouse determines cheeseName by calling getLocalFactory().getCheeseName(cheeseClass), while method setMickeyMouseForTest assumes that cheeseName is not set (even stronger: if it is not null it throws an exception) => instead, it uses the parameter provided in the call.
In the end, the difference between the two lies in the way a certain condition is handled in the code: the method written for testing purposes is capable of dealing with an exceptional condition differently. You’ve created that method exactly to simplify the amount of setup required for tests, so that it doesn’t matter if getLocalFactory().getCheeseName(cheeseClass) is not capable of providing a cheeseName.

So, the code is committed, lots of tests are now possible, the team celebrates your genius.

Some weeks later you’re pair programming with a colleague. You’re still dealing with the same framework, but this time it’s a different corner-case. After lots of mumbling, back and forth, debugging sessions, you superficially look at the signature of that method, obviously without remembering what you have done a month before. You just vaguely remember that it was capable of dealing in a better way with a corner-case. And the method is called “…ForTest” so it has to be good, in the end we are testing indeed.

It doesn’t work. An exception is thrown. The exception hints to “Use standard setMickeyMouse instead…“. Oh gosh, let’s take a look: ops, I know what it is, we’ve forgotten to set cheeseName for cheeseClass in the local factory. Let’s do that.

<type> <type> <type> <type> <type> <type> <type> <type> <type> <type> <type> <type> <type> <type> <type> <type>

Damn, it’s still not working. What is it happening?!?

<debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug> <debug>

It’s very strange.

getLocalFactory().getCheeseName(cheeseClass) is not null, so how can it possibly say that it is null instead?!?

<more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug> <more debug>

Oh my gosh! The exception is actually thrown WHEN getLocalFactory().getCheeseName(cheeseClass) is not null, not viceversa!

What happened is that you and your colleague just wasted 20′ because you misread “!= null” (reading “== null” instead). What does that tell you?

It told me a few things:

  • Managing null values is always evil and error prone. At the very least, checking for a null disrupts the business logic and it makes the code less readable, ultimately more likely to break.
  • It’s too easy to misread a check based on != vs ==. Not only that: the fact that an exception is being thrown if the expression evaluates to true hints that the check is to verify if the value is null, because that is what you would typically do when you are adding a guard to your code. Which brings me to the next point.
  • Moreover, the intent of the code is not clear. It’s not enough to call a method “somethingMeaningfulForTest“, that is the suffix “ForTest” is misleading if the method is not truly applicable for every test. This is even reinforced by the fact that the exception message is not enough communicative.
  • As usual, test code has to follow the same standard of quality as application code, which means readability, expressiveness, simplicity, lack of duplication. This is particularly important for accessory code that is written to support tests.

So, in the end, how should that method be called? What about:

public void setMickeyMouseForTestWhenThereIsNoAvailableCheeseName(
    Class cheeseClass, String cheeseName, Mouse mouse);

The underlying precondition is that method getLocalFactory().getCheeseName(cheeseClass) returns null for this code to execute properly. This should be made clearer with a checked exception being thrown by the method, or, even better, by a custom annotation that marks the method with the defined precondition. In Java land this doesn’t go very much beyond documentation purposes for a variety of reasons:

  • First of all there are limitations on the attributes that an Annotation can exhibit. Specifically, only String, Class and primitive types are allowed, not even object version of scalar type like Integer can be used.
  • Second and foremost, annotations can be queried to provide metadata about code, but they don’t really affect code execution. An annotation is useful only to the degree it is recognized by the runner executing some code. There is in the end no native and effective support for Design By Contract in Java and there will probably never be. A useful result can be achieved by combining together AOP with Annotations, but I have mixed feelings about it => in the end I always have the impression that the mumbo jumbo typical of the supporting infrastructure it’s there just to circumvent a limitation that should be inherently addressed by the language itself.

Or shouldn’t it?

Using functors to handle test connections

Thursday, October 9, 2008

A few days ago I was working on some functional tests that have the following structure:

      public void testSomething() throws Exception {
        // 1. istantiate the system
        // 2. interact with it through the UI
        connectDB();
        // 3. assert the state of the system by examining the DB
        relinquishDB();       

        // 4. interact a bit more through the UI
        connectDB();
        // 5. assert again the state of the system by examining the DB
        relinquishDB();        

        // etc.
    }

Some notes about them:

  • Tests are implemented with JUnit 3.8.2, but this is irrilevant to what will follow.
  • The application is a webapp implemented with a proprietary web framework.
  • Those tests are implemented running the servlet engine in process through ServletUnit.
  • The DB is interfaced through a proprietary framework and it is accessed behind a mock JDBC driver that works translating SQL statements in memory.
  • The state of the system is examined by fetching either an Entity or EntityList, abstract classes that represent respectively a single record or a collection of records in a table.
  • Methods connectDB() and relinquishDB() are used to manage the test connection, which needs to be different than the one used by the application for structural reasons.

All the tests that follow the above structure inherit from an abstract TestCase which implements its own setUp and tearDown, treated like template methods whose behaviour can be refined further (for example, choosing to install a different mock DB). connectDB() and relinquishDB() use connection parameters that are potentially established differently for every test, but in reality they are seamlessly copied over from test to test.

While this structure makes sense, I thought that repeating connectDB() and relinquishDB() several times in the test is extremely annoying. It would be nice if we can wrap some logic into a block that takes care of the repetition for us: calling connectDB() first, executing our code and then calling relinquishDB() in the end. If I was programming in ruby, that would have been extremely easy to implement with proper blocks, wouldn’t it be? Well, in java is not that difficult either, if you tolerate the syntactic noise of course. Let’s see how.

To start, we can define a Functor interface that can be used to implement a block without parameters:

public interface Functor<T> {
    T exec();
}

Now we need to wrap a bit of logic around it that takes care of connecting the DB and relinquishing it afterwards:

public class Fetcher {
    public static <T> T queryStateFor(Functor<T> f) {
        connectDB();
        T result = f.exec();
        relinquishDB();
        return result;
    }
    private static void connectDB() {
        // ...
    }
    private static void relinquishDB() {
        // ...
    }
}

The actual implementations of Fetcher.connectDB() and Fetcher.relinquishDB() are fairly trivial and the methods are in reality inlined: I’ve written them in a Compose Method style just to make them easier to understand and to trace back Fetcher structure to the test structure above.

The Fetcher can be used in the following way (assuming that it is imported statically in the test case):

    public void testSomething() throws Exception {
        // 1. istantiate the system
        // 2. interact with it through the UI
        final PurchaseOrder po = //...
        PurchaseOrderStatus status = queryStateFor(new Functor<PurchaseOrderStatus>() {
            public PurchaseOrderStatus exec() {
                return po.getLastStateChangeStatus();
            }
        });

        // etc.
    }

As you can see, this refactoring is not about saving keystrokes.

A developer was asking me which kind of advantage this style achieves over the other. In general, I think that functors are a powerful tool, often underestimated in statically typed languages like java. But these are the reasons why I think they are particularly relevant in the context of queries and tests:

  1. Avoid repeating connectDB(); … relinquishDB(); without the introduction of methods that would have to be rewritten for every test.
  2. Ensure that queries for checking purposes correctly connect to the proper DB and relinquish it in the end.
  3. Make the test more readable, without hiding relevant details (connecting and relinquishing the DB are irrelevant details to the test, only the query matters in the eyes of the reader / maintainer).
  4. In general, make the test less dependant upon lower level details whose relevance is marginal.
  5. Using a functor the query can be executed as a generified method, hence it definitely scores better in terms of type safety.

I’ve also thought about any disadvantage:

  1. The syntax looks a bit odd for the java world, but it’s not entirely a stranger.
  2. Current functor implementation can handle only one result per call, but it can certainly be extended and in general it doesn’t hurt significantly calling several functors in a row.

And you? What do you think about this way of handling a test connection to the DB?

PS: While I think that this style provides a better alternative to repeat calls to connectDB() and relinquishDB() all over the code, there’s actually a simpler way to solve the problem ==> the tests can be provided with their own connection that has nothing to deal with the connection handled by application code running in the servlet engine (credits to Mark for recognizing it). Nevertheless, functors are a powerful tool typically neglected in the java world and they definitely deserve more attention in situations like this one.

The consequences of shipping

Thursday, September 11, 2008

A friend of mine pointed me towards one of last Kent Beck’s mini-essays: Just Ship, Baby. While reading it I felt more and more surprised: I was hoping to find a controversial but sophisticated provocation directed towards those who hide behind the process and fail to deliver because of it. That can be quite common among those teams that are approaching agile methods for the first time and fail to understand and embrace the pragmatic essence of methods like XP. It’s a kind of error that can happen frequently for teams coming from a waterfallish culture and it’s always done in good faith. The rules are tools and they change over time, based on the circumstances.

Despite his good intentions, though, I think that Beck has failed to ship the right message, so to speak: the examples are quite misleading because when I’m exploring a technology that I don’t know well enough, the right way to go is spiking and prototyping until I get the answer that I’m looking for. Under these circumstances, only at a later stage it makes sense to apply TDD, typically when you have played enough with the technicalities of the specific API. Certainly, this shouldn’t happen anytime sooner than when you feel safe filling the gaps related to the expectations in your tests. So, I’d say that Beck’s problem in this case has been VIOLATING what I consider one of the implicit rules of TDD: don’t write your tests until you have enough understanding of the modelling mechanisms of the API / framework / system you are dealing with. TDD can be used to explore those mechanisms, but that’s a completely different story.

There’s actually even more there: if I’m just interested in evaluating the feasibility of something, then a throwaway prototype seems a better idea. When you have confirmed the feasibility, and only after that, then you make tabula rasa and develop your system applying TDD consistently. No shortcuts, but different approaches depending on your intent. I don’t think there’s anything new in this.

Going back to Beck’s essay original intent (as I understand it), I think that the target has been missed: the issue whether you follow the rules or you deliver the system has not been addressed. First of all, it’s a faulty dilemma if you put it like that. If there’s one thing that agile methods have taught me it’s that delivering a system and/or completing a project are the result of trade-offs, continuous negotiation, constant planning throughout the entire project life, reassessing current status and goals based on feedback. But let’s put all those considerations aside for a while and let’s consider a situation where a short-term deliverable can be achieved if some of the practices are relaxed. In this situation, the team incurs typically into some significant technical debt or, even worse, the external quality of the system is severely affected. What should you do? Well, it depends and as usual it’s a judgement call.

There are situations in which it makes perfect sense and it buys you time to clean the system immediately afterwards. I haven’t seen this happening many times, especially because it requires a perfectly healthy, open and honest environment, but I accept the idea that it can happen. On this subject, for example, there was a blog post by James Shore a couple of years ago which triggered an interesting and lengthy discussion in the XP mailing list.

There are other circumstances where the system can already be so compromised that no more design debt can be tolerated or the lack of correctness of the software can cause financial loss (or worse). My experience is that this is much more likely to happen and when it does its consequences can be devastating. Saying “Just ship, baby” can have catastrophic effects and it’s ultimately irresponsible.

It’s so annoying having to test all that code and solve those bugs. Real programmers write correct code and eventually solve bugs instead of sleeping. That’s what we are paid for, aren’t we? I fear that Beck’s post can ultimately feed the despicable culture of Macho Programming.

Delivering software is not like winning a football game: it has consequences that need to be taken into account. In sports, when a game is over, it’s over: unless you have broken the law, got some dope or other things that have nothing to deal with sport itself, every game is self-contained.

Software development though it’s different. More than often the consequences of your actions come back at you, sometimes in very unpleasant and everlasting ways.