As I said here before I was in Brazil some weeks ago to present at a conference. I had a really great time over there with some amazing people and would like to thank Caelum for their hard work in creating such a great conference. I’m making the slide deck and notes available in my Brazilian Portuguese website and thought it would be nice to translate them to English and publish it here too.

The following paragraphs are a consolidated -but really long!- version of my slide notes. The slide deck you can find in slideshare:

In my experience it is really hard to translate material I write for the Brazilian community to English so please be advised that some bits may not make much sense, please report problems in the comments or drop me a mail.

The world has changed

If you’ve been working in this industry for a while you probably know what I am talking about: nowadays things are harder. Years ago our job was to build systems, today our job is to make sure old systems talk to each other.

When you get to the Tech Lead role in a company or project you often come full of ideas, excitement and plans. And you will see all these being destroyed by your customers and colleagues. No one will trust you, no one thinks you are able to fix those problems; no one will give you any opportunity to change anything for real –even though they said that they would.

The problem is that you clients have been bitten before. They’ve played this “let’s develop a new system” game and hate it! They’ve got their plans and dreams crushed by a bunch of code cowboys that came in with wonderful tools and techniques and delivered that buggy piece of crap -and charged 2x the agreed price. Your customers can’t take any risks. No, not this time. Not again. Regardless of what you agreed they will tell you to do what they think works.

And in this situation what happen to the standard coder? She just gives up. It’s so hard to change anything that let’s just join the dance and play the 9-to-5-job game. She starts writing all those silly specs that the client asked for, she asks the Ivory Tower architect about every single stupid detail in her application… and she starts producing pretty much nothing useful.

But after spending that many nights awake reading books, blogs and trying stuff out… after fighting to deliver on schedule and on budget for so long… will you really accept this?

There are options, resistance is not futile! It is not going to be easy. You will be openly attacked, secretly hated and people will be terribly picky even about what you wear or how you say things. But you can survive as long as you keep in mind that there is only one thing that guarantees survival: deliver value. Make sure your customer reaches his goals –and those are not to have a complete spec for the system but to earn them some money.

To make sure your customers trust you it is important that you deliver value frequently –something your client can count on. It is also important that you deliver at the right time –do not miss a deadline. And it is important that you are never late –make sure you have realistic estimates and know when something is going to be late.

But to get those there is a big enemy to fight: surprises. If there were no surprises in software development most teams would be able to deliver quality software right on time. Surprises are what make those 10% left in a task last forever. As a Tech Lead your goal is to make sure your team is delivering in a timely fashion. As a Tech lead it is your goal to avoid surprises.

And to avoid those, my suggestion is that you build barriers to keep them away. Like in any good zombie movie.

The House of the 17 Specs

That reminds me of when I used to work for this big product company. They had a funny process where before the first line of code was written you had to come up with a spec for the full product, then one spec for each of the modules. Each of those documents should be paired with a test strategy/case document. I think that if you followed the full process –and this was the usual way- for a large system one would write 17 documents before the first line of code.

In theory that would not be a problem, the specs were detailed enough so, theoretically, the coding task would be just to convert what was written in there to Java, C# or whatever we were using for that piece. That wasn’t the case, as you probably imagined. There was a huge impedance mismatch between the world described in those specs and the real world. The library we were be using would happen not to be thread-safe, the operating system that the client wants would be missing that very system call we relied on or something like that.

After so many weekends in the office trying to make the real world look more like the spec we started to think about doing things a bit differently. We came up with all these cool ideas about how to improve our development process and took them to middle and senior management types. Of course that didn’t work, management was definitely not interested in what a bunch of punks with a couple of years of experience in that company had to say about their 10-years-delivering-software process.

We got really upset and for the next couple of weeks I would see my team visiting job boards for most of the day. Eventually someone came with an interesting idea: the process says that we must have such and such document but it doesn’t actually say that the docs should be all ready before coding starts. Leaving the documents to the end will probably mean that they will never be finished and the team is going to be in trouble… but what about we try to write them incrementally?

For the next project our approach was simple: we kept Microsoft Word and our IDE open at the same time. We would spend two hours coding and 20 minutes updating the specifications. In the end we had fairly accurate specs and we managed to deliver on time for the first time ever.

But that didn’t solve all of our problems. We still had many issues that were only found in production –the system reflected the specs but the specs were wrong. One thing we learned is that there are many feedback cycles that are important during development, we managed to shorten one of them but we needed more. But to get this working we had to convince other departments –QA Department, BA Department, Senior Architects(tm)… – that this was a good thing.

And that never happened. After trying very hard to convince people that it would be nice to have short feedback most of us gave up and resigned. This company is still in business but they’ve lost 70% of their clients in the past couple of years. People I know there say that the problem is that their competitors launch a new product every year but them –still following the old process- can barely fix the bugs for the 10 years old version.

Getting Cyclic

But this taught me a lot. It showed me that even by creating one single barrier between my team and “the unknown” we could achieve a fantastic improvement. I was dying to try new barriers and see what we could achieve.

And after some years applying those I now believe that there are at least five “layers” of feedback while writing normal business software:

1. Development
2. Integration
3. Verification
4. Acceptance
5. Production

The first cycle is what happens in the developer’s box while she is writing code. A big waste of time –and waste causes feedback cycles to get longer- happens if it takes long to check if a piece of code you just changed works or not. Waiting for the build, starting up the application server or something like that just to see if what was showing a 2 now shows a 3 is a big time consumer. Having to manually click around your application to check if a bug was fixed is not only wasteful but also stupid.

And here comes the second cycle: how long does it take to integrate what you’ve done with what the rest of your team is working on? Conflicts and merges are a commonplace when build processes take long. That is terrible for feedback as a long integration cycle will make developers assume the code is working even though she only tested it on her machine.

Eventually someone has to verify your code, could either be a tester or some other developer doing code reviews. This is the verification cycle and it answers many questions about your changes. Does it really work? Does it break what was already there? Is it reliable? Follows the architecture? What about standards? Finding problems only at this stage is often a problem as it will require the developer to get back to her desk, fix it and go through the whole cycle again.

But, as we said before, following the spec may be good but if the spec is wrong your system will still be wrong. At some point you have to show it to the customer and see if his expectations were met, what I call the acceptance cycle. Usually this is done late in development and because of that it becomes such a painful thing that at this stage that developers will try to hide bugs and problems.

In the end there is always the production environment. It doesn’t really matter how much you test: production is always production and it will often show problems that you had no idea about. And, of course, it is often the most expensive place to find surprises.

Your Bricks

Those five cycles –and more cycles you may find- need to be short, they are the barriers that will keep the zombies of surprise out of your team. Let’s discuss some techniques and tools to help in this process.

For the first cycle, there are some very good tools that your team should be aware of. The first tools would be Test-Driven Development. Using TDD you set your goal before you write a piece of code and you know straight away when you met that goal. It will also help you testing the system, most code can be verified without firing up a browser and wasting time clicking and rendering.

To make sure this process is efficient you should also make sure you have a fast build process. The build should execute all tests and verifications. Every developer should run it before even thinking about checking in the code. That requires the build process to be short. 10 minutes is a good goal for most Java and .Net projects.

And fast builds will also help out in the integration cycle. A build that performs all the required verifications can be executed continuously in a dedicated machine every time a change is committed to the code base. This continuous integration process should also run the system in a non-dev machine –a machine without compilers, IDEs and other dev tools- to make sure that it is ready to be deployed.

This will also help you in the third cycle, verification. Testers should be able to write their own tests –using either a regular programming language or some kind of Domain-Specific Language- and those test cases should be part of the build and therefore run in the continuous integration process.

The best scenario is when you get the testers to work close to the developers, make them part of the development cycle. Developers should know immediately when they are breaking a test case that was working before. If you can’t make the tests written by the testers to be part of the developer normal build you have to make sure that at least they run -automatically and frequently- in the integration service.

And then we get to the acceptance cycle. This is extremely important: you don’t want to waste time developing something that is not what your client wants. Keep your client close to the development team. This is risky as the client may want to change things all the time but you should make sure that there are high-bandwidth channels between all stakeholders –and developers are stakeholders- and that you don’t spend too much money before realizing that you are building the wrong thing.

A nice way to collect early feedback from customers and other stakeholders is the Sandwich Model that we discussed here the other day:

One practice that I’ve used to help make this process more straightforward is the Sandwich Model for stories. Before a card is played the users, BAs, QAs and developers get together. The user and/or BA present the card to the technical people. First questions and answered and acceptance criteria is defined. After the developers finish the card they perform a walkthrough with BAs/users and QAs so that they can verify if it looks like done. If more work is required the developers get back to work, if the card was accepted it moves to the next stage.

To achieve better communication between developers and business experts it is also useful to make sure that they speak the same language. Domain-Driven Design can help you a lot with that as it aims to achieve a Ubiquitous Language used for developers, users and everyone else in that team.

And there will always be the production feedback cycle. Unfortunately in most of my projects these days it is not possible to replicate the production environment during development. There are techniques that can help you ease the pain but it is often impossible to replicate all the small details of production.

To minimize surprises in that level it is important to deliver increments instead of going for a big bang approach. Although everyone agrees with that, the reality is that most software development projects make just one or a couple of production rollouts. Don’t be like that, make sure that at least at the end of your iteration –but ideally always- your codebase is production-ready and try hard to get it deployed frequently. Not only you will see if there are problems that you didn’t find while in development but you will also be delivering value frequently to your customers –and maybe learning that the new features are not wanted at all!

But even though we can’t replicate the full production environment during day-to-day development, it is extremely important to have a production-like environment freely available to developers. Please avoid the “you must schedule an environment” behavior, have a box –or a virtual machine- somewhere where your developers, QAs and everyone else can log in, deploy some version of the application and mess around.

They’ve got inside!

As I said previously, these are not the only barriers you will find but in my experience these five are found in every project. An important thing to remember, though is that barriers exist only for one reason: to be broken.

For sure you want to minimize the number of times that those are broken but don’t even think about trying to protect your team from everything. Not only it is too expensive but also it is extremely inefficient as your barriers only protect you from the risks you know about.

Instead of doing that, you should focus in making sure that every time a barrier is broken everyone knows about that. Red lights, sirens, notification emails, broken builds… make sure that when something fails it fails out and loud.

In Case of Emergency Keep Those in Mind

In case you can’t remember all that we talked about in this ridiculously huge post, please keep four things in mind:

1 – Your team is not the centre of the universe: Software development is extremely important for a healthy business but it is far from being the only relevant moving part –or even the most important. You and your team have to learn how important you are, where do you sit in the value chain and how your work can help or hurt the business.

2 – The goal is not to be fail proof but to make it safe to fail: If you try to make sure that there is never a problem the first thing that probably will happen is that your team will stop innovating as they stop trying new things. You will also end up with different problems, you can’t avoid what you don’t know. Instead of fail proofing, make sure everyone is warned about problems as soon as possible.

3 – Do not assume people will give you support: I don’t care what your business card says or who hired you: if you try to change things you will make more enemies then friends. Keep in mind that it is very hard to criticize your team if you are delivering value, though. And be prepared.

4 – Keep your career goals apart from your team goals: It is probably a good thing that you want to progress in your carrier and get to a more important project or a raise. Make sure you don’t let this affect your leading style though. Your job is not to make management see how good you are, your job is to enable that team to deliver the best thing you possibly can. And in my experience a professional that can lead a development team is much more important than someone that can write the whole system by himself.