Coders at Work: My Review

Published by Phillip Calçado December 29th, 2009 in agile, books, software design and trends. 2 Comments

I just finished reading Peter Seibel’s new book, Coders at Work.

I was a bit skeptical at first. I only picked the book because of the big names on the cover and because Peter Siebel’s Practical Common Lisp is one of my favourite books on learn-a-new-programming-language. I thought that a book filled only with interviews with some of the big names in our industry would feel a bit too much like an issue of Who Magazine, with completely irrelevant questions like “Are you a Mac or PC?”. I was terribly wrong. The book is amazing.

Peter does a great job in guiding the interviewees into telling us about how they got into the industry and how they do their thing. For me it was really good to see how pretty much every single person interviewed -regardless of tech platform, background or expertise- tries to somehow come up with short feedback cycles during development.

And that makes it really funny to look at Joel Spolsky’s article on the book. He should have read a bit more before writing such a misinformed article, pretty much all of the interviewees write some other form of specification –being that a test, formal proof or a “test program”- before or while writing the program.

My favourite interview was with Bernie Cosell. Bernie is an old school hacker, he was part of what became the ARPANET. The way he describes his work really maps to what I do, and it can probably be summarised in this quote:

Peter Seibel: Have you heard of refactoring?
Bernie Cosell: No, what is that?
Peter Seibel: What you just described.

Duke Nukem Forever and Magic Bags of Money

Published by Phillip Calçado December 22nd, 2009 in agile, business, case study, digital media, economics, management, thoughtworks and trends. 2 Comments

Wired has a very interesting piece on how the Duke Nukem Forever project failed. It’s not only relevant because DNF is part of the nerd culture but also because it is a very interesting tale about a company that could not achieve a reasonable Definition of Done (DoD).

In software development we often talk about Done as if it was only a technical thing. It is not.

When developing a normal and bureaucratic system for a large bank or some similar client it is very unlikely that the DoD will be a problem at all –and if it becomes one it will most certainly be because of technical aspects as those mentioned in the list linked above. Things get a bit different when we are dealing with passionate people.

Projects with passionate people are always exciting. You can feel the energy, everyone is eager to help and improve the product. Everyone love what they do, really depend on the project to be a success or both. It is a fairly common situation for startups, creative agencies, the entertainment industry and media companies in general. And it is an awesome thing!

Having been to these kind of projects for most of my career, I know that the problems described in the Wired article are very common in there. Startups don’t want to present something that is not fully-featured and end up missing opportunities to enter a market, creative companies want to show their clients and users only the high-quality final version of the product, media companies are afraid to look like Muppets if their products are not one step ahead of everyone else since day one…

The Wired article seems to suggest that feature freeze is the key to overcome this problem, at least for the videogames industry:

It’s a dilemma all artists confront, of course. When do you stop creating and send your work out to face the public? Plenty of Hollywood directors have delayed for months, dithering in the editing room. But in videogames, the problem is particularly acute, because the longer you delay, the more genuinely antiquated your product begins to look — and the more likely it is that you’ll need to rip things down and start again. All game designers know this, so they pick a point to stop improving — to “lock the game down” — and then spend a frantic year polishing. But Broussard never seemed willing to do that.

I am no one to talk about how to manage a large code base written in esoteric C++ engines –I did that only once in my life and there is a reason I’m not going back there– and can’t really say if that is how that industry works/should be working; but I have to say that this is often not the best solution in the domains I mentioned before.

There are many strategies that I have used in this kind of environment. They often are either about showing that good enough is good enough or making sure people understand the costs of what they are doing.

A while ago I was employed by a big media conglomerate. Everything we did had to go through so many levels of approval and review that it could easily take months from a project kick-off meeting to when my development team received the graphic assets (PSD files, static HTML, etc.) and started writing code for it.

We had UAT servers where my team would deploy the application after each iteration, for the showcase and review. For those meeting we let the demo application point to the QA database. That database was populated with a dataset good enough to showcase the latest changes, but also had some garbage that was created by testers and developers in the process. One thing we realized at those reviews sessions was that the more database garbage displayed in a given demo the more rejected stories we would get in that meeting. After finding this out we decided to do something different.

For the upcoming releases, instead of using a UAT server to host the showcase we deployed the application straight to production. The old version would still be available in the expected URI but if you typed a URI that only our stakeholders knew you would get to the new version and that was pointing to the production database –sure you can’t do this with any application but most public facing websites wouldn’t have a problem. We used this secret URI in the showcases and by doing that we managed to get things approved much quicker: there was no noise in what the users were seeing.

But even with short feedback cycles there is still the problem of people changing the product all the time. Don’t get me wrong, it should be OK for a software development team to change things as requested, embrace change and all that. The problem is that, in the end of the day, if you change things all the time you will not deliver anything.

As Wired points out, that is a problem for the team, and an even bigger problem for the business as a whole.

Then a staff rebellion broke out. For longtime employees, the incessant delays posed two big problems. One was professional cred: Duke Nukem Forever was the only modern 3-D game some of them had worked on; if it didn’t ship soon, they’d have spent nearly a decade with nothing to show for it. The other was money. 3D Realms paid its designers less than many competitors, most notably id Software down the road. Broussard motivated them by offering profit-sharing. “Their business model was to pay the developers very low, but with a potential payday at the end that was pretty substantial,” says former employee Schuytema. As Duke Nukem Forever failed to arrive, so did that big payday.

As a consultant it is easier. I just remember the client that there is a bill to be paid in the end of the month and she would probably spend her budget more effectively if she tries to get some money from the system as soon as she can and then think about making it better. As a permanent employee for those organizations, I tried many different ways to convince people of the same thing. Turns out that this is really hard. Often business people are not in the same hierarchy as your team and you act as a service provider to them, with the difference that there is no bill to be explicitly paid. They just don’t care.

One thing that works in this scenario is the poker chips technique developer by Jason Yip. I’ve used that both as a consultant -pairing with Jason himself- and as a permanent employee for organizations -we didn’t actually have poker chips but tokens representing development hours.

It is amazing how one’s mindset changes when they stop thinking that there is a magic bag of infinite money and have to rationalize their decisions. You should give this a try.

I Wish I Knew That Before Getting This Job – Slides and (Long) Notes

Published by Phillip Calçado November 24th, 2009 in agile, business, c#, components, domain driven design, domain specific languages, economics, events, java, layers, management, object orientation, software architecture, software design, thoughtworks and trends. 0 Comments

As I said here before I was in Brazil some weeks ago to present at a conference. I had a really great time over there with some amazing people and would like to thank Caelum for their hard work in creating such a great conference. I’m making the slide deck and notes available in my Brazilian Portuguese website and thought it would be nice to translate them to English and publish it here too.

The following paragraphs are a consolidated -but really long!- version of my slide notes. The slide deck you can find in slideshare:

I Wish i Knew that Before Getting This Job
View more documents from Phillip Calçado.

In my experience it is really hard to translate material I write for the Brazilian community to English so please be advised that some bits may not make much sense, please report problems in the comments or drop me a mail.

The world has changed

If you’ve been working in this industry for a while you probably know what I am talking about: nowadays things are harder. Years ago our job was to build systems, today our job is to make sure old systems talk to each other.

When you get to the Tech Lead role in a company or project you often come full of ideas, excitement and plans. And you will see all these being destroyed by your customers and colleagues. No one will trust you, no one thinks you are able to fix those problems; no one will give you any opportunity to change anything for real –even though they said that they would.

The problem is that you clients have been bitten before. They’ve played this “let’s develop a new system” game and hate it! They’ve got their plans and dreams crushed by a bunch of code cowboys that came in with wonderful tools and techniques and delivered that buggy piece of crap -and charged 2x the agreed price. Your customers can’t take any risks. No, not this time. Not again. Regardless of what you agreed they will tell you to do what they think works.

And in this situation what happen to the standard coder? She just gives up. It’s so hard to change anything that let’s just join the dance and play the 9-to-5-job game. She starts writing all those silly specs that the client asked for, she asks the Ivory Tower architect about every single stupid detail in her application… and she starts producing pretty much nothing useful.

But after spending that many nights awake reading books, blogs and trying stuff out… after fighting to deliver on schedule and on budget for so long… will you really accept this?

There are options, resistance is not futile! It is not going to be easy. You will be openly attacked, secretly hated and people will be terribly picky even about what you wear or how you say things. But you can survive as long as you keep in mind that there is only one thing that guarantees survival: deliver value. Make sure your customer reaches his goals –and those are not to have a complete spec for the system but to earn them some money.

To make sure your customers trust you it is important that you deliver value frequently –something your client can count on. It is also important that you deliver at the right time –do not miss a deadline. And it is important that you are never late –make sure you have realistic estimates and know when something is going to be late.

But to get those there is a big enemy to fight: surprises. If there were no surprises in software development most teams would be able to deliver quality software right on time. Surprises are what make those 10% left in a task last forever. As a Tech Lead your goal is to make sure your team is delivering in a timely fashion. As a Tech lead it is your goal to avoid surprises.

And to avoid those, my suggestion is that you build barriers to keep them away. Like in any good zombie movie.

The House of the 17 Specs

That reminds me of when I used to work for this big product company. They had a funny process where before the first line of code was written you had to come up with a spec for the full product, then one spec for each of the modules. Each of those documents should be paired with a test strategy/case document. I think that if you followed the full process –and this was the usual way- for a large system one would write 17 documents before the first line of code.

In theory that would not be a problem, the specs were detailed enough so, theoretically, the coding task would be just to convert what was written in there to Java, C# or whatever we were using for that piece. That wasn’t the case, as you probably imagined. There was a huge impedance mismatch between the world described in those specs and the real world. The library we were be using would happen not to be thread-safe, the operating system that the client wants would be missing that very system call we relied on or something like that.

After so many weekends in the office trying to make the real world look more like the spec we started to think about doing things a bit differently. We came up with all these cool ideas about how to improve our development process and took them to middle and senior management types. Of course that didn’t work, management was definitely not interested in what a bunch of punks with a couple of years of experience in that company had to say about their 10-years-delivering-software process.

We got really upset and for the next couple of weeks I would see my team visiting job boards for most of the day. Eventually someone came with an interesting idea: the process says that we must have such and such document but it doesn’t actually say that the docs should be all ready before coding starts. Leaving the documents to the end will probably mean that they will never be finished and the team is going to be in trouble… but what about we try to write them incrementally?

For the next project our approach was simple: we kept Microsoft Word and our IDE open at the same time. We would spend two hours coding and 20 minutes updating the specifications. In the end we had fairly accurate specs and we managed to deliver on time for the first time ever.

But that didn’t solve all of our problems. We still had many issues that were only found in production –the system reflected the specs but the specs were wrong. One thing we learned is that there are many feedback cycles that are important during development, we managed to shorten one of them but we needed more. But to get this working we had to convince other departments –QA Department, BA Department, Senior Architects(tm)… – that this was a good thing.

And that never happened. After trying very hard to convince people that it would be nice to have short feedback most of us gave up and resigned. This company is still in business but they’ve lost 70% of their clients in the past couple of years. People I know there say that the problem is that their competitors launch a new product every year but them –still following the old process- can barely fix the bugs for the 10 years old version.

Getting Cyclic

But this taught me a lot. It showed me that even by creating one single barrier between my team and “the unknown” we could achieve a fantastic improvement. I was dying to try new barriers and see what we could achieve.

And after some years applying those I now believe that there are at least five “layers” of feedback while writing normal business software:

  1. Development
  2. Integration
  3. Verification
  4. Acceptance
  5. Production

The first cycle is what happens in the developer’s box while she is writing code. A big waste of time –and waste causes feedback cycles to get longer- happens if it takes long to check if a piece of code you just changed works or not. Waiting for the build, starting up the application server or something like that just to see if what was showing a 2 now shows a 3 is a big time consumer. Having to manually click around your application to check if a bug was fixed is not only wasteful but also stupid.

And here comes the second cycle: how long does it take to integrate what you’ve done with what the rest of your team is working on? Conflicts and merges are a commonplace when build processes take long. That is terrible for feedback as a long integration cycle will make developers assume the code is working even though she only tested it on her machine.

Eventually someone has to verify your code, could either be a tester or some other developer doing code reviews. This is the verification cycle and it answers many questions about your changes. Does it really work? Does it break what was already there? Is it reliable? Follows the architecture? What about standards? Finding problems only at this stage is often a problem as it will require the developer to get back to her desk, fix it and go through the whole cycle again.

But, as we said before, following the spec may be good but if the spec is wrong your system will still be wrong. At some point you have to show it to the customer and see if his expectations were met, what I call the acceptance cycle. Usually this is done late in development and because of that it becomes such a painful thing that at this stage that developers will try to hide bugs and problems.

In the end there is always the production environment. It doesn’t really matter how much you test: production is always production and it will often show problems that you had no idea about. And, of course, it is often the most expensive place to find surprises.

Your Bricks

Those five cycles –and more cycles you may find- need to be short, they are the barriers that will keep the zombies of surprise out of your team. Let’s discuss some techniques and tools to help in this process.

For the first cycle, there are some very good tools that your team should be aware of. The first tools would be Test-Driven Development. Using TDD you set your goal before you write a piece of code and you know straight away when you met that goal. It will also help you testing the system, most code can be verified without firing up a browser and wasting time clicking and rendering.

To make sure this process is efficient you should also make sure you have a fast build process. The build should execute all tests and verifications. Every developer should run it before even thinking about checking in the code. That requires the build process to be short. 10 minutes is a good goal for most Java and .Net projects.

And fast builds will also help out in the integration cycle. A build that performs all the required verifications can be executed continuously in a dedicated machine every time a change is committed to the code base. This continuous integration process should also run the system in a non-dev machine –a machine without compilers, IDEs and other dev tools- to make sure that it is ready to be deployed.

This will also help you in the third cycle, verification. Testers should be able to write their own tests –using either a regular programming language or some kind of Domain-Specific Language- and those test cases should be part of the build and therefore run in the continuous integration process.

The best scenario is when you get the testers to work close to the developers, make them part of the development cycle. Developers should know immediately when they are breaking a test case that was working before. If you can’t make the tests written by the testers to be part of the developer normal build you have to make sure that at least they run -automatically and frequently- in the integration service.

And then we get to the acceptance cycle. This is extremely important: you don’t want to waste time developing something that is not what your client wants. Keep your client close to the development team. This is risky as the client may want to change things all the time but you should make sure that there are high-bandwidth channels between all stakeholders –and developers are stakeholders- and that you don’t spend too much money before realizing that you are building the wrong thing.

A nice way to collect early feedback from customers and other stakeholders is the Sandwich Model that we discussed here the other day:

One practice that I’ve used to help make this process more straightforward is the Sandwich Model for stories. Before a card is played the users, BAs, QAs and developers get together. The user and/or BA present the card to the technical people. First questions and answered and acceptance criteria is defined. After the developers finish the card they perform a walkthrough with BAs/users and QAs so that they can verify if it looks like done. If more work is required the developers get back to work, if the card was accepted it moves to the next stage.

To achieve better communication between developers and business experts it is also useful to make sure that they speak the same language. Domain-Driven Design can help you a lot with that as it aims to achieve a Ubiquitous Language used for developers, users and everyone else in that team.

And there will always be the production feedback cycle. Unfortunately in most of my projects these days it is not possible to replicate the production environment during development. There are techniques that can help you ease the pain but it is often impossible to replicate all the small details of production.

To minimize surprises in that level it is important to deliver increments instead of going for a big bang approach. Although everyone agrees with that, the reality is that most software development projects make just one or a couple of production rollouts. Don’t be like that, make sure that at least at the end of your iteration –but ideally always- your codebase is production-ready and try hard to get it deployed frequently. Not only you will see if there are problems that you didn’t find while in development but you will also be delivering value frequently to your customers –and maybe learning that the new features are not wanted at all!

But even though we can’t replicate the full production environment during day-to-day development, it is extremely important to have a production-like environment freely available to developers. Please avoid the “you must schedule an environment” behavior, have a box –or a virtual machine- somewhere where your developers, QAs and everyone else can log in, deploy some version of the application and mess around.

They’ve got inside!

As I said previously, these are not the only barriers you will find but in my experience these five are found in every project. An important thing to remember, though is that barriers exist only for one reason: to be broken.

For sure you want to minimize the number of times that those are broken but don’t even think about trying to protect your team from everything. Not only it is too expensive but also it is extremely inefficient as your barriers only protect you from the risks you know about.

Instead of doing that, you should focus in making sure that every time a barrier is broken everyone knows about that. Red lights, sirens, notification emails, broken builds… make sure that when something fails it fails out and loud.

In Case of Emergency Keep Those in Mind

In case you can’t remember all that we talked about in this ridiculously huge post, please keep four things in mind:

1 – Your team is not the centre of the universe: Software development is extremely important for a healthy business but it is far from being the only relevant moving part –or even the most important. You and your team have to learn how important you are, where do you sit in the value chain and how your work can help or hurt the business.

2 – The goal is not to be fail proof but to make it safe to fail: If you try to make sure that there is never a problem the first thing that probably will happen is that your team will stop innovating as they stop trying new things. You will also end up with different problems, you can’t avoid what you don’t know. Instead of fail proofing, make sure everyone is warned about problems as soon as possible.

3 – Do not assume people will give you support: I don’t care what your business card says or who hired you: if you try to change things you will make more enemies then friends. Keep in mind that it is very hard to criticize your team if you are delivering value, though. And be prepared.

4 – Keep your career goals apart from your team goals: It is probably a good thing that you want to progress in your carrier and get to a more important project or a raise. Make sure you don’t let this affect your leading style though. Your job is not to make management see how good you are, your job is to enable that team to deliver the best thing you possibly can. And in my experience a professional that can lead a development team is much more important than someone that can write the whole system by himself.

Speaking in Brazil and Last Year’s Slide Deck

Published by Phillip Calçado October 24th, 2009 in agile, domain driven design, events and thoughtworks. 3 Comments

As I said here earlied this year, next month I will be presenting in a conference called Caelum Day. This will be held in Rio de Janeiro, my home town, and I’m really excited to be there.

My presentation will be a keynote on the role of a Tech Lead and what I wish I knew before getting my first gig as one. Just like last year, I will also be helding a Workshop on Domain-Driven Design. I really tried to change the subject of the workshop to something else -DDD is getting a bit boring- but it turns out that there is a massive demand for this.

The presentation and workshops will be held in Portuguese. As most readers of this blog don’t really understand coisa nenhuma of Portuguese I jsut translated last year’s slide deck.

The Curse of The Agile Software Factory
View more documents from Phillip Calçado.

I’m not quite happy with the translation, some expressions require a Brazilian background and sound dull in English. Anyway it’s better than nothing :)

Don’t Trust Fake IDs

Published by Phillip Calçado October 12th, 2009 in c#, domain driven design, java, layers, object orientation, software architecture, software design and web. 5 Comments

After so many years I thought that this was a dead topic but recently I saw at least two reasonably experienced developers having trouble with object identity and thought that it would be good to write about it.

Most languages define some kind of equality operator on objects, in Java, for example, that is the equals method. In Effective Java, Joshua Bloch says that the easiest thing to do is not to implement an equals method at all, but you should implement one when “the notion of logical equality differs from mere object identity” and your superclass doesn’t have already implemented a suitable equality comparisson.

I think most people understand this when it comes to numbers, dates, strings and other Value Objects but I’ve seemed people having trouble understanding what object identity means for Entities.

Entities are objects whose identity is defined by rules in the domain we are modelling. In his Domain Driven Design book, Eric Evans defines objects following this pattern as objects that change state over time but keep their identity. As the object’s identity comes from the business domain reference equality is too fragile to properly compare objects.

Problems with fragile object identity are fairly well-documented in the Object-Orientation literature. In 1986, Setrag N. Khoshafian and George P. Copeland wrote a paper titled Object Identity. In this document they present a list of known problems with fragile identity, some of these are:

  • Multiple references to the object: It is very common to have multiple instances of a given object at a given time and it should be possible to infer that those multiple instances are actually the same object.
  • Uniqueness Not Respected: Objects are abstractions and as such they don’t model every single detail present in the modelled concept. If we don’t define identity correctly we may end up with a model in which different objects are seem as equal.
  • Data and Identity Confusion: The identity of an object should not be allowed to change but, as Evans says, data will change over time. There should be a clear separation of what is data and what is identity.

People use fragile things for object identity for multiple reasons, let’s address two of those.

I refuse to write test-only code!

One of the problems pointed out by those using reference equality instead of object equality is that equals() is commonly not called by anything but tests and writing production code that is used only in tests is a bad smell.

Let’s look at a very simple class and its tests:

class Point {
	private int y, x;

	public Point(int x, int y) {
		this.y = y;
		this.x = x;
	}

	public int getY() {
		return y;
	}

	public int getX() {
		return x;
	}
}

class Square {
	private Point letfTop, leftBottom, rightTop, rightBottom;

	public Square(Point a, Point b, Point c, Point d) {
		this.letfTop = a;
		this.leftBottom = b;
		this.rightTop = c;
		this.rightBottom = d;
	}

	public Point getRightTop() {
		return rightTop;
	}

	public Point getRightBottom() {
		return rightBottom;
	}

	public Point getLeftTop() {
		return letfTop;
	}

	public Point getLeftBottom() {
		return leftBottom;
	}
}

public class SquareTest {
	@Test
	public void ShouldHavePointsInTheRightPlace() {
		Point first = new Point(1, 1);
		Point second = new Point(2, 2);
		Point third = new Point(3, 3);
		Point fourth = new Point(4, 4);
		Square square = new Square(first, second, third, fourth);

		// how do we assert that the right points are in place?
	}
}

Althought the value of the test above is questionable, it exposes the problem: How do we make sure that the points are in the right place? As we said before we could implement an equals() method in the Point class in order to be able to compare X and Y, and then do something like:

assertEquals(first, square.getLeftTop());
assertEquals(second, square.getLeftBottom());
assertEquals(third, square.getRightTop());
assertEquals(fourth, square.getRightBottom());

That looks OK but we just wrote a piece of production code (Point.equals()) that are used only by tests. I personally don’t think of this as a big deal -for me equals, hashcode and toString are part of the language protocol- but I can understand why people don’t like to do that.

So maybe we can just use the == operator as we wouldn’t have to write any extra code for that. Using that operator our assertions wold become something like:

		assertTrue(first == square.getLeftTop());
		assertTrue(second == square.getLeftBottom());
		assertTrue(third == square.getRightTop());
		assertTrue(fourth == square.getRightBottom());

That would work but the problem is that your tests are relying on the wrong thing. Think about that for a second: in the test above do you really care that the same instance is in the top left corner? Suppose we have new requirements and for some reason we can’t store the data as Points. The class contract didn’t change at all, the fact that we can’t use points internally is just an implementation details inside Square: 

class Square {

	class SimplePoint {
		public int x, y;

		public SimplePoint(Point p) {
			this.x = p.getX();
			this.y = p.getY();
		}

		public Point toPoint() {
			return new Point(x, y);
		}
	}

	SimplePoint letfTop, leftBottom, rightTop, rightBottom;

	public Square(Point a, Point b, Point c, Point d) {
		this.letfTop = new SimplePoint(a);
		this.leftBottom = new SimplePoint(b);
		this.rightTop = new SimplePoint(c);
		this.rightBottom = new SimplePoint(d);
	}

	public Point getRightTop() {
		return rightTop.toPoint();
	}

	public Point getRightBottom() {
		return rightBottom.toPoint();
	}

	public Point getLeftTop() {
		return letfTop.toPoint();
	}

	public Point getLeftBottom() {
		return leftBottom.toPoint();
	}
}

Now our assertions based on the == operator fail and if you are testing the behavior of the class –as opposed to its implementation- this shouldn’t happen.

Tests should follow the Single Responsibility Principle and have only one reason to change. In that case the only reason for that test to change should be if the points are not in the places we expect them to be. It doesn’t matter if they are stored internally as plain Points, if they are stored as integers over a matrix or even serialized to disk: the test should not fail if the storage strategy has to be modified.

This is the kind of assertion that creates extremely fragile test suites. A fragile test suite will require constant maintenance and will break with no clear reason. Tests start to cause more problems than help development and it is very common for the team to think about either testing less or dropping tests entirely.

As we saw before, the domain defines what makes on object equivalent to another. In the case above the Point class, it is a Value Object and what that means is that identity for that object is given by the value of its attributes, in this specific case that would be the integers Y and X.

If you don’t want to add an equals() method to your class at least make sure you rely on the real identity of the object and not in some accident:

public class PointAssertions {
	public static void assertPointsEquals(Point one, Point another) {
		assertEquals("Expected [" + another + "] to have X equals to ["+ one.getX() + "]", one.getX(), another.getX());
		assertEquals("Expected [" + another + "] to have Y equals to ["+ one.getY() + "]", one.getY(), another.getY());
	}
}

And use that in your tests:

assertPointsEquals(first, square.getLeftTop());
assertPointsEquals(second, square.getLeftBottom());
assertPointsEquals(third, square.getRightTop());
assertPointsEquals(fourth, square.getRightBottom());

So although you had to write some more code you didn’t change production code just to make your tests pass. Also you are probably going to reuse this type of code in other tests that deal with points.

It’s a kind of magic!

Another cause for not correctly defining object identity is when people are using things like Hibernate or NHibernate that, in some cases, return the same instance over multiple calls. One quick example would be:

public class User
    {
        public string Login { get; set; }
        public string Name { get; set; }
        //business logic should be here somewhere
    }
    public class Group
    {
        public User Admin { get; set; }
        public bool CanBeChangedBy(User u)
        {
            return u == Admin;
        }
    }
    public class UserRepository : HibernateRepository
    {
        public User FindByLogin(string login)
        {
            return session.Load<User> (login);
        }
    }

    public class GroupRepository : HibernateRepository
    {
        public Group FindById(string groupId)
        {
            return session.Load<Group> (groupId);
        }
    }

    public class GroupsController
    {

//Some other stuff

        public void Change(string groupId, string login)
        {
            var userTryingToChange = userRepository.FindByLogin(login);
            var groupTobeChanged = groupRepository.FindById(groupId);
            if (groupTobeChanged.CanBeChangedBy(userTryingToChange))
            {
                //do something that lets the guy change the group
            }
            else
            {
                // show error message or something like that
            }
        }
    }

In this case NHibernate happens to return the same object for every call. This code is extremely brittle, though. If you look at NHibernate’s documentation (or Hibernate’s) you will see that multiple queries using the same Session object are guaranteed to return the same instance. The problem is that if you use more than one Session your objects will be different.

That may sound like a small problem given that most web applications (and most applications are web applications) use only one Session per request. But the real problem hides in the fact that now your Business Layer depends on an implementation detail of the Persistence Layer.

Changes in the implementation of the Persistence Layer will directly affect your Business Layer and this is the kind of accidental complexity that creates buggy and unreliable software.

And it gets worse. The original Hibernate’s documentation adds some more information in a section named “Considering object identity”:

11.1.3. Considering object identity
An application can concurrently access the same persistent state in two different Sessions. However, an instance of a persistent class is never shared between two Session instances. It is for this reason that there are two different notions of identity:

Database Identity
foo.getId().equals( bar.getId() )

JVM Identity
foo==bar

For objects attached to a particular Session (i.e., in the scope of a Session), the two notions are equivalent and JVM identity for database identity is guaranteed by Hibernate. […]
However, an application that uses == outside of a Session might produce unexpected results. This might occur even in some unexpected places. For example, if you put two detached instances into the same Set, both might have the same database identity (i.e., they represent the same row). JVM identity, however, is by definition not guaranteed for instances in a detached state. The developer has to override the equals() and hashCode() methods in persistent classes and implement their own notion of object equality.

Now what that means is that I cannot detach my object from Hibernate to, for example, store it in a HTTP session. Even if you don’t use HTTP sessions, that means you can’t cache objects as the cached copy will be a different instance than those returned by a Session.

Suddenly your architecture is extremely constrained just because someone decided that implementing a proper object comparisson method was too much effort.

Before anyone writes a comment with YAGNI in it I will say once more that the simplest thing is different from the hackiest thing. Relying on how your Persistence Layer happens to be implemented is not being simple, it is coupling your components in a way that even small changes requrie hard work.

No half-baked IDs

In the first example we saw a situation where implementing proper object equality wasn’t necessary for the production code. In that case you just have to make sure that your tests are using something reliable when comparing objects.

In the second case we have a need for object identity and using weak identity may save some lines of code but ultimately creates a crittle and inflexible architecture.

Those are just a couple of examples of problems you may find. The most important thing I can say is that every time you need to compare objects you should make sure to use strong identity. You can write the comparisson operations yourself or use a library or anything, just make sure you follow the contract that your language imposes on objects.

Note: I didn’t say or implement anything regarding Hashcode methods here for the sake of simplicity. Anyway I suggest that you follow the recommendations for your language.

Ubiquitous Language, Tiny Types and Responsibility

Published by Phillip Calçado August 21st, 2009 in books, c#, domain driven design, object orientation and software design. 8 Comments

As a consultant I have to log my hours in a timesheet application. This could be a test inside the timesheet app:

 [TestFixture]
    public class BillableTimeTest
    {
        [Test]
        public void ShouldhaveCorrectNumberOfWorkedHours()
        {
            var beerOClock = 17;
            var earlyInTheMorning = 9;
            var workDay = new BillableTime(earlyInTheMorning, beerOClock);
            Assert.That(workDay.Hours, Is.EqualTo(8));
        }

        [Test]
        public void ShouldRejectNonBusinessHours()
        {
            var afterHours = 23;
            var earlyInTheMorning = 9;
            Assert.Throws<ArgumentException>(() => new BillableTime(earlyInTheMorning, afterHours));
        }
    }

Given the test above it looks that in this system you can only log billable hours if they between 9a.m. and 5p.m. The implementation of the BillableTime object could be something like this:

    public class BillableTime
    {
        private readonly int _from;
        private readonly int _to;

        public BillableTime(int from, int to)
        {
            ValidateTimePeriod(from, to);
            _from = from;
            _to = to;
        }

        private void ValidateTimePeriod(int from, int to)
        {
            if (from < 9 || to > 17) throw new ArgumentException("BillableTime only accepts business hours (9am-5pm)");
        }

        public int Hours { get { return _to - _from; } }
    }

Let’s move on. If the BillableTime object is used to log only office hours we will probably have an object that represents overtime. Here is the test for our Overtime object:

    [TestFixture]
    public class OvertimeTest
    {
        [Test]
        public void ShouldhaveCorrectNumberOfWorkedHours()
        {
            var beerOClock = 17;
            var wayTooLate = 22;
            var workDay = new Overtime(beerOClock, wayTooLate);
            Assert.That(workDay.Hours, Is.EqualTo(5));
        }

        [Test]
        public void ShouldRejectIfNotInOVertimePeriod()
        {
            var afterHours = 23;
            var earlyInTheMorning = 9;
            Assert.Throws<ArgumentException>(() => new Overtime(earlyInTheMorning, afterHours));
        }
    }

You may have guessed that Overtime is implemented as something like this:

    public class Overtime
    {
        private readonly int _from;
        private readonly int _to;

        public Overtime(int from, int to)
        {
            ValidateTimePeriod(from, to);
            _from = from;
            _to = to;
        }

        private void ValidateTimePeriod(int from, int to)
        {
            if (from < 17 ) throw new ArgumentException("OvertimeTime only starts after 5pm");
        }

        public int Hours { get { return _to - _from; } }
    }

Code like the examples above is pretty common but there is a problem in there. Right now a BillableTime represents a period of time in which someone was working AND controls what constitutes “business hours”. The overtime object represents a period of time in which someone did some work after ours AND define what “after hours” means. When you can’t describe an object’s responsibilities without using and there is a chance that your model has something wrong in it.

In an Object-Oriented system it is generally a bad idea to have the implementation of a concept spread across multiple places. In a language like C# the best thing to do is often to have this implemented in its own class:

 [TestFixture]
    public class TimeIntervalTest
    {
        [Test]
        public void ShouldBeBusinessHoursIfBetween9AmAnd5Pm()
        {
            var beerOClock = 17;
            var earlyInTheMorning = 9;
            var businessHoursInterval = new TimeInterval(earlyInTheMorning, beerOClock);
            Assert.That(businessHoursInterval.Kind, Is.EqualTo(TimeInterval.Kinds.BusinessHours));
        }

        [Test]
        public void ShouldBeOvertimeIfAfter5Pm()
        {
            var beerOClock = 17;
            var wayTooLate = 22;
            var businessHoursInterval = new TimeInterval(beerOClock, wayTooLate);
            Assert.That(businessHoursInterval.Kind, Is.EqualTo(TimeInterval.Kinds.AfterHours));
        }

        [Test]
        public void ShouldHaveCorrectNumberofHours()
        {
            var startingTime = 9;
            var oneHourAfter = 10;
            var businessHoursInterval = new TimeInterval(startingTime, oneHourAfter);
            Assert.That(businessHoursInterval.Hours, Is.EqualTo(1));
        }

    }

    public class TimeInterval
    {
        private readonly int _from;
        private readonly int _to;
        private readonly Kinds _kind;

        public TimeInterval(int from, int to)
        {
            _from = from;
            _to = to;

            if (from >= 9 && to <= 17)
                _kind = Kinds.BusinessHours;
            else
            {
             _kind = Kinds.AfterHours;
            }
        }

        public enum Kinds { AfterHours, BusinessHours }

        public Kinds Kind
        {
            get { return _kind; }
        }

        public int Hours { get { return _to - _from; } }
    }

You can get fancy and create multiple types but this implementation is probably enough for this example. Now we have a class representing the time interval. Notice that this concept was present in our domain since the very beginning; we just didn’t have it explicitly.

But why would explicitly make anything better? Suppose that you are working on this code base and suddenly you need to change something. Say that your company decided that counting hours is not acceptable and that they want to track everything as minutes.

In the first example you would have to change both Overtime and BillableTime classes as they share the implementation of the concept. In the second example you would have to change only the class that actually represents the concept.

But even to realize which classes you would have to change it would take possibly hours in searching through the code base and tests to figure out all the places that implement time interval related logic. Something that for the business is very simple –as the business considers time interval to be a “strong concept”- would take ages to be implemented, possibly frustrating your stakeholders.

I like Eric Evans’ take on these situations. In his classical Domain Driven Design book he says:

[…]A deep model has power because it contains the central concepts and abstractions that can succinctly and flexibly express essential knowledge of the users’ activities, their problems, and their solutions. The first step is to somehow represent the essential concepts of the domain in the model. Refinement comes later, after successive iterations of knowledge crunching and refactoring. But this process really gets into gear when an important concept is recognized and made explicit in the model and design.
Many transformations of domain models and the corresponding code happen when developers recognize a concept that has been hinted at in discussion or present implicitly in the design, and they then represent it explicitly in the model with one or more objects or relationships.
Occasionally, this transformation of a formerly implicit concept into an explicit one is a breakthrough that leads to a deep model. More often, though, the breakthrough comes later, after a number of important concepts are explicit in the model; after successive refactorings have tweaked their responsibilities repeatedly, changed their relationships with other objects, and even changed their names a few times. Everything finally snaps into focus. But the process starts with recognizing the implied concepts in some form, however crude.
[…]
Listen to the language the domain experts use. Are there terms that succinctly state something complicated? Are they correcting your word choice (perhaps diplomatically)? Do the puzzled looks on their faces go away when you use a particular phrase? These are hints of a concept that might benefit the model.

The example was extremely simple but it’s not that different from real-world situations. What I’ve found in many projects is that people use things like metadata-based validators to hide business logic. Here is an example using NHibernate Validator:

 public class Contract
    {
        private int _lengthInYears;

        [Range(Min=1, Max = 3)]
        public int length
        {
            get { return _lengthInYears; }
            set { _lengthInYears = value; }
        }
    }

I think those validator frameworks can really help in many situations but when I look at the code above I can’t get some very important concepts. The method tells me it accepts an integer (and that means a lot of numbers, including negatives), but then it rejects most of them. In this case wouldn’t it be better if we explicitly say what we accept?

public class Contract
    {
        public Years length { get; set; }
    }

    public class Years
    {
        public int NumberOf { get; set; }

        public Years(int numberOf)
        {
            NumberOf = numberOf;
            if (numberOf < 1 || numberOf > 3)
                throw new ArgumentException();
        }
    }

It is quite clear that this also adds a lot of noise as you have to wrap and unwrap integers into Years. I am not saying that you must validate all your integers, strings and the like every time. What I am trying to say is that whenever you have this kind of logic in a validator or something else it is very likely that the use of metadata is actually hiding information about an important domain concept. I’d rather have it explicit in my code base than hidden in some metadata.

And this gets us into theTiny Types (or Micro Types) idea. I like the concept and I use it when it makes my domain more explicit. What really bugs me is when I see something like:

    public class User
    {
        Name name { get; set; }
        //other stuff
    }

    public class Name
    {
        public string Value { get; set; }

        public Name(string value)
        {
            Value = value;
        }
    }

The tiny types in the example below not only are often not relevant to the Ubiquitous Language –I don’t think I ever worked in a system where name wasn’t a string… any string!- but also most languages (C#, Java, Ruby…) make it so bureaucratic to create and maintain those little types that in the end you waste a lot of effort doing something that is not relevant at all.

It is a wonderful thing to make your domain concepts explicit using types. Just make sure you don’t overdo it, it’s not worth the effort.

3 Things Agile Teams Should Care About

Published by Phillip Calçado August 17th, 2009 in agile, management, software architecture, software design and thoughtworks. 3 Comments

Regardless of what kind of team you have, it is a given that if you keep the wrong focus you are going to be in some kind of trouble. It is very important for a team to have a clear understanding of the project’s goals and constraints. That sounds like PM-speak but it’s not only relevant for business/management side of things but for the technical team as well. I’ve been to many projects in big trouble just because the technical team focused on the wrong goals.

In short, there are three things that I think all software development teams should be obsessed with:

1 – What does “Done” Mean?

The whole team has to get a shared understanding of what done means. This must be defined by the team, including non-technical people and it is expected that no one accepts half-baked software.

This is not about test coverage or any other metrics. This is about having software that does what the user wants with the quality that the team considers acceptable.

One practice that I’ve used to help make this process more straightforward is the Sandwich Model for stories. Before a card is played the users, BAs, QAs and developers get together. The user and/or BA present the card to the technical people. First questions and answered and acceptance criteria is defined. After the developers finish the card they perform a walkthrough with BAs/users and QAs so that they can verify if it looks like Done. If more work is required the developers get back to work, if the card was accepted it moves to the next stage.

Other practice, one that I would expect all teams that call themselves Agile to follow, is to always have Potentially Shippable Software, from day zero. Unfortunately it is very common for teams to have a product that can only be tested in a development environment and can’t be deployed to a server or workstation.

I am not saying that you need a fancy Capistrano deployment script or to make your application WebStart/ClickOnce ready. What I am saying is that there must be an easy way to install and run your application in a production-like environment -and production like probably means no compilers, no build tools and no damn Visual Studio.

An user story is only done when it can be tested in the production-like environment, before that it is only “90% Done”.

2 – Flow

After deciding what Done means, the team should be concerned about how fast things are getting there. Agile software development is based on short feedback cycles. When cards move from Doing to Done at a reasonable Velocity you empower the stakeholders to make the right business decisions. When changing is cheap you can delay technical commitments until you have enough information to make your decision. If you don’t have a short feedback cycle you may be iterative and may be incremental but for me you are certainly not Agile.

Feedback cycle is often related to technical issues and the team must make sure that those issues are not slowing it down. My main metric on that is the Rule of The Second Card: The first card played on a given area of the code base is expected to take a long time. The second card should take at most half of the first card’s time.

This is just a rule of thumb but it helps measuring the pain. The first card played in a new area will always require a lot of thinking and learning. Even when the simplest solution is implemented I expect it to take longer as people learn how to connect the dots –remember: it’s about the Simplest solution, not the Hackiest.

The second card shouldn’t take that long. User stories represent small chunks of change and the second one is often just an addition to the first (e.g. First is “User should search by state”; Second is “User should search by state and city”) and if the first card was played with reasonable quality standards then we can only expect that it has built the scaffold for the second card.

If your second card takes about the same time as the first one -or even longer!- then your team may be in trouble. It may be that the pair playing the first card took too many shortcuts in order to get the first card delivered and produced a lot of technical debt. If it becomes clear that this is a common pattern in your project you guys should sit down and work out what is happening.

Another thing to pay attention to is the build time. The build is the main feedback cycle for a developer; builds should not be long and should not fail for unknown reasons. There are many techniques one can use to keep build times slow, we are going to talk about them in future posts.

One level above the build time is how Continuous Integration is done in your project. Julio Maia has a very interesting article describing its importance.

3 – Readability

It may be odd to see this as the third most important thing in a blog from someone that works for ThoughtWorks. I do believe that writing code that is readable is extremely important, I just don’t think that, in a project, it comes before the two other topics we previously talked about.

The Tech Lead should make sure that the code being written is readable by people that have enough knowledge. What is “enough” means depends on who you ask but for me there are at least two groups in this situation:

  1. People that are proficient in the programming platform (e.g. Java, .Net, Rails, etc.) but not necessarily in the domain
  2. People that are programmers, know the domain but don’t master the platform used in that system.

There are multiple ways of doing that. In ThoughtWorks we love to have people coming to our projects and working with us for a couple of days. When a skilled ThoughtWorker has trouble trying to understand the code this is a very bad sign and we need to work on that. I’ve heard about people that are going even further and asking for people from other companies to come and pair with the development team for some time.

Everything is Connected

The items above are presented as three independent units but that is not the case. There are many dependencies between items and they become clear as you start thinking about them.

Of course the list is not exhaustive, there are many other technical aspects that must bee considered in a project, but in my experience if you get at least those three right it is much more likely that your project will be in good shape.

ThoughtWorks Away Day Presentation: Common Myths about Type Systems

Published by Phillip Calçado August 12th, 2009 in c#, clojure, groovy, haskell, java, language oriented programming, lisp, object orientation, rails, ruby, software design, thoughtworks and trends. 7 Comments

In ThoughtWorks we get together at least once an year for a whole weekend to drink beer and do whatever people consider interesting. This year’s ThoughtWorks Australia Away Day (AKA Team Hug) was somewhere in Victoria and among other activities (and a bus crash) we had technical sessions.

I used one of those slots to do a quick presentation on some fairly common myths regarding types and typesystems. Although there was nothing really advanced in it this -to be fair I barely know the basics about types- the session was triggered by my perception that most programmers know absolutely nothing about this topic.

one or two things you may not know about typesystems
View more documents from Phillip Calçado.

Real World Haskell – Book Review

Published by Phillip Calçado July 8th, 2009 in books, haskell and software design. 1 Comment

I just finished reading Real World Haskell. This was my first book on Haskell, I’ve read only articles and papers before and I believe it was a very good introduction to the language.

The book tries to use real world problems and I feel like it is not very successful as it. The examples focus too much in a tiny feature of the language but don’t show the bigger picture. There are many exercises on parsing, from JSON to binary formats, but nothing that shows you how Haskell deals with abstractions and layers, the bread and butter of what we usually do. I was expecting something more like what Practical Common Lisp does in its examples, with quick examples showing how features behave followed by more complete examples showing how everything fits together.

The description of features, including the type system, is good though. I read the book mostly in the morning train and even without a handy computer I still got a very good understanding of those. I am definitely more comfortable reading a blog about Haskell now.

The introduction to monads wasn’t that great. I can see how the authors try to create the need for monads’ features and then show how to use them but I think that the approach dies in implementation details and forgets about the bigger picture. I believe that this was deliberately made to avoid lengthy and complicated explanations that are present in many articles but I don’t think it gets it right either.

The book is freely available here, full of comments written by the Haskell community. As I said before it may have its problems but is a very good introduction.

methodologists

Published by Phillip Calçado July 7th, 2009 in agile, business, management and trends. 1 Comment

Nick showed me this today. My favourite part:

At a workshop once, Pete McBreen said “The Agile methods are methods created by people who like to program.” While that’s not entirely true, we bet it’s more true of that particular bunch of people than of any previous gaggle of methodologists. And their oddity went even further: a surprising number of the authors of the Agile Manifesto had programmed in Smalltalk. They were technology enthusiasts, and that enthusiasm—a “gosh wow!” enthusiasm for continuous integration tools, refactoring tools, programming languages, testing tools, and the like—has been a continuous part of the growth of Agile. It’s been a touch disreputable, though—because of the Agile Manifesto’s “individuals and interactions over processes and tools”—so we’ve chosen the artistic style called retro-futurism to call renewed attention to it.

Think about that next time someone tries to convince you that a pseudo-software development methodology based merely on assembly-line queue management and with no direct relation to software is a good thing.

« Previous Entries

About

My name is Phillip Calçado, I'm a Brazilian software developer currently working for ThoughtWorks in Sydney, Australia. More info? Check my LinkedIn profile, twitter or even my Brazilian Portuguese blog.

Latest

RSS




Creative Commons License

This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License.