For some months now I’ve being playing around the idea of writing a testing framework for Clojure. It started as just a more extensible fork of the fact library but now I’m trying to explore some funny ideas in the testing semantics.

Although this project is progressing too slowly it already spawned some other pet projects. One of those projects is the struct-quack library.

This is just an extremely naïve implementation of duck-typed structs for Clojure. Although I am ok using maps for pretty much everything I often need “maps that only accept certain keys” – a register/struct pretty much.

As a TDD practitioner I need some testing support for it. As I already had a somewhat working version of my testing library by that time I used it.

After finishing struct-quack’s first iteration I started to rewrite the testing framework itself to use that and perform some smart multimethods magic.

And now I have a hard problem to solve: what am I going to use to test the struct-quack library in its next iteration? Given that neither Java nor Clojure have versioned packages, if I use the library in the testing tool I just put myself in a circular dependency chain.

Circular dependencies are bad in multiple contexts. In this case it’s not only bad but also dangerous. As I modify struct-quack I may introduce a bug. This bug may affect the testing framework itself and tests that should fail could pass. There is no way to assert that struct-quack has no known bugs if its tests rely (even indirectly) on itself.

Shoulder of Giants?

I’m reading The Annotated Turing (great book so far) and the author does a great job in covering some basic concepts behind Turing’s work, including explanations of those in a way non-mathematicians (like me) can understand.

One of those building blocks is the work of Kurt Gödel. This called my attention:

[Gödel] had used the arithmetic developed within the system to associate every formula and every proof with a number. He was then able to develop a formula that asserted its own unprovability. […] The formula asserts nothing about its own truth or falsehood, but instead it’s unprovable. If arithmetic is consistent, then this formula can’t be false, because that would lead to a contradiction. The formula must be true –but true only in a metamathematical sense because truth is not a concept of the logical system itself- which means that it really is unprovable.
[…]
One crucial premise for the Incompleteness Theorem is that arithmetic is consistent. As a corollary, Gödel also showed that a consistency proof for arithmetic within the system was impossible. Because certain formulas could not be proved or disproved, it was possible that these formulas were inconsistent.[…]

As I said before I’m no mathematician but this looks like the same problem we saw in the previous section. It is just impossible to prove –again, in the usual unit testing sense- that a system works if the test code verifying that depends on code from the system under test.

In the Real World

But we all know that tests are not formal proofs and most of us are no Dijkstra or Knuth to validate every single line of code we write. What’s in there for me then?

Whenever I’m in a consulting gig I often find people immersed in the problem described above. The rationale for this is simple: testing requires a lot of code so we must reuse code we already have.

At lower level testing -unit-testing classes or functions- probably all you need is a testing framework and some mocking library. Units are small and easy to test.

The problem arises in higher-level testing. When you start thinking about page models and story runners you often need to create data structures to use during tests. The issue here is that you already have those structures in your domain code.

It is really tempting to just reuse your production code with a thin mapper from HTML back to the class.

But if you think about this what you have is a circular dependency between code under test and test code, just like what I described in sections above:

When I find this scenario it is often common that a huge change will make no tests break as the bug has corrupted the tests themselves. The level of confidence in testing decreases and people stop writing and fixing tests.

Your tests are compromised. In this scenario a failed test says that there is something wrong but a successful test doesn’t mean that everything is ok. If the test pass does it mean that the code is correct? What if the user class has a bug and, therefore, your very test has a bug?

For me a correct test is a test that verifies the right behaviour in the right way. A correct test won’t necessarily be a passing test. A correct system (or code, class, function… whatever unit you use) is one that passes a correct test.

In a good design there must be a clear separation of test code and code under test. They have to be orthogonal, one must not depend on the other to be correct and bugs in one must not interfere in the other in any way.

The duplication of the user class in this example is really annoying and wasteful but the solution proposed above affects the effectiveness of the whole testing process.

I can’t think of an easy solution for this problem. Currently my suggestion for most cases is to avoid using a bureaucratic language like Java to write your testing model. Languages like Clojure, Ruby, JavaScript or Python make it so easy to create data structures that the amount of code required to create your test model is actually less than what it would be required to reuse production code.