A perfect continuous integration system

One of the fundamental concept of agile software development is Continuous Integration (CI). You can read more about it here, for instance. In this post I'll try to define elementary things that in my humble opinion could build a ground for perfect continuous integration environment. Your comments on my concept will be also appreciated.

For this post I assume GNU environment, and statically-compiled languages like C++. However, such recipe can be easily rearranged to other compilers, languages, and environments.

1. CI server

First things go first. In order to facilitate the concept of CI we need some kind of automation software. Amount of this kind of software is fair big, so we can pick up one that fits our needs. The most popular is Jenkins (derivative from Hudson), as it is free and covers a lot of functionality. It is also easily extendable, thanks to over 400 plugins. From the other side, we have commercial applications, like for instance, Bamboo by Atlassian. I can't compare these two, but I assume that such a comparison would be similar to comparing MS Word with OpenOffice.org. For full list of well-known CI servers refer to this list.

2. Unit & Module tests

When we have our CI server running we are able to start defining jobs it will take car of. I do not mention basic "build" job, as it is obvious. Every knows that good code should be tested as much as possible. We can distinguish between standard Unit Tests (UT), that are likely to be written by developers and Module Tests (MT), that are preferably written by software testers. Personally I prefer to have one test runner to evaluate all UT and MT tests. This runner should output report in a form that is consumable by the CI server.

3. Lint for code and documentation

Before development on specific project is started, it is good idea to establish kind of coding convention and have a tool that will check whether software that is written follows rules specified in that document. This tool can be merged with standard lint tools like, for instance, this one, so hard-to-find errors are reported in the early stages of software development. Examples of such errors are: uninitialized variable, lack of assignment operator while copy constructor is defined, and so on.
We should also pick documentation system like, for example, doxygen and a tool that will check whether documentation is well written. People tends to ignore that, which is sad, because undocumented code is kind of useless for future maintainers and developers.

4. Coverage and profiling

When we have UT and MT integrated into our automation process, we likely are want to know how much code is really tested and what is overall performance of this code. For former case we have tools like gcov that will help us determine final test coverage in percents. For latter case there is available gprof, which output can be reorganized, so CI server can handle it. Example of such a tool is gprof2dot.py. This simply way we are able to see call graphs directly from our CI site and find out bottlenecks in the software we are developing.

5. Memory leaks hunter

Another thing that may be useful is tracing memory leaks (in case of languages that are statically compiled). We can use tool like valgrind to perform such operations. By using dedicated plugins for our CI server we are able to see overall trend of memory that was leaked in the software during run-time and figure out from where those leaks come from.

6. Cyclomatic complexity

Cyclomatic complexity helps us to determine how complex the software under production actually is. You can find more information about it here. I find this kind of tools very useful, therefore they are included in this list.

That's it! If you think that something crucial is missing here do not hesitate to write your opinions in comments!