CMake, CTest and CDash: Creating portable c++ software

CMake is a powerful build system for C++ applications (and other languages) used by many software projects including KDE. The main difference between CMake and other build systems is that it generates native unix make files, Microsoft Visual C++ solutions or XCode projects. Before we adopted cmake, we had to maintain separate build files for windows and unix/mac platforms. A developer using a specific platform would not update the other platform's build files which often resulted in compile errors when another developer checked out the new code. While fixing these errors is usually a trivial task, it's still an inconvenience. Using CMake solves this problem without adding any additional dependencies. CMake is small and binaries for a large number of platforms are provided.

CTest is part of CMake and can be used to test software. It doesn't provide any code but functions as a way to run the tests. For example, in OpenBabel we have a header where we define some functions we can use to write unit tests: OB_ASSERT, OB_REQUIRE and OB_COMPARE. Implementations of these functions are simple but any unit testing framework (code library, headers) can be used. Our functions print "FAIL" to the standard output and we use this as regular expression to allow CTest to determine if a test has failed. The advantage of using CTest is that we can now use the tests to improve the quality of our software. This becomes possible because CMake/CTest can submit the results to CDash.



The OpenBabel dashboard is available here

CDash is a web application containing the status of building and testing your software. KitWare provides CDash hosting but you still need to set up build "sites". These sites are just computers that run a CTest script as scheduled task or cronjob. For example, several developers and users from OpenBabel run these scripts every night and these results are submitted to CDash for all developers to see. We have multiple configurations to cover the most frequently used platforms and compilers (win, linux, mac, gcc, cygwin, MSVC, 32/64 bit, ...) but this depends on the platforms you want to support for your project. CTest synchronizes the Nightly builds so all sites will use the same revision which makes it easy to spot differences between platforms.

We also use Continuous builds which means that every time a developer checks in changes, an (incremental) build will be done, the tests will run and the results are submitted to CDash. This is complementary to the Nightly builds. For example, if for a Nightly run, a unit test failed on all platforms and there were 20 commits. The continuous results will identify which commit caused the test to start failing.

CDash can also be used for other information beyond compile warnings/errors and test results. For example, it is easy to include code code coverage and memory leak detection. For code coverage, CTest has support for GNU GCov and a commercial tool Bullseye. For OpenBabel, we split the coverage in two to differentiate between the core library and plugins. This is possible by using subprojects but it currently takes two builds and I'm still experimenting with the script. I'm told support for this will be improved in future versions but if the number of subprojects is small, the multiuiple build approach works. It runs cmake and defines some variables that enable code coverage for the library or formats (i.e. -fprofile-ars -ftest-coverage for gcov). The building directory is emptied between builds to ensure the gcov files are removed before analysing the new coverage results. The script can be found here and the changes to the OpenBabel CMakeLists.txt files here.

Another supported feature is memory checking to find memory leaks. In linux, valgrind is supported. This takes some time to run if the number of tests is large but running this Nightly is enough. It is also possible to submit additional files.

Acknowledgements: I'd like to thank Marcus Hanwell and Kitware for providing CDash hosting and other contributions to OpenBabel and Avogadro.