bda's blog

Lately I’ve found maintaining Makefiles tedious, even for small projects. Much can be done with implicit rules, but header file dependencies must still be specified manually (or so I though – more on this later). For a small project, maybe you only have five or so of these rules. However there is no reason to manage these manually, and for small projects I am more lazy and feel more entitled to a completely hassle free build system. If it’s so simple, why should I be doing any work?

What I want is a build system that handles include dependencies automatically, allows me to build to a separate directory, and allows me to clean without specifying all the files that need to be deleted. I also really like compact and easy to understand syntax. Advanced features that are useful for large projects are also a plus – many large projects start out small.

I created build files for SCons, BJam/Boost.Build V2, and CMake. For each tool, I will list my build file(s) and discuss my impressions. Note that the analysis is necessarily shallow – I am mainly concerned with how well they work on small projects with very modest build needs, and I have spent very little time with each system. I also discuss a solution to the header dependency problem using GNU Make.

The Source Tree

The project I will use as an example is an intelligent brute force attack on textbook ElGamal encryption, which works well when the message encrypted is small (ElGamal is not vulnerable to this attack when properly implemented).

Here is the source tree:

code
|-- basicmimattack.cc
|-- basicmimattack.h
|-- diskmimattack.cc
|-- diskmimattack.h
|-- dlogtest.cc
|-- elgamalmgr.cc
|-- elgamaltest.cc
|-- hashmimattack.cc
|-- hashmimattack.h
|-- include
|   |-- dlog.h
|   |-- elgamal.h
|   `-- randomhelpers.h
|-- lib
|   |-- dlog.cc
|   |-- elgamal.cc
|   `-- randomhelpers.cc
|-- mimattackmain.cc
`-- randomfac.cc

mimattackmain.cc contains the main program, and it uses different attacks from the *mimattack.cc files depending on its command line arguments. dlogtest.cc, elgamaltest.cc, elgamalmgr.cc, and randomfac.cc also define main functions for other helper and test programs. All programs use the GNU MP Bignum library (GMP), which I link to statically. The mimattack main program also links dynamically to tokyocabinet, which is used for the disk-based attack. There are various other dependencies, mostly on the files in the lib directory, which should be clear from the build files below.

I run Ubuntu 8.04 (Hardy Haron) 64bit and use the g++ compiler, and I make no attempt at portable builds. However one of the strength of these tools is portability – with slight modifications these build files could be used under windows or with a different compiler.

SCons

SCons is designed to replace make, automake/autoconf, and ccache, and supports many different programming languages. Blender is probably the most popular open source project that uses SCons. The build files – SConstruct and SConscript files – are Python code. This make for an extremely flexible system. If you want to do something that is not directly supported, chances are you can figure out how to do it with some extra Python glue. I am using SCons version 1.01, built and installed from the “source” tarball, since the Ubuntu package is out of date.

Here is my SConstruct file, placed in the code root:

SConscript(‘SConscript’, variant_dir=’build’, duplicate=0)

Yup, that’s it. It would be more typical to just put all the build commands here, but I wanted a separate build directory and this was the easies way I found to do that. The rules are actually in the SConscript:

env = Environment()

gmp = '/usr/lib/libgmp.a'

randomhelpers_obj = env.Object('lib/randomhelpers.cc')
elgamal_obj = env.Object('lib/elgamal.cc')
dlog_obj = env.Object('lib/dlog.cc')

env.Program('mimattack', ['mimattackmain.cc', elgamal_obj, randomhelpers_obj, gmp] + Glob('*mimattack.cc'),
            LIBS=['tokyocabinet'])
env.Program(['elgamalmgr.cc', elgamal_obj, randomhelpers_obj, gmp])

env.Program(['elgamaltest.cc', elgamal_obj, randomhelpers_obj, gmp])
env.Program(['dlogtest.cc', dlog_obj, randomhelpers_obj, gmp])
env.Program(['randomfac.cc', randomhelpers_obj, gmp])

Just run ‘scons’ in the code root directory to build. The Environment is not really necessary here, but it will make it easier for me to add custom compilation flags later. The basic idea is that you define the nodes of the dependency tree. I could also have left out the object definitions for the files in lib, and just referenced the .cc source in each program requiring them – SCons is smart enough to only compile the the object files once. However this way I don’t have the paths hardcoded in multiple places. I defined a variable containing the path to libgmp.a so I don’t have to type it every time, and I use a glob for the *mimattack.cc files required by mimattack. Note that I only specify a program name for mimattack – all the other programs will be given names based on their first source – ie elgamaltest.cc will be compiled as simply ‘elgamaltest’ in Unix/Linux, ‘elgamaltest.exe’ on Windows.

Notice that there are no header file dependencies – SCons does that for you. You can see a dependency tree with scons -n –tree, to make sure it’s correctly recognizing the header dependencies. SCons uses MD5 signatures instead of time stamps by default to determine what needs rebuilding, so if an object file ends up being identical even after a source change, it may not force the builds you expect. I temporarily set Decider(‘timestamp-new’) to convice myself that my SConscript file is correct.

Also notice the lack of any sort of clean target or node. SCons also takes care of that – just run ‘scons -c’ to perform the equivalent of a make clean.

SCons supports using multiple build directories, which it calls varient directories. I really just wanted all build output in a subdirectory, and using the above SConstruct file was the easiest solution I found. The duplicate=0 tells SCons not to copy all the source files to the build directory (SCons does this by default for maximum compatibility with different tools).

Here are some other useful command line options:

• -Q: Suppress some of the less useful output. I recommend using this always – it will still display the compiler commands and output, which is what we’re really interested in.
• –debug=explain: Explain why certain targets are being build. I wish it would also explain why certain targets are not being build.
• -n: Dry run. This displays what build command will be run without actually executing them. Note that it will not display builds lower in the dependency hiearchy that depend on the output of the initial builds. In particular if a program build depends on object files, SCons won’t know if the object files actually change until the objects are actually built, so it may be that the program is not listed with -n but is actually built when you drop the -n.

Pros

• It’s Python!
• id software used it to build Doom 3!

Cons

• Awkward to specify separate build directory.
• Requires the Python interpreter.

Boost.Build (bjam)

Boost.Build V2 is a C/C++ build tool. bjam (short for Boost.Jam) is the command you run instead of make – bjam is a dependency of the boost-build package in ubuntu, but I’m not sure how many features of Boost.Build I’m using. Boost.Build is used to build the Boost C++ libraries, but can be used independently. There are other versions of Jam, but Boost.Jam was recommended by this freshmeat article so I decided to start with boost. Here is my Jamroot file:

lib tokyocabinet ;

lib gmp : : <file>/usr/lib/libgmp.a ;

exe mimattack : mimattackmain.cc lib/elgamal.cc lib/randomhelpers.cc gmp [ glob *mimattack.cc ] tokyocabinet ;

exe randomfac : randomfac.cc lib/randomhelpers.cc gmp ;

exe dlogtest : dlogtest.cc lib/dlog.cc lib/randomhelpers.cc gmp ;

exe elgamalmgr : elgamalmgr.cc lib/elgamal.cc lib/randomhelpers.cc gmp ;

exe elgamaltest : elgamaltest.cc lib/elgamal.cc lib/randomhelpers.cc gmp ;

bjam has the most make-like and most compact syntax. You must create a user-config.jam file before bjam will run – the ubuntu package installed a sample config file in /etc/user-config.jam, and I just copied it to my homedir and commented out the using gcc line. Once you have a Jamroot and a proper config file, just run ‘bjam’ to build. Notice that you must declare system libraries before you use them – see the Library task section of the manual for more details. I used the most compact target for tokyocabinet – it searches the system library path for library. Note that you don’t include the ‘lib’ in front or a suffix such as .so or .a. bjam seems to use shared libraries if available. For GMP I wanted to use the static library, so I specified it explicitly with the <file> property.

bjam, like SCons, handles header dependencies and single compilation of common files automatically. It also has a really slick way of managing dependencies accross projects.

Notice that there is no mention of build directories in the Jamroot – bjam decides that for you, and I can’t figure out how to change it. By default (at least when using gcc), bjam creates a debug build (passing in the -g option) and puts all the output in bin/gcc/debug. If I run ‘bjam release’, it will compile without -g and the output goes in bin/gcc/release. The lack of options here is annoying, but you can always create symlinks or just keeping a terminal open in your most used build dirs. I actually find this less annoying than building into the source directory by default.

Pros

• Awesome compact syntax!
• Works well for a hierarchy of dependent projects. (I have not used it for this, but Boost has.)

Cons

• No library discovery and configuration features?
• No control of build output directories?
• Requires a config file.

CMake

CMake is the new build system for KDE, so it has some serious critical mass. CMake is different from SCons and bjam in that it generates platform dependent build files, and then you do the actual build with another tool. In Linux it generates make files. Like SCons it has autoconf-like features for finding libraries. Here is my CMakeList.txt:

PROJECT(mimattacks)

set(GMP /usr/lib/libgmp.a)

file(GLOB ATTACKSRCS *mimattack.cc)

add_executable(dlogtest dlogtest.cc lib/dlog.cc lib/randomhelpers.cc)
target_link_libraries(dlogtest ${GMP})

add_executable(elgamalmgr elgamalmgr.cc lib/elgamal.cc lib/randomhelpers.cc)
target_link_libraries(elgamalmgr ${GMP})

add_executable(elgamaltest elgamaltest.cc lib/elgamal.cc lib/randomhelpers.cc)
target_link_libraries(elgamaltest ${GMP})

add_executable(randomfac randomfac.cc lib/randomhelpers.cc)
target_link_libraries(randomfac ${GMP})

add_executable(mimattack mimattackmain.cc lib/elgamal.cc lib/randomhelpers.cc ${ATTACKSRCS})
target_link_libraries(mimattack tokyocabinet ${GMP})

CMake has the most verbose syntax. Building also takes two steps – change to the build directory, run cmake with the source directory as a command line argument, and then run make. The call to cmake generates the Makefiles and make does the actual build.

Pros

• Used by large projects such as KDE.

Cons

• Verbose syntax. Still much better than make.
• Multi-step build process.

GNU Make

Many compilers have options to generate make dependencies for a file – if I run ‘g++ -MM mimattackmain.cc’, I get this output

mimattackmain.o: mimattackmain.cc include/randomhelpers.h \
  include/elgamal.h basicmimattack.h hashmimattack.h diskmimattack.h

The old fashioned way of doing things is to create a ‘depends’ target, so that ‘make depends’ will update your make file with the proper dependencies, calling gcc or other tools to do the hard work. With GNU Make, which is the version installed on most Linux distributions, you can do this without a special target. Using this section of the user’s guide, I created a new Makefile:

%.d: %.cc
        @set -e; rm -f $@; \
     $(CC) -MM $(CPPFLAGS) $< > $@.$$$$; \
     sed 's,\($*\)\.o[ :]*,\1.o $@ : ,g' < $@.$$$$ > $@; \
     rm -f $@.$$$$

CC=g++
CXXFLAGS=-Wall
GMP=/usr/lib/libgmp.a

SRCS := $(wildcard *.cc lib/*.cc)

.PHONY : all
all: elgamaltest randomfac mimattack elgamalmgr dlogtest

include $(SRCS:.cc=.d)

mimattack: mimattackmain.cc basicmimattack.o hashmimattack.o diskmimattack.o \
                lib/elgamal.o lib/randomhelpers.o ${GMP}
        g++ -o mimattack $^ -ltokyocabinet $(CXXFLAGS)

elgamaltest: elgamaltest.o lib/elgamal.o lib/randomhelpers.o ${GMP}

dlogtest: dlogtest.o lib/dlog.o lib/randomhelpers.o ${GMP}

elgamalmgr: elgamalmgr.o lib/elgamal.o lib/randomhelpers.o ${GMP}

randomfac: randomfac.o lib/randomhelpers.o ${GMP}

.PHONY : clean
clean:
        -rm -f *.o lib/*.o *.d lib/*.d mimattack elgamaltest elgamalmgr \
                                       randomfac dlogtest core

Basically what this does is include the files mimattackmain.d, lib/elgamal.d, etc, one for every .cc file. At first the files don’t exist, so the make rule at the top is used to create them. It calls g++ -MM to genrate the dependencies and puts the output in the .d files. Then make can actually include these dependencies. I won’t go into more details here – see the user’s guide link above for more information. This is by far the longest of the build files, and it still doesn’t build to a separate directory or automatically manage cleaning. It is also far more obtuse – you need to know a lot about make to really understand what’s going on here. The advantage of GNU Make is that it has a huge install base, but I really think it’s time the world moved on to more modern tools.

Conclusion

All three build systems are very easy to use for small projects once you get over the initial learning curve. Of the three, bjam is my favorite in terms of syntax. However I don’t think it has autoconf-like features, so if you want to distribute an open source application so it will compile in virtually any environment without extra work, SCons or CMake may be better options. The focus of this article is small projects, but sometimes small projects get bigger, and you may need those features someday. I would consider CMake the “safe” option now that it has been adopted by KDE, but I prefer SCons because I know and love Python, and it has more compact syntax*.

I will probably use bjam for this project. It is my part of my thesis and the code is never going to see wide release. I would use SCons for projects that may gather complex library dependencies and which I may want to distribute to the world, and I would use CMake for a KDE application.

* All three build systems have very clear syntax, and if you are working on a real world project technical considerations are far more important than wishy wash things like “I like this compact syntax better”. However for small time personal hacking (and academic hacking), one can afford to be a syntax snob.