Setting up a new project

Carlo Wood, Jun/Jul 2007

Introduction

About this document

Despite the many automation tools, there are still a lot of things "the same" for every project. Things that have to be done over and over again. The initial goal of this document is to describe these things, so that starting a new project is easier. At the same time I'll try to turn this document into a reference, or at least overview, so that beginners (and also more experienced developers) can use it for that purpose.

What is a project?

In the light of this document, a project is a collection of files— code, a build system and documentation— that is released together as a tar-ball or package.

The main ingredients of a project are:

  1. The build system
  2. Documentation
  3. Code files

Each of these three categories can have files that are part of the project but which are not released or part of the final "package".

On top of that, there are many files which are generated automatically, and that are therefore not part of the project in the sense that you have to maintain them, but which are released as part of the package.

What is a build system?

The "build system" are those files that are intended for the maintainer of the package. This includes those files that allow you to generate other files. A build system exists of four major parts:

  1. External resources
  2. The version control system
  3. The automake/autoconf System
  4. Third party files that are part of the package.

Documentation?

This document is about open source. This means that your package needs to be usable and even maintainable by arbitrary strangers. The documentation is a vital part of your project. Without it you will be the only one ever using it. Documentation can be roughly devided into two groups of files:

  1. Documentation for (the) maintainer(s) of the package.
  2. Documentation for users (this might be developers using the package).

What are code files?

Files containing the code of the application. These are the most interesting part to write from the viewpoint of the developer. The reason for that is that it forms the heart of the project: it causes the application to do what it is intended for. This document will not go into much detail of the content of source files as most of it will be different from project to project. Also, programming falls outside the scope of this document. Nevertheless, we can make a differentiation between several types of source files:

  1. Header files or Source files.
  2. Files that are or are not installed as part of installing the package.
  3. Files that are part of one compilation unit or multiple compilation units.
  4. Files that are part of one executable or library, or files that are common between more than one.

Project Layout

Each project should have a single directory containing all files related to that project. It is logical if the name of that directory and the name of the project are the same, which puts a restriction on the name of a project. A good project name would exist of only lower case characters, no underscores and certainly no spaces.

Furthermore, it is advisable to put all your projects of a given language in on subdirectory as well. Lets call this root PROJECTS. For example,

PROJECTS="$HOME/projects"

Let use define the name of the current project as PROJECTNAME, for example,

PROJECTNAME=cwchessboard

The directory $PROJECTS/$PROJECTNAME will not be under version control, and therefore not contain source files. Instead it will contain external resources (for example documentation related to the project) as well as at least two directories:

  1. A directory with files under revision control.
  2. The build directory, with files that are generated as part of the configuration and compilation step of the build system.

The convention that I use is that the directory that is under revision control has the same name as the project name, hence $PROJECTS/$PROJECTNAME/$PROJECTNAME, and the object directory has -objdir appended to that, thus $PROJECTS/$PROJECTNAME/$PROJECTNAME-objdir.

External Resources

Most "external resource" files are entirely circumstantial and too specific for a particular project to say anything about. Let it suffice to give an example from practise therefore,

$ ls $PROJECTS
cwautomacros/  cwchessboard/  cwdb/  edragon/  ircproxy/  libcw/  libcwd/  libcws/  libecc/

which shows the nine projects that I currently have. The "external resource files" are kept in $PROJECTS/$PROJECTNAME. For example,

$ ls $PROJECTS/cwchessboard
cwchessboard/         env.source                knight_fill_curve.path  knight_head.path    knight_mouth.path       knight_right_ear.path
cwchessboard-objdir/  knight_between_ears.path  knight_front.path       knight_jaw.path     knight_nose.path        knight_shadow.path
env.compiler          knight_eye.path           knight_hair.path        knight_mouth2.path  knight_outer_back.path

This shows that apart from the env.source and env.compiler project files (see Directory dependent history and environment), the only external resources are *.path XML files: Inkscape SVG vector graphics describing different parts of the Knight chess piece.

Another example,

$ ls $PROJECTS/libecc
a126671.txt  bspace_data   csin.nb       env.source  foo.nb   libecc-objdir/   partchk.cc  pink.cc  theory-docs/  Untitled-1.nb
a.out*       bspace_data2  env.compiler  eta.nb      libecc/  Mathematica.txt  pink*       Pink.nb  thirdparty/

This shows the external resources related to libecc. Also the subdirectories theory-docs and thirdparty contain external resources (respectively mathematical documents and third party cryptographic source code).

The directory structure inside $PROJECTS/$PROJECTNAME/$PROJECTNAME

Also here try to organize the data by using a few subdirectories. For example, put all sources in a subdirectory src and the documentation in a subdirectory doc. If you are creating a library then put the headers in a separate directory, for example src/include. Some people prefer to put also application- header files in a separate directory but I think it's easier to have those in the same directory as the source files (src) on account of overview and opening files with an editor.

The build system

The only two files that return in every project directory are env.source and env.compiler. The meaning of these files are described in the HOWTO "Directory dependent history and environment", that also gives a specific example of what the content could be. Here I'll give the content that can be used for a new project. Once the project evolves, it will be needed to update the content of env.source, simply because some things are not known yet. Other things that you might want in an env.source file are too project specific to be given here.

source.env for applications

This is block of text is not the content of the env.source file; you have to copy and paste it to your command prompt. The reason for that is so each environment variable ($PROJECTS, $PROJECTNAME) will be expanded, while the backslash escape before other variables ($INSTALL_PREFIX etc) will be removed.

### COPY&PASTE EVERYTHING BELOW TO THE COMMAND PROMPT.
cat > $PROJECTS/$PROJECTNAME/env.source << EOF
export TOPPROJECT=$PROJECTS/$PROJECTNAME
source /env.source
source \$TOPPROJECT/env.compiler

# Flags.
CPPFLAGS=
LDFLAGS=
CFLAGS=
CXXFLAGS=

export CPPFLAGS LDFLAGS CFLAGS CXXFLAGS

# Helper variable.
GCCVER=\`\$CXX -v 2>&1 | grep '^gcc[ -][Vv]ersion' | sed -e 's/gcc[ -][Vv]ersion //' -e 's/ (.*//' -e 's/ /-/g'\`

# The install prefix.
INSTALL_PREFIX="/usr/local/install/\$GCCVER"

# Set the correct paths.
pre_path "\$INSTALL_PREFIX/lib/pkgconfig" PKG_CONFIG_PATH
pre_path "\$INSTALL_PREFIX/bin" PATH
pre_path "\$INSTALL_PREFIX/lib" LD_LIBRARY_PATH

export PKG_CONFIG_PATH PATH LD_LIBRARY_PATH INSTALL_PREFIX

# Aliases.
alias s='ls $PROJECTS/$PROJECTNAME/$PROJECTNAME/*.cc $PROJECTS/$PROJECTNAME/$PROJECTNAME/*.h'
alias vi='vim -c "set tags=$PROJECTS/$PROJECTNAME/$PROJECTNAME-objdir/tags"'

# The default configure options.
export CONFIGURE_OPTIONS="--enable-maintainer-mode --enable-debug --prefix=\$INSTALL_PREFIX"
alias configure='../$PROJECTNAME/configure \$CONFIGURE_OPTIONS'

# Doxygen output directory.
export OUTPUT_DIRECTORY=$HOME/www

# External source trees that need to be scanned with ctag.
export CTAGS_ROOT_SRCDIRS="/usr/src/gtk/glib-current /usr/src/gtk/gtk+-current"
EOF

source.env for libraries

The source.env for libraries is the same as the above source.env for applications, except that the first line that sets TOPPROJECT has to be removed.

compiler.env

The file compiler.env is the same for both, applications and libraries.

### COPY&PASTE EVERYTHING BELOW TO THE COMMAND PROMPT.
cat > $PROJECTS/$PROJECTNAME/env.compiler << EOF
CC="gcc"
CXX="g++"

# Use the correct cpp version.
CPP="\$(\$CC -print-libgcc-file-name | sed -e 's%/lib/.*%/bin/cpp%')
CXXCPP="\$CPP -x c++"

export CC CXX CPP CXXCPP
EOF

The Version Control System

The version control system would be subversion, or "svn" for short. Most files inside $PROJECTS/$PROJECTNAME/$PROJECTNAME are under version control. Files outside that directory are not. To create the initial $PROJECTS/$PROJECTNAME/$PROJECTNAME directory, first create an empty project and import it into your SVN repository, then check out the trunk:

cd "$PROJECTS/$PROJECTNAME"
mkdir tmp
cd tmp
mkdir branches tags trunk
svn import -m 'Initial directory structure' "svn+ssh://$PROJECT/repos"
cd ..
rm -rf tmp
svn checkout "svn+ssh://$PROJECT/repos/trunk" "$PROJECT"

The arguments containing "svn+ssh://$PROJECT/repos" are what you'd need if you set up your repository as is described in the mini-HOWTO Setting up a jail rooted SVN repository. If you set up your own repository, than use whatever works for you.

How to use svn is outside the scope of this document. Let me suffice with listing the most frequently used commands:

svn status		# List the status of all files in the current directory.
svn add somefile.cc	# Put somefile.cc under version control.
svn commit		# Synchronize repository with local copy.
svn update		# Synchronize local copy with repository (merging local changes).
svn revert somefile.cc	# Revert all local changes made to somefile.cc.
svn diff		# Show a diff of all local changes.

A complete overview of all svn subcommands can be found in Chapter 9 of the online subversion book.

The automake/autoconf System

With this I mean every file (or symbolic link) that is used or generated by the GNU build tools. The files that you, as maintainer, need to create are:

Every (sub)directory containing source (or header) files needs a Makefile.am.

As a convention, the base directory/package should also contain the files, to be distributed:

Both, LICENSE and README may be appended with a keyword that highlights the specific contents of that file. For example, LICENSE.AGPL (GPL v3), LICENSE.QPL, LICENSE.WTFPL, README.SVN, etc. Many projects that are completely built with the GNU autotools use a generic INSTALL file, that basically tells people to type configure, followed by make. You should have a copy of this file at /usr/share/autoconf/INSTALL. The README file is highly project specific, but usually contains things like what the project is about, how to contact the developer(s), the goals of the project and things that user should know when they want to use the project. Finally, the NEWS file is an incremental that highlights the most important bug fixes and features of each new release by version number. An initial NEWS file can for example look like:

$PROJECTNAME-0.0.0

        First public release.

New entries (for new versions) are added at the top of the file.

configure.ac

The configure script is generated (by autoconf) from configure.ac and aclocal.m4, where the latter (which is generated by aclocal) contains all M4 macro definitions. Those can be existing definitions (copied from your local installation of tools like autoconf, automake, libtool, etc), or they can be custom macro definitions. If you wrote your own macros for this project, put each in a macroname.m4 file in a m4 directory.

Here is a is an initial configure.ac: