QMake hackery: Dependencies & external preprocessing

2011-11-13

  • TODO: Put the code here into a Gist?

Qt Creator is a favorite IDE of mine for when I have to deal with miserably large C++ projects. At my job I ported a build in Visual Studio of one such large project over to Qt Creator so that builds and development could be done on OS X and Linux, and in the process, learned a good deal about QMake and how to make it do some unexpected things.

While I find Qt Creator to be a vastly cleaner, lighter IDE than Visual Studio, and find QMake to be a far more straightforward build system for the majority of things than Visual Studio’s build system, some things the build needed were very tricky to set up in QMake. The two main shortcomings I ran into were:

  • Managing dependencies between projects, as building the application in question involved building 40-50 separate subprojects as libraries, many of which depended on each other.
  • Having external build events, as the application also had to call an external tool (no, not moc, this is different) to generate some source files and headers from a series of templates.

QMake, as it happens, has some commands that actually make the project files Turing-complete, albeit in a rather ugly way. The eval command is the main source of this, and I made heavy use of it.

First is the dependency management system. It’s a little large, but I’m including it inline here.

# This file is meant to be included in from other project files, but it needs
# a particular context:
# (1) Make sure that the variable TEMPLATE is set to: subdirs, lib, or app.
#     Your project file really should be doing this anyway.
# (2) Set DEPENDS to a list of dependencies that must be linked in.
# (3) Set DEPENDS_NOLINK to a list of dependencies from which headers are
#     needed, but which are not linked in. (Doesn't matter for 'subdirs'
#     template)
# (4) Make sure BASEDIR is set.
#
# This script may modify SUBDIRS, INCLUDEPATH, and LIBS. It should always add,
# not replace.
# It will halt execution if BASEDIR or TEMPLATE are not set, or if DEPENDS or
# DEPENDS_NOLINK reference something not defined in the table.
#
# Order does matter in DEPENDS for the "subdirs" template. Items which come
# first should satisfy dependencies for items that come later.
# You'll often see:
# include ($$(BASEDIR)/qmakeDefault.pri)
# which includes this file automatically.
#
# -CMH 2011-06

# ----------------------------------------------------------------------------
# Messages and sanity checks
# ----------------------------------------------------------------------------
message("Included Dependencies.pro!")
message("Dependencies: " $$DEPENDS)
message("Dependencies (INCLUDEPATH only): " $$DEPENDS_NOLINK)
#message("TEMPLATE is: " $$TEMPLATE)

isEmpty(BASEDIR) {
    error("BASEDIR variable is empty here. Make sure it is set!")
}
isEmpty(TEMPLATE) {
    error("TEMPLATE variable is empty here. Make sure it is set!")
}

# ----------------------------------------------------------------------------
# Table of project locations
# ----------------------------------------------------------------------------

# Some common locations, here only to shorten descriptions in the _PROJ table.
_PROJECT1   = $$BASEDIR/SomeProject
_PROJECT2   = $$BASEDIR/SomeOtherProject
_DEPENDENCY = $$BASEDIR/SomeDependency

# Table of project file locations
# (Include paths are also generated based off of these)
_PROJ.FooLib               = $$_PROJECT1/Libs/FooLib
_PROJ.BarLib               = $$_PROJECT1/Libs/BarLib
_PROJ.OtherStuff           = $$_PROJECT2/Libs/BarLib
_PROJ.MoreStuff            = $$_PROJECT2/Libs/BarLib
_PROJ.ExternalLib          = $$BASEDIR/SomeLibrary

# ----------------------------------------------------------------------------
# Iterate over dependencies and update variables, as appropriate for the given
# template type
# ----------------------------------------------------------------------------

# _valid is a flag telling whether TEMPLATE has matched anything yet
_valid = false

contains(TEMPLATE, "subdirs") {
    for(dependency, DEPENDS) {
        # Look for an item like: _PROJ.(dependency)

        # Disclaimer: I wrote this and it works. I have no idea why precisely
        # why it works. However, I repeat the pattern several times.
        eval(_dep = $$"_PROJ.$${dependency}")
        isEmpty(_dep) {
            error("Unknown dependency " $${dependency} "!")
        }

        # If that looks okay, then update SUBDIRS.
        eval(SUBDIRS += $$"_PROJ.$${dependency}")
    }
    message("Setting SUBDIRS=" $$SUBDIRS)
    _valid = true
}

contains(TEMPLATE, "app") | contains(TEMPLATE, "lib") {
    # Loop over every dependency listed in DEPENDS.
    for(dependency, DEPENDS) {
        # Look for an item like: _PROJ.(dependency)
        eval(_dep = $$"_PROJ.$${dependency}")
        isEmpty(_dep) {
            error("Unknown dependency " $${dependency} "!")
        }

        # If that looks okay, then update both INCLUDEPATH and LIBS.
        eval(INCLUDEPATH += $$"_PROJ.$${dependency}"/include)
        eval(LIBS += -l$${dependency}$${LIBSUFFIX})
    }
    for(dependency, DEPENDS_NOLINK) {
        # Look for an item like: _PROJ.(dependency)
        eval(_dep = $$"_PROJ.$${dependency}")
        isEmpty(_dep) {
            error("Unknown dependency " $${dependency} "!")
        }

        # If that looks okay, then update INCLUDEPATH.
        eval(INCLUDEPATH += $$"_PROJ.$${dependency}"/include)
    }
    #message("Setting INCLUDEPATH=" $$INCLUDEPATH)
    #message("Setting LIBS=" $$LIBS)
    _valid = true
}

# If no template type has matched, throw an error.
contains(_valid, "false") {
    error("Don't recognize template type: " $${TEMPLATE})
}

It’s been sanitized heavily to remove all sorts of details from the huge project it was taken from. Mostly, you need to add your dependent projects into the “Table of Project Locations” section, and perhaps make another file that set up the necessary variables mentioned at the top. Then set the DEPENDS variable to a list of project names, and then include this QMake file from all of your individual projects (it may be necessary to include it pretty close to the top of the file).

In general, in this large application, each sub-project had two project files:

  • One with TEMPLATE = lib (a few were app instead as well). This is the project file that is included in as a dependency from any project that has TEMPLATE = subdirs, and this project file makes use of the QMake monstrosity above to set up the include and library paths for any dependencies.
  • One with TEMPLATE = subdirs. The same QMake monstrosity is used here to include in the project files (of the sort in #1) of dependencies so that they are built in the first place, and permit you to build the sub-project standalone if needed.

…and both are needed if you want to be able to build sub-project independently and without making to take care of dependencies individually.

The next project down below sort of shows the use of that QMake monstrosity above, though in a semi-useless sanitized form. Its purpose is to show another system, but I’ll explain that below it.

QT -= gui
QT -= core
TEMPLATE = lib

## Include our qmake defaults
DEPENDS = FooLib BarLib
include ($$(BASEDIR)/qmakeDefault.pri)

TARGET = Project$${LIBSUFFIX}
LIBS += -llua5.1 -lrt -lLua$${LIBSUFFIX}
DEFINES += PROJECT_EXPORTS

INCLUDEPATH += /usr/include/lua5.1 
    ./include

HEADERS += include/SomeHeader.h 
    include/SomeOtherHeader.h

SOURCES += source/SomeClass.cpp 
    source/SomeOtherClass.cpp

# The rest of this is done with custom build steps:
GENERATOR_INPUTS = templates/TemplateFile.ext 
    templates/OtherTemplate.ext

gen.input = GENERATOR_INPUTS
gen.commands = $${DESTDIR}/generator -i $${QMAKE_FILE_IN}
# -s source$(InputName).cpp -h include$(InputName).h

# Set the destination of the source and header files.
SOURCE_DIR = "source/"
HEADER_DIR = "include/"
# What prefix and suffix to replace with paths and .h.cpp, respectively.
TEMPLATE_PREFIX = "external/"
TEMPLATE_EXTN = ".ext"

#
# Warning: Here be black magic.
#
# We need to use QMAKE_EXTRA_COMPILERS but its functionality does not give us
# an easy way to explicitly specify the names of multiple output files with a
# single QMAKE_EXTRA_COMPILERS entry. So, we get around this by making one
# entry for each input template (the .ext files).
# The part where this gets tricky is that each entry requires a unique
# variable name, so we must create these variables dynamically, which would
# be impossible in QMake ordinarily since it does only a single eval pass.
# Luckily, QMake has an eval(...) command which explicitly performs an eval
# pass on a string. We repeatedly use constructs like this:
#    $$CONTENTS = "Some string data"
#    $$VARNAME = "STRING"
#    eval($$VARNAME = $$CONTENTS)
# These let us dynamically define variables. For sanity, I've tried to use a
# suffix of _VARNAME on any variable which contains the name of another
# variable.
#

# Iterate over every filename in GENERATOR_INPUTS
for(templatefile, GENERATOR_INPUTS) {
    # Generate the name of the header file.
    H1 = $$replace(templatefile, $$TEMPLATE_PREFIX, $$HEADER_DIR)
    HEADER = $$replace(H1, $$TEMPLATE_EXTN, ".h")
    # Generate the name of the source file.
    S1 = $$replace(templatefile, $TEMPLATE_PREFIX, $$SOURCE_DIR)
    SOURCE = $$replace(S1, $$TEMPLATE_EXTN, ".cpp")
    # Generate unique variable name to populate & pass to QMAKE_EXTRA_COMPILERS
    QEC_VARNAME = $$replace(templatefile, ".", "")
    QEC_VARNAME = $$replace(QEC_VARNAME, "/", "")
    VARNAME = $$replace(QEC_VARNAME, "\", "")
    # Append _INPUT to generate another variable name for the input filename
    INPUT_VARNAME = $${QEC_VARNAME}_INPUT
    eval($${INPUT_VARNAME} = $$templatefile)

    # Now generate an entry to pass to QMAKE_EXTRA_COMPILERS.
    eval($${VARNAME}.commands = $${DESTDIR}/generator -i ${QMAKE_FILE_IN} -s ${QMAKE_FILE_OUT} -h $${HEADER})
    eval($${VARNAME}.name = $$VARNAME)
    # ACHTUNG! The 'input' field is the _variable name_ which contains the
    # input filename, not the filename itself. If you put in a filename or
    # either of those variables don't exist, this will fail, silently, and
    # all attempts at diagnosis will lead you nowhere.
    eval($${VARNAME}.input = $${INPUT_VARNAME})
    eval($${VARNAME}.output = $${SOURCE})
    eval($${VARNAME}.variable_out = SOURCES)

    # Now tell QMake to actually do this step we meticulously built.
    eval(QMAKE_EXTRA_COMPILERS += $$VARNAME)
    # Also add our header files. I doubt it's really necessary, but here it is.
    HEADERS += $${HEADER}
}

This one uses a bit more black magic. The entire GENERATOR_INPUTS list is a set of files that are inputs to an external program that is called to generate some code, which then must be built with the rest of the project. This uses undocumented QMake features, and a couple kludges to generate some things dynamically (i.e. the filenames of the generated code) from a variable-length list. I highly recommend avoiding it. However, it does work.

These two links proved indispensable in the creation of this:

QMake Variable Reference

Undocumented qmake

ProjectTechnobabble

Obscure features of JPEG

Context Free