7. Build Process

foxBMS uses waf The meta build system for building binaries and the documentation.

For detailed information on waf see waf.io. A short introcution to waf is given at waf.io/apidocs/tutorial. The more detailed version of how to use waf is found at waf.io/book.

7.1. General

7.1.1. Where to find the toolchain?

The waf toolchain is located in the directory foxbms\tools, in the binary waf. This archive is automatically unpacked in a directory named something like waf-{{X}-{{some-hash-value}} containing the waf library, where {{X}} is the dash-separated version number of waf. It is unpacked into foxbms\tools. It is generally assumed that all commands are run from directory foxbms. Therefore waf has to be always called by python tools\waf some-command where some-command is an argument defined in the wscript.

Additional build tools are located in foxbms\tools\waftools. These are the tooles needed for building the documentation, i.e., doxygen and sphinx.

7.1.2. Where are the build steps described?

The build process is described in files named wscript, that can be found nearly everywhere inside the different directories inside the foxBMS project. Later in this documentation this is explained in detail.

7.1.3. General

waf needs always to be run form the top level of the repository and the path to waf must be given relative to that directory. This path is tools\waf.

Listing 7.1 How to call waf
cd path\to\foxbms
python tools\waf {{some-command}}

7.1.4. What commands can be used?

To get an overview of support commands run --help or -h on the waf binary:

Listing 7.2 How to call help on waf
python tools\waf --help

This is the output in foxBMS version 1.6.2

Listing 7.3 Waf help in foxBMS
waf [commands] [options]

Main commands (example: ./waf build -j4)
  build                 : executes the build
  build_all             : builds all parts of the project (binaries and documentation)
  build_libs            : executes the build of libs
  build_primary         : executes the build of primary
  build_primary_bare    : executes the build of primary_bare
  build_secondary       : executes the build of secondary
  build_secondary_bare  : executes the build of secondary_bare
  clean                 : cleans the project
  clean_all             : cleans all parts of the project
  clean_libs            : cleans the project libs
  clean_primary         : cleans the project primary
  clean_primary_bare    : cleans the project primary_bare
  clean_secondary       : cleans the project secondary
  clean_secondary_bare  : cleans the project secondary_bare
  configure             : configures the project
  cpplint               : configures cpplint
  dist                  : creates an archive containing the project source code
  distcheck             : creates tar.bz form the source directory and tries to run a build
  distclean             : removes build folders and data
  doxygen               : creates doxygen documentation
  doxygen_libs          : creates the doxygen documentation of libs
  doxygen_primary       : creates the doxygen documentation of primary
  doxygen_primary_bare  : creates the doxygen documentation of primary_bare
  doxygen_secondary     : creates the doxygen documentation of secondary
  doxygen_secondary_bare: creates the doxygen documentation of secondary_bare
  flake8                : runs flake8 on the foxBMS repository
  list                  : lists the targets to execute
  list_libs             : lists the targets to execute for libs
  list_primary          : lists the targets to execute for primary
  list_primary_bare     : lists the targets to execute for primary_bare
  list_secondary        : lists the targets to execute for secondary
  list_secondary_bare   : lists the targets to execute for secondary_bare
  sphinx                : creates the sphinx documentation of the project
  step                  : executes tasks in a step-by-step fashion, for debugging
  step_libs             : executes tasks in a step-by-step fashion, for debugging of libs
  step_primary          : executes tasks in a step-by-step fashion, for debugging of primary
  step_primary_bare     : executes tasks in a step-by-step fashion, for debugging of primary_bare
  step_secondary        : executes tasks in a step-by-step fashion, for debugging of secondary
  step_secondary_bare   : executes tasks in a step-by-step fashion, for debugging of secondary_bare

Options:
  --version             show program's version number and exit
  -c COLORS, --color=COLORS
                        whether to use colors (yes/no/auto) [default: auto]
  -j JOBS, --jobs=JOBS  amount of parallel jobs (8)
  -k, --keep            continue despite errors (-kk to try harder)
  -v, --verbose         verbosity level -v -vv or -vvv [default: 0]
  --zones=ZONES         debugging zones (task_gen, deps, tasks, etc)
  -h, --help            show this help message and exit
  --cpplint-conf-file=CPPLINT_CONF
                        cpplint configuration file (default: cpplint.yml)
  -t TARGET, --target=TARGET
                        build target: debug (default)/release
  -l LIBS, --libs=LIBS  name of the library to be used

  Configuration options:

  Build and installation options:
    -p, --progress      -p: progress bar; -pp: ide output

  Step options:
    --files=FILES       files to process, by regexp, e.g. "*/main.c,*/test/main.o"

  Installation and uninstallation options:
    --distcheck-args=ARGS
                        arguments to pass to distcheck

7.1.5. The configure command

Before building any binaries or documentation is possible, the project needs to be configured. A successfull configure command and its ouput is shown below:

Listing 7.4 Configuration of a the project
python tools\waf configure
(...)
'configure' finished successfully (0.340s)

7.1.6. The build commands

After the project has been configured, a build can be triggered and it is generally exectued by the build commands. As foxBMS requries building variants, one has to use e.g., build_primary in order to build binaries for the primary MCU.

Listing 7.5 Example of a wrong and a correct build command.
python tools\waf build
Waf: Entering directory `.\foxbms\build'
A build variant must be specified, run 'python tools\waf --help'
python tools\waf build_primary
(...)
'build_primary' finished successfully (8.800s)

The possible build commands, the definition of the and corresponding targets and the targets itself is listet below:

  • Primary MCU
    Listing 7.6 Build primary binaries
    python tools\waf build_primary
    

    The targets are defined at:

    • foxbms\embedded-software\mcu-primary\src\application\wscript

    • foxbms\embedded-software\mcu-common\src\engine\wscript

    • foxbms\embedded-software\mcu-common\src\module\wscript

    • foxbms\embedded-software\mcu-primary\src\engine\wscript

    • foxbms\embedded-software\mcu-primary\src\module\wscript

    • foxbms\embedded-software\mcu-freertos\wscript

    • foxbms\embedded-software\mcu-hal\STM32F4xx_HAL_Driver\wscript

    • foxbms\embedded-software\mcu-primary\src\general\wscript

    Listing 7.7 Primary targets
    foxbms-application
    foxbms-common-driver
    foxbms-common-engine
    foxbms-common-module
    foxbms-common-util
    foxbms-driver
    foxbms-engine
    foxbms-module
    foxbms-os
    foxbms-stmhal
    foxbms_primary.elf
    'list_primary' finished successfully (0.052s)
    
  • Secondary MCU
    Listing 7.8 Build secondary binaries
    python tools\waf build_secondary
    

    The targets are defined at:

    • foxbms\embedded-software\mcu-secondary\src\application\wscript

    • foxbms\embedded-software\mcu-common\src\engine\wscript

    • foxbms\embedded-software\mcu-common\src\module\wscript

    • foxbms\embedded-software\mcu-secondary\src\engine\wscript

    • foxbms\embedded-software\mcu-secondary\src\module\wscript

    • foxbms\embedded-software\mcu-freertos\wscript

    • foxbms\embedded-software\mcu-hal\STM32F4xx_HAL_Driver\wscript

    • foxbms\embedded-software\mcu-secondary\src\general\wscript

    Listing 7.9 Secondary targets
    foxbms-application
    foxbms-common-driver
    foxbms-common-engine
    foxbms-common-module
    foxbms-common-util
    foxbms-engine
    foxbms-os
    foxbms-stmhal
    foxbms_secondary.elf
    'list_secondary' finished successfully (0.100s)
    
  • General documentation

    The general documenation is build by

    Listing 7.10 Build the general foxBMS documentation
    python tools\waf sphinx
    
  • API documentation

    The API documentation is build using the doxygen_{{variant}}, therefore

    Listing 7.11 Build the general foxBMS documentation
    python tools\waf doxygen_primary
    python tools\waf doxygen_secondary
    
  • Cleaning

    It is also possible to clean the binaries and Doxygen documentation. This step is performed by the clean command.

    As seen from --help the possible clean commands are

    • clean_all

    • clean_libs

    • clean_primary

    • clean_primary_bare

    • clean_secondary

    • clean_secondary_bare

    • distclean

    Each command cleans the specified option, except for distclean. However it is possible to make a complete clean by distclean. After distclean the entire build directory and all lock files etc. are deleted and the project needs to be configured again. Cleaning the general sphinx documentation alone is currently not supported, but it can be achivied by running distclean.

7.2. Targets

As stated above the targets and sub targets of the build process are shown by list_x where x is the specified target. The main target is the *.elf.unpatched file. The final targets are build afterwards. After successfully linking the map file is generated.

These logging files are found in build and build\{{target}}. Additional to the *.elf.unpatched and *.elf files a *.hex and two *.bin files of the binary are generated. The *.bin files are separated into the flash and the flashheader. The size of each object/binary is written to a log file.

The targets are build as follows (final targets are filled gray):

digraph {
        rankdir=TB;
        graph [fontname = "monospace"];
        node [fontname = "monospace"];
        edge [fontname = "monospace"];
        "foxbms_flash.bin" [style=filled];
        "foxbms_flashheader.bin" [style=filled];
        "foxbms.hex" [style=filled];
        "foxbms.elf" [style=filled];
        "foxbms.elf.unpatched" -> "foxbms_flash.bin" [label="objcopy"];
        "foxbms.elf.unpatched" -> "foxbms_flashheader.bin.unpatchted" [label="objcopy"];
        "foxbms_flash.bin" -> "checksum.yml" [label="tsk_cal_chksum"];
        "foxbms_flashheader.bin.unpatchted" -> "checksum.yml" [label="tsk_cal_chksum"];
        "checksum.yml" -> "foxbms.elf" [label="objdump"];
        "foxbms.elf" -> "foxbms.hex" [label="objcopy"];
        "foxbms.elf" -> "foxbms_flashheader.bin" [label="objcopy"];
 }

Fig. 7.1 foxBMS build process targets

7.3. Build Process

Note

For testing the following explanations it is assumed that python tools\waf configure has been run.

This sections gives an overview how the build process is defined. All features are generally defined in the top wscript located at foxbms\wscript.

The minimum functions that are needed to be defined a build in waf are:

  • configure and

  • build.

As the toolchain needs more targets the following functions need to be implemented: doxygen and sphinx.

Furthermore the following features are needed:

  • for calculating the checksum based on the *.elf.unpatched file the class tsk_cal_chksum and for creating the *.elf file the tsk_wrt_chksum class and for adding these features the function add_chksum_task,

  • for stripping the debug symbols in release mode the class strip and the function add_strip_task,

  • for creating a hex file from the elf file the class hexgen and the function add_hexgen_task,

  • for generating bin files from elf files the classes tsk_binflashheadergen, tsk_binflashgen and the function add_bingen_task and the class tsk_binflashheaderpatch and the function add_patch_bin_task,

  • for generating size information of the objects and binaries the class size and the function process_sizes,

  • for copying the libraries build by build_libs into the correct directories the class copy_libs and the function add_copy_libs,

  • for compiling assembler files *.s the class Sasm and the function asm_hook.

For implementation details see the wscript itself.

Some of these functionalities require scripts from foxbms\tools.

Overview of the build process

7.3.1. General Documentation

This build target uses the function def sphinx(bld). Since this definition of a function called sphinx, it is accepted as command to waf.

This general documentation is generated by running

Listing 7.12 Generate general foxBMS documentation
python tools\waf sphinx

The implementation details of the sphinx command can be found in foxbms\tools\waftools\sphinx_build.py.

7.3.2. Primary and Secondary Binaries and Doxygen Documentation

In order to have different build variants, these variants have to be defined. This is done at the top of the main wscript at foxbms\wscript. The variants have to be defined for the binary build and Doxygen documentation.

Listing 7.13 Implementation of the variant build
from waflib.Build import BuildContext, CleanContext, InstallContext, \
    UninstallContext, ListContext, StepContext
for x in variants:
    for y in (
        BuildContext,
        CleanContext,
        InstallContext,
        UninstallContext,
        ListContext,
        StepContext
        ):
        name = y.__name__.replace('Context','').lower()
        class tmp(y):
            if name == 'build':
                __doc__ = '''executes the {} of {}'''.format(name, x)
            elif name == 'clean':
                __doc__ = '''cleans the project {}'''.format(x)
            elif name == 'install' or name == 'uninstall':
                __doc__ = '''CURRENTLY NOT SUPPORTED:{}s the project {}'''.format(name, x)
            elif name == 'list':
                __doc__ = '''lists the targets to execute for {}'''.format(x)
            elif name == 'step':
                __doc__ = '''executes tasks in a step-by-step fashion, for \
debugging of {}'''.format(x)
            cmd = name + '_' + x
            variant = x

    dox = 'doxygen'
    class tmp(BuildContext):
        __doc__ = '''creates the {} documentation of {}'''.format(dox, x)
        cmd = dox + '_' + x
        fun = dox
        variant = x

In the function build and doxygen the build variant is checked, the the correct sources are selected. If no build variant is specified, an error message is displayed, telling to specify a variant. This is generally implemented something like this:

Listing 7.14 Implementation to ensure a variant build
def build(bld):
    import sys
    import logging
    from waflib import Logs
    if not bld.variant:
        bld.fatal('A {} variant must be specified, run \'{} {} --help\'\
'.format(bld.cmd, sys.executable, sys.argv[0]))

    bld.env.__sw_dir = os.path.normpath('embedded-software')

    src_dir = os.path.normpath('mcu-{}'.format(bld.variant))
    ldscript = os.path.join(bld.env.__sw_dir, src_dir, 'src', bld.env.ldscript_filename)

For doxygen it is implemented very similar:

Listing 7.15 Implementation to ensure a variant doxgen API documentation
def doxygen(bld):
    import sys
    import logging
    from waflib import Logs

    if not bld.variant:
        bld.fatal('A build variant must be specified, run \'{} {} --help\'\
'.format(sys.executable, sys.argv[0]))

    if not bld.env.DOXYGEN:
        bld.fatal('Doxygen was not configured. Run \'{} {} --help\'\
'.format(sys.executable, sys.argv[0]))

    _docbuilddir = os.path.normpath(bld.bldnode.abspath())
    doxygen_conf_dir = os.path.join('documentation', 'doxygen')
    os.makedirs(_docbuilddir, exist_ok=True)
    conf_file = 'doxygen-{}.conf'.format(bld.variant)
    doxygenconf = os.path.join(doxygen_conf_dir, conf_file)

7.4. wscripts

As mentioned above, the build process is described in wscripts, which are itself valid python scripts. The top is foxbms\wscript which defines the functions needed for the build, e.g., configure, build etc.

From the top wscript the other wscript s are called recursive by bld.recurse(..).

To get a detailed view on the single build steps, see these files.

7.5. Building and Linking with a Library

The toolchain enables to build a library and then links against the library. It is possible to build and link multiple libraries into the binaries.

The wscript in embedded-software\libs lists the libraries to be build. Libraries that should be build have to be listed here. Based on the example library testlib it is shown how to include a library in foxBMS.

Note

In fact the libs directory contains two test libraries (foxbms-user-lib and my-foxbms-library) in order to show how multiple libraries can be used. The first example shows how to build one single library and in the second example it is shown, how to build and use more than one library.

General Setup

The wscript in embedded-software\libs lists in the function bld.recurse(...) the directories containing the sources for the to be build library (see line 11 in Listing 7.16).

Listing 7.16 The wscript in embedded-software\libs
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
def build(bld):
    header_files_src = bld.path.ant_glob('**/*.h')
    header_filenames = [n.name for n in header_files_src]
    if not len(header_filenames) == len(set(header_filenames)):
        duplicates = list(set([x for x in header_filenames
                          if header_filenames.count(x) > 1]))
        err_msg = 'There are headers with the same file name recursively ' \
                  ' inside directory \'{}\':\n' \
                  '{}'.format(bld.path.abspath(), '\n'.join(duplicates))
        bld.fatal(err_msg)
    bld.recurse('testlib myfoxbmslibrary')

Note

For every additional library that should be build, the directory containing the library must be added to this line, e.g., if the library sources are in a directory called advancedalgorithms this lines needs to look like this:

Listing 7.17 Adding the library source directory advancedalgorithms
1
2
def build(bld):
    bld.recurse('testlib myfoxbmslibrary advancedalgorithms')

The actual build of the library is defined in the wscript in embedded-software\libs\testlib. All source and header files have to be in the library directory, for this example these are are testlib.c and testlib.h. The library is then build by the wscript in embedded-software\libs\testlib.

Listing 7.18 explained in detail:

  • All source files that should be build have to be listed in the srcs list (see line 5-6.).

  • The name of the library is set to foxbms-user-lib (see line 11).

    Note

    For later on further expanding the advanced-algorithms example, the library name my-advanced-algorithm is assumed.

Listing 7.18 wscript
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
def build(bld):
    header_files_src = bld.path.ant_glob('*.h')
    header_files_tar = [os.path.join(bld.env.INCLUDE_DIR_LIBS,
                        x.path_from(bld.path)) for x in header_files_src]
    srcs = ' '.join([
        os.path.join('testlib.c')])

    includes = os.path.join(bld.bldnode.abspath()) + ' '
    includes += ' '.join([bld.path.get_src().abspath()])

    bld.stlib(target='foxbms-user-lib',
              source=srcs,
              includes=includes,
              cflags=bld.env.CFLAGS_foxbms,
              features=['size', 'copy_libs', 'check_includes'])
    bld(features='subst',
        source=header_files_src,
        target=header_files_tar,
        is_copy=True)
  • The object files (*.o) and the library (*.a) are found in build\libs\embedded-software\libs\testlib\.

  • The libraries (the *.a-files) are copied in build\lib\*.a. When building the default dummy libraries these are build\lib\libfoxbms-user-lib.a and build\lib\libmy-foxbms-library.a. The lib-prefix is generated automatically. This task is generated by the copy_lib feature (see line 14).

  • The headers are copied to build\include (see line 15-17).

Warning

The header names for all library headers are checked for uniqueness. Header files with the same name recursively inside the libs directory will lead to a build error. This check needs to be performed, as all headers get copied to include directory at build\include.

The Library

The library declaration of super_function(uint8_t a, uint8_t b) is in testlib.h:

Listing 7.19 testlib.h
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
#ifndef TESTLIB_H_
#define TESTLIB_H_

/*================== Includes =============================================*/
#include <stdint.h>

/*================== Macros and Definitions ===============================*/

/*================== Constant and Variable Definitions ====================*/
extern uint8_t super_variable;

/*================== Function Prototypes ==================================*/
extern uint16_t super_function(uint8_t a, uint8_t b);

/*================== Function Implementations =============================*/

#endif /* TESTLIB_H_ */

The library defines a function super_function(uint8_t a, uint8_t b) in testlib.c:

Listing 7.20 testlib.c
1
2
3
uint16_t super_function(uint8_t a, uint8_t b) {
    return a + b;
}

Building

  1. Build the library (or libraries):

    Listing 7.21 Build the libraries
    python tools\waf build_libs
    

    Now all libraries are present in build\libs and the headers are in build\include.

  2. Configure the foxBMS project to work with a library, in the first example it is the foxbms-user-lib library.

    Listing 7.22 Configuration with library useage
    python tools\waf configure --libs=foxbms-user-lib
    

Note

For including the hypothetical my-advanced-algorithm library the command would be:

python tools\waf configure --libs=my-advanced-algorithm
  1. Include the headers needed for the functions in the sources and use the functions as needed.

    Listing 7.23 Include header and use a function from the library
     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    /*================== Includes =============================================*/
    /* some other includes  */
    #include "testlib.h"
    
    /*================== Function Prototypes ==================================*/
    
    /*================== Function Implementations =============================*/
    int main(void) {
        uint16_t a = 0;
        /* Use the function super_function from the library */
        a = super_function(2,2);
        /* other code */
    }
    
  2. Build the foxBMS binary as usual.

    Listing 7.24 Build the foxBMS binary
    python tools\waf build_primary
    

Building with Multiple Libraries

A project may want to use multiple libraries. For this example the two provided dummy libraries are assumed (foxbms-user-lib and my-foxbms-library).

  1. The project is configured to work with both libraries. The library names are given as command line argument separated by comma (no additional whitespace).

    Listing 7.25 Configuration with multiple library useage
    python tools\waf configure --libs=foxbms-user-lib,my-foxbms-library
    
  2. Build the library (or libraries):

    Listing 7.26 Build the libraries
    python tools\waf build_libs
    
  3. Include the headers needed for the functions in the sources and use the functions as needed.

    Listing 7.27 Include header and use a function from the library
     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    /*================== Includes =============================================*/
    /* some other includes  */
    #include "testlib.h"
    #include "myfoxbmsalgorithms.h"
    
    /*================== Function Prototypes ==================================*/
    
    /*================== Function Implementations =============================*/
    int main(void) {
        uint16_t a = 0;
        /* Use the function super_function from the library foxbms-user-lib */
        a = super_function(2,2);
        uint16_t b = 0;
        /* Use the function another_super_function from the library my-foxbms-library */
        a = another_super_function(2,2);
        /* other code */
    }
    
  4. Build the foxBMS binary as usual.

    Listing 7.28 Build the foxBMS binary
    python tools\waf build_primary
    

7.6. Building the Test

Note

The test builds described in this section are not mandatory. They can be used as a simple check that the software architecture is kept (see foxBMS software architecture).

In order to verify that low level drivers (i.e., the drivers in embedded-software\mcu-common\driver) do not relay on higher level modules (e.g., FreeRTOS, database, etc.) two tests are included. These can be build by

Listing 7.29 Build bare tests for primary and secondary mcu
python tools\waf configure
python tools\waf configure build_primary_bare
python tools\waf configure build_secondary_bare