5. Unit Tests

We are using the ceedling framework for unit testing. It is a tool based on rake that automatically generates the mocks (with CMock) and test-runners and executes the unit tests with Unity as assertion toolset.

Unit testing tries to test properties of the interfaces of the smallest units in your software. (The linked Wikipedia article is a good read for getting into the topic.) This test tries not to catch every detail of the implementation on the specific MCU or the behavior of modules to each other. For this reason the tests are compiled on a development machine instead of the target.

5.1. Usage

The tool is encapsulated in our waf-script. Additionally it is automatically executed on our CI-jobs. The usage as a waf-tool is explained in How to Use Unit tests. This part also explains how to use the tool directly in a shell. The benefit is that the output in the shell is colored and single tests can be executed.

5.2. Writing tests

A unit test should always cover exactly one file from our source code. See the test-files in the tests/unit subdirectory for reference.

5.2.1. Headers of unit tests

Typically, the header of a unit-test-file consists of three parts.

/* part one: include the unity-header */
#include "unity.h"

/* part two: include Mocks */
#include "MockHL_spi.h"
#include "Mockbeta.h"
#include "Mockdatabase.h"

/* part three: include the code under test */
#include "adc.h"

With this information ceedling will gather all necessary c-files to these headers and compile a test-runner.

5.2.2. Using mocks

Most of the time the code that we want to test will rely on other parts of the codebase. A proper unit test should however only test single units and not more. (This would otherwise be a integration- or high-level-test.)

In order to isolate our code from the rest, we can replace the calls to the surrounding software with so-called Mocks.

You can tell ceedling to create a mock of a file by including the header file and prepending the word Mock to it. The mocks will be created on demand. If you are running into issues with certain parts of the API not being available in your Mock, then check the generated header file, if they are present. Ceedling will have issues parsing the header if the functions are marked with special attributes or if they are defined as inline. In this case it might be necessary to tell CMock that those attributes are “strippable” with the :strippables: yaml in project.yml.

Before calling a function in your test that is calling one of your mocked functions internally, you must tell the test-runner that you are expecting the mock to be called. The generated mocks will have a *_Expect-function for this.

For further information on using CMock please refer to the linked documents in Further reading.

5.2.3. Using assertions

The main part of a test will be asserting that the output of a function is as expected. Unity supplies many assertions that are summed up pretty well on GitHub.

A typical usage might look like this:

/* the function to be tested is sum(a, b) */
TEST_ASSERT_EQUAL(5, sum(2, 3));

5.2.4. Getting started with a new test

As a start for a unit test you can create an empty file that compiles a test-runner together with all needed mocks and the file under test.

You can start this by creating the test-c-file. If the file that you want to test is called adc.c with header adc.h then you would name the unit test test_adc.c. The directory structure of our application is mirrored in the unit-test-directory.

Start with the basic includes:

#include "unity.h"
#include "adc.h"

When you compile with this, the test-runner will complain about missing symbols. Read the output closely and find where the called functions are defined. Then add mocks for those by including their headers:

#include "unity.h"
#include "MockHL_spi.h"
#include "Mockbeta.h"
#include "Mockdatabase.h"
#include "Mockepcos_b57251v5103j060.h"

#include "adc.h"

When you are done with that there will sometimes still be missing variables. In this example definitions from spi_cfg.c are missing. You have two options: add the variables directly to the test-file (this can be useful if you have to access them in the unit test) or adding the header that introduces those variables:

#include "unity.h"
#include "MockHL_spi.h"
#include "Mockbeta.h"
#include "Mockdatabase.h"
#include "Mockepcos_b57251v5103j060.h"

#include "spi_cfg.h"

#include "adc.h"

You can start with this variant in order to get the unit test to compile and then later on add the variables directly to your test and remove the added header.

Special care has to be taken when registers are directly accessed in code. This will most certainly lead to a SEGFAULT in the test-runner, as this access to some random address is not allowed or sensible on the native-platform. Ceedling has a guide on replacing these register accesses with variables, but since we are using a HAL it can often be enough to just replace HAL-calls with mocks.

To finish it, your test needs a startup and teardown function. These functions will be called between each single test in a file.

5.2.5. Do not be too explicit

Take care to make your tests not too explicit. That means that you should not care to much about internals of the functions that you are testing. Focus on the public interfaces of the functions.

One option is to ignore the calls to irrelevant mock functions. You can add a call of *_Ignore()-mocks at the start of your test.

If necessary, tests can be supported with helper functions in the file under test. For example it is possible to add a getter in order to obtain the internal instance of the struct that tracks the state of a state-machine of a module. This should however be kept to a minimum. If a function is not testable at all, it is better to rewrite the function than to try to add complicated tests for it.

5.2.6. Checklist for unit tests

This section aims to be a help for writing complete unit tests. It is an adaption of the checklist described in [MF14] with comments based on experience.

  1. Unit tests MUST show that the module under test performs its intended action. For example if a module transforms values, then a test should show this mechanism in action.

  2. Unit tests SHOULD show that the module under test does not perform any unintended side effects. This is not a strict requirement, as it is hard to prove. Generally, side effects should be omitted whenever possible in order to make the code clearer and in the end more testable. During development and execution of the unit tests special care should be taken in order to spot potential side effects.

  3. Equivalence classes SHALL be created for every input of a module. An equivalence class describes which class of input values takes the same internal path inside the function. For example, when writing plausibility checks, all values that are plausible would be one group and the values that are not plausible would be another group. All values that are loaded by the module or are passed to the module when calling it are considered input.

  4. Boundary values MUST be tested for every equivalence class. By testing the values at the limits of an equivalence class, it is made sure that the function is able to handle the outer limits. This is for many implementations a good heuristic.

  5. The value Zero SHALL be tested as input value. This applies to both variables and pointers. Zero value is a good indicator for unveiling critical edge-cases such as a division by zero or access to a null-pointer.

  6. Outputs SHALL be forced to their limits. By writing a test that forces the output values to their maximum values, it can be made sure that the function is able to handle them.

  7. Outputs SHOULD be forced above their limits. Depending on the module and its implementation, it can be impossible to force the output values of the module beyond its limits. Otherwise, this should be tested.

  8. The first and last element of a sequence SHALL be tested. Especially for arrays, the first and last element shall be checked. This can help to unveil off-by-one and out-of-bounds-access to arrays.

  9. Sequences SHALL be tested with zero, one, two and maximum elements. If the module under test consumes a sequence such as an array, the cases where the array has zero, one, two and the maximum number of elements shall be tested.

  10. The test SHOULD reach full branch and line-coverage. This is only a requirement for safety critical modules. However, it is helpful nevertheless for writing better code by avoiding untestable conditions.

5.3. Typical test failures

This part tries to sum up typical causes for failing unit tests.

5.3.1. Ceedling crash

Sometimes you will see a unit test failing locally with output similar to the following:

Test 'test_bal.c'
-----------------
ERROR: Ceedling Failed

ERROR: Shell command failed.
> Shell executed command:
[gcc call redacted, not relevant here]
> And exited with status: [1].

rake aborted!
ShellExecutionException: ShellExecutionException
ceedling:23:in 'load'
ceedling:23:in '<main>'
Tasks: TOP => test:all
(See full trace by running task with --trace)

This happens most of the time on the first run after changing the branch. It seems to be an internal issue with how we are using ceedling, but is resolved by simply executing the task again (which will then work normally).

5.3.2. Missing coverage

On CI (or when computing the coverage locally), you can encounter error messages similar to this:

Could not find coverage results for ../../src/app/driver/io/io.c

Could not find coverage results for ../../src/app/driver/mic/ltc/common/mic_dma.c

Could not find coverage results for ../../src/app/driver/mic/nxp/common/mic_dma.c

[...]

There were files with no coverage results: aborting.

This means that new files have been introduced that are not covered by any unit test (this test can be even empty). In order to solve this you have two options:

  1. add the offending file to :uncovered_ignore_list: in project.yml

  2. add an (empty) test to the unit tests.

While the first option is quicker, it is preferred to use mainly the second option. This way we can have a good overview of the actual coverage of our code.

5.3.3. SEGFAULT of the test-runner

If the test-runner aborts an ceedling provides not much output this most of the time means that you have run into a segmentation fault. This happens when the code that is currently tested tries to access a register of the MCU directly. In a native build this register address lies outside of the program space and will thus be terminated by your operating system.

5.3.4. Linker issues and unknown symbols

In most other cases it will be a missing symbol during compilation or linking. Read the output closely and try to mock or add any symbols that are missing.

5.4. Using coverage reports

When you build the unit tests with coverage enabled you will find a coverage report locally in build/unit_test/artifacts/gcov as HTML.

This report knows two metrics: line-coverage and branch-coverage. Line-coverage is the percentage of lines of code that have been passed while running the unit test. Branch-coverage is the percentage of decisions (e.g., when passing an if-instruction) that have been covered by the test-code.

The goal is to find uncovered lines and branches and to extend the unit tests in order to reach finally 100% coverage.

5.5. Full example of a unit test

This section explores how a full unit test could look like. As an example we have a math.h:

Listing 5.1 Header for math.h
1
extern uint16_t addition(uint8_t a, uint8_t b);

This header is implemented in math.c:

Listing 5.2 Implementation of math.h
1
2
3
4
5
#include "math.h"

uint16_t addition(uint8_t a, uint8_t b) {
    return a + b;
}

In order to make sure that the function is implemented as expected, we can test its properties with a unit test. Since this is an addition-function it should be commutative. That means a + b == b + a. Another issue could be that the implementation has an overflow when adding two full-scale uint8_t. The following unit-test-fixture has two tests that will be executed one after another.

Listing 5.3 Unit test for math.h
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
/* The test-library */
#include "unity.h"

/* The file under test */
#include "math.h"

void setUp(void) {
    /* this function is called before executing a test;
    we can use it in more complex setups in order to initialize
    the code under test into a defined state */
}

void tearDown(void) {
    /* this function is called after executing a test;
    we can use it for cleaning up if needed */
}

/* The title that we use should be as explicit as possible */
void testAdditionIsCommutative(void) {
    /* In order to test whether this function is commutative we have
    different options. For a unit test it is OK to have hard-coded values. */

    /* we call the function addition with 2 and 3 and
    expect the output to be 5 */
    TEST_ASSERT_EQUAL(5u, addition(2u, 3u));

    /* the other way around calling the function with 3 and 2
    should also return 5 */
    TEST_ASSERT_EQUAL(5u, addition(3u, 2u));
}

void testAdditionOfTwoUint8Max(void) {
    /* The result of adding two UINT8_MAX should be 510 */
    TEST_ASSERT_EQUAL(510, addition(UINT8_MAX, UINT8_MAX));
}

When we run these tests, ceedling will automatically search the dependencies, generate the boilerplate-code for an executable, compile all fragments, link a test-runner and run this test-runner in order to check that the assertions are correct.

The described code is isolated from any other code. In “real” implementations, however, we will often find code that depends on other modules and functions. Unit testing aims to test code-fragments in insulation. Therefore it is often used with a mocking-framework which replaces dependencies with stubs whose behavior can be controlled by test.

Let’s assume that we want to test a function in another module that depends on math.h. This modules checks the plausibility of two values of which the sum should be same as a third value.

Listing 5.4 Header of plausibility.h
1
2
3
4
/* return 0 if values are not plausible, 1 if plausible */
extern uint8_t check_values_plausible(uint8_t voltage_1,
                                      uint8_t voltage_2,
                                      uint16_t sum_of_voltage);
Listing 5.5 Implementation of plausibility.h
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
/* These values would be filled in from some other function in this module */
uint8_t voltage_1 = 6;
uint8_t voltage_2 = 12;
uint16_t sum_of_voltage = 18;

uint8_t check_values_plausible(uint8_t voltage_1,
                               uint8_t voltage_2,
                               uint16_t sum_of_voltage) {
    uint16_t calculated_sum = addition(voltage_1, voltage_2);
    return calculated_sum == sum_of_voltage;
}

In the unit test we replace addition with a mock. That means we do not care if math.h is implemented correctly and replace it with our own implementation.

Listing 5.6 Unit test for plausibility.h
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
#include "unity.h"

/* ceedling will recognize by this that we want to create a mock of
math.h and analyze the file and create it */
#include "Mockmath.h"

/* this is the file under test */
#include "plausibility.h"

void setUp(void) {
}

void tearDown(void) {
}

void testCheckValuesPlausibleWithPlausibleValues(void) {
    /* we tell the mock to expect that it is called with 5 and 4 and that
    it should return 9; This will be added to a list and the mocks will
    be "consumed" from this list during execution */
    addition_ExpectAndReturn(5, 4, 9);

    /* assert that check_values_plausible returns true */
    TEST_ASSERT(check_values_plausible(5, 4, 9));
}

void testCheckValuesPlausibleWithImplausibleValues(void) {
    addition_ExpectAndReturn(5, 4, 9);

    /* assert that implausible values will return false */
    TEST_ASSERT_FALSE(check_values_plausible(5, 4, 10));
}

Both tests should pass. The second test asserts that function returns false if the values are not plausible. For both tests the mock is told what it will receive and what it should return. Variants of these mock-functions exist that allow to ignore calls to this function, or to add a callback for more complex return values.