Post

Software Testing & Methodologies

Posted Updated
By Robert Caddy 6 min read

Useful Links & Guides

Terminology

  • “Unit test” - Test a single “unit” of code such as a function, class method, constructor, etc.
  • “Integration Test” - Test how multiple units of code work together. Example: testing an entire Riemann solver instead of just the individual functions within it
  • “End-to-End (E2E) testing” or “System testing” - Testing the entire code to make sure it produces the correct result or action. Tests should probably be comparing floating point values not strings.
  • “Regression test” - Test that new changes have not broken the code or introduced new bugs. Unit, integration, and E2E tests can all be regression tests
  • “Performance Test” - Make sure the thing you’re testing doesn’t take more time, memory, or other resources than expected
  • “Mocking” - Some tests might have a dependency that doesn’t behave in a predictable way for testing and so we need to make fake or “mocked” version that does. An example might be a random number generator. In production we want it to be random but in testing we might always want it to return the same value.
  • “Automated Testing” - Running tests automatically based on some event. Typically this event is something like pushing to GitHub or opening a pull request.
  • “Continuous Integration (CI)” - Quickly integrating changes from different developers using some sort of automated build, test, deployment system. Read the link in the Useful Links & Guides section for more info.
  • “Test Driven Development (TDD)” - Writing the tests BEFORE you write the code that will be tested. This can be done as part of the design phase to make sure the end product matches all the requirements.

Testing Strategies

  • What should you test exactly?
    • Test each path within the code. I.e. each branch in every if-elif-else statement or any similar branching
    • Test with good inputs AND bad inputs, the code should fail when given bad inputs and it should fail the way you expect it to.
    • Test edge cases. Things that are uncommon but could throw a wrench in things. Examples might be zero velocity etc
    • For system tests make sure to test the number of total time steps
  • Death Tests
    • Special naming convention should be used. Specifically the name of all test suites with a death test in them should end in DeathTest Reason
  • We need to have a rigorous naming scheme for tests and test suites. Google likely has suggestions
  • If you have series of tests that require the same setup (initializing a class etc) then you can use a test fixture to automate that. Each test will use its own instance of the fixture
    • Resources can be shared between tests in the same suite, it just requires some extra syntax. Look at GTests documentation for details
  • Performance Tests
    1. Benchmark the code. There’s a library for this or you can do it yourself
    2. Compare the execution time to the fiducial time with EXPECT_NEAR using an appropriate margin of error
  • Tests should also be done with different compilers(both across vendors and versions), OS’s, hardware, etc.

Naming & Location of Tests

  • Tests should be named clearly and similarly to what they test so they can be easily found. I.e. the code that tests hllc.cpp should be name something like hllc-tests.cpp
  • Where to store the tests?
    1. Right next to the file it tests. This can lead to very cluttered directories but means that the testing is never “out of sight and out of mind”
    2. If the source directory is broken up into subdirectories for each “topic” then in a subsubdirectory within the topic. I.e. within repo-root/src/topic-dir/topic-dir-test. This is probably the most organized and cleanest way of doing things but it requires a really solid directory structure for source files
    3. In repo-root/tests. This might be the best option when your source files are just all in a single directory without sub directories. However it means that the tests are out of sight when editing, and therefore likely out of mind. Also, since the testing directory should maintain the same structure as the source directory it can be hard to maintain.

C++ Testing Frameworks

  • GoogleTest
    • GoogleTest Primer
    • Feature rich
    • Reasonably easy to use
    • Includes death tests, important since Cholla doesn’t have much in the way of error handling
    • Has an extensive list of related projects that enhance its features
    • Threadsafe if the pthreads library is available. i.e. everywhere except Windows
    • Requires an entire external library
    • Is only designed with CMake and Bazel in mind. Integration with Make works fine.
  • Catch2
    • Very simple and easy to implement, though lacking some features
    • Header only
    • Popular
    • Due to the header only nature it can significantly increase compile times if used improperly
    • No death tests
    • Not thread safe
  • DOCtest
    • A stripped down, faster version of Catch2
    • Fast but not very feature rich
    • No death tests
    • Not thread safe
  • Boost Test
    • Not bad, just doesn’t really stand out compared to other options
    • No death tests
    • Not thread safe
  • CTest
    • Might work if you’re using CMake already, if not then it’s not worth it.

Automated Testing

Naming Scheme

Googletest allows you to choose which tests to run at runtime via standard pattern matching. To facilitate this we need to have rigorous system for naming test suites and tests. Here you will find those conventions. Details of the naming scheme we used for Cholla can be found here.

Testing, Guide
Comparing Floats C++ Floating Point Numbers Framework GitHub Actions
This post is licensed under CC BY 4.0 by the author.