1  <a id="top"></a>
  2  # Best practices and other tips on using Catch2
  3  
  4  ## Running tests
  5  
  6  Your tests should be run in a manner roughly equivalent with:
  7  
  8  ```
  9  ./tests --order rand --warn NoAssertions
 10  ```
 11  
 12  Notice that all the tests are run in a large batch, their relative order
 13  is randomized, and that you ask Catch2 to fail test whose leaf-path
 14  does not contain an assertion.
 15  
 16  The reason I recommend running all your tests in the same process is that
 17  this exposes your tests to interference from their runs. This can be both
 18  positive interference, where the changes in global state from previous
 19  test allow later tests to pass, but also negative interference, where
 20  changes in global state from previous test causes later tests to fail.
 21  
 22  In my experience, interference, especially destructive interference,
 23  usually comes from errors in the code under test, rather than the tests
 24  themselves. This means that by allowing interference to happen, our tests
 25  can find these issues. Obviously, to shake out interference coming from
 26  different orderings of tests, the test order also need to be shuffled
 27  between runs.
 28  
 29  However, running all tests in a single batch eventually becomes impractical
 30  as they will take too long to run, and you will want to run your tests
 31  in parallel.
 32  
 33  
 34  <a id="parallel-tests"></a>
 35  ## Running tests in parallel
 36  
 37  There are multiple ways of running tests in parallel, with various level
 38  of structure. If you are using CMake and CTest, then we provide a helper
 39  function [`catch_discover_tests`](cmake-integration.md#automatic-test-registration)
 40  that registers each Catch2 `TEST_CASE` as a single CTest test, which
 41  is then run in a separate process. This is an easy way to set up parallel
 42  tests if you are already using CMake & CTest to run your tests, but you
 43  will lose the advantage of running tests in batches.
 44  
 45  
 46  Catch2 also supports [splitting tests in a binary into multiple
 47  shards](command-line.md#test-sharding). This can be used by any test
 48  runner to run batches of tests in parallel. Do note that when selecting
 49  on the number of shards, you should have more shards than there are cores,
 50  to avoid issues with long-running tests getting accidentally grouped in
 51  the same shard, and causing long-tailed execution time.
 52  
 53  **Note that naively composing sharding and random ordering of tests will break.**
 54  
 55  Invoking Catch2 test executable like this
 56  
 57  ```text
 58  ./tests --order rand --shard-index 0 --shard-count 3
 59  ./tests --order rand --shard-index 1 --shard-count 3
 60  ./tests --order rand --shard-index 2 --shard-count 3
 61  ```
 62  
 63  does not guarantee covering all tests inside the executable, because
 64  each invocation will have its own random seed, thus it will have its own
 65  random order of tests and thus the partitioning of tests into shards will
 66  be different as well.
 67  
 68  To do this properly, you need the individual shards to share the random
 69  seed, e.g.
 70  ```text
 71  ./tests --order rand --shard-index 0 --shard-count 3 --rng-seed 0xBEEF
 72  ./tests --order rand --shard-index 1 --shard-count 3 --rng-seed 0xBEEF
 73  ./tests --order rand --shard-index 2 --shard-count 3 --rng-seed 0xBEEF
 74  ```
 75  
 76  Catch2 actually provides a helper to automatically register multiple shards
 77  as CTest tests, with shared random seed that changes each CTest invocation.
 78  For details look at the documentation of
 79  [`CatchShardTests.cmake` CMake script](cmake-integration.md#catchshardtestscmake).
 80  
 81  
 82  ## Organizing tests into binaries
 83  
 84  Both overly large and overly small test binaries can cause issues. Overly
 85  large test binaries have to be recompiled and relinked often, and the
 86  link times are usually also long. Overly small test binaries in turn pay
 87  significant overhead from linking against Catch2 more often per compiled
 88  test case, and also make it hard/impossible to run tests in batches.
 89  
 90  Because there is no hard and fast rule for the right size of a test binary,
 91  I recommend having 1:1 correspondence between libraries in project and test
 92  binaries. (At least if it is possible, in some cases it is not.) Having
 93  a test binary for each library in project keeps related tests together,
 94  and makes tests easy to navigate by reflecting the project's organizational
 95  structure.
 96  
 97  
 98  ---
 99  
100  [Home](Readme.md#top)