testing.md
1 # Unit Testing 2 3 In Erlang you would typically call three things a unit in the context of 4 unit testing: a function, a module and a process (or rather the code a 5 process runs). A Unit test suite is a collection of tests that describe 6 which unit they address and then provide a reasonable coverage over that 7 unit. If the unit depends on other parts, such as file system, libraries, 8 etc, then those might be mocked (i.e., be replaced by code that is 9 predictable and repeatable). 10 11 We aim for having a unit test suite for each module, because very often our 12 functions are too little and simple to spend a complete unit test suite on 13 and our processes are mainly contained in one module with some library 14 modules to build upon. 15 16 **If a unit test fails, then we know that the error is in that module** 17 18 Best practice is when unit tests are side effect free. That is, when you 19 run the test, you always get the same result. That means that one may need 20 to mock the parts that can influence the outcome of a test. For side-effect 21 free, or pure, functions, this is no issue. For processes however, or 22 functions that interact with the operating system or other parts of the 23 system, this is highly relevant. 24 25 26 ## Running unit tests 27 28 We place our unit tests in a directory called `test`, for example, unit 29 tests for modules in the `aecore` application are in 30 [`apps/aecore/test`](../tree/master/apps/aecore/test/). 31 32 Running all unit tests in the project can be done by 33 34 ```shell 35 make eunit-latest 36 ``` 37 38 It will automatically search in all `apps/*/test` directories after files 39 named `*_tests.erl` which are considered containing Eunit tests. Note that 40 it is important to name the unit tests with the same name as the Erlang 41 module in order to have the tool chain recognize them. Thus an Erlang 42 module called M.erl in `apps/*/src/` shall have a module called M_tests.erl 43 in `apps/*/test/`. 44 45 Unit tests can be run per tests file, for example 46 47 ```shell 48 make eunit-latest TEST=aec_peers 49 ``` 50 51 will run the unit tests in 52 [`apps/aecore/test/aec_peers_tests.erl`](../tree/master/apps/aecore/test/aec_peers_tests.erl). 53 54 Note that we also have a directory [`test`](../tree/master/test/) on the 55 top level. One should NOT put Eunit tests in there, but only test suites 56 run with common test that test interaction between the different 57 applications in `apps`. In short, this directory is meant for application 58 integration tests. 59 60 61 # Integration Testing 62 63 The purpose of integration testing is to ensure that newly added code does 64 not break the system. We run some basic tests on three Erlang nodes These 65 nodes should be able to perform the basic operations, such as mining 66 blocks, synchronizing blocks, handling transactions, etc. 67 68 For each application there might be some integration tests provided that 69 focus on the integration of processes in that application. The vision is to 70 have tests that integrate a number of different applications in the top 71 level test directory. 72 73 Integration tests are executed by common test and are therefore stored in 74 files named `*_SUITE.erl`. Running all integration tests is done by 75 76 ```shell 77 make ct-latest 78 ``` 79 80 Since the complete collection of integration tests is run at each pull 81 request of new software, the tests have to be able to run in a reasonably 82 short time interval (aim for running the complete suite to completion in 15 83 minutes or less). 84 85 The API used in these tests are Erlang functions, typically the API of gen 86 servers and the like. We mock the mining algorithm or use one that can mine 87 very fast, such as lean16 or so. 88 89 90 ## Running Specific Tests 91 92 To run a specific suite, group or test case, you can use the environment 93 variables `SUITE`, `GROUP` and `TEST`. Examples: 94 95 ```shell 96 $ make ct-latest SUITE=apps/aecore/test/aecore_sync 97 $ make ct-latest SUITE=apps/aehttp/test/aehttp_integration GROUP=external_endpoints 98 $ make ct-latest SUITE=apps/aehttp/test/aehttp_integration GROUP=external_endpoints TEST=get_transaction 99 ``` 100 101 The environment variables can be combined to narrow the scope of which 102 tests to run, however `SUITE` must always be used when `GROUP` and/or 103 `TEST` is used. The prefix `_SUITE` is automatically added to the specified 104 suite name. Note that we need to specify the full path to the suite, as the 105 code is divided into apps in subfolders. 106 107 The `DIR` environment variable can be used to run all the tests in a 108 specific directory (or directories - most of the test related flags above 109 accept comma separated lists): 110 111 ```shell 112 $ make ct-latest DIR=apps/aehttp/test 113 $ make ct-latest DIR=apps/aehttp/test,apps/aecore/test 114 ``` 115 116 117 ## Typical Requirements to Test 118 119 1. Each node should be able to mine a block and add it to the chain of the 120 system. 121 122 2. If a fork is created then discarded the fork with lowest PoW (exact 123 requirements to be added) 124 125 3. We are able to post a spend transaction that is occurring in the chain. 126 127 4. We are able to pay a fee for a spend transaction. 128 129 5. We collect a transaction fee by mining. 130 131 6. We can start and restart the applications. 132 133 134 # System Testing 135 136 The purpose of system testing is to make sure the system as a whole works. 137 That means that we want at least 3 nodes running, preferably on different 138 platforms (OS and architecture). These nodes should be able to perform the 139 basic operations, such as mining blocks, synchronizing blocks, handling 140 transactions, etc. 141 142 We establish this using docker. First read the information on [how to work 143 with our docker 144 images](https://github.com/aeternity/epoch/blob/master/docs/docker.md). 145 146 ```shell 147 $ make system-test-deps 148 ``` 149 150 The API used in these tests (apart from building and starting) is purely 151 the web interface. We use a mining algorithm that can mine very fast, such 152 as lean16 or so. In the [User Acceptance Test](User-Acceptance-testing) the 153 real miner will be tested. 154 155 How to run the system tests: 156 157 ```shell 158 $ make system-test 159 ``` 160 161 How to keep the docker containers and networks created during the tests: 162 163 ```shell 164 $ EPOCH_DISABLE_NODE_CLEANUP=1 make system-test 165 ``` 166 167 168 ## Running Specific Tests 169 170 To run a specific suite, group or test case, you can use the environment 171 variables `SUITE`, `GROUP` and `TEST`. Examples: 172 173 ```shell 174 $ make system-test SUITE=aest_sync 175 $ make system-test SUITE=aest_perf GROUP=long_chain 176 $ make system-test SUITE=aest_sync TEST=new_node_joins_network 177 ``` 178 179 The environment variables can be combined to narrow the scope of which 180 tests to run, however `SUITE` must always be used when `GROUP` and/or 181 `TEST` is used. The prefix `_SUITE` is automatically added to the specified 182 suite name. 183 184 185 ## Running Special Groups of Test Suites 186 187 How to run a fast subset of the system tests: 188 189 ```shell 190 $ make system-smoke-test-deps 191 $ make smoke-test-run 192 ``` 193 194 How to run the system tests meant to be run only locally i.e. not run by 195 CI: 196 197 ```shell 198 $ make local-system-test 199 ``` 200 201 202 ## Typical Requirements to Test 203 204 1. The nodes in the system should be able to connect to its peers. 205 - If the node has been back-listed by the peers before, then we are able 206 to re-connect after X 207 208 2. Each node should be able to mine a block and add it to the chain of the 209 system. 210 211 3. If one node disconnects, a fork is created, that is: 212 1. Discarded when re-connecting to the other nodes if other chain has 213 more PoW (exact requirements to be added) 214 2. Accepted by the other nodes if this fork has a larger PoW (exact 215 requirements to be added) 216 217 4. Each node is able to: 218 1. Post a spend transaction that is occurring in the chain. 219 2. Pay a fee for a spend transaction. 220 3. Collect a transaction fee by mining. 221 222 5. We can access all endpoints specified in the swagger.yaml file with 223 expected results. (This needs further refinement) 224 225 6. A node that is stopped for 5 minutes (enough to create additional blocks 226 on the chain) should be able to restart and satisfy 1, 2 and 5 above. 227 228 7. A node that sends invalid data to other nodes is black-listed. 229 230 231 ## System Testing on Continuous Integration Infrastructure 232 233 On branch `master`: 234 * System tests (`make system-test`) are run twice a day. 235 * No system tests are run whenever a pull request is merged. 236 237 On "normal" branches (i.e. not `env/*`, not `system-tests`): 238 * System *smoke* tests are run (`make smoke-test-run`). 239 240 On branch `system-tests`: 241 * System tests (`make system-test`) are run. 242 243 On tags: 244 * No system tests are run. 245 246 247 # User Acceptance Testing 248 249 Although the terminology might be a bit confusing, we test system 250 requirements in the user acceptance testing phase (see Testing-general). 251 252 The purpose of this test is to make sure that when users download our 253 software, they can start working with it the way they expect. This also 254 includes that they are able to install it according to the release notes 255 and other documents describing what to do. However, automation of these 256 tasks is out of reach and therefore this is not done systematically. 257 258 For the user acceptance test we need an infrastructure with quite some 259 miners, but few enough to make sure we can contribute with a mined block 260 in, say, an hour. We refer to this as the uat-net. If we can connect our 261 updated miner software to a uat-net in a user acceptable way, then we 262 assume it will also work for the main net. It is important that the uat-net 263 has miners that run the same release software as miners on the main net as 264 well as some other versions (which is also possible on the main net). 265 266 The API used in these tests (apart from building and starting) is purely 267 the web interface. 268 269 Typical requirements we should test: 270 271 1. A user clone or pull of the master branch of the repository should build 272 using `make` without errors. 273 2. A user clone or pull of the master branch of the repository should run 274 `make test` without errors. 275 276 3. A user downloaded or created production package (`make prod-package`) 277 that is correctly configured against the uat-net: 278 1. Should be able to connect to the uat-net and show the correct top 279 block of the uat-net. 280 2. Should be able to mine a block and add it to the chain of the uat-net 281 (within an hour). 282 3. If disconnected from the uat-net, a fork is created, that is 283 discarded when re-connecting to the uat-net (we assume the uat-net to 284 have more mining power than a single miner). 285 286 4. A user downloaded or created production package is, by running for one 287 hour, able to: 288 1. Post a spend transaction that is occurring in the uat-net chain. 289 2. Pay a fee for a spend transaction. 290 3. Collect a transaction fee by mining. 291 292 5. A user can access all endpoints specified in the swagger.yaml file with 293 expected results. (This needs further refinement) 294 295 6. A user should be able to stop the miner and restart it after 15 minutes 296 with the effect that 3.i, 3.ii and 5 above hold. 297 298 7. A user should be able to update the software and continue with the saved 299 chain as starting point. In that case 3, 4 and 5 above should be 300 fulfilled.