1  ## Integration Tests
  2  
  3  Unit tests are automatically run as part of the regular build. Integration tests
  4  aren't run automatically since they have nontrivial requirements to run.
  5  
  6  ### Three ways of running the tests
  7  
  8  We are reusing the same set of integration tests in 3 ways. In each of
  9  those cases, we get the code we're testing to test from a different source.
 10  
 11  - Run them as part of development. In this case, we get the CLI
 12    and the framework libraries from the source repository.
 13  - Run them as integration tests in the pipeline. In this case, we get a specific
 14    version of the CLI and the framework libraries from a set of candidate NPM
 15    packages.
 16  - Run them continuously, as a canary. In this case, we get the latest CLI and
 17    the framework libraries directly from the package managers, same as an
 18    end user would do.
 19  
 20  To hide the differences between these different ways of running the tests,
 21  there are 3 scripts. They all take as command-line argument the ACTUAL test
 22  script to run, and prepare the environment in such a way that the tests
 23  will use the `cdk` command and the libraries from the distribution selected.
 24  
 25  To run the CLI integ tests in each configuration:
 26  
 27  ```
 28  $ test/integ/run-against-repo test/integ/cli/test.sh
 29  $ test/integ/run-against-dist test/integ/cli/test.sh
 30  $ test/integ/run-against-release test/integ/cli/test.sh
 31  ```
 32  
 33  To run a single integ test in the source tree:
 34  
 35  ```
 36  $ test/integ/run-against-repo test/integ/cli/test.sh -t 'SUBSTRING OF THE TEST NAME'
 37  ```
 38  
 39  To run regression tests in the source tree:
 40  
 41  ```
 42  $ test/integ/test-cli-regression-against-current-code.sh [-t '...']
 43  ```
 44  
 45  Integ tests can run in parallel across multiple regions. Set the `AWS_REGIONS`
 46  environment variable to a comma-separate list of regions:
 47  
 48  ```
 49  $ env AWS_REGIONS=us-west-2,us-west-1,eu-central-1,eu-west-2,eu-west-3 test/integ/run-against-repo test/integ/cli/test.sh
 50  ```
 51  
 52  Elements from the list of region will be exclusively allocated to one test at
 53  a time. The tests will run in parallel up to the concurrency limit imposed by
 54  jest (default of 5, controllable by `--maxConcurrency`) and the available
 55  number of elements. Regions may be repeated in the list in which case more
 56  than one test will run at a time in that region.
 57  
 58  If `AWS_REGIONS` is not set, all tests will sequentially run in the one
 59  region set in `AWS_REGION`.
 60  
 61  Run with `env INTEG_NO_CLEAN=1` to forego cleaning up the temporary directory,
 62  in order to be able to debug 'cdk synth' output.
 63  
 64  ### CLI integration tests
 65  
 66  CLI tests will exercise a number of common CLI scenarios, and deploy actual
 67  stacks to your AWS account.
 68  
 69  REQUIREMENTS
 70  
 71  * All packages have been compiled.
 72  * Shell has been preloaded with AWS credentials.
 73  
 74  Run:
 75  
 76  ```
 77  yarn integ-cli
 78  ```
 79  
 80  This command runs two types of tests:
 81  
 82  #### Functional
 83  
 84  These tests simply run the local integration tests located in [`test/integ/cli`](./test/integ/cli). They test the proper deployment of stacks and in general the correctness of the actions performed by the CLI.
 85  
 86  You can also run just these tests by executing:
 87  
 88  ```console
 89  yarn integ-cli-no-regression
 90  ```
 91  
 92  #### Regression
 93  
 94  Validate that previously tested functionality still works in light of recent changes to the CLI. This is done by fetching the functional tests of the previous published release, and running them against the new CLI code.
 95  
 96  These tests run in two variations:
 97  
 98  - **against local framework code**
 99  
100    Use your local framework code. This is important to make sure the new CLI version
101    will work properly with the new framework version.
102  
103    > See a concrete failure [example](https://github.com/aws/aws-cdk-rfcs/blob/master/text/00110-cli-framework-compatibility-strategy.md#remove---target-from-docker-build-command)
104  
105  - **against previously release code**
106  
107    Fetches the framework code from the previous release. This is important to make sure
108    the new CLI version does not rely on new framework features to provide the same functionality.
109  
110    > See a concrete failure [example](https://github.com/aws/aws-cdk-rfcs/blob/master/text/00110-cli-framework-compatibility-strategy.md#change-artifact-metadata-type-value)
111  
112  You can also run just these tests by executing:
113  
114  ```console
115  yarn integ-cli-regression
116  ```
117  
118  Note that these tests can only be executed using the `run-against-dist` wrapper. Why? well, it doesn't really make sense to `run-against-repo` when testing the **previously released code**, since we obviously cannot use the repo. Granted, running **against local framework code** can somehow work, but it required a few too many hacks in the current codebase to make it seem worthwhile.
119  
120  ##### Implementation
121  
122  The implementation of the regression suites is not trivial to reason about and follow. Even though the code includes inline comments, we break down the exact details to better serve us in maintaining it and regaining context.
123  
124  Before diving into it, we establish a few key concepts:
125  
126  - `CANDIDATE_VERSION` - This is the version of the code that is being built in the pipeline, and its value is stored in the `build.json` file of the packaged artifact of the repo.
127  - `PREVIOUS_VERSION` - This is the version previous to the `CANDIDATE_VERSION`.
128  - `CLI_VERSION` - This is the version of the CLI we are testing. It is **always** the same as the `CANDIDATE_VERSION` since we want to test the latest CLI code.
129  - `FRAMEWORK_VERSION` - This is the version of the framework we are testing. It varries between the two variation of the regression suites.
130  Its value can either be that of `CANDIDATE_VERSION` (for testing against the latest framework code), or `PREVIOUS_VERSION` (for testing against the previously published version of the framework code).
131  
132  Following are the steps involved in running these tests:
133  
134  1. Run [`./bump-candidate.sh`](../../bump-candidate.sh) to differentiate between the local version and the published version. For example, if the version in `lerna.json` is `1.67.0`, this script will result in a version `1.67.0-rc.0`. This is needed so that we can launch a verdaccio instance serving local tarballs without worrying about conflicts with the public npm uplink. This will help us avoid version quirks that might happen during the *post-release-pre-merge-back* time window.
135  
136  2. Run [`./align-version.sh`](../../scripts/align-version.sh) to configure the above version in all our packages.
137  
138  3. Build and Pack the repository. The produced tarballs will be versioned with the above version.
139  
140  4. Run `test/integ/run-against-dist test/integ/test-cli-regression-against-latest-release.sh` (or `test/integ/test-cli-regression-against-latest-code.sh`)
141  
142  5. First, the `run-against-dist` wrapper will run and:
143  
144      - Read the `CANDIDATE_VERSION` from `build.json` and export it.
145      - [Launch verdaccio](./test/integ/run-against-dist#L29) to serve all local tarballs (serves the `CANDIDATE_VERSION` now)
146      - [Install the CLI](./test/integ/run-against-dist#L30) using the `CANDIDATE_VERSION` version `CANDIDATE_VERSION` env variable.
147      - Execute the given script.
148  
149  6. Both cli regression test scripts run the same [`run_regression_against_framework_version`](./test/integ/test-cli-regression.bash#L22) function. This function accepts which framework version should the regression run against, it can be either `CANDIDATE_VERSION` or `PREVIOUS_VERSION`. Note that the argument is not the actual value of the version, but instead is just an [indirection indentifier](./test/integ/test-cli-regression.bash#L81). The function will:
150  
151      - Calculate the actual value of the previous version based on the candidate version. (fetches from github)
152      - Download the previous version tarball from npm and extract the integration tests.
153      - Export a `FRAMWORK_VERSION` env variable based on the caller, and execute the integration tests of the previous version.
154  
155  7. Our integration tests now run and have knowledge of which framework version they should [install](./test/integ/helpers/cdk.ts#L74).
156  
157  That "basically" it, hope it makes sense...
158  
159  ### Init template integration tests
160  
161  Init template tests will initialize and compile the init templates that the
162  CLI ships with.
163  
164  REQUIREMENTS
165  
166  * Running on a machine that has all language tools available (JDK, .NET Core,
167    Python installed).
168  * All packages have been compiled.
169  * All packages have been packaged to their respective languages (`pack.sh`).
170  
171  Run:
172  
173  ```
174  npm run integ-init
175  ```
176  
177  ## Integration test modes
178  
179  These two sets of integration tests have 3 running modes:
180  
181  - Developer mode, when called through `npm run`. Will use the source tree.
182  - Integration test, when called from a directory with the build artifacts
183    (the `dist` directory).
184  - Canaries, when called with `IS_CANARY=true`. Will use the build artifacts
185    up on the respective package managers.
186  
187  The integration test and canary modes are used in the CDK publishing pipeline
188  and the CDK canaries, respectively. You wouldn't normally need to run
189  them directly that way.