Release.md
1 # How to release Arti 2 3 ## Tools and notation 4 5 We're going to use the following. 6 Why not upgrade to the latest version before you start? 7 8 * cargo-semver-checks 9 * cargo-edit 10 11 In the documentation below, 12 we sometimes use environment variables to indicate 13 particular versions of Arti. 14 For example, in the release I just did, I had: 15 16 ``` 17 LAST_VERSION=1.1.5 18 THIS_VERSION=1.1.6 19 ``` 20 21 ## Are we ready to release? 22 23 Before we can finally release, we need to check a few things 24 to make sure we aren't going to break our users. 25 26 1. Make sure CI is passing. 27 28 2. After making sure that the pipeline as a whole has passed, 29 look at every part of the pipeline that "passed with warnings". 30 Are the warnings what we expect? 31 If it's failing, is is it failing for the reasons we anticipated, 32 or have new failures crept in? 33 34 3. Look at the current list of exceptions in our automated tooling. 35 Are they still relevant? 36 (There are exceptions in 37 `cargo_audit`, 38 `check_doc_features`, 39 and 40 `check_licenses`.) 41 42 4. Do we have any open [issues] or [merge requests] tagged "Blocker"? 43 44 [issues]: https://gitlab.torproject.org/tpo/core/arti/-/issues/?label_name%5B%5D=Blocker 45 [merge requests]: https://gitlab.torproject.org/tpo/core/arti/-/merge_requests/?label_name[]=Blocker 46 47 5. Does `maint/fixup-features` produce any results? 48 If so, fix them. 49 50 Note: fixup-features should be run with the top-level Cargo.toml 51 as an argument: 52 ``` 53 cargo run -p fixup-features -- --exclude examples/ --exclude maint/ Cargo.toml 54 ``` 55 56 6. Does `maint/semver-checks "arti-v$LAST_VERSION"` find any issues 57 not noted in our semver.md files? 58 If so, add them. 59 60 Note that you can do these steps _in parallel_ with "preparing for the 61 release" below. 62 63 ## Preparing for the release 64 65 Note that you can do these steps _in parallel_ with "are we ready to 66 release?" above. 67 68 1. Check for breaking changes to our dependencies. 69 In the weeks between releases, I try to run: 70 `cargo upgrade --dry-run --compatible=ignore --incompatible=allow`. 71 This will tell us if any of our dependencies 72 have new versions that will not upgrade automatically. 73 74 2. Check for non-breaking changes to our dependencies. 75 A day or two before release, I try to run: 76 `cargo update`. 77 This will replace each of our dependencies in Cargo.lock 78 with the latest version. 79 (I recommend doing this a bit before the release 80 to make sure that we have time 81 to deal with any surprising breakage.) 82 83 3. Write a changelog. 84 85 I start by running 86 `git log --topo-order --reverse arti-v${LAST_VERSION}..`, 87 and writing a short summary of everything I find into 88 new sections of the changelog. 89 I try to copy the list of sections 90 from previous changelogs, to keep them consistent. 91 92 If I have to take a break in the middle, 93 or if time will pass between now and the release, 94 I add a note like 95 `UP TO DATE WITH 726f666c2d2d6d61646520796f75206c6f6f6b21` 96 to remind me where I need to start again. 97 98 The script `maint/changelog-syntax-fiddle` 99 can be helpful to write the cross-refernces more easily. 100 101 4. Finish the changelog. 102 103 When the changelog is done, run 104 `maint/update-md-links CHANGELOG.md` 105 to auto-generate markdown links 106 to our gitlab repositories. 107 (This script may need a Python venv, as it depends on 108 a specific version of the `mistune` library.) 109 Then, fill in the URLs for any links that the script couldn't find - 110 they'll be marked with X X X todo markers. 111 112 Run `maint/thanks arti-v${LAST_VERSION}` 113 to generate our list of acknowledgments; 114 insert this into the changelog. 115 116 Add an acknowledgement for the current sponsor(s). 117 118 4. Determine what semver/version update to do to each crate. 119 120 We need to sort our crates into the following tiers. 121 * Unstable (0.x) `tor-*` and `arti-*` crates. 122 (Bump minor version, to the same value for each crate.) 123 * No changes were made. 124 (No version bump needed) 125 * Only non-functional changes were made. 126 (Bump the version of the crate, but not the depended-on version.) 127 * Functional changes were made, but no APIs were added or broken. 128 (Bump patchlevel.) 129 * APIs were added. 130 (Bump patchlevel if major == 0; else bump minor.) 131 * APIs were broken. 132 (Bump minor if major == 0; else bump major.) 133 134 For all `tor-*` and `arti-*` crates with 0.x version numbers 135 (which, as of March 2024 includes all `tor-*` crates, 136 and all `arti-*` crates apart from `arti` itself), 137 we always bump the minor version and release, 138 even if there have been no changes. 139 For other crates, things are more complicated: 140 141 You can identify crates that have no changes using `maint/changed_crates`: 142 ``` 143 maint/changed_crates -v "arti-v$LAST_VERSION" 2>&1 >/dev/null | grep -i "no change" 144 ``` 145 146 To see whether a crate has only non-functional changes, 147 you have to use `git diff`. Sorry! 148 Historically, trivial changes 149 are small tweaks in the clippy lints, 150 or documentation/comment improvements. 151 152 To determine whether any APIs were added, 153 look in the semver.md files. 154 (We often forget to add these; 155 but fortunately, 156 most of our crates currently have major version 0, 157 so we don't need to distinguish "functional changes" 158 from "new APIs".) 159 160 To determine whether any APIs were broken, 161 look in the semver.md files. 162 The `cargo semver-checks` tool 163 can also identify some of these, 164 but its false-negative rate is high. 165 166 You may also want to look over the crate-by-crate diffs. 167 This can be quite time-consuming. 168 169 ## Final preparation for a release. 170 171 Wait! Go back and make sure 172 that you have done everything in the previous sections 173 before you continue! 174 175 0. Declare a merge freeze. 176 177 Tell `network-team` 178 (via email and IRC) 179 that the tree is now frozen, 180 and no MRs should be merged. 181 182 1. Finalize the changelog. 183 184 Make sure that the date is correct. 185 Make sure that the acknowledgments and links are correct, 186 if they might have gotten stale. 187 188 2. Increase all appropriate version numbers. 189 190 For unstable (0.x) `tor-*` and `arti-*` crates, 191 determine the new minor number. 192 `maint/list_crates --versions | grep -P '^tor|^arti'` 193 will show you the existing versions, 194 which should usually all be the same. 195 Pick the next minor version, and, for each such crate: 196 `cargo set-version -p ${CRATE} 0.${MINOR}.0`. 197 198 For other crates: 199 200 * For each crate with functional changes: 201 `cargo set-version --bump {patch|minor|major} -p ${CRATE}`. 202 203 * For crates with non-functional changes, 204 you can use the `bump_nodep` script: 205 `./maint/bump_nodep crate1 crate2 crate3` ... 206 207 In all cases, make sure you commit `Cargo.lock` changes too. 208 209 If `hashx` or `equix` have changed since the last release, you must also update 210 `crates/{hashx,equix}/bench/Cargo.lock`, 211 which aren't in the workspace for 212 [Reasons](https://gitlab.torproject.org/tpo/core/arti/-/issues/1351): 213 214 ``` 215 (cd crates/hashx/bench && cargo update) 216 (cd crates/equix/bench && cargo update) 217 ``` 218 219 3. Check for side effects from bumping versions! 220 221 As of March 2024, you can skip this section 222 for `tor-*` and `arti-*`, since: 223 * `arti` is the only non-0.x `arti-*` or `tor-*` crate; 224 * `arti` does not expose types from our lower-layer crates; 225 * None other of our crates depend on `tor-*` or `arti-*` crates. 226 Therefore all necessary bumps have been done. 227 228 You may need to perform these checks 229 if there have been semver bumps 230 to non-`arti-*` or `tor-*` crates, 231 when other such crates expose their types. 232 233 Does a previously unchanged crate 234 depend on a crate that got a version bump? 235 Now _that_ crate counts as changed, 236 and needs a version bump of its own. 237 238 You can list all crates that have changed 239 since the last version, 240 but not had their versions bumped, 241 with the command 242 `./maint/changed_crates -u ${LAST_VERSION}`. 243 (Note the `-u`.) 244 245 Run `maint/semver-checks` again: 246 It should be quiet now that you bumped all the versions. 247 248 4. Then make sure that CI passes, again. 249 250 ## The actual release itself. 251 252 1. Run `maint/cargo-publish --dry-run ${THIS_VERSION}` 253 to see what it thinks. 254 255 If all seems well, run it without the `--dry-run` option. 256 257 If it fails, you may be able to rerun the script 258 after fixing the cause. 259 It is supposed to be idempotent. 260 261 2. We tag the repository with `arti-v${THIS_VERSION}` 262 263 To do this, run 264 `maint/tag-arti-release ${THIS_VERSION}` 265 (which will include the output of `maint/list_crates --version`). 266 267 (Note to self: if you find that gpg can't find your yubikey, 268 you'll need to run 269 `sudo systemctl restart pcscd` 270 to set things right. 271 I hope that nobody else has this problem.) 272 273 3. Tell `network-team` 274 (via email and IRC) 275 that the tree is open 276 for new MRs to be merged! 277 278 ## Post-release 279 280 1. Remove all of the semver.md files: 281 `git rm crates/*/semver.md`. 282 283 2. Write a blog post. 284 285 3. There's a "pages" branch in the arti repository 286 that controls the contents of <https:/arti.torproject.org/>. 287 Once the blog post is published, 288 update that "pages" branch to refer to the new version. 289 (For an example MR, see [!2271].) 290 291 4. Once the blog post is published, 292 send Bekeela an email. 293 (For the duration of the Zcash grant, 294 Tor needs to inform the funder when we put out a new release. 295 This is being handled by Bekeela, our Grants Manager.) 296 297 5. Did you create any new crates? 298 If so, you need to make sure that they are owned (on crates.io) 299 by the right set of developers. 300 If you aren't sure, run `maint/cargo-crate-owners`. 301 302 [!2271]: https://gitlab.torproject.org/tpo/core/arti/-/merge_requests/2271 303 304 <!-- ================================================== --> 305 306 # Making a patch release of a crate, eg to fix a semver breakage 307 308 If one crate needs to be updated urgently 309 (eg to fix a build breakage), 310 we can release it separately as follows. 311 312 (The underlying principle is to try to 313 make precisely the minimal intended change; 314 and avoid picking up other changes made to Arti crates 315 since the last release.) 316 317 0. File a ticket describing the problem. 318 This will be referenced in MRs, release docs, etc. 319 320 1. Prepare the changes on a git branch 321 starting from the most recent `arti-v...` tag, **not main**. 322 323 (If this happens more than once per Arti release cycle, 324 start the branch from the previous crate-specific tag; 325 we should perhaps create a `patch-releases` branch 326 if this is other than a very rare case.) 327 328 2. Changes on this branch should include: 329 * The underlying desired change (whether to code or `Cargo.toml`s) 330 * Version bump to the crate being released 331 * Absolutely minimal necessary changes to `Cargo.lock`. 332 (`Cargo.lock` is mostly for CI here.) 333 * New stanza in `CHANGELOG.md` describing the point release 334 (and cross-referencing to the ticket). 335 336 3. Make an MR of this branch. 337 State in the MR that the branch is intended for a patch update. 338 Obtain a review. 339 340 4. If the CI passes, and review is favourable, 341 merge the MR and go on to step 5. 342 343 5. If the CI failed, you may want to proceed anyway. 344 For example, `cargo audit`, or builds with Nightly, 345 can rot, without us having done anything. 346 If you think you wish to do this: 347 348 1. Review the CI output for the failing command, 349 and make a decision about it, 350 in consultation with your reviewer. 351 If you wish to tolerate this failure: 352 353 2. Create a new git branch `bodge` 354 off the patch release branch. 355 356 3. In `.gitlab-ci.yml`, 357 prepend the failing shell command with `true X X X` 358 (only with the three Xs cuddled together). 359 This will cause it to not run, but "succeed". 360 The `XXXs` will cause the blocking-todos CI job to fail, 361 hindering accidental merge of this branch to main. 362 363 4. Make a new MR of this `bodge` branch. 364 Say "do not merge" in the title 365 and mark it draft. 366 367 5. Await CI on the bodge MR. 368 If all that fails is the blocking todos, 369 you're good to go. 370 Otherwise, go back to sub-step 1 371 and review the further failures. 372 373 6. Double check which MR page you are looking at: 374 you must be looking at the real patch MR, 375 not the CI bodge MR, or some unrelated MR. 376 Then, tell gitlab to merge the branch despite CI failure. 377 378 6. After the patch branch is merged, 379 check it out locally and make releases: 380 381 1. Check that you are on the patch branch, not `main`. 382 383 2. `cargo publish -p the-affected-crate`. 384 385 3. `git tag -s the-affected-crate-vX.Y.Z` 386 and push the tag. 387 388 7. File any followup tickets and/or do any post-patch cleanup. 389 390 8. Consider whether to make a blog post about the patch release. 391 392