003-consolidate-core-packages.md
1 # Consolidate Core Atom Packages 2 3 ## Status 4 5 Accepted 6 7 ## Summary 8 9 Atom's official distribution is comprised of 92 core packages which provide its built-in functionality. These packages currently live in their own independent repositories in the Atom organization, all with their own separate issues, PRs, releases, and CI configurations. This RFC proposes that by consolidating most, if not all, of these core packages back into the `atom/atom` repo, we will see the following benefits: 10 11 - Less confusion for new contributors 12 - Simpler core package contribution experience 13 - Greatly reduced burden for maintainers 14 15 ## Motivation 16 17 Let's cover each of the bullet points mentioned above: 18 19 ### Less confusion for contributors 20 21 Imagine that a new contributor wants to add a small new feature to the `tree-view` package. The first place they are likely to look is the `atom/atom` repository. Scanning through the folders will lead to a dead end as nothing that looks like `tree-view` code can be found. They might take one of the following steps next: 22 23 - By reading README.md, maybe they will decide to click the link to the Atom Flight Manual and _maybe_ find the [Contributing to Official Atom Packages](https://flight-manual.atom.io/hacking-atom/sections/contributing-to-official-atom-packages/) page there 24 - They could read the CONTRIBUTING.md file which [has a section](https://github.com/atom/atom/blob/master/CONTRIBUTING.md#atom-and-packages) that explains where to find the repos for core packages and how to contribute, but we don't really have a clear pointer to that in our README.md 25 - If they don't happen to find that page, they might use Google to search for "atom tree view" and find the atom/tree-view repo and _maybe_ read the CONTRIBUTING.md file which sends them to Atom's overall contribution documentation 26 - They might go to the Atom Forum or Slack community to ask how to contribute to a particular part of Atom and *hopefully* get a helpful response that points them in the right direction 27 28 Having all of the core Atom packages represented in a top-level `packages` folder, even if they all don't actually live in the repo, will go a long way to making the core package code more discoverable. 29 30 ### Simpler core package contribution experience 31 32 Separating core Atom features out into individual repositories and delivering them to Atom builds via `apm` is a great idea in theory because it validates the Atom package ecosystem and gives developers many examples of how to develop an Atom package. It also gives Atom developers real-world experience working with Atom's APIs so that we ensure community package authors have the same hackability that Atom developers enjoy. 33 34 On the other hand, having these packages live in separate repositories and released "independently" introduces a great deal of overhead when adding new features. Here is a comparison of the current package development workflow contrasted to what we could achieve with consolidated packages: 35 36 #### Current Package Development Workflow 37 38 For example, to add a single feature to the `tree-view` package, one must: 39 40 1. Fork and clone the `tree-view` repository to their computer (making sure to pull the commit relevant to the version of Atom they are working with) 41 1. Run `apm install` and `apm link` inside of the repo folder 42 1. Make their desired changes to the code 43 1. Open a PR to the `tree-view` repo and wait for CI to pass and a maintainer to review it 44 1. Work with maintainers to get the PR approved and merged 45 46 After this is finished, an Atom maintainer must take the following steps: 47 48 1. Clone the `tree-view` repo 49 2. Run `apm publish` to publish a new release of the package 50 3. Edit `package.json` in the Atom repo to reflect the new version of `tree-view` 51 4. Commit and push the changes to the relevant branch where the change belongs (`master` or `1.nn-releases`) 52 53 #### Simplified Package Development 54 55 If we were to move `tree-view` (or any other core Atom package) back into `atom/atom`, the development workflow would look more like this: 56 57 1. Fork and clone `atom/atom` and switch to a release branch if necessary 58 1. Build Atom and launch it in dev mode 59 1. Make desired changes to the code in `packages/tree-view` 60 1. Open a PR on `atom/atom` and wait for CI to pass and a maintainer to review it 61 1. Work with maintainers to get the PR approved and merged 62 63 At this point, the change is merged into Atom and ready for inclusion in the next release. 64 65 ### Greatly reduced burden for maintainers 66 67 Since packages all have their own repositories, this means that we have to watch 91 different repos for issues and pull requests. This also means that we have to redirect issues filed on `atom/atom` to the appropriate repository when a user doesn't know where it belongs. Even more importantly, there's not an easy way to prioritize and track issues across the Atom organization without using GitHub Projects. 68 69 Also, as mentioned above, there's the added duty of doing the package "version dance" when we merge any new PRs to a package repository: publish the package update, update `package.json` in Atom. It's very easy to forget to do this and not have community contributions included in the next Atom release! 70 71 The more core packages live in `atom/atom`, the less work Atom maintainers have to do overall. 72 73 ## Explanation 74 75 Many of Atom's core packages now live in the core `atom/atom` repository. To the Atom user, this change will be imperceptible as these packages still show up in the list of Core Packages in the Settings View. Users can still optionally disable these packages. 76 77 For maintainers and contributors, there will be less juggling of repositories and no more publishing of updates to these packages with `apm`: 78 79 Contributors now clone and build `atom/atom` to work on improvements to core packages. They will no longer have to use `apm link` in dev mode to test changes they make to packages in the repo's `packages` folder. Core packages that aren't consolidated still have folders under `packages` with README.md files that point to the home repository for that package. 80 81 When a contributor sends a PR to `atom/atom` that only affects files in a folder under `packages`, only the specs for the relevant package folders will be executed using Atom's CI scripts. This means that a full Atom build will not be required when no Atom Core code is changed in a PR. Package specs are also now run against all 3 OSes on Atom `master` and release builds. 82 83 Atom maintainers no longer have to publish new versions to consolidated core packages and then edit `package.json` to bump the package version in a particular Atom release branch (Stable, Beta, or `master`). When a PR against a consolidated core package in `atom/atom` is merged, no version number change is required and the changes will immediately be a part of the next release from that branch. 84 85 ## Drawbacks 86 87 One possible drawback of this approach is that there might be some initial confusion where core Atom packages live, especially if some are consolidated into `atom/atom` and others still live in their own repositories. We will manage this confusion by doing the following: 88 89 - Include a `README.md` file in the `packages` folder which lists core packages that are not consolidated in the Atom repo. This will enable users to find the home repositories of those packages. 90 91 - Archive the repositories for consolidated core packages, but only after migrating existing issues, merging or closing existing PRs, and updating the README.md to point to the new home of the package code. 92 93 Also, contributors will now have to fork, clone, and build `atom/atom` to contribute to core packages where they would previously just need to clone the package repository. This might put added burden on them such as installing necessary build dependencies on their machine that they wouldn't otherwise need. It is very likely we could simplify this process for them, though. 94 95 One final drawback is that it will now be harder to have single-package maintainers. We currently have 7 core packages where there is a maintainer who isn't a part of the core Atom maintainers team. These maintainers generally are able to merge community PRs and make commits to those packages with their own judgement. If we get rid of individual package repositories, do we now make those maintainers full Atom maintainers? 96 97 ## Rationale and alternatives 98 99 The Motivation section explains most of the rationale, so this section will focus on the process of consolidating packages back into `atom/atom`. The set of packages we've chosen to consolidate were evaluated based on a few factors: 100 101 - Number of open issues and PRs (exclude any with > 10 open PRs) 102 - Time since last update (longer duration since last update is prioritized) 103 - Number of package-only maintainers on the repo (exclude any with package maintainers for now) 104 105 Using this criteria, all 91 packages have been evaluated and categorized to determine whether they are good candidates for consolidation: 106 107 #### Initial Consolidation Candidates 108 109 | Package | Open Issues | Open PRs | Outside Maintainers | Last Updated | 110 |---------|-------------|----------|---------------------| -------------| 111 | **[about]** | 2 | 0 | 0 | 7/11/18 | 112 | **[archive-view]** | 10 | 0 | 0 | 6/3/18 | 113 | **[atom-dark-syntax]** | 5 | 0 | 0 | 12/6/17 | 114 | **[atom-dark-ui]** | 1 | 2 | 0 | 2/13/18 | 115 | **[atom-light-syntax]** | 1 | 0 | 0 | 10/17/16 | 116 | **[atom-light-ui]** | 1 | 0 | 0 | 2/13/18 | 117 | **[autoflow]** | 17 | 4 | 0 | 4/17/18 | 118 | **[autosave]** | 13 | 0 | 0 | 9/16/17 | 119 | **[background-tips]** | 3 | 2 | 0 | 2/17/18 | 120 | **[base16-tomorrow-dark-theme]** | 5 | 0 | 0 | 1/10/17 | 121 | **[base16-tomorrow-light-theme]** | 1 | 0 | 0 | 1/10/17 | 122 | **[bookmarks]** | 19 | 4 | 0 | 12/10/17 | 123 | **[bracket-matcher]** | 74 | 8 | 0 | 3/20/18 | 124 | **[command-palette]** | 18 | 6 | 0 | 2/27/18 | 125 | **[dalek]** | 2 | 0 | 0 | 2/28/18 | 126 | **[deprecation-cop]** | 5 | 0 | 0 | 9/7/17 | 127 | **[dev-live-reload]** | 4 | 0 | 0 | 11/14/17 | 128 | **[encoding-selector]** | 11 | 2 | 0 | 4/19/18 | 129 | **[exception-reporting]** | 5 | 0 | 0 | 2/6/18 | 130 | **[git-diff]** | 38 | 1 | 0 | 1/18/18 | 131 | **[go-to-line]** | 5 | 2 | 0 | 1/25/18 | 132 | **[grammar-selector]** | 3 | 1 | 0 | 4/12/18 | 133 | **[image-view]** | 4 | 4 | 0 | 7/9/18 | 134 | **[incompatible-packages]** | 1 | 0 | 0 | 4/25/17 | 135 | **[keybinding-resolver]** | 11 | 3 | 0 | 7/6/18 | 136 | **[language-clojure]** | 13 | 3 | 0 | 1/26/18 | 137 | **[language-coffee-script]** | 9 | 2 | 0 | 11/1/17 | 138 | **[language-csharp]** | 1 | 1 | 0 | 4/27/18 | 139 | **[language-css]** | 6 | 7 | 0 | 6/11/18 | 140 | **[language-gfm]** | 52 | 9 | 0 | 6/15/18 | 141 | **[language-git]** | 4 | 2 | 0 | 4/18/17 | 142 | **[language-html]** | 11 | 4 | 0 | 7/5/18 | 143 | **[language-hyperlink]** | 2 | 3 | 0 | 10/25/17 | 144 | **[language-json]** | 1 | 0 | 0 | 5/11/18 | 145 | **[language-less]** | 5 | 1 | 0 | 6/11/18 | 146 | **[language-make]** | 7 | 3 | 0 | 11/26/16 | 147 | **[language-mustache]** | 0 | 0 | 0 | 2/5/18 | 148 | **[language-objective-c]** | 2 | 0 | 0 | 12/1/15 | 149 | **[language-php]** | 25 | 7 | 0 | 6/11/18 | 150 | **[language-property-list]** | 1 | 0 | 0 | 3/11/17 | 151 | **[language-python]** | 33 | 4 | 0 | 6/18/18 | 152 | **[language-ruby]** | 38 | 10 | 0 | 10/25/17 | 153 | **[language-ruby-on-rails]** | 9 | 6 | 0 | 12/7/17 | 154 | **[language-sass]** | 12 | 5 | 0 | 5/2/18 | 155 | **[language-shellscript]** | 12 | 3 | 0 | 6/18/18 | 156 | **[language-source]** | 0 | 0 | 0 | 1/6/15 | 157 | **[language-sql]** | 6 | 4 | 0 | 1/26/18 | 158 | **[language-text]** | 1 | 0 | 0 | 3/9/18 | 159 | **[language-todo]** | 10 | 6 | 0 | 1/26/18 | 160 | **[language-toml]** | 1 | 0 | 0 | 1/6/18 | 161 | **[language-typescript]** | 6 | 0 | 0 | 6/18/18 | 162 | **[language-xml]** | 2 | 1 | 0 | 6/12/17 | 163 | **[language-yaml]** | 8 | 2 | 0 | 3/9/18 | 164 | **[line-ending-selector]** | 10 | 0 | 0 | 5/18/18 | 165 | **[link]** | 0 | 1 | 0 | 11/14/17 | 166 | **[metrics]** | 1 | 2 | 0 | 7/5/18 | 167 | **[notifications]** | 29 | 8 | 0 | 3/22/18 | 168 | **[one-dark-syntax]** | 4 | 0 | 0 | 5/27/18 | 169 | **[one-dark-ui]** | 13 | 1 | 0 | 5/1/18 | 170 | **[one-light-syntax]** | 2 | 1 | 0 | 5/27/18 | 171 | **[one-light-ui]** | 2 | 0 | 0 | 5/1/18 | 172 | **[open-on-github]** | 8 | 3 | 0 | 11/21/17 | 173 | **[package-generator]** | 10 | 2 | 0 | 11/16/17 | 174 | **[status-bar]** | 25 | 3 | 0 | 11/6/17 | 175 | **[styleguide]** | 12 | 2 | 0 | 4/12/18 | 176 | **[tabs]** | 66 | 7 | 0 | 5/13/18 | 177 | **[timecop]** | 5 | 0 | 0 | 11/4/17 | 178 | **[update-package-dependencies]** | 0 | 0 | 0 | 12/10/17 | 179 | **[welcome]** | 0 | 0 | 0 | 11/21/17 | 180 | **[whitespace]** | 31 | 6 | 0 | 5/30/18 | 181 | **[wrap-guide]** | 3 | 4 | 0 | 11/27/17 | 182 183 #### Packages to be Consolidated Later 184 185 The following packages will not be consolidated until the stated reasons can be resolved or we decide on a consolidation strategy for them: 186 187 | Package | Open Issues | Open PRs | Outside Maintainers | Last Updated | Reason | 188 |---------|-------------|----------|---------------------|--------------|-------| 189 | **[find-and-replace]** | 219 | 17 | 0 | 6/4/18 | Too many open PRs | 190 | **[fuzzy-finder]** | 89 | 22 | 0 | 5/17/18 | Too many open PRs | 191 | **[github]** | | | | | Independent project | 192 | **[language-c]** | 53 | 15 | 0 | 7/10/18 | Too many open PRs | 193 | **[language-go]** | 12 | 2 | **1** | 6/18/18 | Package maintainer, possibly inactive? | 194 | **[language-java]** | 8 | 2 | **1** | 6/11/18 | Package maintainer | 195 | **[language-javascript]** | 66 | 12 | 0 | 7/6/18 | Too many open PRs | 196 | **[language-perl]** | 17 | 1 | **1** | 10/30/17 | Package maintainer, possibly inactive? | 197 | **[markdown-preview]** | 139 | 12 | 0 | 1/8/18 | Too many open PRs | 198 | **[settings-view]** | 137 | 18 | 0 | 5/17/18 | Too many open PRs | 199 | **[snippets]** | 57 | 4 | **1** | 4/17/18 | Package maintainer | 200 | **[solarized-dark-syntax]** | 8 | 3 | **1** | 5/27/18 | Package maintainer | 201 | **[solarized-light-syntax]** | 2 | 3 | **1** | 5/27/18 | Package maintainer | 202 | **[spell-check]** | 68 | 14 | **1** | 5/25/18 | Too many open PRs, package maintainer | 203 | **[symbols-view]** | 86 | 13 | 0 | 12/10/17 | Too many open PRs | 204 | **[tree-view]** | 210 | 36 | 0 | 3/21/18 | Too many open PRs | 205 206 #### Packages to Never Consolidate 207 208 These packages will not be consolidated because they would inhibit contributions from our friends in the Nuclide team at Facebook: 209 210 - **[autocomplete-atom-api]** 211 - **[autocomplete-css]** 212 - **[autocomplete-html]** 213 - **[autocomplete-plus]** 214 - **[autocomplete-snippets]** 215 216 ### Consolidation Process 217 218 To consolidate a single core package repository back into `atom/atom`, the following steps will be taken: 219 220 1. All open pull requests on the package's repository must either be closed or merged before consolidation can proceed 221 1. The package repository's code in `master` will be copied over to Atom's `packages` folder in a subfolder bearing that package's name. 222 1. Atom's `package.json` file will be updated to change the package's `packageDependencies` entry to reference its local path with the following syntax: `"tree-view": "file:./packages/tree-view"` 223 1. A test build will be created locally to manually verify that the package loads and works correctly at first glance 224 1. The package specs for the newly-consolidated package will be run against the local Atom build 225 1. A PR will be sent to `atom/atom` to verify that CI passes with the introduction of the consolidated package 226 1. Once CI is clean and the PR is approved, the PR will be merged 227 1. The package's original repository will have all of its existing issues moved over to `atom/atom` using a bulk issue mover tool, assigning a label to those issues relative to the package name, like `packages/tree-view` 228 1. The package's original repository will have its README.md updated to point contributors to the code's new home in `atom/atom` 229 1. The package's original repository will now be archived on GitHub 230 231 ### Alternative Approaches 232 233 One alternative approach would be to break this core Atom functionality out of packages and put it directly in the Atom codebase without treating them as packages. This would simplify the development process even further but with the following drawbacks: 234 235 - The Atom team would have less regular exposure to Atom package development 236 - Users would no longer be able to disable core packages to replace their behavior with other packages (different tree views, etc) 237 238 ## Unresolved questions 239 240 - Is there a good reason to not move the `language-*` packages into `atom/atom`? 241 242 One concern here is that there exist projects which depend directly on these repositories for the TextMate syntax grammars they contain. Moving the code into `atom/atom` would require that we notify the consumers of the grammars so that they can redirect their requests to the `atom/atom` repo. 243 244 - Should we use `git subtree` to migrate the entire commit history of these packages over or just depend on the history from a package's original repository? 245 246 For now, we won't use `git subtree` due to the possibility that bringing over thousands of commits could cause unknown problems in the Atom repo. We may try this for newly consolidated packages in the future if we decide that not having the package commit history is a sufficient impediment to problem investigations. 247 248 - What are the criteria we might use to eventually decide to move larger packages like `tree-view`, `settings-view`, and `find-and-replace` back into `atom/atom`? 249 250 - Will we be losing any useful data about these packages if we don't have standalone repositories anymore? 251 252 - Should we use this as an opportunity to remove any unnecessary packages from the core Atom distribution? 253 254 [about]: https://github.com/atom/about 255 [archive-view]: https://github.com/atom/archive-view 256 [atom-dark-syntax]: https://github.com/atom/atom-dark-syntax 257 [atom-dark-ui]: https://github.com/atom/atom-dark-ui 258 [atom-light-syntax]: https://github.com/atom/atom-light-syntax 259 [atom-light-ui]: https://github.com/atom/atom-light-ui 260 [autocomplete-atom-api]: https://github.com/atom/autocomplete-atom-api 261 [autocomplete-css]: https://github.com/atom/autocomplete-css 262 [autocomplete-html]: https://github.com/atom/autocomplete-html 263 [autocomplete-plus]: https://github.com/atom/autocomplete-plus 264 [autocomplete-snippets]: https://github.com/atom/autocomplete-snippets 265 [autoflow]: https://github.com/atom/autoflow 266 [autosave]: https://github.com/atom/autosave 267 [background-tips]: https://github.com/atom/background-tips 268 [base16-tomorrow-dark-theme]: https://github.com/atom/base16-tomorrow-dark-theme 269 [base16-tomorrow-light-theme]: https://github.com/atom/base16-tomorrow-light-theme 270 [bookmarks]: https://github.com/atom/bookmarks 271 [bracket-matcher]: https://github.com/atom/bracket-matcher 272 [command-palette]: https://github.com/atom/command-palette 273 [dalek]: https://github.com/atom/dalek 274 [deprecation-cop]: https://github.com/atom/deprecation-cop 275 [dev-live-reload]: https://github.com/atom/dev-live-reload 276 [encoding-selector]: https://github.com/atom/encoding-selector 277 [exception-reporting]: https://github.com/atom/exception-reporting 278 [find-and-replace]: https://github.com/atom/find-and-replace 279 [fuzzy-finder]: https://github.com/atom/fuzzy-finder 280 [git-diff]: https://github.com/atom/git-diff 281 [github]: https://github.com/atom/github 282 [go-to-line]: https://github.com/atom/go-to-line 283 [grammar-selector]: https://github.com/atom/grammar-selector 284 [image-view]: https://github.com/atom/image-view 285 [incompatible-packages]: https://github.com/atom/incompatible-packages 286 [keybinding-resolver]: https://github.com/atom/keybinding-resolver 287 [language-c]: https://github.com/atom/language-c 288 [language-clojure]: https://github.com/atom/language-clojure 289 [language-coffee-script]: https://github.com/atom/language-coffee-script 290 [language-csharp]: https://github.com/atom/language-csharp 291 [language-css]: https://github.com/atom/language-css 292 [language-gfm]: https://github.com/atom/language-gfm 293 [language-git]: https://github.com/atom/language-git 294 [language-go]: https://github.com/atom/language-go 295 [language-html]: https://github.com/atom/language-html 296 [language-hyperlink]: https://github.com/atom/language-hyperlink 297 [language-java]: https://github.com/atom/language-java 298 [language-javascript]: https://github.com/atom/language-javascript 299 [language-json]: https://github.com/atom/language-json 300 [language-less]: https://github.com/atom/language-less 301 [language-make]: https://github.com/atom/language-make 302 [language-mustache]: https://github.com/atom/language-mustache 303 [language-objective-c]: https://github.com/atom/language-objective-c 304 [language-perl]: https://github.com/atom/language-perl 305 [language-php]: https://github.com/atom/language-php 306 [language-property-list]: https://github.com/atom/language-property-list 307 [language-python]: https://github.com/atom/language-python 308 [language-ruby]: https://github.com/atom/language-ruby 309 [language-ruby-on-rails]: https://github.com/atom/language-ruby-on-rails 310 [language-sass]: https://github.com/atom/language-sass 311 [language-shellscript]: https://github.com/atom/language-shellscript 312 [language-source]: https://github.com/atom/language-source 313 [language-sql]: https://github.com/atom/language-sql 314 [language-text]: https://github.com/atom/language-text 315 [language-todo]: https://github.com/atom/language-todo 316 [language-toml]: https://github.com/atom/language-toml 317 [language-typescript]: https://github.com/atom/language-typescript 318 [language-xml]: https://github.com/atom/language-xml 319 [language-yaml]: https://github.com/atom/language-yaml 320 [line-ending-selector]: https://github.com/atom/line-ending-selector 321 [link]: https://github.com/atom/link 322 [markdown-preview]: https://github.com/atom/markdown-preview 323 [metrics]: https://github.com/atom/metrics 324 [notifications]: https://github.com/atom/notifications 325 [one-dark-syntax]: https://github.com/atom/one-dark-syntax 326 [one-dark-ui]: https://github.com/atom/one-dark-ui 327 [one-light-syntax]: https://github.com/atom/one-light-syntax 328 [one-light-ui]: https://github.com/atom/one-light-ui 329 [open-on-github]: https://github.com/atom/open-on-github 330 [package-generator]: https://github.com/atom/package-generator 331 [settings-view]: https://github.com/atom/settings-view 332 [snippets]: https://github.com/atom/snippets 333 [solarized-dark-syntax]: https://github.com/atom/solarized-dark-syntax 334 [solarized-light-syntax]: https://github.com/atom/solarized-light-syntax 335 [spell-check]: https://github.com/atom/spell-check 336 [status-bar]: https://github.com/atom/status-bar 337 [styleguide]: https://github.com/atom/styleguide 338 [symbols-view]: https://github.com/atom/symbols-view 339 [tabs]: https://github.com/atom/tabs 340 [timecop]: https://github.com/atom/timecop 341 [tree-view]: https://github.com/atom/tree-view 342 [update-package-dependencies]: https://github.com/atom/update-package-dependencies 343 [welcome]: https://github.com/atom/welcome 344 [whitespace]: https://github.com/atom/whitespace 345 [wrap-guide]: https://github.com/atom/wrap-guide