1  # Common parts of the build system used by Nimbus and related projects
  2  
  3  We focus on building Nim software on multiple platforms, without having to deal
  4  with language-specific package managers.
  5  
  6  We care about dependencies specified with commit-level accuracy (including the
  7  Nim compiler), reproducible builds, bisectable Git histories and self-contained
  8  projects that don't create dirs/files outside their main directory.
  9  
 10  We try to minimise complexity, but we will trade implementation complexity
 11  increases for a simpler user experience.
 12  
 13  ## Prerequisites
 14  
 15  GNU Make, Bash and the usual POSIX utilities.
 16  
 17  ## Usage
 18  
 19  Add this repository to your project as a Git submodule. You can use our handy shell script:
 20  
 21  ```bash
 22  curl -OLs https://raw.githubusercontent.com/status-im/nimbus-build-system/master/scripts/add_submodule.sh
 23  less add_submodule.sh # you do read random Internet scripts before running them, right?
 24  chmod 755 add_submodule.sh
 25  ./add_submodule.sh status-im/nimbus-build-system
 26  ```
 27  
 28  Or you can do it by hand:
 29  
 30  ```bash
 31  git submodule add https://github.com/status-im/nimbus-build-system.git vendor/nimbus-build-system
 32  # specify a branch
 33  git config -f .gitmodules submodule.vendor/nimbus-build-system.branch master
 34  # hide submodule working tree changes in `git diff`
 35  git config -f .gitmodules submodule.vendor/nimbus-build-system.ignore dirty
 36  ```
 37  
 38  Write your own top-level Makefile, taking our
 39  "[Makefile.superproject.example](./Makefile.superproject.example)" as an...
 40  example.
 41  
 42  See also the Makefiles we wrote for
 43  [Nimbus-eth1](https://github.com/status-im/nimbus-eth1),
 44  [Nimbus-eth2](https://github.com/status-im/nimbus-eth2),
 45  [nim-waku](https://github.com/status-im/nim-waku),
 46  [Stratus](https://github.com/status-im/nim-stratus),
 47  [nim-status-client](https://github.com/status-im/nim-status-client).
 48  
 49  Instruct your users to run `make update` after cloning your project, after a
 50  `git pull` or after switching branches.
 51  
 52  ## Make variables
 53  
 54  ### V
 55  
 56  Control the verbosity level. Defaults to 0 for a nice, quiet build.
 57  
 58  ```bash
 59  make V=1 # verbose
 60  make V=2 test # even more verbose
 61  ```
 62  
 63  ### LOG_LEVEL
 64  
 65  Set the [Chronicles log
 66  level](https://github.com/status-im/nim-chronicles#chronicles_log_level) to one
 67  of: TRACE, DEBUG, INFO, NOTICE, WARN, ERROR, FATAL, NONE.
 68  
 69  This Make variable is unset by default, which means that Chronicles' default
 70  kicks in (DEBUG in debug builds and INFO in release mode) or some
 71  application-specific default takes precedence.
 72  
 73  Note that this sets the compile-time log level. If runtime log level selection
 74  is implemented (which cannot have larger values than what was set at compile
 75  time), additional steps need to be taken to pass the proper command line
 76  argument to your binary.
 77  
 78  ```bash
 79  make LOG_LEVEL=DEBUG foo # this is the default
 80  make LOG_LEVEL=TRACE bar # log everything
 81  ```
 82  
 83  ### NIMFLAGS
 84  
 85  Pass arbitrary parameters to the Nim compiler. Uses an internal `NIM_PARAMS`
 86  variable that should not be overridden by the user.
 87  
 88  ```bash
 89  make NIMFLAGS="-d:release"
 90  ```
 91  
 92  Defaults to Nim parameters mirroring the selected verbosity and log level:
 93  
 94  ```bash
 95  make V=0 # NIMFLAGS="--verbosity:0 --hints:off"
 96  make V=1 LOG_LEVEL=TRACE # NIMFLAGS="--verbosity:1 -d:chronicles_log_level=TRACE"
 97  make V=2 # NIMFLAGS="--verbosity:2"
 98  ```
 99  
100  Projects using this build system may choose to add other default flags like
101  `-d:release` in their Makefiles (usually those that can't be placed in a
102  top-level "config.nims" or "nim.cfg"). This will be done by appending to the
103  internal variable:
104  
105  ```make
106  NIM_PARAMS += -d:release
107  ```
108  
109  ### CI_CACHE
110  
111  Specify a directory where Nim compiler binaries should be cached, in a CI service like AppVeyor:
112  
113  ```yml
114  build_script:
115    # the 32-bit build is done on a 64-bit image, so we need to override the architecture
116    - mingw32-make -j2 ARCH_OVERRIDE=%PLATFORM% CI_CACHE=NimBinaries update
117  ```
118  
119  ### USE_SYSTEM_NIM
120  
121  Use the system Nim instead of our shipped version (usually for testing Nim
122  devel versions). Defaults to 0. Setting it to 1 means you're on your own, when
123  it comes to support.
124  
125  `make USE_SYSTEM_NIM=1 test`
126  
127  ### PARTIAL_STATIC_LINKING
128  
129  Statically link some libraries (currently libgcc\_s and libpcre). Defaults to 0.
130  
131  This is useful when you can't statically link Glibc because you use NSS functions.
132  
133  `make PARTIAL_STATIC_LINKING=1 beacon_node`
134  
135  ### LINK_PCRE
136  
137  Link PCRE, defaults to 1.
138  
139  `make LINK_PCRE=0`
140  
141  ### QUICK_AND_DIRTY_COMPILER
142  
143  Skip some Nim compiler bootstrap iterations and tool building. Useful in
144  CI. Defaults to 0.
145  
146  `make QUICK_AND_DIRTY_COMPILER=1 build-nim`
147  
148  ### QUICK_AND_DIRTY_NIMBLE
149  
150  Build Nimble when it wouldn't be normally built as part of the tools. Only
151  useful with `QUICK_AND_DIRTY_COMPILER=1`. Defaults to 0.
152  
153  `make QUICK_AND_DIRTY_COMPILER=1 QUICK_AND_DIRTY_NIMBLE=1 build-nim`
154  
155  ### NIM_COMMIT
156  
157  Build and use a different Nim compiler version than the default one.
158  
159  Possible values: (partial) commit hashes, tags, branches and anything else recognised by `git checkout ...`.
160  
161  `make -j8 NIM_COMMIT="v1.2.6" build-nim`
162  
163  You also need to specify it when using this non-default Nim compiler version:
164  
165  `make -j8 NIM_COMMIT="v1.2.6" nimbus_beacon_node`
166  
167  ### EXCLUDED_NIM_PACKAGES
168  
169  List of relative paths (incomplete ones also work) to Git submodules that
170  should not end up as Nim packages in "vendor/.nimble" - usually because they
171  duplicate more high-level ones.
172  
173  For example, say we have:
174  
175  ```text
176  $ find vendor -name "nim-chronos" | sort
177  vendor/nim-chronos
178  vendor/nim-waku/vendor/nim-chronos
179  vendor/nim-waku/vendor/nim-dnsdisc/vendor/nim-chronos
180  ```
181  
182  We only want the top-level "vendor/nim-chronos", so we put the rest in
183  `EXCLUDED_NIM_PACKAGES`, in the top-level Makefile:
184  
185  ```text
186  EXCLUDED_NIM_PACKAGES := vendor/nim-waku/vendor/nim-chronos \
187  			 vendor/nim-waku/vendor/nimbus-build-system \
188  			 vendor/nim-waku/vendor/nim-dnsdisc/vendor
189  ```
190  
191  As you see, we can exclude all those "nim-dnsdisc" submodules with a single
192  line, because the pattern is not anchored during the match.
193  
194  ### OVERRIDE
195  
196  Whether to override any uncommitted changes to Git submodules during `make
197  update`. Defaults to 1, in order to keep the old behaviour for users building
198  from source.
199  
200  Set to 0 inside `make update-dev`, to help developers avoid losing work.
201  
202  ## Make targets
203  
204  ### build
205  
206  Internal target that creates the directory with the same name.
207  
208  ### sanity-checks
209  
210  Internal target used to check that a C compiler is installed.
211  
212  ### warn-update
213  
214  Internal target that checks if `make update` was executed for the current Git commit.
215  
216  ### warn-jobs
217  
218  Internal target that checks if Make's parallelism was enabled by specifying the number of jobs.
219  
220  ### deps-common
221  
222  Internal target that needs to be a dependency for a custom "deps" target which,
223  in turn, will be a dependency for various compilation targets.
224  
225  The superproject's Makefile would look like this:
226  
227  ```make
228  deps: | deps-common
229  	# Have custom deps? Add them above.
230  
231  # building Nim programs
232  foo bar: | build deps
233  	echo -e $(BUILD_MSG) "build/$@" && \
234  		$(ENV_SCRIPT) nim c -o:build/$@ $(NIM_PARAMS) "$@.nim"
235  ```
236  
237  The user should never have to run `make deps` directly.
238  
239  ### build-nim
240  
241  Internal target that builds the Nim compiler if it's not built yet or if the
242  corresponding submodule points to a newer commit than the existing binary.
243  
244  It's being executed as part of "update-common" and "$(NIM\_BINARY)" targets. It
245  may be executed directly, rarely, when testing new Nim versions.
246  
247  ### update-common
248  
249  Internal target that needs to become the dependency of a custom "update" target.
250  
251  ```make
252  update: | update-common
253  	# Do you need to do something extra for this target?
254  ```
255  
256  Initialises and updates all Git submodules, with various ugly hacks to account
257  for corner cases like submodules changing URLs or being replaced with regularly
258  committed files.
259  
260  Tell your users to run `make update` after cloning the superproject, after a
261  `git pull` and after changing branches or checking out older commits.
262  
263  ## update-dev
264  
265  Alternative to "update" for developers who want to avoid losing uncommitted
266  work in Git submodules. `make update-dev` simply runs `make OVERRIDE=0 update`.
267  
268  ### update-remote
269  
270  Dangerous target that updates all submodules to their latest remote commit.
271  
272  As you may imagine, it's usually necessary to roll back one or two of these
273  automatic bumps. You do it like this:
274  
275  ```bash
276  git submodule update --recursive vendor/news
277  ```
278  
279  ### clean-cross
280  
281  Confusingly named target that deletes libnatpmp and miniupnp objects and static
282  libraries. Useful when you're cross-compiling and don't want to `make clean`
283  and have to rebuild the compiler.
284  
285  ### clean-common
286  
287  Internal target that needs to be a dependency for a custom "clean" target that
288  deletes any additional build artefacts:
289  
290  ```make
291  clean: | clean-common
292  	rm -rf build/{foo,bar}
293  ```
294  
295  Don't run `make clean` if you don't really need to, since it also deletes the Nim compiler.
296  
297  Unlike C/C++ projects, we always recompile our Nim targets (because it's too
298  hard to tell Make what are all the files involved in the build process), so
299  there's no need to delete them to force a recompilation.
300  
301  ### mrproper
302  
303  Dangerous target that, in addition to depending on "clean", deletes the
304  "vendor" directory and any not-yet-pushed modification you may have in there.
305  Don't use it.
306  
307  ### github-ssh
308  
309  Changes submodule URLs, without affecting .gitmodules, so you connect to GitHub
310  using your SSH key - very useful when you have write access to some submodule
311  repos and you want to work on them without cloning them separately.
312  
313  ### status
314  
315  Run `git status` in all your submodules and in your superproject.
316  
317  ### ctags
318  
319  Run [Universal Ctags](https://github.com/universal-ctags/ctags) with a bunch of Nim-specific options.
320  
321  ### show-deps
322  
323  List all Git submodules, including the nested ones.
324  
325  ### fetch-dlls
326  
327  Windows-specific target. Downloads and unpacks in the "build" dir some DLLs we may not want to build ourselves (PCRE, RocksDB, libcurl, pdcurses, SQLite3, OpenSSL, zlib, etc.).
328  
329  ### nat-libs
330  
331  Build "libminiupnpc.a" and "libnatpmp.a" from [nim-nat-traversal](https://github.com/status-im/nim-nat-traversal)'s submodules - not included in this repo.
332  
333  ## Scripts
334  
335  ### add_submodule.sh
336  
337  Add a new Git submodule to your superproject, setting the branch to "master"
338  and hiding submodule working tree changes in `git diff`.
339  
340  Usage: `./add_submodule.sh some/repo [destdir]` # "destdir" defaults to "vendor/repo"
341  
342  Examples:
343  
344  `./add_submodule.sh status-im/nimbus-build-system`
345  
346  `./vendor/nimbus-build-system/scripts/add_submodule.sh status-im/nim-nat-traversal`
347  
348  ### build_nim.sh
349  
350  Build the Nim compiler and some associated tools, parallelising the
351  bootstrap-from-C phase, with binary caching and conditional recompilation
352  support (based on the timestamp of the latest commit in the Nim repo).
353  
354  Usage: `./build_nim.sh nim_dir csources_dir nimble_dir ci_cache_dir`
355  
356  This script is not usually used directly, but through the `update` target
357  (which depends on `update-common` which runs `build-nim`).
358  
359  Our `build-nim` target uses it like this:
360  
361  ```make
362  build-nim: | sanity-checks
363  	+ NIM_BUILD_MSG="$(BUILD_MSG) Nim compiler" \
364  		V=$(V) \
365  		CC=$(CC) \
366  		MAKE=$(MAKE) \
367  		ARCH_OVERRIDE=$(ARCH_OVERRIDE) \
368  		"$(CURDIR)/$(BUILD_SYSTEM_DIR)/scripts/build_nim.sh" "$(NIM_DIR)" ../Nim-csources ../nimble "$(CI_CACHE)"
369  ```
370  
371  Other Nim projects that don't use this build system use the script directly in their CI. From a ".travis.yml":
372  
373  ```yaml
374  install:
375    - curl -O -L -s -S https://raw.githubusercontent.com/status-im/nimbus-build-system/master/scripts/build_nim.sh
376    - env MAKE="make -j${NPROC}" bash build_nim.sh Nim csources dist/nimble NimBinaries
377    - export PATH="$PWD/Nim/bin:$PATH"
378  ```
379  
380  Or an ".appveyor.yml":
381  
382  ```yaml
383  install:
384    # [...]
385    - curl -O -L -s -S https://raw.githubusercontent.com/status-im/nimbus-build-system/master/scripts/build_nim.sh
386    - env MAKE="mingw32-make -j2" ARCH_OVERRIDE=%PLATFORM% bash build_nim.sh Nim csources dist/nimble NimBinaries
387    - SET PATH=%CD%\Nim\bin;%PATH%
388  ```
389  
390  Notice how the number of Make jobs is set through the "MAKE" env var.
391  
392  ### build_rocksdb.sh
393  
394  Builds RocksDB. No longer used.
395  
396  Usage: `./build_rocksdb.sh ci_cache_dir`
397  
398  ### create_nimble_link.sh
399  
400  Cheeky little script used to fake a Nimble package repository in the
401  `$(NIMBLE_DIR)` target (a dependency of `deps-common` which is a dependency of
402  `deps`):
403  
404  ```make
405  $(NIMBLE_DIR):
406  	mkdir -p $(NIMBLE_DIR)/pkgs
407  	NIMBLE_DIR="$(CURDIR)/$(NIMBLE_DIR)" PWD_CMD="$(PWD)" \
408  		git submodule foreach --quiet '$(CURDIR)/$(BUILD_SYSTEM_DIR)/scripts/create_nimble_link.sh "$$sm_path"'
409  ```
410  
411  That's how the Nim compiler knows how to find all these Nim packages we have in
412  our submodules: we set the "NIMBLE\_DIR" env var to the path of this fake
413  Nimble package repo.
414  
415  ### env.sh
416  
417  Script responsible for setting up environment variables and shell aliases
418  necessary for using this build system. It's being sourced by a script with the
419  same name in the superproject's top directory:
420  
421  ```bash
422  #!/usr/bin/env bash
423  
424  # We use ${BASH_SOURCE[0]} instead of $0 to allow sourcing this file
425  # and we fall back to a Zsh-specific special var to also support Zsh.
426  REL_PATH="$(dirname ${BASH_SOURCE[0]:-${(%):-%x}})"
427  ABS_PATH="$(cd ${REL_PATH}; pwd)"
428  source ${ABS_PATH}/vendor/nimbus-build-system/scripts/env.sh
429  ```
430  
431  Supported usage: `./env.sh nim --version`
432  
433  Unsupported usage: `source env.sh; nim --version`
434  
435  An interesting alias is `nimble` which calls the "nimble.sh" script which pretends to be Nimble:
436  
437  ```bash
438  cd vendor/nim-metrics
439  ../../env.sh nimble test
440  ```
441  
442  In order to get autocompletion and code navigation functionality working
443  properly in your text editor, you need the environment variables set by this script:
444  
445  ```bash
446  ./env.sh vim
447  ```
448  
449  ### nimble.sh
450  
451  Simple script that symlinks the first \*.nimble file it finds to \*.nims and
452  runs it using `nim`. Easier to access using the `nimble` alias in "env.sh".
453  
454  Of very limited use, it can execute \*.nimble targets, as long as there are no
455  "before install:" blocks that the real Nimble strips before doing the same thing we do.
456  
457  If you need the real Nimble, it's in "vendor/nimbus-build-system/vendor/Nim/bin/nimble".
458  
459  ## License
460  
461  Licensed and distributed under either of
462  
463  * MIT license: [LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT
464  
465  or
466  
467  * Apache License, Version 2.0, ([LICENSE-APACHEv2](LICENSE-APACHEv2) or http://www.apache.org/licenses/LICENSE-2.0)
468  
469  at your option. These files may not be copied, modified, or distributed except according to those terms.
470