nix-pills-luca-bruno.org
1 #+title: Nix Pills 2 3 #+author: Luca Bruno 4 <<00-preface.html>> 5 6 * Preface 7 This is a ported version of the *Nix Pills*, a series of blog posts 8 written by *Luca Bruno* (aka Lethalman) and originally published in 2014 9 and 2015. It provides a tutorial introduction into the Nix package 10 manager and Nixpkgs package collection, in the form of short chapters 11 called 'pills'. 12 13 Since the Nix Pills are considered a classic introduction to Nix, an 14 effort to port them to the current format was led by Graham Christensen 15 (aka grahamc / gchristensen) and other contributors in 2017. 16 17 For an up-to-date version, please visit 18 [[https://nixos.org/guides/nix-pills/]]. An 19 [[https://nixos.org/guides/nix-pills/nix-pills.epub][EPUB version]] is 20 also available. 21 22 If you encounter problems, please report them on the 23 [[https://github.com/NixOS/nix-pills/issues][nixos/nix-pills]] issue 24 tracker. 25 26 <<01-why-you-should-give-it-a-try.html>> 27 28 * Why You Should Give it a Try 29 ** Introduction 30 Welcome to the first post of the "[[https://nixos.org/nix][Nix]] in 31 pills" series. Nix is a purely functional package manager and deployment 32 system for POSIX. 33 34 There's a lot of documentation that describes what Nix, 35 [[https://nixos.org/nixos][NixOS]] and related projects are. But the 36 purpose of this post is to convince you to give Nix a try. Installing 37 NixOS is not required, but sometimes I may refer to NixOS as a real 38 world example of Nix usage for building a whole operating system. 39 40 ** Rationale for this series 41 The [[https://nixos.org/manual/nix][Nix]], 42 [[https://nixos.org/manual/nixpkgs/][Nixpkgs]], and 43 [[https://nixos.org/manual/nixos/][NixOS]] manuals along with 44 [[https://wiki.nixos.org/][the wiki]] are excellent resources for 45 explaining how Nix/NixOS works, how you can use it, and how cool things 46 are being done with it. However, at the beginning you may feel that some 47 of the magic which happens behind the scenes is hard to grasp. 48 49 This series aims to complement the existing explanations from the more 50 formal documents. 51 52 The following is a description of Nix. Just as with pills, I'll try to 53 be as short as possible. 54 55 ** Not being purely functional 56 Most, if not all, widely used package managers 57 ([[https://wiki.debian.org/dpkg][dpkg]], [[http://www.rpm.org/][rpm]], 58 ...) mutate the global state of the system. If a package =foo-1.0= 59 installs a program to =/usr/bin/foo=, you cannot install =foo-1.1= as 60 well, unless you change the installation paths or the binary name. But 61 changing the binary names means breaking users of that binary. 62 63 There are some attempts to mitigate this problem. Debian, for example, 64 partially solves the problem with the 65 [[https://wiki.debian.org/DebianAlternatives][alternatives]] system. 66 67 So while in theory it's possible with some current systems to install 68 multiple versions of the same package, in practice it's very painful. 69 70 Let's say you need an nginx service and also an nginx-openresty service. 71 You have to create a new package that changes all the paths to have, for 72 example, an =-openresty= suffix. 73 74 Or suppose that you want to run two different instances of mysql: 5.2 75 and 5.5. The same thing applies, plus you have to also make sure the two 76 mysqlclient libraries do not collide. 77 78 This is not impossible but it /is/ very inconvenient. If you want to 79 install two whole stacks of software like GNOME 3.10 and GNOME 3.12, you 80 can imagine the amount of work. 81 82 From an administrator's point of view: you can use containers. The 83 typical solution nowadays is to create a container per service, 84 especially when different versions are needed. That somewhat solves the 85 problem, but at a different level and with other drawbacks. For example, 86 needing orchestration tools, setting up a shared cache of packages, and 87 new machines to monitor rather than simple services. 88 89 From a developer's point of view: you can use virtualenv for python, or 90 jhbuild for gnome, or whatever else. But then how do you mix the two 91 stacks? How do you avoid recompiling the same thing when it could 92 instead be shared? Also you need to set up your development tools to 93 point to the different directories where libraries are installed. Not 94 only that, there's the risk that some of the software incorrectly uses 95 system libraries. 96 97 And so on. Nix solves all this at the packaging level and solves it 98 well. A single tool to rule them all. 99 100 ** Being purely functional 101 Nix makes no assumptions about the global state of the system. This has 102 many advantages, but also some drawbacks of course. The core of a Nix 103 system is the Nix store, usually installed under =/nix/store=, and some 104 tools to manipulate the store. In Nix there is the notion of a 105 /derivation/ rather than a package. The difference can be subtle at the 106 beginning, so I will often use the words interchangeably. 107 108 Derivations/packages are stored in the Nix store as follows: 109 =/nix/store/«hash-name»=, where the hash uniquely identifies the 110 derivation (this isn't quite true, it's a little more complex), and the 111 name is the name of the derivation. 112 113 Let's take a bash derivation as an example: 114 =/nix/store/s4zia7hhqkin1di0f187b79sa2srhv6k-bash-4.2-p45/=. This is a 115 directory in the Nix store which contains =bin/bash=. 116 117 What that means is that there's no =/bin/bash=, there's only that 118 self-contained build output in the store. The same goes for coreutils 119 and everything else. To make them convenient to use from the shell, Nix 120 will arrange for binaries to appear in your =PATH= as appropriate. 121 122 What we have is basically a store of all packages (with different 123 versions occupying different locations), and everything in the Nix store 124 is immutable. 125 126 In fact, there's no ldconfig cache either. So where does bash find libc? 127 128 #+begin_example 129 $ ldd `which bash` 130 libc.so.6 => /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/libc.so.6 (0x00007f0248cce000) 131 #+end_example 132 133 It turns out that when bash was built, it was built against that 134 specific version of glibc in the Nix store, and at runtime it will 135 require exactly that glibc version. 136 137 Don't be confused by the version in the derivation name: it's only a 138 name for us humans. You may end up having two derivations with the same 139 name but different hashes: it's the hash that really matters. 140 141 What does all this mean? It means that you could run mysql 5.2 with 142 glibc-2.18, and mysql 5.5 with glibc-2.19. You could use your python 143 module with python 2.7 compiled with gcc 4.6 and the same python module 144 with python 3 compiled with gcc 4.8, all in the same system. 145 146 In other words: no dependency hell, not even a dependency resolution 147 algorithm. Straight dependencies from derivations to other derivations. 148 149 From an administrator's point of view: if you want an old PHP version 150 for one application, but want to upgrade the rest of the system, that's 151 not painful any more. 152 153 From a developer's point of view: if you want to develop webkit with 154 llvm 3.4 and 3.3, that's not painful any more. 155 156 ** Mutable vs. immutable 157 When upgrading a library, most package managers replace it in-place. All 158 new applications run afterwards with the new library without being 159 recompiled. After all, they all refer dynamically to =libc6.so=. 160 161 Since Nix derivations are immutable, upgrading a library like glibc 162 means recompiling all applications, because the glibc path to the Nix 163 store has been hardcoded. 164 165 So how do we deal with security updates? In Nix we have some tricks 166 (still pure) to solve this problem, but that's another story. 167 168 Another problem is that unless software has in mind a pure functional 169 model, or can be adapted to it, it can be hard to compose applications 170 at runtime. 171 172 Let's take Firefox for example. On most systems, you install flash, and 173 it starts working in Firefox because Firefox looks in a global path for 174 plugins. 175 176 In Nix, there's no such global path for plugins. Firefox therefore must 177 know explicitly about the path to flash. The way we handle this problem 178 is to wrap the Firefox binary so that we can setup the necessary 179 environment to make it find flash in the nix store. That will produce a 180 new Firefox derivation: be aware that it takes a few seconds, and it 181 makes composition harder at runtime. 182 183 There are no upgrade/downgrade scripts for your data. It doesn't make 184 sense with this approach, because there's no real derivation to be 185 upgraded. With Nix you switch to using other software with its own stack 186 of dependencies, but there's no formal notion of upgrade or downgrade 187 when doing so. 188 189 If there is a data format change, then migrating to the new data format 190 remains your own responsibility. 191 192 ** Conclusion 193 Nix lets you compose software at build time with maximum flexibility, 194 and with builds being as reproducible as possible. Not only that, due to 195 its nature deploying systems in the cloud is so easy, consistent, and 196 reliable that in the Nix world all existing self-containment and 197 orchestration tools are deprecated by 198 [[http://nixos.org/nixops/][NixOps]]. 199 200 It however /currently/ falls short when working with dynamic composition 201 at runtime or replacing low level libraries, due to the need to rebuild 202 dependencies. 203 204 That may sound scary, however after running NixOS on both a server and a 205 laptop desktop, I'm very satisfied so far. Some of the architectural 206 problems just need some man-power, other design problems still need to 207 be solved as a community. 208 209 Considering [[https://nixos.org/nixpkgs/][Nixpkgs]] 210 ([[https://github.com/NixOS/nixpkgs][github link]]) is a completely new 211 repository of all the existing software, with a completely fresh 212 concept, and with few core developers but overall year-over-year 213 increasing contributions, the current state is more than acceptable and 214 beyond the experimental stage. In other words, it's worth your 215 investment. 216 217 ** Next pill... 218 ...we will install Nix on top of your current system (I assume 219 GNU/Linux, but we also have OSX users) and start inspecting the 220 installed software. 221 222 <<02-install-on-your-running-system.html>> 223 224 * Install on Your Running System 225 Welcome to the second Nix pill. In the 226 [[#01-why-you-should-give-it-a-try.html][first]] pill we briefly 227 described Nix. 228 229 Now we'll install Nix on our running system and understand what changed 230 in our system after the installation. *If you're using NixOS, Nix is 231 already installed; you can skip to the 232 [[#03-enter-environment.html][next]] pill.* 233 234 For installation instructions, please refer to the Nix Reference Manual 235 on 236 [[https://nix.dev/manual/nix/stable/installation/installing-binary][Installing 237 Nix]]. 238 239 ** Installation 240 These articles are not a tutorial on /using/ Nix. Instead, we're going 241 to walk through the Nix system to understand the fundamentals. 242 243 The first thing to note: derivations in the Nix store refer to other 244 derivations which are themselves in the Nix store. They don't use =libc= 245 from our system or anywhere else. It's a self-contained store of all the 246 software we need to bootstrap up to any particular package. 247 248 Note: In a multi-user installation, such as the one used in NixOS, the 249 store is owned by root and multiple users can install and build software 250 through a Nix daemon. You can read more about 251 [[https://nix.dev/manual/nix/stable/installation/installing-binary#multi-user-installation][multi-user 252 installations here]]. 253 254 ** The beginnings of the Nix store 255 Start looking at the output of the install command: 256 257 #+begin_example 258 copying Nix to /nix/store.......................... 259 #+end_example 260 261 That's the =/nix/store= we were talking about in the first article. 262 We're copying in the necessary software to bootstrap a Nix system. You 263 can see bash, coreutils, the C compiler toolchain, perl libraries, 264 sqlite and Nix itself with its own tools and libnix. 265 266 You may have noticed that =/nix/store= can contain not only directories, 267 but also files, still always in the form «hash-name». 268 269 ** The Nix database 270 Right after copying the store, the installation process initializes a 271 database: 272 273 #+begin_example 274 initialising Nix database... 275 #+end_example 276 277 Yes, Nix also has a database. It's stored under =/nix/var/nix/db=. It is 278 a sqlite database that keeps track of the dependencies between 279 derivations. 280 281 The schema is very simple: there's a table of valid paths, mapping from 282 an auto increment integer to a store path. 283 284 Then there's a dependency relation from path to paths upon which they 285 depend. 286 287 You can inspect the database by installing sqlite 288 (=nix-env -iA sqlite -f '<nixpkgs>'=) and then running 289 =sqlite3 /nix/var/nix/db/db.sqlite=. 290 291 Note: If this is the first time you're using Nix after the initial 292 installation, remember you must close and open your terminals first, so 293 that your shell environment will be updated. 294 295 Important: Never change =/nix/store= manually. If you do, then it will 296 no longer be in sync with the sqlite db, unless you /really/ know what 297 you are doing. 298 299 ** The first profile 300 Next in the installation, we encounter the concept of the 301 [[https://nix.dev/manual/nix/stable/package-management/profiles][profile]]: 302 303 #+begin_example 304 creating /home/nix/.nix-profile 305 installing 'nix-2.1.3' 306 building path(s) `/nix/store/a7p1w3z2h8pl00ywvw6icr3g5l9vm5r7-user-environment' 307 created 7 symlinks in user environment 308 #+end_example 309 310 A profile in Nix is a general and convenient concept for realizing 311 rollbacks. Profiles are used to compose components that are spread among 312 multiple paths under a new unified path. Not only that, but profiles are 313 made up of multiple "generations": they are versioned. Whenever you 314 change a profile, a new generation is created. 315 316 Generations can be switched and rolled back atomically, which makes them 317 convenient for managing changes to your system. 318 319 Let's take a closer look at our profile: 320 321 #+begin_example 322 $ ls -l ~/.nix-profile/ 323 bin -> /nix/store/ig31y9gfpp8pf3szdd7d4sf29zr7igbr-nix-2.1.3/bin 324 [...] 325 manifest.nix -> /nix/store/q8b5238akq07lj9gfb3qb5ycq4dxxiwm-env-manifest.nix 326 [...] 327 share -> /nix/store/ig31y9gfpp8pf3szdd7d4sf29zr7igbr-nix-2.1.3/share 328 #+end_example 329 330 That =nix-2.1.3= derivation in the Nix store is Nix itself, with 331 binaries and libraries. The process of "installing" the derivation in 332 the profile basically reproduces the hierarchy of the =nix-2.1.3= store 333 derivation in the profile by means of symbolic links. 334 335 The contents of this profile are special, because only one program has 336 been installed in our profile, therefore e.g. the =bin= directory points 337 to the only program which has been installed (Nix itself). 338 339 But that's only the contents of the latest generation of our profile. In 340 fact, =~/.nix-profile= itself is a symbolic link to 341 =/nix/var/nix/profiles/default=. 342 343 In turn, that's a symlink to =default-1-link= in the same directory. 344 Yes, that means it's the first generation of the =default= profile. 345 346 Finally, =default-1-link= is a symlink to the nix store 347 "user-environment" derivation that you saw printed during the 348 installation process. 349 350 We'll talk about =manifest.nix= more in the next article. 351 352 ** Nixpkgs expressions 353 More output from the installer: 354 355 #+begin_example 356 downloading Nix expressions from `http://releases.nixos.org/nixpkgs/nixpkgs-14.10pre46060.a1a2851/nixexprs.tar.xz'... 357 unpacking channels... 358 created 2 symlinks in user environment 359 modifying /home/nix/.profile... 360 #+end_example 361 362 Nix expressions are written in the 363 [[https://nix.dev/tutorials/nix-language][Nix language]] and used to 364 describe packages and how to build them. 365 [[https://nixos.org/nixpkgs/][Nixpkgs]] is the repository containing all 366 of the expressions: [[https://github.com/NixOS/nixpkgs]]. 367 368 The installer downloaded the package descriptions from commit =a1a2851=. 369 370 The second profile we discover is the channels profile. 371 =~/.nix-defexpr/channels= points to 372 =/nix/var/nix/profiles/per-user/nix/channels= which points to 373 =channels-1-link= which points to a Nix store directory containing the 374 downloaded Nix expressions. 375 376 Channels are a set of packages and expressions available for download. 377 Similar to Debian stable and unstable, there's a stable and unstable 378 channel. In this installation, we're tracking =nixpkgs-unstable=. 379 380 Don't worry about Nix expressions yet, we'll get to them later. 381 382 Finally, for your convenience, the installer modified =~/.profile= to 383 automatically enter the Nix environment. What 384 =~/.nix-profile/etc/profile.d/nix.sh= really does is simply to add 385 =~/.nix-profile/bin= to =PATH= and =~/.nix-defexpr/channels/nixpkgs= to 386 =NIX_PATH=. We'll discuss =NIX_PATH= later. 387 388 Read =nix.sh=, it's short. 389 390 ** FAQ: Can I change /nix to something else? 391 You can, but there's a good reason to keep using =/nix= instead of a 392 different directory. All the derivations depend on other derivations by 393 using absolute paths. We saw in the first article that bash referenced a 394 =glibc= under a specific absolute path in =/nix/store=. 395 396 You can see for yourself, don't worry if you see multiple bash 397 derivations: 398 399 #+begin_example 400 $ ldd /nix/store/*bash*/bin/bash 401 [...] 402 #+end_example 403 404 Keeping the store in =/nix= means we can grab the binary cache from 405 nixos.org (just like you grab packages from debian mirrors) otherwise: 406 407 - =glibc= would be installed under =/foo/store= 408 409 - Thus bash would need to point to =glibc= under =/foo/store=, instead 410 of under =/nix/store= 411 412 - So the binary cache can't help, because we need a /different/ bash, 413 and so we'd have to recompile everything ourselves. 414 415 After all =/nix= is a sensible place for the store. 416 417 ** Conclusion 418 We've installed Nix on our system, fully isolated and owned by the =nix= 419 user as we're still coming to terms with this new system. 420 421 We learned some new concepts like profiles and channels. In particular, 422 with profiles we're able to manage multiple generations of a composition 423 of packages, while with channels we're able to download binaries from 424 =nixos.org=. 425 426 The installation put everything under =/nix=, and some symlinks in the 427 Nix user home. That's because every user is able to install and use 428 software in her own environment. 429 430 I hope I left nothing uncovered so that you think there's some kind of 431 magic going on behind the scenes. It's all about putting components in 432 the store and symlinking these components together. 433 434 ** Next pill... 435 ...we will enter the Nix environment and learn how to interact with the 436 store. 437 438 <<03-enter-environment.html>> 439 440 * Enter the Environment 441 :PROPERTIES: 442 :CUSTOM_ID: 03-enter-environment.html#enter-environment 443 :END: 444 Welcome to the third Nix pill. In the 445 [[#02-install-on-your-running-system.html][second pill]] we installed 446 Nix on our running system. Now we can finally play with it a little, 447 these things also apply to NixOS users. 448 449 ** Enter the environment 450 *If you're using NixOS, you can skip to the 451 [[#03-enter-environment.html#install-something][next]] step.* 452 453 In the previous article we created a Nix user, so let's start by 454 switching to it with =su - nix=. If your =~/.profile= got evaluated, 455 then you should now be able to run commands like =nix-env= and 456 =nix-store=. 457 458 If that's not the case: 459 460 #+begin_example 461 $ source ~/.nix-profile/etc/profile.d/nix.sh 462 #+end_example 463 464 To remind you, =~/.nix-profile/etc= points to the =nix-2.1.3= 465 derivation. At this point, we are in our Nix user profile. 466 467 ** Install something 468 :PROPERTIES: 469 :CUSTOM_ID: 03-enter-environment.html#install-something 470 :END: 471 Finally something practical! Installation into the Nix environment is an 472 interesting process. Let's install =hello=, a simple CLI tool which 473 prints =Hello world= and is mainly used to test compilers and package 474 installations. 475 476 Back to the installation: 477 478 #+begin_example 479 $ nix-env -i hello 480 installing 'hello-2.10' 481 [...] 482 building '/nix/store/0vqw0ssmh6y5zj48yg34gc6macr883xk-user-environment.drv'... 483 created 36 symlinks in user environment 484 #+end_example 485 486 Now you can run =hello=. Things to notice: 487 488 - We installed software as a user, and only for the Nix user. 489 490 - It created a new user environment. That's a new generation of our Nix 491 user profile. 492 493 - The [[https://nix.dev/manual/nix/stable/command-ref/nix-env][nix-env]] 494 tool manages environments, profiles and their generations. 495 496 - We installed =hello= by derivation name minus the version. I repeat: 497 we specified the *derivation name* (minus the version) to install it. 498 499 We can list generations without walking through the =/nix= hierarchy: 500 501 #+begin_example 502 $ nix-env --list-generations 503 1 2014-07-24 09:23:30 504 2 2014-07-25 08:45:01 (current) 505 #+end_example 506 507 Listing installed derivations: 508 509 #+begin_example 510 $ nix-env -q 511 nix-2.1.3 512 hello-2.10 513 #+end_example 514 515 So, where did =hello= really get installed? =which hello= is 516 =~/.nix-profile/bin/hello= which points to the store. We can also list 517 the derivation paths with =nix-env -q --out-path=. So that's what those 518 derivation paths are called: the *output* of a build. 519 520 ** Path merging 521 At this point you probably want to run =man= to get some documentation. 522 Even if you already have man system-wide outside of the Nix environment, 523 you can install and use it within Nix with =nix-env -i man-db=. As 524 usual, a new generation will be created, and =~/.nix-profile= will point 525 to it. 526 527 Let's inspect the 528 [[https://nix.dev/manual/nix/stable/package-management/profiles][profile]] 529 a bit: 530 531 #+begin_example 532 $ ls -l ~/.nix-profile/ 533 dr-xr-xr-x 2 nix nix 4096 Jan 1 1970 bin 534 lrwxrwxrwx 1 nix nix 55 Jan 1 1970 etc -> /nix/store/ig31y9gfpp8pf3szdd7d4sf29zr7igbr-nix-2.1.3/etc 535 [...] 536 #+end_example 537 538 Now that's interesting. When only =nix-2.1.3= was installed, =bin= was a 539 symlink to =nix-2.1.3=. Now that we've actually installed some things 540 (=man=, =hello=), it's a real directory, not a symlink. 541 542 #+begin_example 543 $ ls -l ~/.nix-profile/bin/ 544 [...] 545 man -> /nix/store/83cn9ing5sc6644h50dqzzfxcs07r2jn-man-1.6g/bin/man 546 [...] 547 nix-env -> /nix/store/ig31y9gfpp8pf3szdd7d4sf29zr7igbr-nix-2.1.3/bin/nix-env 548 [...] 549 hello -> /nix/store/58r35bqb4f3lxbnbabq718svq9i2pda3-hello-2.10/bin/hello 550 [...] 551 #+end_example 552 553 Okay, that's clearer now. =nix-env= merged the paths from the installed 554 derivations. =which man= points to the Nix profile, rather than the 555 system =man=, because =~/.nix-profile/bin= is at the head of =$PATH=. 556 557 ** Rolling back and switching generation 558 The last command installed =man=. We should be at generation 3, unless 559 you changed something in the middle. Let's say we want to rollback to 560 the old generation: 561 562 #+begin_example 563 $ nix-env --rollback 564 switching from generation 3 to 2 565 #+end_example 566 567 Now =nix-env -q= does not list =man= anymore. =ls -l `which man`= should 568 now be your system copy. 569 570 Enough with the rollback, let's go back to the most recent generation: 571 572 #+begin_example 573 $ nix-env -G 3 574 switching from generation 2 to 3 575 #+end_example 576 577 I invite you to read the manpage of =nix-env=. =nix-env= requires an 578 operation to perform, then there are common options for all operations, 579 as well as options specific to each operation. 580 581 You can of course also 582 [[https://nix.dev/manual/nix/stable/command-ref/nix-env/uninstall][uninstall]] 583 and 584 [[https://nix.dev/manual/nix/stable/command-ref/nix-env/upgrade][upgrade]] 585 packages. 586 587 ** Querying the store 588 So far we learned how to query and manipulate the environment. But all 589 of the environment components point to the store. 590 591 To query and manipulate the store, there's the =nix-store= command. We 592 can do some interesting things, but we'll only see some queries for now. 593 594 To show the direct runtime dependencies of =hello=: 595 596 #+begin_example 597 $ nix-store -q --references `which hello` 598 /nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27 599 /nix/store/58r35bqb4f3lxbnbabq718svq9i2pda3-hello-2.10 600 #+end_example 601 602 The argument to =nix-store= can be anything as long as it points to the 603 Nix store. It will follow symlinks. 604 605 It may not make sense to you right now, but let's print reverse 606 dependencies of =hello=: 607 608 #+begin_example 609 $ nix-store -q --referrers `which hello` 610 /nix/store/58r35bqb4f3lxbnbabq718svq9i2pda3-hello-2.10 611 /nix/store/fhvy2550cpmjgcjcx5rzz328i0kfv3z3-env-manifest.nix 612 /nix/store/yzdk0xvr0b8dcwhi2nns6d75k2ha5208-env-manifest.nix 613 /nix/store/mp987abm20c70pl8p31ljw1r5by4xwfw-user-environment 614 /nix/store/ppr3qbq7fk2m2pa49i2z3i32cvfhsv7p-user-environment 615 #+end_example 616 617 Was it what you expected? It turns out that our environments depend upon 618 =hello=. Yes, that means that the environments are in the store, and 619 since they contain symlinks to =hello=, therefore the environment 620 depends upon =hello=. 621 622 Two environments were listed, generation 2 and generation 3, since these 623 are the ones that had =hello= installed in them. 624 625 The =manifest.nix= file contains metadata about the environment, such as 626 which derivations are installed. So that =nix-env= can list, upgrade or 627 remove them. And yet again, the current =manifest.nix= can be found at 628 =~/.nix-profile/manifest.nix=. 629 630 ** Closures 631 The closures of a derivation is a list of all its dependencies, 632 recursively, including absolutely everything necessary to use that 633 derivation. 634 635 #+begin_example 636 $ nix-store -qR `which man` 637 [...] 638 #+end_example 639 640 Copying all those derivations to the Nix store of another machine makes 641 you able to run =man= out of the box on that other machine. That's the 642 base of deployment using Nix, and you can already foresee the potential 643 when deploying software in the cloud (hint: =nix-copy-closures= and 644 =nix-store --export=). 645 646 A nicer view of the closure: 647 648 #+begin_example 649 $ nix-store -q --tree `which man` 650 [...] 651 #+end_example 652 653 With the above command, you can find out exactly why a /runtime/ 654 dependency, be it direct or indirect, exists for a given derivation. 655 656 The same applies to environments. As an exercise, run 657 =nix-store -q --tree ~/.nix-profile=, and see that the first children 658 are direct dependencies of the user environment: the installed 659 derivations, and the =manifest.nix=. 660 661 ** Dependency resolution 662 There isn't anything like =apt= which solves a SAT problem in order to 663 satisfy dependencies with lower and upper bounds on versions. There's no 664 need for this because all the dependencies are static: if a derivation X 665 depends on a derivation Y, then it always depends on it. A version of X 666 which depended on Z would be a different derivation. 667 668 ** Recovering the hard way 669 #+begin_example 670 $ nix-env -e '*' 671 uninstalling 'hello-2.10' 672 uninstalling 'nix-2.1.3' 673 [...] 674 #+end_example 675 676 Oops, that uninstalled all derivations from the environment, including 677 Nix. That means we can't even run =nix-env=, what now? 678 679 Previously we got =nix-env= from the environment. Environments are a 680 convenience for the user, but Nix is still there in the store! 681 682 First, pick one =nix-2.1.3= derivation: =ls /nix/store/*nix-2.1.3=, say 683 =/nix/store/ig31y9gfpp8pf3szdd7d4sf29zr7igbr-nix-2.1.3=. 684 685 The first option is to rollback: 686 687 #+begin_example 688 $ /nix/store/ig31y9gfpp8pf3szdd7d4sf29zr7igbr-nix-2.1.3/bin/nix-env --rollback 689 #+end_example 690 691 The second option is to install Nix, thus creating a new generation: 692 693 #+begin_example 694 $ /nix/store/ig31y9gfpp8pf3szdd7d4sf29zr7igbr-nix-2.1.3/bin/nix-env -i /nix/store/ig31y9gfpp8pf3szdd7d4sf29zr7igbr-nix-2.1.3/bin/nix-env 695 #+end_example 696 697 ** Channels 698 So where are we getting packages from? We said something about this 699 already in the [[#02-install-on-your-running-system.html][second 700 article]]. There's a list of channels from which we get packages, 701 although usually we use a single channel. The tool to manage channels is 702 [[https://nix.dev/manual/nix/stable/command-ref/nix-channel][nix-channel]]. 703 704 #+begin_example 705 $ nix-channel --list 706 nixpkgs http://nixos.org/channels/nixpkgs-unstable 707 #+end_example 708 709 If you're using NixOS, you may not see any output from the above command 710 (if you're using the default), or you may see a channel whose name 711 begins with "nixos-" instead of "nixpkgs". 712 713 That's essentially the contents of =~/.nix-channels=. 714 715 Note: =~/.nix-channels= is not a symlink to the nix store! 716 717 To update the channel run =nix-channel --update=. That will download the 718 new Nix expressions (descriptions of the packages), create a new 719 generation of the channels profile and unpack it under 720 =~/.nix-defexpr/channels=. 721 722 This is quite similar to =apt-get update=. (See 723 [[https://wiki.nixos.org/wiki/Cheatsheet][this table]] for a rough 724 mapping between Ubuntu and NixOS package management.) 725 726 ** Conclusion 727 We learned how to query the user environment and to manipulate it by 728 installing and uninstalling software. Upgrading software is also 729 straightforward, as you can read in 730 [[https://nix.dev/manual/nix/stable/command-ref/nix-env/upgrade][the 731 manual]] (=nix-env -u= will upgrade all packages in the environment). 732 733 Every time we change the environment, a new generation is created. 734 Switching between generations is easy and immediate. 735 736 Then we learned how to query the store. We inspected the dependencies 737 and reverse dependencies of store paths. 738 739 We saw how symlinks are used to compose paths from the Nix store, a 740 useful trick. 741 742 A quick analogy with programming languages: you have the heap with all 743 the objects, that corresponds to the Nix store. You have objects that 744 point to other objects, those correspond to derivations. This is a 745 suggestive metaphor, but will it be the right path? 746 747 ** Next pill 748 ...we will learn the basics of the Nix language. The Nix language is 749 used to describe how to build derivations, and it's the basis for 750 everything else, including NixOS. Therefore it's very important to 751 understand both the syntax and the semantics of the language. 752 753 <<04-basics-of-language.html>> 754 755 * The Basics of the Language 756 :PROPERTIES: 757 :CUSTOM_ID: 04-basics-of-language.html#basics-of-language 758 :END: 759 Welcome to the fourth Nix pill. In the 760 [[#03-enter-environment.html][previous article]] we learned about Nix 761 environments. We installed software as a user, managed their profile, 762 switched between generations, and queried the Nix store. Those are the 763 very basics of system administration using Nix. 764 765 The [[https://nix.dev/manual/nix/stable/language/][Nix language]] is 766 used to write expressions that produce derivations. The 767 [[https://nix.dev/manual/nix/stable/command-ref/nix-build][nix-build]] 768 tool is used to build derivations from an expression. Even as a system 769 administrator that wants to customize the installation, it's necessary 770 to master Nix. Using Nix for your jobs means you get the features we saw 771 in the previous articles for free. 772 773 The syntax of Nix is quite unfamiliar, so looking at existing examples 774 may lead you to think that there's a lot of magic happening. In reality, 775 it's mostly about writing utility functions to make things convenient. 776 777 On the other hand, the same syntax is great for describing packages, so 778 learning the language itself will pay off when writing package 779 expressions. 780 781 Important: In Nix, everything is an expression, there are no statements. 782 This is common in functional languages. 783 784 Important: Values in Nix are immutable. 785 786 ** Value types 787 Nix 2.0 contains a command named =nix repl= which is a simple command 788 line tool for playing with the Nix language. In fact, Nix is a 789 [[https://nix.dev/manual/nix/stable/language/][pure, lazy, functional 790 language]], not only a set of tools to manage derivations. The 791 =nix repl= syntax is slightly different to Nix syntax when it comes to 792 assigning variables, but it shouldn't be confusing so long as you bear 793 it in mind. I prefer to start with =nix repl= before cluttering your 794 mind with more complex expressions. 795 796 Launch =nix repl=. First of all, Nix supports basic arithmetic 797 operations: =+=, =-=, =*= and =/=. (To exit =nix repl=, use the command 798 =:q=. Help is available through the =:?= command.) 799 800 #+begin_example 801 nix-repl> 1+3 802 4 803 804 nix-repl> 7-4 805 3 806 807 nix-repl> 3*2 808 6 809 #+end_example 810 811 Attempting to perform division in Nix can lead to some surprises. 812 813 #+begin_example 814 nix-repl> 6/3 815 /home/nix/6/3 816 #+end_example 817 818 What happened? Recall that Nix is not a general purpose language, it's a 819 domain-specific language for writing packages. Integer division isn't 820 actually that useful when writing package expressions. Nix parsed =6/3= 821 as a relative path to the current directory. To get Nix to perform 822 division instead, leave a space after the =/=. Alternatively, you can 823 use =builtins.div=. 824 825 #+begin_example 826 nix-repl> 6/ 3 827 2 828 829 nix-repl> builtins.div 6 3 830 2 831 #+end_example 832 833 Other operators are =||=, =&&= and =!= for booleans, and relational 834 operators such as =!==, ====, =<=, =>=, =<==, =>==. In Nix, =<=, =>=, 835 =<== and =>== are not much used. There are also other operators we will 836 see in the course of this series. 837 838 Nix has integer, floating point, string, path, boolean and null 839 [[https://nix.dev/manual/nix/stable/language/#overview][simple]] types. 840 Then there are also lists, sets and functions. These types are enough to 841 build an operating system. 842 843 Nix is strongly typed, but it's not statically typed. That is, you 844 cannot mix strings and integers, you must first do the conversion. 845 846 As demonstrated above, expressions will be parsed as paths as long as 847 there's a slash not followed by a space. Therefore to specify the 848 current directory, use =./.= In addition, Nix also parses urls 849 specially. 850 851 Not all urls or paths can be parsed this way. If a syntax error occurs, 852 it's still possible to fallback to plain strings. Literal urls and paths 853 are convenient for additional safety. 854 855 ** Identifier 856 There's not much to say here, except that dash (=-=) is allowed in 857 identifiers. That's convenient since many packages use dash in their 858 names. In fact: 859 860 #+begin_example 861 nix-repl> a-b 862 error: undefined variable `a-b' at (string):1:1 863 nix-repl> a - b 864 error: undefined variable `a' at (string):1:1 865 #+end_example 866 867 As you can see, =a-b= is parsed as identifier, not as a subtraction. 868 869 ** Strings 870 It's important to understand the syntax for strings. When learning to 871 read Nix expressions, you may find dollars (=$=) ambiguous, but they are 872 very important . Strings are enclosed by double quotes (="=), or two 873 single quotes (=''=). 874 875 #+begin_example 876 nix-repl> "foo" 877 "foo" 878 nix-repl> ''foo'' 879 "foo" 880 #+end_example 881 882 In other languages like Python you can also use single quotes for 883 strings (e.g. ='foo'=), but not in Nix. 884 885 It's possible to 886 [[https://nix.dev/manual/nix/stable/language/string-interpolation][interpolate]] 887 whole Nix expressions inside strings with the =${...}= syntax and only 888 that syntax, not =$foo= or ={$foo}= or anything else. 889 890 #+begin_example 891 nix-repl> foo = "strval" 892 nix-repl> "$foo" 893 "$foo" 894 nix-repl> "${foo}" 895 "strval" 896 nix-repl> "${2+3}" 897 error: cannot coerce an integer to a string, at (string):1:2 898 #+end_example 899 900 Note: ignore the =foo = "strval"= assignment, special syntax in 901 =nix repl=. 902 903 As said previously, you cannot mix integers and strings. You need to 904 explicitly include conversions. We'll see this later: function calls are 905 another story. 906 907 Using the syntax with two single quotes is useful for writing double 908 quotes inside strings without needing to escape them: 909 910 #+begin_example 911 nix-repl> ''test " test'' 912 "test \" test" 913 nix-repl> ''${foo}'' 914 "strval" 915 #+end_example 916 917 Escaping =${...}= within double quoted strings is done with the 918 backslash. Within two single quotes, it's done with =''=: 919 920 #+begin_example 921 nix-repl> "\${foo}" 922 "${foo}" 923 nix-repl> ''test ''${foo} test'' 924 "test ${foo} test" 925 #+end_example 926 927 ** Lists 928 Lists are a sequence of expressions delimited by space (/not/ comma): 929 930 #+begin_example 931 nix-repl> [ 2 "foo" true (2+3) ] 932 [ 2 "foo" true 5 ] 933 #+end_example 934 935 Lists, like everything else in Nix, are immutable. Adding or removing 936 elements from a list is possible, but will return a new list. 937 938 ** Attribute sets 939 An attribute set is an association between string keys and Nix values. 940 Keys can only be strings. When writing attribute sets you can also use 941 unquoted identifiers as keys. 942 943 #+begin_example 944 nix-repl> s = { foo = "bar"; a-b = "baz"; "123" = "num"; } 945 nix-repl> s 946 { "123" = "num"; a-b = "baz"; foo = "bar"; } 947 #+end_example 948 949 For those reading Nix expressions from nixpkgs: do not confuse attribute 950 sets with argument sets used in functions. 951 952 To access elements in the attribute set: 953 954 #+begin_example 955 nix-repl> s.a-b 956 "baz" 957 nix-repl> s."123" 958 "num" 959 #+end_example 960 961 Yes, you can use strings to address keys which aren't valid identifiers. 962 963 Inside an attribute set you cannot normally refer to elements of the 964 same attribute set: 965 966 #+begin_example 967 nix-repl> { a = 3; b = a+4; } 968 error: undefined variable `a' at (string):1:10 969 #+end_example 970 971 To do so, use 972 [[https://nix.dev/manual/nix/stable/language/constructs#recursive-sets][recursive 973 attribute sets]]: 974 975 #+begin_example 976 nix-repl> rec { a = 3; b = a+4; } 977 { a = 3; b = 7; } 978 #+end_example 979 980 This is very convenient when defining packages, which tend to be 981 recursive attribute sets. 982 983 ** If expressions 984 These are expressions, not statements. 985 986 #+begin_example 987 nix-repl> a = 3 988 nix-repl> b = 4 989 nix-repl> if a > b then "yes" else "no" 990 "no" 991 #+end_example 992 993 You can't have only the =then= branch, you must specify also the =else= 994 branch, because an expression must have a value in all cases. 995 996 ** Let expressions 997 This kind of expression is used to define local variables for inner 998 expressions. 999 1000 #+begin_example 1001 nix-repl> let a = "foo"; in a 1002 "foo" 1003 #+end_example 1004 1005 The syntax is: first assign variables, then =in=, then an expression 1006 which can use the defined variables. The value of the whole =let= 1007 expression will be the value of the expression after the =in=. 1008 1009 #+begin_example 1010 nix-repl> let a = 3; b = 4; in a + b 1011 7 1012 #+end_example 1013 1014 Let's write two =let= expressions, one inside the other: 1015 1016 #+begin_example 1017 nix-repl> let a = 3; in let b = 4; in a + b 1018 7 1019 #+end_example 1020 1021 With =let= you cannot assign twice to the same variable. However, you 1022 can shadow outer variables: 1023 1024 #+begin_example 1025 nix-repl> let a = 3; a = 8; in a 1026 error: attribute `a' at (string):1:12 already defined at (string):1:5 1027 nix-repl> let a = 3; in let a = 8; in a 1028 8 1029 #+end_example 1030 1031 You cannot refer to variables in a =let= expression outside of it: 1032 1033 #+begin_example 1034 nix-repl> let a = (let c = 3; in c); in c 1035 error: undefined variable `c' at (string):1:31 1036 #+end_example 1037 1038 You can refer to variables in the =let= expression when assigning 1039 variables, like with recursive attribute sets: 1040 1041 #+begin_example 1042 nix-repl> let a = 4; b = a + 5; in b 1043 9 1044 #+end_example 1045 1046 So beware when you want to refer to a variable from the outer scope, but 1047 it's also defined in the current let expression. The same applies to 1048 recursive attribute sets. 1049 1050 ** With expression 1051 This kind of expression is something you rarely see in other languages. 1052 You can think of it like a more granular version of =using= from C++, or 1053 =from module import *= from Python. You decide per-expression when to 1054 include symbols into the scope. 1055 1056 #+begin_example 1057 nix-repl> longName = { a = 3; b = 4; } 1058 nix-repl> longName.a + longName.b 1059 7 1060 nix-repl> with longName; a + b 1061 7 1062 #+end_example 1063 1064 That's it, it takes an attribute set and includes symbols from it in the 1065 scope of the inner expression. Of course, only valid identifiers from 1066 the keys of the set will be included. If a symbol exists in the outer 1067 scope and would also be introduced by the =with=, it will /not/ be 1068 shadowed. You can however still refer to the attribute set: 1069 1070 #+begin_example 1071 nix-repl> let a = 10; in with longName; a + b 1072 14 1073 nix-repl> let a = 10; in with longName; longName.a + b 1074 7 1075 #+end_example 1076 1077 ** Laziness 1078 Nix evaluates expressions only when needed. This is a great feature when 1079 working with packages. 1080 1081 #+begin_example 1082 nix-repl> let a = builtins.div 4 0; b = 6; in b 1083 6 1084 #+end_example 1085 1086 Since =a= is not needed, there's no error about division by zero, 1087 because the expression is not in need to be evaluated. That's why we can 1088 have all the packages defined on demand, yet have access to specific 1089 packages very quickly. 1090 1091 ** Next pill 1092 ...we will talk about functions and imports. In this pill I've tried to 1093 avoid function calls as much as possible, otherwise the post would have 1094 been too long. 1095 1096 <<05-functions-and-imports.html>> 1097 1098 * Functions and Imports 1099 Welcome to the fifth Nix pill. In the previous 1100 [[#04-basics-of-language.html][fourth pill]] we touched the Nix language 1101 for a moment. We introduced basic types and values of the Nix language, 1102 and basic expressions such as =if=, =with= and =let=. I invite you to 1103 re-read about these expressions and play with them in the repl. 1104 1105 Functions help to build reusable components in a big repository like 1106 [[https://github.com/NixOS/nixpkgs/][nixpkgs]]. The Nix manual has a 1107 [[https://nix.dev/manual/nix/stable/language/constructs#functions][great 1108 explanation of functions]]. Let's go: pill on one hand, Nix manual on 1109 the other hand. 1110 1111 I remind you how to enter the Nix environment: 1112 =source ~/.nix-profile/etc/profile.d/nix.sh= 1113 1114 ** Nameless and single parameter 1115 Functions are anonymous (lambdas), and only have a single parameter. The 1116 syntax is extremely simple. Type the parameter name, then "=:=", then 1117 the body of the function. 1118 1119 #+begin_example 1120 nix-repl> x: x*2 1121 «lambda» 1122 #+end_example 1123 1124 So here we defined a function that takes a parameter =x=, and returns 1125 =x*2=. The problem is that we cannot use it in any way, because it's 1126 unnamed... joke! 1127 1128 We can store functions in variables. 1129 1130 #+begin_example 1131 nix-repl> double = x: x*2 1132 nix-repl> double 1133 «lambda» 1134 nix-repl> double 3 1135 6 1136 #+end_example 1137 1138 As usual, please ignore the special syntax for assignments inside 1139 =nix repl=. So, we defined a function =x: x*2= that takes one parameter 1140 =x=, and returns =x*2=. This function is then assigned to the variable 1141 =double=. Finally we did our first function call: =double 3=. 1142 1143 Big note: it's not like many other programming languages where you write 1144 =double(3)=. It really is =double 3=. 1145 1146 In summary: to call a function, name the variable, then space, then the 1147 argument. Nothing else to say, it's as easy as that. 1148 1149 ** More than one parameter 1150 How do we create a function that accepts more than one parameter? For 1151 people not used to functional programming, this may take a while to 1152 grasp. Let's do it step by step. 1153 1154 #+begin_example 1155 nix-repl> mul = a: (b: a*b) 1156 nix-repl> mul 1157 «lambda» 1158 nix-repl> mul 3 1159 «lambda» 1160 nix-repl> (mul 3) 4 1161 12 1162 #+end_example 1163 1164 We defined a function that takes the parameter =a=, the body returns 1165 another function. This other function takes a parameter =b= and returns 1166 =a*b=. Therefore, calling =mul 3= returns this kind of function: 1167 =b: 3*b=. In turn, we call the returned function with =4=, and get the 1168 expected result. 1169 1170 You don't have to use parentheses at all, Nix has sane priorities when 1171 parsing the code: 1172 1173 #+begin_example 1174 nix-repl> mul = a: b: a*b 1175 nix-repl> mul 1176 «lambda» 1177 nix-repl> mul 3 1178 «lambda» 1179 nix-repl> mul 3 4 1180 12 1181 nix-repl> mul (6+7) (8+9) 1182 221 1183 #+end_example 1184 1185 Much more readable, you don't even notice that functions only receive 1186 one argument. Since the argument is separated by a space, to pass more 1187 complex expressions you need parentheses. In other common languages you 1188 would write =mul(6+7, 8+9)=. 1189 1190 Given that functions have only one parameter, it is straightforward to 1191 use *partial application*: 1192 1193 #+begin_example 1194 nix-repl> foo = mul 3 1195 nix-repl> foo 4 1196 12 1197 nix-repl> foo 5 1198 15 1199 #+end_example 1200 1201 We stored the function returned by =mul 3= into a variable foo, then 1202 reused it. 1203 1204 ** Argument set 1205 Now this is a very cool feature of Nix. It is possible to pattern match 1206 over a set in the parameter. We write an alternative version of 1207 =mul = a: b: a*b= first by using a set as argument, then using pattern 1208 matching. 1209 1210 #+begin_example 1211 nix-repl> mul = s: s.a*s.b 1212 nix-repl> mul { a = 3; b = 4; } 1213 12 1214 nix-repl> mul = { a, b }: a*b 1215 nix-repl> mul { a = 3; b = 4; } 1216 12 1217 #+end_example 1218 1219 In the first case we defined a function that accepts a single parameter. 1220 We then access attributes =a= and =b= from the given set. Note how the 1221 parentheses-less syntax for function calls is very elegant in this case, 1222 instead of doing =mul({ a=3; b=4; })= in other languages. 1223 1224 In the second case we defined an argument set. It's like defining a set, 1225 except without values. We require that the passed set contains the keys 1226 =a= and =b=. Then we can use those =a= and =b= in the function body 1227 directly. 1228 1229 #+begin_example 1230 nix-repl> mul = { a, b }: a*b 1231 nix-repl> mul { a = 3; b = 4; c = 6; } 1232 error: anonymous function at (string):1:2 called with unexpected argument `c', at (string):1:1 1233 nix-repl> mul { a = 3; } 1234 error: anonymous function at (string):1:2 called without required argument `b', at (string):1:1 1235 #+end_example 1236 1237 Only a set with exactly the attributes required by the function is 1238 accepted, nothing more, nothing less. 1239 1240 ** Default and variadic attributes 1241 It is possible to specify *default values* of attributes in the argument 1242 set: 1243 1244 #+begin_example 1245 nix-repl> mul = { a, b ? 2 }: a*b 1246 nix-repl> mul { a = 3; } 1247 6 1248 nix-repl> mul { a = 3; b = 4; } 1249 12 1250 #+end_example 1251 1252 Also you can allow passing more attributes (*variadic*) than the 1253 expected ones: 1254 1255 #+begin_example 1256 nix-repl> mul = { a, b, ... }: a*b 1257 nix-repl> mul { a = 3; b = 4; c = 2; } 1258 #+end_example 1259 1260 However, in the function body you cannot access the "c" attribute. The 1261 solution is to give a name to the given set with the *@-pattern*: 1262 1263 #+begin_example 1264 nix-repl> mul = s@{ a, b, ... }: a*b*s.c 1265 nix-repl> mul { a = 3; b = 4; c = 2; } 1266 24 1267 #+end_example 1268 1269 That's it, you give a name to the whole parameter with name@ before the 1270 set pattern. 1271 1272 Advantages of using argument sets: 1273 1274 - Named unordered arguments: you don't have to remember the order of the 1275 arguments. 1276 1277 - You can pass sets, that adds a whole new layer of flexibility and 1278 convenience. 1279 1280 Disadvantages: 1281 1282 - Partial application does not work with argument sets. You have to 1283 specify the whole attribute set, not part of it. 1284 1285 You may find similarities with 1286 [[https://docs.python.org/3/faq/programming.html#how-can-i-pass-optional-or-keyword-parameters-from-one-function-to-another][Python 1287 **kwargs]]. 1288 1289 ** Imports 1290 The =import= function is built-in and provides a way to parse a =.nix= 1291 file. The natural approach is to define each component in a =.nix= file, 1292 then compose by importing these files. 1293 1294 Let's start with the bare metal. 1295 1296 =a.nix=: 1297 1298 #+begin_example 1299 3 1300 #+end_example 1301 1302 =b.nix=: 1303 1304 #+begin_example 1305 4 1306 #+end_example 1307 1308 =mul.nix=: 1309 1310 #+begin_example 1311 a: b: a*b 1312 #+end_example 1313 1314 #+begin_example 1315 nix-repl> a = import ./a.nix 1316 nix-repl> b = import ./b.nix 1317 nix-repl> mul = import ./mul.nix 1318 nix-repl> mul a b 1319 12 1320 #+end_example 1321 1322 Yes it's really that simple. You import a file, and it gets parsed as an 1323 expression. Note that the scope of the imported file does not inherit 1324 the scope of the importer. 1325 1326 =test.nix=: 1327 1328 #+begin_example 1329 x 1330 #+end_example 1331 1332 #+begin_example 1333 nix-repl> let x = 5; in import ./test.nix 1334 error: undefined variable `x' at /home/lethal/test.nix:1:1 1335 #+end_example 1336 1337 So how do we pass information to the module? Use functions, like we did 1338 with =mul.nix=. A more complex example: 1339 1340 =test.nix=: 1341 1342 #+begin_example 1343 { a, b ? 3, trueMsg ? "yes", falseMsg ? "no" }: 1344 if a > b 1345 then builtins.trace trueMsg true 1346 else builtins.trace falseMsg false 1347 #+end_example 1348 1349 #+begin_example 1350 nix-repl> import ./test.nix { a = 5; trueMsg = "ok"; } 1351 trace: ok 1352 true 1353 #+end_example 1354 1355 Explaining: 1356 1357 - In =test.nix= we return a function. It accepts a set, with default 1358 attributes =b=, =trueMsg= and =falseMsg=. 1359 1360 - =builtins.trace= is a 1361 [[https://nix.dev/manual/nix/stable/language/builtins][built-in 1362 function]] that takes two arguments. The first is the message to 1363 display, the second is the value to return. It's usually used for 1364 debugging purposes. 1365 1366 - Then we import =test.nix=, and call the function with that set. 1367 1368 So when is the message shown? Only when it needs to be evaluated. 1369 1370 ** Next pill 1371 ...we will finally write our first derivation. 1372 1373 <<06-our-first-derivation.html>> 1374 1375 * Our First Derivation 1376 Welcome to the sixth Nix pill. In the previous 1377 [[#05-functions-and-imports.html][fifth pill]] we introduced functions 1378 and imports. Functions and imports are very simple concepts that allow 1379 for building complex abstractions and composition of modules to build a 1380 flexible Nix system. 1381 1382 In this post we finally arrived to writing a derivation. Derivations are 1383 the building blocks of a Nix system, from a file system view point. The 1384 Nix language is used to describe such derivations. 1385 1386 I remind you how to enter the Nix environment: 1387 =source ~/.nix-profile/etc/profile.d/nix.sh= 1388 1389 ** The derivation function 1390 The [[https://nix.dev/manual/nix/stable/language/derivations][derivation 1391 built-in function]] is used to create derivations. I invite you to read 1392 the link in the Nix manual about the derivation built-in. A derivation 1393 from a Nix language view point is simply a set, with some attributes. 1394 Therefore you can pass the derivation around with variables like 1395 anything else. 1396 1397 That's where the real power comes in. 1398 1399 The =derivation= function receives a set as its first argument. This set 1400 requires at least the following three attributes: 1401 1402 - name: the name of the derivation. In the nix store the format is 1403 hash-name, that's the name. 1404 1405 - system: is the name of the system in which the derivation can be 1406 built. For example, x86_64-linux. 1407 1408 - builder: is the binary program that builds the derivation. 1409 1410 First of all, what's the name of our system as seen by nix? 1411 1412 #+begin_example 1413 nix-repl> builtins.currentSystem 1414 "x86_64-linux" 1415 #+end_example 1416 1417 Let's try to fake the name of the system: 1418 1419 #+begin_example 1420 nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = "mysystem"; } 1421 nix-repl> d 1422 «derivation /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv» 1423 #+end_example 1424 1425 Oh oh, what's that? Did it build the derivation? No it didn't, but it 1426 *did create the .drv file*. =nix repl= does not build derivations unless 1427 you tell it to do so. 1428 1429 ** Digression about .drv files 1430 What's that =.drv= file? It is the specification of how to build the 1431 derivation, without all the Nix language fuzz. 1432 1433 Before continuing, some analogies with the C language: 1434 1435 - =.nix= files are like =.c= files. 1436 1437 - =.drv= files are intermediate files like =.o= files. The =.drv= 1438 describes how to build a derivation; it's the bare minimum 1439 information. 1440 1441 - out paths are then the product of the build. 1442 1443 Both drv paths and out paths are stored in the nix store as you can see. 1444 1445 What's in that =.drv= file? You can read it, but it's better to pretty 1446 print it: 1447 1448 Note: If your version of nix doesn't have =nix derivation show=, use 1449 =nix show-derivation= instead. 1450 1451 #+begin_example 1452 $ nix derivation show /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv 1453 { 1454 "/nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv": { 1455 "outputs": { 1456 "out": { 1457 "path": "/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname" 1458 } 1459 }, 1460 "inputSrcs": [], 1461 "inputDrvs": {}, 1462 "platform": "mysystem", 1463 "builder": "mybuilder", 1464 "args": [], 1465 "env": { 1466 "builder": "mybuilder", 1467 "name": "myname", 1468 "out": "/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname", 1469 "system": "mysystem" 1470 } 1471 } 1472 } 1473 #+end_example 1474 1475 Ok, we can see there's an out path, but it does not exist yet. We never 1476 told Nix to build it, but we know beforehand where the build output will 1477 be. Why? 1478 1479 Think, if Nix ever built the derivation just because we accessed it in 1480 Nix, we would have to wait a long time if it was, say, Firefox. That's 1481 why Nix let us know the path beforehand and kept evaluating the Nix 1482 expressions, but it's still empty because no build was ever made. 1483 1484 Important: the hash of the out path is based solely on the input 1485 derivations in the current version of Nix, not on the contents of the 1486 build product. It's possible however to have 1487 [[https://en.wikipedia.org/wiki/Content-addressable_storage][content-addressable]] 1488 derivations for e.g. tarballs as we'll see later on. 1489 1490 Many things are empty in that =.drv=, however I'll write a summary of 1491 the [[http://nixos.org/~eelco/pubs/phd-thesis.pdf][.drv format]] for 1492 you: 1493 1494 1. The output paths (there can be multiple ones). By default nix creates 1495 one out path called "out". 1496 1497 2. The list of input derivations. It's empty because we are not 1498 referring to any other derivation. Otherwise, there would be a list 1499 of other .drv files. 1500 1501 3. The system and the builder executable (yes, it's a fake one). 1502 1503 4. Then a list of environment variables passed to the builder. 1504 1505 That's it, the minimum necessary information to build our derivation. 1506 1507 Important note: the environment variables passed to the builder are just 1508 those you see in the .drv plus some other Nix related configuration 1509 (number of cores, temp dir, ...). The builder will not inherit any 1510 variable from your running shell, otherwise builds would suffer from 1511 [[https://wiki.debian.org/ReproducibleBuilds][non-determinism]]. 1512 1513 Back to our fake derivation. 1514 1515 Let's build our really fake derivation: 1516 1517 #+begin_example 1518 nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = "mysystem"; } 1519 nix-repl> :b d 1520 [...] 1521 these derivations will be built: 1522 /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv 1523 building path(s) `/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname' 1524 error: a `mysystem' is required to build `/nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv', but I am a `x86_64-linux' 1525 #+end_example 1526 1527 The =:b= is a =nix repl= specific command to build a derivation. You can 1528 see more commands with =:?= . So in the output you can see that it takes 1529 the =.drv= as information on how to build the derivation. Then it says 1530 it's trying to produce our out path. Finally the error we were waiting 1531 for: that derivation can't be built on our system. 1532 1533 We're doing the build inside =nix repl=, but what if we don't want to 1534 use =nix repl=? You can *realise* a =.drv= with: 1535 1536 #+begin_example 1537 $ nix-store -r /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv 1538 #+end_example 1539 1540 You will get the same output as before. 1541 1542 Let's fix the system attribute: 1543 1544 #+begin_example 1545 nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = builtins.currentSystem; } 1546 nix-repl> :b d 1547 [...] 1548 build error: invalid file name `mybuilder' 1549 #+end_example 1550 1551 A step forward: of course, that =mybuilder= executable does not really 1552 exist. Stop for a moment. 1553 1554 ** What's in a derivation set 1555 It is useful to start by inspecting the return value from the derivation 1556 function. In this case, the returned value is a plain set: 1557 1558 #+begin_example 1559 nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = "mysystem"; } 1560 nix-repl> builtins.isAttrs d 1561 true 1562 nix-repl> builtins.attrNames d 1563 [ "all" "builder" "drvAttrs" "drvPath" "name" "out" "outPath" "outputName" "system" "type" ] 1564 #+end_example 1565 1566 You can guess what =builtins.isAttrs= does; it returns true if the 1567 argument is a set. While =builtins.attrNames= returns a list of keys of 1568 the given set. Some kind of reflection, you might say. 1569 1570 Start from drvAttrs: 1571 1572 #+begin_example 1573 nix-repl> d.drvAttrs 1574 { builder = "mybuilder"; name = "myname"; system = "mysystem"; } 1575 #+end_example 1576 1577 That's basically the input we gave to the derivation function. Also the 1578 =d.name=, =d.system= and =d.builder= attributes are exactly the ones we 1579 gave as input. 1580 1581 #+begin_example 1582 nix-repl> (d == d.out) 1583 true 1584 #+end_example 1585 1586 So out is just the derivation itself, it seems weird but the reason is 1587 that we only have one output from the derivation. That's also the reason 1588 why =d.all= is a singleton. We'll see multiple outputs later. 1589 1590 The =d.drvPath= is the path of the =.drv= file: 1591 =/nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv=. 1592 1593 Something interesting is the =type= attribute. It's ="derivation"=. Nix 1594 does add a little of magic to sets with type derivation, but not that 1595 much. To help you understand, you can create yourself a set with that 1596 type, it's a simple set: 1597 1598 #+begin_example 1599 nix-repl> { type = "derivation"; } 1600 «derivation ???» 1601 #+end_example 1602 1603 Of course it has no other information, so Nix doesn't know what to say 1604 :-) But you get it, the =type = "derivation"= is just a convention for 1605 Nix and for us to understand the set is a derivation. 1606 1607 When writing packages, we are interested in the outputs. The other 1608 metadata is needed for Nix to know how to create the drv path and the 1609 out path. 1610 1611 The =outPath= attribute is the build path in the nix store: 1612 =/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname=. 1613 1614 ** Referring to other derivations 1615 Just like dependencies in other package managers, how do we refer to 1616 other packages? How do we refer to other derivations in terms of files 1617 on the disk? We use the =outPath=. The =outPath= describes the location 1618 of the files of that derivation. To make it more convenient, Nix is able 1619 to do a conversion from a derivation set to a string. 1620 1621 #+begin_example 1622 nix-repl> d.outPath 1623 "/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname" 1624 nix-repl> builtins.toString d 1625 "/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname" 1626 #+end_example 1627 1628 Nix does the "set to string conversion" as long as there is the 1629 =outPath= attribute (much like a toString method in other languages): 1630 1631 #+begin_example 1632 nix-repl> builtins.toString { outPath = "foo"; } 1633 "foo" 1634 nix-repl> builtins.toString { a = "b"; } 1635 error: cannot coerce a set to a string, at (string):1:1 1636 #+end_example 1637 1638 Say we want to use binaries from coreutils (ignore the nixpkgs etc.): 1639 1640 #+begin_example 1641 nix-repl> :l <nixpkgs> 1642 Added 3950 variables. 1643 nix-repl> coreutils 1644 «derivation /nix/store/1zcs1y4n27lqs0gw4v038i303pb89rw6-coreutils-8.21.drv» 1645 nix-repl> builtins.toString coreutils 1646 "/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21" 1647 #+end_example 1648 1649 Apart from the nixpkgs stuff, just think we added to the scope a series 1650 of variables. One of them is coreutils. It is the derivation of the 1651 coreutils package you all know of from other Linux distributions. It 1652 contains basic binaries for GNU/Linux systems (you may have multiple 1653 derivations of coreutils in the nix store, no worries): 1654 1655 #+begin_example 1656 $ ls /nix/store/*coreutils*/bin 1657 [...] 1658 #+end_example 1659 1660 I remind you, inside strings it's possible to interpolate Nix 1661 expressions with =${...}=: 1662 1663 #+begin_example 1664 nix-repl> "${d}" 1665 "/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname" 1666 nix-repl> "${coreutils}" 1667 "/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21" 1668 #+end_example 1669 1670 That's very convenient, because then we could refer to e.g. the bin/true 1671 binary like this: 1672 1673 #+begin_example 1674 nix-repl> "${coreutils}/bin/true" 1675 "/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21/bin/true" 1676 #+end_example 1677 1678 ** An almost working derivation 1679 In the previous attempt we used a fake builder, =mybuilder= which 1680 obviously does not exist. But we can use for example bin/true, which 1681 always exits with 0 (success). 1682 1683 #+begin_example 1684 nix-repl> :l <nixpkgs> 1685 nix-repl> d = derivation { name = "myname"; builder = "${coreutils}/bin/true"; system = builtins.currentSystem; } 1686 nix-repl> :b d 1687 [...] 1688 builder for `/nix/store/qyfrcd53wmc0v22ymhhd5r6sz5xmdc8a-myname.drv' failed to produce output path `/nix/store/ly2k1vswbfmswr33hw0kf0ccilrpisnk-myname' 1689 #+end_example 1690 1691 Another step forward, it executed the builder (bin/true), but the 1692 builder did not create the out path of course, it just exited with 0. 1693 1694 Obvious note: every time we change the derivation, a new hash is 1695 created. 1696 1697 Let's examine the new =.drv= now that we referred to another derivation: 1698 1699 #+begin_example 1700 $ nix derivation show /nix/store/qyfrcd53wmc0v22ymhhd5r6sz5xmdc8a-myname.drv 1701 { 1702 "/nix/store/qyfrcd53wmc0v22ymhhd5r6sz5xmdc8a-myname.drv": { 1703 "outputs": { 1704 "out": { 1705 "path": "/nix/store/ly2k1vswbfmswr33hw0kf0ccilrpisnk-myname" 1706 } 1707 }, 1708 "inputSrcs": [], 1709 "inputDrvs": { 1710 "/nix/store/hixdnzz2wp75x1jy65cysq06yl74vx7q-coreutils-8.29.drv": [ 1711 "out" 1712 ] 1713 }, 1714 "platform": "x86_64-linux", 1715 "builder": "/nix/store/qrxs7sabhqcr3j9ai0j0cp58zfnny0jz-coreutils-8.29/bin/true", 1716 "args": [], 1717 "env": { 1718 "builder": "/nix/store/qrxs7sabhqcr3j9ai0j0cp58zfnny0jz-coreutils-8.29/bin/true", 1719 "name": "myname", 1720 "out": "/nix/store/ly2k1vswbfmswr33hw0kf0ccilrpisnk-myname", 1721 "system": "x86_64-linux" 1722 } 1723 } 1724 } 1725 #+end_example 1726 1727 Aha! Nix added a dependency to our myname.drv, it's the coreutils.drv. 1728 Before doing our build, Nix should build the coreutils.drv. But since 1729 coreutils is already in our nix store, no build is needed, it's already 1730 there with out path 1731 =/nix/store/qrxs7sabhqcr3j9ai0j0cp58zfnny0jz-coreutils-8.29=. 1732 1733 ** When is the derivation built 1734 Nix does not build derivations *during evaluation* of Nix expressions. 1735 In fact, that's why we have to do ":b drv" in =nix repl=, or use 1736 nix-store -r in the first place. 1737 1738 An important separation is made in Nix: 1739 1740 - *Instantiate/Evaluation time*: the Nix expression is parsed, 1741 interpreted and finally returns a derivation set. During evaluation, 1742 you can refer to other derivations because Nix will create .drv files 1743 and we will know out paths beforehand. This is achieved with 1744 [[https://nix.dev/manual/nix/stable/command-ref/nix-instantiate][nix-instantiate]]. 1745 1746 - *Realise/Build time*: the .drv from the derivation set is built, first 1747 building .drv inputs (build dependencies). This is achieved with 1748 [[https://nix.dev/manual/nix/stable/command-ref/nix-store/realise][nix-store 1749 -r]]. 1750 1751 Think of it as of compile time and link time like with C/C++ projects. 1752 You first compile all source files to object files. Then link object 1753 files in a single executable. 1754 1755 In Nix, first the Nix expression (usually in a .nix file) is compiled to 1756 .drv, then each .drv is built and the product is installed in the 1757 relative out paths. 1758 1759 ** Conclusion 1760 Is it that complicated to create a package for Nix? No, it's not. 1761 1762 We're walking through the fundamentals of Nix derivations, to understand 1763 how they work, how they are represented. Packaging in Nix is certainly 1764 easier than that, but we're not there yet in this post. More Nix pills 1765 are needed. 1766 1767 With the derivation function we provide a set of information on how to 1768 build a package, and we get back the information about where the package 1769 was built. Nix converts a set to a string when there's an =outPath=; 1770 that's very convenient. With that, it's easy to refer to other 1771 derivations. 1772 1773 When Nix builds a derivation, it first creates a .drv file from a 1774 derivation expression, and uses it to build the output. It does so 1775 recursively for all the dependencies (inputs). It "executes" the .drv 1776 files like a machine. Not much magic after all. 1777 1778 ** Next pill 1779 ...we will finally write our first *working* derivation. Yes, this post 1780 is about "our first derivation", but I never said it was a working one 1781 ;) 1782 1783 <<07-working-derivation.html>> 1784 1785 * Working Derivation 1786 ** Introduction 1787 Welcome to the seventh nix pill. In the previous 1788 [[#06-our-first-derivation.html][sixth pill]] we introduced the notion 1789 of derivation in the Nix language --- how to define a raw derivation and 1790 how to (try to) build it. 1791 1792 In this post we continue along the path, by creating a derivation that 1793 actually builds something. Then, we try to package a real program: we 1794 compile a simple C file and create a derivation out of it, given a 1795 blessed toolchain. 1796 1797 I remind you how to enter the Nix environment: 1798 =source ~/.nix-profile/etc/profile.d/nix.sh= 1799 1800 ** Using a script as a builder 1801 What's the easiest way to run a sequence of commands for building 1802 something? A bash script. We write a custom bash script, and we want it 1803 to be our builder. Given a =builder.sh=, we want the derivation to run 1804 =bash builder.sh=. 1805 1806 We don't use hash bangs in =builder.sh=, because at the time we are 1807 writing it we do not know the path to bash in the nix store. Yes, even 1808 bash is in the nix store, everything is there. 1809 1810 We don't even use /usr/bin/env, because then we lose the cool stateless 1811 property of Nix. Not to mention that =PATH= gets cleared when building, 1812 so it wouldn't find bash anyway. 1813 1814 In summary, we want the builder to be bash, and pass it an argument, 1815 =builder.sh=. Turns out the =derivation= function accepts an optional 1816 =args= attribute which is used to pass arguments to the builder 1817 executable. 1818 1819 First of all, let's write our =builder.sh= in the current directory: 1820 1821 #+begin_src sh 1822 declare -xp 1823 echo foo > $out 1824 #+end_src 1825 1826 The command =declare -xp= lists exported variables (=declare= is a 1827 builtin bash function). As we covered in the previous pill, Nix computes 1828 the output path of the derivation. The resulting =.drv= file contains a 1829 list of environment variables passed to the builder. One of these is 1830 =$out=. 1831 1832 What we have to do is create something in the path =$out=, be it a file 1833 or a directory. In this case we are creating a file. 1834 1835 In addition, we print out the environment variables during the build 1836 process. We cannot use env for this, because env is part of coreutils 1837 and we don't have a dependency to it yet. We only have bash for now. 1838 1839 Like for coreutils in the previous pill, we get a blessed bash for free 1840 from our magic nixpkgs stuff: 1841 1842 #+begin_example 1843 nix-repl> :l <nixpkgs> 1844 Added 3950 variables. 1845 nix-repl> "${bash}" 1846 "/nix/store/ihmkc7z2wqk3bbipfnlh0yjrlfkkgnv6-bash-4.2-p45" 1847 #+end_example 1848 1849 So with the usual trick, we can refer to bin/bash and create our 1850 derivation: 1851 1852 #+begin_example 1853 nix-repl> d = derivation { name = "foo"; builder = "${bash}/bin/bash"; args = [ ./builder.sh ]; system = builtins.currentSystem; } 1854 nix-repl> :b d 1855 [1 built, 0.0 MiB DL] 1856 1857 this derivation produced the following outputs: 1858 out -> /nix/store/gczb4qrag22harvv693wwnflqy7lx5pb-foo 1859 #+end_example 1860 1861 We did it! The contents of 1862 =/nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-foo= is really foo. We've 1863 built our first derivation. 1864 1865 Note that we used =./builder.sh= and not ="./builder.sh"=. This way, it 1866 is parsed as a path, and Nix performs some magic which we will cover 1867 later. Try using the string version and you will find that it cannot 1868 find =builder.sh=. This is because it tries to find it relative to the 1869 temporary build directory. 1870 1871 ** The builder environment 1872 We can use =nix-store --read-log= to see the logs our builder produced: 1873 1874 #+begin_example 1875 $ nix-store --read-log /nix/store/gczb4qrag22harvv693wwnflqy7lx5pb-foo 1876 declare -x HOME="/homeless-shelter" 1877 declare -x NIX_BUILD_CORES="4" 1878 declare -x NIX_BUILD_TOP="/tmp/nix-build-foo.drv-0" 1879 declare -x NIX_LOG_FD="2" 1880 declare -x NIX_STORE="/nix/store" 1881 declare -x OLDPWD 1882 declare -x PATH="/path-not-set" 1883 declare -x PWD="/tmp/nix-build-foo.drv-0" 1884 declare -x SHLVL="1" 1885 declare -x TEMP="/tmp/nix-build-foo.drv-0" 1886 declare -x TEMPDIR="/tmp/nix-build-foo.drv-0" 1887 declare -x TMP="/tmp/nix-build-foo.drv-0" 1888 declare -x TMPDIR="/tmp/nix-build-foo.drv-0" 1889 declare -x builder="/nix/store/q1g0rl8zfmz7r371fp5p42p4acmv297d-bash-4.4-p19/bin/bash" 1890 declare -x name="foo" 1891 declare -x out="/nix/store/gczb4qrag22harvv693wwnflqy7lx5pb-foo" 1892 declare -x system="x86_64-linux" 1893 #+end_example 1894 1895 Let's inspect those environment variables printed during the build 1896 process. 1897 1898 - =$HOME= is not your home directory, and =/homeless-shelter= doesn't 1899 exist at all. We force packages not to depend on =$HOME= during the 1900 build process. 1901 1902 - =$PATH= plays the same game as =$HOME= 1903 1904 - =$NIX_BUILD_CORES= and =$NIX_STORE= are 1905 [[https://nix.dev/manual/nix/stable/command-ref/conf-file][nix 1906 configuration options]] 1907 1908 - =$PWD= and =$TMP= clearly show that nix created a temporary build 1909 directory 1910 1911 - Then =$builder=, =$name=, =$out=, and =$system= are variables set due 1912 to the .drv file's contents. 1913 1914 And that's how we were able to use =$out= in our derivation and put 1915 stuff in it. It's like Nix reserved a slot in the nix store for us, and 1916 we must fill it. 1917 1918 In terms of autotools, =$out= will be the =--prefix= path. Yes, not the 1919 make =DESTDIR=, but the =--prefix=. That's the essence of stateless 1920 packaging. You don't install the package in a global common path under 1921 =/=, you install it in a local isolated path under your nix store slot. 1922 1923 ** The .drv contents 1924 We added something else to the derivation this time: the args attribute. 1925 Let's see how this changed the .drv compared to the previous pill: 1926 1927 #+begin_example 1928 $ nix derivation show /nix/store/i76pr1cz0za3i9r6xq518bqqvd2raspw-foo.drv 1929 { 1930 "/nix/store/i76pr1cz0za3i9r6xq518bqqvd2raspw-foo.drv": { 1931 "outputs": { 1932 "out": { 1933 "path": "/nix/store/gczb4qrag22harvv693wwnflqy7lx5pb-foo" 1934 } 1935 }, 1936 "inputSrcs": [ 1937 "/nix/store/lb0n38r2b20r8rl1k45a7s4pj6ny22f7-builder.sh" 1938 ], 1939 "inputDrvs": { 1940 "/nix/store/hcgwbx42mcxr7ksnv0i1fg7kw6jvxshb-bash-4.4-p19.drv": [ 1941 "out" 1942 ] 1943 }, 1944 "platform": "x86_64-linux", 1945 "builder": "/nix/store/q1g0rl8zfmz7r371fp5p42p4acmv297d-bash-4.4-p19/bin/bash", 1946 "args": [ 1947 "/nix/store/lb0n38r2b20r8rl1k45a7s4pj6ny22f7-builder.sh" 1948 ], 1949 "env": { 1950 "builder": "/nix/store/q1g0rl8zfmz7r371fp5p42p4acmv297d-bash-4.4-p19/bin/bash", 1951 "name": "foo", 1952 "out": "/nix/store/gczb4qrag22harvv693wwnflqy7lx5pb-foo", 1953 "system": "x86_64-linux" 1954 } 1955 } 1956 } 1957 #+end_example 1958 1959 Much like the usual .drv, except that there's a list of arguments in 1960 there passed to the builder (bash) with =builder.sh=... In the nix 1961 store..? Nix automatically copies files or directories needed for the 1962 build into the store to ensure that they are not changed during the 1963 build process and that the deployment is stateless and independent of 1964 the building machine. =builder.sh= is not only in the arguments passed 1965 to the builder, it's also in the input sources. 1966 1967 Given that =builder.sh= is a plain file, it has no .drv associated with 1968 it. The store path is computed based on the filename and on the hash of 1969 its contents. Store paths are covered in detail in 1970 [[#18-nix-store-paths.html][a later pill]]. 1971 1972 ** Packaging a simple C program 1973 Start off by writing a simple C program called =simple.c=: 1974 1975 #+begin_src C 1976 void main() { 1977 puts("Simple!"); 1978 } 1979 #+end_src 1980 1981 And its =simple_builder.sh=: 1982 1983 #+begin_src sh 1984 export PATH="$coreutils/bin:$gcc/bin" 1985 mkdir $out 1986 gcc -o $out/simple $src 1987 #+end_src 1988 1989 Don't worry too much about where those variables come from yet; let's 1990 write the derivation and build it: 1991 1992 #+begin_example 1993 nix-repl> :l <nixpkgs> 1994 nix-repl> simple = derivation { name = "simple"; builder = "${bash}/bin/bash"; args = [ ./simple_builder.sh ]; gcc = gcc; coreutils = coreutils; src = ./simple.c; system = builtins.currentSystem; } 1995 nix-repl> :b simple 1996 this derivation produced the following outputs: 1997 1998 out -> /nix/store/ni66p4jfqksbmsl616llx3fbs1d232d4-simple 1999 #+end_example 2000 2001 Now you can run 2002 =/nix/store/ni66p4jfqksbmsl616llx3fbs1d232d4-simple/simple= in your 2003 shell. 2004 2005 ** Explanation 2006 We added two new attributes to the derivation call, =gcc= and 2007 =coreutils=. In =gcc = gcc;=, the name on the left is the name in the 2008 derivation set, and the name on the right refers to the gcc derivation 2009 from nixpkgs. The same applies for coreutils. 2010 2011 We also added the =src= attribute, nothing magical --- it's just a name, 2012 to which the path =./simple.c= is assigned. Like =simple-builder.sh=, 2013 =simple.c= will be added to the store. 2014 2015 The trick: every attribute in the set passed to =derivation= will be 2016 converted to a string and passed to the builder as an environment 2017 variable. This is how the builder gains access to coreutils and gcc: 2018 when converted to strings, the derivations evaluate to their output 2019 paths, and appending =/bin= to these leads us to their binaries. 2020 2021 The same goes for the =src= variable. =$src= is the path to =simple.c= 2022 in the nix store. As an exercise, pretty print the .drv file. You'll see 2023 =simple_builder.sh= and =simple.c= listed in the input derivations, 2024 along with bash, gcc and coreutils .drv files. The newly added 2025 environment variables described above will also appear. 2026 2027 In =simple_builder.sh= we set the =PATH= for gcc and coreutils binaries, 2028 so that our build script can find the necessary utilities like mkdir and 2029 gcc. 2030 2031 We then create =$out= as a directory and place the binary inside it. 2032 Note that gcc is found via the =PATH= environment variable, but it could 2033 equivalently be referenced explicitly using =$gcc/bin/gcc=. 2034 2035 ** Enough of =nix repl= 2036 Drop out of nix repl and write a file =simple.nix=: 2037 2038 #+begin_example 2039 let 2040 pkgs = import <nixpkgs> { }; 2041 in 2042 derivation { 2043 name = "simple"; 2044 builder = "${pkgs.bash}/bin/bash"; 2045 args = [ ./simple_builder.sh ]; 2046 gcc = pkgs.gcc; 2047 coreutils = pkgs.coreutils; 2048 src = ./simple.c; 2049 system = builtins.currentSystem; 2050 } 2051 #+end_example 2052 2053 Now you can build it with =nix-build simple.nix=. This will create a 2054 symlink =result= in the current directory, pointing to the out path of 2055 the derivation. 2056 2057 nix-build does two jobs: 2058 2059 - [[https://nix.dev/manual/nix/stable/command-ref/nix-instantiate][nix-instantiate]]: 2060 parse and evaluate =simple.nix= and return the .drv file corresponding 2061 to the parsed derivation set 2062 2063 - [[https://nix.dev/manual/nix/stable/command-ref/nix-store/realise][=nix-store -r=]]: 2064 realise the .drv file, which actually builds it. 2065 2066 Finally, it creates the symlink. 2067 2068 In the second line of =simple.nix=, we have an =import= function call. 2069 Recall that =import= accepts one argument, a nix file to load. In this 2070 case, the contents of the file evaluate to a function. 2071 2072 Afterwards, we call the function with the empty set. We saw this already 2073 in [[#05-functions-and-imports.html][the fifth pill]]. To reiterate: 2074 =import <nixpkgs> {}= is calling two functions, not one. Reading it as 2075 =(import <nixpkgs>) {}= makes this clearer. 2076 2077 The value returned by the nixpkgs function is a set; more specifically, 2078 it's a set of derivations. Calling =import <nixpkgs> {}= into a 2079 =let=-expression creates the local variable =pkgs= and brings it into 2080 scope. This has an effect similar to the =:l <nixpkgs>= we used in nix 2081 repl, in that it allows us to easily access derivations such as =bash=, 2082 =gcc=, and =coreutils=, but those derivations will have to be explicitly 2083 referred to as members of the =pkgs= set (e.g., =pkgs.bash= instead of 2084 just =bash=). 2085 2086 Below is a revised version of the =simple.nix= file, using the =inherit= 2087 keyword: 2088 2089 #+begin_example 2090 let 2091 pkgs = import <nixpkgs> { }; 2092 in 2093 derivation { 2094 name = "simple"; 2095 builder = "${pkgs.bash}/bin/bash"; 2096 args = [ ./simple_builder.sh ]; 2097 inherit (pkgs) gcc coreutils; 2098 src = ./simple.c; 2099 system = builtins.currentSystem; 2100 } 2101 #+end_example 2102 2103 Here we also take the opportunity to introduce the 2104 [[https://nix.dev/manual/nix/stable/language/constructs#inheriting-attributes][=inherit= 2105 keyword]]. =inherit foo;= is equivalent to =foo = foo;=. Similarly, 2106 =inherit gcc coreutils;= is equivalent to 2107 =gcc = gcc; coreutils = coreutils;=. Lastly, 2108 =inherit (pkgs) gcc coreutils;= is equivalent to 2109 =gcc = pkgs.gcc; coreutils = pkgs.coreutils;=. 2110 2111 This syntax only makes sense inside sets. There's no magic involved, 2112 it's simply a convenience to avoid repeating the same name for both the 2113 attribute name and the value in scope. 2114 2115 ** Next pill 2116 We will generalize the builder. You may have noticed that we wrote two 2117 separate =builder.sh= scripts in this post. We would like to have a 2118 generic builder script instead, especially since each build script goes 2119 in the nix store: a bit of a waste. 2120 2121 /Is it really that hard to package stuff in Nix? No/, here we're 2122 studying the fundamentals of Nix. 2123 2124 <<08-generic-builders.html>> 2125 2126 * Generic Builders 2127 Welcome to the 8th Nix pill. In the previous 2128 [[#07-working-derivation.html][7th pill]] we successfully built a 2129 derivation. We wrote a builder script that compiled a C file and 2130 installed the binary under the nix store. 2131 2132 In this post, we will generalize the builder script, write a Nix 2133 expression for [[https://www.gnu.org/software/hello/][GNU hello world]] 2134 and create a wrapper around the derivation built-in function. 2135 2136 ** Packaging GNU hello world 2137 In the previous pill we packaged a simple .c file, which was being 2138 compiled with a raw gcc call. That's not a good example of a project. 2139 Many use autotools, and since we're going to generalize our builder, it 2140 would be better to do it with the most used build system. 2141 2142 [[https://www.gnu.org/software/hello/][GNU hello world]], despite its 2143 name, is a simple yet complete project which uses autotools. Fetch the 2144 latest tarball here: 2145 [[https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz]]. 2146 2147 Let's create a builder script for GNU hello world, hello_builder.sh: 2148 2149 #+begin_src sh 2150 export PATH="$gnutar/bin:$gcc/bin:$gnumake/bin:$coreutils/bin:$gawk/bin:$gzip/bin:$gnugrep/bin:$gnused/bin:$bintools/bin" 2151 tar -xzf $src 2152 cd hello-2.12.1 2153 ./configure --prefix=$out 2154 make 2155 make install 2156 #+end_src 2157 2158 And the derivation hello.nix: 2159 2160 #+begin_example 2161 let 2162 pkgs = import <nixpkgs> { }; 2163 in 2164 derivation { 2165 name = "hello"; 2166 builder = "${pkgs.bash}/bin/bash"; 2167 args = [ ./hello_builder.sh ]; 2168 inherit (pkgs) 2169 gnutar 2170 gzip 2171 gnumake 2172 gcc 2173 coreutils 2174 gawk 2175 gnused 2176 gnugrep 2177 ; 2178 bintools = pkgs.binutils.bintools; 2179 src = ./hello-2.12.1.tar.gz; 2180 system = builtins.currentSystem; 2181 } 2182 #+end_example 2183 2184 **** Nix on darwin 2185 Darwin (i.e. macOS) builds typically use =clang= rather than =gcc= for a 2186 C compiler. We can adapt this early example for darwin by using this 2187 modified version of =hello.nix=: 2188 2189 #+begin_example 2190 let 2191 pkgs = import <nixpkgs> { }; 2192 in 2193 derivation { 2194 name = "hello"; 2195 builder = "${pkgs.bash}/bin/bash"; 2196 args = [ ./hello_builder.sh ]; 2197 inherit (pkgs) 2198 gnutar 2199 gzip 2200 gnumake 2201 coreutils 2202 gawk 2203 gnused 2204 gnugrep 2205 ; 2206 gcc = pkgs.clang; 2207 bintools = pkgs.clang.bintools.bintools_bin; 2208 src = ./hello-2.12.1.tar.gz; 2209 system = builtins.currentSystem; 2210 } 2211 #+end_example 2212 2213 Later, we will show how Nix can automatically handle these differences. 2214 For now, please be just aware that changes similar to the above may be 2215 needed in what follows. 2216 2217 Now build it with =nix-build hello.nix= and you can launch 2218 =result/bin/hello=. Nothing easier, but do we have to create a 2219 builder.sh for each package? Do we always have to pass the dependencies 2220 to the =derivation= function? 2221 2222 Please note the =--prefix=$out= we were talking about in the 2223 [[#07-working-derivation.html][previous pill]]. 2224 2225 ** A generic builder 2226 Let's create a generic =builder.sh= for autotools projects: 2227 2228 #+begin_src sh 2229 set -e 2230 unset PATH 2231 for p in $buildInputs; do 2232 export PATH=$p/bin${PATH:+:}$PATH 2233 done 2234 2235 tar -xf $src 2236 2237 for d in *; do 2238 if [ -d "$d" ]; then 2239 cd "$d" 2240 break 2241 fi 2242 done 2243 2244 ./configure --prefix=$out 2245 make 2246 make install 2247 #+end_src 2248 2249 What do we do here? 2250 2251 1. Exit the build on any error with =set -e=. 2252 2253 2. First =unset PATH=, because it's initially set to a non-existent 2254 path. 2255 2256 3. We'll see this below in detail, however for each path in 2257 =$buildInputs=, we append =bin= to =PATH=. 2258 2259 4. Unpack the source. 2260 2261 5. Find a directory where the source has been unpacked and =cd= into it. 2262 2263 6. Once we're set up, compile and install. 2264 2265 As you can see, there's no reference to "hello" in the builder anymore. 2266 It still makes several assumptions, but it's certainly more generic. 2267 2268 Now let's rewrite =hello.nix=: 2269 2270 #+begin_example 2271 let 2272 pkgs = import <nixpkgs> { }; 2273 in 2274 derivation { 2275 name = "hello"; 2276 builder = "${pkgs.bash}/bin/bash"; 2277 args = [ ./builder.sh ]; 2278 buildInputs = with pkgs; [ 2279 gnutar 2280 gzip 2281 gnumake 2282 gcc 2283 coreutils 2284 gawk 2285 gnused 2286 gnugrep 2287 binutils.bintools 2288 ]; 2289 src = ./hello-2.12.1.tar.gz; 2290 system = builtins.currentSystem; 2291 } 2292 #+end_example 2293 2294 All clear, except that buildInputs. However it's easier than any black 2295 magic you are thinking of at this moment. 2296 2297 Nix is able to convert a list to a string. It first converts the 2298 elements to strings, and then concatenates them separated by a space: 2299 2300 #+begin_example 2301 nix-repl> builtins.toString 123 2302 "123" 2303 nix-repl> builtins.toString [ 123 456 ] 2304 "123 456" 2305 #+end_example 2306 2307 Recall that derivations can be converted to a string, hence: 2308 2309 #+begin_example 2310 nix-repl> :l <nixpkgs> 2311 Added 3950 variables. 2312 nix-repl> builtins.toString gnugrep 2313 "/nix/store/g5gdylclfh6d224kqh9sja290pk186xd-gnugrep-2.14" 2314 nix-repl> builtins.toString [ gnugrep gnused ] 2315 "/nix/store/g5gdylclfh6d224kqh9sja290pk186xd-gnugrep-2.14 /nix/store/krgdc4sknzpw8iyk9p20lhqfd52kjmg0-gnused-4.2.2" 2316 #+end_example 2317 2318 Simple! The buildInputs variable is a string with out paths separated by 2319 space, perfect for bash usage in a for loop. 2320 2321 ** A more convenient derivation function 2322 We managed to write a builder that can be used for multiple autotools 2323 projects. But in the hello.nix expression we are specifying tools that 2324 are common to more projects; we don't want to pass them every time. 2325 2326 A natural approach would be to create a function that accepts an 2327 attribute set, similar to the one used by the derivation function, and 2328 merge it with another attribute set containing values common to many 2329 projects. 2330 2331 Create =autotools.nix=: 2332 2333 #+begin_example 2334 pkgs: attrs: 2335 let 2336 defaultAttrs = { 2337 builder = "${pkgs.bash}/bin/bash"; 2338 args = [ ./builder.sh ]; 2339 baseInputs = with pkgs; [ 2340 gnutar 2341 gzip 2342 gnumake 2343 gcc 2344 coreutils 2345 gawk 2346 gnused 2347 gnugrep 2348 binutils.bintools 2349 ]; 2350 buildInputs = [ ]; 2351 system = builtins.currentSystem; 2352 }; 2353 in 2354 derivation (defaultAttrs // attrs) 2355 #+end_example 2356 2357 Ok now we have to remember a little about 2358 [[#05-functions-and-imports.html][Nix functions]]. The whole nix 2359 expression of this =autotools.nix= file will evaluate to a function. 2360 This function accepts a parameter =pkgs=, then returns a function which 2361 accepts a parameter =attrs=. 2362 2363 The body of the function is simple, yet at first sight it might be hard 2364 to grasp: 2365 2366 1. First drop in the scope the magic =pkgs= attribute set. 2367 2368 2. Within a let expression we define a helper variable, =defaultAttrs=, 2369 which serves as a set of common attributes used in derivations. 2370 2371 3. Finally we create the derivation with that strange expression, 2372 (=defaultAttrs // attrs=). 2373 2374 The 2375 [[https://nix.dev/manual/nix/stable/language/operators.html#update][// 2376 operator]] is an operator between two sets. The result is the union of 2377 the two sets. In case of conflicts between attribute names, the value on 2378 the right set is preferred. 2379 2380 So we use =defaultAttrs= as base set, and add (or override) the 2381 attributes from =attrs=. 2382 2383 A couple of examples ought to be enough to clear out the behavior of the 2384 operator: 2385 2386 #+begin_example 2387 nix-repl> { a = "b"; } // { c = "d"; } 2388 { a = "b"; c = "d"; } 2389 nix-repl> { a = "b"; } // { a = "c"; } 2390 { a = "c"; } 2391 #+end_example 2392 2393 *Exercise:* Complete the new =builder.sh= by adding =$baseInputs= in the 2394 =for= loop together with =$buildInputs=. As you noticed, we passed that 2395 new variable in the derivation. Instead of merging buildInputs with the 2396 base ones, we prefer to preserve buildInputs as seen by the caller, so 2397 we keep them separated. Just a matter of choice. 2398 2399 Then we rewrite =hello.nix= as follows: 2400 2401 #+begin_example 2402 let 2403 pkgs = import <nixpkgs> { }; 2404 mkDerivation = import ./autotools.nix pkgs; 2405 in 2406 mkDerivation { 2407 name = "hello"; 2408 src = ./hello-2.12.1.tar.gz; 2409 } 2410 #+end_example 2411 2412 Finally! We got a very simple description of a package! Below are a 2413 couple of remarks that you may find useful as you're continuing to 2414 understand the nix language: 2415 2416 - We assigned to pkgs the import that we did in the previous expressions 2417 in the "with". Don't be afraid, it's that straightforward. 2418 2419 - The mkDerivation variable is a nice example of partial application, 2420 look at it as (=import ./autotools.nix=) =pkgs=. First we import the 2421 expression, then we apply the =pkgs= parameter. That will give us a 2422 function that accepts the attribute set =attrs=. 2423 2424 - We create the derivation specifying only name and src. If the project 2425 eventually needed other dependencies to be in PATH, then we would 2426 simply add those to buildInputs (not specified in hello.nix because 2427 empty). 2428 2429 Note we didn't use any other library. Special C flags may be needed to 2430 find include files of other libraries at compile time, and ld flags at 2431 link time. 2432 2433 ** Conclusion 2434 Nix gives us the bare metal tools for creating derivations, setting up a 2435 build environment and storing the result in the nix store. 2436 2437 Out of this pill we managed to create a generic builder for autotools 2438 projects, and a function =mkDerivation= that composes by default the 2439 common components used in autotools projects instead of repeating them 2440 in all the packages we would write. 2441 2442 We are familiarizing ourselves with the way a Nix system grows up: it's 2443 about creating and composing derivations with the Nix language. 2444 2445 Analogy: in C you create objects in the heap, and then you compose them 2446 inside new objects. Pointers are used to refer to other objects. 2447 2448 In Nix you create derivations stored in the nix store, and then you 2449 compose them by creating new derivations. Store paths are used to refer 2450 to other derivations. 2451 2452 ** Next pill 2453 ...we will talk a little about runtime dependencies. Is the GNU hello 2454 world package self-contained? What are its runtime dependencies? We only 2455 specified build dependencies by means of using other derivations in the 2456 "hello" derivation. 2457 2458 <<09-automatic-runtime-dependencies.html>> 2459 2460 * Automatic Runtime Dependencies 2461 Welcome to the 9th Nix pill. In the previous 2462 [[#08-generic-builders.html][8th pill]] we wrote a generic builder for 2463 autotools projects. We fed in build dependencies and a source tarball, 2464 and we received a Nix derivation as a result. 2465 2466 Today we stop by the GNU =hello= program to analyze build and runtime 2467 dependencies, and we enhance our builder to eliminate unnecessary 2468 runtime dependencies. 2469 2470 ** Build dependencies 2471 Let's start analyzing build dependencies for our GNU =hello= package: 2472 2473 #+begin_example 2474 $ nix-instantiate hello.nix 2475 /nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv 2476 $ nix-store -q --references /nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv 2477 /nix/store/0q6pfasdma4as22kyaknk4kwx4h58480-hello-2.10.tar.gz 2478 /nix/store/1zcs1y4n27lqs0gw4v038i303pb89rw6-coreutils-8.21.drv 2479 /nix/store/2h4b30hlfw4fhqx10wwi71mpim4wr877-gnused-4.2.2.drv 2480 /nix/store/39bgdjissw9gyi4y5j9wanf4dbjpbl07-gnutar-1.27.1.drv 2481 /nix/store/7qa70nay0if4x291rsjr7h9lfl6pl7b1-builder.sh 2482 /nix/store/g6a0shr58qvx2vi6815acgp9lnfh9yy8-gnugrep-2.14.drv 2483 /nix/store/jdggv3q1sb15140qdx0apvyrps41m4lr-bash-4.2-p45.drv 2484 /nix/store/pglhiyp1zdbmax4cglkpz98nspfgbnwr-gnumake-3.82.drv 2485 /nix/store/q9l257jn9lndbi3r9ksnvf4dr8cwxzk7-gawk-4.1.0.drv 2486 /nix/store/rgyrqxz1ilv90r01zxl0sq5nq0cq7v3v-binutils-2.23.1.drv 2487 /nix/store/qzxhby795niy6wlagfpbja27dgsz43xk-gcc-wrapper-4.8.3.drv 2488 /nix/store/sk590g7fv53m3zp0ycnxsc41snc2kdhp-gzip-1.6.drv 2489 #+end_example 2490 2491 It has precisely the derivations referenced in the =derivation= 2492 function; nothing more, nothing less. Of course, we may not use some of 2493 them at all. However, given that our generic =mkDerivation= function 2494 always pulls such dependencies (think of it like 2495 [[https://packages.debian.org/unstable/build-essential][build-essential]] 2496 from Debian), we will already have these packages in the nix store for 2497 any future packages that need them. 2498 2499 Why are we looking at =.drv= files? Because the =hello.drv= file is the 2500 representation of the build action that builds the =hello= out path. As 2501 such, it contains the input derivations needed before building =hello=. 2502 2503 ** Digression about NAR files 2504 The =NAR= format is the "Nix ARchive". This format was designed due to 2505 existing archive formats, such as =tar=, being insufficient. Nix 2506 benefits from deterministic build tools, but commonly used archivers 2507 lack this property: they add padding, they do not sort files, they add 2508 timestamps, and so on. This can result in directories containing 2509 bit-identical files turning into non-bit-identical archives, which leads 2510 to different hashes. 2511 2512 Thus the =NAR= format was developed as a simple, deterministic archive 2513 format. =NAR=s are used extensively within Nix, as we will see below. 2514 2515 For more rationale and implementation details behind =NAR= see 2516 [[http://nixos.org/~eelco/pubs/phd-thesis.pdf][Dolstra's PhD Thesis]]. 2517 2518 To create NAR archives from store paths, we can use =nix-store --dump= 2519 and =nix-store --restore=. 2520 2521 ** Runtime dependencies 2522 We now note that Nix automatically recognized build dependencies once 2523 our =derivation= call referred to them, but we never specified the 2524 runtime dependencies. 2525 2526 Nix handles runtime dependencies for us automatically. The technique it 2527 uses to do so may seem fragile at first glance, but it works so well 2528 that the NixOS operating system is built off of it. The underlying 2529 mechanism relies on the hash of the store paths. It proceeds in three 2530 steps: 2531 2532 1. Dump the derivation as a NAR. Recall that this is a serialization of 2533 the derivation output -- meaning this works fine whether the output 2534 is a single file or a directory. 2535 2536 2. For each build dependency =.drv= and its relative out path, search 2537 the contents of the NAR for this out path. 2538 2539 3. If the path is found, then it's a runtime dependency. 2540 2541 The snippet below shows the dependencies for =hello=. 2542 2543 #+begin_example 2544 $ nix-instantiate hello.nix 2545 /nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv 2546 $ nix-store -r /nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv 2547 /nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello 2548 $ nix-store -q --references /nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello 2549 /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19 2550 /nix/store/8jm0wksask7cpf85miyakihyfch1y21q-gcc-4.8.3 2551 /nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello 2552 #+end_example 2553 2554 We see that =glibc= and =gcc= are runtime dependencies. Intuitively, 2555 =gcc= shouldn't be in this list! Displaying the printable strings in the 2556 =hello= binary shows that the out path of =gcc= does indeed appear: 2557 2558 #+begin_example 2559 $ strings result/bin/hello|grep gcc 2560 /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib:/nix/store/8jm0wksask7cpf85miyakihyfch1y21q-gcc-4.8.3/lib64 2561 #+end_example 2562 2563 This is why Nix added =gcc=. But why is that path present in the first 2564 place? The answer is that it is the 2565 [[http://en.wikipedia.org/wiki/Rpath][ld rpath]]: the list of 2566 directories where libraries can be found at runtime. In other 2567 distributions, this is usually not abused. But in Nix, we have to refer 2568 to particular versions of libraries, and thus the rpath has an important 2569 role. 2570 2571 The build process adds the =gcc= lib path thinking it may be useful at 2572 runtime, but this isn't necessary. To address issues like these, Nix 2573 provides a tool called [[https://nixos.org/patchelf.html][patchelf]], 2574 which reduces the rpath to the paths that are actually used by the 2575 binary. 2576 2577 Even after reducing the rpath, the =hello= binary would still depend 2578 upon =gcc= because of some debugging information. This unnecessarily 2579 increases the size of our runtime dependencies. We'll explore how 2580 =strip= can help us with that in the next section. 2581 2582 ** Another phase in the builder 2583 We will add a new phase to our autotools builder. The builder has six 2584 phases already: 2585 2586 1. The "environment setup" phase 2587 2588 2. The "unpack phase": we unpack the sources in the current directory 2589 (remember, Nix changes to a temporary directory first) 2590 2591 3. The "change directory" phase, where we change source root to the 2592 directory that has been unpacked 2593 2594 4. The "configure" phase: =./configure= 2595 2596 5. The "build" phase: =make= 2597 2598 6. The "install" phase: =make install= 2599 2600 Now we will add a new phase after the installation phase, which we call 2601 the "fixup" phase. At the end of the =builder.sh=, we append: 2602 2603 #+begin_example 2604 find $out -type f -exec patchelf --shrink-rpath '{}' \; -exec strip '{}' \; 2>/dev/null 2605 #+end_example 2606 2607 That is, for each file we run =patchelf --shrink-rpath= and =strip=. 2608 Note that we used two new commands here, =find= and =patchelf=. These 2609 must be added to our derivation. 2610 2611 *Exercise:* Add =findutils= and =patchelf= to the =baseInputs= of 2612 =autotools.nix=. 2613 2614 Now, we rebuild =hello.nix=... 2615 2616 #+begin_example 2617 $ nix-build hello.nix 2618 [...] 2619 $ nix-store -q --references result 2620 /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19 2621 /nix/store/md4a3zv0ipqzsybhjb8ndjhhga1dj88x-hello 2622 #+end_example 2623 2624 and we see that =glibc= is a runtime dependency but =gcc= is not there 2625 anymore. This is exactly what we wanted. 2626 2627 The package is self-contained. This means that we can copy its closure 2628 onto another machine and we will be able to run it. Remember, only a 2629 very few components under the =/nix/store= are required to 2630 [[#02-install-on-your-running-system.html][run nix]]. The =hello= binary 2631 will use the exact version of =glibc= library and interpreter referred 2632 to in the binary, rather than the system one: 2633 2634 #+begin_example 2635 $ ldd result/bin/hello 2636 linux-vdso.so.1 (0x00007fff11294000) 2637 libc.so.6 => /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/libc.so.6 (0x00007f7ab7362000) 2638 /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/ld-linux-x86-64.so.2 (0x00007f7ab770f000) 2639 #+end_example 2640 2641 Of course, the executable will run fine as long as everything is under 2642 the =/nix/store= path. 2643 2644 ** Conclusion 2645 We saw some of the tools Nix provides, along with their features. In 2646 particular, we saw how Nix is able to compute runtime dependencies 2647 automatically. This is not limited to only shared libraries, but can 2648 also reference executables, scripts, Python libraries, and so forth. 2649 2650 Approaching builds in this way makes packages self-contained, ensuring 2651 (apart from data and configuration) that copying the runtime closure 2652 onto another machine is sufficient to run the program. This enables us 2653 to run programs without installation using =nix-shell=, and forms the 2654 basis for [[https://github.com/NixOS/nixops][reliable deployment in the 2655 cloud]]. 2656 2657 ** Next pill 2658 The next pill will introduce =nix-shell=. With =nix-build=, we've always 2659 built derivations from scratch: the source gets unpacked, configured, 2660 built, and installed. But this can take a long time for large packages. 2661 What if we want to apply some small changes and compile incrementally 2662 instead, yet still want to keep a self-contained environment similar to 2663 =nix-build=? =nix-shell= enables this. 2664 2665 <<10-developing-with-nix-shell.html>> 2666 2667 * Developing with =nix-shell= 2668 Welcome to the 10th Nix pill. In the previous 2669 [[#09-automatic-runtime-dependencies.html][9th pill]] we saw one of the 2670 powerful features of Nix: automatic discovery of runtime dependencies. 2671 We also finalized the GNU =hello= package. 2672 2673 In this pill, we will introduce the =nix-shell= tool and use it to hack 2674 on the GNU =hello= program. We will see how =nix-shell= gives us an 2675 isolated environment while we modify the source files of the project, 2676 similar to how =nix-build= gave us an isolated environment while 2677 building the derivation. 2678 2679 Finally, we will modify our builder to work more ergonomically with a 2680 =nix-shell=-focused workflow. 2681 2682 ** What is =nix-shell=? 2683 The 2684 [[https://nix.dev/manual/nix/stable/command-ref/nix-shell][nix-shell]] 2685 tool drops us in a shell after setting up the environment variables 2686 necessary to hack on a derivation. It does not build the derivation; it 2687 only serves as a preparation so that we can run the build steps 2688 manually. 2689 2690 Recall that in a nix environment, we don't have access to libraries or 2691 programs unless they have been installed with =nix-env=. However, 2692 installing libraries with =nix-env= is not good practice. We prefer to 2693 have isolated environments for development, which =nix-shell= provides 2694 for us. 2695 2696 We can call =nix-shell= on any Nix expression which returns a 2697 derivation, but the resulting =bash= shell's =PATH= does not have the 2698 utilities we want: 2699 2700 #+begin_example 2701 $ nix-shell hello.nix 2702 [nix-shell]$ make 2703 bash: make: command not found 2704 [nix-shell]$ echo $baseInputs 2705 /nix/store/jff4a6zqi0yrladx3kwy4v6844s3swpc-gnutar-1.27.1 [...] 2706 #+end_example 2707 2708 This shell is rather useless. It would be reasonable to expect that the 2709 GNU =hello= build inputs are available in =PATH=, including GNU =make=, 2710 but this is not the case. 2711 2712 However, we do have the environment variables that we set in the 2713 derivation, like =$baseInputs=, =$buildInputs=, =$src=, and so on. 2714 2715 This means that we can =source= our =builder.sh=, and it will build the 2716 derivation. You may get an error in the installation phase, because your 2717 user may not have the permission to write to =/nix/store=: 2718 2719 #+begin_example 2720 [nix-shell]$ source builder.sh 2721 ... 2722 #+end_example 2723 2724 The derivation didn't install, but it did build. Note the following: 2725 2726 - We sourced =builder.sh= and it ran all of the build steps, including 2727 setting up the =PATH= for us. 2728 2729 - The working directory is no longer a temp directory created by 2730 =nix-build=, but is instead the directory in which we entered the 2731 shell. Therefore, =hello-2.10= has been unpacked in the current 2732 directory. 2733 2734 We are able to =cd= into =hello-2.10= and type =make=, because =make= is 2735 now available. 2736 2737 The take-away is that =nix-shell= drops us in a shell with the same (or 2738 very similar) environment used to run the builder. 2739 2740 ** A builder for nix-shell 2741 The previous steps require some manual commands to be run and are not 2742 optimized for a workflow centered on =nix-shell=. We will now improve 2743 our builder to be more =nix-shell= friendly. 2744 2745 There are a few things that we would like to change. 2746 2747 First, when we =source=d the =builder.sh= file, we obtained the file in 2748 the current directory. What we really wanted was the =builder.sh= that 2749 is stored in the nix store, as this is the file that would be used by 2750 =nix-build=. To achieve this, the correct technique is to pass an 2751 environment variable through the derivation. (Note that =$builder= is 2752 already defined, but it points to the bash executable rather than our 2753 =builder.sh=. Our =builder.sh= is passed as an argument to bash.) 2754 2755 Second, we don't want to run the whole builder: we only want to setup 2756 the necessary environment for manually building the project. Thus, we 2757 can break =builder.sh= into two files: a =setup.sh= for setting up the 2758 environment, and the real =builder.sh= that =nix-build= expects. 2759 2760 During our refactoring, we will wrap the build phases in functions to 2761 give more structure to our design. Additionally, we'll move the =set -e= 2762 to the builder file instead of the setup file. The =set -e= is annoying 2763 in =nix-shell=, as it will terminate the shell if an error is 2764 encountered (such as a mistyped command.) 2765 2766 Here is our modified =autotools.nix=. Noteworthy is the 2767 =setup = ./setup.sh;= attribute in the derivation, which adds =setup.sh= 2768 to the nix store and correspondingly adds a =$setup= environment 2769 variable in the builder. 2770 2771 #+begin_example 2772 pkgs: attrs: 2773 let 2774 defaultAttrs = { 2775 builder = "${pkgs.bash}/bin/bash"; 2776 args = [ ./builder.sh ]; 2777 setup = ./setup.sh; 2778 baseInputs = with pkgs; [ 2779 gnutar 2780 gzip 2781 gnumake 2782 gcc 2783 coreutils 2784 gawk 2785 gnused 2786 gnugrep 2787 binutils.bintools 2788 patchelf 2789 findutils 2790 ]; 2791 buildInputs = [ ]; 2792 system = builtins.currentSystem; 2793 }; 2794 in 2795 derivation (defaultAttrs // attrs) 2796 #+end_example 2797 2798 Thanks to that, we can split =builder.sh= into =setup.sh= and 2799 =builder.sh=. What =builder.sh= does is =source= =$setup= and call the 2800 =genericBuild= function. Everything else is just some changes to the 2801 bash script. 2802 2803 Here is the modified =builder.sh=: 2804 2805 #+begin_src sh 2806 set -e 2807 source $setup 2808 genericBuild 2809 #+end_src 2810 2811 Here is the newly added =setup.sh=: 2812 2813 #+begin_src sh 2814 unset PATH 2815 for p in $baseInputs $buildInputs; do 2816 export PATH=$p/bin${PATH:+:}$PATH 2817 done 2818 2819 function unpackPhase() { 2820 tar -xzf $src 2821 2822 for d in *; do 2823 if [ -d "$d" ]; then 2824 cd "$d" 2825 break 2826 fi 2827 done 2828 } 2829 2830 function configurePhase() { 2831 ./configure --prefix=$out 2832 } 2833 2834 function buildPhase() { 2835 make 2836 } 2837 2838 function installPhase() { 2839 make install 2840 } 2841 2842 function fixupPhase() { 2843 find $out -type f -exec patchelf --shrink-rpath '{}' \; -exec strip '{}' \; 2>/dev/null 2844 } 2845 2846 function genericBuild() { 2847 unpackPhase 2848 configurePhase 2849 buildPhase 2850 installPhase 2851 fixupPhase 2852 } 2853 #+end_src 2854 2855 Finally, here is =hello.nix=: 2856 2857 #+begin_example 2858 let 2859 pkgs = import <nixpkgs> { }; 2860 mkDerivation = import ./autotools.nix pkgs; 2861 in 2862 mkDerivation { 2863 name = "hello"; 2864 src = ./hello-2.12.1.tar.gz; 2865 } 2866 #+end_example 2867 2868 Now back to nix-shell: 2869 2870 #+begin_example 2871 $ nix-shell hello.nix 2872 [nix-shell]$ source $setup 2873 [nix-shell]$ 2874 #+end_example 2875 2876 Now, for example, you can run =unpackPhase= which unpacks =$src= and 2877 enters the directory. And you can run commands like =./configure=, 2878 =make=, and so forth manually, or run phases with their respective 2879 functions. 2880 2881 The process is that straightforward. =nix-shell= builds the =.drv= file 2882 and its input dependencies, then drops into a shell by setting up the 2883 environment variables necessary to build the =.drv=. In particular, the 2884 environment variables in the shell match those passed to the 2885 =derivation= function. 2886 2887 ** Conclusion 2888 With =nix-shell= we are able to drop into an isolated environment 2889 suitable for developing a project. This environment provides the 2890 necessary dependencies for the development shell, similar to how 2891 =nix-build= provides the necessary dependencies to a builder. 2892 Additionally, we can build and debug the project manually, executing 2893 step-by-step like we would in any other operating system. Note that we 2894 never installed tools such =gcc= or =make= system-wide; these tools and 2895 libraries are isolated and available per-build. 2896 2897 ** Next pill 2898 In the next pill, we will clean up the nix store. We have written and 2899 built derivations which add to the nix store, but until now we haven't 2900 worried about cleaning up the used space in the store. 2901 2902 <<11-garbage-collector.html>> 2903 2904 * The Garbage Collector 2905 :PROPERTIES: 2906 :CUSTOM_ID: 11-garbage-collector.html#garbage-collector 2907 :END: 2908 Welcome to the 11th Nix pill. In the previous 2909 [[#10-developing-with-nix-shell.html][10th pill]], we drew a parallel 2910 between the isolated build environment provided by =nix-build= and the 2911 isolated development shell provided by =nix-shell=. Using =nix-shell= 2912 allowed us to debug, modify, and manually build software using an 2913 environment that is almost identical to the one provided by =nix-build=. 2914 2915 Today, we will stop focusing on packaging and instead look at a critical 2916 component of Nix: the garbage collector. When we use Nix tools, we are 2917 often building derivations. This includes =.drv= files as well as out 2918 paths. These artifacts go in the Nix store and take up space in our 2919 storage. Eventually we may wish to free up some space by removing 2920 derivations we no longer need. This is the focus of the 11th pill. By 2921 default, Nix takes a relatively conservative approach when automatically 2922 deciding which derivations are "needed". In this pill, we will also see 2923 a technique to conduct more destructive upgrade and deletion operations. 2924 2925 ** How does garbage collection work? 2926 Programming languages with garbage collectors use the concept of a set 2927 of "garbage collector (or 'GC') roots" to keep track of "live" objects. 2928 A GC root is an object that is always considered "live" (unless 2929 explicitly removed as GC root). The garbage collection process starts 2930 from the GC roots and proceeds by recursively marking object references 2931 as "live". All other objects can be collected and deleted. 2932 2933 Instead of objects, Nix's garbage collection operates on store paths, 2934 [[https://nix.dev/manual/nix/stable/package-management/garbage-collector-roots][with 2935 the GC roots themselves being store paths]]. . This approach is much 2936 more principled than traditional package managers such as =dpkg= or 2937 =rpm=, which may leave around unused packages or dangling files. 2938 2939 The implementation is very simple and transparent to the user. The 2940 primary GC roots are stored under =/nix/var/nix/gcroots=. If there is a 2941 symlink to a store path, then the linked store path is a GC root. 2942 2943 Nix allows this directory to have subdirectories: it will simply 2944 recursively traverse the subdirectories in search of symlinks to store 2945 paths. When a symlink is encountered, its target is added to the list of 2946 live store paths. 2947 2948 In summary, Nix maintains a list of GC roots. These roots can then be 2949 used to compute a list of all live store paths. Any other store paths 2950 are considered dead. Deleting these paths is now straightforward. Nix 2951 first moves dead store paths to =/nix/store/trash=, which is an atomic 2952 operation. Afterwards, the trash is emptied. 2953 2954 ** Playing with the GC 2955 Before we begin we first run the 2956 [[https://nix.dev/manual/nix/stable/command-ref/nix-collect-garbage][nix 2957 garbage collector]] so that we have a clean setup for our experiments: 2958 2959 #+begin_example 2960 $ nix-collect-garbage 2961 finding garbage collector roots... 2962 [...] 2963 deleting unused links... 2964 note: currently hard linking saves -0.00 MiB 2965 1169 store paths deleted, 228.43 MiB freed 2966 #+end_example 2967 2968 If we run the garbage collector again it won't find anything new to 2969 delete, as we expect. After running the garbage collector, the nix store 2970 only contains paths with references from the GC roots. 2971 2972 We now install a new program, =bsd-games=, inspect its store path, and 2973 examine its GC root. The =nix-store -q --roots= command is used to query 2974 the GC roots that refer to a given derivation. In this case, our current 2975 user environment refers to =bsd-games=: 2976 2977 #+begin_example 2978 $ nix-env -iA nixpkgs.bsdgames 2979 $ readlink -f `which fortune` 2980 /nix/store/b3lxx3d3ggxcggvjw5n0m1ya1gcrmbyn-bsd-games-2.17/bin/fortune 2981 $ nix-store -q --roots `which fortune` 2982 /nix/var/nix/profiles/default-9-link 2983 $ nix-env --list-generations 2984 [...] 2985 9 2014-08-20 12:44:14 (current) 2986 #+end_example 2987 2988 Now we remove it and run the garbage collector, and note that 2989 =bsd-games= is still in the nix store: 2990 2991 #+begin_example 2992 $ nix-env -e bsd-games 2993 uninstalling `bsd-games-2.17' 2994 $ nix-collect-garbage 2995 [...] 2996 $ ls /nix/store/b3lxx3d3ggxcggvjw5n0m1ya1gcrmbyn-bsd-games-2.17 2997 bin share 2998 #+end_example 2999 3000 The old generation is still in the nix store because it is a GC root. As 3001 we will see below, all profiles and their generations are automatically 3002 GC roots. 3003 3004 Removing a GC root is simple. In our case, we delete the generation that 3005 refers to =bsd-games=, run the garbage collector, and note that 3006 =bsd-games= is no longer in the nix store: 3007 3008 #+begin_example 3009 $ rm /nix/var/nix/profiles/default-9-link 3010 $ nix-env --list-generations 3011 [...] 3012 8 2014-07-28 10:23:24 3013 10 2014-08-20 12:47:16 (current) 3014 $ nix-collect-garbage 3015 [...] 3016 $ ls /nix/store/b3lxx3d3ggxcggvjw5n0m1ya1gcrmbyn-bsd-games-2.17 3017 ls: cannot access /nix/store/b3lxx3d3ggxcggvjw5n0m1ya1gcrmbyn-bsd-games-2.17: No such file or directory 3018 #+end_example 3019 3020 Note: =nix-env --list-generations= does not rely on any particular 3021 metadata. It is able to list generations based solely on the file names 3022 under the profiles directory. 3023 3024 Note that we removed the link from =/nix/var/nix/profiles=, not from 3025 =/nix/var/nix/gcroots=. In addition to the latter, Nix treats 3026 =/nix/var/nix/profiles= as a GC root. This is useful because it means 3027 that any profile and its generations are GC roots. Other paths are 3028 considered GC roots as well; for example, =/run/booted-system= on NixOS. 3029 The command =nix-store --gc --print-roots= prints all paths considered 3030 as GC roots when running the garbage collector. 3031 3032 ** Indirect roots 3033 Recall that building the GNU =hello= package with =nix-build= produces a 3034 =result= symlink in the current directory. Despite the garbage 3035 collection done above, the =hello= program is still working. Therefore, 3036 it has not been garbage collected. Since there is no other derivation 3037 that depends upon the GNU =hello= package, it must be a GC root. 3038 3039 In fact, =nix-build= automatically adds the =result= symlink as a GC 3040 root. Note that this is not the built derivation, but the symlink 3041 itself. These GC roots are added under =/nix/var/nix/gcroots/auto=. 3042 3043 #+begin_example 3044 $ ls -l /nix/var/nix/gcroots/auto/ 3045 total 8 3046 drwxr-xr-x 2 nix nix 4096 Aug 20 10:24 ./ 3047 drwxr-xr-x 3 nix nix 4096 Jul 24 10:38 ../ 3048 lrwxrwxrwx 1 nix nix 16 Jul 31 10:51 xlgz5x2ppa0m72z5qfc78b8wlciwvgiz -> /home/nix/result/ 3049 #+end_example 3050 3051 The name of the GC root symlink is not important to us at this time. 3052 What is important is that such a symlink exists and points to 3053 =/home/nix/result=. This is called an *indirect GC root*. A GC root is 3054 considered indirect if its specification is outside of 3055 =/nix/var/nix/gcroots=. In this case, this means that the target of the 3056 =result= symlink will not be garbage collected. 3057 3058 To remove a derivation considered "live" by an indirect GC root, there 3059 are two possibilities: 3060 3061 - Remove the indirect GC root from =/nix/var/nix/gcroots/auto=. 3062 3063 - Remove the =result= symlink. 3064 3065 In the first case, the derivation will be deleted from the nix store 3066 during garbage collection, and =result= becomes a dangling symlink. In 3067 the second case, the derivation is removed as well as the indirect root 3068 in =/nix/var/nix/gcroots/auto=. 3069 3070 Running =nix-collect-garbage= after deleting the GC root or the indirect 3071 GC root will remove the derivation from the store. 3072 3073 ** Cleanup everything 3074 The main source of software duplication in the nix store comes from GC 3075 roots, due to =nix-build= and profile generations. Running =nix-build= 3076 results in a GC root for the build that refers to a specific version of 3077 specific libraries, such as =glibc=. After an upgrade, we must delete 3078 the previous build if we want the garbage collector to remove the 3079 corresponding derivation, as well as if we want old dependencies cleaned 3080 up. 3081 3082 The same holds for profiles. Manipulating the =nix-env= profile will 3083 create further generations. Old generations refer to old software, thus 3084 increasing duplication in the nix store after an upgrade. 3085 3086 Other systems typically "forget" everything about their previous state 3087 after an upgrade. With Nix, we can perform this type of upgrade (having 3088 Nix remove all old derivations, including old generations), but we do so 3089 manually. There are four steps to doing this: 3090 3091 - First, we download a new version of the nixpkgs channel, which holds 3092 the description of all the software. This is done via 3093 =nix-channel --update=. 3094 3095 - Then we upgrade our installed packages with =nix-env -u=. This will 3096 bring us into a new generation with updated software. 3097 3098 - Then we remove all the indirect roots generated by =nix-build=: 3099 beware, as this will result in dangling symlinks. A smarter strategy 3100 would also remove the target of those symlinks. 3101 3102 - Finally, the =-d= option of =nix-collect-garbage= is used to delete 3103 old generations of all profiles, then collect garbage. After this, you 3104 lose the ability to rollback to any previous generation. It is 3105 important to ensure the new generation is working well before running 3106 this command. 3107 3108 The four steps are shown below: 3109 3110 #+begin_example 3111 $ nix-channel --update 3112 $ nix-env -u --always 3113 $ rm /nix/var/nix/gcroots/auto/* 3114 $ nix-collect-garbage -d 3115 #+end_example 3116 3117 ** Conclusion 3118 Garbage collection in Nix is a powerful mechanism to clean up your 3119 system. The =nix-store= commands allow us to know why a certain 3120 derivation is present in the nix store, and whether or not it is 3121 eligible for garbage collection. We also saw how to conduct more 3122 destructive deletion and upgrade operations. 3123 3124 ** Next pill 3125 In the next pill, we will package another project and introduce the 3126 "inputs" design pattern. We've only played with a single derivation 3127 until now; however we'd like to start organizing a small repository of 3128 software. The "inputs" pattern is widely used in nixpkgs; it allows us 3129 to decouple derivations from the repository itself and increase 3130 customization opportunities. 3131 3132 <<12-inputs-design-pattern.html>> 3133 3134 * Package Repositories and the Inputs Design Pattern 3135 :PROPERTIES: 3136 :CUSTOM_ID: 12-inputs-design-pattern.html#inputs-design-pattern 3137 :END: 3138 Welcome to the 12th Nix pill. In the previous 3139 [[#11-garbage-collector.html][11th pill]], we stopped packaging and 3140 cleaned up the system with the garbage collector. 3141 3142 This time, we will resume packaging and improve different aspects of it. 3143 We will also demonstrate how to create a repository of multiple 3144 packages. 3145 3146 ** Repositories in Nix 3147 Package repositories in Nix arose naturally from the need to organize 3148 packages. There is no preset directory structure or packaging policy 3149 prescribed by Nix itself; Nix, as a full, functional programming 3150 language, is powerful enough to support multiple different repository 3151 formats. 3152 3153 Over time, the =nixpkgs= repository evolved a particular structure. This 3154 structure reflects the history of Nix as well as the design patterns 3155 adopted by its users as useful tools in building and organizing 3156 packages. Below, we will examine some of these patterns in detail. 3157 3158 ** The single repository pattern 3159 Different operating system distributions have different opinions about 3160 how package repositories should be organized. Systems like Debian 3161 scatter packages in several small repositories (which tends to make 3162 tracking interdependent changes more difficult, and hinders 3163 contributions to the repositories), while systems like Gentoo put all 3164 package descriptions in a single repository. 3165 3166 Nix follows the "single repository" pattern by placing all descriptions 3167 of all packages into [[https://github.com/NixOS/nixpkgs][nixpkgs]]. This 3168 approach has proven natural and attractive for new contributions. 3169 3170 For the rest of this pill, we will adopt the single repository pattern. 3171 The natural implementation in Nix is to create a top-level Nix 3172 expression, followed by one expression for each package. The top-level 3173 expression imports and combines all package expressions in an attribute 3174 set mapping names to packages. 3175 3176 In some programming languages, such an approach -- including every 3177 possible package description in a single data structure -- would be 3178 untenable due to the language needing to load the entire data structure 3179 into memory before operating on it. Nix, however, is a lazy language and 3180 only evaluates what is needed. 3181 3182 ** Packaging =graphviz= 3183 We have already packaged GNU =hello=. Next, we will package a 3184 graph-drawing program called =graphviz= so that we can create a 3185 repository containing multiple packages. The =graphviz= package was 3186 selected because it uses the standard autotools build system and 3187 requires no patching. It also has optional dependencies, which will give 3188 us an opportunity to illustrate a technique to configure builds to a 3189 particular situation. 3190 3191 First, we download =graphviz= from 3192 [[https://gitlab.com/api/v4/projects/4207231/packages/generic/graphviz-releases/2.49.3/graphviz-2.49.3.tar.gz][gitlab]]. 3193 The =graphviz.nix= expression is straightforward: 3194 3195 #+begin_example 3196 let 3197 pkgs = import <nixpkgs> { }; 3198 mkDerivation = import ./autotools.nix pkgs; 3199 in 3200 mkDerivation { 3201 name = "graphviz"; 3202 src = ./graphviz-2.49.3.tar.gz; 3203 } 3204 #+end_example 3205 3206 If we build the project with =nix-build graphviz.nix=, we will get 3207 runnable binaries under =result/bin=. Notice how we reused the same 3208 =autotools.nix= of =hello.nix.= 3209 3210 By default, =graphviz= does not compile with the ability to produce 3211 =png= files. Thus, the derivation above will build a binary supporting 3212 only the native output formats, as we see below: 3213 3214 #+begin_example 3215 $ echo 'graph test { a -- b }'|result/bin/dot -Tpng -o test.png 3216 Format: "png" not recognized. Use one of: canon cmap [...] 3217 #+end_example 3218 3219 If we want to produce a =png= file with =graphviz=, we must add it to 3220 our derivation. The place to do so is in =autotools.nix=, where we 3221 created a =buildInputs= variable that gets concatenated to =baseInputs=. 3222 This is the exact reason for this variable: to allow users of 3223 =autotools.nix= to add additional inputs from package expressions. 3224 3225 Version 2.49 of =graphviz= has several plugins to output =png=. For 3226 simplicity, we will use =libgd=. 3227 3228 ** Passing library information to =pkg-config= via environment variables 3229 The =graphviz= configuration script uses =pkg-config= to specify which 3230 flags are passed to the compiler. Since there is no global location for 3231 libraries, we need to tell =pkg-config= where to find its description 3232 files, which tell the configuration script where to find headers and 3233 libraries. 3234 3235 In classic POSIX systems, =pkg-config= just finds the =.pc= files of all 3236 installed libraries in system folders like =/usr/lib/pkgconfig=. 3237 However, these files are not present in the isolated environments 3238 presented to Nix. 3239 3240 As an alternative, we can inform =pkg-config= about the location of 3241 libraries via the =PKG_CONFIG_PATH= environment variable. We can 3242 populate this environment variable using the same trick we used for 3243 =PATH=: automatically filling the variables from =buildInputs=. This is 3244 the relevant snippet of =setup.sh=: 3245 3246 #+begin_src sh 3247 for p in $baseInputs $buildInputs; do 3248 if [ -d $p/bin ]; then 3249 export PATH="$p/bin${PATH:+:}$PATH" 3250 fi 3251 if [ -d $p/lib/pkgconfig ]; then 3252 export PKG_CONFIG_PATH="$p/lib/pkgconfig${PKG_CONFIG_PATH:+:}$PKG_CONFIG_PATH" 3253 fi 3254 done 3255 #+end_src 3256 3257 Now if we add derivations to =buildInputs=, their =lib/pkgconfig= and 3258 =bin= paths are automatically added in =setup.sh=. 3259 3260 ** Completing graphviz with =gd= 3261 Below, we finish the expression for =graphviz= with =gd= support. Note 3262 the use of the =with= expression in =buildInputs= to avoid repeating 3263 =pkgs=: 3264 3265 #+begin_example 3266 let 3267 pkgs = import <nixpkgs> { }; 3268 mkDerivation = import ./autotools.nix pkgs; 3269 in 3270 mkDerivation { 3271 name = "graphviz"; 3272 src = ./graphviz-2.49.3.tar.gz; 3273 buildInputs = with pkgs; [ 3274 pkg-config 3275 (pkgs.lib.getLib gd) 3276 (pkgs.lib.getDev gd) 3277 ]; 3278 } 3279 #+end_example 3280 3281 We add =pkg-config= to the derivation to make this tool available for 3282 the configure script. As =gd= is a package with 3283 [[https://nixos.org/manual/nixpkgs/stable/#sec-multiple-outputs-][split 3284 outputs]], we need to add both the library and development outputs. 3285 3286 After building, =graphviz= can now create =png=s. 3287 3288 ** The repository expression 3289 Now that we have two packages, we want to combine them into a single 3290 repository. To do so, we'll mimic what =nixpkgs= does: we will create a 3291 single attribute set containing derivations. This attribute set can then 3292 be imported, and derivations can be selected by accessing the top-level 3293 attribute set. 3294 3295 Using this technique we are able to abstract from the file names. 3296 Instead of referring to a package by =REPO/some/sub/dir/package.nix=, 3297 this technique allows us to select a derivation as 3298 =importedRepo.package= (or =pkgs.package= in our examples). 3299 3300 To begin, create a default.nix in the current directory: 3301 3302 #+begin_example 3303 { 3304 hello = import ./hello.nix; 3305 graphviz = import ./graphviz.nix; 3306 } 3307 #+end_example 3308 3309 This file is ready to use with =nix repl=: 3310 3311 #+begin_example 3312 $ nix repl 3313 nix-repl> :l default.nix 3314 Added 2 variables. 3315 nix-repl> hello 3316 «derivation /nix/store/dkib02g54fpdqgpskswgp6m7bd7mgx89-hello.drv» 3317 nix-repl> graphviz 3318 «derivation /nix/store/zqv520v9mk13is0w980c91z7q1vkhhil-graphviz.drv» 3319 #+end_example 3320 3321 With =nix-build=, we can pass the -A option to access an attribute of 3322 the set from the given =.nix= expression: 3323 3324 #+begin_example 3325 $ nix-build default.nix -A hello 3326 [...] 3327 $ result/bin/hello 3328 Hello, world! 3329 #+end_example 3330 3331 The =default.nix= file is special. When a directory contains a 3332 =default.nix= file, it is used as the implicit nix expression of the 3333 directory. This, for example, allows us to run =nix-build -A hello= 3334 without specifying =default.nix= explicitly. 3335 3336 We can now use =nix-env= to install the package into our user 3337 environment: 3338 3339 #+begin_example 3340 $ nix-env -f . -iA graphviz 3341 [...] 3342 $ dot -V 3343 #+end_example 3344 3345 Taking a closer look at the above command, we see the following options: 3346 3347 - The -f option is used to specify the expression to use. In this case, 3348 the expression is the =./default.nix= of the current directory. 3349 3350 - The -i option stands for "installation". 3351 3352 - The -A is the same as above for =nix-build=. 3353 3354 We reproduced the very basic behavior of =nixpkgs=: combining multiple 3355 derivations into a single, top-level attribute set. 3356 3357 ** The inputs pattern 3358 The approach we've taken so far has a few problems: 3359 3360 - First, =hello.nix= and =graphviz.nix= are dependent on =nixpkgs=, 3361 which they import directly. A better approach would be to pass in 3362 =nixpkgs= as an argument, as we did in =autotools.nix=. 3363 3364 - Second, we don't have a straightforward way to compile different 3365 variants of the same software, such as =graphviz= with or without 3366 =libgd= support. 3367 3368 - Third, we don't have a way to test =graphviz= with a particular 3369 =libgd= version. 3370 3371 Until now, our approach to addressing the above problems has been 3372 inadequate and required changing the nix expression to match our needs. 3373 With the =inputs= pattern, we provide another answer: let the user 3374 change the =inputs= of the expression. 3375 3376 When we talk about "the inputs of an expression", we are referring to 3377 the set of derivations needed to build that expression. In this case: 3378 3379 - =mkDerivation= from =autotools=. Recall that =mkDerivation= has an 3380 implicit dependency on the toolchain. 3381 3382 - =libgd= and its dependencies. 3383 3384 The =./src= directory is also an input, but we wouldn't change the 3385 source from the caller. In =nixpkgs= we prefer to write another 3386 expression for version bumps (e.g. because patches or different inputs 3387 are needed). 3388 3389 Our goal is to make package expressions independent of the repository. 3390 To achieve this, we use functions to declare inputs for a derivation. 3391 For example, with =graphviz.nix=, we make the following changes to make 3392 the derivation independent of the repository and customizable: 3393 3394 #+begin_example 3395 { mkDerivation, lib, gdSupport ? true, gd, pkg-config }: 3396 3397 mkDerivation { 3398 name = "graphviz"; 3399 src = ./graphviz-2.49.3.tar.gz; 3400 buildInputs = 3401 if gdSupport 3402 then [ 3403 pkg-config 3404 (lib.getLib gd) 3405 (lib.getDev gd) 3406 ] 3407 else []; 3408 } 3409 #+end_example 3410 3411 Recall that "={...}: ...=" is the syntax for defining functions 3412 accepting an attribute set as argument; the above snippet just defines a 3413 function. 3414 3415 We made =gd= and its dependencies optional. If =gdSupport= is true 3416 (which it is by default), we will fill =buildInputs= and =graphviz= will 3417 be built with =gd= support. Otherwise, if an attribute set is passed 3418 with =gdSupport = false;=, the build will be completed without =gd= 3419 support. 3420 3421 Going back to =default.nix=, we modify our expression to utilize the 3422 inputs pattern: 3423 3424 #+begin_example 3425 let 3426 pkgs = import <nixpkgs> { }; 3427 mkDerivation = import ./autotools.nix pkgs; 3428 in 3429 with pkgs; 3430 { 3431 hello = import ./hello.nix { inherit mkDerivation; }; 3432 graphviz = import ./graphviz.nix { 3433 inherit 3434 mkDerivation 3435 lib 3436 gd 3437 pkg-config 3438 ; 3439 }; 3440 graphvizCore = import ./graphviz.nix { 3441 inherit 3442 mkDerivation 3443 lib 3444 gd 3445 pkg-config 3446 ; 3447 gdSupport = false; 3448 }; 3449 } 3450 #+end_example 3451 3452 We factorized the import of =nixpkgs= and =mkDerivation=, and also added 3453 a variant of =graphviz= with =gd= support disabled. The result is that 3454 both =hello.nix= (left as an exercise for the reader) and =graphviz.nix= 3455 are independent of the repository and customizable by passing specific 3456 inputs. 3457 3458 If we wanted to build =graphviz= with a specific version of =gd=, it 3459 would suffice to pass =gd = ...;=. 3460 3461 If we wanted to change the toolchain, we would simply pass a different 3462 =mkDerivation= function. 3463 3464 Let's take a closer look at the snippet and dissect the syntax: 3465 3466 - The entire expression in =default.nix= returns an attribute set with 3467 the keys =hello=, =graphviz=, and =graphvizCore=. 3468 3469 - With "=let=", we define some local variables. 3470 3471 - We bring =pkgs= into the scope when defining the package set. This 3472 saves us from having to type =pkgs=" repeatedly. 3473 3474 - We import =hello.nix= and =graphviz.nix=, which each return a 3475 function. We call the functions with a set of inputs to get back the 3476 derivation. 3477 3478 - The "=inherit x=" syntax is equivalent to "=x = x=". This means that 3479 the "=inherit gd=" here, combined with the above "=with pkgs;=", is 3480 equivalent to "=gd = pkgs.gd=". 3481 3482 The entire repository of this can be found at the 3483 [[https://gist.github.com/tfc/ca800a444b029e85a14e530c25f8e872][pill 3484 12]] gist. 3485 3486 ** Conclusion 3487 The "=inputs=" pattern allows our expressions to be easily customizable 3488 through a set of arguments. These arguments could be flags, derivations, 3489 or any other customizations enabled by the nix language. Our package 3490 expressions are simply functions: there is no extra magic present. 3491 3492 The "=inputs=" pattern also makes the expressions independent of the 3493 repository. Given that we pass all needed information through arguments, 3494 it is possible to use these expressions in any other context. 3495 3496 ** Next pill 3497 In the next pill, we will talk about the "=callPackage=" design pattern. 3498 This removes the tedium of specifying the names of the inputs twice: 3499 once in the top-level =default.nix=, and once in the package expression. 3500 With =callPackage=, we will implicitly pass the necessary inputs from 3501 the top-level expression. 3502 3503 <<13-callpackage-design-pattern.html>> 3504 3505 * Callpackage Design Pattern 3506 Welcome to the 13th Nix pill. In the previous 3507 [[#12-inputs-design-pattern.html][12th pill]], we introduced the first 3508 basic design pattern for organizing a repository of software. In 3509 addition, we packaged =graphviz= so that we had two packages to bundle 3510 into an example repository. 3511 3512 The next design pattern we will examine is called the =callPackage= 3513 pattern. This technique is extensively used in 3514 [[https://github.com/NixOS/nixpkgs][nixpkgs]], and it's the current de 3515 facto standard for importing packages in a repository. Its purpose is to 3516 reduce the duplication of identifiers between package derivation inputs 3517 and repository derivations. 3518 3519 ** The callPackage convenience 3520 In the previous pill, we demonstrated how the =inputs= pattern decouples 3521 packages from the repository. This allowed us to manually pass the 3522 inputs to the derivation; the derivation declares its inputs, and the 3523 caller passes the arguments. 3524 3525 However, as with usual programming languages, there is some duplication 3526 of work: we declare parameter names and then we pass arguments, 3527 typically with the same name. For example, if we define a package 3528 derivation using the =inputs= pattern such as: 3529 3530 #+begin_example 3531 { input1, input2, ... }: 3532 ... 3533 #+end_example 3534 3535 We would likely want to bundle that package derivation into a repository 3536 via an attribute set defined as something like: 3537 3538 #+begin_example 3539 rec { 3540 lib1 = import package1.nix { inherit input1 input2; }; 3541 program2 = import package2.nix { inherit inputX inputY lib1; }; 3542 } 3543 #+end_example 3544 3545 There are two things to note. First, that inputs often have the same 3546 name as attributes in the repository itself. Second, that (due to the 3547 =rec= keyword), the inputs to a package derivation may be other packages 3548 in the repository itself. 3549 3550 Rather than passing the inputs twice, we would prefer to pass those 3551 inputs from the repository automatically and allow for manually 3552 overriding defaults. 3553 3554 To achieve this, we will define a =callPackage= function with the 3555 following calling convention: 3556 3557 #+begin_example 3558 { 3559 lib1 = callPackage package1.nix { }; 3560 program2 = callPackage package2.nix { someoverride = overriddenDerivation; }; 3561 } 3562 #+end_example 3563 3564 We want =callPackage= to be a function of two arguments, with the 3565 following behavior: 3566 3567 - Import the given expression contained in the file of the first 3568 argument, and return a function. This function returns a package 3569 derivation that uses the inputs pattern. 3570 3571 - Determine the name of the arguments to the function (i.e., the names 3572 of the inputs to the package derivation). 3573 3574 - Pass default arguments from the repository set, and let us override 3575 those arguments if we wish to customize the package derivation. 3576 3577 ** Implementing =callPackage= 3578 In this section, we will build up the =callPackages= pattern from 3579 scratch. To start, we need a way to obtain the argument names of a 3580 function (in this case, the function that takes "inputs" and produces a 3581 package derivation) at runtime. This is because we want to automatically 3582 pass such arguments. 3583 3584 Nix provides a builtin function to do this: 3585 3586 #+begin_example 3587 nix-repl> add = { a ? 3, b }: a+b 3588 nix-repl> builtins.functionArgs add 3589 { a = true; b = false; } 3590 #+end_example 3591 3592 In addition to returning the argument names, the attribute set returned 3593 by =functionArgs= indicates whether or not the argument has a default 3594 value. For our purposes, we are only interested in the argument names; 3595 we do not care about the default values right now. 3596 3597 The next step is to make =callPackage= automatically pass inputs to our 3598 package derivations based on the argument names we've just obtained with 3599 =functionArgs=. 3600 3601 To do this, we need two things: 3602 3603 - A package repository set containing package derivations that match the 3604 arguments names we've obtained 3605 3606 - A way to obtain an auto-populated attribute set combining the package 3607 repository and the return value of =functionArgs=. 3608 3609 The former is easy: we just have to set our package derivation's inputs 3610 to be package names in a repository, such as =nixpkgs=. For the latter, 3611 Nix provides another builtin function: 3612 3613 #+begin_example 3614 nix-repl> values = { a = 3; b = 5; c = 10; } 3615 nix-repl> builtins.intersectAttrs values (builtins.functionArgs add) 3616 { a = true; b = false; } 3617 nix-repl> builtins.intersectAttrs (builtins.functionArgs add) values 3618 { a = 3; b = 5; } 3619 #+end_example 3620 3621 The =intersectAttrs= returns an attribute set whose names are the 3622 intersection of both arguments' attribute names, with the attribute 3623 values taken from the second argument. 3624 3625 This is all we need to do: we have obtained the argument names from a 3626 function, and populated these with an existing set of attributes. This 3627 is our simple implementation of =callPackage=: 3628 3629 #+begin_example 3630 nix-repl> callPackage = set: f: f (builtins.intersectAttrs (builtins.functionArgs f) set) 3631 nix-repl> callPackage values add 3632 8 3633 nix-repl> with values; add { inherit a b; } 3634 8 3635 #+end_example 3636 3637 Let's dissect the above snippet: 3638 3639 - We define a =callPackage= variable which is a function. 3640 3641 - The first parameter to the =callPackage= function is a set of 3642 name-value pairs that may appear in the argument set of the function 3643 we wish to "autocall". 3644 3645 - The second parameter is the function to "autocall" 3646 3647 - We take the argument names of the function and intersect with the set 3648 of all values. 3649 3650 - Finally, we call the passed function =f= with the resulting 3651 intersection. 3652 3653 In the snippet above, we've also demonstrated that the =callPackage= 3654 call is equivalent to directly calling =add a b=. 3655 3656 We achieved most of what we wanted: to automatically call functions 3657 given a set of possible arguments. If an argument is not found within 3658 the set we used to call the function, then we receive an error (unless 3659 the function has variadic arguments denoted with =...=, as explained in 3660 the [[#05-functions-and-imports.html][5th pill]]). 3661 3662 The last missing piece is allowing users to override some of the 3663 parameters. We may not want to always call functions with values taken 3664 from the big set. Thus, we add a third parameter which takes a set of 3665 overrides: 3666 3667 #+begin_example 3668 nix-repl> callPackage = set: f: overrides: f ((builtins.intersectAttrs (builtins.functionArgs f) set) // overrides) 3669 nix-repl> callPackage values add { } 3670 8 3671 nix-repl> callPackage values add { b = 12; } 3672 15 3673 #+end_example 3674 3675 Apart from the increasing number of parentheses, it should be clear that 3676 we simply take a set union between the default arguments and the 3677 overriding set. 3678 3679 ** Using callPackage to simplify the repository 3680 Given our =callPackages=, we can simplify the repository expression in 3681 =default.nix=: 3682 3683 #+begin_example 3684 let 3685 nixpkgs = import <nixpkgs> { }; 3686 allPkgs = nixpkgs // pkgs; 3687 callPackage = 3688 path: overrides: 3689 let 3690 f = import path; 3691 in 3692 f ((builtins.intersectAttrs (builtins.functionArgs f) allPkgs) // overrides); 3693 pkgs = with nixpkgs; { 3694 mkDerivation = import ./autotools.nix nixpkgs; 3695 hello = callPackage ./hello.nix { }; 3696 graphviz = callPackage ./graphviz.nix { }; 3697 graphvizCore = callPackage ./graphviz.nix { gdSupport = false; }; 3698 }; 3699 in 3700 pkgs 3701 #+end_example 3702 3703 Let's examine this in detail: 3704 3705 - The expression above defines our own package repository, which we call 3706 =pkgs=, that contains =hello= along with our two variants of 3707 =graphviz=. 3708 3709 - In the =let= expression, we import =nixpkgs=. Note that previously, we 3710 referred to this import with the variable =pkgs=, but now that name is 3711 taken by the repository we are creating ourselves. 3712 3713 - We needed a way to pass =pkgs= to =callPackage= somehow. Instead of 3714 returning the set of packages directly from =default.nix=, we first 3715 assign it to a =let= variable and reuse it in =callPackage=. 3716 3717 - For convenience, in =callPackage= we first import the file instead of 3718 calling it directly. Otherwise we would have to write the =import= for 3719 each package. 3720 3721 - Since our expressions use packages from =nixpkgs=, in =callPackage= we 3722 use =allPkgs=, which is the union of =nixpkgs= and our packages. 3723 3724 - We moved =mkDerivation= into =pkgs= itself, so that it also gets 3725 passed automatically. 3726 3727 Note how easily we overrode arguments in the case of =graphviz= without 3728 =gd=. In addition, note how easy it was to merge two repositories: 3729 =nixpkgs= and our =pkgs=! 3730 3731 The reader should notice a magic thing happening. We're defining =pkgs= 3732 in terms of =callPackage=, and =callPackage= in terms of =pkgs=. That 3733 magic is possible thanks to lazy evaluation: =builtins.intersectAttrs= 3734 doesn't need to know the values in =allPkgs= in order to perform 3735 intersection, only the keys that do not require =callPackage= 3736 evaluation. 3737 3738 ** Conclusion 3739 The "=callPackage=" pattern has simplified our repository considerably. 3740 We were able to import packages that require named arguments and call 3741 them automatically, given the set of all packages sourced from 3742 =nixpkgs=. 3743 3744 We've also introduced some useful builtin functions that allows us to 3745 introspect Nix functions and manipulate attributes. These builtin 3746 functions are not usually used when packaging software, but rather act 3747 as tools for packaging. They are documented in the 3748 [[https://nix.dev/manual/nix/stable/language/builtins][Nix manual]]. 3749 3750 Writing a repository in Nix is an evolution of writing convenient 3751 functions for combining the packages. This pill demonstrates how Nix can 3752 be a generic tool to build and deploy software, and how suitable it is 3753 to create software repositories with our own conventions. 3754 3755 ** Next pill 3756 In the next pill, we will talk about the "=override=" design pattern. 3757 The =graphvizCore= seems straightforward. It starts from =graphviz.nix= 3758 and builds it without =gd=. In the next pill, we will consider another 3759 point of view: starting from =pkgs.graphviz= and disabling =gd=? 3760 3761 <<14-override-design-pattern.html>> 3762 3763 * Override Design Pattern 3764 Welcome to the 14th Nix pill. In the previous 3765 [[#13-callpackage-design-pattern.html][13th]] pill, we introduced the 3766 =callPackage= pattern and used it to simplify the composition of 3767 software in a repository. 3768 3769 The next design pattern is less necessary, but is useful in many cases 3770 and is a good exercise to learn more about Nix. 3771 3772 ** About composability 3773 Functional languages are known for being able to compose functions. In 3774 particular, these languages gain expressivity from functions that 3775 manipulate an original value into a new value having the same structure. 3776 This allows us to compose multiple functions to perform the desired 3777 modifications. 3778 3779 In Nix, we mostly talk about *functions* that accept inputs in order to 3780 return *derivations*. In our world, we want utility functions that are 3781 able to manipulate those structures. These utilities add some useful 3782 properties to the original value, and we'd like to be able to apply more 3783 utilities on top of the result. 3784 3785 For example, let's say we have an initial derivation =drv= and we want 3786 to transform it into a =drv= with debugging information and custom 3787 patches: 3788 3789 #+begin_example 3790 debugVersion (applyPatches [ ./patch1.patch ./patch2.patch ] drv) 3791 #+end_example 3792 3793 The final result should be the original derivation with some changes. 3794 This is both interesting and very different from other packaging 3795 approaches, which is a consequence of using a functional language to 3796 describe packages. 3797 3798 Designing such utilities is not trivial in a functional language without 3799 static typing, because understanding what can or cannot be composed is 3800 difficult. But we try to do our best. 3801 3802 ** The override pattern 3803 In [[#12-inputs-design-pattern.html][pill 12]] we introduced the inputs 3804 design pattern. We do not return a derivation picking dependencies 3805 directly from the repository; rather we declare the inputs and let the 3806 callers pass the necessary arguments. 3807 3808 In our repository we have a set of attributes that import the 3809 expressions of the packages and pass these arguments, getting back a 3810 derivation. Let's take for example the =graphviz= attribute: 3811 3812 #+begin_example 3813 graphviz = import ./graphviz.nix { inherit mkDerivation gd fontconfig libjpeg bzip2; }; 3814 #+end_example 3815 3816 If we wanted to produce a derivation of =graphviz= with a customized 3817 =gd= version, we would have to repeat most of the above plus specifying 3818 an alternative =gd=: 3819 3820 #+begin_example 3821 { 3822 mygraphviz = import ./graphviz.nix { 3823 inherit 3824 mkDerivation 3825 fontconfig 3826 libjpeg 3827 bzip2 3828 ; 3829 gd = customgd; 3830 }; 3831 } 3832 #+end_example 3833 3834 That's hard to maintain. Using =callPackage= would be easier: 3835 3836 #+begin_example 3837 mygraphviz = callPackage ./graphviz.nix { gd = customgd; }; 3838 #+end_example 3839 3840 But we may still be diverging from the original graphviz in the 3841 repository. 3842 3843 We would like to avoid specifying the nix expression again. Instead, we 3844 would like to reuse the original =graphviz= attribute in the repository 3845 and add our overrides like so: 3846 3847 #+begin_example 3848 mygraphviz = graphviz.override { gd = customgd; }; 3849 #+end_example 3850 3851 The difference is obvious, as well as the advantages of this approach. 3852 3853 Note: that =.override= is not a "method" in the OO sense as you may 3854 think. Nix is a functional language. The=.override= is simply an 3855 attribute of a set. 3856 3857 ** The override implementation 3858 Recall that the =graphviz= attribute in the repository is the derivation 3859 returned by the function imported from =graphviz.nix=. We would like to 3860 add a further attribute named "=override=" to the returned set. 3861 3862 Let's start by first creating a function "=makeOverridable=". This 3863 function will take two arguments: a function (that must return a set) 3864 and the set of original arguments to be passed to the function. 3865 3866 We will put this function in a =lib.nix=: 3867 3868 #+begin_example 3869 { 3870 makeOverridable = 3871 f: origArgs: 3872 let 3873 origRes = f origArgs; 3874 in 3875 origRes // { override = newArgs: f (origArgs // newArgs); }; 3876 } 3877 #+end_example 3878 3879 =makeOverridable= takes a function and a set of original arguments. It 3880 returns the original returned set, plus a new =override= attribute. 3881 3882 This =override= attribute is a function taking a set of new arguments, 3883 and returns the result of the original function called with the original 3884 arguments unified with the new arguments. This is admittedly somewhat 3885 confusing, but the examples below should make it clear. 3886 3887 Let's try it with =nix repl=: 3888 3889 #+begin_example 3890 $ nix repl 3891 nix-repl> :l lib.nix 3892 Added 1 variables. 3893 nix-repl> f = { a, b }: { result = a+b; } 3894 nix-repl> f { a = 3; b = 5; } 3895 { result = 8; } 3896 nix-repl> res = makeOverridable f { a = 3; b = 5; } 3897 nix-repl> res 3898 { override = «lambda»; result = 8; } 3899 nix-repl> res.override { a = 10; } 3900 { result = 15; } 3901 #+end_example 3902 3903 Note that, as we specified above, the function =f= does not return the 3904 plain sum. Instead, it returns a set with the sum bound to the name 3905 =result=. 3906 3907 The variable =res= contains the result of the function call without any 3908 override. It's easy to see in the definition of =makeOverridable=. In 3909 addition, you can see that the new =override= attribute is a function. 3910 3911 Calling =res.override= with a set will invoke the original function with 3912 the overrides, as expected. 3913 3914 This is a good start, but we can't override again! This is because the 3915 returned set (with =result = 15=) does not have an =override= attribute 3916 of its own. This is bad; it breaks further composition. 3917 3918 The solution is simple: the =.override= function should make the result 3919 overridable again: 3920 3921 #+begin_example 3922 rec { 3923 makeOverridable = 3924 f: origArgs: 3925 let 3926 origRes = f origArgs; 3927 in 3928 origRes // { override = newArgs: makeOverridable f (origArgs // newArgs); }; 3929 } 3930 #+end_example 3931 3932 Please note the =rec= keyword. It's necessary so that we can refer to 3933 =makeOverridable= from =makeOverridable= itself. 3934 3935 Now let's try overriding twice: 3936 3937 #+begin_example 3938 nix-repl> :l lib.nix 3939 Added 1 variables. 3940 nix-repl> f = { a, b }: { result = a+b; } 3941 nix-repl> res = makeOverridable f { a = 3; b = 5; } 3942 nix-repl> res2 = res.override { a = 10; } 3943 nix-repl> res2 3944 { override = «lambda»; result = 15; } 3945 nix-repl> res2.override { b = 20; } 3946 { override = «lambda»; result = 30; } 3947 #+end_example 3948 3949 Success! The result is 30 (as expected) because =a= is overridden to 10 3950 in the first override, and =b= is overridden to 20 in the second. 3951 3952 Now it would be nice if =callPackage= made our derivations overridable. 3953 This is an exercise for the reader. 3954 3955 ** Conclusion 3956 The "=override=" pattern simplifies the way we customize packages 3957 starting from an existing set of packages. This opens a world of 3958 possibilities for using a central repository like =nixpkgs= and defining 3959 overrides on our local machine without modifying the original package. 3960 3961 We can dream of a custom, isolated =nix-shell= environment for testing 3962 =graphviz= with a custom =gd=: 3963 3964 #+begin_example 3965 debugVersion (graphviz.override { gd = customgd; }) 3966 #+end_example 3967 3968 Once a new version of the overridden package comes out in the 3969 repository, the customized package will make use of it automatically. 3970 3971 The key in Nix is to find powerful yet simple abstractions in order to 3972 let the user customize their environment with highest consistency and 3973 lowest maintenance time, by using predefined composable components. 3974 3975 ** Next pill 3976 In the next pill, we will talk about Nix search paths. By "search path", 3977 we mean a place in the file system where Nix looks for expressions. This 3978 answers the question of where =<nixpkgs>= comes from. 3979 3980 <<15-nix-search-paths.html>> 3981 3982 * Nix Search Paths 3983 Welcome to the 15th Nix pill. In the previous 3984 [[#14-override-design-pattern.html][14th]] pill we have introduced the 3985 "override" pattern, useful for writing variants of derivations by 3986 passing different inputs. 3987 3988 Assuming you followed the previous posts, I hope you are now ready to 3989 understand =nixpkgs=. But we have to find =nixpkgs= in our system first! 3990 So this is the step: introducing some options and environment variables 3991 used by nix tools. 3992 3993 ** The NIX_PATH 3994 The 3995 [[https://nix.dev/manual/nix/stable/command-ref/env-common?highlight=NIX_PATH][NIX_PATH 3996 environment variable]] is very important. It's very similar to the 3997 =PATH= environment variable. The syntax is similar, several paths are 3998 separated by a colon =:=. Nix will then search for something in those 3999 paths from left to right. 4000 4001 Who uses =NIX_PATH=? The nix expressions! Yes, =NIX_PATH= is not of much 4002 use by the nix tools themselves, rather it's used when writing nix 4003 expressions. 4004 4005 In the shell for example, when you execute the command =ping=, it's 4006 being searched in the =PATH= directories. The first one found is the one 4007 being used. 4008 4009 In nix it's exactly the same, however the syntax is different. Instead 4010 of just typing =ping= you have to type =<ping>=. Yes, I know... you are 4011 already thinking of =<nixpkgs>=. However, don't stop reading here, let's 4012 keep going. 4013 4014 What's =NIX_PATH= good for? Nix expressions may refer to an "abstract" 4015 path such as =<nixpkgs>=, and it's possible to override it from the 4016 command line. 4017 4018 For ease we will use =nix-instantiate --eval= to do our tests. I remind 4019 you, 4020 [[https://nix.dev/manual/nix/stable/command-ref/nix-instantiate][nix-instantiate]] 4021 is used to evaluate nix expressions and generate the .drv files. Here we 4022 are not interested in building derivations, so evaluation is enough. It 4023 can be used for one-shot expressions. 4024 4025 ** Fake it a little 4026 It's useless from a nix view point, but I think it's useful for your own 4027 understanding. Let's use =PATH= itself as =NIX_PATH=, and try to locate 4028 =ping= (or another binary if you don't have it). 4029 4030 #+begin_example 4031 $ nix-instantiate --eval -E '<ping>' 4032 error: file `ping' was not found in the Nix search path (add it using $NIX_PATH or -I) 4033 $ NIX_PATH=$PATH nix-instantiate --eval -E '<ping>' 4034 /bin/ping 4035 $ nix-instantiate -I /bin --eval -E '<ping>' 4036 /bin/ping 4037 #+end_example 4038 4039 Great. At first attempt nix obviously said could not be found anywhere 4040 in the search path. Note that the =-I= option accepts a single 4041 directory. Paths added with =-I= take precedence over =NIX_PATH=. 4042 4043 The =NIX_PATH= also accepts a different yet very handy syntax: 4044 "=somename=somepath=". That is, instead of searching inside a directory 4045 for a name, we specify exactly the value of that name. 4046 4047 #+begin_example 4048 $ NIX_PATH="ping=/bin/ping" nix-instantiate --eval -E '<ping>' 4049 /bin/ping 4050 $ NIX_PATH="ping=/bin/foo" nix-instantiate --eval -E '<ping>' 4051 error: file `ping' was not found in the Nix search path (add it using $N 4052 #+end_example 4053 4054 Note in the second case how Nix checks whether the path exists or not. 4055 4056 ** The path to repository 4057 You are out of curiosity, right? 4058 4059 #+begin_example 4060 $ nix-instantiate --eval -E '<nixpkgs>' 4061 /home/nix/.nix-defexpr/channels/nixpkgs 4062 $ echo $NIX_PATH 4063 nixpkgs=/home/nix/.nix-defexpr/channels/nixpkgs 4064 #+end_example 4065 4066 You may have a different path, depending on how you added channels etc.. 4067 Anyway that's the whole point. The =<nixpkgs>= stranger that we used in 4068 our nix expressions, is referring to a path in the filesystem specified 4069 by =NIX_PATH=. 4070 4071 You can list that directory and realize it's simply a checkout of the 4072 nixpkgs repository at a specific commit (hint: =.version-suffix=). 4073 4074 The =NIX_PATH= variable is exported by =nix.sh=, and that's the reason 4075 why I always asked you to 4076 [[https://nix.dev/manual/nix/stable/installation/env-variables][source 4077 nix.sh]] at the beginning of my posts. 4078 4079 You may wonder: then I can also specify a different 4080 [[https://github.com/NixOS/nixpkgs][nixpkgs]] path to, e.g., a 4081 =git checkout= of =nixpkgs=? Yes, you can and I encourage doing that. 4082 We'll talk about this in the next pill. 4083 4084 Let's define a path for our repository, then! Let's say all the 4085 =default.nix=, =graphviz.nix= etc. are under =/home/nix/mypkgs=: 4086 4087 #+begin_example 4088 $ export NIX_PATH=mypkgs=/home/nix/mypkgs:$NIX_PATH 4089 $ nix-instantiate --eval '<mypkgs>' 4090 { graphviz = <code>; graphvizCore = <code>; hello = <code>; mkDerivation = <code>; } 4091 #+end_example 4092 4093 Yes, =nix-build= also accepts paths with angular brackets. We first 4094 evaluate the whole repository (=default.nix=) and then pick the 4095 =graphviz= attribute. 4096 4097 ** A big word about nix-env 4098 The [[https://nix.dev/manual/nix/stable/command-ref/nix-env][nix-env]] 4099 command is a little different than =nix-instantiate= and =nix-build=. 4100 Whereas =nix-instantiate= and =nix-build= require a starting nix 4101 expression, =nix-env= does not. 4102 4103 You may be crippled by this concept at the beginning, you may think 4104 =nix-env= uses =NIX_PATH= to find the =nixpkgs= repository. But that's 4105 not it. 4106 4107 The =nix-env= command uses =~/.nix-defexpr=, which is also part of 4108 =NIX_PATH= by default, but that's only a coincidence. If you empty 4109 =NIX_PATH=, =nix-env= will still be able to find derivations because of 4110 =~/.nix-defexpr=. 4111 4112 So if you run =nix-env -i graphviz= inside your repository, it will 4113 install the nixpkgs one. Same if you set =NIX_PATH= to point to your 4114 repository. 4115 4116 In order to specify an alternative to =~/.nix-defexpr= it's possible to 4117 use the -f option: 4118 4119 #+begin_example 4120 $ nix-env -f '<mypkgs>' -i graphviz 4121 warning: there are multiple derivations named `graphviz'; using the first one 4122 replacing old `graphviz' 4123 installing `graphviz' 4124 #+end_example 4125 4126 Oh why did it say there's another derivation named =graphviz=? Because 4127 both =graphviz= and =graphvizCore= attributes in our repository have the 4128 name "graphviz" for the derivation: 4129 4130 #+begin_example 4131 $ nix-env -f '<mypkgs>' -qaP 4132 graphviz graphviz 4133 graphvizCore graphviz 4134 hello hello 4135 #+end_example 4136 4137 By default =nix-env= parses all derivations and uses the derivation 4138 names to interpret the command line. So in this case "graphviz" matched 4139 two derivations. Alternatively, like for =nix-build=, one can use -A to 4140 specify an attribute name instead of a derivation name: 4141 4142 #+begin_example 4143 $ nix-env -f '<mypkgs>' -i -A graphviz 4144 replacing old `graphviz' 4145 installing `graphviz' 4146 #+end_example 4147 4148 This form, other than being more precise, it's also faster because 4149 =nix-env= does not have to parse all the derivations. 4150 4151 For completeness: you must install =graphvizCore= with -A, since without 4152 the -A switch it's ambiguous. 4153 4154 In summary, it may happen when playing with nix that =nix-env= picks a 4155 different derivation than =nix-build=. In that case you probably 4156 specified =NIX_PATH=, but =nix-env= is instead looking into 4157 =~/.nix-defexpr=. 4158 4159 Why is =nix-env= having this different behavior? I don't know 4160 specifically by myself either, but the answers could be: 4161 4162 - =nix-env= tries to be generic, thus it does not look for =nixpkgs= in 4163 =NIX_PATH=, rather it looks in =~/.nix-defexpr=. 4164 4165 - =nix-env= is able to merge multiple trees in =~/.nix-defexpr= by 4166 looking at all the possible derivations 4167 4168 It may also happen to you *that you cannot match a derivation name when 4169 installing*, because of the derivation name vs -A switch described 4170 above. Maybe =nix-env= wanted to be more friendly in this case for 4171 default user setups. 4172 4173 It may or may not make sense for you, or it's like that for historical 4174 reasons, but that's how it works currently, unless somebody comes up 4175 with a better idea. 4176 4177 ** Conclusion 4178 The =NIX_PATH= variable is the search path used by nix when using the 4179 angular brackets syntax. It's possible to refer to "abstract" paths 4180 inside nix expressions and define the "concrete" path by means of 4181 =NIX_PATH=, or the usual =-I= flag in nix tools. 4182 4183 We've also explained some of the uncommon =nix-env= behaviors for 4184 newcomers. The =nix-env= tool does not use =NIX_PATH= to search for 4185 packages, but rather for =~/.nix-defexpr=. Beware of that! 4186 4187 In general do not abuse =NIX_PATH=, when possible use relative paths 4188 when writing your own nix expressions. Of course, in the case of 4189 =<nixpkgs>= in our repository, that's a perfectly fine usage of 4190 =NIX_PATH=. Instead, inside our repository itself, refer to expressions 4191 with relative paths like =./hello.nix=. 4192 4193 ** Next pill 4194 ...we will finally dive into =nixpkgs=. Most of the techniques we have 4195 developed in this series are already in =nixpkgs=, like =mkDerivation=, 4196 =callPackage=, =override=, etc., but of course better. With time, those 4197 base utilities get enhanced by the community with more features in order 4198 to handle more and more use cases and in a more general way. 4199 4200 <<16-nixpkgs-parameters.html>> 4201 4202 * Nixpkgs Parameters 4203 Welcome to the 16th Nix pill. In the previous 4204 [[#15-nix-search-paths.html][15th]] pill we've realized how nix finds 4205 expressions with the angular brackets syntax, so that we finally know 4206 where =<nixpkgs>= is located on our system. 4207 4208 We can start diving into the [[https://github.com/NixOS/nixpkgs][nixpkgs 4209 repository]], through all the various tools and design patterns. Please 4210 note that also =nixpkgs= has its own manual, underlying the difference 4211 between the general =nix= language and the =nixpkgs= repository. 4212 4213 ** The default.nix expression 4214 We will not start inspecting packages at the beginning, rather the 4215 general structure of =nixpkgs=. 4216 4217 In our custom repository we created a =default.nix= which composed the 4218 expressions of the various packages. 4219 4220 Also =nixpkgs= has its own 4221 [[https://github.com/NixOS/nixpkgs/blob/master/default.nix][default.nix]], 4222 which is the one being loaded when referring to =<nixpkgs>=. It does a 4223 simple thing: check whether the =nix= version is at least 1.7 (at the 4224 time of writing this blog post). Then import 4225 [[https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix][pkgs/top-level/all-packages.nix]]. 4226 From now on, we will refer to this set of packages as *pkgs*. 4227 4228 The =all-packages.nix= is then the file that composes all the packages. 4229 Note the =pkgs/= subdirectory, while nixos is in the =nixos/= 4230 subdirectory. 4231 4232 The =all-packages.nix= is a bit contrived. First of all, it's a 4233 function. It accepts a couple of interesting parameters: 4234 4235 - =system=: defaults to the current system 4236 4237 - =config=: defaults to null 4238 4239 - others... 4240 4241 The *system* parameter, as per comment in the expression, it's the 4242 system for which the packages will be built. It allows for example to 4243 install i686 packages on amd64 machines. 4244 4245 The *config* parameter is a simple attribute set. Packages can read some 4246 of its values and change the behavior of some derivations. 4247 4248 ** The system parameter 4249 You will find this parameter in many other .nix expressions (e.g. 4250 release expressions). The reason is that, given pkgs accepts a system 4251 parameter, then whenever you want to import pkgs you also want to pass 4252 through the value of system. E.g.: 4253 4254 =myrelease.nix=: 4255 4256 #+begin_example 4257 { system ? builtins.currentSystem }: 4258 4259 let pkgs = import <nixpkgs> { inherit system; }; 4260 ... 4261 #+end_example 4262 4263 Why is it useful? With this parameter it's very easy to select a set of 4264 packages for a particular system. For example: 4265 4266 #+begin_example 4267 nix-build -A psmisc --argstr system i686-linux 4268 #+end_example 4269 4270 This will build the =psmisc= derivation for i686-linux instead of 4271 x86_64-linux. This concept is very similar to multi-arch of Debian. 4272 4273 The setup for cross compiling is also in =nixpkgs=, however it's a 4274 little contrived to talk about it and I don't know much of it either. 4275 4276 ** The config parameter 4277 I'm sure on the wiki or other manuals you've read about 4278 =~/.config/nixpkgs/config.nix= (previously =~/.nixpkgs/config.nix=) and 4279 I'm sure you've wondered whether that's hardcoded in nix. It's not, it's 4280 in 4281 [[https://github.com/NixOS/nixpkgs/blob/32c523914fdb8bf9cc7912b1eba023a8daaae2e8/pkgs/top-level/impure.nix#L28][nixpkgs]]. 4282 4283 The =all-packages.nix= expression accepts the =config= parameter. If 4284 it's =null=, then it reads the =NIXPKGS_CONFIG= environment variable. If 4285 not specified, =nixpkgs= will pick =$HOME/.config/nixpkgs/config.nix=. 4286 4287 After determining =config.nix=, it will be imported as a nix expression, 4288 and that will be the value of =config= (in case it hasn't been passed as 4289 parameter to import =<nixpkgs>=). 4290 4291 The =config= is available in the resulting repository: 4292 4293 #+begin_example 4294 $ nix repl 4295 nix-repl> pkgs = import <nixpkgs> {} 4296 nix-repl> pkgs.config 4297 { } 4298 nix-repl> pkgs = import <nixpkgs> { config = { foo = "bar"; }; } 4299 nix-repl> pkgs.config 4300 { foo = "bar"; } 4301 #+end_example 4302 4303 What attributes go in =config= is a matter of convenience and 4304 conventions. 4305 4306 For example, =config.allowUnfree= is an attribute that forbids building 4307 packages that have an unfree license by default. The =config.pulseaudio= 4308 setting tells whether to build packages with =pulseaudio= support or not 4309 where applicable and when the derivation obeys to the setting. 4310 4311 ** About .nix functions 4312 A =.nix= file contains a nix expression. Thus it can also be a function. 4313 I remind you that =nix-build= expects the expression to return a 4314 derivation. Therefore it's natural to return straight a derivation from 4315 a =.nix= file. However, it's also very natural for the =.nix= file to 4316 accept some parameters, in order to tweak the derivation being returned. 4317 4318 In this case, nix does a trick: 4319 4320 - If the expression is a derivation, build it. 4321 4322 - If the expression is a function, call it and build the resulting 4323 derivation. 4324 4325 For example you can nix-build the =.nix= file below: 4326 4327 #+begin_example 4328 { pkgs ? import <nixpkgs> {} }: 4329 4330 pkgs.psmisc 4331 #+end_example 4332 4333 Nix is able to call the function because the pkgs parameter has a 4334 default value. This allows you to pass a different value for pkgs using 4335 the =--arg= option. 4336 4337 Does it work if you have a function returning a function that returns a 4338 derivation? No, Nix only calls the function it encounters once. 4339 4340 ** Conclusion 4341 We've unleashed the =<nixpkgs>= repository. It's a function that accepts 4342 some parameters, and returns the set of all packages. Due to laziness, 4343 only the accessed derivations will be built. 4344 4345 You can use this repository to build your own packages as we've seen in 4346 the previous pill when creating our own repository. 4347 4348 Lately I'm a little busy with the NixOS 14.11 release and other stuff, 4349 and I'm also looking toward migrating from blogger to a more 4350 coder-oriented blogging platform. So sorry for the delayed and shorter 4351 pills :) 4352 4353 ** Next pill 4354 ...we will talk about overriding packages in the =nixpkgs= repository. 4355 What if you want to change some options of a library and let all other 4356 packages pick the new library? One possibility is to use, like described 4357 above, the =config= parameter when applicable. The other possibility is 4358 to override derivations. 4359 4360 <<17-nixpkgs-overriding-packages.html>> 4361 4362 * Nixpkgs Overriding Packages 4363 Welcome to the 17th Nix pill. In the previous 4364 [[#16-nixpkgs-parameters.html][16th]] pill we have started to dive into 4365 the [[http://github.com/NixOS/nixpkgs][nixpkgs repository]]. =Nixpkgs= 4366 is a function, and we've looked at some parameters like =system= and 4367 =config=. 4368 4369 Today we'll talk about a special attribute: =config.packageOverrides=. 4370 Overriding packages in a set with fixed point can be considered another 4371 design pattern in nixpkgs. 4372 4373 ** Overriding a package 4374 Recall the override design pattern from the 4375 [[#14-override-design-pattern.html][nix pill 14]]. Instead of calling a 4376 function with parameters directly, we make the call (function + 4377 parameters) overridable. 4378 4379 We put the override function in the returned attribute set of the 4380 original function call. 4381 4382 Take for example =graphviz=. It has an input parameter =xorg=. If it's 4383 null, then =graphviz= will build without X support. 4384 4385 #+begin_example 4386 $ nix repl 4387 nix-repl> :l <nixpkgs> 4388 Added 4360 variables. 4389 nix-repl> :b graphviz.override { withXorg = false; } 4390 #+end_example 4391 4392 This will build =graphviz= without X support, it's as simple as that. 4393 4394 However, let's say a package =P= depends on =graphviz=, how do we make 4395 =P= depend on the new =graphviz= without X support? 4396 4397 ** In an imperative world... 4398 ...you could do something like this: 4399 4400 #+begin_example 4401 pkgs = import <nixpkgs> {}; 4402 pkgs.graphviz = pkgs.graphviz.override { withXorg = false; }; 4403 build(pkgs.P) 4404 #+end_example 4405 4406 Given =pkgs.P= depends on =pkgs.graphviz=, it's easy to build =P= with 4407 the replaced =graphviz=. In a pure functional language it's not that 4408 easy because you can assign to variables only once. 4409 4410 ** Fixed point 4411 The fixed point with lazy evaluation is crippling but about necessary in 4412 a language like Nix. It lets us achieve something similar to what we'd 4413 do imperatively. 4414 4415 Follows the definition of fixed point in 4416 [[https://github.com/NixOS/nixpkgs/blob/f224a4f1b32b3e813783d22de54e231cd8ea2448/lib/fixed-points.nix#L19][nixpkgs]]: 4417 4418 #+begin_example 4419 { 4420 # Take a function and evaluate it with its own returned value. 4421 fix = 4422 f: 4423 let 4424 result = f result; 4425 in 4426 result; 4427 } 4428 #+end_example 4429 4430 It's a function that accepts a function =f=, calls =f result= on the 4431 result just returned by =f result= and returns it. In other words it's 4432 =f(f(f(....= 4433 4434 At first sight, it's an infinite loop. With lazy evaluation it isn't, 4435 because the call is done only when needed. 4436 4437 #+begin_example 4438 nix-repl> fix = f: let result = f result; in result 4439 nix-repl> pkgs = self: { a = 3; b = 4; c = self.a+self.b; } 4440 nix-repl> fix pkgs 4441 { a = 3; b = 4; c = 7; } 4442 #+end_example 4443 4444 Without the =rec= keyword, we were able to refer to =a= and =b= of the 4445 same set. 4446 4447 - First =pkgs= gets called with an unevaluated thunk =(pkgs(pkgs(...)= 4448 4449 - To set the value of =c= then =self.a= and =self.b= are evaluated. 4450 4451 - The =pkgs= function gets called again to get the value of =a= and =b=. 4452 4453 The trick is that =c= is not needed to be evaluated in the inner call, 4454 thus it doesn't go in an infinite loop. 4455 4456 Won't go further with the explanation here. A good post about fixed 4457 point and Nix can be [[http://r6.ca/blog/20140422T142911Z.html][found 4458 here]]. 4459 4460 *** Overriding a set with fixed point 4461 Given that =self.a= and =self.b= refer to the passed set and not to the 4462 literal set in the function, we're able to override both =a= and =b= and 4463 get a new value for =c=: 4464 4465 #+begin_example 4466 nix-repl> overrides = { a = 1; b = 2; } 4467 nix-repl> let newpkgs = pkgs (newpkgs // overrides); in newpkgs 4468 { a = 3; b = 4; c = 3; } 4469 nix-repl> let newpkgs = pkgs (newpkgs // overrides); in newpkgs // overrides 4470 { a = 1; b = 2; c = 3; } 4471 #+end_example 4472 4473 In the first case we computed pkgs with the overrides, in the second 4474 case we also included the overridden attributes in the result. 4475 4476 ** Overriding nixpkgs packages 4477 We've seen how to override attributes in a set such that they get 4478 recursively picked by dependent attributes. This approach can be used 4479 for derivations too, after all =nixpkgs= is a giant set of attributes 4480 that depend on each other. 4481 4482 To do this, =nixpkgs= offers =config.packageOverrides=. So =nixpkgs= 4483 returns a fixed point of the package set, and =packageOverrides= is used 4484 to inject the overrides. 4485 4486 Create a =config.nix= file like this somewhere: 4487 4488 #+begin_example 4489 { 4490 packageOverrides = pkgs: { 4491 graphviz = pkgs.graphviz.override { 4492 # disable xorg support 4493 withXorg = false; 4494 }; 4495 }; 4496 } 4497 #+end_example 4498 4499 Now we can build e.g. =asciidoc-full= and it will automatically use the 4500 overridden =graphviz=: 4501 4502 #+begin_example 4503 nix-repl> pkgs = import <nixpkgs> { config = import ./config.nix; } 4504 nix-repl> :b pkgs.asciidoc-full 4505 #+end_example 4506 4507 Note how we pass the =config= with =packageOverrides= when importing 4508 =nixpkgs=. Then =pkgs.asciidoc-full= is a derivation that has =graphviz= 4509 input (=pkgs.asciidoc= is the lighter version and doesn't use =graphviz= 4510 at all). 4511 4512 Since there's no version of =asciidoc= with =graphviz= without X support 4513 in the binary cache, Nix will recompile the needed stuff for you. 4514 4515 ** The ~/.config/nixpkgs/config.nix file 4516 In the previous pill we already talked about this file. The above 4517 =config.nix= that we just wrote could be the content of 4518 =~/.config/nixpkgs/config.nix= (or the deprecated location 4519 =~/.nixpkgs/config.nix=). 4520 4521 Instead of passing it explicitly whenever we import =nixpkgs=, it will 4522 be automatically 4523 [[https://github.com/NixOS/nixpkgs/blob/32c523914fdb8bf9cc7912b1eba023a8daaae2e8/pkgs/top-level/impure.nix#L28][imported 4524 by nixpkgs]]. 4525 4526 ** Conclusion 4527 We've learned about a new design pattern: using fixed point for 4528 overriding packages in a package set. 4529 4530 Whereas in an imperative setting, like with other package managers, a 4531 library is installed replacing the old version and applications will use 4532 it, in Nix it's not that straight and simple. But it's more precise. 4533 4534 Nix applications will depend on specific versions of libraries, hence 4535 the reason why we have to recompile =asciidoc= to use the new =graphviz= 4536 library. 4537 4538 The newly built =asciidoc= will depend on the new =graphviz=, and old 4539 =asciidoc= will keep using the old =graphviz= undisturbed. 4540 4541 ** Next pill 4542 ...we will stop studying =nixpkgs= for a moment and talk about store 4543 paths. How does Nix compute the path in the store where to place the 4544 result of builds? How to add files to the store for which we have an 4545 integrity hash? 4546 4547 <<18-nix-store-paths.html>> 4548 4549 * Nix Store Paths 4550 Welcome to the 18th Nix pill. In the previous 4551 [[#17-nixpkgs-overriding-packages.html][17th]] pill we have scratched 4552 the surface of the =nixpkgs= repository structure. It is a set of 4553 packages, and it's possible to override such packages so that all other 4554 packages will use the overrides. 4555 4556 Before reading existing derivations, I'd like to talk about store paths 4557 and how they are computed. In particular we are interested in fixed 4558 store paths that depend on an integrity hash (e.g. a sha256), which is 4559 usually applied to source tarballs. 4560 4561 The way store paths are computed is a little contrived, mostly due to 4562 historical reasons. Our reference will be the 4563 [[https://github.com/NixOS/nix/blob/07f992a74b64f4376d5b415d0042babc924772f3/src/libstore/store-api.cc#L197][Nix 4564 source code]]. 4565 4566 ** Source paths 4567 Let's start simple. You know nix allows relative paths to be used, such 4568 that the file or directory is stored in the nix store, that is 4569 =./myfile= gets stored into =/nix/store/.......= We want to understand 4570 how is the store path generated for such a file: 4571 4572 #+begin_example 4573 $ echo mycontent > myfile 4574 #+end_example 4575 4576 I remind you, the simplest derivation you can write has a =name=, a 4577 =builder= and the =system=: 4578 4579 #+begin_example 4580 $ nix repl 4581 nix-repl> derivation { system = "x86_64-linux"; builder = ./myfile; name = "foo"; } 4582 «derivation /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv» 4583 #+end_example 4584 4585 Now inspect the .drv to see where is =./myfile= being stored: 4586 4587 #+begin_example 4588 $ nix derivation show /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv 4589 { 4590 "/nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv": { 4591 "outputs": { 4592 "out": { 4593 "path": "/nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo" 4594 } 4595 }, 4596 "inputSrcs": [ 4597 "/nix/store/xv2iccirbrvklck36f1g7vldn5v58vck-myfile" 4598 ], 4599 "inputDrvs": {}, 4600 "platform": "x86_64-linux", 4601 "builder": "/nix/store/xv2iccirbrvklck36f1g7vldn5v58vck-myfile", 4602 "args": [], 4603 "env": { 4604 "builder": "/nix/store/xv2iccirbrvklck36f1g7vldn5v58vck-myfile", 4605 "name": "foo", 4606 "out": "/nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo", 4607 "system": "x86_64-linux" 4608 } 4609 } 4610 } 4611 #+end_example 4612 4613 Great, how did nix decide to use =xv2iccirbrvklck36f1g7vldn5v58vck= ? 4614 Keep looking at the nix comments. 4615 4616 *Note:* doing =nix-store --add myfile= will store the file in the same 4617 store path. 4618 4619 *** Step 1, compute the hash of the file 4620 The comments tell us to first compute the sha256 of the NAR 4621 serialization of the file. Can be done in two ways: 4622 4623 #+begin_example 4624 $ nix-hash --type sha256 myfile 4625 2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3 4626 #+end_example 4627 4628 Or: 4629 4630 #+begin_example 4631 $ nix-store --dump myfile|sha256sum 4632 2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3 4633 #+end_example 4634 4635 In general, Nix understands two contents: flat for regular files, or 4636 recursive for NAR serializations which can be anything. 4637 4638 *** Step 2, build the string description 4639 Then nix uses a special string which includes the hash, the path type 4640 and the file name. We store this in another file: 4641 4642 #+begin_example 4643 $ echo -n "source:sha256:2bfef67de873c54551d884fdab3055d84d573e654efa79db3c0d7b98883f9ee3:/nix/store:myfile" > myfile.str 4644 #+end_example 4645 4646 *** Step 3, compute the final hash 4647 Finally the comments tell us to compute the base-32 representation of 4648 the first 160 bits (truncation) of a sha256 of the above string: 4649 4650 #+begin_example 4651 $ nix-hash --type sha256 --truncate --base32 --flat myfile.str 4652 xv2iccirbrvklck36f1g7vldn5v58vck 4653 #+end_example 4654 4655 ** Output paths 4656 Output paths are usually generated for derivations. We use the above 4657 example because it's simple. Even if we didn't build the derivation, nix 4658 knows the out path =hs0yi5n5nw6micqhy8l1igkbhqdkzqa1=. This is because 4659 the out path only depends on inputs. 4660 4661 It's computed in a similar way to source paths, except that the .drv is 4662 hashed and the type of derivation is =output:out=. In case of multiple 4663 outputs, we may have different =output:<id>=. 4664 4665 At the time nix computes the out path, the .drv contains an empty string 4666 for each out path. So what we do is getting our .drv and replacing the 4667 out path with an empty string: 4668 4669 #+begin_example 4670 $ cp -f /nix/store/y4h73bmrc9ii5bxg6i7ck6hsf5gqv8ck-foo.drv myout.drv 4671 $ sed -i 's,/nix/store/hs0yi5n5nw6micqhy8l1igkbhqdkzqa1-foo,,g' myout.drv 4672 #+end_example 4673 4674 The =myout.drv= is the .drv state in which nix is when computing the out 4675 path for our derivation: 4676 4677 #+begin_example 4678 $ sha256sum myout.drv 4679 1bdc41b9649a0d59f270a92d69ce6b5af0bc82b46cb9d9441ebc6620665f40b5 myout.drv 4680 $ echo -n "output:out:sha256:1bdc41b9649a0d59f270a92d69ce6b5af0bc82b46cb9d9441ebc6620665f40b5:/nix/store:foo" > myout.str 4681 $ nix-hash --type sha256 --truncate --base32 --flat myout.str 4682 hs0yi5n5nw6micqhy8l1igkbhqdkzqa1 4683 #+end_example 4684 4685 Then nix puts that out path in the .drv, and that's it. 4686 4687 In case the .drv has input derivations, that is it references other 4688 .drv, then such .drv paths are replaced by this same algorithm which 4689 returns a hash. 4690 4691 In other words, you get a final .drv where every other .drv path is 4692 replaced by its hash. 4693 4694 ** Fixed-output paths 4695 Finally, the other most used kind of path is when we know beforehand an 4696 integrity hash of a file. This is usual for tarballs. 4697 4698 A derivation can take three special attributes: =outputHashMode=, 4699 =outputHash= and =outputHashAlgo= which are well documented in the 4700 [[https://nix.dev/manual/nix/stable/language/advanced-attributes][nix 4701 manual]]. 4702 4703 The builder must create the out path and make sure its hash is the same 4704 as the one declared with =outputHash=. 4705 4706 Let's say our builder should create a file whose contents is 4707 =mycontent=: 4708 4709 #+begin_example 4710 $ echo mycontent > myfile 4711 $ sha256sum myfile 4712 f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb myfile 4713 nix-repl> derivation { name = "bar"; system = "x86_64-linux"; builder = "none"; outputHashMode = "flat"; outputHashAlgo = "sha256"; outputHash = "f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb"; } 4714 «derivation /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv» 4715 #+end_example 4716 4717 Inspect the .drv and see that it also stored the fact that it's a 4718 fixed-output derivation with sha256 algorithm, compared to the previous 4719 examples: 4720 4721 #+begin_example 4722 $ nix derivation show /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv 4723 { 4724 "/nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv": { 4725 "outputs": { 4726 "out": { 4727 "path": "/nix/store/a00d5f71k0vp5a6klkls0mvr1f7sx6ch-bar", 4728 "hashAlgo": "sha256", 4729 "hash": "f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb" 4730 } 4731 }, 4732 [...] 4733 } 4734 #+end_example 4735 4736 It doesn't matter which input derivations are being used, the final out 4737 path must only depend on the declared hash. 4738 4739 What nix does is to create an intermediate string representation of the 4740 fixed-output content: 4741 4742 #+begin_example 4743 $ echo -n "fixed:out:sha256:f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb:" > mycontent.str 4744 $ sha256sum mycontent.str 4745 423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5798fb374dbcfd7639 myfile.str 4746 #+end_example 4747 4748 Then proceed as it was a normal derivation output path: 4749 4750 #+begin_example 4751 $ echo -n "output:out:sha256:423e6fdef56d53251c5939359c375bf21ea07aaa8d89ca5798fb374dbcfd7639:/nix/store:bar" > myfile.str 4752 $ nix-hash --type sha256 --truncate --base32 --flat myfile.str 4753 a00d5f71k0vp5a6klkls0mvr1f7sx6ch 4754 #+end_example 4755 4756 Hence, the store path only depends on the declared fixed-output hash. 4757 4758 ** Conclusion 4759 There are other types of store paths, but you get the idea. Nix first 4760 hashes the contents, then creates a string description, and the final 4761 store path is the hash of this string. 4762 4763 Also we've introduced some fundamentals, in particular the fact that Nix 4764 knows beforehand the out path of a derivation since it only depends on 4765 the inputs. We've also introduced fixed-output derivations which are 4766 especially used by the nixpkgs repository for downloading and verifying 4767 source tarballs. 4768 4769 ** Next pill 4770 ...we will introduce =stdenv=. In the previous pills we rolled our own 4771 =mkDerivation= convenience function for wrapping the builtin derivation, 4772 but the =nixpkgs= repository also has its own convenience functions for 4773 dealing with =autotools= projects and other build systems. 4774 4775 <<19-fundamentals-of-stdenv.html>> 4776 4777 * Fundamentals of Stdenv 4778 Welcome to the 19th Nix pill. In the previous 4779 [[#18-nix-store-paths.html][18th]] pill we dived into the algorithm used 4780 by Nix to compute the store paths, and also introduced fixed-output 4781 store paths. 4782 4783 This time we will instead look into =nixpkgs=, in particular one of its 4784 core derivations: =stdenv=. 4785 4786 The =stdenv= is not treated as a special derivation by Nix, but it's 4787 very important for the =nixpkgs= repository. It serves as a base for 4788 packaging software. It is used to pull in dependencies such as the GCC 4789 toolchain, GNU make, core utilities, patch and diff utilities, and so 4790 on: basic tools needed to compile a huge pile of software currently 4791 present in =nixpkgs=. 4792 4793 ** What is stdenv? 4794 First of all, =stdenv= is a derivation, and it's a very simple one: 4795 4796 #+begin_example 4797 $ nix-build '<nixpkgs>' -A stdenv 4798 /nix/store/k4jklkcag4zq4xkqhkpy156mgfm34ipn-stdenv 4799 $ ls -R result/ 4800 result/: 4801 nix-support/ setup 4802 4803 result/nix-support: 4804 propagated-user-env-packages 4805 #+end_example 4806 4807 It has just two files: =/setup= and 4808 =/nix-support/propagated-user-env-packages=. Don't worry about the 4809 latter. It's empty, in fact. The important file is =/setup=. 4810 4811 How can this simple derivation pull in all of the toolchain and basic 4812 tools needed to compile packages? Let's look at the runtime 4813 dependencies: 4814 4815 #+begin_example 4816 $ nix-store -q --references result 4817 /nix/store/3a45nb37s0ndljp68228snsqr3qsyp96-bzip2-1.0.6 4818 /nix/store/a457ywa1haa0sgr9g7a1pgldrg3s798d-coreutils-8.24 4819 /nix/store/zmd4jk4db5lgxb8l93mhkvr3x92g2sx2-bash-4.3-p39 4820 /nix/store/47sfpm2qclpqvrzijizimk4md1739b1b-gcc-wrapper-4.9.3 4821 ... 4822 #+end_example 4823 4824 How can it be? The package must be referring to those other packages 4825 somehow. In fact, they are hardcoded in the =/setup= file: 4826 4827 #+begin_example 4828 $ head result/setup 4829 export SHELL=/nix/store/zmd4jk4db5lgxb8l93mhkvr3x92g2sx2-bash-4.3-p39/bin/bash 4830 initialPath="/nix/store/a457ywa1haa0sgr9g7a1pgldrg3s798d-coreutils-8.24 ..." 4831 defaultNativeBuildInputs="/nix/store/sgwq15xg00xnm435gjicspm048rqg9y6-patchelf-0.8 ..." 4832 #+end_example 4833 4834 ** The setup file 4835 Remember our generic =builder.sh= in [[#08-generic-builders.html][Pill 4836 8]]? It sets up a basic =PATH=, unpacks the source and runs the usual 4837 =autotools= commands for us. 4838 4839 The 4840 [[https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh][stdenv 4841 setup file]] is exactly that. It sets up several environment variables 4842 like =PATH= and creates some helper bash functions to build a package. I 4843 invite you to read it. 4844 4845 The hardcoded toolchain and utilities are used to initially fill up the 4846 environment variables so that it's more pleasant to run common commands, 4847 similar to what we did with our builder with =baseInputs= and 4848 =buildInputs=. 4849 4850 The build with =stdenv= works in phases. Phases are like =unpackPhase=, 4851 =configurePhase=, =buildPhase=, =checkPhase=, =installPhase=, 4852 =fixupPhase=. You can see the default list in the =genericBuild= 4853 function. 4854 4855 What =genericBuild= does is just run these phases. Default phases are 4856 just bash functions. You can easily read them. 4857 4858 Every phase has hooks to run commands before and after the phase has 4859 been executed. Phases can be overwritten, reordered, whatever, it's just 4860 bash code. 4861 4862 How to use this file? Like our old builder. To test it, we enter a fake 4863 empty derivation, source the =stdenv= =setup=, unpack the hello sources 4864 and build it: 4865 4866 #+begin_example 4867 $ nix-shell -E 'derivation { name = "fake"; builder = "fake"; system = "x86_64-linux"; }' 4868 nix-shell$ unset PATH 4869 nix-shell$ source /nix/store/k4jklkcag4zq4xkqhkpy156mgfm34ipn-stdenv/setup 4870 nix-shell$ tar -xf hello-2.10.tar.gz 4871 nix-shell$ cd hello-2.10 4872 nix-shell$ configurePhase 4873 ... 4874 nix-shell$ buildPhase 4875 ... 4876 #+end_example 4877 4878 /I unset =PATH= to further show that the =stdenv= is sufficiently 4879 self-contained to build autotools packages that have no other 4880 dependencies./ 4881 4882 So we ran the =configurePhase= function and =buildPhase= function and 4883 they worked. These bash functions should be self-explanatory. You can 4884 read the code in the =setup= file. 4885 4886 ** How the setup file is built 4887 Until now we worked with plain bash scripts. What about the Nix side? 4888 The =nixpkgs= repository offers a useful function, like we did with our 4889 old builder. It is a wrapper around the raw derivation function which 4890 pulls in the =stdenv= for us, and runs =genericBuild=. It's 4891 [[https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/make-derivation.nix][stdenv.mkDerivation]]. 4892 4893 Note how =stdenv= is a derivation but it's also an attribute set which 4894 contains some other attributes, like =mkDerivation=. Nothing fancy here, 4895 just convenience. 4896 4897 Let's write a =hello.nix= expression using this newly discovered 4898 =stdenv=: 4899 4900 #+begin_example 4901 with import <nixpkgs> { }; 4902 stdenv.mkDerivation { 4903 name = "hello"; 4904 src = ./hello-2.10.tar.gz; 4905 } 4906 #+end_example 4907 4908 Don't be scared by the =with= expression. It pulls the =nixpkgs= 4909 repository into scope, so we can directly use =stdenv=. It looks very 4910 similar to the hello expression in [[#08-generic-builders.html][Pill 4911 8]]. 4912 4913 It builds, and runs fine: 4914 4915 #+begin_example 4916 $ nix-build hello.nix 4917 ... 4918 /nix/store/6y0mzdarm5qxfafvn2zm9nr01d1j0a72-hello 4919 $ result/bin/hello 4920 Hello, world! 4921 #+end_example 4922 4923 ** The stdenv.mkDerivation builder 4924 Let's take a look at the builder used by =mkDerivation=. You can read 4925 the code 4926 [[https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/make-derivation.nix][here 4927 in nixpkgs]]: 4928 4929 #+begin_example 4930 { 4931 # ... 4932 builder = attrs.realBuilder or shell; 4933 args = 4934 attrs.args or [ 4935 "-e" 4936 (attrs.builder or ./default-builder.sh) 4937 ]; 4938 stdenv = result; 4939 # ... 4940 } 4941 #+end_example 4942 4943 Also take a look at our old derivation wrapper in previous pills! The 4944 builder is bash (that shell variable), the argument to the builder 4945 (bash) is =default-builder.sh=, and then we add the environment variable 4946 =$stdenv= in the derivation which is the =stdenv= derivation. 4947 4948 You can open 4949 [[https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/default-builder.sh][default-builder.sh]] 4950 and see what it does: 4951 4952 #+begin_src sh 4953 source $stdenv/setup 4954 genericBuild 4955 #+end_src 4956 4957 It's what we did in [[#10-developing-with-nix-shell.html][Pill 10]] to 4958 make the derivations =nix-shell= friendly. When entering the shell, the 4959 setup file only sets up the environment without building anything. When 4960 doing =nix-build=, it actually runs the build process. 4961 4962 To get a clear understanding of the environment variables, look at the 4963 .drv of the hello derivation: 4964 4965 #+begin_example 4966 $ nix derivation show $(nix-instantiate hello.nix) 4967 warning: you did not specify '--add-root'; the result might be removed by the garbage collector 4968 { 4969 "/nix/store/abwj50lycl0m515yblnrvwyydlhhqvj2-hello.drv": { 4970 "outputs": { 4971 "out": { 4972 "path": "/nix/store/6y0mzdarm5qxfafvn2zm9nr01d1j0a72-hello" 4973 } 4974 }, 4975 "inputSrcs": [ 4976 "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh", 4977 "/nix/store/svc70mmzrlgq42m9acs0prsmci7ksh6h-hello-2.10.tar.gz" 4978 ], 4979 "inputDrvs": { 4980 "/nix/store/hcgwbx42mcxr7ksnv0i1fg7kw6jvxshb-bash-4.4-p19.drv": [ 4981 "out" 4982 ], 4983 "/nix/store/sfxh3ybqh97cgl4s59nrpi78kgcc8f3d-stdenv-linux.drv": [ 4984 "out" 4985 ] 4986 }, 4987 "platform": "x86_64-linux", 4988 "builder": "/nix/store/q1g0rl8zfmz7r371fp5p42p4acmv297d-bash-4.4-p19/bin/bash", 4989 "args": [ 4990 "-e", 4991 "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh" 4992 ], 4993 "env": { 4994 "buildInputs": "", 4995 "builder": "/nix/store/q1g0rl8zfmz7r371fp5p42p4acmv297d-bash-4.4-p19/bin/bash", 4996 "configureFlags": "", 4997 "depsBuildBuild": "", 4998 "depsBuildBuildPropagated": "", 4999 "depsBuildTarget": "", 5000 "depsBuildTargetPropagated": "", 5001 "depsHostBuild": "", 5002 "depsHostBuildPropagated": "", 5003 "depsTargetTarget": "", 5004 "depsTargetTargetPropagated": "", 5005 "name": "hello", 5006 "nativeBuildInputs": "", 5007 "out": "/nix/store/6y0mzdarm5qxfafvn2zm9nr01d1j0a72-hello", 5008 "propagatedBuildInputs": "", 5009 "propagatedNativeBuildInputs": "", 5010 "src": "/nix/store/svc70mmzrlgq42m9acs0prsmci7ksh6h-hello-2.10.tar.gz", 5011 "stdenv": "/nix/store/6kz2vbh98s2r1pfshidkzhiy2s2qdw0a-stdenv-linux", 5012 "system": "x86_64-linux" 5013 } 5014 } 5015 } 5016 #+end_example 5017 5018 It's so short I decided to paste it entirely above. The builder is bash, 5019 with =-e default-builder.sh= arguments. Then you can see the =src= and 5020 =stdenv= environment variables. 5021 5022 The last bit, the =unpackPhase= in the setup, is used to unpack the 5023 sources and enter the directory. Again, like we did in our old builder. 5024 5025 ** Conclusion 5026 The =stdenv= is the core of the =nixpkgs= repository. All packages use 5027 the =stdenv.mkDerivation= wrapper instead of the raw derivation. It does 5028 a bunch of operations for us and also sets up a pleasant build 5029 environment. 5030 5031 The overall process is simple: 5032 5033 - =nix-build= 5034 5035 - =bash -e default-builder.sh= 5036 5037 - =source $stdenv/setup= 5038 5039 - =genericBuild= 5040 5041 That's it. Everything you need to know about the stdenv phases is in the 5042 [[https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh][setup 5043 file]]. 5044 5045 Really, take your time to read that file. Don't forget that juicy docs 5046 are also available in the 5047 [[http://nixos.org/nixpkgs/manual/#chap-stdenv][nixpkgs manual]]. 5048 5049 ** Next pill... 5050 ...we will talk about how to add dependencies to our packages with 5051 =buildInputs= and =propagatedBuildInputs=, and influence downstream 5052 builds with setup hooks and env hooks. These concepts are crucial to how 5053 =nixpkgs= packages are composed. 5054 5055 <<20-basic-dependencies-and-hooks.html>> 5056 5057 * Basic Dependencies and Hooks 5058 Welcome to the 20th Nix pill. In the previous 5059 [[#19-fundamentals-of-stdenv.html][19th]] pill we introduced Nixpkgs' 5060 stdenv, including =setup.sh= script, =default-builder.sh= helper script, 5061 and =stdenv.mkDerivation= builder. We focused on how stdenv is put 5062 together, and how it's used, and a bit about the phases of 5063 =genericBuild=. 5064 5065 This time, we'll focus on the interaction of packages built with 5066 =stdenv.mkDerivation=. Packages need to depend on each other, of course. 5067 For this we have =buildInputs= and =propagatedBuildInputs= attributes. 5068 We've also found that dependencies sometimes need to influence their 5069 dependents in ways the dependents can't or shouldn't predict. For this 5070 we have setup hooks and env hooks. Together, these 4 concepts support 5071 almost all build-time package interactions. 5072 5073 Note: The complexity of the dependencies and hooks infrastructure has 5074 increased, over time, to support cross compilation. Once you learn the 5075 core concepts, you will be able to understand the extra complexity. As a 5076 starting point, you might want to refer to nixpkgs commit 5077 [[https://github.com/nixos/nixpkgs/tree/6675f0a52c0962042a1000c7f20e887d0d26ae25][6675f0a5]], 5078 the last version of stdenv without cross-compilation complexity. 5079 5080 ** The =buildInputs= Attribute 5081 For the simplest dependencies where the current package directly needs 5082 another, we use the =buildInputs= attribute. This is exactly the pattern 5083 used in our builder in [[#08-generic-builders.html][Pill 8]]. To demo 5084 this, let's build GNU Hello, and then another package which provides a 5085 shell script that =exec=s it. 5086 5087 #+begin_example 5088 let 5089 5090 nixpkgs = import <nixpkgs> { }; 5091 5092 inherit (nixpkgs) stdenv fetchurl which; 5093 5094 actualHello = stdenv.mkDerivation { 5095 name = "hello-2.3"; 5096 5097 src = fetchurl { 5098 url = "mirror://gnu/hello/hello-2.3.tar.bz2"; 5099 sha256 = "0c7vijq8y68bpr7g6dh1gny0bff8qq81vnp4ch8pjzvg56wb3js1"; 5100 }; 5101 }; 5102 5103 wrappedHello = stdenv.mkDerivation { 5104 name = "hello-wrapper"; 5105 5106 buildInputs = [ 5107 actualHello 5108 which 5109 ]; 5110 5111 unpackPhase = "true"; 5112 5113 installPhase = '' 5114 mkdir -p "$out/bin" 5115 echo "#! ${stdenv.shell}" >> "$out/bin/hello" 5116 echo "exec $(which hello)" >> "$out/bin/hello" 5117 chmod 0755 "$out/bin/hello" 5118 ''; 5119 }; 5120 in 5121 wrappedHello 5122 #+end_example 5123 5124 Notice that the wrappedHello derivation finds the =hello= binary from 5125 the =PATH=. This works because stdenv contains something like: 5126 5127 #+begin_src sh 5128 pkgs="" 5129 for i in $buildInputs; do 5130 findInputs $i 5131 done 5132 #+end_src 5133 5134 where =findInputs= is defined like: 5135 5136 #+begin_src sh 5137 findInputs() { 5138 local pkg=$1 5139 5140 ## Don't need to repeat already processed package 5141 case $pkgs in 5142 *\ $pkg\ *) 5143 return 0 5144 ;; 5145 esac 5146 5147 pkgs="$pkgs $pkg " 5148 5149 ## More goes here in reality that we can ignore for now. 5150 } 5151 #+end_src 5152 5153 then after this is run: 5154 5155 #+begin_src sh 5156 for i in $pkgs; do 5157 addToEnv $i 5158 done 5159 #+end_src 5160 5161 where =addToEnv= is defined like: 5162 5163 #+begin_src sh 5164 addToEnv() { 5165 local pkg=$1 5166 5167 if test -d $1/bin; then 5168 addToSearchPath _PATH $1/bin 5169 fi 5170 5171 ## More goes here in reality that we can ignore for now. 5172 } 5173 #+end_src 5174 5175 The =addToSearchPath= call adds =$1/bin= to =_PATH= if the former exists 5176 (code 5177 [[https://github.com/NixOS/nixpkgs/blob/6675f0a52c0962042a1000c7f20e887d0d26ae25/pkgs/stdenv/generic/setup.sh#L60-L73][here]]). 5178 Once all the packages in =buildInputs= have been processed, then content 5179 of =_PATH= is added to =PATH=, as follows: 5180 5181 #+begin_src sh 5182 PATH="${_PATH-}${_PATH:+${PATH:+:}}$PATH" 5183 #+end_src 5184 5185 With the real =hello= on the =PATH=, the =installPhase= should hopefully 5186 make sense. 5187 5188 ** The =propagatedBuildInputs= Attribute 5189 The =buildInputs= covers direct dependencies, but what about indirect 5190 dependencies where one package needs a second package which needs a 5191 third? Nix itself handles this just fine, understanding various 5192 dependency closures as covered in previous builds. But what about the 5193 conveniences that =buildInputs= provides, namely accumulating in =pkgs= 5194 environment variable and inclusion of =«pkg»/bin= directories on the 5195 =PATH=? For this, stdenv provides the =propagatedBuildInputs=: 5196 5197 #+begin_example 5198 let 5199 5200 nixpkgs = import <nixpkgs> { }; 5201 5202 inherit (nixpkgs) stdenv fetchurl which; 5203 5204 actualHello = stdenv.mkDerivation { 5205 name = "hello-2.3"; 5206 5207 src = fetchurl { 5208 url = "mirror://gnu/hello/hello-2.3.tar.bz2"; 5209 sha256 = "0c7vijq8y68bpr7g6dh1gny0bff8qq81vnp4ch8pjzvg56wb3js1"; 5210 }; 5211 }; 5212 5213 intermediary = stdenv.mkDerivation { 5214 name = "middle-man"; 5215 5216 propagatedBuildInputs = [ actualHello ]; 5217 5218 unpackPhase = "true"; 5219 5220 installPhase = '' 5221 mkdir -p "$out" 5222 ''; 5223 }; 5224 5225 wrappedHello = stdenv.mkDerivation { 5226 name = "hello-wrapper"; 5227 5228 buildInputs = [ 5229 intermediary 5230 which 5231 ]; 5232 5233 unpackPhase = "true"; 5234 5235 installPhase = '' 5236 mkdir -p "$out/bin" 5237 echo "#! ${stdenv.shell}" >> "$out/bin/hello" 5238 echo "exec $(which hello)" >> "$out/bin/hello" 5239 chmod 0755 "$out/bin/hello" 5240 ''; 5241 }; 5242 in 5243 wrappedHello 5244 #+end_example 5245 5246 See how the intermediate package has a =propagatedBuildInputs= 5247 dependency, but the wrapper only needs a =buildInputs= dependency on the 5248 intermediary. 5249 5250 How does this work? You might think we do something in Nix, but actually 5251 it's done not at eval time but at build time in bash. let's look at part 5252 of the =fixupPhase= of stdenv: 5253 5254 #+begin_src sh 5255 fixupPhase() { 5256 5257 ## Elided 5258 5259 if test -n "$propagatedBuildInputs"; then 5260 mkdir -p "$out/nix-support" 5261 echo "$propagatedBuildInputs" > "$out/nix-support/propagated-build-inputs" 5262 fi 5263 5264 ## Elided 5265 5266 } 5267 #+end_src 5268 5269 This dumps the propagated build inputs in a so-named file in 5270 =$out/nix-support/=. Then, back in =findInputs= look at the lines at the 5271 bottom we elided before: 5272 5273 #+begin_src sh 5274 findInputs() { 5275 local pkg=$1 5276 5277 ## More goes here in reality that we can ignore for now. 5278 5279 if test -f $pkg/nix-support/propagated-build-inputs; then 5280 for i in $(cat $pkg/nix-support/propagated-build-inputs); do 5281 findInputs $i 5282 done 5283 fi 5284 } 5285 #+end_src 5286 5287 See how =findInputs= is actually recursive, looking at the propagated 5288 build inputs of each dependency, and those dependencies' propagated 5289 build inputs, etc. 5290 5291 We actually simplified the =findInputs= call site from before; 5292 =propagatedBuildInputs= is also looped over in reality: 5293 5294 #+begin_src sh 5295 pkgs="" 5296 for i in $buildInputs $propagatedBuildInputs; do 5297 findInputs $i 5298 done 5299 #+end_src 5300 5301 This demonstrates an important point. For the /current/ package alone, 5302 it doesn't matter whether a dependency is propagated or not. It will be 5303 processed the same way: called with =findInputs= and =addToEnv=. (The 5304 packages discovered by =findInputs=, which are also accumulated in 5305 =pkgs= and passed to =addToEnv=, are also the same in both cases.) 5306 Downstream however, it certainly does matter because only the propagated 5307 immediate dependencies are put in the 5308 =$out/nix-support/propagated-build-inputs=. 5309 5310 ** Setup Hooks 5311 As we mentioned above, sometimes dependencies need to influence the 5312 packages that use them in ways other than just /being/ a dependency. 5313 ^{[[#20-basic-dependencies-and-hooks.html#fn1][1]]} 5314 =propagatedBuildInputs= can actually be seen as an example of this: 5315 packages using that are effectively "injecting" those dependencies as 5316 extra =buildInputs= in their downstream dependents. But in general, a 5317 dependency might affect the packages it depends on in arbitrary ways. 5318 /Arbitrary/ is the key word here. We could teach =setup.sh= things about 5319 upstream packages like =«pkg»/nix-support/propagated-build-inputs=, but 5320 not arbitrary interactions. 5321 5322 Setup hooks are the basic building block we have for this. In nixpkgs, a 5323 "hook" is basically a bash callback, and a setup hook is no exception. 5324 Let's look at the last part of =findInputs= we haven't covered: 5325 5326 #+begin_src sh 5327 findInputs() { 5328 local pkg=$1 5329 5330 ## More goes here in reality that we can ignore for now. 5331 5332 if test -f $pkg/nix-support/setup-hook; then 5333 source $pkg/nix-support/setup-hook 5334 fi 5335 5336 ## More goes here in reality that we can ignore for now. 5337 5338 } 5339 #+end_src 5340 5341 If a package includes the path =«pkg»/nix-support/setup-hook=, it will 5342 be sourced by any stdenv-based build including that as a dependency. 5343 5344 This is strictly more general than any of the other mechanisms 5345 introduced in this chapter. For example, try writing a setup hook that 5346 has the same effect as a /propagatedBuildInputs/ entry. One can almost 5347 think of this as an escape hatch around Nix's normal isolation 5348 guarantees, and the principle that dependencies are immutable and inert. 5349 We're not actually doing something unsafe or modifying dependencies, but 5350 we are allowing arbitrary ad-hoc behavior. For this reason, setup-hooks 5351 should only be used as a last resort. 5352 5353 ** Environment Hooks 5354 As a final convenience, we have environment hooks. Recall in 5355 [[#12-inputs-design-pattern.html][Pill 12]] how we created 5356 =NIX_CFLAGS_COMPILE= for =-I= flags and =NIX_LDFLAGS= for =-L= flags, in 5357 a similar manner to how we prepared the =PATH=. One point of ugliness 5358 was how anti-modular this was. It makes sense to build the =PATH= in a 5359 generic builder, because the =PATH= is used by the shell, and the 5360 generic builder is intrinsically tied to the shell. But =-I= and =-L= 5361 flags are only relevant to the C compiler. The stdenv isn't wedded to 5362 including a C compiler (though it does by default), and there are other 5363 compilers too which may take completely different flags. 5364 5365 As a first step, we can move that logic to a setup hook on the C 5366 compiler; indeed that's just what we do in CC Wrapper. 5367 ^{[[#20-basic-dependencies-and-hooks.html#fn2][2]]} But this pattern 5368 comes up fairly often, so somebody decided to add some helper support to 5369 reduce boilerplate. 5370 5371 The other half of =addToEnv= is: 5372 5373 #+begin_src sh 5374 addToEnv() { 5375 local pkg=$1 5376 5377 ## More goes here in reality that we can ignore for now. 5378 5379 # Run the package-specific hooks set by the setup-hook scripts. 5380 for i in "${envHooks[@]}"; do 5381 $i $pkg 5382 done 5383 } 5384 #+end_src 5385 5386 Functions listed in =envHooks= are applied to every package passed to 5387 =addToEnv=. One can write a setup hook like: 5388 5389 #+begin_src sh 5390 anEnvHook() { 5391 local pkg=$1 5392 5393 echo "I'm depending on \"$pkg\"" 5394 } 5395 5396 envHooks+=(anEnvHook) 5397 #+end_src 5398 5399 and if one dependency has that setup hook then all of them will be so 5400 =echo=ed. Allowing dependencies to learn about their /sibling/ 5401 dependencies is exactly what compilers need. 5402 5403 ** Next pill... 5404 ...I'm not sure! We could talk about the additional dependency types and 5405 hooks which cross compilation necessitates, building on our knowledge 5406 here to cover stdenv as it works today. We could talk about how nixpkgs 5407 is bootstrapped. Or we could talk about how =localSystem= and 5408 =crossSystem= are elaborated into the =buildPlatform=, =hostPlatform=, 5409 and =targetPlatform= each bootstrapping stage receives. Let us know 5410 which most interests you! 5411 5412 <<20-basic-dependencies-and-hooks.html#fn1>> 5413 ^{1} 5414 We can now be precise and consider what =addToEnv= does alone the 5415 minimal treatment of a dependency: i.e. a package that is /just/ a 5416 dependency would /only/ have =addToEnv= applied to it. 5417 5418 <<20-basic-dependencies-and-hooks.html#fn2>> 5419 ^{2} 5420 It was called 5421 [[https://github.com/NixOS/nixpkgs/tree/6675f0a52c0962042a1000c7f20e887d0d26ae25/pkgs/build-support/gcc-wrapper][GCC 5422 Wrapper]] in the version of nixpkgs suggested for following along in 5423 this pill; Darwin and Clang support hadn't yet motivated the rename.