aleph-012-render.md
1 # RFC-012: render.nix - Typed Shell Scripts 2 3 ## Status 4 5 Draft 6 7 ## Abstract 8 9 A type system, LSP, formatter, and linter for bash scripts embedded in Nix. Enforces a safe subset of bash with no heredocs, no bare commands, no eval. Provides IDE features: hover types, go-to-definition, completions. Generates documentation from inferred types. 10 11 ## Motivation 12 13 ### The Problem 14 15 Bash scripts in Nix are write-only: 16 17 ```nix 18 pkgs.writeShellScriptBin "deploy" '' 19 PORT=''${PORT:-8080} 20 cat << EOF > /tmp/config.json 21 { 22 "port": $PORT, 23 "host": "''${HOST:-localhost}" 24 } 25 EOF 26 curl -X POST http://localhost:$PORT/deploy -d @/tmp/config.json 27 '' 28 ``` 29 30 What's wrong: 31 1. **Heredoc** - Three escaping systems (Nix, bash, JSON). Injection bugs. Untyped. 32 2. **Bare command** - `curl` could be `/usr/bin/curl` or nothing. Not reproducible. 33 3. **No types** - Is `PORT` a string or int? What's required? What's optional? 34 4. **No tooling** - No hover, no go-to-def, no completions, no docs. 35 36 ### The Solution 37 38 Enforce a typed subset. Reject the bad patterns at commit time. Provide IDE support for the good patterns. 39 40 ```nix 41 pkgs.writeShellScriptBin "deploy" '' 42 # render.nix infers: 43 # PORT : Int = 8080 44 # HOST : String = "localhost" 45 46 PORT="''${PORT:-8080}" 47 HOST="''${HOST:-localhost}" 48 49 ${pkgs.curl}/bin/curl -X POST \ 50 "http://$HOST:$PORT/deploy" \ 51 -d "$(${pkgs.dhall-json}/bin/dhall-to-json <<< '{ status = "starting" }')" 52 '' 53 ``` 54 55 What's right: 56 1. **No heredoc** - Dhall for structured data, type-checked 57 2. **Store paths** - `${pkgs.curl}` pins the exact version 58 3. **Inferred types** - PORT is Int (from `8080`), HOST is String 59 4. **Full tooling** - Hover, completions, docs, all work 60 61 ## Design 62 63 ### Architecture 64 65 ``` 66 ┌─────────────────────────────────────────────────────────────┐ 67 │ render.nix │ 68 ├─────────────────────────────────────────────────────────────┤ 69 │ │ 70 │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ 71 │ │ Nix Parser │───▶│ Bash Parser │───▶│ Type │ │ 72 │ │ (hnix) │ │ (ShellCheck)│ │ Inference │ │ 73 │ └─────────────┘ └─────────────┘ └─────────────┘ │ 74 │ │ │ │ │ 75 │ ▼ ▼ ▼ │ 76 │ ┌─────────────────────────────────────────────────┐ │ 77 │ │ Unified AST │ │ 78 │ │ - Nix expressions with interpolation sites │ │ 79 │ │ - Bash AST with store path references │ │ 80 │ │ - Type annotations on all variables │ │ 81 │ └─────────────────────────────────────────────────┘ │ 82 │ │ │ │ │ 83 │ ▼ ▼ ▼ │ 84 │ ┌───────────┐ ┌───────────┐ ┌───────────┐ │ 85 │ │ LSP │ │ Formatter │ │ Linter │ │ 86 │ │ Server │ │ │ │ │ │ 87 │ └───────────┘ └───────────┘ └───────────┘ │ 88 │ │ │ │ │ 89 │ ▼ ▼ ▼ │ 90 │ ┌───────────┐ ┌───────────┐ ┌───────────┐ │ 91 │ │ Hover │ │ Pretty │ │ Pre-commit│ │ 92 │ │ Go-to-def│ │ Print + │ │ Gate │ │ 93 │ │ Complete │ │ Types │ │ │ │ 94 │ └───────────┘ └───────────┘ └───────────┘ │ 95 │ │ 96 └─────────────────────────────────────────────────────────────┘ 97 ``` 98 99 ### Type System 100 101 #### Base Types 102 103 ``` 104 Type ::= TInt -- integers: 8080, -1, 0 105 | TString -- strings: "hello", localhost 106 | TBool -- booleans: true, false 107 | TPath -- store paths: /nix/store/... 108 | TArray Type -- arrays: ("a" "b" "c") 109 | TUnknown -- not yet inferred 110 ``` 111 112 #### Inference Rules 113 114 ``` 115 ───────────────────────────────────────────────────── 116 VAR="${VAR:-8080}" ⊢ VAR : TInt 117 118 ───────────────────────────────────────────────────── 119 VAR="${VAR:-true}" ⊢ VAR : TBool 120 121 ───────────────────────────────────────────────────── 122 VAR="${VAR:-hello}" ⊢ VAR : TString 123 124 ───────────────────────────────────────────────────── 125 VAR="${VAR:?}" ⊢ VAR : TString, required 126 127 ───────────────────────────────────────────────────── 128 VAR="${VAR:-$OTHER}" ⊢ VAR : typeof(OTHER) 129 130 ───────────────────────────────────────────────────── 131 ${pkgs.foo}/bin/bar ⊢ StorePath(/nix/store/...-foo) 132 ``` 133 134 #### Constraint Generation 135 136 Each pattern generates constraints. Unification solves them. 137 138 ```haskell 139 data Constraint 140 = Type :~: Type -- equality 141 | Required Text -- var must be provided 142 | StorePath Text Path -- interpolation resolves to path 143 | BareCommand Text Span -- error: unresolved command 144 | Heredoc Span -- error: heredoc detected 145 | Eval Span -- error: eval detected 146 | Backtick Span -- error: backtick detected 147 ``` 148 149 ### Forbidden Constructs 150 151 These are errors, not warnings. No override flag. 152 153 #### 1. Heredocs 154 155 ```bash 156 # FORBIDDEN 157 cat << EOF 158 {"port": $PORT} 159 EOF 160 161 # FORBIDDEN 162 cat << 'EOF' 163 literal text 164 EOF 165 166 # FORBIDDEN 167 cat <<< "here string with $VAR" 168 ``` 169 170 **Why**: Three escaping systems. Injection vulnerabilities. Untyped output. 171 172 **Instead**: 173 ```bash 174 # Dhall for structured data 175 ${dhall-json}/bin/dhall-to-json <<< '{ port = 8080 }' 176 177 # printf for simple strings 178 printf '{"port": %d}\n' "$PORT" 179 180 # Or generate at Nix level, not bash level 181 ``` 182 183 #### 2. Bare Commands 184 185 ```bash 186 # FORBIDDEN 187 curl http://example.com 188 grep "pattern" file 189 jq '.foo' 190 191 # ALLOWED 192 ${pkgs.curl}/bin/curl http://example.com 193 ${pkgs.gnugrep}/bin/grep "pattern" file 194 ${pkgs.jq}/bin/jq '.foo' 195 ``` 196 197 **Why**: Non-reproducible. Depends on `$PATH`. Version skew. 198 199 #### 3. eval 200 201 ```bash 202 # FORBIDDEN 203 eval "$DYNAMIC_CODE" 204 eval "$(generate_commands)" 205 ``` 206 207 **Why**: Unanalyzable. Security risk. Type system can't help. 208 209 #### 4. Backticks 210 211 ```bash 212 # FORBIDDEN 213 result=`some_command` 214 215 # ALLOWED 216 result=$(some_command) 217 ``` 218 219 **Why**: Deprecated syntax. Nesting issues. Confusing escaping. 220 221 #### 5. Unquoted Nix Interpolations 222 223 ```nix 224 # FORBIDDEN (in Nix) 225 pkgs.writeShellScriptBin "foo" '' 226 ${someVariable} # might not be a store path 227 '' 228 229 # ALLOWED 230 pkgs.writeShellScriptBin "foo" '' 231 ${pkgs.curl}/bin/curl # definitely a store path 232 '' 233 ``` 234 235 **Why**: Interpolations must resolve to store paths, not arbitrary strings. 236 237 ### LSP Server 238 239 #### Capabilities 240 241 | Feature | Nix Files | Bash in Nix | 242 |---------|-----------|-------------| 243 | Hover | Package info, type | Var type, default, required | 244 | Go to definition | Package def | Var definition | 245 | Find references | Usages | Var usages | 246 | Completion | Package names | Env vars, store paths | 247 | Diagnostics | Type errors | Heredocs, bare cmds | 248 | Code actions | Add store path | Fix bare command | 249 250 #### Hover Examples 251 252 Hovering over `${pkgs.curl}`: 253 ``` 254 curl 8.5.0 255 256 Outputs: 257 bin: /nix/store/abc...-curl-8.5.0-bin 258 dev: /nix/store/def...-curl-8.5.0-dev 259 260 From: nixpkgs#curl 261 ``` 262 263 Hovering over `$PORT`: 264 ``` 265 PORT : Int 266 267 Default: 8080 268 Required: no 269 Defined: line 5 270 Used: lines 8, 12, 15 271 ``` 272 273 Hovering over `curl` (bare command): 274 ``` 275 ⚠ Bare command: curl 276 277 This command is not pinned to a store path. 278 Use: ${pkgs.curl}/bin/curl 279 280 [Quick Fix: Add store path] 281 ``` 282 283 #### Diagnostics 284 285 ``` 286 error[E001]: heredoc not allowed 287 --> deploy.nix:5:3 288 | 289 5 | cat << EOF 290 | ^^^^^^^^^^ heredocs are forbidden 291 | 292 = help: use Dhall for structured output 293 = help: use printf for simple strings 294 295 error[E002]: bare command 296 --> deploy.nix:8:3 297 | 298 8 | curl http://example.com 299 | ^^^^ command not pinned to store path 300 | 301 = help: use ${pkgs.curl}/bin/curl 302 303 error[E003]: eval not allowed 304 --> deploy.nix:12:3 305 | 306 12 | eval "$cmd" 307 | ^^^^ eval is forbidden 308 | 309 = help: refactor to avoid dynamic code execution 310 ``` 311 312 ### Formatter 313 314 Takes bash (in Nix), infers types, outputs bash with type annotations. 315 316 #### Input 317 318 ```nix 319 pkgs.writeShellScriptBin "deploy" '' 320 PORT=''${PORT:-8080} 321 HOST=''${HOST:-localhost} 322 DB=''${DB_URL:?} 323 ${pkgs.curl}/bin/curl "http://$HOST:$PORT" 324 '' 325 ``` 326 327 #### Output 328 329 ```nix 330 pkgs.writeShellScriptBin "deploy" '' 331 # @env PORT : Int = 8080 332 # @env HOST : String = "localhost" 333 # @env DB_URL : String (required) 334 # @uses curl : /nix/store/abc...-curl-8.5.0 335 336 PORT="''${PORT:-8080}" 337 HOST="''${HOST:-localhost}" 338 DB="''${DB_URL:?}" 339 340 ${pkgs.curl}/bin/curl "http://$HOST:$PORT" 341 '' 342 ``` 343 344 The annotations are: 345 - Generated, not source of truth 346 - Re-derived on each format 347 - Machine-readable for doc generation 348 349 #### Formatting Rules 350 351 1. **Header block** with type signatures 352 2. **Blank line** after header 353 3. **Variable declarations** grouped at top 354 4. **Store path comment** showing resolved paths 355 5. **Consistent quoting** - always quote `"${VAR}"` 356 6. **Consistent spacing** - one space around operators 357 358 ### Documentation Generator 359 360 Reads formatted scripts, emits markdown/HTML. 361 362 #### Output 363 364 ```markdown 365 # deploy 366 367 Deployment script for the API service. 368 369 ## Environment Variables 370 371 | Name | Type | Default | Required | Description | 372 |------|------|---------|----------|-------------| 373 | `PORT` | Int | `8080` | No | Server port | 374 | `HOST` | String | `"localhost"` | No | Server host | 375 | `DB_URL` | String | - | **Yes** | Database connection URL | 376 377 ## Dependencies 378 379 | Package | Version | Path | 380 |---------|---------|------| 381 | curl | 8.5.0 | `/nix/store/abc...-curl-8.5.0` | 382 383 ## Source 384 385 \`\`\`bash 386 PORT="${PORT:-8080}" 387 HOST="${HOST:-localhost}" 388 DB="${DB_URL:?}" 389 390 ${curl}/bin/curl "http://$HOST:$PORT" 391 \`\`\` 392 ``` 393 394 ### Pre-commit Hook 395 396 Enforces everything. No bypass. 397 398 ```yaml 399 # .pre-commit-config.yaml 400 repos: 401 - repo: local 402 hooks: 403 - id: render-check 404 name: render.nix typecheck 405 entry: render check 406 language: system 407 files: '\.nix$' 408 pass_filenames: true 409 ``` 410 411 The hook: 412 1. Finds all `writeShellScript*` calls 413 2. Extracts and parses bash 414 3. Runs type inference 415 4. Checks for forbidden constructs 416 5. Fails commit if any errors 417 418 **No `--force` flag.** No `# render-ignore` comments. Fix the code or don't commit. 419 420 ### Integration with Existing Tools 421 422 #### treefmt 423 424 ```nix 425 # treefmt.nix 426 { 427 programs.render = { 428 enable = true; 429 includes = [ "*.nix" ]; 430 }; 431 } 432 ``` 433 434 #### nil/nixd (Nix LSP) 435 436 render.nix can run as a child process of nil/nixd, providing bash-specific features while the parent handles Nix. 437 438 Communication via LSP's `workspace/executeCommand` or a custom protocol. 439 440 #### ShellCheck 441 442 We use ShellCheck's parser but add: 443 - Type inference 444 - Store path tracking 445 - Stricter rules (no heredocs) 446 - Nix integration 447 448 ShellCheck warnings become errors where appropriate. 449 450 ## Implementation 451 452 ### Phase 1: Linter (Blocking) 453 454 Detect and reject: 455 - [ ] Heredocs (`<<`, `<<<`) 456 - [ ] Bare commands 457 - [ ] `eval` 458 - [ ] Backticks 459 - [ ] Unquoted Nix interpolations 460 461 This is the enforcement gate. Must work before anything else. 462 463 ### Phase 2: Type Inference (Foundation) 464 465 - [ ] Parse Nix to find embedded bash 466 - [ ] Track interpolation sites and their Nix values 467 - [ ] Infer env var types from patterns 468 - [ ] Unification with error recovery 469 - [ ] Schema output (JSON) 470 471 ### Phase 3: Formatter (Developer Experience) 472 473 - [ ] Pretty printer for bash-in-Nix 474 - [ ] Type annotation comments 475 - [ ] Store path comments 476 - [ ] Integration with treefmt 477 478 ### Phase 4: LSP (Full IDE) 479 480 - [ ] Hover for types 481 - [ ] Go to definition 482 - [ ] Find references 483 - [ ] Completions 484 - [ ] Diagnostics 485 - [ ] Code actions 486 487 ### Phase 5: Documentation 488 489 - [ ] Markdown generator 490 - [ ] HTML generator 491 - [ ] Integration with mdbook 492 - [ ] Automatic README sections 493 494 ## File Structure 495 496 ``` 497 nix/render/ 498 ├── app/ 499 │ ├── render.hs # CLI entry point 500 │ └── lsp.hs # LSP server entry point 501 ├── lib/ 502 │ ├── Render.hs # Main module 503 │ ├── Render/ 504 │ │ ├── Types.hs # Core types 505 │ │ ├── Nix/ 506 │ │ │ ├── Parse.hs # Nix parser (hnix wrapper) 507 │ │ │ ├── Extract.hs # Find bash in Nix 508 │ │ │ └── Interp.hs # Track interpolations 509 │ │ ├── Bash/ 510 │ │ │ ├── Parse.hs # Bash parser (ShellCheck) 511 │ │ │ ├── Facts.hs # Extract facts 512 │ │ │ └── Patterns.hs# Pattern matchers 513 │ │ ├── Infer/ 514 │ │ │ ├── Constraint.hs 515 │ │ │ └── Unify.hs 516 │ │ ├── Lint/ 517 │ │ │ ├── Heredoc.hs 518 │ │ │ ├── BareCmd.hs 519 │ │ │ ├── Eval.hs 520 │ │ │ └── Policy.hs 521 │ │ ├── Format/ 522 │ │ │ ├── Pretty.hs 523 │ │ │ └── Annotate.hs 524 │ │ ├── LSP/ 525 │ │ │ ├── Server.hs 526 │ │ │ ├── Hover.hs 527 │ │ │ ├── Complete.hs 528 │ │ │ └── Diagnostic.hs 529 │ │ └── Doc/ 530 │ │ ├── Markdown.hs 531 │ │ └── Html.hs 532 └── test/ 533 ├── Props.hs # Property tests 534 └── Golden/ # Golden tests 535 ``` 536 537 ## CLI 538 539 ``` 540 render - typed shell scripts 541 542 USAGE: 543 render <COMMAND> 544 545 COMMANDS: 546 check Typecheck and lint (exit 1 on errors) 547 fmt Format with type annotations 548 infer Output inferred schema (JSON) 549 docs Generate documentation 550 lsp Run LSP server 551 552 OPTIONS: 553 --help Show help 554 --version Show version 555 556 EXAMPLES: 557 render check *.nix # Lint all Nix files 558 render fmt --write *.nix # Format in place 559 render infer deploy.nix # Show schema 560 render docs --out docs/ *.nix # Generate docs 561 ``` 562 563 ## Examples 564 565 ### Before: Untyped Heredoc Hell 566 567 ```nix 568 { pkgs }: 569 570 pkgs.writeShellScriptBin "deploy" '' 571 PORT=''${PORT:-8080} 572 HOST=''${HOST:-localhost} 573 574 cat << EOF > /tmp/config.json 575 { 576 "server": { 577 "port": $PORT, 578 "host": "$HOST" 579 }, 580 "database": { 581 "url": "''${DB_URL}" 582 } 583 } 584 EOF 585 586 curl -X POST http://admin:''${ADMIN_PASS}@$HOST:$PORT/deploy \ 587 -d @/tmp/config.json 588 '' 589 ``` 590 591 Problems: 592 - Heredoc with mixed escaping 593 - Password in URL (visible in logs) 594 - Bare `curl` command 595 - Config file in /tmp (race condition) 596 - No type information 597 598 ### After: Typed and Safe 599 600 ```nix 601 { pkgs }: 602 603 pkgs.writeShellScriptBin "deploy" '' 604 # @env PORT : Int = 8080 605 # @env HOST : String = "localhost" 606 # @env DB_URL : String (required) 607 # @env ADMIN_PASS : String (required) 608 # @uses curl dhall-json 609 610 set -euo pipefail 611 612 PORT="''${PORT:-8080}" 613 HOST="''${HOST:-localhost}" 614 DB_URL="''${DB_URL:?}" 615 ADMIN_PASS="''${ADMIN_PASS:?}" 616 617 config=$(${pkgs.dhall-json}/bin/dhall-to-json << 'DHALL' 618 { server = { port = env:PORT, host = env:HOST } 619 , database = { url = env:DB_URL } 620 } 621 DHALL 622 ) 623 624 ${pkgs.curl}/bin/curl \ 625 --netrc-file <(printf 'machine %s login admin password %s\n' "$HOST" "$ADMIN_PASS") \ 626 -X POST "http://$HOST:$PORT/deploy" \ 627 -H 'Content-Type: application/json' \ 628 -d "$config" 629 '' 630 ``` 631 632 Improvements: 633 - Dhall for config (typed, no escaping issues) 634 - netrc for credentials (not in URL) 635 - Store path for curl 636 - Type annotations (generated) 637 - Config in variable (no temp file) 638 639 Wait, that still has a heredoc (`<< 'DHALL'`). Let me fix: 640 641 ### After: Actually Correct 642 643 ```nix 644 { pkgs }: 645 646 let 647 # Config schema in Nix/Dhall, not bash 648 configDhall = pkgs.writeText "config.dhall" '' 649 { server = { port = env:PORT, host = env:HOST } 650 , database = { url = env:DB_URL } 651 } 652 ''; 653 in 654 pkgs.writeShellScriptBin "deploy" '' 655 # @env PORT : Int = 8080 656 # @env HOST : String = "localhost" 657 # @env DB_URL : String (required) 658 # @env ADMIN_PASS : String (required) 659 # @uses curl dhall-json 660 661 set -euo pipefail 662 663 PORT="''${PORT:-8080}" 664 HOST="''${HOST:-localhost}" 665 DB_URL="''${DB_URL:?}" 666 ADMIN_PASS="''${ADMIN_PASS:?}" 667 668 export PORT HOST DB_URL # for Dhall env: references 669 670 config=$(${pkgs.dhall-json}/bin/dhall-to-json --file ${configDhall}) 671 672 ${pkgs.curl}/bin/curl \ 673 -u "admin:$ADMIN_PASS" \ 674 -X POST "http://$HOST:$PORT/deploy" \ 675 -H 'Content-Type: application/json' \ 676 -d "$config" 677 '' 678 ``` 679 680 Now: 681 - Config is a separate Dhall file (Nix handles interpolation) 682 - No heredocs in bash at all 683 - Credentials via `-u` (still visible in ps, but better than URL) 684 - All commands are store paths 685 - Type annotations generated from inference 686 687 ## FAQ 688 689 ### Why no heredocs at all? 690 691 Heredocs mix three languages (Nix, bash, target format). Each has its own escaping rules. The result is always wrong in subtle ways. 692 693 ```nix 694 '' 695 cat << EOF 696 {"path": "''${PATH//\\/\\\\}"} 697 EOF 698 '' 699 ``` 700 701 Is this correct? Who knows. Use Dhall. 702 703 ### What about simple cases? 704 705 ```bash 706 cat << EOF 707 Hello, $NAME 708 EOF 709 ``` 710 711 Still no. Use: 712 713 ```bash 714 printf 'Hello, %s\n' "$NAME" 715 ``` 716 717 Or: 718 719 ```bash 720 echo "Hello, $NAME" 721 ``` 722 723 Heredocs are never necessary. They're a convenience that costs clarity. 724 725 ### What if I need multiline strings? 726 727 ```nix 728 let 729 message = pkgs.writeText "message.txt" '' 730 Line 1 731 Line 2 732 Line 3 733 ''; 734 in 735 pkgs.writeShellScriptBin "foo" '' 736 cat ${message} 737 '' 738 ``` 739 740 Generate the content in Nix. Reference it in bash. 741 742 ### What about <<< (here strings)? 743 744 Also forbidden. Use: 745 746 ```bash 747 echo "string" | command 748 # or 749 command < <(echo "string") 750 # or 751 printf '%s' "string" | command 752 ``` 753 754 ### Can I disable checks for legacy code? 755 756 No. 757 758 ### Can I add ignore comments? 759 760 No. 761 762 ### Can I use --force? 763 764 No. 765 766 Fix the code. 767 768 ## References 769 770 - [ShellCheck](https://www.shellcheck.net/) - Bash parser and linter 771 - [hnix](https://github.com/haskell-nix/hnix) - Nix parser in Haskell 772 - [Dhall](https://dhall-lang.org/) - Typed configuration language 773 - [LSP Specification](https://microsoft.github.io/language-server-protocol/) 774 - [resholve](https://github.com/abathur/resholve) - Store path resolution for bash