bare-metal-deployment.md
1 # Bare-Metal Radicle Seed Node Deployment Guide 2 ## Project Auxo - Sovereign Infrastructure 3 4 **Version**: 1.0 5 **Last Updated**: 2025-01-11 6 **Audience**: Infrastructure Team 7 8 --- 9 10 ## Overview 11 12 This guide covers deploying Radicle seed nodes on Auxo-owned hardware for maximum sovereignty and control. 13 14 ### Design Principles 15 - **Full Sovereignty**: No cloud providers, complete control 16 - **Geographic Distribution**: Multiple physical locations for redundancy 17 - **Security First**: Physical and network security 18 - **Cost Effective**: One-time hardware investment vs. ongoing cloud fees 19 - **Maintainable**: Automated configuration and monitoring 20 21 --- 22 23 ## Hardware Requirements 24 25 ### Minimum Specifications (per seed node) 26 - **CPU**: 2 cores (4 cores recommended) 27 - **RAM**: 4 GB (8 GB recommended) 28 - **Storage**: 100 GB SSD (500 GB recommended) 29 - **Network**: Stable broadband (10 Mbps up/down minimum) 30 - **Power**: UPS recommended for uptime 31 32 ### Recommended Hardware Options 33 34 #### Option 1: Repurpose Existing Hardware 35 - Old workstations or servers 36 - **Cost**: $0 37 - **Pros**: Immediate availability, no cost 38 - **Cons**: May consume more power, larger footprint 39 40 #### Option 2: Mini PCs (Intel NUC, Beelink, etc.) 41 - Modern mini PCs with low power consumption 42 - **Cost**: $400-600 per unit 43 - **Pros**: Small, quiet, efficient (15-30W) 44 - **Cons**: Limited expandability 45 46 #### Option 3: Dedicated Servers (Dell, HP, Supermicro) 47 - Enterprise-grade refurbished servers 48 - **Cost**: $500-1000 per unit 49 - **Pros**: Reliable, expandable, enterprise features 50 - **Cons**: Higher power consumption, noise, space 51 52 ### Recommended Configuration: Intel NUC or Similar 53 ``` 54 Model: Intel NUC 11 or newer (or equivalent Beelink, ASUS PN) 55 CPU: Intel i5 or better (4 cores) 56 RAM: 16 GB DDR4 57 Storage: 500 GB NVMe SSD 58 Power: 65W max (typically 15-30W idle) 59 Cost: ~$500-600 60 61 Benefits: 62 - Silent operation 63 - Low power consumption (~$5-10/month electricity) 64 - Small footprint (4" x 4") 65 - Reliable 66 - Easy to maintain 67 ``` 68 69 --- 70 71 ## Network Requirements 72 73 ### Internet Connectivity 74 - **Upload**: 10 Mbps minimum, 50+ Mbps recommended 75 - **Download**: 10 Mbps minimum, 100+ Mbps recommended 76 - **Latency**: <50ms to major internet hubs preferred 77 - **Stability**: 99%+ uptime (business internet preferred) 78 79 ### Static IP vs. Dynamic DNS 80 - **Option 1**: Static IP from ISP (ideal) 81 - **Option 2**: Dynamic DNS (DynDNS, No-IP) - works fine 82 - **Option 3**: Tailscale only (private network) - recommended start 83 84 ### Firewall Configuration 85 ```bash 86 # Required ports (if exposing publicly) 87 22/tcp - SSH (change to non-standard port like 2222) 88 8776/tcp - Radicle protocol 89 8080/tcp - HTTP API (optional, can be Tailscale only) 90 91 # For Tailscale-only setup (recommended for Auxo) 92 - All communication over Tailscale network 93 - No public port exposure needed 94 - Maximum security 95 ``` 96 97 --- 98 99 ## Operating System Installation 100 101 ### Recommended: Ubuntu Server 22.04 LTS 102 103 #### 1. Download Ubuntu Server 104 ```bash 105 # Download from: https://ubuntu.com/download/server 106 # Verify checksum 107 sha256sum ubuntu-22.04-live-server-amd64.iso 108 ``` 109 110 #### 2. Create Bootable USB 111 ```bash 112 # macOS 113 sudo diskutil list 114 sudo diskutil unmountDisk /dev/diskX 115 sudo dd if=ubuntu-22.04-live-server-amd64.iso of=/dev/rdiskX bs=1m 116 117 # Linux 118 sudo dd if=ubuntu-22.04-live-server-amd64.iso of=/dev/sdX bs=1M status=progress 119 ``` 120 121 #### 3. Install Ubuntu 122 ``` 123 - Boot from USB 124 - Select "Install Ubuntu Server" 125 - Configure: 126 - Hostname: seed1.auxo.local (or seed2, seed3) 127 - Username: auxoadmin 128 - Enable full disk encryption (LUKS) 129 - Install OpenSSH server 130 - No additional packages needed 131 ``` 132 133 #### 4. Initial Configuration 134 ```bash 135 # SSH into the new server 136 ssh auxoadmin@seed1.auxo.local 137 138 # Update system 139 sudo apt update && sudo apt upgrade -y 140 141 # Install essential tools 142 sudo apt install -y curl wget git vim htop ufw fail2ban 143 144 # Configure timezone 145 sudo timedatectl set-timezone America/New_York # Adjust to your location 146 147 # Enable automatic security updates 148 sudo apt install -y unattended-upgrades 149 sudo dpkg-reconfigure --priority=low unattended-upgrades 150 ``` 151 152 --- 153 154 ## Security Hardening 155 156 ### 1. SSH Configuration 157 158 #### Generate ED25519 SSH Key (from your workstation) 159 ```bash 160 # On your workstation 161 ssh-keygen -t ed25519 -C "auxo-seed-access" 162 163 # Copy to server 164 ssh-copy-id -i ~/.ssh/id_ed25519.pub auxoadmin@seed1.auxo.local 165 ``` 166 167 #### Harden SSH Configuration 168 ```bash 169 # On the server 170 sudo vim /etc/ssh/sshd_config 171 172 # Add/modify these settings: 173 Port 2222 # Non-standard port 174 PermitRootLogin no # Disable root login 175 PasswordAuthentication no # Only SSH keys 176 PubkeyAuthentication yes 177 AuthenticationMethods publickey 178 MaxAuthTries 3 179 MaxSessions 5 180 ClientAliveInterval 300 181 ClientAliveCountMax 2 182 183 # Restart SSH 184 sudo systemctl restart sshd 185 ``` 186 187 #### Configure fail2ban 188 ```bash 189 sudo vim /etc/fail2ban/jail.local 190 ``` 191 ```ini 192 [DEFAULT] 193 bantime = 3600 194 findtime = 600 195 maxretry = 3 196 197 [sshd] 198 enabled = true 199 port = 2222 200 logpath = /var/log/auth.log 201 ``` 202 ```bash 203 sudo systemctl enable fail2ban 204 sudo systemctl restart fail2ban 205 ``` 206 207 ### 2. Firewall Configuration 208 209 ```bash 210 # Using Tailscale-only approach (recommended) 211 sudo ufw default deny incoming 212 sudo ufw default allow outgoing 213 sudo ufw allow 2222/tcp comment 'SSH' 214 sudo ufw allow in on tailscale0 comment 'Tailscale' 215 216 # Enable firewall 217 sudo ufw enable 218 sudo ufw status verbose 219 ``` 220 221 ### 3. Full Disk Encryption Verification 222 223 ```bash 224 # Verify LUKS encryption is active 225 sudo cryptsetup status /dev/mapper/ubuntu--vg-ubuntu--lv 226 227 # Should show: "type: LUKS2" 228 ``` 229 230 --- 231 232 ## Tailscale Installation 233 234 ### 1. Install Tailscale 235 ```bash 236 # Add Tailscale repository 237 curl -fsSL https://tailscale.com/install.sh | sh 238 239 # Start Tailscale 240 sudo tailscale up --advertise-tags=tag:radicle-seed 241 242 # Get Tailscale IP 243 tailscale ip -4 244 # Example: 100.97.158.51 245 ``` 246 247 ### 2. Configure Tailscale for Stability 248 ```bash 249 # Enable exit node capability (optional) 250 sudo tailscale up --advertise-exit-node 251 252 # Configure to start on boot 253 sudo systemctl enable tailscaled 254 ``` 255 256 ### 3. Set Hostname in Tailscale 257 ```bash 258 # In Tailscale admin console (https://login.tailscale.com) 259 # Set machine name: seed1-auxo, seed2-auxo, seed3-auxo 260 # Enable MagicDNS 261 # Disable key expiry for seed nodes 262 ``` 263 264 --- 265 266 ## Radicle Installation 267 268 ### 1. Install Radicle CLI 269 270 Use the existing automated script: 271 ```bash 272 # Clone the repository 273 git clone https://github.com/auxo/radicle.git 274 cd radicle 275 276 # Run installation script 277 ./scripts/setup/install-radicle.sh 278 ``` 279 280 Or manual installation: 281 ```bash 282 # Install dependencies 283 sudo apt install -y build-essential libssl-dev pkg-config 284 285 # Download and install Radicle 286 RADICLE_VERSION="1.2.1" 287 curl -sSL https://files.radicle.xyz/releases/radicle-cli-${RADICLE_VERSION}-x86_64-unknown-linux-musl.tar.gz -o radicle.tar.gz 288 tar -xzf radicle.tar.gz 289 sudo mv rad radicle-node radicle-remote-helper radicle-httpd /usr/local/bin/ 290 rm radicle.tar.gz 291 292 # Verify installation 293 rad --version 294 radicle-node --version 295 ``` 296 297 ### 2. Configure Radicle Node 298 299 #### Create seed user 300 ```bash 301 # Create dedicated user for Radicle 302 sudo useradd -r -m -d /home/seed -s /bin/bash seed 303 sudo passwd -l seed # Lock password login 304 ``` 305 306 #### Initialize Radicle identity 307 ```bash 308 # Switch to seed user 309 sudo -u seed bash 310 311 # Initialize identity (with passphrase for security) 312 rad auth --alias "Seed1 @ Auxo" 313 314 # IMPORTANT: Backup the keys immediately! 315 mkdir -p ~/.radicle-backup 316 cp -r ~/.radicle/keys ~/.radicle-backup/ 317 chmod 600 ~/.radicle-backup/keys/* 318 319 # Encrypt the backup 320 tar -czf radicle-keys-backup.tar.gz ~/.radicle-backup/ 321 gpg --encrypt --recipient ops@auxo.com radicle-keys-backup.tar.gz 322 323 # Copy encrypted backup to secure location 324 # Then delete unencrypted files 325 rm -rf ~/.radicle-backup/ 326 rm radicle-keys-backup.tar.gz 327 ``` 328 329 #### Configure node settings 330 ```bash 331 # Copy Auxo private network configuration 332 sudo -u seed bash 333 cat > ~/.radicle/config.json << 'EOF' 334 { 335 "node": { 336 "alias": "Seed1 @ Auxo", 337 "listen": ["0.0.0.0:8776"], 338 "peers": { 339 "type": "static", 340 "peers": [ 341 "seed2.auxo.ts.net:8776", 342 "seed3.auxo.ts.net:8776" 343 ] 344 }, 345 "connect": [ 346 "seed2.auxo.ts.net:8776", 347 "seed3.auxo.ts.net:8776" 348 ], 349 "externalAddresses": [], 350 "network": "auxo", 351 "relay": "never", 352 "limits": { 353 "routingMaxSize": 1000, 354 "routingMaxAge": 604800, 355 "gossipMaxAge": 1209600, 356 "fetchConcurrency": 1, 357 "maxOpenFiles": 4096, 358 "connection": { 359 "inbound": 128, 360 "outbound": 16 361 } 362 }, 363 "workers": 4, 364 "policy": { 365 "seedingPolicy": { 366 "default": "block", 367 "scope": "followed" 368 } 369 } 370 }, 371 "preferredSeeds": [] 372 } 373 EOF 374 ``` 375 376 ### 3. Configure Radicle as systemd Service 377 378 ```bash 379 # Create systemd service file 380 sudo vim /etc/systemd/system/radicle-node.service 381 ``` 382 383 ```ini 384 [Unit] 385 Description=Radicle Node 386 After=network-online.target 387 Wants=network-online.target 388 389 [Service] 390 Type=simple 391 User=seed 392 Group=seed 393 ExecStart=/usr/local/bin/radicle-node --listen 0.0.0.0:8776 394 Restart=always 395 RestartSec=10 396 StandardOutput=journal 397 StandardError=journal 398 SyslogIdentifier=radicle-node 399 400 # Security hardening 401 NoNewPrivileges=true 402 PrivateTmp=true 403 ProtectSystem=strict 404 ProtectHome=true 405 ReadWritePaths=/home/seed/.radicle 406 407 # Resource limits 408 LimitNOFILE=65535 409 LimitNPROC=4096 410 411 [Install] 412 WantedBy=multi-user.target 413 ``` 414 415 ```bash 416 # Enable and start service 417 sudo systemctl daemon-reload 418 sudo systemctl enable radicle-node 419 sudo systemctl start radicle-node 420 421 # Check status 422 sudo systemctl status radicle-node 423 sudo journalctl -u radicle-node -f 424 ``` 425 426 --- 427 428 ## Monitoring Setup 429 430 ### 1. Install Prometheus Node Exporter 431 432 ```bash 433 # Download and install 434 EXPORTER_VERSION="1.7.0" 435 wget https://github.com/prometheus/node_exporter/releases/download/v${EXPORTER_VERSION}/node_exporter-${EXPORTER_VERSION}.linux-amd64.tar.gz 436 tar -xzf node_exporter-${EXPORTER_VERSION}.linux-amd64.tar.gz 437 sudo mv node_exporter-${EXPORTER_VERSION}.linux-amd64/node_exporter /usr/local/bin/ 438 rm -rf node_exporter-${EXPORTER_VERSION}* 439 440 # Create systemd service 441 sudo vim /etc/systemd/system/node_exporter.service 442 ``` 443 444 ```ini 445 [Unit] 446 Description=Prometheus Node Exporter 447 After=network-online.target 448 449 [Service] 450 Type=simple 451 ExecStart=/usr/local/bin/node_exporter 452 Restart=always 453 RestartSec=10 454 455 [Install] 456 WantedBy=multi-user.target 457 ``` 458 459 ```bash 460 # Enable and start 461 sudo systemctl daemon-reload 462 sudo systemctl enable node_exporter 463 sudo systemctl start node_exporter 464 465 # Verify (should see metrics) 466 curl http://localhost:9100/metrics 467 ``` 468 469 ### 2. Install Health Check Script 470 471 ```bash 472 # Copy from repository 473 sudo cp /path/to/radicle/scripts/monitoring/health-check.sh /usr/local/bin/ 474 sudo chmod +x /usr/local/bin/health-check.sh 475 476 # Test 477 /usr/local/bin/health-check.sh 478 ``` 479 480 ### 3. Configure Automated Health Checks 481 482 ```bash 483 # Add cron job for health monitoring 484 sudo -u seed crontab -e 485 ``` 486 487 ```cron 488 # Health check every 5 minutes, log to file 489 */5 * * * * /usr/local/bin/health-check.sh >> /home/seed/health-check.log 2>&1 490 491 # Email alert if unhealthy (requires mail configured) 492 */5 * * * * /usr/local/bin/health-check.sh || echo "Seed node unhealthy on $(hostname)" | mail -s "Radicle Alert" ops@auxo.com 493 ``` 494 495 --- 496 497 ## Backup Configuration 498 499 ### 1. Automated Backup Script 500 501 ```bash 502 # Copy backup script 503 sudo cp /path/to/radicle/scripts/backup/backup-seed.sh /usr/local/bin/ 504 sudo chmod +x /usr/local/bin/backup-seed.sh 505 506 # Configure backup destination 507 sudo vim /usr/local/bin/backup-seed.sh 508 # Update BACKUP_DEST to point to Auxo backup server 509 ``` 510 511 ### 2. Schedule Daily Backups 512 513 ```bash 514 # Add to root crontab 515 sudo crontab -e 516 ``` 517 518 ```cron 519 # Daily backup at 2 AM 520 0 2 * * * /usr/local/bin/backup-seed.sh 521 ``` 522 523 ### 3. Test Backup and Restore 524 525 ```bash 526 # Create test backup 527 sudo /usr/local/bin/backup-seed.sh 528 529 # Verify backup exists and is encrypted 530 ls -lh /backups/ 531 532 # Test restore (on test system) 533 # See disaster recovery documentation 534 ``` 535 536 --- 537 538 ## Post-Deployment Verification 539 540 ### 1. Verify Radicle Node 541 542 ```bash 543 # Check node is running 544 sudo systemctl status radicle-node 545 546 # Check logs 547 sudo journalctl -u radicle-node -n 50 548 549 # Verify listening on port 550 sudo ss -tlnp | grep 8776 551 552 # Check Radicle identity 553 sudo -u seed rad self 554 ``` 555 556 ### 2. Verify Network Connectivity 557 558 ```bash 559 # Ping other seeds via Tailscale 560 ping seed2.auxo.ts.net 561 ping seed3.auxo.ts.net 562 563 # Check Radicle peer connections 564 # (Check logs for peer connection messages) 565 sudo journalctl -u radicle-node | grep -i "peer" 566 ``` 567 568 ### 3. Verify Monitoring 569 570 ```bash 571 # Check node exporter 572 curl http://localhost:9100/metrics | head 573 574 # Check health status 575 /usr/local/bin/health-check.sh 576 echo $? # Should be 0 577 ``` 578 579 ### 4. Verify Backup 580 581 ```bash 582 # Check backup exists 583 ls -lh /backups/ 584 585 # Verify backup is recent (last 24 hours) 586 find /backups/ -name "*.tar.gz.gpg" -mtime -1 587 ``` 588 589 --- 590 591 ## Geographic Distribution Strategy 592 593 ### Recommended 3-Seed Topology 594 595 ``` 596 Seed 1: Primary Location (e.g., Main Office) 597 - Role: Primary development seed 598 - Network: Business internet 599 - Backup: Daily to Seed 2 & 3 600 601 Seed 2: Secondary Location (e.g., Co-working Space / Home Office) 602 - Role: Redundancy & backup 603 - Network: Residential or business 604 - Backup: Daily to Seed 1 & 3 605 606 Seed 3: Tertiary Location (e.g., Another Office / Data Center) 607 - Role: Geographic redundancy 608 - Network: Business internet or colo 609 - Backup: Daily to Seed 1 & 2 610 ``` 611 612 ### Considerations 613 - **Same City**: Good for development (low latency) 614 - **Different Cities**: Better for disaster recovery 615 - **Mix of Both**: 2 in same metro + 1 remote (recommended) 616 617 ### Internet Connectivity 618 - **Business Internet** (preferred): Better uptime, SLA, support 619 - **Residential Internet** (acceptable): Works fine with Tailscale 620 - **Avoid**: Mobile hotspots, satellite (high latency, unreliable) 621 622 --- 623 624 ## Maintenance Procedures 625 626 ### Monthly Maintenance 627 628 ```bash 629 # Update system packages 630 sudo apt update && sudo apt upgrade -y 631 632 # Update Radicle (check for new versions) 633 rad --version # Compare with https://radicle.xyz/downloads 634 635 # Check disk space 636 df -h 637 638 # Review logs for errors 639 sudo journalctl -u radicle-node --since "1 month ago" | grep -i error 640 641 # Test backup restore 642 # (Follow disaster recovery procedures on test system) 643 644 # Verify health checks 645 /usr/local/bin/health-check.sh 646 647 # Check Tailscale connectivity 648 tailscale status 649 ``` 650 651 ### Quarterly Maintenance 652 653 ```bash 654 # Review and rotate logs 655 sudo journalctl --vacuum-time=90d 656 657 # Review security updates 658 sudo unattended-upgrade --dry-run -d 659 660 # Review firewall rules 661 sudo ufw status verbose 662 663 # Review SSH logs for unauthorized attempts 664 sudo grep "Failed password" /var/log/auth.log | tail -20 665 666 # Test disaster recovery procedures 667 # (Follow full DR test plan) 668 ``` 669 670 --- 671 672 ## Troubleshooting 673 674 ### Node Won't Start 675 676 ```bash 677 # Check logs 678 sudo journalctl -u radicle-node -n 100 679 680 # Common issues: 681 1. Port 8776 already in use 682 sudo ss -tlnp | grep 8776 683 684 2. Permission issues 685 sudo chown -R seed:seed /home/seed/.radicle 686 687 3. Config file errors 688 sudo -u seed rad config show 689 ``` 690 691 ### Can't Connect to Peers 692 693 ```bash 694 # Check Tailscale connectivity 695 tailscale status 696 ping seed2.auxo.ts.net 697 698 # Check firewall 699 sudo ufw status verbose 700 701 # Check Radicle is listening 702 sudo ss -tlnp | grep 8776 703 704 # Check peer configuration 705 sudo -u seed cat ~/.radicle/config.json | grep -A 10 peers 706 ``` 707 708 ### High Resource Usage 709 710 ```bash 711 # Check resource usage 712 htop 713 714 # Check Radicle process 715 ps aux | grep radicle 716 717 # Check disk space 718 df -h 719 du -sh /home/seed/.radicle/storage 720 721 # Adjust limits in systemd service if needed 722 sudo systemctl edit radicle-node 723 ``` 724 725 --- 726 727 ## Security Considerations 728 729 ### Physical Security 730 - Keep hardware in locked room/cabinet 731 - Restrict physical access to authorized personnel 732 - Use cable locks for smaller devices (optional) 733 - Monitor access logs 734 735 ### Network Security 736 - Use Tailscale for all inter-seed communication 737 - Avoid exposing Radicle ports to public internet unless necessary 738 - Use SSH keys only (no passwords) 739 - Monitor fail2ban logs regularly 740 741 ### Data Security 742 - Full disk encryption (LUKS) mandatory 743 - Encrypt all backups with GPG 744 - Store backup encryption keys securely (offline or HSM) 745 - Regular backup testing 746 747 ### Access Control 748 - Limit SSH access to authorized keys only 749 - Use sudo for administrative tasks 750 - Maintain audit logs 751 - Regular access reviews 752 753 --- 754 755 ## Cost Summary 756 757 ### One-Time Hardware Costs (per seed) 758 - Mini PC (Intel NUC or similar): $500-600 759 - UPS (APC Back-UPS): $100-150 760 - Network equipment (if needed): $50-100 761 - **Total per seed**: ~$650-850 762 - **Total for 3 seeds**: ~$1,950-2,550 763 764 ### Ongoing Costs (per seed) 765 - Power (@ $0.12/kWh, 20W avg): ~$2-3/month 766 - Internet (existing): $0 767 - **Total per seed**: ~$2-3/month 768 - **Total for 3 seeds**: ~$6-9/month 769 770 ### 5-Year Total Cost of Ownership 771 - Hardware: $1,950-2,550 (one-time) 772 - Power: $360-540 (5 years) 773 - **Total 5-Year TCO**: $2,310-3,090 774 775 ### Comparison to Cloud (3 seeds, 5 years) 776 - Digital Ocean/Hetzner: ~$600/year × 5 = $3,000 777 - **Savings**: Roughly break-even, but **infinite sovereignty value** 778 779 --- 780 781 ## Conclusion 782 783 Deploying Radicle on Auxo-owned hardware provides: 784 785 ✅ **Maximum Sovereignty** - Complete control over infrastructure 786 ✅ **Cost Effective** - Comparable to cloud after 3-4 years 787 ✅ **Security** - Physical and network control 788 ✅ **Privacy** - No third-party access 789 ✅ **Flexibility** - Configure exactly as needed 790 ✅ **Reliability** - Business-grade hardware and internet 791 792 Your Tailscale integration makes this approach even better by providing stable networking regardless of physical location changes. 793 794 --- 795 796 ## Next Steps 797 798 1. Acquire hardware (or repurpose existing) 799 2. Install Ubuntu Server with full disk encryption 800 3. Configure Tailscale 801 4. Deploy Radicle using this guide 802 5. Repeat for seeds 2 and 3 803 6. Configure monitoring and backups 804 7. Test disaster recovery procedures 805 806 --- 807 808 ## References 809 810 - Radicle Documentation: https://radicle.xyz/guides 811 - Ubuntu Server Guide: https://ubuntu.com/server/docs 812 - Tailscale Documentation: https://tailscale.com/kb 813 - Auxo Radicle Repository: rad:z4Hc9ECSRSuZbYSQFLTqtJuNoQJDP 814 815 **Version**: 1.0 816 **Maintainer**: Auxo Infrastructure Team 817 **Last Review**: 2025-01-11