ALPHA-DELTA_CI_Setup_Guide_v2.md
1 # Alpha/Delta Protocol: High-Performance CI/CD Setup on Digital Ocean 2 ## Complete Implementation Guide — Optimized Edition 3 4 **Last Updated:** January 2026 5 **Version:** 2.0 (Optimized for High-Performance Runner) 6 **Target Stack:** Forgejo + Forgejo-Runner + Radicle Integration 7 **Optimization Focus:** Maximum parallelization for Rust builds on 32-core runner 8 9 > **For Claude coding sessions**: Load the cspec files in `devops/machine/` instead (~80% fewer tokens). 10 > 11 > **cspec sources**: 12 > - `machine/ci-infrastructure.cspec` - Phase 1: DO infrastructure 13 > - `machine/ci-forgejo.cspec` - Phase 2: Forgejo server 14 > - `machine/ci-runner.cspec` - Phase 3: Build runner 15 > - `machine/ci-pipelines.cspec` - Phase 4: CI workflows 16 > - `machine/ci-radicle.cspec` - Phase 5: Radicle integration 17 > - `machine/ci-maintenance.cspec` - Phase 6: Monitoring 18 19 --- 20 21 ## Executive Summary: Hardware-Optimized Architecture 22 23 This guide is optimized for a **two-server model** with the following specifications: 24 25 | Server | Role | Specs | Monthly Cost | 26 |--------|------|-------|--------------| 27 | **Forgejo Primary** | Git hosting, Web UI, Actions coordination | 2 vCPU / 2GB RAM / 50GB SSD | ~$12 | 28 | **CI Runner** | Build, test, compilation | **32 vCPU / 64GB RAM / 1TB NVMe** | ~$336 | 29 30 **Key Optimizations for 32-Core Runner:** 31 - Parallel job capacity: **8 concurrent jobs** (vs. 2 in baseline) 32 - Parallel compilation: **CARGO_BUILD_JOBS=28** (leaving headroom for system) 33 - Aggressive sccache: **30GB cache** (leveraging NVMe speed) 34 - tmpfs for builds: RAM-backed `/tmp` for maximum I/O throughput 35 - Multi-workspace parallel builds: ALPHA + DELTA simultaneously 36 37 --- 38 39 ## Table of Contents 40 41 1. [Architecture Overview](#architecture-overview) 42 2. [Phase 1: Digital Ocean Infrastructure](#phase-1-digital-ocean-infrastructure) 43 3. [Phase 2: Forgejo Instance Setup](#phase-2-forgejo-instance-setup) 44 4. [Phase 3: High-Performance Runner Configuration](#phase-3-high-performance-runner-configuration) 45 5. [Phase 4: Rust-Optimized CI Pipeline](#phase-4-rust-optimized-ci-pipeline) 46 6. [Phase 5: Radicle Integration](#phase-5-radicle-integration) 47 7. [Phase 6: Monitoring & Maintenance](#phase-6-monitoring--maintenance) 48 8. [Appendix: Performance Tuning Reference](#appendix-performance-tuning-reference) 49 50 --- 51 52 ## Architecture Overview 53 54 ``` 55 ┌─────────────────────────────────────────────────────────────────────────────┐ 56 │ Digital Ocean VPC (Private Network) │ 57 ├─────────────────────────────────────────────────────────────────────────────┤ 58 │ │ 59 │ ┌────────────────────┐ ┌─────────────────────────────────────┐│ 60 │ │ Forgejo Server │ │ HIGH-PERFORMANCE RUNNER ││ 61 │ │ (2 vCPU / 2GB) │◄────────────►│ (32 vCPU / 64GB RAM / 1TB NVMe) ││ 62 │ │ │ VPC Link │ ││ 63 │ │ • Git repos │ │ ┌─────────────────────────────────┐││ 64 │ │ • Web UI │ │ │ Resource Allocation: │││ 65 │ │ • Actions coord. │ │ │ • 8 concurrent jobs │││ 66 │ │ • PostgreSQL │ │ │ • 28 cores for cargo │││ 67 │ └────────────────────┘ │ │ • 30GB sccache (NVMe) │││ 68 │ │ │ │ • 16GB tmpfs build dir │││ 69 │ │ │ │ • 900GB artifact storage │││ 70 │ ▼ │ └─────────────────────────────────┘││ 71 │ ┌────────────────────┐ │ ││ 72 │ │ External Access │ │ ┌─────────────────────────────────┐││ 73 │ │ • HTTPS (443) │ │ │ Optimizations: │││ 74 │ │ • SSH (22) │ │ │ • mold linker (10x faster) │││ 75 │ └────────────────────┘ │ │ • sccache (S3 + local hybrid) │││ 76 │ │ │ • Native host execution │││ 77 │ │ │ • Cranelift dev builds │││ 78 │ │ └─────────────────────────────────┘││ 79 │ └─────────────────────────────────────┘│ 80 │ │ │ 81 │ ┌─────────────────────────┘ │ 82 │ ▼ │ 83 │ ┌───────────────────────────┐ │ 84 │ │ DO Spaces (S3-Compatible)│ │ 85 │ │ • Distributed sccache │ │ 86 │ │ • Build artifacts │ │ 87 │ │ • CI cache layers │ │ 88 │ └───────────────────────────┘ │ 89 └─────────────────────────────────────────────────────────────────────────────┘ 90 ``` 91 92 **Design Rationale for Two-Server Model:** 93 94 1. **Security Isolation**: Runner has no public network exposure; all traffic routes through VPC 95 2. **Resource Dedication**: Forgejo's lightweight footprint doesn't compete with compilation 96 3. **Failure Isolation**: Runner crashes don't affect git hosting availability 97 4. **Cost Efficiency**: Forgejo needs minimal resources; budget concentrated on build power 98 99 --- 100 101 ## Phase 1: Digital Ocean Infrastructure 102 103 ### Step 1.1: Create VPC 104 105 ```bash 106 doctl compute vpc create \ 107 --name alpha-delta-vpc \ 108 --description "Alpha/Delta Protocol CI Infrastructure" \ 109 --ip-range 10.118.0.0/20 110 ``` 111 112 Record the VPC UUID: `______________` 113 114 ### Step 1.2: Create Forgejo Droplet (Lightweight) 115 116 ```bash 117 doctl compute droplet create ci-forgejo-primary \ 118 --region nyc3 \ 119 --image debian-12-x64 \ 120 --size s-2vcpu-2gb \ 121 --ssh-keys <your-ssh-key-id> \ 122 --enable-backups \ 123 --vpc-uuid <vpc-id> \ 124 --tag-names "ci,forgejo,alpha-delta" 125 ``` 126 127 **Specs:** 2 vCPU / 2GB RAM / 50GB SSD (~$12/month) 128 129 ### Step 1.3: Create High-Performance Runner Droplet 130 131 For the 32-core / 64GB / 1TB NVMe configuration, use: 132 133 ```bash 134 # Option A: General Purpose (Recommended for balanced workloads) 135 doctl compute droplet create ci-runner-build \ 136 --region nyc3 \ 137 --image debian-12-x64 \ 138 --size gd-32vcpu-128gb \ 139 --ssh-keys <your-ssh-key-id> \ 140 --vpc-uuid <vpc-id> \ 141 --tag-names "ci,runner,alpha-delta" 142 143 # Option B: CPU-Optimized (Better for pure compilation) 144 doctl compute droplet create ci-runner-build \ 145 --region nyc3 \ 146 --image debian-12-x64 \ 147 --size c-32-64gib-400gb-intel \ 148 --ssh-keys <your-ssh-key-id> \ 149 --vpc-uuid <vpc-id> \ 150 --tag-names "ci,runner,alpha-delta" 151 ``` 152 153 > **Note:** Exact slug names vary. Check `doctl compute size list | grep -E "32.*64"` for available options matching your specs. If using a custom/dedicated droplet, ensure NVMe storage is attached. 154 155 ### Step 1.4: Attach Additional Block Storage (If Needed) 156 157 If your base droplet has less than 1TB, attach block storage: 158 159 ```bash 160 doctl compute volume create ci-runner-storage \ 161 --region nyc3 \ 162 --size 1000GiB \ 163 --desc "CI Runner build cache and artifacts" 164 165 doctl compute volume-action attach <volume-id> <runner-droplet-id> 166 ``` 167 168 ### Step 1.5: Configure Firewall 169 170 ```bash 171 doctl compute firewall create \ 172 --name alpha-delta-fw \ 173 --inbound-rules \ 174 "protocol:tcp,ports:22,address:<your-ip>/32" \ 175 "protocol:tcp,ports:80,address:0.0.0.0/0" \ 176 "protocol:tcp,ports:443,address:0.0.0.0/0" \ 177 "protocol:tcp,ports:3000,address:10.118.0.0/20" \ 178 --outbound-rules \ 179 "protocol:tcp,ports:all,address:0.0.0.0/0" \ 180 "protocol:udp,ports:all,address:0.0.0.0/0" \ 181 "protocol:icmp,address:0.0.0.0/0" \ 182 --droplet-ids <forgejo-id>,<runner-id> 183 ``` 184 185 **Critical:** Port 3000 is restricted to VPC range only — runner communicates internally. 186 187 ### Step 1.6: Reserve Static IP & Configure DNS 188 189 ```bash 190 doctl compute reserved-ip create --region nyc3 --droplet-id <forgejo-droplet-id> 191 ``` 192 193 Create DNS record: `ci.yourdomain.com` → `<reserved-ip>` 194 195 **Record Your Infrastructure:** 196 - Forgejo Public IP: `______________` 197 - Forgejo VPC IP: `______________` 198 - Runner VPC IP: `______________` 199 - VPC UUID: `______________` 200 201 --- 202 203 ## Phase 2: Forgejo Instance Setup 204 205 ### Step 2.1: Initial System Setup 206 207 ```bash 208 ssh root@<forgejo-public-ip> 209 210 # System update 211 apt-get update && apt-get upgrade -y 212 213 # Install dependencies 214 apt-get install -y \ 215 git curl wget nano htop \ 216 certbot python3-certbot-nginx nginx \ 217 postgresql postgresql-contrib \ 218 jq gnupg 219 ``` 220 221 ### Step 2.2: Create Forgejo User & Directories 222 223 ```bash 224 useradd -r -s /bin/bash -d /home/forgejo -m forgejo 225 mkdir -p /home/forgejo/{custom/conf,data} 226 mkdir -p /var/lib/forgejo/{repositories,data,log} 227 chown -R forgejo:forgejo /home/forgejo /var/lib/forgejo 228 ``` 229 230 ### Step 2.3: Install Forgejo Binary 231 232 ```bash 233 export ARCH=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/') 234 export FORGEJO_VERSION=$(curl -s https://codeberg.org/api/v1/repos/forgejo/forgejo/releases/latest | jq -r .tag_name | cut -c 2-) 235 236 cd /tmp 237 wget -q "https://codeberg.org/forgejo/forgejo/releases/download/v${FORGEJO_VERSION}/forgejo-${FORGEJO_VERSION}-linux-${ARCH}" 238 wget -q "https://codeberg.org/forgejo/forgejo/releases/download/v${FORGEJO_VERSION}/forgejo-${FORGEJO_VERSION}-linux-${ARCH}.asc" 239 240 # Verify GPG signature 241 gpg --keyserver keys.openpgp.org --recv 39C6B5B3244C57C9F29001B8CECD0BE6FB79DCC6 242 gpg --verify "forgejo-${FORGEJO_VERSION}-linux-${ARCH}.asc" 243 244 # Install 245 install -m 755 "forgejo-${FORGEJO_VERSION}-linux-${ARCH}" /usr/local/bin/forgejo 246 forgejo --version 247 ``` 248 249 ### Step 2.4: Configure PostgreSQL 250 251 ```bash 252 sudo -u postgres psql << 'EOSQL' 253 CREATE USER forgejo WITH PASSWORD 'CHANGE_ME_SECURE_PASSWORD'; 254 CREATE DATABASE forgejo OWNER forgejo; 255 GRANT ALL PRIVILEGES ON DATABASE forgejo TO forgejo; 256 \q 257 EOSQL 258 259 # Test connection 260 sudo -u forgejo psql -h localhost -U forgejo -d forgejo -c "SELECT version();" 261 ``` 262 263 ### Step 2.5: Configure Forgejo (app.ini) 264 265 ```bash 266 # Generate secrets 267 SECRET_KEY=$(openssl rand -hex 32) 268 INTERNAL_TOKEN=$(openssl rand -hex 32) 269 JWT_SECRET=$(openssl rand -hex 32) 270 271 cat > /home/forgejo/custom/conf/app.ini << EOF 272 [app] 273 APP_NAME = Alpha/Delta CI 274 RUN_MODE = prod 275 RUN_USER = forgejo 276 WORK_PATH = /var/lib/forgejo 277 278 [server] 279 APP_DATA_PATH = /var/lib/forgejo 280 DOMAIN = ci.yourdomain.com 281 SSH_DOMAIN = ci.yourdomain.com 282 ROOT_URL = https://ci.yourdomain.com/ 283 HTTP_ADDR = 127.0.0.1 284 HTTP_PORT = 3000 285 DISABLE_SSH = false 286 SSH_PORT = 22 287 START_SSH_SERVER = false 288 OFFLINE_MODE = false 289 LANDING_PAGE = explore 290 291 [database] 292 DB_TYPE = postgres 293 HOST = 127.0.0.1:5432 294 NAME = forgejo 295 USER = forgejo 296 PASSWD = CHANGE_ME_SECURE_PASSWORD 297 LOG_SQL = false 298 MAX_OPEN_CONNS = 60 299 MAX_IDLE_CONNS = 30 300 CONN_MAX_LIFETIME = 10m 301 302 [repository] 303 ROOT = /var/lib/forgejo/repositories 304 DEFAULT_BRANCH = main 305 DEFAULT_PRIVATE = private 306 ENABLE_PUSH_CREATE_USER = true 307 ENABLE_PUSH_CREATE_ORG = true 308 309 [service] 310 DISABLE_REGISTRATION = true 311 REQUIRE_SIGNIN_VIEW = false 312 DEFAULT_KEEP_EMAIL_PRIVATE = true 313 DEFAULT_ALLOW_CREATE_ORGANIZATION = true 314 315 [security] 316 INSTALL_LOCK = true 317 SECRET_KEY = ${SECRET_KEY} 318 INTERNAL_TOKEN = ${INTERNAL_TOKEN} 319 320 [oauth2] 321 JWT_SECRET = ${JWT_SECRET} 322 323 [actions] 324 ENABLED = true 325 DEFAULT_ACTIONS_URL = https://github.com 326 327 [queue] 328 TYPE = level 329 DATADIR = /var/lib/forgejo/queues 330 331 [cache] 332 ENABLED = true 333 ADAPTER = memory 334 335 [session] 336 PROVIDER = file 337 PROVIDER_CONFIG = /var/lib/forgejo/sessions 338 339 [log] 340 MODE = file 341 LEVEL = info 342 ROOT_PATH = /var/lib/forgejo/log 343 344 [cors] 345 ENABLED = true 346 ALLOW_DOMAIN = * 347 EOF 348 349 chown forgejo:forgejo /home/forgejo/custom/conf/app.ini 350 chmod 600 /home/forgejo/custom/conf/app.ini 351 ``` 352 353 ### Step 2.6: Create Systemd Service 354 355 ```bash 356 cat > /etc/systemd/system/forgejo.service << 'EOF' 357 [Unit] 358 Description=Forgejo Git Server 359 After=syslog.target network.target postgresql.service 360 Wants=postgresql.service 361 362 [Service] 363 Type=simple 364 User=forgejo 365 Group=forgejo 366 WorkingDirectory=/var/lib/forgejo 367 ExecStart=/usr/local/bin/forgejo web --config /home/forgejo/custom/conf/app.ini 368 Restart=always 369 RestartSec=3 370 Environment="USER=forgejo" "HOME=/home/forgejo" "GITEA_WORK_DIR=/var/lib/forgejo" 371 372 # Security hardening 373 ProtectSystem=full 374 PrivateDevices=true 375 NoNewPrivileges=true 376 377 [Install] 378 WantedBy=multi-user.target 379 EOF 380 381 systemctl daemon-reload 382 systemctl enable --now forgejo 383 systemctl status forgejo 384 ``` 385 386 ### Step 2.7: Configure Nginx with HTTPS 387 388 ```bash 389 cat > /etc/nginx/sites-available/forgejo << 'EOF' 390 upstream forgejo { 391 server 127.0.0.1:3000; 392 keepalive 32; 393 } 394 395 server { 396 listen 80; 397 listen [::]:80; 398 server_name ci.yourdomain.com; 399 400 location /.well-known/acme-challenge/ { 401 root /var/www/certbot; 402 } 403 404 location / { 405 return 301 https://$server_name$request_uri; 406 } 407 } 408 409 server { 410 listen 443 ssl http2; 411 listen [::]:443 ssl http2; 412 server_name ci.yourdomain.com; 413 414 ssl_certificate /etc/letsencrypt/live/ci.yourdomain.com/fullchain.pem; 415 ssl_certificate_key /etc/letsencrypt/live/ci.yourdomain.com/privkey.pem; 416 ssl_session_timeout 1d; 417 ssl_session_cache shared:SSL:50m; 418 ssl_session_tickets off; 419 420 ssl_protocols TLSv1.2 TLSv1.3; 421 ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384; 422 ssl_prefer_server_ciphers off; 423 424 client_max_body_size 512M; 425 426 location / { 427 proxy_pass http://forgejo; 428 proxy_http_version 1.1; 429 proxy_set_header Host $host; 430 proxy_set_header X-Real-IP $remote_addr; 431 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 432 proxy_set_header X-Forwarded-Proto $scheme; 433 proxy_set_header Connection ""; 434 435 # WebSocket support for live updates 436 proxy_set_header Upgrade $http_upgrade; 437 proxy_set_header Connection "upgrade"; 438 439 proxy_buffering off; 440 proxy_read_timeout 300s; 441 } 442 } 443 EOF 444 445 ln -sf /etc/nginx/sites-available/forgejo /etc/nginx/sites-enabled/ 446 rm -f /etc/nginx/sites-enabled/default 447 nginx -t && systemctl restart nginx 448 ``` 449 450 ### Step 2.8: Obtain SSL Certificate 451 452 ```bash 453 mkdir -p /var/www/certbot 454 certbot certonly --webroot -w /var/www/certbot \ 455 -d ci.yourdomain.com \ 456 --non-interactive --agree-tos -m admin@yourdomain.com 457 458 systemctl restart nginx 459 systemctl enable certbot.timer 460 ``` 461 462 ### Step 2.9: Initial Forgejo Setup 463 464 1. Navigate to `https://ci.yourdomain.com` 465 2. Create admin account 466 3. Go to **Site Administration → Actions → Runners** and note registration token 467 468 --- 469 470 ## Phase 3: High-Performance Runner Configuration 471 472 This is where we leverage the **32 cores / 64GB RAM / 1TB NVMe**. 473 474 ### Step 3.1: SSH into Runner & Initial Setup 475 476 ```bash 477 ssh root@<runner-vpc-ip> # Via VPC from Forgejo, or directly if configured 478 479 apt-get update && apt-get upgrade -y 480 apt-get install -y \ 481 git curl wget htop iotop \ 482 build-essential pkg-config libssl-dev \ 483 clang llvm lld \ 484 docker.io docker-compose \ 485 jq gnupg unzip 486 ``` 487 488 ### Step 3.2: Configure System for High-Performance Builds 489 490 #### 3.2.1: Kernel Tuning 491 492 ```bash 493 cat >> /etc/sysctl.conf << 'EOF' 494 # Increase file descriptor limits 495 fs.file-max = 2097152 496 fs.nr_open = 2097152 497 498 # Increase inotify limits (for cargo watch, etc.) 499 fs.inotify.max_user_watches = 524288 500 fs.inotify.max_user_instances = 1024 501 502 # Network optimizations 503 net.core.somaxconn = 65535 504 net.core.netdev_max_backlog = 65535 505 506 # Memory management for large builds 507 vm.swappiness = 10 508 vm.dirty_ratio = 60 509 vm.dirty_background_ratio = 5 510 vm.vfs_cache_pressure = 50 511 512 # Allow more memory overcommit for cargo 513 vm.overcommit_memory = 1 514 EOF 515 516 sysctl -p 517 ``` 518 519 #### 3.2.2: Configure Limits 520 521 ```bash 522 cat >> /etc/security/limits.conf << 'EOF' 523 * soft nofile 1048576 524 * hard nofile 1048576 525 * soft nproc 65535 526 * hard nproc 65535 527 root soft nofile 1048576 528 root hard nofile 1048576 529 EOF 530 ``` 531 532 #### 3.2.3: Create tmpfs for Ultra-Fast Builds 533 534 ```bash 535 # 16GB RAM-backed build directory for maximum I/O speed 536 mkdir -p /mnt/ramdisk 537 echo 'tmpfs /mnt/ramdisk tmpfs rw,nodev,nosuid,size=16G,mode=1777 0 0' >> /etc/fstab 538 mount /mnt/ramdisk 539 540 # Verify 541 df -h /mnt/ramdisk 542 ``` 543 544 #### 3.2.4: Configure NVMe Storage Layout 545 546 ```bash 547 # Create optimized directory structure on NVMe 548 mkdir -p /opt/ci/{cache,artifacts,workspaces,sccache,cargo-registry} 549 550 # If using attached block storage, mount it first: 551 # mkfs.ext4 /dev/sda # Adjust device name 552 # mount /dev/sda /opt/ci 553 # echo '/dev/sda /opt/ci ext4 defaults,noatime,nodiratime 0 2' >> /etc/fstab 554 555 # Set permissions 556 chown -R root:root /opt/ci 557 chmod 755 /opt/ci 558 ``` 559 560 ### Step 3.3: Install Rust Toolchain (System-Wide) 561 562 ```bash 563 # Install rustup system-wide 564 export RUSTUP_HOME=/opt/rust/rustup 565 export CARGO_HOME=/opt/rust/cargo 566 567 curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- \ 568 --default-toolchain stable \ 569 --profile default \ 570 -y 571 572 # Add to system-wide profile 573 cat >> /etc/profile.d/rust.sh << 'EOF' 574 export RUSTUP_HOME=/opt/rust/rustup 575 export CARGO_HOME=/opt/rust/cargo 576 export PATH="$CARGO_HOME/bin:$PATH" 577 EOF 578 579 source /etc/profile.d/rust.sh 580 581 # Install additional toolchains 582 rustup install nightly 583 rustup component add rustfmt clippy llvm-tools-preview 584 rustup component add rustfmt clippy --toolchain nightly 585 586 # Verify 587 rustc --version 588 cargo --version 589 ``` 590 591 ### Step 3.4: Install High-Performance Build Tools 592 593 #### 3.4.1: mold Linker (10-20x Faster Than ld) 594 595 ```bash 596 MOLD_VERSION="2.35.1" 597 wget -q "https://github.com/rui314/mold/releases/download/v${MOLD_VERSION}/mold-${MOLD_VERSION}-x86_64-linux.tar.gz" 598 tar -xzf "mold-${MOLD_VERSION}-x86_64-linux.tar.gz" 599 cp "mold-${MOLD_VERSION}-x86_64-linux/bin/mold" /usr/local/bin/ 600 chmod +x /usr/local/bin/mold 601 ln -sf /usr/local/bin/mold /usr/local/bin/ld.mold 602 603 mold --version 604 ``` 605 606 #### 3.4.2: sccache (Distributed Compilation Cache) 607 608 ```bash 609 SCCACHE_VERSION="0.8.2" 610 wget -q "https://github.com/mozilla/sccache/releases/download/v${SCCACHE_VERSION}/sccache-v${SCCACHE_VERSION}-x86_64-unknown-linux-musl.tar.gz" 611 tar -xzf "sccache-v${SCCACHE_VERSION}-x86_64-unknown-linux-musl.tar.gz" 612 cp "sccache-v${SCCACHE_VERSION}-x86_64-unknown-linux-musl/sccache" /usr/local/bin/ 613 chmod +x /usr/local/bin/sccache 614 615 sccache --version 616 ``` 617 618 #### 3.4.3: Configure sccache for S3 + Local Hybrid 619 620 ```bash 621 mkdir -p /etc/sccache 622 cat > /etc/sccache/config << 'EOF' 623 [cache.s3] 624 bucket = "alpha-delta-cache" 625 endpoint = "https://nyc3.digitaloceanspaces.com" 626 region = "nyc3" 627 key_prefix = "sccache/" 628 use_ssl = true 629 630 [cache.disk] 631 dir = "/opt/ci/sccache" 632 size = 32212254720 # 30GB local cache 633 EOF 634 635 # Environment for sccache 636 cat > /etc/profile.d/sccache.sh << 'EOF' 637 export SCCACHE_BUCKET="alpha-delta-cache" 638 export SCCACHE_REGION="nyc3" 639 export SCCACHE_ENDPOINT="https://nyc3.digitaloceanspaces.com" 640 export SCCACHE_S3_KEY_PREFIX="sccache/" 641 export SCCACHE_DIR="/opt/ci/sccache" 642 export SCCACHE_CACHE_SIZE="30G" 643 export RUSTC_WRAPPER="/usr/local/bin/sccache" 644 645 # S3 credentials (replace with actual values) 646 export AWS_ACCESS_KEY_ID="YOUR_SPACES_ACCESS_KEY" 647 export AWS_SECRET_ACCESS_KEY="YOUR_SPACES_SECRET_KEY" 648 EOF 649 650 source /etc/profile.d/sccache.sh 651 ``` 652 653 #### 3.4.4: Install Additional Rust Tools 654 655 ```bash 656 source /etc/profile.d/rust.sh 657 658 # Faster test runner 659 cargo install cargo-nextest --locked 660 661 # Security auditing 662 cargo install cargo-audit --locked 663 664 # Dependency analysis 665 cargo install cargo-deny --locked 666 667 # Code coverage 668 cargo install cargo-llvm-cov --locked 669 670 # Benchmarking 671 cargo install cargo-criterion --locked 672 673 # Move binaries to system path 674 cp /opt/rust/cargo/bin/cargo-* /usr/local/bin/ 675 ``` 676 677 ### Step 3.5: Create Runner User with Optimized Environment 678 679 ```bash 680 useradd -r -s /bin/bash -d /var/lib/forgejo-runner -m forgejo-runner 681 usermod -aG docker forgejo-runner 682 683 # Create runner directories 684 mkdir -p /var/lib/forgejo-runner/{config,logs,.cache} 685 mkdir -p /opt/ci/workspaces 686 chown -R forgejo-runner:forgejo-runner /var/lib/forgejo-runner /opt/ci 687 688 # Create optimized cargo config for runner 689 mkdir -p /var/lib/forgejo-runner/.cargo 690 cat > /var/lib/forgejo-runner/.cargo/config.toml << 'EOF' 691 [build] 692 # Use 28 of 32 cores (leave headroom for system + other jobs) 693 jobs = 28 694 695 # Use mold linker by default 696 rustflags = ["-C", "link-arg=-fuse-ld=mold"] 697 698 # Target directory on NVMe for fast I/O 699 target-dir = "/opt/ci/workspaces/target" 700 701 [target.x86_64-unknown-linux-gnu] 702 linker = "clang" 703 rustflags = ["-C", "link-arg=-fuse-ld=mold", "-C", "target-cpu=native"] 704 705 [net] 706 git-fetch-with-cli = true 707 retry = 3 708 709 [registries.crates-io] 710 protocol = "sparse" 711 712 [profile.dev] 713 # Use Cranelift for faster dev builds (when available) 714 # codegen-backend = "cranelift" 715 716 [profile.dev.package."*"] 717 opt-level = 0 718 debug = false 719 720 [profile.release] 721 lto = "thin" 722 codegen-units = 16 723 EOF 724 725 chown -R forgejo-runner:forgejo-runner /var/lib/forgejo-runner/.cargo 726 ``` 727 728 ### Step 3.6: Install Forgejo Runner 729 730 ```bash 731 export ARCH=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/') 732 RUNNER_VERSION=$(curl -s https://codeberg.org/api/v1/repos/forgejo/runner/releases/latest | jq -r .tag_name | cut -c 2-) 733 734 cd /tmp 735 wget -q "https://codeberg.org/forgejo/runner/releases/download/v${RUNNER_VERSION}/forgejo-runner-${RUNNER_VERSION}-linux-${ARCH}" 736 737 install -m 755 "forgejo-runner-${RUNNER_VERSION}-linux-${ARCH}" /usr/local/bin/forgejo-runner 738 forgejo-runner --version 739 ``` 740 741 ### Step 3.7: Configure Runner for Maximum Parallelism 742 743 ```bash 744 cat > /var/lib/forgejo-runner/config/config.yaml << 'EOF' 745 log: 746 level: info 747 748 runner: 749 file: /var/lib/forgejo-runner/.runner 750 # 8 concurrent jobs - each can use multiple cores 751 capacity: 8 752 timeout: 3h 753 insecure: false 754 fetch_timeout: 30s 755 fetch_interval: 2s 756 labels: 757 # Host execution labels (fastest - no container overhead) 758 - native:host 759 - rust-native:host 760 - linux-x64:host 761 # Docker labels for isolation when needed 762 - ubuntu-latest:docker://ubuntu:24.04 763 - ubuntu-22.04:docker://ubuntu:22.04 764 - rust-docker:docker://rust:1.83-bookworm 765 - alpine:docker://alpine:3.19 766 767 cache: 768 enabled: true 769 dir: /opt/ci/cache 770 host: 771 # S3 distributed cache 772 s3: 773 server_address: https://nyc3.digitaloceanspaces.com 774 access_key_id: YOUR_SPACES_ACCESS_KEY 775 access_key_secret: YOUR_SPACES_SECRET_KEY 776 bucket: alpha-delta-cache 777 path_style: false 778 779 container: 780 network: host 781 docker_host: unix:///var/run/docker.sock 782 privileged: false 783 options: | 784 -v /opt/ci:/opt/ci 785 -v /opt/rust:/opt/rust:ro 786 -v /usr/local/bin/mold:/usr/local/bin/mold:ro 787 -v /usr/local/bin/sccache:/usr/local/bin/sccache:ro 788 valid_volumes: 789 - /opt/ci/** 790 - /opt/rust/** 791 - /mnt/ramdisk/** 792 793 host: 794 workdir_parent: /opt/ci/workspaces 795 EOF 796 797 chown forgejo-runner:forgejo-runner /var/lib/forgejo-runner/config/config.yaml 798 chmod 600 /var/lib/forgejo-runner/config/config.yaml 799 ``` 800 801 ### Step 3.8: Register Runner with Forgejo 802 803 ```bash 804 # Get token from Forgejo Admin Panel → Actions → Runners → Create new runner 805 export FORGEJO_URL="https://ci.yourdomain.com" 806 export RUNNER_TOKEN="<40-char-token-from-forgejo>" 807 export RUNNER_NAME="alpha-delta-powerhouse" 808 809 sudo -u forgejo-runner /usr/local/bin/forgejo-runner register \ 810 --instance "${FORGEJO_URL}" \ 811 --token "${RUNNER_TOKEN}" \ 812 --name "${RUNNER_NAME}" \ 813 --labels "native,rust-native,linux-x64,ubuntu-latest,ubuntu-22.04,rust-docker,alpine" \ 814 --config /var/lib/forgejo-runner/config/config.yaml 815 ``` 816 817 ### Step 3.9: Create Systemd Service with Resource Controls 818 819 ```bash 820 cat > /etc/systemd/system/forgejo-runner.service << 'EOF' 821 [Unit] 822 Description=Forgejo Actions Runner (High-Performance) 823 After=network.target docker.service 824 Requires=docker.service 825 826 [Service] 827 Type=simple 828 User=forgejo-runner 829 Group=forgejo-runner 830 WorkingDirectory=/var/lib/forgejo-runner 831 832 # Environment setup 833 Environment="PATH=/usr/local/bin:/opt/rust/cargo/bin:/usr/bin:/bin" 834 Environment="RUSTUP_HOME=/opt/rust/rustup" 835 Environment="CARGO_HOME=/opt/rust/cargo" 836 Environment="SCCACHE_DIR=/opt/ci/sccache" 837 Environment="SCCACHE_CACHE_SIZE=30G" 838 Environment="RUSTC_WRAPPER=/usr/local/bin/sccache" 839 Environment="CARGO_BUILD_JOBS=28" 840 Environment="CARGO_INCREMENTAL=1" 841 Environment="CARGO_TARGET_DIR=/opt/ci/workspaces/target" 842 843 # S3 credentials for sccache 844 Environment="AWS_ACCESS_KEY_ID=YOUR_SPACES_ACCESS_KEY" 845 Environment="AWS_SECRET_ACCESS_KEY=YOUR_SPACES_SECRET_KEY" 846 Environment="SCCACHE_BUCKET=alpha-delta-cache" 847 Environment="SCCACHE_REGION=nyc3" 848 Environment="SCCACHE_ENDPOINT=https://nyc3.digitaloceanspaces.com" 849 850 ExecStart=/usr/local/bin/forgejo-runner daemon --config /var/lib/forgejo-runner/config/config.yaml 851 Restart=always 852 RestartSec=5 853 854 # Resource limits - allow full utilization 855 LimitNOFILE=1048576 856 LimitNPROC=65535 857 LimitCORE=infinity 858 859 # Don't kill on OOM - let cargo handle memory 860 OOMScoreAdjust=-500 861 862 [Install] 863 WantedBy=multi-user.target 864 EOF 865 866 systemctl daemon-reload 867 systemctl enable --now forgejo-runner 868 systemctl status forgejo-runner 869 ``` 870 871 ### Step 3.10: Start sccache Server (Background) 872 873 ```bash 874 cat > /etc/systemd/system/sccache.service << 'EOF' 875 [Unit] 876 Description=sccache Server 877 After=network.target 878 879 [Service] 880 Type=simple 881 User=forgejo-runner 882 Environment="SCCACHE_DIR=/opt/ci/sccache" 883 Environment="SCCACHE_CACHE_SIZE=30G" 884 Environment="AWS_ACCESS_KEY_ID=YOUR_SPACES_ACCESS_KEY" 885 Environment="AWS_SECRET_ACCESS_KEY=YOUR_SPACES_SECRET_KEY" 886 Environment="SCCACHE_BUCKET=alpha-delta-cache" 887 Environment="SCCACHE_REGION=nyc3" 888 Environment="SCCACHE_ENDPOINT=https://nyc3.digitaloceanspaces.com" 889 ExecStart=/usr/local/bin/sccache --start-server 890 Restart=always 891 RestartSec=5 892 893 [Install] 894 WantedBy=multi-user.target 895 EOF 896 897 systemctl daemon-reload 898 systemctl enable --now sccache 899 ``` 900 901 ### Step 3.11: Verify Runner Registration 902 903 In Forgejo web UI: 904 1. Go to **Site Administration → Actions → Runners** 905 2. Verify `alpha-delta-powerhouse` shows as **Idle** 906 3. Check labels are correctly registered 907 908 --- 909 910 ## Phase 4: Rust-Optimized CI Pipeline 911 912 ### Step 4.1: Create Repository Structure 913 914 In your Alpha/Delta Protocol repository: 915 916 ```bash 917 mkdir -p .forgejo/workflows 918 ``` 919 920 ### Step 4.2: Main CI Pipeline (Maximizing Parallelism) 921 922 Create `.forgejo/workflows/ci.yml`: 923 924 ```yaml 925 name: Alpha/Delta CI Pipeline 926 927 on: 928 push: 929 branches: [main, develop] 930 pull_request: 931 branches: [main] 932 933 env: 934 CARGO_TERM_COLOR: always 935 CARGO_INCREMENTAL: 1 936 CARGO_NET_RETRY: 10 937 RUSTFLAGS: "-C link-arg=-fuse-ld=mold -C target-cpu=native" 938 RUSTC_WRAPPER: /usr/local/bin/sccache 939 SCCACHE_DIR: /opt/ci/sccache 940 CARGO_TARGET_DIR: /opt/ci/workspaces/target/${{ github.repository }} 941 942 # Cancel in-progress runs on same branch 943 concurrency: 944 group: ${{ github.workflow }}-${{ github.ref }} 945 cancel-in-progress: true 946 947 jobs: 948 # ============================================================ 949 # STAGE 1: Quick Checks (Run in Parallel) 950 # ============================================================ 951 952 formatting: 953 name: "📝 Format Check" 954 runs-on: native 955 steps: 956 - uses: actions/checkout@v4 957 958 - name: Check formatting 959 run: cargo fmt --all -- --check 960 961 lint: 962 name: "🔍 Clippy Lint" 963 runs-on: native 964 steps: 965 - uses: actions/checkout@v4 966 967 - name: Cache cargo registry 968 uses: actions/cache@v4 969 with: 970 path: | 971 ~/.cargo/registry 972 ~/.cargo/git 973 key: ${{ runner.os }}-cargo-registry-${{ hashFiles('**/Cargo.lock') }} 974 restore-keys: ${{ runner.os }}-cargo-registry- 975 976 - name: Run Clippy (all features) 977 run: | 978 cargo clippy --workspace --all-targets --all-features -- -D warnings 979 980 - name: Run Clippy (no default features) 981 run: | 982 cargo clippy --workspace --all-targets --no-default-features -- -D warnings 983 984 security-audit: 985 name: "🔒 Security Audit" 986 runs-on: native 987 steps: 988 - uses: actions/checkout@v4 989 990 - name: Run cargo-audit 991 run: cargo audit --deny warnings 992 993 - name: Run cargo-deny 994 run: cargo deny check 995 996 # ============================================================ 997 # STAGE 2: Unit Tests (Parallel by Workspace) 998 # ============================================================ 999 1000 test-alpha-chain: 1001 name: "🧪 Test ALPHA Chain" 1002 runs-on: native 1003 needs: [formatting, lint] 1004 steps: 1005 - uses: actions/checkout@v4 1006 1007 - name: Run ALPHA unit tests 1008 run: | 1009 cargo nextest run \ 1010 --workspace \ 1011 --package 'alpha-*' \ 1012 --release \ 1013 --no-fail-fast \ 1014 --test-threads=24 1015 1016 test-delta-exchange: 1017 name: "🧪 Test DELTA Exchange" 1018 runs-on: native 1019 needs: [formatting, lint] 1020 steps: 1021 - uses: actions/checkout@v4 1022 1023 - name: Run DELTA unit tests 1024 run: | 1025 cargo nextest run \ 1026 --workspace \ 1027 --package 'delta-*' \ 1028 --release \ 1029 --no-fail-fast \ 1030 --test-threads=24 1031 1032 test-shared-libs: 1033 name: "🧪 Test Shared Libraries" 1034 runs-on: native 1035 needs: [formatting, lint] 1036 steps: 1037 - uses: actions/checkout@v4 1038 1039 - name: Run shared lib tests 1040 run: | 1041 cargo nextest run \ 1042 --workspace \ 1043 --exclude 'alpha-*' \ 1044 --exclude 'delta-*' \ 1045 --release \ 1046 --no-fail-fast \ 1047 --test-threads=24 1048 1049 # ============================================================ 1050 # STAGE 3: Build Binaries (Parallel) 1051 # ============================================================ 1052 1053 build-alpha: 1054 name: "🔨 Build ALPHA Chain" 1055 runs-on: native 1056 needs: [test-alpha-chain, test-shared-libs] 1057 steps: 1058 - uses: actions/checkout@v4 1059 1060 - name: Build ALPHA (Release) 1061 env: 1062 CARGO_PROFILE_RELEASE_LTO: thin 1063 CARGO_PROFILE_RELEASE_CODEGEN_UNITS: 16 1064 run: | 1065 cargo build --release --package alpha-chain --package alpha-node 1066 1067 - name: Upload ALPHA artifacts 1068 uses: actions/upload-artifact@v4 1069 with: 1070 name: alpha-binaries 1071 path: | 1072 /opt/ci/workspaces/target/${{ github.repository }}/release/alpha-chain 1073 /opt/ci/workspaces/target/${{ github.repository }}/release/alpha-node 1074 retention-days: 7 1075 1076 build-delta: 1077 name: "🔨 Build DELTA Exchange" 1078 runs-on: native 1079 needs: [test-delta-exchange, test-shared-libs] 1080 steps: 1081 - uses: actions/checkout@v4 1082 1083 - name: Build DELTA (Release) 1084 env: 1085 CARGO_PROFILE_RELEASE_LTO: thin 1086 CARGO_PROFILE_RELEASE_CODEGEN_UNITS: 16 1087 run: | 1088 cargo build --release --package delta-exchange --package delta-node 1089 1090 - name: Upload DELTA artifacts 1091 uses: actions/upload-artifact@v4 1092 with: 1093 name: delta-binaries 1094 path: | 1095 /opt/ci/workspaces/target/${{ github.repository }}/release/delta-exchange 1096 /opt/ci/workspaces/target/${{ github.repository }}/release/delta-node 1097 retention-days: 7 1098 1099 # ============================================================ 1100 # STAGE 4: Integration Tests (Requires Both Chains) 1101 # ============================================================ 1102 1103 integration-tests: 1104 name: "🔗 Integration Tests" 1105 runs-on: native 1106 needs: [build-alpha, build-delta] 1107 steps: 1108 - uses: actions/checkout@v4 1109 1110 - name: Download ALPHA artifacts 1111 uses: actions/download-artifact@v4 1112 with: 1113 name: alpha-binaries 1114 path: ./bin/ 1115 1116 - name: Download DELTA artifacts 1117 uses: actions/download-artifact@v4 1118 with: 1119 name: delta-binaries 1120 path: ./bin/ 1121 1122 - name: Make binaries executable 1123 run: chmod +x ./bin/* 1124 1125 - name: Run integration tests 1126 env: 1127 ALPHA_BIN: ./bin/alpha-node 1128 DELTA_BIN: ./bin/delta-node 1129 run: | 1130 cargo nextest run \ 1131 --workspace \ 1132 --features integration-tests \ 1133 --test '*integration*' \ 1134 --test-threads=8 \ 1135 --no-fail-fast 1136 1137 # ============================================================ 1138 # STAGE 5: Documentation & Metrics 1139 # ============================================================ 1140 1141 docs: 1142 name: "📚 Build Documentation" 1143 runs-on: native 1144 needs: [formatting] 1145 steps: 1146 - uses: actions/checkout@v4 1147 1148 - name: Build docs 1149 env: 1150 RUSTDOCFLAGS: "--enable-index-page -Zunstable-options" 1151 run: | 1152 cargo +nightly doc \ 1153 --workspace \ 1154 --no-deps \ 1155 --document-private-items 1156 1157 - name: Upload documentation 1158 uses: actions/upload-artifact@v4 1159 with: 1160 name: documentation 1161 path: /opt/ci/workspaces/target/${{ github.repository }}/doc 1162 retention-days: 30 1163 1164 sccache-stats: 1165 name: "📊 Cache Statistics" 1166 runs-on: native 1167 needs: [build-alpha, build-delta] 1168 if: always() 1169 steps: 1170 - name: Show sccache stats 1171 run: | 1172 echo "=== sccache Statistics ===" 1173 sccache --show-stats 1174 echo "" 1175 echo "=== Cache Hit Rate ===" 1176 sccache --show-stats | grep -E "(hit|miss|Cache)" 1177 1178 # ============================================================ 1179 # FINAL: Summary & Notifications 1180 # ============================================================ 1181 1182 ci-success: 1183 name: "✅ CI Complete" 1184 runs-on: native 1185 needs: [integration-tests, docs, security-audit] 1186 steps: 1187 - name: CI Summary 1188 run: | 1189 echo "🎉 All CI checks passed!" 1190 echo "" 1191 echo "Build Summary:" 1192 echo " - Format: ✅" 1193 echo " - Lint: ✅" 1194 echo " - Security: ✅" 1195 echo " - Tests: ✅" 1196 echo " - Build: ✅" 1197 echo " - Integration: ✅" 1198 echo " - Docs: ✅" 1199 ``` 1200 1201 ### Step 4.3: Nightly Benchmark Pipeline 1202 1203 Create `.forgejo/workflows/benchmarks.yml`: 1204 1205 ```yaml 1206 name: Performance Benchmarks 1207 1208 on: 1209 push: 1210 branches: [main] 1211 schedule: 1212 - cron: '0 3 * * *' # Daily at 3 AM UTC 1213 workflow_dispatch: 1214 1215 env: 1216 CARGO_TERM_COLOR: always 1217 RUSTFLAGS: "-C link-arg=-fuse-ld=mold -C target-cpu=native" 1218 1219 jobs: 1220 benchmarks: 1221 name: "⚡ Run Benchmarks" 1222 runs-on: native 1223 steps: 1224 - uses: actions/checkout@v4 1225 1226 - name: Run criterion benchmarks 1227 run: | 1228 cargo criterion \ 1229 --workspace \ 1230 --message-format=json \ 1231 > benchmark-results.json 1232 1233 - name: Generate benchmark report 1234 run: | 1235 cargo criterion --workspace 1236 1237 - name: Upload benchmark results 1238 uses: actions/upload-artifact@v4 1239 with: 1240 name: benchmark-results-${{ github.sha }} 1241 path: | 1242 benchmark-results.json 1243 target/criterion/ 1244 retention-days: 90 1245 1246 - name: Compare with baseline 1247 if: github.event_name == 'push' 1248 run: | 1249 # Compare against stored baseline 1250 if [ -f /opt/ci/baselines/benchmark-baseline.json ]; then 1251 echo "Comparing against baseline..." 1252 # Add comparison logic here 1253 fi 1254 1255 # Store new baseline for main branch 1256 cp benchmark-results.json /opt/ci/baselines/benchmark-baseline.json 1257 ``` 1258 1259 ### Step 4.4: Release Pipeline 1260 1261 Create `.forgejo/workflows/release.yml`: 1262 1263 ```yaml 1264 name: Release Build 1265 1266 on: 1267 push: 1268 tags: 1269 - 'v*' 1270 1271 env: 1272 CARGO_TERM_COLOR: always 1273 RUSTFLAGS: "-C link-arg=-fuse-ld=mold -C target-cpu=native" 1274 1275 jobs: 1276 release-build: 1277 name: "📦 Release Build" 1278 runs-on: native 1279 strategy: 1280 matrix: 1281 target: 1282 - x86_64-unknown-linux-gnu 1283 - x86_64-unknown-linux-musl 1284 steps: 1285 - uses: actions/checkout@v4 1286 1287 - name: Install target 1288 run: rustup target add ${{ matrix.target }} 1289 1290 - name: Install musl tools 1291 if: contains(matrix.target, 'musl') 1292 run: apt-get update && apt-get install -y musl-tools 1293 1294 - name: Build release binaries 1295 env: 1296 CARGO_PROFILE_RELEASE_LTO: fat 1297 CARGO_PROFILE_RELEASE_CODEGEN_UNITS: 1 1298 CARGO_PROFILE_RELEASE_OPT_LEVEL: 3 1299 CARGO_PROFILE_RELEASE_STRIP: symbols 1300 run: | 1301 cargo build \ 1302 --release \ 1303 --target ${{ matrix.target }} \ 1304 --workspace 1305 1306 - name: Package binaries 1307 run: | 1308 VERSION=${GITHUB_REF#refs/tags/} 1309 mkdir -p release-${{ matrix.target }} 1310 1311 cp target/${{ matrix.target }}/release/alpha-* release-${{ matrix.target }}/ 1312 cp target/${{ matrix.target }}/release/delta-* release-${{ matrix.target }}/ 1313 1314 tar -czvf alpha-delta-${VERSION}-${{ matrix.target }}.tar.gz \ 1315 -C release-${{ matrix.target }} . 1316 1317 - name: Upload release artifacts 1318 uses: actions/upload-artifact@v4 1319 with: 1320 name: release-${{ matrix.target }} 1321 path: "*.tar.gz" 1322 retention-days: 90 1323 ``` 1324 1325 ### Step 4.5: Add Cargo Workspace Optimization 1326 1327 Create/update `Cargo.toml` at repository root: 1328 1329 ```toml 1330 [workspace] 1331 resolver = "2" 1332 members = [ 1333 "crates/alpha-chain", 1334 "crates/alpha-node", 1335 "crates/delta-exchange", 1336 "crates/delta-node", 1337 "crates/common/*", 1338 ] 1339 1340 [workspace.package] 1341 version = "0.1.0" 1342 edition = "2021" 1343 rust-version = "1.75" 1344 authors = ["Alpha/Delta Team"] 1345 license = "MIT OR Apache-2.0" 1346 1347 [workspace.dependencies] 1348 # Define shared dependencies here for consistency 1349 tokio = { version = "1.40", features = ["full"] } 1350 serde = { version = "1.0", features = ["derive"] } 1351 # ... other shared deps 1352 1353 # Optimize dev builds for faster iteration 1354 [profile.dev] 1355 opt-level = 0 1356 debug = 1 # Reduced debug info for faster builds 1357 incremental = true 1358 1359 [profile.dev.package."*"] 1360 opt-level = 2 # Optimize dependencies even in dev 1361 1362 # Optimize release builds 1363 [profile.release] 1364 opt-level = 3 1365 lto = "thin" 1366 codegen-units = 16 1367 panic = "abort" 1368 strip = "debuginfo" 1369 1370 # Maximum optimization for production releases 1371 [profile.release-prod] 1372 inherits = "release" 1373 lto = "fat" 1374 codegen-units = 1 1375 strip = "symbols" 1376 1377 # Fast test builds 1378 [profile.test] 1379 opt-level = 1 1380 debug = 1 1381 incremental = true 1382 1383 # Benchmarks need full optimization 1384 [profile.bench] 1385 opt-level = 3 1386 lto = "thin" 1387 codegen-units = 1 1388 ``` 1389 1390 --- 1391 1392 ## Phase 5: Radicle Integration 1393 1394 ### Step 5.1: Install Radicle CLI 1395 1396 On your local development machine: 1397 1398 ```bash 1399 curl -fsSL https://releases.radicle.xyz/radicle-0.12.0-x86_64-unknown-linux-musl.tar.gz | tar -xz 1400 sudo mv radicle-*/bin/* /usr/local/bin/ 1401 rad --version 1402 ``` 1403 1404 ### Step 5.2: Initialize Radicle Project 1405 1406 ```bash 1407 cd alpha-delta-protocol 1408 1409 # Initialize Radicle identity (first time only) 1410 rad auth 1411 1412 # Initialize project 1413 rad init --name "alpha-delta-protocol" \ 1414 --description "Dual-chain sovereign monetary infrastructure" \ 1415 --default-branch main 1416 ``` 1417 1418 ### Step 5.3: Configure Forgejo-Radicle Sync Workflow 1419 1420 Create `.forgejo/workflows/radicle-sync.yml`: 1421 1422 ```yaml 1423 name: Sync to Radicle 1424 1425 on: 1426 push: 1427 branches: [main, develop] 1428 workflow_dispatch: 1429 1430 jobs: 1431 radicle-sync: 1432 name: "🌐 Push to Radicle Network" 1433 runs-on: native 1434 steps: 1435 - uses: actions/checkout@v4 1436 with: 1437 fetch-depth: 0 1438 1439 - name: Install Radicle CLI 1440 run: | 1441 if ! command -v rad &> /dev/null; then 1442 curl -fsSL https://releases.radicle.xyz/radicle-0.12.0-x86_64-unknown-linux-musl.tar.gz | tar -xz 1443 cp radicle-*/bin/* /usr/local/bin/ 1444 fi 1445 1446 - name: Configure Radicle 1447 env: 1448 RADICLE_PASSPHRASE: ${{ secrets.RADICLE_PASSPHRASE }} 1449 run: | 1450 mkdir -p ~/.radicle 1451 echo "${{ secrets.RADICLE_SSH_KEY }}" > ~/.radicle/keys/radicle 1452 chmod 600 ~/.radicle/keys/radicle 1453 1454 - name: Sync to Radicle 1455 run: | 1456 rad sync --announce 1457 1458 - name: Verify sync 1459 run: | 1460 rad inspect 1461 echo "Synced to Radicle at $(rad inspect --rid)" 1462 ``` 1463 1464 ### Step 5.4: Add Radicle Secrets to Forgejo 1465 1466 1. Go to Repository → Settings → Secrets and Variables → Actions 1467 2. Add: 1468 - `RADICLE_SSH_KEY`: Your Radicle private key 1469 - `RADICLE_PASSPHRASE`: Key passphrase (if set) 1470 1471 --- 1472 1473 ## Phase 6: Monitoring & Maintenance 1474 1475 ### Step 6.1: Runner Monitoring Script 1476 1477 Create `/usr/local/bin/runner-health.sh` on the runner: 1478 1479 ```bash 1480 #!/bin/bash 1481 # Runner health check script 1482 1483 echo "=== Runner Health Check $(date) ===" 1484 echo "" 1485 1486 # System resources 1487 echo "📊 System Resources:" 1488 echo " CPU Usage: $(top -bn1 | grep 'Cpu(s)' | awk '{print $2}')%" 1489 echo " Memory: $(free -h | awk '/^Mem:/ {print $3 "/" $2}')" 1490 echo " Disk /opt/ci: $(df -h /opt/ci | awk 'NR==2 {print $3 "/" $2 " (" $5 " used)"}')" 1491 echo " Load Average: $(uptime | awk -F'load average:' '{print $2}')" 1492 echo "" 1493 1494 # sccache stats 1495 echo "🗄️ sccache Statistics:" 1496 sccache --show-stats 2>/dev/null | grep -E "(Compile requests|Cache hits|Cache misses|Hit rate)" || echo " sccache not running" 1497 echo "" 1498 1499 # Runner status 1500 echo "🏃 Runner Status:" 1501 systemctl is-active forgejo-runner >/dev/null && echo " Service: ✅ Running" || echo " Service: ❌ Stopped" 1502 echo " Jobs capacity: $(grep 'capacity' /var/lib/forgejo-runner/config/config.yaml | awk '{print $2}')" 1503 echo "" 1504 1505 # Docker status 1506 echo "🐳 Docker:" 1507 echo " Containers: $(docker ps -q | wc -l) running" 1508 echo " Images: $(docker images -q | wc -l)" 1509 echo " Disk usage: $(docker system df --format '{{.Size}}' 2>/dev/null | head -1)" 1510 echo "" 1511 1512 # Active builds 1513 echo "🔨 Active Builds:" 1514 pgrep -a cargo 2>/dev/null | head -5 || echo " No active cargo processes" 1515 ``` 1516 1517 ```bash 1518 chmod +x /usr/local/bin/runner-health.sh 1519 ``` 1520 1521 ### Step 6.2: Automated Cache Cleanup 1522 1523 Create `/etc/cron.daily/ci-cleanup`: 1524 1525 ```bash 1526 #!/bin/bash 1527 # Daily CI cleanup script 1528 1529 LOG="/var/log/ci-cleanup.log" 1530 exec >> "$LOG" 2>&1 1531 1532 echo "=== CI Cleanup $(date) ===" 1533 1534 # Clean old target directories (keep last 7 days) 1535 find /opt/ci/workspaces -type d -name "target" -mtime +7 -exec rm -rf {} \; 2>/dev/null 1536 1537 # Clean Docker 1538 docker system prune -af --volumes --filter "until=168h" 1539 1540 # Trim sccache if over 25GB 1541 CACHE_SIZE=$(du -sb /opt/ci/sccache 2>/dev/null | awk '{print $1}') 1542 if [ "$CACHE_SIZE" -gt 26843545600 ]; then 1543 echo "Trimming sccache (current: $(du -sh /opt/ci/sccache | awk '{print $1}'))" 1544 sccache --stop-server 2>/dev/null 1545 find /opt/ci/sccache -type f -atime +14 -delete 1546 sccache --start-server 1547 fi 1548 1549 # Clean old artifacts 1550 find /opt/ci/artifacts -type f -mtime +30 -delete 2>/dev/null 1551 1552 # Clean cargo registry (keep downloaded crates for 30 days) 1553 find /opt/rust/cargo/registry/cache -type f -atime +30 -delete 2>/dev/null 1554 1555 echo "Cleanup complete" 1556 ``` 1557 1558 ```bash 1559 chmod +x /etc/cron.daily/ci-cleanup 1560 ``` 1561 1562 ### Step 6.3: Forgejo Backup Script 1563 1564 Create `/usr/local/bin/backup-forgejo.sh` on Forgejo server: 1565 1566 ```bash 1567 #!/bin/bash 1568 set -e 1569 1570 BACKUP_DIR="/var/backups/forgejo" 1571 DATE=$(date +%Y%m%d_%H%M%S) 1572 RETENTION_DAYS=14 1573 1574 mkdir -p "$BACKUP_DIR" 1575 1576 echo "Starting Forgejo backup at $(date)" 1577 1578 # Database backup 1579 sudo -u postgres pg_dump forgejo | gzip > "$BACKUP_DIR/db-$DATE.sql.gz" 1580 1581 # Repository backup (incremental) 1582 rsync -av --delete /var/lib/forgejo/repositories/ "$BACKUP_DIR/repos-latest/" 1583 1584 # Config backup 1585 tar -czf "$BACKUP_DIR/config-$DATE.tar.gz" \ 1586 /home/forgejo/custom/conf/app.ini \ 1587 /etc/nginx/sites-available/forgejo \ 1588 /etc/systemd/system/forgejo.service 1589 1590 # Cleanup old backups 1591 find "$BACKUP_DIR" -name "*.sql.gz" -mtime +$RETENTION_DAYS -delete 1592 find "$BACKUP_DIR" -name "*.tar.gz" -mtime +$RETENTION_DAYS -delete 1593 1594 echo "Backup completed: $BACKUP_DIR" 1595 ls -lh "$BACKUP_DIR"/*.gz | tail -5 1596 ``` 1597 1598 ### Step 6.4: Maintenance Checklist 1599 1600 **Weekly:** 1601 - [ ] Check runner disk usage: `df -h /opt/ci` 1602 - [ ] Review sccache hit rate: `sccache --show-stats` 1603 - [ ] Check Forgejo logs: `journalctl -u forgejo --since "1 week ago" | grep -i error` 1604 - [ ] Verify backup integrity 1605 1606 **Monthly:** 1607 - [ ] Update Forgejo: Download new binary, restart service 1608 - [ ] Update Forgejo-Runner: Download new binary, restart service 1609 - [ ] Update Rust toolchain: `rustup update` 1610 - [ ] Update mold linker if new version available 1611 - [ ] Review and rotate secrets/tokens 1612 - [ ] Test certificate renewal: `certbot renew --dry-run` 1613 1614 **Quarterly:** 1615 - [ ] Full security audit: `cargo audit`, review CVEs 1616 - [ ] Backup restoration test 1617 - [ ] Performance baseline comparison 1618 - [ ] Review and optimize CI pipeline times 1619 - [ ] Clean up old artifacts in S3 1620 1621 --- 1622 1623 ## Appendix: Performance Tuning Reference 1624 1625 ### Expected Build Times (32-core Runner) 1626 1627 | Build Type | Cold Cache | Warm Cache | Incremental | 1628 |------------|------------|------------|-------------| 1629 | `cargo check` | ~45s | ~15s | ~5s | 1630 | `cargo test` (all) | ~3min | ~1min | ~30s | 1631 | `cargo build --release` (single crate) | ~2min | ~45s | ~20s | 1632 | Full release (ALPHA + DELTA) | ~5min | ~2min | ~1min | 1633 | Integration tests | ~4min | ~2min | ~1min | 1634 | **Complete CI pipeline** | ~12min | ~6min | ~4min | 1635 1636 ### Resource Utilization Targets 1637 1638 | Resource | Target | Alert Threshold | 1639 |----------|--------|-----------------| 1640 | CPU (during build) | 80-95% | <50% (underutilized) | 1641 | Memory | 40-70% | >85% | 1642 | Disk I/O | NVMe saturated OK | >95% util for >5min | 1643 | sccache hit rate | >70% | <50% | 1644 | Job queue | <3 pending | >8 pending | 1645 1646 ### Troubleshooting Quick Reference 1647 1648 **Slow builds:** 1649 ```bash 1650 # Check sccache is working 1651 sccache --show-stats | grep "Cache hits" 1652 1653 # Verify mold is being used 1654 readelf -p .comment target/release/alpha-chain | grep mold 1655 1656 # Check for I/O bottleneck 1657 iostat -x 1 5 1658 1659 # Monitor compilation parallelism 1660 watch -n1 'ps aux | grep rustc | wc -l' 1661 ``` 1662 1663 **Runner not picking up jobs:** 1664 ```bash 1665 # Check runner connection 1666 journalctl -u forgejo-runner -n 50 | grep -i error 1667 1668 # Verify network to Forgejo 1669 curl -I https://ci.yourdomain.com/api/v1/version 1670 1671 # Re-register if needed 1672 systemctl stop forgejo-runner 1673 rm /var/lib/forgejo-runner/.runner 1674 # Re-run registration command 1675 systemctl start forgejo-runner 1676 ``` 1677 1678 **OOM during compilation:** 1679 ```bash 1680 # Reduce parallelism temporarily 1681 CARGO_BUILD_JOBS=16 cargo build --release 1682 1683 # Or enable disk swap for large builds 1684 fallocate -l 16G /swapfile 1685 chmod 600 /swapfile 1686 mkswap /swapfile 1687 swapon /swapfile 1688 ``` 1689 1690 --- 1691 1692 ## Summary 1693 1694 ### Cost Estimate (Monthly) 1695 1696 | Item | Cost | 1697 |------|------| 1698 | Forgejo Droplet (2GB/2vCPU) | $12 | 1699 | Runner Droplet (64GB/32vCPU) | ~$336 | 1700 | DO Spaces (100GB) | ~$5 | 1701 | Reserved IP | $4 | 1702 | Bandwidth (~500GB) | ~$45 | 1703 | **Total** | **~$402/month** | 1704 1705 ### Performance Improvements vs. Baseline 1706 1707 | Metric | Baseline (4-core) | Optimized (32-core) | Improvement | 1708 |--------|-------------------|---------------------|-------------| 1709 | Full CI cycle | ~25min | ~6min | **4x faster** | 1710 | Parallel jobs | 2 | 8 | **4x capacity** | 1711 | sccache hit rate | ~40% | ~75% | **Reduced rebuilds** | 1712 | Release build | ~15min | ~2min | **7x faster** | 1713 1714 --- 1715 1716 **Document Version:** 2.0 1717 **Last Updated:** January 2026 1718 **Status:** Production-Ready for High-Performance Deployment