/ INTEGRATION_PLAN_v2.md
INTEGRATION_PLAN_v2.md
1 # Deboot Integration Plan v2 2 3 ## The Goal 4 5 ``` 6 Physical Machine Swarm Network 7 ┌─────────────┐ ┌─────────────┐ 8 │ SWFETCH.EFI │◄────────►│ Peers │ 9 │ │ TCP │ │ 10 │ - Swarm │ Noise │ bzz://... │ 11 │ - BMT │ Yamux │ │ 12 │ - Kademlia │ │ │ 13 └─────────────┘ └─────────────┘ 14 15 No bridge. No intermediary. Pure decentralized boot. 16 ``` 17 18 Machine powers on → EFI fetches OS from Swarm → boots. 19 20 ## Development Strategy 21 22 ``` 23 Dev: SWFETCH.EFI ──libp2p──► Bridge ──Swarm protocols──► Swarm Network 24 │ 25 └── Bridge does the Swarm work 26 27 Prod: SWFETCH.EFI ──Swarm protocols──► Swarm Network 28 │ 29 └── EFI does it directly 30 ``` 31 32 The "sneaky dev trick" (credit: aata): 33 - Bridge talks to **real Swarm** - not just local files 34 - SWFETCH.EFI is a "boilerplate" that just speaks libp2p to the bridge 35 - Full flow works immediately (real content from Swarm) 36 - Later, "squeeze" real Swarm code into SWFETCH.EFI 37 - No blockers - parallel development 38 39 ## Architecture Overview 40 41 ``` 42 ┌─────────────────────────────────────────────────────────────────────────────────┐ 43 │ Production (Fully Decentralized) │ 44 ├─────────────────────────────────────────────────────────────────────────────────┤ 45 │ │ 46 │ ┌─────────────────────┐ │ 47 │ │ Physical Machine │ ┌─────────────────────────────────┐ │ 48 │ │ (UEFI firmware) │ TCP │ Swarm Network │ │ 49 │ │ │ ◄─────► │ │ │ 50 │ │ ┌───────────────┐ │ Noise │ ┌─────────┐ ┌─────────┐ │ │ 51 │ │ │ SWFETCH.EFI │ │ Yamux │ │ Peer │ │ Peer │ │ │ 52 │ │ │ │ │ │ └────┬────┘ └────┬────┘ │ │ 53 │ │ │ - Swarm proto │ │ │ │ │ │ │ 54 │ │ │ - BMT verify │ │ │ ▼ ▼ │ │ 55 │ │ │ - Kademlia │ │ │ ┌─────────────────────┐ │ │ 56 │ │ └───────────────┘ │ │ │ Boot artifacts │ │ │ 57 │ │ │ │ │ │ (content-addressed)│ │ │ 58 │ │ ▼ │ │ │ bzz://abc123... │ │ │ 59 │ │ Loads & boots │ │ └─────────────────────┘ │ │ 60 │ │ │ │ │ │ 61 │ └─────────────────────┘ └─────────────────────────────────┘ │ 62 │ │ 63 │ No bridge. No intermediary. EFI talks directly to Swarm peers. │ 64 │ │ 65 └─────────────────────────────────────────────────────────────────────────────────┘ 66 67 ┌─────────────────────────────────────────────────────────────────────────────────┐ 68 │ Development Setup │ 69 ├─────────────────────────────────────────────────────────────────────────────────┤ 70 │ │ 71 │ ┌─────────────────────┐ ┌─────────────────────────────────┐ │ 72 │ │ QEMU + OVMF │ UDP │ Host OS (your machine) │ │ 73 │ │ │ ◄─────► │ │ │ 74 │ │ ┌───────────────┐ │ libp2p │ ┌───────────────────────────┐ │ │ 75 │ │ │ SWFETCH.EFI │ │ QUIC │ │ Bridge (Go) │ │ │ 76 │ │ │ (boilerplate) │ │ │ │ │ │ │ 77 │ │ │ │ │ │ │ - Speaks libp2p to EFI │ │ │ 78 │ │ │ + nectar BMT │ │ │ │ - Fetches from Swarm │ │ │ 79 │ │ │ verification│ │ │ └─────────────┬─────────────┘ │ │ 80 │ │ └───────────────┘ │ │ │ │ │ 81 │ │ │ │ │ │ Swarm protocols│ │ 82 │ │ ▼ │ │ ▼ │ │ 83 │ │ Loads into memory │ │ ┌─────────────────┐ │ │ 84 │ │ & boots kernel │ │ │ Swarm Network │ │ │ 85 │ │ │ │ │ bzz://... │ │ │ 86 │ └─────────────────────┘ │ └─────────────────┘ │ │ 87 │ │ │ │ 88 │ SWFETCH.EFI is "dumb" - just speaks libp2p. Bridge does Swarm. │ │ 89 │ Later: squeeze real Swarm code into SWFETCH.EFI, remove bridge. │ │ 90 │ │ 91 └─────────────────────────────────────────────────────────────────────────────────┘ 92 ``` 93 94 ## What We're Using 95 96 | Component | Role | Why | 97 |-----------|------|-----| 98 | **libp2p.efi** | EFI bootloader | Core - will implement native Swarm | 99 | **debootd** | Post-boot daemon | Core - manages updates | 100 | **nectar-primitives** | BMT verification | no_std compatible, used in EFI | 101 | **kabashira** | Protocol reference | Documents Swarm wire protocols | 102 | **bridge/** | Dev/test only | Temporary scaffold, not production | 103 104 ## Phase 1: Bridge + Boilerplate SWFETCH.EFI (Week 1) 105 106 **Goal**: Bridge that fetches from real Swarm, boilerplate EFI that talks libp2p to bridge 107 108 ### 1.1 Bridge structure 109 110 ``` 111 deboot/ 112 ├── bridge/ 113 │ ├── go.mod 114 │ ├── main.go # Entry point, libp2p host setup 115 │ ├── handlers/ 116 │ │ └── fetch.go # /deboot/fetch/1.0.0 protocol handler 117 │ └── swarm/ 118 │ └── client.go # Fetches from real Swarm network 119 ``` 120 121 ### 1.2 Bridge implementation 122 123 ```go 124 // bridge/main.go 125 package main 126 127 import ( 128 "github.com/libp2p/go-libp2p" 129 quic "github.com/libp2p/go-libp2p/p2p/transport/quic" 130 ) 131 132 func main() { 133 // Initialize Swarm client (connects to Swarm network) 134 swarmClient := swarm.NewClient() 135 136 host, _ := libp2p.New( 137 libp2p.Transport(quic.NewTransport), 138 libp2p.ListenAddrStrings("/ip4/0.0.0.0/udp/4001/quic-v1"), 139 ) 140 141 host.SetStreamHandler("/deboot/fetch/1.0.0", func(s network.Stream) { 142 handleFetch(s, swarmClient) 143 }) 144 145 select {} 146 } 147 ``` 148 149 ```go 150 // bridge/handlers/fetch.go 151 func handleFetch(s network.Stream, swarm *swarm.Client) { 152 defer s.Close() 153 154 var req FetchRequest 155 json.NewDecoder(s).Decode(&req) 156 157 switch req.Type { 158 case "manifest": 159 // Fetch manifest from Swarm 160 data, _ := swarm.Fetch(req.ManifestHash) 161 sendResponse(s, data) 162 case "artifact": 163 // Fetch artifact from Swarm 164 data, _ := swarm.Fetch(req.Hash) 165 sendResponse(s, data) 166 } 167 } 168 ``` 169 170 ### 1.3 Protocol format 171 172 ``` 173 REQUEST: { "type": "manifest", "hash": "<manifest-swarm-hash>" } 174 REQUEST: { "type": "artifact", "hash": "<swarm-hash>" } 175 176 RESPONSE: { "ok": true, "size": 1234 } 177 <raw bytes follow> 178 179 RESPONSE: { "ok": false, "error": "not found" } 180 ``` 181 182 ### 1.4 Bridge Swarm client 183 184 The bridge uses kabashira or bee API to fetch from real Swarm: 185 186 ```go 187 // bridge/swarm/client.go 188 type Client struct { 189 // Option A: Use bee node API 190 beeURL string 191 192 // Option B: Use kabashira's abu 193 // (shell out or use as library) 194 } 195 196 func (c *Client) Fetch(hash string) ([]byte, error) { 197 // Fetch from real Swarm network 198 resp, _ := http.Get(fmt.Sprintf("%s/bzz/%s", c.beeURL, hash)) 199 return io.ReadAll(resp.Body) 200 } 201 ``` 202 203 ## Phase 2: Boilerplate SWFETCH.EFI (Week 1-2) 204 205 **Goal**: SWFETCH.EFI talks libp2p to bridge, verifies with nectar 206 207 This is the "dumb" version - it just speaks libp2p to the bridge. The bridge does the real Swarm work. 208 209 ### 2.1 SWFETCH.EFI structure 210 211 ``` 212 libp2p/src/ 213 ├── main.rs # Entry point → SWFETCH.EFI 214 ├── fetch/ # NEW 215 │ ├── mod.rs 216 │ ├── protocol.rs # /deboot/fetch/1.0.0 client (talks to bridge) 217 │ └── manifest.rs # Parse manifest JSON 218 ├── verify/ # NEW 219 │ └── bmt.rs # Uses nectar-primitives 220 └── boot/ # NEW 221 ├── mod.rs 222 ├── loader.rs # Load kernel/initramfs into memory 223 └── handoff.rs # Exit boot services, jump to kernel 224 ``` 225 226 ### 2.2 Add nectar dependency 227 228 ```toml 229 # libp2p/Cargo.toml 230 [dependencies] 231 nectar-primitives = { path = "../nectar/crates/primitives", default-features = false } 232 ``` 233 234 ```rust 235 // libp2p/src/verify/bmt.rs 236 use nectar_primitives::{DefaultHasher, Chunk}; 237 238 pub fn verify_content(data: &[u8], expected_hash: &[u8; 32]) -> bool { 239 let mut hasher = DefaultHasher::new(); 240 hasher.set_span(data.len() as u64); 241 hasher.update(data); 242 hasher.sum() == *expected_hash 243 } 244 ``` 245 246 ### 2.3 Boot sequence (dev mode) 247 248 1. Initialize UEFI networking 249 2. Connect to bridge at hardcoded IP (10.0.2.2:4001 for QEMU) 250 3. Request manifest (bridge fetches from Swarm) 251 4. Request vmlinuz (bridge fetches from Swarm), verify BMT hash 252 5. Request initramfs (bridge fetches from Swarm), verify BMT hash 253 6. Load into memory, set up boot params 254 7. Exit boot services, jump to kernel 255 256 **Note**: SWFETCH.EFI doesn't know about Swarm yet - it just talks libp2p to the bridge. BMT verification ensures content integrity regardless of source. 257 258 ## Phase 3: Integration Testing (Week 2) 259 260 Test the full flow: SWFETCH.EFI → Bridge → Swarm → Boot 261 262 ```bash 263 #!/bin/bash 264 # scripts/run-test.sh 265 266 # 1. Build everything 267 (cd debootd/image && sudo ./build.sh) 268 (cd libp2p && cargo build --release) # Produces SWFETCH.EFI 269 (cd bridge && go build -o deboot-bridge) 270 271 # 2. Upload artifacts to Swarm (requires bee node) 272 MANIFEST_HASH=$(swarm-cli upload ./debootd/image/host-configuration-manifest.json) 273 echo "Manifest hash: $MANIFEST_HASH" 274 275 # 3. Start bridge (connects to Swarm) 276 ./bridge/deboot-bridge --bee-url http://localhost:1633 & 277 BRIDGE_PID=$! 278 279 # 4. Create test disk with SWFETCH.EFI 280 ./scripts/create-test-disk.sh --manifest-hash $MANIFEST_HASH 281 282 # 5. Boot QEMU 283 qemu-system-x86_64 \ 284 -bios /usr/share/OVMF/OVMF_CODE.fd \ 285 -drive file=test-disk.img,format=raw \ 286 -netdev user,id=net0 \ 287 -device virtio-net-pci,netdev=net0 \ 288 -serial stdio | tee boot.log & 289 290 # 6. Wait and check 291 sleep 60 292 grep "debootd ready" boot.log && echo "SUCCESS" 293 294 kill $BRIDGE_PID 295 ``` 296 297 **What's happening:** 298 1. Artifacts uploaded to real Swarm 299 2. Bridge connects to Swarm via bee node 300 3. SWFETCH.EFI asks bridge for content 301 4. Bridge fetches from Swarm, returns to EFI 302 5. EFI verifies BMT hashes, boots kernel 303 304 ## Phase 4: Debootd Integration (Week 3) 305 306 Debootd monitors ENS and fetches updates. Initially can use the same bridge, later direct Swarm access. 307 308 ## Phase 5: Native Swarm in SWFETCH.EFI (The Goal) 309 310 **This is the end state. No compromise.** 311 312 SWFETCH.EFI speaks Swarm protocols directly - no bridge, no intermediary. 313 314 "Squeeze" real Swarm code into SWFETCH.EFI, replacing the libp2p-to-bridge communication. 315 316 ### 5.1 What SWFETCH.EFI needs to implement 317 318 ``` 319 ┌─────────────────────────────────────┐ 320 │ SWFETCH.EFI (production) │ 321 ├─────────────────────────────────────┤ 322 │ Swarm Protocols: │ 323 │ - /swarm/handshake/14.0.0 │ 324 │ - /swarm/retrieval/1.4.0 │ 325 │ - /swarm/hive/1.1.0 (peer discovery)│ 326 ├─────────────────────────────────────┤ 327 │ Transport: │ 328 │ - TCP (smoltcp, no_std) │ 329 │ - Noise encryption │ 330 │ - Yamux multiplexing │ 331 ├─────────────────────────────────────┤ 332 │ Verification: │ 333 │ - BMT hashing (nectar-primitives) │ 334 │ - Content addressing │ 335 ├─────────────────────────────────────┤ 336 │ Bootstrap: │ 337 │ - Hardcoded bootnodes │ 338 │ - Or ENS lookup for bootnode list │ 339 └─────────────────────────────────────┘ 340 ``` 341 342 ### 5.2 Production boot flow 343 344 1. EFI starts, initializes networking via UEFI SNP 345 2. Connects to hardcoded Swarm bootnode(s) 346 3. Performs Hive protocol to discover peers 347 4. Requests manifest hash (from ENS or hardcoded) 348 5. Fetches vmlinuz, initramfs via /swarm/retrieval 349 6. Verifies BMT hashes 350 7. Boots kernel 351 352 ### 5.3 The transition 353 354 ``` 355 Before (dev): SWFETCH.EFI ──libp2p──► Bridge ──► Swarm 356 After (prod): SWFETCH.EFI ──────────────────────► Swarm 357 ``` 358 359 The external interface stays the same (EFI boots from content hash). Only the internals change. 360 361 ### 5.4 Reference implementations 362 363 - **kabashira** - Rust Swarm client (std, but protocol reference) 364 - **bee** - Go reference implementation 365 - **nectar** - Low-level primitives (no_std compatible) 366 367 ## Next Steps 368 369 | Task | Priority | Notes | 370 |------|----------|-------| 371 | Create bridge/ with Go libp2p server | P0 | Dev scaffold only | 372 | Verify nectar-primitives builds no_std | P0 | Required for EFI | 373 | Add fetch module to libp2p.efi | P0 | Start with bridge protocol | 374 | Add BMT verification using nectar | P0 | | 375 | Integration test harness | P1 | | 376 | Implement TCP in libp2p.efi (smoltcp) | P1 | For native Swarm | 377 | Implement Noise encryption in EFI | P1 | For native Swarm | 378 | Implement /swarm/handshake in EFI | P2 | | 379 | Implement /swarm/retrieval in EFI | P2 | | 380 | Implement /swarm/hive in EFI | P2 | Peer discovery | 381 | Debootd update logic | P2 | | 382 | Remove bridge dependency | P3 | Production ready | 383 384 ## Repository Structure 385 386 ``` 387 deboot/ 388 ├── bridge/ # Go libp2p bridge (serves artifacts) 389 ├── scripts/ # Test scripts 390 ├── libp2p/ # libp2p.efi (will use nectar) 391 ├── debootd/ # Boot daemon 392 ├── nectar/ # BMT primitives (used as dependency) 393 ├── kabashira/ # Reference only 394 └── kabashira-docs/ # Reference only 395 ```