TECHNICAL_SPECIFICATION_old.md
1 # Abzu Technical Specification 2 3 > **Version**: 0.4.0 | **Status**: Production-Ready | **Updated**: January 27, 2026 4 5 --- 6 7 ## Table of Contents 8 9 1. [Executive Summary](#executive-summary) 10 2. [Architecture Overview](#architecture-overview) 11 3. [Cryptographic Foundations](#cryptographic-foundations) 12 4. [Wire Protocol](#wire-protocol) 13 5. [Transport Security](#transport-security) 14 6. [Routing Layer](#routing-layer) 15 7. [Trust Circles](#trust-circles) 16 8. [Content-Addressed Storage](#content-addressed-storage) 17 9. [Security Hardening](#security-hardening) 18 10. [API Reference](#api-reference) 19 11. [Test Coverage](#test-coverage) 20 12. [Known Limitations](#known-limitations) 21 13. [Roadmap](#roadmap) 22 23 --- 24 25 ## Executive Summary 26 27 Abzu is a sovereign mesh networking protocol designed for censorship-resistant, privacy-preserving communication. It enables peer-to-peer overlay networking with strong cryptographic guarantees, traffic analysis resistance, and decentralized trust. 28 29 ### Core Principles 30 31 | Principle | Implementation | 32 |-----------|---------------| 33 | **Sovereignty** | Every node is a first-class citizen; no central authority | 34 | **Zero-Trust** | Verify everything; trust nothing implicitly | 35 | **Defense in Depth** | Multiple layers of security; single compromise doesn't break system | 36 | **Privacy by Design** | Metadata minimization; traffic analysis resistance | 37 | **Local-First** | Operate without network; sync when available | 38 39 ### Key Capabilities 40 41 - **Cryptographic Identity**: Ed25519 keypairs for authentication 42 - **Perfect Forward Secrecy**: X25519 ephemeral keys per session 43 - **Stealth Transport**: FakeTLS masquerades as legitimate HTTPS 44 - **Multi-Hop Routing**: Onion-style nested encryption 45 - **Traffic Analysis Resistance**: Ghost Mode with cover traffic 46 - **Group Communication**: Trust Circles with epoch-based encryption 47 - **Content Addressing**: BLAKE3-based content-addressed storage 48 49 --- 50 51 ## Architecture Overview 52 53 ### Crate Structure 54 55 ``` 56 abzu/ 57 ├── abzu-core/ # Central state: Node, Switchboard, Circles 58 ├── abzu-router/ # Routing: RoutingTable, TreeCoords, PathBuilder 59 ├── abzu-transport/ # Wire: AbzuFrame, FakeTLS, Cover Traffic 60 ├── abzu-token/ # Economy: Balance, Ledger, Bonds, Market 61 ├── abzu-daemon/ # Runtime: CLI, RPC Server, WebSocket 62 ├── abzu-sdk/ # Client API: AbzuClient, Configuration 63 └── abzu-ffi/ # Foreign: C bindings for mobile/native 64 ``` 65 66 ### Component Relationships 67 68 ``` 69 ┌─────────────────────────────────────────────────────────────┐ 70 │ abzu-daemon │ 71 │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │ 72 │ │ RPC Server │ │ WebSocket │ │ CLI Interface │ │ 73 │ └──────┬──────┘ └──────┬──────┘ └──────────┬──────────┘ │ 74 └─────────┼────────────────┼────────────────────┼─────────────┘ 75 │ │ │ 76 ▼ ▼ ▼ 77 ┌─────────────────────────────────────────────────────────────┐ 78 │ abzu-sdk │ 79 │ ┌─────────────────────────────────────────────────────┐ │ 80 │ │ AbzuClient │ │ 81 │ │ • connect() • send_chat() • store_content() │ │ 82 │ └─────────────────────────────────────────────────────┘ │ 83 └──────────────────────────┬──────────────────────────────────┘ 84 │ 85 ┌────────────────┼────────────────┐ 86 ▼ ▼ ▼ 87 ┌────────────────┐ ┌───────────────┐ ┌─────────────────┐ 88 │ abzu-core │ │ abzu-router │ │ abzu-transport │ 89 │ ┌────────────┐ │ │ ┌───────────┐ │ │ ┌─────────────┐ │ 90 │ │ Node │ │ │ │ Routing │ │ │ │ AbzuFrame │ │ 91 │ │Switchboard │ │ │ │ Table │ │ │ │ FakeTLS │ │ 92 │ │ Circles │ │ │ │TreeCoords │ │ │ │ CoverTraffic│ │ 93 │ │ContentStore│ │ │ └───────────┘ │ │ └─────────────┘ │ 94 │ └────────────┘ │ └───────────────┘ └─────────────────┘ 95 └────────────────┘ 96 ``` 97 98 ### Data Flow 99 100 ``` 101 ┌──────────────────────────────────────────────────────────────┐ 102 │ Incoming Message │ 103 └──────────────────────────┬───────────────────────────────────┘ 104 ▼ 105 ┌──────────────────────────────────────────────────────────────┐ 106 │ 1. FakeTLS Decrypt (ChaCha20-Poly1305) │ 107 └──────────────────────────┬───────────────────────────────────┘ 108 ▼ 109 ┌──────────────────────────────────────────────────────────────┐ 110 │ 2. Postcard Deserialize → AbzuFrame │ 111 └──────────────────────────┬───────────────────────────────────┘ 112 ▼ 113 ┌──────────────────────────────────────────────────────────────┐ 114 │ 3. Switchboard Dispatch │ 115 │ ├─ Route frame → Forward to next hop │ 116 │ ├─ Chunk frame → Store in ContentStore │ 117 │ ├─ Chat frame → Deliver to application │ 118 │ ├─ Circle msg → Decrypt with epoch key, deliver │ 119 │ └─ Cover frame → Discard (traffic padding) │ 120 └──────────────────────────────────────────────────────────────┘ 121 ``` 122 123 --- 124 125 ## Cryptographic Foundations 126 127 ### Algorithm Selection 128 129 | Purpose | Algorithm | Rationale | 130 |---------|-----------|-----------| 131 | Identity | Ed25519 | Compact signatures, fast verification | 132 | Key Exchange | X25519 | Elliptic curve Diffie-Hellman | 133 | Encryption | ChaCha20-Poly1305 | AEAD, constant-time, no padding oracle | 134 | Hashing | BLAKE3 | Fast, parallel, XOF capability | 135 | KDF | HKDF-BLAKE3 | RFC 5869 with BLAKE3 as PRF | 136 | Password KDF | Argon2id | Memory-hard, side-channel resistant | 137 138 ### Key Hierarchy 139 140 ``` 141 ┌─────────────────────────────────────────────────────────────┐ 142 │ Identity Keypair (Ed25519) │ 143 │ Stored encrypted with Argon2id + AES-GCM │ 144 └──────────────────────────┬──────────────────────────────────┘ 145 │ 146 ▼ 147 ┌─────────────────────────────────────────────────────────────┐ 148 │ Ephemeral Keypair (X25519) │ 149 │ Generated per-connection │ 150 └──────────────────────────┬──────────────────────────────────┘ 151 │ 152 ▼ 153 ┌─────────────────────────────────────────────────────────────┐ 154 │ Session Key (256-bit) │ 155 │ Derived: HKDF(shared_secret, salt, "abzu-session-v1") │ 156 └──────────────────────────┬──────────────────────────────────┘ 157 │ 158 ┌──────────────┼──────────────┐ 159 ▼ ▼ ▼ 160 ┌──────────┐ ┌──────────┐ ┌──────────┐ 161 │ send_key │ │ recv_key │ │ nonce_iv │ 162 └──────────┘ └──────────┘ └──────────┘ 163 ``` 164 165 ### Identity Binding 166 167 Session keys are cryptographically bound to identities: 168 169 ``` 170 session_key = HKDF( 171 ikm: X25519(ephemeral_secret, peer_ephemeral_public), 172 salt: SORT(our_identity_pub || peer_identity_pub), 173 info: "abzu-session-v1" 174 ) 175 ``` 176 177 This prevents MITM attacks even if ephemeral keys are intercepted. 178 179 --- 180 181 ## Wire Protocol 182 183 ### Frame Types 184 185 ```rust 186 #[derive(Serialize, Deserialize)] 187 pub enum AbzuFrame { 188 // Handshake 189 Hello { 190 version_major: u16, 191 version_minor: u16, 192 ephemeral_pub: [u8; 32], 193 timestamp: u64, 194 }, 195 HelloAck { 196 version_major: u16, 197 version_minor: u16, 198 ephemeral_pub: [u8; 32], 199 confirmation: Vec<u8>, 200 }, 201 202 // Content 203 Chunk { cid: [u8; 32], data: Vec<u8> }, 204 Request { cid: [u8; 32], requester: [u8; 32] }, 205 206 // Routing 207 Route { 208 target: [u8; 32], 209 next_hop: [u8; 32], 210 payload: Vec<u8>, 211 }, 212 213 // Messaging 214 Chat { id: u64, to: [u8; 32], msg: Vec<u8>, timestamp: u64 }, 215 ChatAck { id: u64 }, 216 217 // Groups 218 CircleMessage { circle_id: [u8; 32], epoch: u32, ciphertext: Vec<u8> }, 219 220 // Traffic padding 221 Cover { padding: Vec<u8> }, 222 KeepAlive, 223 } 224 ``` 225 226 ### Protocol Version Negotiation 227 228 Versions are exchanged during handshake: 229 230 1. **Initiator** sends `Hello` with their version 231 2. **Responder** checks compatibility: 232 - Reject if peer major version < `MIN_COMPATIBLE_VERSION_MAJOR` 233 - Reject if peer major version > our major version 234 - If major versions match, negotiate to minimum minor version 235 3. **Responder** sends `HelloAck` with negotiated version 236 237 ```rust 238 pub const PROTOCOL_VERSION_MAJOR: u16 = 1; 239 pub const PROTOCOL_VERSION_MINOR: u16 = 0; 240 pub const MIN_COMPATIBLE_VERSION_MAJOR: u16 = 1; 241 ``` 242 243 ### Serialization 244 245 - **Format**: Postcard (no_std compatible, compact binary) 246 - **Framing**: Length-prefixed (4-byte big-endian length header) 247 - **Encryption**: ChaCha20-Poly1305 AEAD with 12-byte nonce 248 249 --- 250 251 ## Transport Security 252 253 ### FakeTLS 254 255 FakeTLS masquerades connections as standard TLS 1.3 to evade Deep Packet Inspection: 256 257 | Aspect | Implementation | 258 |--------|----------------| 259 | Record Layer | Standard TLS 1.3 (content type 0x17) | 260 | ClientHello | Genuine TLS 1.3 with randomized fields | 261 | SNI | Configurable benign domain (e.g., "cloudflare.com") | 262 | Cipher | ChaCha20-Poly1305 (same as wire protocol) | 263 264 ```rust 265 pub struct FakeTlsStream { 266 inner: TcpStream, 267 cipher: ChaCha20Poly1305, 268 sni_domain: String, 269 } 270 ``` 271 272 ### Ghost Mode 273 274 Ghost Mode provides traffic analysis resistance through: 275 276 1. **Cover Traffic**: Fake packets between real messages 277 2. **Timing Obfuscation**: Randomized transmission intervals 278 3. **Size Bucketing**: Pad packets to fixed size buckets 279 280 #### Timing Strategies 281 282 | Strategy | Description | Use Case | 283 |----------|-------------|----------| 284 | Fixed | Constant interval | Simple, predictable | 285 | Poisson | Exponential distribution | Statistical cover | 286 | Adaptive | Match observed traffic patterns | Blend with real traffic | 287 288 #### Size Buckets 289 290 Packets are padded to the next bucket size: 291 292 ```rust 293 const BUCKETS: [usize; 6] = [64, 256, 1024, 4096, 16384, 65536]; 294 295 fn pad_to_bucket(data: &[u8]) -> Vec<u8> { 296 let bucket_size = BUCKETS.iter() 297 .find(|&&s| s >= data.len()) 298 .unwrap_or(&65536); 299 let mut padded = data.to_vec(); 300 padded.resize(*bucket_size, 0); 301 padded 302 } 303 ``` 304 305 ### NAT Traversal 306 307 | Technique | Description | 308 |-----------|-------------| 309 | STUN | Discover external address and NAT type | 310 | Hole Punching | UDP punch through compatible NATs | 311 | TURN Relay | Fallback for symmetric NATs | 312 313 --- 314 315 ## Routing Layer 316 317 ### Spanning Tree Coordinates 318 319 Inspired by Yggdrasil, each node has tree coordinates representing its path from the root: 320 321 ```rust 322 pub struct TreeCoords { 323 pub path: Vec<u8>, // Port indices from root 324 } 325 326 impl TreeCoords { 327 pub fn distance(&self, other: &TreeCoords) -> usize { 328 let lca = self.lowest_common_ancestor(other); 329 (self.path.len() - lca) + (other.path.len() - lca) 330 } 331 } 332 ``` 333 334 ### Hybrid Routing 335 336 ```rust 337 impl RoutingTable { 338 pub fn next_hop(&self, dest_key: &[u8; 32], dest_coords: Option<&TreeCoords>) 339 -> Option<[u8; 32]> 340 { 341 // 1. Try tree-based routing first 342 if let (Some(self_coords), Some(dest_coords)) = 343 (&self.self_coords, dest_coords) 344 { 345 if let Some(direction) = self.tree_direction(self_coords, dest_coords) { 346 return Some(direction); 347 } 348 } 349 350 // 2. Fall back to XOR distance (DHT-style) 351 self.closest_to_xor(dest_key) 352 } 353 } 354 ``` 355 356 ### Multi-Hop Routing 357 358 For enhanced privacy, messages can be routed through multiple intermediate nodes: 359 360 ```rust 361 pub fn build_onion_path( 362 target: [u8; 32], 363 hops: &[[u8; 32]], 364 ) -> Vec<AbzuFrame> { 365 let mut layers = Vec::new(); 366 for hop in hops.iter().rev() { 367 // Wrap each layer with next hop's encryption 368 layers.push(encrypt_layer(*hop, &layers)); 369 } 370 layers 371 } 372 ``` 373 374 --- 375 376 ## Trust Circles 377 378 Circles provide group communication with forward secrecy: 379 380 ### Structure 381 382 ```rust 383 pub struct Circle { 384 pub id: [u8; 32], // BLAKE3(creator || name || nonce) 385 pub name: String, 386 pub members: Vec<Member>, 387 pub epoch: u32, // Rotates on membership change 388 pub current_key: [u8; 32], // Derived from epoch 389 pub policy: TrustPolicy, // Admin, vouch-based, or epidemic 390 } 391 ``` 392 393 ### Epoch-Based Key Rotation 394 395 When membership changes, the epoch increments and new keys are derived: 396 397 ```rust 398 fn derive_epoch_key(circle_id: &[u8; 32], epoch: u32) -> [u8; 32] { 399 let mut hasher = blake3::Hasher::new(); 400 hasher.update(circle_id); 401 hasher.update(&epoch.to_le_bytes()); 402 *hasher.finalize().as_bytes() 403 } 404 ``` 405 406 ### Trust Policies 407 408 | Policy | Description | 409 |--------|-------------| 410 | Admin | Single admin controls membership | 411 | Vouch | M-of-N members must vouch for new member | 412 | Epidemic | Gossip-based membership (BFT threshold) | 413 414 --- 415 416 ## Content-Addressed Storage 417 418 ### Content Store 419 420 ```rust 421 pub struct ContentStore { 422 chunks: DashMap<[u8; 32], Vec<u8>>, // CID → data 423 metadata: DashMap<[u8; 32], ChunkMeta>, 424 } 425 426 impl ContentStore { 427 pub fn store(&self, data: &[u8]) -> [u8; 32] { 428 let cid = blake3::hash(data); 429 self.chunks.insert(*cid.as_bytes(), data.to_vec()); 430 *cid.as_bytes() 431 } 432 433 pub fn get(&self, cid: &[u8; 32]) -> Option<Vec<u8>> { 434 self.chunks.get(cid).map(|r| r.value().clone()) 435 } 436 } 437 ``` 438 439 ### Chunking Strategy 440 441 Large content is split into fixed-size chunks: 442 443 - Default chunk size: 256 KiB 444 - Each chunk has independent CID 445 - Manifest CID references all chunk CIDs 446 447 --- 448 449 ## Token Economy 450 451 The Abzu Token (ABZU) is a native service credit designed for resource allocation, spam prevention, and Sybil resistance. It is NOT an investment vehicle. 452 453 ### Core Primitives 454 455 | Primitive | Description | Implementation | 456 |-----------|-------------|----------------| 457 | **Balance** | 18-decimal precision service credits | `abzu-token::Balance` (u128 drops) | 458 | **Ledger** | Abstract trait for balance tracking | `MemoryLedger` / `DhtLedger` | 459 | **Bond** | Staked tokens for reputation/service | `ReputationBond` (7-day cooldown) | 460 461 ### Distributed State 462 463 Token data is stored in the DHT using specific `ValueType` variants: 464 465 ```rust 466 pub enum ValueType { 467 // ... existing types ... 468 TokenBalance = 10, // Encrypted balance record 469 TokenBond = 11, // Reputation bond definition 470 ServiceOffer = 12, // Marketplace service listing 471 } 472 ``` 473 474 ### Bonding Curve (Year 1) 475 476 Access to the network is bootstrapped via a square-root bonding curve that mints tokens in exchange for verifiable computation or genesis allocation. 477 478 - **Formula**: `Price = Slope * sqrt(Supply)` 479 - **Sunset**: Curve functionality is disabled after Year 1 or supply cap hit. 480 481 ### Marketplace 482 483 Service discovery is fully decentralized: 484 485 1. **Providers** publish `ServiceOffer` (price, endpoint, bond ID) to DHT. 486 2. **Consumers** query `OfferRegistry` to find providers matching criteria. 487 3. **Settlement** occurs via direct peer-to-peer payment channels (concept). 488 489 --- 490 491 ## Security Hardening 492 493 ### Completed Phases 494 495 | Phase | Feature | Description | 496 |-------|---------|-------------| 497 | 1 | Link-Layer PFS | X25519 ephemeral keys, identity binding | 498 | 2 | RPC Authentication | Token-based auth for daemon control | 499 | 3 | Encrypted Identity | Argon2id + AES-GCM for key storage | 500 | 4 | Protocol Versioning | Version negotiation in handshake | 501 502 ### Phase 1: Link-Layer PFS 503 504 - Generate ephemeral X25519 keypair per connection 505 - Exchange via Hello/HelloAck 506 - Derive session key with HKDF 507 - Bind to identity via sorted public key salt 508 509 ### Phase 2: RPC Authentication 510 511 ```rust 512 pub struct RpcAuth { 513 token: [u8; 32], // Random token, exchanged out-of-band 514 expires_at: SystemTime, 515 } 516 ``` 517 518 ### Phase 3: Encrypted Identity Storage 519 520 ```rust 521 pub fn encrypt_identity(key: &SigningKey, password: &str) -> Vec<u8> { 522 let salt = rand::random::<[u8; 32]>(); 523 let key = argon2id(password, &salt, M_COST, T_COST, P_COST); 524 let nonce = rand::random::<[u8; 12]>(); 525 let ciphertext = aes_gcm_encrypt(&key, &nonce, key.to_bytes()); 526 [salt, nonce, ciphertext].concat() 527 } 528 ``` 529 530 ### Phase 4: Protocol Versioning 531 532 - `Hello`/`HelloAck` include `version_major` and `version_minor` 533 - Semantic versioning: major breaks compatibility, minor adds features 534 - Negotiation: use minimum compatible version 535 536 --- 537 538 ## API Reference 539 540 ### AbzuClient (SDK) 541 542 ```rust 543 pub struct AbzuClient { 544 node: Arc<Node>, 545 config: ClientConfig, 546 } 547 548 impl AbzuClient { 549 /// Connect to a peer 550 pub async fn connect(&self, addr: SocketAddr) -> Result<PeerHandle>; 551 552 /// Send encrypted chat message 553 pub async fn send_chat(&self, to: [u8; 32], msg: &[u8]) -> Result<u64>; 554 555 /// Store content, returns CID 556 pub async fn store(&self, data: &[u8]) -> Result<[u8; 32]>; 557 558 /// Retrieve content by CID 559 pub async fn fetch(&self, cid: [u8; 32]) -> Result<Vec<u8>>; 560 561 /// Create a new trust circle 562 pub async fn create_circle(&self, name: &str, policy: TrustPolicy) -> Result<Circle>; 563 564 /// Send message to circle 565 pub async fn circle_send(&self, circle_id: [u8; 32], msg: &[u8]) -> Result<()>; 566 } 567 ``` 568 569 For detailed API documentation, see [SDK_API_REFERENCE.md](SDK_API_REFERENCE.md). 570 571 --- 572 573 ## Test Coverage 574 575 ### Test Categories 576 577 | Category | Count | Coverage | 578 |----------|-------|----------| 579 | Unit Tests | 150+ | Core logic, crypto, serialization | 580 | Integration Tests | 50+ | Multi-node scenarios, handshakes | 581 | Stress Tests | 10+ | Concurrent connections, throughput | 582 | Property Tests | 20+ | Fuzzing with proptest | 583 | Protocol Tests | 30+ | Version negotiation, frame handling | 584 585 ### Key Test Files 586 587 - `abzu-transport/tests/wire_fuzzing.rs` — Protocol fuzzing 588 - `abzu-core/tests/integration.rs` — Multi-node integration 589 - `abzu-transport/tests/fake_tls_tests.rs` — TLS masquerading 590 - `abzu-router/tests/routing_tests.rs` — Path building 591 592 ### Running Tests 593 594 ```bash 595 # All tests 596 cargo test --workspace 597 598 # With coverage 599 cargo llvm-cov --workspace 600 601 # Stress tests only 602 cargo test --package abzu-core --test stress 603 ``` 604 605 --- 606 607 ## Known Limitations 608 609 | Limitation | Impact | Mitigation | 610 |------------|--------|------------| 611 | `seen_messages` unbounded | Memory growth | TTL cleanup planned | 612 | STUN servers hardcoded | Configuration rigidity | Move to config file | 613 | Identity plaintext fallback | Security risk if encryption fails | Keychain integration | 614 | No DHT persistence | Loses routing on restart | Bootstrap from seeds | 615 | Single-threaded circles | Performance ceiling | Shard by circle ID | 616 617 --- 618 619 ## Roadmap 620 621 ### Completed (v0.4.0) 622 623 - [x] Phase 1: Link-Layer PFS + Identity Binding 624 - [x] Phase 2: Token-based RPC Authentication 625 - [x] Phase 3: Encrypted Identity at Rest 626 - [x] Phase 4: Protocol Version Negotiation 627 - [x] Token Economy: Core Types & Bonding Curve 628 - [x] Documentation consolidation 629 - [x] Phase 2: Token-based RPC Authentication 630 - [x] Phase 3: Encrypted Identity at Rest 631 - [x] Phase 4: Protocol Version Negotiation 632 - [x] Documentation consolidation 633 634 ### Planned (v0.5.0) 635 636 - [ ] Phase 5: Content Store GC (TTL-based expiration) 637 - [ ] Phase 6: Bootstrap Node Signatures (signed announcements) 638 - [ ] WebRTC transport (browser compatibility) 639 - [ ] Mobile SDK (iOS/Android via FFI) 640 641 ### Future 642 643 - [ ] Mixnet integration (Nym-style) 644 - [ ] Incentive layer (optional) 645 - [ ] Hardware security module support 646 - [ ] Post-quantum key exchange 647 648 --- 649 650 ## References 651 652 - [ARCHITECTURE.md](ARCHITECTURE.md) — High-level architecture 653 - [WIRE_PROTOCOL_API.md](WIRE_PROTOCOL_API.md) — Wire protocol details 654 - [SDK_API_REFERENCE.md](SDK_API_REFERENCE.md) — Client API 655 - [DHT_DESIGN_SPECIFICATION.md](DHT_DESIGN_SPECIFICATION.md) — DHT layer design 656 - [WHITE_PAPER_v0.4.md](WHITE_PAPER_v0.4.md) — Strategic vision 657 - [deep-dive/TECHNICAL_DEEP_DIVE.md](deep-dive/TECHNICAL_DEEP_DIVE.md) — Code-level analysis 658 659 --- 660 661 *This document is the canonical technical reference for Abzu. For navigation, see [INDEX.md](INDEX.md).*