understanding.rst
1 .. _understanding-main: 2 3 *********************** 4 Understanding Reticulum 5 *********************** 6 This chapter will briefly describe the overall purpose and operating principles of Reticulum. 7 It should give you an overview of how the stack works, and an understanding of how to 8 develop networked applications using Reticulum. 9 10 This chapter is not an exhaustive source of information on Reticulum, at least not yet. Currently, 11 the only complete repository, and final authority on how Reticulum actually functions, is the Python 12 reference implementation and API reference. That being said, this chapter is an essential resource in 13 understanding how Reticulum works from a high-level perspective, along with the general principles of 14 Reticulum, and how to apply them when creating your own networks or software. 15 16 After reading this chapter, you should be well-equipped to understand how a Reticulum network 17 operates, what it can achieve, and how you can use it yourself. This chapter also seeks to provide an overview of the 18 sentiments and the philosophy behind Reticulum, what problems it seeks to solve, and how it 19 approaches those solutions. 20 21 .. _understanding-motivation: 22 23 Motivation 24 ========== 25 26 The primary motivation for designing and implementing Reticulum has been the current lack of 27 reliable, functional and secure minimal-infrastructure modes of digital communication. It is my 28 belief that it is highly desirable to create a reliable and efficient way to set up long-range digital 29 communication networks that can securely allow exchange of information between people and 30 machines, with no central point of authority, control, censorship or barrier to entry. 31 32 Almost all of the various networking systems in use today share a common limitation: They 33 require large amounts of coordination and centralised trust and power to function. To join such networks, you need approval 34 of gatekeepers in control. This need for coordination and trust inevitably leads to an environment of 35 central control, where it's very easy for infrastructure operators or governments to control or alter 36 traffic, and censor or persecute unwanted actors. It also makes it completely impossible to freely deploy 37 and use networks at will, like one would use other common tools that enhance individual agency and freedom. 38 39 Reticulum aims to require as little coordination and trust as possible. It aims to make secure, 40 anonymous and permissionless networking and information exchange a tool that anyone can just pick up and use. 41 42 Since Reticulum is completely medium agnostic, it can be used to build networks on whatever is best 43 suited to the situation, or whatever you have available. In some cases, this might be packet radio 44 links over VHF frequencies, in other cases it might be a 2.4 GHz 45 network using off-the-shelf radios, or it might be using common LoRa development boards. 46 47 At the time of release of this document, the fastest and easiest setup for development and testing is using 48 LoRa radio modules with an open source firmware (see the section :ref:`Reference Setup<understanding-referencesystem>`), 49 connected to any kind of computer or mobile device that Reticulum can run on. 50 51 The ultimate aim of Reticulum is to allow anyone to be their own network operator, and to make it 52 cheap and easy to cover vast areas with a myriad of independent, interconnectable and autonomous networks. 53 Reticulum **is not** *one network*, it **is a tool** to build *thousands of networks*. Networks without 54 kill-switches, surveillance, censorship and control. Networks that can freely interoperate, associate and disassociate 55 with each other, and require no central oversight. Networks for human beings. *Networks for the people*. 56 57 .. _understanding-goals: 58 59 Goals 60 ===== 61 62 To be as widely usable and efficient to deploy as possible, the following goals have been used to 63 guide the design of Reticulum: 64 65 66 * **Fully useable as open source software stack** 67 Reticulum must be implemented with, and be able to run using only open source software. This is 68 critical to ensuring the availability, security and transparency of the system. 69 * **Hardware layer agnosticism** 70 Reticulum must be fully hardware agnostic, and shall be useable over a wide range of 71 physical networking layers, such as data radios, serial lines, modems, handheld transceivers, 72 wired Ethernet, WiFi, or anything else that can carry a digital data stream. Hardware made for 73 dedicated Reticulum use shall be as cheap as possible and use off-the-shelf components, so 74 it can be easily modified and replicated by anyone interested in doing so. 75 * **Very low bandwidth requirements** 76 Reticulum should be able to function reliably over links with a transmission capacity as low 77 as *5 bits per second*. 78 * **Encryption by default** 79 Reticulum must use strong encryption by default for all communication. 80 * **Initiator Anonymity** 81 It must be possible to communicate over a Reticulum network without revealing any identifying 82 information about oneself. 83 * **Unlicensed use** 84 Reticulum shall be functional over physical communication mediums that do not require any 85 form of license to use. Reticulum must be designed in a way, so it is usable over ISM radio 86 frequency bands, and can provide functional long distance links in such conditions, for example 87 by connecting a modem to a PMR or CB radio, or by using LoRa or WiFi modules. 88 * **Supplied software** 89 In addition to the core networking stack and API, that allows a developer to build 90 applications with Reticulum, a basic set of Reticulum-based communication tools must be 91 implemented and released along with Reticulum itself. These shall serve both as a 92 functional, basic communication suite, and as an example and learning resource to others wishing 93 to build applications with Reticulum. 94 * **Ease of use** 95 The reference implementation of Reticulum is written in Python, to make it easy to use 96 and understand. A programmer with only basic experience should be able to use 97 Reticulum to write networked applications. 98 * **Low cost** 99 It shall be as cheap as possible to deploy a communication system based on Reticulum. This 100 should be achieved by using cheap off-the-shelf hardware that potential users might already 101 own. The cost of setting up a functioning node should be less than $100 even if all parts 102 needs to be purchased. 103 104 .. _understanding-basicfunctionality: 105 106 Introduction & Basic Functionality 107 ================================== 108 109 Reticulum is a networking stack suited for high-latency, low-bandwidth links. Reticulum is at its 110 core a *message oriented* system. It is suited for both local point-to-point or point-to-multipoint 111 scenarios where all nodes are within range of each other, as well as scenarios where packets need 112 to be transported over multiple hops in a complex network to reach the recipient. 113 114 Reticulum does away with the idea of addresses and ports known from IP, TCP and UDP. Instead 115 Reticulum uses the singular concept of *destinations*. Any application using Reticulum as its 116 networking stack will need to create one or more destinations to receive data, and know the 117 destinations it needs to send data to. 118 119 All destinations in Reticulum are *represented* as a 16 byte hash. This hash is derived from truncating a full 120 SHA-256 hash of identifying characteristics of the destination. To users, the destination addresses 121 will be displayed as 16 hexadecimal bytes, like this example: ``<13425ec15b621c1d928589718000d814>``. 122 123 The truncation size of 16 bytes (128 bits) for destinations has been chosen as a reasonable trade-off 124 between address space 125 and packet overhead. The address space accommodated by this size can support many billions of 126 simultaneously active devices on the same network, while keeping packet overhead low, which is 127 essential on low-bandwidth networks. In the very unlikely case that this address space nears 128 congestion, a one-line code change can upgrade the Reticulum address space all the way up to 256 129 bits, ensuring the Reticulum address space could potentially support galactic-scale networks. 130 This is obviously complete and ridiculous over-allocation, and as such, the current 128 bits should 131 be sufficient, even far into the future. 132 133 By default Reticulum encrypts all data using elliptic curve cryptography and AES. Any packet sent to a 134 destination is encrypted with a per-packet derived key. Reticulum can also set up an encrypted 135 channel to a destination, called a *Link*. Both data sent over Links and single packets offer 136 *Initiator Anonymity*. Links additionally offer *Forward Secrecy* by default, employing an Elliptic Curve 137 Diffie Hellman key exchange on Curve25519 to derive per-link ephemeral keys. Asymmetric, link-less 138 packet communication can also provide forward secrecy, with automatic key ratcheting, by enabling 139 ratchets on a per-destination basis. The multi-hop transport, coordination, verification and reliability 140 layers are fully autonomous and also based on elliptic curve cryptography. 141 142 Reticulum also offers symmetric key encryption for group-oriented communications, as well as 143 unencrypted packets (for local broadcast purposes **only**). 144 145 Reticulum can connect to a variety of interfaces such as radio modems, data radios and serial ports, 146 and offers the possibility to easily tunnel Reticulum traffic over IP links such as the Internet or 147 private IP networks. 148 149 .. _understanding-destinations: 150 151 Destinations 152 ------------ 153 154 To receive and send data with the Reticulum stack, an application needs to create one or more 155 destinations. Reticulum uses three different basic destination types, and one special: 156 157 158 * **Single** 159 The *single* destination type is the most common type in Reticulum, and should be used for 160 most purposes. It is always identified by a unique public key. Any data sent to this 161 destination will be encrypted using ephemeral keys derived from an ECDH key exchange, and will 162 only be readable by the creator of the destination, who holds the corresponding private key. 163 * **Plain** 164 A *plain* destination type is unencrypted, and suited for traffic that should be broadcast to a 165 number of users, or should be readable by anyone. Traffic to a *plain* destination is not encrypted. 166 Generally, *plain* destinations can be used for broadcast information intended to be public. 167 Plain destinations are only reachable directly, and packets addressed to plain destinations are 168 never transported over multiple hops in the network. To be transportable over multiple hops in Reticulum, information 169 *must* be encrypted, since Reticulum uses the per-packet encryption to verify routing paths and 170 keep them alive. 171 * **Group** 172 The *group* special destination type, that defines a symmetrically encrypted virtual destination. 173 Data sent to this destination will be encrypted with a symmetric key, and will be readable by 174 anyone in possession of the key, but as with the *plain* destination type, packets to this type 175 of destination are not currently transported over multiple hops, although a planned upgrade 176 to Reticulum will allow globally reachable *group* destinations. 177 * **Link** 178 A *link* is a special destination type, that serves as an abstract channel to a *single* 179 destination, directly connected or over multiple hops. The *link* also offers reliability and 180 more efficient encryption, forward secrecy, initiator anonymity, and as such can be useful even 181 when a node is directly reachable. It also offers a more capable API and allows easily carrying 182 out requests and responses, large data transfers and more. 183 184 .. _understanding-destinationnaming: 185 186 Destination Naming 187 ^^^^^^^^^^^^^^^^^^ 188 189 Destinations are created and named in an easy to understand dotted notation of *aspects*, and 190 represented on the network as a hash of this value. The hash is a SHA-256 truncated to 128 bits. The 191 top level aspect should always be a unique identifier for the application using the destination. 192 The next levels of aspects can be defined in any way by the creator of the application. 193 194 Aspects can be as long and as plentiful as required, and a resulting long destination name will not 195 impact efficiency, as names are always represented as truncated SHA-256 hashes on the network. 196 197 As an example, a destination for a environmental monitoring application could be made up of the 198 application name, a device type and measurement type, like this: 199 200 .. code-block:: text 201 202 app name : environmentlogger 203 aspects : remotesensor, temperature 204 205 full name : environmentlogger.remotesensor.temperature 206 hash : 4faf1b2e0a077e6a9d92fa051f256038 207 208 For the *single* destination, Reticulum will automatically append the associated public key as a 209 destination aspect before hashing. This is done to ensure only the correct destination is reached, 210 since anyone can listen to any destination name. Appending the public key ensures that a given 211 packet is only directed at the destination that holds the corresponding private key to decrypt the 212 packet. 213 214 **Take note!** There is a very important concept to understand here: 215 216 * Anyone can use the destination name ``environmentlogger.remotesensor.temperature`` 217 218 * Each destination that does so will still have a unique destination hash, and thus be uniquely 219 addressable, because their public keys will differ. 220 221 In actual use of *single* destination naming, it is advisable not to use any uniquely identifying 222 features in aspect naming. Aspect names should be general terms describing what kind of destination 223 is represented. The uniquely identifying aspect is always achieved by appending the public key, 224 which expands the destination into a uniquely identifiable one. Reticulum does this automatically. 225 226 Any destination on a Reticulum network can be addressed and reached simply by knowing its 227 destination hash (and public key, but if the public key is not known, it can be requested from the 228 network simply by knowing the destination hash). The use of app names and aspects makes it easy to 229 structure Reticulum programs and makes it possible to filter what information and data your program 230 receives. 231 232 To recap, the different destination types should be used in the following situations: 233 234 * **Single** 235 When private communication between two endpoints is needed. Supports multiple hops. 236 * **Group** 237 When private communication between two or more endpoints is needed. Supports multiple hops 238 indirectly, but must first be established through a *single* destination. 239 * **Plain** 240 When plain-text communication is desirable, for example when broadcasting information, or for local discovery purposes. 241 242 To communicate with a *single* destination, you need to know its public key. Any method for 243 obtaining the public key is valid, but Reticulum includes a simple mechanism for making other 244 nodes aware of your destinations public key, called the *announce*. It is also possible to request 245 an unknown public key from the network, as all transport instances serve as a distributed ledger 246 of public keys. 247 248 Note that public key information can be shared and verified in other ways than using the 249 built-in *announce* functionality, and that it is therefore not required to use the *announce* and *path request* 250 functionality to obtain public keys. It is by far the easiest though, and should definitely be used 251 if there is not a very good reason for doing it differently. 252 253 .. _understanding-keyannouncements: 254 255 Public Key Announcements 256 ------------------------ 257 258 An *announce* will send a special packet over any relevant interfaces, containing all needed 259 information about the destination hash and public key, and can also contain some additional, 260 application specific data. The entire packet is signed by the sender to ensure authenticity. It is not 261 required to use the announce functionality, but in many cases it will be the simplest way to share 262 public keys on the network. The announce mechanism also serves to establish end-to-end connectivity 263 to the announced destination, as the announce propagates through the network. 264 265 As an example, an announce in a simple messenger application might contain the following information: 266 267 268 * The announcers destination hash 269 * The announcers public key 270 * Application specific data, in this case the users nickname and availability status 271 * A random blob, making each new announce unique 272 * An Ed25519 signature of the above information, verifying authenticity 273 274 With this information, any Reticulum node that receives it will be able to reconstruct an outgoing 275 destination to securely communicate with that destination. You might have noticed that there is one 276 piece of information lacking to reconstruct full knowledge of the announced destination, and that is 277 the aspect names of the destination. These are intentionally left out to save bandwidth, since they 278 will be implicit in almost all cases. The receiving application will already know them. If a destination 279 name is not entirely implicit, information can be included in the application specific data part that 280 will allow the receiver to infer the naming. 281 282 It is important to note that announces will be forwarded throughout the network according to a 283 certain pattern. This will be detailed in the section 284 :ref:`The Announce Mechanism in Detail<understanding-announce>`. 285 286 In Reticulum, destinations are allowed to move around the network at will. This is very different from 287 protocols such as IP, where an address is always expected to stay within the network segment it was assigned in. 288 This limitation does not exist in Reticulum, and any destination is *completely portable* over the entire topography 289 of the network, and *can even be moved to other Reticulum networks* than the one it was created in, and 290 still become reachable. To update its reachability, a destination simply needs to send an announce on any 291 networks it is part of. After a short while, it will be globally reachable in the network. 292 293 Seeing how *single* destinations are always tied to a private/public key pair leads us to the next topic. 294 295 .. _understanding-identities: 296 297 Identities 298 ---------- 299 300 In Reticulum, an *identity* does not necessarily represent a personal identity, but is an abstraction that 301 can represent any kind of *verifiable entity*. This could very well be a person, but it could also be the 302 control interface of a machine, a program, robot, computer, sensor or something else entirely. In 303 general, any kind of agent that can act, or be acted upon, or store or manipulate information, can be 304 represented as an identity. An *identity* can be used to create any number of destinations. 305 306 A *single* destination will always have an *identity* tied to it, but not *plain* or *group* 307 destinations. Destinations and identities share a multilateral connection. You can create a 308 destination, and if it is not connected to an identity upon creation, it will just create a new one to use 309 automatically. This may be desirable in some situations, but often you will probably want to create 310 the identity first, and then use it to create new destinations. 311 312 As an example, we could use an identity to represent the user of a messaging application. 313 Destinations can then be created by this identity to allow communication to reach the user. 314 In all cases it is of great importance to store the private keys associated with any 315 Reticulum Identity securely and privately, since obtaining access to the identity keys equals 316 obtaining access and controlling reachability to any destinations created by that identity. 317 318 .. _understanding-gettingfurther: 319 320 Getting Further 321 --------------- 322 323 The above functions and principles form the core of Reticulum, and would suffice to create 324 functional networked applications in local clusters, for example over radio links where all interested 325 nodes can directly hear each other. But to be truly useful, we need a way to direct traffic over multiple 326 hops in the network. 327 328 In the following sections, two concepts that allow this will be introduced, *paths* and *links*. 329 330 .. _understanding-transport: 331 332 Reticulum Transport 333 =================== 334 335 The methods of routing used in traditional networks are fundamentally incompatible with the physical medium 336 types and circumstances that Reticulum was designed to handle. These mechanisms mostly assume trust at the physical layer, 337 and often needs a lot more bandwidth than Reticulum can assume is available. Since Reticulum is designed to 338 survive running over open radio spectrum, no such trust can be assumed, and bandwidth is often very limited. 339 340 To overcome such challenges, Reticulum’s *Transport* system uses asymmetric elliptic curve cryptography to 341 implement the concept of *paths* that allow discovery of how to get information closer to a certain 342 destination. It is important to note that no single node in a Reticulum network knows the complete 343 path to a destination. Every Transport node participating in a Reticulum network will only 344 know the most direct way to get a packet one hop closer to it's destination. 345 346 347 .. _understanding-nodetypes: 348 349 Node Types 350 ---------- 351 352 Currently, Reticulum distinguishes between two types of network nodes. All nodes on a Reticulum network 353 are *Reticulum Instances*, and some are also *Transport Nodes*. If a system running Reticulum is fixed in 354 one place, and is intended to be kept available most of the time, it is a good contender to be a *Transport Node*. 355 356 Any Reticulum Instance can become a Transport Node by enabling it in the configuration. 357 This distinction is made by the user configuring the node, and is used to determine what nodes on the 358 network will help forward traffic, and what nodes rely on other nodes for wider connectivity. 359 360 If a node is an *Instance* it should be given the configuration directive ``enable_transport = No``, which 361 is the default setting. 362 363 If it is a *Transport Node*, it should be given the configuration directive ``enable_transport = Yes``. 364 365 366 .. _understanding-announce: 367 368 The Announce Mechanism in Detail 369 -------------------------------- 370 371 When an *announce* for a destination is transmitted by a Reticulum instance, it will be forwarded by 372 any transport node receiving it, but according to some specific rules: 373 374 375 * | If this exact announce has already been received before, ignore it. 376 377 * | If not, record into a table which Transport Node the announce was received from, and how many times in 378 total it has been retransmitted to get here. 379 380 * | If the announce has been retransmitted *m+1* times, it will not be forwarded any more. By default, *m* is 381 set to 128. 382 383 * | After a randomised delay, the announce will be retransmitted on all interfaces that have bandwidth 384 available for processing announces. By default, the maximum bandwidth allocation for processing 385 announces is set at 2%, but can be configured on a per-interface basis. 386 387 * | If any given interface does not have enough bandwidth available for retransmitting the announce, 388 the announce will be assigned a priority inversely proportional to its hop count, and be inserted 389 into a queue managed by the interface. 390 391 * | When the interface has bandwidth available for processing an announce, it will prioritise announces 392 for destinations that are closest in terms of hops, thus prioritising reachability and connectivity 393 of local nodes, even on slow networks that connect to wider and faster networks. 394 395 * | After the announce has been re-transmitted, and if no other nodes are heard retransmitting the announce 396 with a greater hop count than when it left this node, transmitting it will be retried *r* times. By default, 397 *r* is set to 1. 398 399 * | If a newer announce from the same destination arrives, while an identical one is already waiting 400 to be transmitted, the newest announce is discarded. If the newest announce contains different 401 application specific data, it will replace the old announce. 402 403 Once an announce has reached a transport node in the network, any other node in direct contact with that 404 transport node will be able to reach the destination the announce originated from, simply by sending a packet 405 addressed to that destination. Any transport node with knowledge of the announce will be able to direct the 406 packet towards the destination by looking up the most efficient next node to the destination. 407 408 According to these rules, an announce will propagate throughout the network in a predictable way, 409 and make the announced destination reachable in a short amount of time. Fast networks that have the 410 capacity to process many announces can reach full convergence very quickly, even when constantly adding 411 new destinations. Slower segments of such networks might take a bit longer to gain full knowledge about 412 the wide and fast networks they are connected to, but can still do so over time, while prioritising full 413 and quickly converging end-to-end connectivity for their local, slower segments. 414 415 .. tip:: 416 417 Even very slow networks, that simply don't have the capacity to ever reach *full* convergence 418 will generally still be able to reach **any other destination on any connected segments**, since 419 interconnecting transport nodes will prioritize announces into the slower segments that are 420 actually requested by nodes on these. 421 422 This means that slow, low-capacity or low-resource segments **don't** need to have full network 423 knowledge, since paths can always be recursively resolved from other segments that do have 424 knowledge about them. 425 426 In general, even extremely complex networks, that utilize the maximum 128 hops will converge to full 427 end-to-end connectivity in about one minute, given there is enough bandwidth available to process 428 the required amount of announces. 429 430 .. _understanding-paths: 431 432 Reaching the Destination 433 ------------------------ 434 435 In networks with changing topology and trustless connectivity, nodes need a way to establish 436 *verified connectivity* with each other. Since the underlying network mediums are assumed to be trustless, Reticulum 437 must provide a way to guarantee that the peer you are communicating with is actually who you 438 expect. Reticulum offers two ways to do this. 439 440 For exchanges of small amounts of information, Reticulum offers the *Packet* API, which works exactly like you would expect - on a per packet level. The following process is employed when sending a packet: 441 442 * | A packet is always created with an associated destination and some payload data. When the packet is sent 443 to a *single* destination type, Reticulum will automatically create an ephemeral encryption key, perform 444 an ECDH key exchange with the destination's public key (or ratchet key, if available), and encrypt the information. 445 446 * | It is important to note that this key exchange does not require any network traffic. The sender already 447 knows the public key of the destination from an earlier received announce, and can thus perform the ECDH 448 key exchange locally, before sending the packet. 449 450 * | The public part of the newly generated ephemeral key-pair is included with the encrypted token, and sent 451 along with the encrypted payload data in the packet. 452 453 * | When the destination receives the packet, it can itself perform an ECDH key exchange and decrypt the 454 packet. 455 456 * | A new ephemeral key is used for every packet sent in this way. 457 458 * | Once the packet has been received and decrypted by the addressed destination, that destination can opt 459 to *prove* its receipt of the packet. It does this by calculating the SHA-256 hash of the received packet, 460 and signing this hash with its Ed25519 signing key. Transport nodes in the network can then direct this 461 *proof* back to the packets origin, where the signature can be verified against the destination's known 462 public signing key. 463 464 * | In case the packet is addressed to a *group* destination type, the packet will be encrypted with the 465 pre-shared AES-256 key associated with the destination. In case the packet is addressed to a *plain* 466 destination type, the payload data will not be encrypted. Neither of these two destination types can offer 467 forward secrecy. In general, it is recommended to always use the *single* destination type, unless it is 468 strictly necessary to use one of the others. 469 470 471 For exchanges of larger amounts of data, or when longer sessions of bidirectional communication is desired, Reticulum offers the *Link* API. To establish a *link*, the following process is employed: 472 473 * | First, the node that wishes to establish a link will send out a *link request* packet, that 474 traverses the network and locates the desired destination. Along the way, the Transport Nodes that 475 forward the packet will take note of this *link request*, and mark it as pending. 476 477 * | Second, if the destination accepts the *link request* , it will send back a packet that proves the 478 authenticity of its identity (and the receipt of the link request) to the initiating node. All 479 nodes that initially forwarded the packet will also be able to verify this proof, and thus 480 accept the validity of the *link* throughout the network. The link is now marked as *established*. 481 482 * | When the validity of the *link* has been accepted by forwarding nodes, these nodes will 483 remember the *link* , and it can subsequently be used by referring to a hash representing it. 484 485 * | As a part of the *link request*, an Elliptic Curve Diffie-Hellman key exchange takes place, that sets up an 486 efficiently encrypted tunnel between the two nodes. As such, this mode of communication is preferred, 487 even for situations when nodes can directly communicate, when the amount of data to be exchanged numbers 488 in the tens of packets, or whenever the use of the more advanced API functions is desired. 489 490 * | When a *link* has been set up, it automatically provides message receipt functionality, through 491 the same *proof* mechanism discussed before, so the sending node can obtain verified confirmation 492 that the information reached the intended recipient. 493 494 * | Once the *link* has been set up, the initiator can remain anonymous, or choose to authenticate towards 495 the destination using a Reticulum Identity. This authentication is happening inside the encrypted 496 link, and is only revealed to the verified destination, and no intermediaries. 497 498 In a moment, we will discuss the details of how this methodology is 499 implemented, but let’s first recap what purposes this methodology serves. We 500 first ensure that the node answering our request is actually the one we want to 501 communicate with, and not a malicious actor pretending to be so. At the same 502 time we establish an efficient encrypted channel. The setup of this is 503 relatively cheap in terms of bandwidth, so it can be used just for a short 504 exchange, and then recreated as needed, which will also rotate encryption keys. 505 The link can also be kept alive for longer periods of time, if this is more 506 suitable to the application. The procedure also inserts the *link id* , a hash 507 calculated from the link request packet, into the memory of forwarding nodes, 508 which means that the communicating nodes can thereafter reach each other simply 509 by referring to this *link id*. 510 511 The combined bandwidth cost of setting up a link is 3 packets totalling 297 bytes (more info in the 512 :ref:`Binary Packet Format<understanding-packetformat>` section). The amount of bandwidth used on keeping 513 a link open is practically negligible, at 0.45 bits per second. Even on a slow 1200 bits per second packet 514 radio channel, 100 concurrent links will still leave 96% channel capacity for actual data. 515 516 517 Link Establishment in Detail 518 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 519 520 After exploring the basics of the announce mechanism, finding a path through the network, and an overview 521 of the link establishment procedure, this section will go into greater detail about the Reticulum link 522 establishment process. 523 524 The *link* in Reticulum terminology should not be viewed as a direct node-to-node link on the 525 physical layer, but as an abstract channel, that can be open for any amount of time, and can span 526 an arbitrary number of hops, where information will be exchanged between two nodes. 527 528 529 * | When a node in the network wants to establish verified connectivity with another node, it 530 will randomly generate a new X25519 private/public key pair. It then creates a *link request* 531 packet, and broadcast it. 532 | 533 | *It should be noted that the X25519 public/private keypair mentioned above is two separate keypairs: 534 An encryption key pair, used for derivation of a shared symmetric key, and a signing key pair, used 535 for signing and verifying messages on the link. They are sent together over the wire, and can be 536 considered as single public key for simplicity in this explanation.* 537 538 * | The *link request* is addressed to the destination hash of the desired destination, and 539 contains the following data: The newly generated X25519 public key *LKi*. 540 541 * | The broadcasted packet will be directed through the network according to the rules laid out 542 previously. 543 544 * | Any node that forwards the link request will store a *link id* in it’s *link table* , along with the 545 amount of hops the packet had taken when received. The link id is a hash of the entire link 546 request packet. If the link request packet is not *proven* by the addressed destination within some 547 set amount of time, the entry will be dropped from the *link table* again. 548 549 * | When the destination receives the link request packet, it will decide whether to accept the request. 550 If it is accepted, the destination will also generate a new X25519 private/public key pair, and 551 perform a Diffie Hellman Key Exchange, deriving a new symmetric key that will be used to encrypt the 552 channel, once it has been established. 553 554 * | A *link proof* packet is now constructed and transmitted over the network. This packet is 555 addressed to the *link id* of the *link*. It contains the following data: The newly generated X25519 556 public key *LKr* and an Ed25519 signature of the *link id* and *LKr* made by the *original signing key* of 557 the addressed destination. 558 559 * | By verifying this *link proof* packet, all nodes that originally transported the *link request* 560 packet to the destination from the originator can now verify that the intended destination received 561 the request and accepted it, and that the path they chose for forwarding the request was valid. 562 In successfully carrying out this verification, the transporting nodes marks the link as active. 563 An abstract bi-directional communication channel has now been established along a path in the network. 564 Packets can now be exchanged bi-directionally from either end of the link simply by adressing the 565 packets to the *link id* of the link. 566 567 * | When the source receives the *proof* , it will know unequivocally that a verified path has been 568 established to the destination. It can now also use the X25519 public key contained in the 569 *link proof* to perform it's own Diffie Hellman Key Exchange and derive the symmetric key 570 that is used to encrypt the channel. Information can now be exchanged reliably and securely. 571 572 .. note:: 573 574 It’s important to note that this methodology ensures that the source of the request does not need to 575 reveal any identifying information about itself. **The link initiator remains completely anonymous**. 576 577 When using *links*, Reticulum will automatically verify all data sent over the link, and can also 578 automate retransmissions if *Resources* are used. 579 580 .. _understanding-resources: 581 582 Resources 583 --------- 584 585 For exchanging small amounts of data over a Reticulum network, the :ref:`Packet<api-packet>` interface 586 is sufficient, but for exchanging data that would require many packets, an efficient way to coordinate 587 the transfer is needed. 588 589 This is the purpose of the Reticulum :ref:`Resource<api-resource>`. A *Resource* can automatically 590 handle the reliable transfer of an arbitrary amount of data over an established :ref:`Link<api-link>`. 591 Resources can auto-compress data, will handle breaking the data into individual packets, sequencing 592 the transfer, integrity verification and reassembling the data on the other end. 593 594 :ref:`Resources<api-resource>` are programmatically very simple to use, and only requires a few lines 595 of codes to reliably transfer any amount of data. They can be used to transfer data stored in memory, 596 or stream data directly from files. 597 598 .. _understanding-network_identities: 599 600 Network Identities 601 ================== 602 603 In Reticulum, every peer and application utilizes a cryptographic **Identity** to verify authenticity and establish encrypted channels. While standard identities are typically used to represent a single user, device, or service, Reticulum introduces the concept of a **Network Identity** to represent a logical group of nodes or an entire community infrastructure. 604 605 A Network Identity is, at its core, a standard Reticulum Identity keyset. However, its purpose and usage differ from a personal identity. Instead of identifying a single entity, a Network Identity acts as a shared credential that federates multiple independent Transport Instances under a single, verifiable administrative domain. 606 607 608 Conceptual Overview 609 ------------------- 610 611 You can think of a standard Reticulum Identity as a self-sovereign, privately created passport for a single person. A Network Identity, conversely, is akin to a cryptographic flag, or a charter that flies over a fleet of ships. It signifies that while the ships may operate independently and be physically distant, they belong to the same organization, follow the same protocols, and are expected to act in concert. 612 613 When you configure a Network Identity on one or more of your nodes, you are effectively declaring that these nodes constitute a specific "network" within a broader Reticulum mesh. This allows other peers to recognize interfaces not just as "a node named Alice", but as "a gateway belonging to The Eastern Ret Of Freedom". 614 615 616 Current Usage 617 ------------- 618 619 At present, the primary function of a Network Identity is within the :ref:`Interface Discovery<using-interface_discovery>` system. 620 621 When a Transport Instance broadcasts a discovery announce for an interface, it can optionally sign that announce with a Network Identity, instead of just its local transport identity. Remote peers receiving the announce can then verify the signature. This provides functionality for two important distinctions: 622 623 1. **Authenticity:** It proves that the interface was published by an operator who possesses the private key for that Network Identity. 624 2. **Trust Boundaries:** It allows users to configure their systems to only accept and connect to interfaces that belong to specific Network Identities, effectively creating "whitelisted" zones of trusted infrastructure. 625 626 .. note:: 627 If you enable encryption on your discovery announces, the Network Identity is used as the shared secret. Only peers who have been explicitly provided with the Network Identity's full keyset (and have it configured locally) will be able to decrypt and utilize the connection details. 628 629 This functionality will be expanded in the future, so that peers with delegated keys can be allowed to decrypt discovery announces without holding the root network key. Currently, the functionality is sufficient for sharing interface information privately where you control all nodes that must decrypt the discovered interfaces. 630 631 632 Future Implications 633 ------------------- 634 635 While the current implementation focuses on interface discovery, the concept of Network Identities serves as the foundational building block for future Reticulum features designed to support large-scale, organic mesh formation. 636 637 As the ecosystem evolves, Network Identities will facilitate: 638 639 * **Distributed Name Resolution:** A system where networks can publish name-to-identity mappings, allowing human-readable names to resolve without centralized servers. 640 * **Service Publishing:** Networks will be able to announce specific capabilities, services, or information endpoints available publicly or to their members. 641 * **Inter-Network Federation:** Trust relationships between different networks, allowing for seamless but managed flow of traffic and information across distinct administrative boundaries. 642 * **Distributed Blackhole Management:** A reputation-based system for blackhole list distribution, where trusted Network Identities can sign and publish lists of blackholed identities. This allows communities to collaboratively enforce security standards and filter spam or malicious identities across the parts of the wider mesh that they are responsible for. 643 644 By adopting the use of Network Identities now, you are preparing your infrastructure to be compatible with this future functionality. 645 646 647 Creating and Using a Network Identity 648 ------------------------------------- 649 650 Since a Network Identity is simply a standard Reticulum Identity, you create one using the built-in tools. 651 652 1. **Generate the Identity:** 653 Use the ``rnid`` utility to generate a new identity file that will serve as your Network Identity. 654 655 .. code:: sh 656 657 $ rnid -g ~/.reticulum/storage/identities/my_network 658 659 2. **Distribute the Public Key:** 660 The public key must be distributed to any Transport Instance that needs to verify your network's announces and discovery information. By default, if your node is set up to use a network identity, this happens automatically (using the standard announce mechanism). 661 662 3. **Configure Instances:** 663 In the ``[reticulum]`` section of the configuration file on every node within your network, point the ``network_identity`` option to the file you created. 664 665 .. code:: ini 666 667 [reticulum] 668 ... 669 network_identity = ~/.reticulum/storage/identities/my_network 670 ... 671 672 Once configured, your instances will automatically utilize this identity for signing discovery announces (and potentially decrypting network-private information), presenting a unified front to the wider network. 673 674 .. _understanding-referencesystem: 675 676 Reference Setup 677 ====================== 678 679 This section will detail a recommended *Reference Setup* for Reticulum. It is important to 680 note that Reticulum is designed to be usable on more or less any computing device, and over more 681 or less any medium that allows you to send and receive data, which satisfies some very low 682 minimum requirements. 683 684 The communication channel must support at least half-duplex operation, and provide an average 685 throughput of 5 bits per second or greater, and supports a physical layer MTU of 500 bytes. The 686 Reticulum stack should be able to run on more or less any hardware that can provide a Python 3.x 687 runtime environment. 688 689 That being said, this reference setup has been outlined to provide a common platform for anyone 690 who wants to help in the development of Reticulum, and for everyone who wants to know a 691 recommended setup to get started experimenting. A reference system consists of three parts: 692 693 * **An Interface Device** 694 Which provides access to the physical medium whereupon the communication 695 takes place, for example a radio with an integrated modem. A setup with a separate modem 696 connected to a radio would also be an interface device. 697 * **A Host Device** 698 Some sort of computing device that can run the necessary software, communicate with the 699 interface device, and provide user interaction. 700 * **A Software Stack** 701 The software implementing the Reticulum protocol and applications using it. 702 703 The reference setup can be considered a relatively stable platform to develop on, and also to start 704 building networks or applications on. While details of the implementation might change at the current stage of 705 development, it is the goal to maintain hardware compatibility for as long as entirely possible, and 706 the current reference setup has been determined to provide a functional platform for many years 707 into the future. The current Reference System Setup is as follows: 708 709 710 * **Interface Device** 711 A data radio consisting of a LoRa radio module, and a microcontroller with open source 712 firmware, that can connect to host devices via USB. It operates in either the 430, 868 or 900 713 MHz frequency bands. More details can be found on the `RNode Page <https://github.com/markqvist/rnode_firmware>`_. 714 * **Host Device** 715 Any computer device running Linux and Python. A Raspberry Pi with a Debian based OS is 716 a good place to start, but anything can be used. 717 * **Software Stack** 718 The most recently released Python Implementation of Reticulum, running on a Linux-based 719 operating system. 720 721 .. note:: 722 723 To avoid confusion, it is very important to note, that the reference interface device **does not** 724 use the LoRaWAN standard, but uses a custom MAC layer on top of the plain LoRa modulation! As such, you will 725 need a plain LoRa radio module connected to a controller with the correct firmware. Full details on how to 726 get or make such a device is available on the `RNode Page <https://github.com/markqvist/rnode_firmware>`_. 727 728 With the current reference setup, it should be possible to get on a Reticulum network for around 100$ 729 even if you have none of the hardware already, and need to purchase everything. 730 731 This reference setup is of course just a recommendation for getting started easily, and you should 732 tailor it to your own specific needs, or whatever hardware you have available. 733 734 .. _understanding-protocolspecifics: 735 736 Protocol Specifics 737 ================== 738 739 This chapter will detail protocol specific information that is essential to the implementation of 740 Reticulum, but non-critical in understanding how the protocol works on a general level. It should be 741 treated more as a reference than as essential reading. 742 743 744 Packet Prioritisation 745 --------------------- 746 747 Currently, Reticulum is completely priority-agnostic regarding *general* traffic. All traffic is handled 748 on a first-come, first-serve basis. Announce re-transmission and other maintenance traffic is handled 749 according to the re-transmission times and priorities described earlier in this chapter. 750 751 752 Interface Access Codes 753 ---------------------- 754 755 Reticulum can create named virtual networks, and networks that are only accessible by knowing a preshared 756 passphrase. The configuration of this is detailed in the :ref:`Common Interface Options<interfaces-options>` 757 section. To implement this feature, Reticulum uses the concept of Interface Access Codes, that are calculated 758 and verified per-packet. 759 760 An interface with a named virtual network or passphrase authentication enabled will derive a shared Ed25519 761 signing identity, and for every outbound packet generate a signature of the entire packet. This signature is 762 then inserted into the packet as an Interface Access Code before transmission. Depending on the speed and 763 capabilities of the interface, the IFAC can be the full 512-bit Ed25519 signature, or a truncated version. 764 Configured IFAC length can be inspected for all interfaces with the ``rnstatus`` utility. 765 766 Upon receipt, the interface will check that the signature matches the expected value, and drop the packet if it 767 does not. This ensures that only packets sent with the correct naming and/or passphrase parameters are allowed to 768 pass onto the network. 769 770 771 .. _understanding-packetformat: 772 773 Wire Format 774 ----------- 775 776 .. code-block:: text 777 778 == Reticulum Wire Format ====== 779 780 A Reticulum packet is composed of the following fields: 781 782 [HEADER 2 bytes] [ADDRESSES 16/32 bytes] [CONTEXT 1 byte] [DATA 0-465 bytes] 783 784 * The HEADER field is 2 bytes long. 785 * Byte 1: [IFAC Flag], [Header Type], [Context Flag], [Propagation Type], 786 [Destination Type] and [Packet Type] 787 * Byte 2: Number of hops 788 789 * Interface Access Code field if the IFAC flag was set. 790 * The length of the Interface Access Code can vary from 791 1 to 64 bytes according to physical interface 792 capabilities and configuration. 793 794 * The ADDRESSES field contains either 1 or 2 addresses. 795 * Each address is 16 bytes long. 796 * The Header Type flag in the HEADER field determines 797 whether the ADDRESSES field contains 1 or 2 addresses. 798 * Addresses are SHA-256 hashes truncated to 16 bytes. 799 800 * The CONTEXT field is 1 byte. 801 * It is used by Reticulum to determine packet context. 802 803 * The DATA field is between 0 and 465 bytes. 804 * It contains the packets data payload. 805 806 IFAC Flag 807 ----------------- 808 open 0 Packet for publically accessible interface 809 authenticated 1 Interface authentication is included in packet 810 811 812 Header Types 813 ----------------- 814 type 1 0 Two byte header, one 16 byte address field 815 type 2 1 Two byte header, two 16 byte address fields 816 817 818 Context Flag 819 ----------------- 820 unset 0 The context flag is used for various types 821 set 1 of signalling, depending on packet context 822 823 824 Propagation Types 825 ----------------- 826 broadcast 0 827 transport 1 828 829 830 Destination Types 831 ----------------- 832 single 00 833 group 01 834 plain 10 835 link 11 836 837 838 Packet Types 839 ----------------- 840 data 00 841 announce 01 842 link request 10 843 proof 11 844 845 846 +- Packet Example -+ 847 848 HEADER FIELD DESTINATION FIELDS CONTEXT FIELD DATA FIELD 849 _______|_______ ________________|________________ ________|______ __|_ 850 | | | | | | | | 851 01010000 00000100 [HASH1, 16 bytes] [HASH2, 16 bytes] [CONTEXT, 1 byte] [DATA] 852 || | | | | 853 || | | | +-- Hops = 4 854 || | | +------- Packet Type = DATA 855 || | +--------- Destination Type = SINGLE 856 || +----------- Propagation Type = TRANSPORT 857 |+------------- Header Type = HEADER_2 (two byte header, two address fields) 858 +-------------- Access Codes = DISABLED 859 860 861 +- Packet Example -+ 862 863 HEADER FIELD DESTINATION FIELD CONTEXT FIELD DATA FIELD 864 _______|_______ _______|_______ ________|______ __|_ 865 | | | | | | | | 866 00000000 00000111 [HASH1, 16 bytes] [CONTEXT, 1 byte] [DATA] 867 || | | | | 868 || | | | +-- Hops = 7 869 || | | +------- Packet Type = DATA 870 || | +--------- Destination Type = SINGLE 871 || +----------- Propagation Type = BROADCAST 872 |+------------- Header Type = HEADER_1 (two byte header, one address field) 873 +-------------- Access Codes = DISABLED 874 875 876 +- Packet Example -+ 877 878 HEADER FIELD IFAC FIELD DESTINATION FIELD CONTEXT FIELD DATA FIELD 879 _______|_______ ______|______ _______|_______ ________|______ __|_ 880 | | | | | | | | | | 881 10000000 00000111 [IFAC, N bytes] [HASH1, 16 bytes] [CONTEXT, 1 byte] [DATA] 882 || | | | | 883 || | | | +-- Hops = 7 884 || | | +------- Packet Type = DATA 885 || | +--------- Destination Type = SINGLE 886 || +----------- Propagation Type = BROADCAST 887 |+------------- Header Type = HEADER_1 (two byte header, one address field) 888 +-------------- Access Codes = ENABLED 889 890 891 Size examples of different packet types 892 --------------------------------------- 893 894 The following table lists example sizes of various 895 packet types. The size listed are the complete on- 896 wire size counting all fields including headers, 897 but excluding any interface access codes. 898 899 - Path Request : 51 bytes 900 - Announce : 167 bytes 901 - Link Request : 83 bytes 902 - Link Proof : 115 bytes 903 - Link RTT packet : 99 bytes 904 - Link keepalive : 20 bytes 905 906 907 .. _understanding-announcepropagation: 908 909 Announce Propagation Rules 910 -------------------------- 911 912 The following table illustrates the rules for automatically propagating announces 913 from one interface type to another, for all possible combinations. For the purpose 914 of announce propagation, the *Full* and *Gateway* modes are identical. 915 916 .. image:: graphics/if_mode_graph_b.png 917 918 See the :ref:`Interface Modes<interfaces-modes>` section for a conceptual overview 919 of the different interface modes, and how they are configured. 920 921 .. 922 (.. code-block:: text) 923 Full ────── ✓ ──┐ ┌── ✓ ── Full 924 AP ──────── ✓ ──┼───> Full >───┼── ✕ ── AP 925 Boundary ── ✓ ──┤ ├── ✓ ── Boundary 926 Roaming ─── ✓ ──┘ └── ✓ ── Roaming 927 928 Full ────── ✕ ──┐ ┌── ✓ ── Full 929 AP ──────── ✕ ──┼────> AP >────┼── ✕ ── AP 930 Boundary ── ✕ ──┤ ├── ✓ ── Boundary 931 Roaming ─── ✕ ──┘ └── ✓ ── Roaming 932 933 Full ────── ✓ ──┐ ┌── ✓ ── Full 934 AP ──────── ✓ ──┼─> Roaming >──┼── ✕ ── AP 935 Boundary ── ✕ ──┤ ├── ✕ ── Boundary 936 Roaming ─── ✕ ──┘ └── ✕ ── Roaming 937 938 Full ────── ✓ ──┐ ┌── ✓ ── Full 939 AP ──────── ✓ ──┼─> Boundary >─┼── ✕ ── AP 940 Boundary ── ✓ ──┤ ├── ✓ ── Boundary 941 Roaming ─── ✕ ──┘ └── ✕ ── Roaming 942 943 944 .. _understanding-primitives: 945 946 Cryptographic Primitives 947 ------------------------ 948 949 Reticulum uses a simple suite of efficient, strong and well-tested cryptographic 950 primitives, with widely available implementations that can be used both on 951 general-purpose CPUs and on microcontrollers. 952 953 One of the primary considerations for choosing this particular set of primitives is 954 that they can be implemented *safely* with relatively few pitfalls, on practically 955 all current computing platforms. 956 957 The primitives listed here **are authoritative**. Anything claiming to be Reticulum, 958 but not using these exact primitives **is not** Reticulum, and possibly an 959 intentionally compromised or weakened clone. The utilised primitives are: 960 961 * Ed25519 for signatures 962 963 * X25519 for ECDH key exchanges 964 965 * HKDF for key derivation 966 967 * Encrypted tokens are based on the Fernet spec 968 969 * Ephemeral keys derived from an ECDH key exchange on Curve25519 970 971 * AES-256 in CBC mode with PKCS7 padding 972 973 * HMAC using SHA256 for message authentication 974 975 * IVs must be generated through ``os.urandom()`` or better 976 977 * No Fernet version and timestamp metadata fields 978 979 * SHA-256 980 981 * SHA-512 982 983 In the default installation configuration, the ``X25519``, ``Ed25519`` and ``AES-256-CBC`` 984 primitives are provided by `OpenSSL <https://www.openssl.org/>`_ (via the `PyCA/cryptography <https://github.com/pyca/cryptography>`_ 985 package). The hashing functions ``SHA-256`` and ``SHA-512`` are provided by the standard 986 Python `hashlib <https://docs.python.org/3/library/hashlib.html>`_. The ``HKDF``, ``HMAC``, 987 ``Token`` primitives, and the ``PKCS7`` padding function are always provided by the 988 following internal implementations: 989 990 - ``RNS/Cryptography/HKDF.py`` 991 - ``RNS/Cryptography/HMAC.py`` 992 - ``RNS/Cryptography/Token.py`` 993 - ``RNS/Cryptography/PKCS7.py`` 994 995 996 Reticulum also includes a complete implementation of all necessary primitives in pure Python. 997 If OpenSSL & PyCA are not available on the system when Reticulum is started, Reticulum will 998 instead use the internal pure-python primitives. A trivial consequence of this is performance, 999 with the OpenSSL backend being *much* faster. The most important consequence however, is the 1000 potential loss of security by using primitives that has not seen the same amount of scrutiny, 1001 testing and review as those from OpenSSL. 1002 1003 Using the normal RNS installation procedures, it is not possible to install Reticulum on a 1004 system without the required OpenSSL primitives being available, and if they are not, they will 1005 be resolved and installed as a dependency. It is only possible to use the pure-python primitives 1006 by manually specifying this, for example by using the ``rnspure`` package. 1007 1008 .. warning:: 1009 If you want to use the internal pure-python primitives, it is **highly advisable** that you 1010 have a good understanding of the risks that this pose, and make an informed decision on whether 1011 those risks are acceptable to you.