zmq.md
1 # Block and Transaction Broadcasting with ZeroMQ 2 3 [ZeroMQ](https://zeromq.org/) is a lightweight wrapper around TCP 4 connections, inter-process communication, and shared-memory, 5 providing various message-oriented semantics such as publish/subscribe, 6 request/reply, and push/pull. 7 8 The Bitcoin Core daemon can be configured to act as a trusted "border 9 router", implementing the bitcoin wire protocol and relay, making 10 consensus decisions, maintaining the local blockchain database, 11 broadcasting locally generated transactions into the network, and 12 providing a queryable RPC interface to interact on a polled basis for 13 requesting blockchain related data. However, there exists only a 14 limited service to notify external software of events like the arrival 15 of new blocks or transactions. 16 17 The ZeroMQ facility implements a notification interface through a set 18 of specific notifiers. Currently there are notifiers that publish 19 blocks and transactions. This read-only facility requires only the 20 connection of a corresponding ZeroMQ subscriber port in receiving 21 software; it is not authenticated nor is there any two-way protocol 22 involvement. Therefore, subscribers should validate the received data 23 since it may be out of date, incomplete or even invalid. 24 25 ZeroMQ sockets are self-connecting and self-healing; that is, 26 connections made between two endpoints will be automatically restored 27 after an outage, and either end may be freely started or stopped in 28 any order. 29 30 Because ZeroMQ is message oriented, subscribers receive transactions 31 and blocks all-at-once and do not need to implement any sort of 32 buffering or reassembly. 33 34 ## Prerequisites 35 36 The ZeroMQ feature in Bitcoin Core requires the ZeroMQ API >= 4.0.0 37 [libzmq](https://github.com/zeromq/libzmq/releases). 38 For version information, see [dependencies.md](dependencies.md). 39 Typically, it is packaged by distributions as something like 40 *libzmq3-dev*. The C++ wrapper for ZeroMQ is *not* needed. 41 42 In order to run the example Python client scripts in the `contrib/zmq/` 43 directory, one must also install [PyZMQ](https://github.com/zeromq/pyzmq) 44 (generally with `pip install pyzmq`), though this is not necessary for daemon 45 operation. 46 47 ## Enabling 48 49 By default, the ZeroMQ feature is not automatically compiled. 50 To enable, use `-DWITH_ZMQ=ON` when configuring the build system: 51 52 $ cmake -B build -DWITH_ZMQ=ON 53 54 To actually enable operation, one must set the appropriate options on 55 the command line or in the configuration file. 56 57 ## Usage 58 59 Currently, the following notifications are supported: 60 61 -zmqpubhashtx=address 62 -zmqpubhashblock=address 63 -zmqpubrawblock=address 64 -zmqpubrawtx=address 65 -zmqpubsequence=address 66 67 The socket type is PUB and the address must be a valid ZeroMQ socket 68 address. The same address can be used in more than one notification. 69 The same notification can be specified more than once. 70 71 The option to set the PUB socket's outbound message high water mark 72 (SNDHWM) may be set individually for each notification: 73 74 -zmqpubhashtxhwm=n 75 -zmqpubhashblockhwm=n 76 -zmqpubrawblockhwm=n 77 -zmqpubrawtxhwm=n 78 -zmqpubsequencehwm=n 79 80 The high water mark value must be an integer greater than or equal to 0. 81 82 For instance: 83 84 $ bitcoind -zmqpubhashtx=tcp://127.0.0.1:28332 \ 85 -zmqpubhashtx=tcp://192.168.1.2:28332 \ 86 -zmqpubhashblock="tcp://[::1]:28333" \ 87 -zmqpubrawtx=unix:/tmp/bitcoind.tx.raw \ 88 -zmqpubhashtxhwm=10000 89 90 `bitcoin node` or `bitcoin gui` can also be substituted for `bitcoind`. 91 92 Notification types correspond to message topics (details in next section). For instance, 93 for the notification `-zmqpubhashtx` the topic is `hashtx`. These options can also be 94 provided in bitcoin.conf. 95 96 ### Message format 97 98 All ZMQ messages share the same structure with three parts: _topic_ string, 99 message _body_, and _message sequence number_: 100 101 | topic | body | message sequence number | 102 |-----------+------------------------------------------------------+--------------------------| 103 | rawtx | <serialized transaction> | <4-byte LE uint> | 104 | hashtx | <reversed 32-byte transaction hash> | <4-byte LE uint> | 105 | rawblock | <serialized block> | <4-byte LE uint> | 106 | hashblock | <reversed 32-byte block hash> | <4-byte LE uint> | 107 | sequence | <reversed 32-byte block hash>C | <4-byte LE uint> | 108 | sequence | <reversed 32-byte block hash>D | <4-byte LE uint> | 109 | sequence | <reversed 32-byte transaction hash>R<8-byte LE uint> | <4-byte LE uint> | 110 | sequence | <reversed 32-byte transaction hash>A<8-byte LE uint> | <4-byte LE uint> | 111 112 where: 113 114 - message sequence number represents message count to detect lost messages, distinct for each topic 115 - all transaction and block hashes are in _reversed byte order_ (i. e. with bytes 116 produced by hashing function reversed), the same format as the RPC interface and block 117 explorers use to display transaction and block hashes 118 119 #### rawtx 120 121 Notifies about all transactions, both when they are added to mempool or when a new block 122 arrives. This means a transaction could be published multiple times: first when it enters 123 mempool and then again in each block that includes it. The body part of the message is the 124 serialized transaction. 125 126 #### hashtx 127 128 Notifies about all transactions, both when they are added to mempool or when a new block 129 arrives. This means a transaction could be published multiple times: first when it enters 130 mempool and then again in each block that includes it. The body part of the message is the 131 32-byte transaction hash in reversed byte order. 132 133 #### rawblock 134 135 Notifies when the chain tip is updated. When assumeutxo is in use, this notification will 136 not be issued for historical blocks connected to the background validation chainstate. The 137 body part of the message is the serialized block. 138 139 #### hashblock 140 141 Notifies when the chain tip is updated. When assumeutxo is in use, this notification will 142 not be issued for historical blocks connected to the background validation chainstate. The 143 body part of the message is the 32-byte block hash in reversed byte order. 144 145 #### sequence 146 147 The 8-byte LE uints correspond to _mempool sequence number_ and the types of bodies are: 148 149 - `C` : block with this hash connected 150 - `D` : block with this hash disconnected 151 - `R` : transaction with this hash removed from mempool for non-block inclusion reason 152 - `A` : transaction with this hash added to mempool 153 154 ### Implementing ZMQ client 155 156 ZeroMQ endpoint specifiers for TCP (and others) are documented in the 157 [ZeroMQ API](https://libzmq.readthedocs.io/en/zeromq4-x/). 158 159 Client side, then, the ZeroMQ subscriber socket must have the 160 ZMQ_SUBSCRIBE option set to one or either of these prefixes (for 161 instance, just `hash`); without doing so will result in no messages 162 arriving. Please see [`contrib/zmq/zmq_sub.py`](/contrib/zmq/zmq_sub.py) for a working example. 163 164 The ZMQ_PUB socket's ZMQ_TCP_KEEPALIVE option is enabled. This means that 165 the underlying SO_KEEPALIVE option is enabled when using a TCP transport. 166 The effective TCP keepalive values are managed through the underlying 167 operating system configuration and must be configured prior to connection establishment. 168 169 For example, when running on GNU/Linux, one might use the following 170 to lower the keepalive setting to 10 minutes: 171 172 sudo sysctl -w net.ipv4.tcp_keepalive_time=600 173 174 Setting the keepalive values appropriately for your operating environment may 175 improve connectivity in situations where long-lived connections are silently 176 dropped by network middle boxes. 177 178 Also, the socket's ZMQ_IPV6 option is enabled to accept connections from IPv6 179 hosts as well. If needed, this option has to be set on the client side too. 180 181 ## Remarks 182 183 From the perspective of bitcoind, the ZeroMQ socket is write-only; PUB 184 sockets don't even have a read function. Thus, there is no state 185 introduced into bitcoind directly. Furthermore, no information is 186 broadcast that wasn't already received from the public P2P network. 187 188 No authentication or authorization is done on connecting clients; it 189 is assumed that the ZeroMQ port is exposed only to trusted entities, 190 using other means such as firewalling. 191 192 Note that for `*block` topics, when the block chain tip changes, 193 a reorganisation may occur and just the tip will be notified. 194 It is up to the subscriber to retrieve the chain from the last known 195 block to the new tip. Also note that no notification will occur if the tip 196 was in the active chain, as would be the case after calling the `invalidateblock` RPC. 197 In contrast, the `sequence` topic publishes all block connections and 198 disconnections. 199 200 There are several possibilities that ZMQ notification can get lost 201 during transmission depending on the communication type you are 202 using. Bitcoind appends an up-counting sequence number to each 203 notification which allows listeners to detect lost notifications. 204 205 The `sequence` topic refers specifically to the mempool sequence 206 number, which is also published along with all mempool events. This 207 is a different sequence value than in ZMQ itself in order to allow a total 208 ordering of mempool events to be constructed.