network-troubleshooting.md
1 # Network troubleshooting 2 3 If you're having network issues, refer to this page to debug various 4 issues. If you see inconsistencies in the docs: always trust 5 `${DARKFI_REPO}/bin/darkirc/darkirc_config.toml` or whichever respective 6 apps' repo config file. Documentation updates are a current WIP. 7 8 The default location for config files is `~/.config/darkfi`. 9 10 <u><b>Note</b></u>: throughout this page we generally assume you are using 11 `darkirc` since it's our main p2p app currently. If you're 12 using a different app such as `darkfid` or `taud`, the syntax remains 13 but the app name will change (for example, if using `taud`, 14 the config file `~/.config/darkfi/darkirc_config.toml` 15 would become `~/.config/darkfi/taud_config.toml`). 16 17 ## Common net problems 18 19 The most common problem in connecting to `darkirc` is the following: 20 21 ``` 22 [ERROR] [EVENTGRAPH] Sync: Could not find any DAG tips 23 [ERROR] Failed syncing DAG. Exiting. 24 Error: DagSyncFailed 25 ``` 26 27 This generally indicates that we were *unable to establish any P2P 28 connections*, and thus couldn't retrieve the message history required to 29 sync our messages locally within the time limit (DAG sync failed). 30 31 There are two main reasons why we would fail to establish a P2P connection: 32 33 1. Seed node is down or rejecting our connection 34 2. Our node does not have sufficient peers 35 36 If the seed node is down, you will see this message in the debug output: 37 38 ``` 39 ERROR] [P2P] Network reseed failed! 40 [WARN] [P2P] Unable to connect to seed [tcp+tls://lilith1.dark.fi:5262/]: IO error: connection refused 41 ``` 42 43 If it's a problem related to nodes, you will typically see a successful 44 seed connection like so: 45 46 ``` 47 [INFO] [P2P] Connected seed [tcp+tls://lilith1.dark.fi:5262/] 48 [INFO] [P2P] Disconnecting from seed [tcp+tls://lilith1.dark.fi:5262/] 49 ``` 50 51 Followed by multiple connection failed messages, like so: 52 53 ``` 54 [INFO] [P2P] Unable to connect outbound slot #5 [tcp+tls://example_peer:26661/]: IO error: connection refused 55 [INFO] [P2P] Unable to connect outbound slot #6 [tcp+tls://example_peer2:26661/]: IO error: host unreachable 56 ``` 57 58 ### Seed node is down 59 60 If you get an error like this: 61 62 ``` 63 [WARN] [P2P] Unable to connect to seed [tcp+tls://lilith1.dark.fi:5262/]: IO error: connection refused 64 ``` 65 66 This means you are failing to establish a connection to the seed node. 67 68 <u><b>Note</b></u>: the IO error might not always read `connection refused` 69 but could be some other error such as `host unreachable`. Please note 70 this IO error as it is useful debugging info. 71 72 Here's what to do next: 73 74 #### It's my first time connecting to the network 75 76 If it's your first time connecting to the network, you local node does 77 not have a record of other peers it can connect to in case the seed node 78 is down. Please do the following: 79 80 1. Take careful note of the `IO error` that is written after `Unable to 81 connect to seed`. 82 2. Refer to [Error reporting](#error-reporting) section below. 83 3. You can set a peer such as `tcp+tls://example_peer:26661` in your 84 config file. Ask in the telegram community channel for an active peer 85 (here we are using a fake peer called `example_peer`. Then open the 86 config file at `~/.config/darkfi/darkirc_config.toml` and modify the `peers` 87 field with the provided peer as follows: 88 89 ``` 90 peers = ["tcp+tls://example_peer:26661"] 91 ``` 92 93 #### It's not my first time connecting to the network 94 95 If it's not your first time connecting to the network, you should be 96 able to establish connections to peers even if the seed node is down. 97 98 This is possible via a list of hosts that your darkirc node keeps locally. 99 You can inspect the hostlist as follows: 100 101 ``` 102 cat ~/.local/share/darkfi/darkirc/hostlist.tsv 103 ``` 104 105 If the list is empty, open `~/.config/darkfi/darkirc_config` and ensure 106 that the `hostlist` field is set with a path of your choosing. 107 108 For example: 109 110 ``` 111 hostlist = "~/.local/share/darkfi/darkirc/hostlist.tsv" 112 ``` 113 114 <u><b>Note</b></u>: If you are editing a line that is commented out, don't forget 115 to uncomment the line. 116 117 Then follow the steps in the above section 118 [It's my first time connecting to the network](#its-my-first-time-connecting-to-the-network). 119 120 If the hostlist is not empty, retry the `darkirc` connection and carefully 121 note the connection errors that are happening from peers. See [Error reporting](#error-reporting) 122 section below to report errors. 123 It might be simply the case that there are not enough peers on the 124 network, or perhaps there is another issue we are not aware of. 125 126 You can also check the liveness of peers using the `ping` tool. 127 Refer to the [Ping tool](#ping-tool) section below for instructions. 128 129 ### Cannot establish peer connections 130 131 If you're able to connect to the seed but are failing to establish peer 132 connections, please retry the darkirc connection and carefully note the 133 connection errors that are happening from peers. See the 134 [Error reporting](#error-reporting) section to report errors. 135 136 ### Cannot establish Tor onion connections 137 138 You may get an error like this: 139 ``` 140 [WARN] darError reportingkfi::net::transport::tor: error: tor: Onion Service not found: Failed to obtain hidden service circuit to ????.onion: Unable to download hidden service descriptor 141 ``` 142 This happens when [Arti](https://gitlab.torproject.org/tpo/core/arti/-/blob/main/README.md) 143 gets corrupted due to internet downtime or other triggers. To fix this, 144 we'll delete the directory: 145 146 1. Stop `darkirc` 147 2. Stop `tor` daemon 148 3. Remove `arti` cache folder located at `~/.local/share/arti` 149 4. Start `tor` daemon 150 5. Start `darkirc` 151 152 ### I'm connected but my messages do not go through 153 154 If you see something in the logs like this: 155 156 ``` 157 [INFO] [P2P] Outbound slot #1 connected [tcp-tls://example_peer:25551/] 158 ``` 159 160 That means you are connected. You can verify that by writing `test` in 161 #random and seeing do you get a `test back` message. 162 163 If you do not get a `test back` message, that can mean either: 164 165 1. You need to wait for your DAG to sync (this can take several minutes, 166 especially over Tor or on days with high network activity). 167 168 2. You need to update your system clock. To sync the event graph, 169 darkirc requires that your system clock is correct. You can check your 170 system time by running `date`. The best way to ensure your clock does 171 not drift is to run some timekeeping daemon like `chrony` or `ntpd`. If 172 your clock is wrong, set this up and try to reconnect again. 173 174 ### DagSync spam 175 176 If you see a many rapid `EventReq` messages in the log, it is possible that there is 177 an incompatibility with your local `darkirc` database and the state of the network. 178 179 This can be resolved by deleting `~/.local/share/darkfi/darkirc_db/` 180 181 This is a known bug and we are working on a fix. 182 183 ## dnet 184 185 dnet is a simple tui to explore darkfi p2p network topology. You can use 186 dnet to gather more network information. dnet displays: 187 188 1. Active p2p nodes 189 2. Outgoing, incoming, manual and seed sessions 190 3. Each associated connection and recent messages. 191 192 To install and learn to use dnet, go [here](../learn/dchat/network-tools/using-dnet.md). 193 You can use dnet to view the network topology and see how your node 194 interacts within the network. dnet log information is created in 195 `${DARKFI_REPO}/bin/dnet/dnet.log`. 196 197 ## Ping tool 198 199 You can ping any node to make sure it's online by using the provided 200 `ping` tool located at `${DARKFI_REPO}/script/ping`. Select a peer from 201 your hostlist file. You can now use the `ping` tool by 202 running this command: 203 204 ``` 205 $ cargo run -- tcp+tls://example_peer:26661 206 ``` 207 208 If the peers are reachable, you'll receive a `Connected!` output. 209 210 ## Inbound 211 212 To see if your address is reachable to others in the network, you'll need 213 to use a separate device to `ping` your external address. 214 [You can generate an external address here](nodes/tor-guide.md#inbound-node-settings). 215 For example purposes, let's assume your external address is 216 `jamie3vkiwibfiwucd6vxijskbhpjdyajmzeor4mc4i7yopvpo4p7cyd.onion`. In 217 `${DARKFI_REPO}/script/ping` we can attempt to `ping` your external address 218 from a separate device. 219 220 ``` 221 $ cargo run -- tor://jamie3vkiwiskbhpjdyajmzeor4mc4i7yopvpo4p7cyd.onion 222 ``` 223 224 If your external address is reachable, you'll receive a `Connected!` prompt. 225 226 ## Check tor connection 227 228 You can verify if your local node is running over Tor. Execute this 229 command in `${DARKFI_REPO}/script`. You'll need to install pysocks 230 `pip install pysocks` prior to running `tor-test.py` the first time: 231 232 ``` 233 $ python tor-test.py 234 ``` 235 236 If your local node is running Tor, the response should be an IP address. 237 An error will return if Tor isn't running. 238 239 You can also verify if your node is running over Tor with 240 dnet. If you run `dnet` and you see onion addresses as 241 outbound connections, and localhost connections as inbound 242 connections, this means you're connected to Tor. 243 244 ## Helpful debug information 245 246 If you're looking to debug an issue, try these helpful tools. 247 248 ### Logs in debug mode 249 250 You can run any app in debug mode as follows: 251 252 ``` 253 $ ./darkirc -vv 254 ``` 255 256 Alternatively, modify the config file at `~/.config/darkfi/darkirc.toml` as follows: 257 258 ```toml 259 # Log to file. Off by default. 260 log = "/tmp/darkirc.log" 261 # Set log level. 1 is info (default), 2 is debug, 3 is trace 262 verbose = 2 263 ``` 264 265 ### Peer Discovery 266 267 When running in debug mode, you will see `[INFO]` messages that indicate 268 `PEER DISCOVERY`. This is healthy and expected behavior. 269 270 ``` 271 [INFO] net::outbound_session::peer_discovery(): [P2P] [PEER DISCOVERY] Asking peers for new peers to connect to... 272 ``` 273 274 ### Config file 275 276 Your config files are generated in your `~/.config/darkfi` directory. 277 You'll have to run each daemon once for the app to spawn a config file, 278 which you can review and edit. There is also helpful information within 279 the config files. 280 281 If experiencing connection issues review the configuration file for any mistakes. 282 Check for duplicate variables. 283 284 ### Node information script 285 286 If you're looking for information about your node, including inbound, 287 outbound, and seed connections, execute this command in ``${DARKFI_REPO}/script``: 288 289 ``` 290 $ python node_get-info.py 291 ``` 292 293 ### Hostlist issues 294 295 If you receive DAG sync issues, verify: 296 297 1. A hostlist is set in the config file of the respective app. 298 2. There are hosts in the hostlists (you should get hostlists from the 299 default seed on the first run). You can find the hostlist files within 300 the respective apps' repo. For example `darkirc`'s default hostlist location 301 is `~/.local/share/darkfi/darkirc/hostlist.tsv`. 302 303 ### Error reporting 304 305 If you're receiving errors and need to report them, report using `darkirc` first. If you 306 cannot connect, you can report these errors on the community telegram (t.me/darkfichat). 307 - Don't send screenshots. 308 - Use [pastebin](https://pastebin.com/) (or [termbin](https://termbin.com/) 309 or another paste service) for multi-line errors, or just copy-paste for a single line error.