OLD - Implementing Codex BitTorrent extension.md
1 In supporting BitTorrent on Codex network, it is important to clarify the pre-conditions: what do we expect to have as an input, and when will be the output. 2 3 BitTorrent itself, can have three types of inputs: 4 5 - a `.torrent` manifest file - a b-encoded [[BitTorrent metadata files]] - different formats for torrent version one and version 2 6 - a magnet link - introduced in [[BEP9 - Extension for Peers to Send Metadata Files]] to support trackerless torrent and using DHT for peer discovery 7 8 In both cases there are differences between version 1 and version 2 of metadata files (see [[BitTorrent metadata files]] for details) and version 1 and version 2 of [[Magnet Links|magnet links]]. 9 10 A torrent file, provides a complete description of the torrent, and can be used to compute the corresponding `info` hash. 11 12 Thus, while uploading (or seeding) BitTorrent content to the Codex network, the input is the content itself, while the output will be a (hybrid) magnet link. 13 14 To retrieve previously seeded content, the input can be a torrent file, a magnet link, or directly an info hash (either v1 or v2, tagged or untagged). 15 16 This is illustrated on the following picture: 17 18 ![[BitTorrent-Upload-Download.svg]] 19 20 Thus, from the implementation perspective, the actual input to the Codex network while retrieving previously uploaded content is its `info` hash. 21 22 ### Uploading BitTorrent Content to Codex 23 24 For the time being we only support version 1 and only a single file content (supporting directories and version 2 is work in progress). As a side not, limiting the description to this much simplified version will help to emphasize the important implementation challenges without being distracted with technicalities related to handling multiple file and folders. 25 26 Thus, let's assume we have a single input file: `data40k.bin`. It is a binary file of size `40KiB` (`40960` Bytes). We will be using `16KiB` (`16384` Bytes) block size, and commonly used for such small content `piece length` of `256KiB` (`262144` Bytes). 27 28 Let's go step by step through the code base to understand the upload process and the related challenges. 29 30 First, the upload API: 31 32 ``` 33 /api/codex/v1/torrent 34 ``` 35 36 To upload the content we can use the following `POST` request: 37 38 ```bash 39 curl -X POST \ 40 http://localhost:8001/api/codex/v1/torrent \ 41 -H 'Content-Type: application/octet-stream' \ 42 -H 'Content-Disposition: filename="data40k.bin"' \ 43 -w '\n' \ 44 -T data40k.bin 45 ``` 46 47 We use `Content Disposition` header to indicate the name we want to use for the uploaded content. 48 49 This will land to the API handler in `codex/rest/api.nim` : 50 51 ```nim 52 router.rawApi(MethodPost, "/api/codex/v1/torrent") do() -> RestApiResponse: 53 ## Upload a file in a streaming manner 54 ## 55 ``` 56 57 It will call `node.storeTorrent` to effectively upload the content and get the resulting `info` (multi) hash back: 58 59 ```nim 60 without infoHash =? ( 61 await node.storeTorrent( 62 AsyncStreamWrapper.new(reader = AsyncStreamReader(reader)), 63 filename = filename, 64 mimetype = mimetype, 65 ) 66 ), error: 67 error "Error uploading file", exc = error.msg 68 return RestApiResponse.error(Http500, error.msg) 69 ``` 70 71 This brings us to `node.storeTorrent` in `codex/node.nim: 72 73 ```nim 74 proc storeTorrent*( 75 self: CodexNodeRef, 76 stream: LPStream, 77 filename: ?string = string.none, 78 mimetype: ?string = string.none, 79 ): Future[?!MultiHash] {.async.} = 80 info "Storing BitTorrent data" 81 82 without bitTorrentManifest =? 83 await self.storePieces( 84 stream, filename = filename, mimetype = mimetype, blockSize = BitTorrentBlockSize 85 ): 86 return failure("Unable to store BitTorrent data") 87 88 trace "Created BitTorrent manifest", bitTorrentManifest = $bitTorrentManifest 89 90 let infoBencoded = bencode(bitTorrentManifest.info) 91 92 trace "BitTorrent Info successfully bencoded" 93 94 without infoHash =? MultiHash.digest($Sha1HashCodec, infoBencoded).mapFailure, err: 95 return failure(err) 96 97 trace "computed info hash", infoHash = $infoHash 98 99 without manifestBlk =? await self.storeBitTorrentManifest( 100 bitTorrentManifest, infoHash 101 ), err: 102 error "Unable to store manifest" 103 return failure(err) 104 105 info "Stored BitTorrent data", 106 infoHash = $infoHash, codexManifestCid = bitTorrentManifest.codexManifestCid 107 108 success infoHash 109 ``` 110 111 It starts with `self.storePieces`, which returns a [[BitTorrent Manifest]]. A manifest contains the BitTorrent Info dictionary and the corresponding Codex Manifest Cid: 112 113 ``` 114 type 115 BitTorrentInfo* = ref object 116 length*: uint64 117 pieceLength*: uint32 118 pieces*: seq[MultiHash] 119 name*: ?string 120 121 BitTorrentManifest* = ref object 122 info*: BitTorrentInfo 123 codexManifestCid*: Cid 124 ``` 125 126 `storePieces` does a very similar job to the `store` proc which is used for the regular Codex content, but additionally, it computes the *piece hashes* and creates the `info` dictionary and finally returns the corresponding `BitTorrentManifest`. 127 128 Back in `storeTorrent`, we *b-encode* the `info` dictionary and compute its hash (multihash). This `info` (multi) hash is what we will use to announce the content on the Codex DHT (see [[Announcing BitTorrent Content on Codex DHT]]). 129 130 Finally, `storeBitTorrentManifest` will effectively store the BitTorrent manifest block on the Codex network: 131 132 ``` 133 proc storeBitTorrentManifest*( 134 self: CodexNodeRef, manifest: BitTorrentManifest, infoHash: MultiHash 135 ): Future[?!bt.Block] {.async.} = 136 let encodedManifest = manifest.encode() 137 138 without infoHashCid =? Cid.init(CIDv1, InfoHashV1Codec, infoHash).mapFailure, error: 139 trace "Unable to create CID for BitTorrent info hash" 140 return failure(error) 141 142 without blk =? bt.Block.new(data = encodedManifest, cid = infoHashCid, verify = false), 143 error: 144 trace "Unable to create block from manifest" 145 return failure(error) 146 147 if err =? (await self.networkStore.putBlock(blk)).errorOption: 148 trace "Unable to store BitTorrent manifest block", cid = blk.cid, err = err.msg 149 return failure(err) 150 151 success blk 152 ``` 153 154 Some important things happen here. First, notice, that in Codex we use `Cids` to refer to the content. This is very handy: requesting a cid and getting the corresponding data back, we can immediately check if the content multihash present in the Cid, matches the computed multihash of the received data. If they do not match, we know immediately that the received block is invalid. 155 156 But looking at the code above, a careful reader will spot immediately that we are *cheating* a bit. 157 158 We first create a cid (`infoHashCid`) using precomputed `infoHash`, which we then associate with the `encodedManifest` in the `Block.new` call. Clearly, `info` hash does not identify our `encodedManifest`: if we compute a hash of the `encodedManifest`, it would not match the precomputed `infoHash`. This is because our Bit Torrent Manifest is more than just the `info` dictionary: it also contains the cid of the corresponding Codex Manifest for our content. 159 160 This cid is clearly not a valid cid. 161 162 We could create a valid Cid, by, for instance, creating a hash over the whole `encodedManifest` and appending it to the precomputed `infoHash` in such a Cid. Then, while retrieving the corresponding block back, we could first compare that the computed hash over the retrieved data matches the hash of the `encodedManifest` that we included in our cid, and then after reconstructing the BitTorrent Manifest from the encoded data, we could b-encode the `info` dictionary from the reconstructed BitTorrent Manifest, compute its hash, and compare it with the precomputed `infoHash` included in the cid. This would make the cid valid, but there is a problem with this approach. 163 164 In Codex, we use cids as references to blocks in `RepoStore`. We namely use cids as inputs to functions like `createBlockExpirationMetadataKey` or `makePrefixKey`. The cid itself is not preserved. The uploader (the seeder) has all necessary data to create an extended cid we describe in the paragraph above, but when requesting, the downloader knows only the `info` hash or potentially the contents of the the `.torrent` metadata file. In any case, the downloader does not know the cid of the underlying Codex manifest, pointing to the actual data. This means that the downloader is unable to create a full cid with the appended hash of the full `encodedManifest`. It is technically possible to send such an incomplete cid and use it to retrieve the full cid from the uploader datastore, but we are not making the system any more secure by doing this. The sender, can easily send a forged block with with perfectly valid cid as it has all necessary information to compute the appended hash, but the receiver, not having access to this information beforehand, will not be able to validate it. 165 166 Does it mean we can only be sure that the received content identified by the cid of the Codex manifest matches the requested info hash? No. 167 168 Notice, that BitTorrent does not use cids. The BitTorrent protocol operates at the level of pieces, and in version 1 of the protocol does not even use inclusion proofs. Yet, it does not wait till the whole piece is fetched in order to conclude it is genuine. 169 170 The info dictionary contains the `pieces` attribute, with hashes for all pieces. Once the piece is aggregated from the underlying blocks of `16KiB`, the hash is computed and compared against an entry in the `pieces` array. And this exactly what we do in Codex in order to prove that the received data, identified by the cid of the Codex manifest, matches the requested `info` hash. 171 Moreover, we also validate the received data at the block level, even before being able to validate the complete piece. We get this as a bonus from the Codex protocol, which together with data block, sends also the corresponding inclusion proof. Thus, even though at the moment we validate the individual blocks, we do not know if the received data, identified by the cid of the Codex manifest, matches the requested `info` hash, we do know already if the received data matches the Codex manifest. If this is not the case, if does not even make sense to aggregate pieces. 172 173 Thus, to summarize, while we cannot validate if the received BitTorrent manifest points to the valid data by validating the corresponding cid (`infoHashCid`), we do it later in two phases. Let's look at the download flow, starting from the end. 174 175 ### Downloading BitTorrent Content from Codex 176 177 We start from the `NetworkPeer.readLoop` (in `codex/blockexchange/network/networkpeer.nim`), where we decode the protocol `Message` with: 178 179 ```nim 180 data = await conn.readLp(MaxMessageSize.int) 181 msg = Message.protobufDecode(data).mapFailure().tryGet() 182 ``` 183 184 There, for each data item, we call: 185 186 ```nim 187 BlockDelivery.decode(initProtoBuffer(item, maxSize = MaxBlockSize)) 188 ``` 189 190 and this is where we get the cid, `Block`, `BlockAddress`, and the corresponding `proof` (for regular data, or *leaf* blocks): 191 192 ```nim 193 proc decode*(_: type BlockDelivery, pb: ProtoBuffer): ProtoResult[BlockDelivery] = 194 var 195 value = BlockDelivery() 196 dataBuf = newSeq[byte]() 197 cidBuf = newSeq[byte]() 198 cid: Cid 199 ipb: ProtoBuffer 200 201 if ?pb.getField(1, cidBuf): 202 cid = ?Cid.init(cidBuf).mapErr(x => ProtoError.IncorrectBlob) 203 if ?pb.getField(2, dataBuf): 204 value.blk = 205 ?Block.new(cid, dataBuf, verify = true).mapErr(x => ProtoError.IncorrectBlob) 206 if ?pb.getField(3, ipb): 207 value.address = ?BlockAddress.decode(ipb) 208 209 if value.address.leaf: 210 var proofBuf = newSeq[byte]() 211 if ?pb.getField(4, proofBuf): 212 let proof = ?CodexProof.decode(proofBuf).mapErr(x => ProtoError.IncorrectBlob) 213 value.proof = proof.some 214 else: 215 value.proof = CodexProof.none 216 else: 217 value.proof = CodexProof.none 218 219 ok(value) 220 ``` 221 222 We see that we while constructing instance of `Block`, we already request the block validation by setting `verify = true`: 223 224 ```nim 225 proc new*( 226 T: type Block, cid: Cid, data: openArray[byte], verify: bool = true 227 ): ?!Block = 228 ## creates a new block for both storage and network IO 229 ## 230 231 without isTorrent =? cid.isTorrentCid, err: 232 return "Unable to determine if cid is torrent info hash".failure 233 234 # info hash cids are "fake cids" - they will not validate 235 # info hash validation is done outside of the cid itself 236 if verify and not isTorrent: 237 let 238 mhash = ?cid.mhash.mapFailure 239 computedMhash = ?MultiHash.digest($mhash.mcodec, data).mapFailure 240 computedCid = ?Cid.init(cid.cidver, cid.mcodec, computedMhash).mapFailure 241 if computedCid != cid: 242 return "Cid doesn't match the data".failure 243 244 return Block(cid: cid, data: @data).success 245 ``` 246 247 Here we see that because as explained above, the cids corresponding to the BitTorrent manifest blocks cannot be immediately validated, we make sure, the validation is skipped here for Torrent cids. 248 249 Once the `Message` is decoded, back in `NetworkPeer.readLoop`, it is passed to `NetworkPeer.handler` which is set to `Network.rpcHandler` while creating the instance of `NetworkPeer` in `Network.getOrCreatePeer`. For block deliveries, `Network.rpcHandler` forwards `msg.payload` (`seq[BlockDelivery]`) to `Network.handleBlocksDelivery`, which in turn, calls `Network.handlers.onBlocksDelivery`. The `Network.handlers.onBlocksDelivery` is set by the constructor of `BlockExcEngine`. Thus, in the end of its journey, a `seq[BlockDelivery]` from the `msg.payload` ends up in `BlockExcEngine.blocksDeliveryHandler`. This is where the data blocks are further validated against the inclusion proof and then the validated data (*leafs*) blocks or non-data blocks (non-*leafs*, e.g. a BitTorrent or Codex Manifest block), are stored in the `localStore` and then *resolved* against pending blocks via `BlockExcEngine.resolveBlocks` that calls `pendingBlocks.resolve(blocksDelivery)` (`PendingBlocksManager`). This is where `blockReq.handle.complete(bd.blk)` is called on the matching pending blocks, which completes future awaited in `BlockExcEngine.requestBlock`, which completes the future awaited in `NetworkStore.getBlock`: `await self.engine.requestBlock(address)`. And `NetworkStore.getBlock` was awaited either in `CodexNodeRef.fetchPieces` for data blocks or in `CodexNodeRef.fetchTorrentManifest`. 250 251 So, how do we get to `CodexNodeRef.fetchPieces` and `CodexNodeRef.fetchTorrentManifest` in the download flow. 252 253 It starts with the API handler of `/api/codex/v1/torrent/{infoHash}/network/stream`: 254 255 ```nim 256 router.api(MethodGet, "/api/codex/v1/torrent/{infoHash}/network/stream") do( 257 infoHash: MultiHash, resp: HttpResponseRef 258 ) -> RestApiResponse: 259 var headers = buildCorsHeaders("GET", allowedOrigin) 260 261 without infoHash =? infoHash.mapFailure, error: 262 return RestApiResponse.error(Http400, error.msg, headers = headers) 263 264 if infoHash.mcodec != Sha1HashCodec: 265 return RestApiResponse.error( 266 Http400, "Only torrents version 1 are currently supported!", headers = headers 267 ) 268 269 if corsOrigin =? allowedOrigin: 270 resp.setCorsHeaders("GET", corsOrigin) 271 resp.setHeader("Access-Control-Headers", "X-Requested-With") 272 273 trace "torrent requested: ", multihash = $infoHash 274 275 await node.retrieveInfoHash(infoHash, resp = resp) 276 ``` 277 278 `CodexNodeRef.retrieveInfoHash` first tries to fetch the `Torrent` object, which consists of `torrentManifest` and `codexManifest`. To get it, it calls `node.retrieveTorrent(infoHash)` with the `infoHash` as the argument. And then in the `retrieveTorrent` we get to the above mentioned `fetchTorrentManifest`: 279 280 ```nim 281 proc retrieveTorrent*( 282 self: CodexNodeRef, infoHash: MultiHash 283 ): Future[?!Torrent] {.async.} = 284 without infoHashCid =? Cid.init(CIDv1, InfoHashV1Codec, infoHash).mapFailure, error: 285 trace "Unable to create CID for BitTorrent info hash" 286 return failure(error) 287 288 without torrentManifest =? (await self.fetchTorrentManifest(infoHashCid)), err: 289 trace "Unable to fetch Torrent Manifest" 290 return failure(err) 291 292 without codexManifest =? (await self.fetchManifest(torrentManifest.codexManifestCid)), 293 err: 294 trace "Unable to fetch Codex Manifest for torrent info hash" 295 return failure(err) 296 297 success (torrentManifest: torrentManifest, codexManifest: codexManifest) 298 ``` 299 300 We first create `infoHashCid`, using only the precomputed `infoHash` and we pass it to `fetchTorrentManifest`: 301 302 ```nim 303 proc fetchTorrentManifest*( 304 self: CodexNodeRef, infoHashCid: Cid 305 ): Future[?!BitTorrentManifest] {.async.} = 306 if err =? infoHashCid.isTorrentInfoHash.errorOption: 307 return failure "CID has invalid content type for torrent info hash {$cid}" 308 309 trace "Retrieving torrent manifest for infoHashCid", infoHashCid 310 311 without blk =? await self.networkStore.getBlock(BlockAddress.init(infoHashCid)), err: 312 trace "Error retrieve manifest block", infoHashCid, err = err.msg 313 return failure err 314 315 trace "Successfully retrieved torrent manifest with given block cid", 316 cid = blk.cid, infoHashCid 317 trace "Decoding torrent manifest" 318 319 without torrentManifest =? BitTorrentManifest.decode(blk), err: 320 trace "Unable to decode torrent manifest", err = err.msg 321 return failure("Unable to decode torrent manifest") 322 323 trace "Decoded torrent manifest", infoHashCid, torrentManifest = $torrentManifest 324 325 without isValid =? torrentManifest.validate(infoHashCid), err: 326 trace "Error validating torrent manifest", infoHashCid, err = err.msg 327 return failure(err.msg) 328 329 if not isValid: 330 trace "Torrent manifest does not match torrent info hash", infoHashCid 331 return failure "Torrent manifest does not match torrent info hash {$infoHashCid}" 332 333 return torrentManifest.success 334 ``` 335 336 Here we will be awaiting for the `networkStore.getBlock`, which will get completed with the block delivery flow we describe at the beginning of this section. We restore the `BitTorrentManifest` object using `BitTorrentManifest.decode(blk)`, and then we validate if the `info` dictionary from the received BitTorrent manifest matches the request `infoHash`: 337 338 ```nim 339 without isValid =? torrentManifest.validate(infoHashCid), err: 340 trace "Error validating torrent manifest", infoHashCid, err = err.msg 341 return failure(err.msg) 342 ``` 343 344 Thus, now we know that we have genuine `info` dictionary. 345 346 Now, we still need to get and validate the actual data. 347 348 BitTorrent manifest includes the cid of the Codex manifest in `codexManifestCid` attribute. Back in `retrieveTorrent`, we thus now fetch the Codex manifest, and we return both to `retrieveInfoHash`, where the download effectively started. 349 350 The `retrieveInfoHash` calls `streamTorrent` passing both manifests as arguments: 351 352 ```nim 353 let stream = await node.streamTorrent(torrentManifest, codexManifest) 354 ``` 355 356 Let's take a look at `streamTorrent`: 357 358 ```nim 359 proc streamTorrent*( 360 self: CodexNodeRef, torrentManifest: BitTorrentManifest, codexManifest: Manifest 361 ): Future[LPStream] {.async: (raises: []).} = 362 trace "Retrieving pieces from torrent" 363 let stream = LPStream(StoreStream.new(self.networkStore, codexManifest, pad = false)) 364 var jobs: seq[Future[void]] 365 366 proc onPieceReceived(blocks: seq[bt.Block], pieceIndex: int): ?!void {.raises: [].} = 367 trace "Fetched torrent piece - verifying..." 368 369 var pieceHashCtx: sha1 370 pieceHashCtx.init() 371 372 for blk in blocks: 373 pieceHashCtx.update(blk.data) 374 375 let pieceHash = pieceHashCtx.finish() 376 377 if (pieceHash != torrentManifest.info.pieces[pieceIndex]): 378 error "Piece verification failed", pieceIndex = pieceIndex 379 return failure("Piece verification failed") 380 381 trace "Piece verified", pieceIndex, pieceHash 382 # great success 383 success() 384 385 proc prefetch(): Future[void] {.async: (raises: []).} = 386 try: 387 if err =? ( 388 await self.fetchPieces(torrentManifest, codexManifest, onPieceReceived) 389 ).errorOption: 390 error "Unable to fetch blocks", err = err.msg 391 await stream.close() 392 except CancelledError: 393 trace "Prefetch cancelled" 394 395 jobs.add(prefetch()) 396 397 # Monitor stream completion and cancel background jobs when done 398 proc monitorStream() {.async: (raises: []).} = 399 try: 400 await stream.join() 401 except CancelledError: 402 trace "Stream cancelled" 403 finally: 404 await noCancel allFutures(jobs.mapIt(it.cancelAndWait)) 405 406 self.trackedFutures.track(monitorStream()) 407 408 trace "Creating store stream for torrent manifest" 409 stream 410 ``` 411 412 `streamTorrent` does three things: 413 414 1. starts background `prefetch` job 415 2. monitors the stream using `monitorStream` 416 3. validates the aggregated pieces 417 418 The `prefetch` job calls `fetchPieces`: 419 420 ```nim 421 proc fetchPieces*( 422 self: CodexNodeRef, 423 torrentManifest: BitTorrentManifest, 424 codexManifest: Manifest, 425 onPiece: PieceProc, 426 ): Future[?!void] {.async: (raw: true, raises: [CancelledError]).} = 427 trace "Fetching torrent pieces" 428 429 let numOfPieces = torrentManifest.info.pieces.len 430 let numOfBlocksPerPiece = 431 torrentManifest.info.pieceLength.int div codexManifest.blockSize.int 432 let blockIter = Iter[int].new(0 ..< codexManifest.blocksCount) 433 let pieceIter = Iter[int].new(0 ..< numOfPieces) 434 self.fetchPieces( 435 codexManifest.treeCid, blockIter, pieceIter, numOfBlocksPerPiece, onPiece 436 ) 437 ``` 438 439 At this level, we create the iterators to manage the sequential processing of blocks and pieces with each piece containing `numOfBlocksPerPiece` blocks. Subsequently, we call the overloaded version of `fetchPieces` that will perform the actual (pre) fetching: 440 441 ```nim 442 proc fetchPieces*( 443 self: CodexNodeRef, 444 cid: Cid, 445 blockIter: Iter[int], 446 pieceIter: Iter[int], 447 numOfBlocksPerPiece: int, 448 onPiece: PieceProc, 449 ): Future[?!void] {.async: (raises: [CancelledError]).} = 450 while not blockIter.finished: 451 let blockFutures = collect: 452 for i in 0 ..< numOfBlocksPerPiece: 453 if not blockIter.finished: 454 let address = BlockAddress.init(cid, blockIter.next()) 455 self.networkStore.getBlock(address) 456 457 without blocks =? await allFinishedValues(blockFutures), err: 458 return failure(err) 459 460 if pieceErr =? (onPiece(blocks, pieceIter.next())).errorOption: 461 return failure(pieceErr) 462 463 await sleepAsync(1.millis) 464 465 success() 466 ``` 467 468 We fetch blocks in batches, or rather per pieces. We trigger fetching blocks with `self.networkStore.getBlock(address)`, which will resolve by either getting the block from the local store or from the network through block delivery described above. 469 470 Notice, we need to get all the blocks here, not only trigger fetching the blocks that are not yet available in the local store. This is necessary, because we need to get all the blocks in a piece so that we can validate the piece and potentially stop streaming if the piece turns out to be invalid. 471 472 Before calling `onPiece`, where validation will take place, we wait for all `Futures` to complete returning the requested blocks. 473 474 `onPiece` is set to `onPieceReceived` in `streamTorrent` and it basically computes the SHA1 hash of the concatenated blocks and checks if it matches the (multi) hash from the `info` dictionary. This steps forms the second validation step: after we checked that the `info` dictionary matches the requested `info` hash in the first step described above, here we are making sure that the received content matches the metadata in the `info` dictionary, and thus it is indeed the content identified by the `info` hash from the request. 475 476 `fetchPieces` operates in background, and thus very likely after first piece has been fetched the stream will be returned to `retrieveInfoHash` where streaming the blocks down to the client will take place: 477 478 ```nim 479 proc retrieveInfoHash( 480 node: CodexNodeRef, infoHash: MultiHash, resp: HttpResponseRef 481 ): Future[void] {.async.} = 482 ## Download torrent from the node in a streaming 483 ## manner 484 ## 485 var stream: LPStream 486 487 var bytes = 0 488 try: 489 without torrent =? (await node.retrieveTorrent(infoHash)), err: 490 error "Unable to fetch Torrent Metadata", err = err.msg 491 resp.status = Http404 492 await resp.sendBody(err.msg) 493 return 494 let (torrentManifest, codexManifest) = torrent 495 496 if codexManifest.mimetype.isSome: 497 resp.setHeader("Content-Type", codexManifest.mimetype.get()) 498 else: 499 resp.addHeader("Content-Type", "application/octet-stream") 500 501 if codexManifest.filename.isSome: 502 resp.setHeader( 503 "Content-Disposition", 504 "attachment; filename=\"" & codexManifest.filename.get() & "\"", 505 ) 506 else: 507 resp.setHeader("Content-Disposition", "attachment") 508 509 await resp.prepareChunked() 510 511 let stream = await node.streamTorrent(torrentManifest, codexManifest) 512 513 while not stream.atEof: 514 var 515 buff = newSeqUninitialized[byte](BitTorrentBlockSize.int) 516 len = await stream.readOnce(addr buff[0], buff.len) 517 518 buff.setLen(len) 519 if buff.len <= 0: 520 break 521 522 bytes += buff.len 523 524 await resp.sendChunk(addr buff[0], buff.len) 525 await resp.finish() 526 codex_api_downloads.inc() 527 except CancelledError as exc: 528 raise exc 529 except CatchableError as exc: 530 warn "Error streaming blocks", exc = exc.msg 531 resp.status = Http500 532 if resp.isPending(): 533 await resp.sendBody(exc.msg) 534 finally: 535 info "Sent bytes for torrent", infoHash = $infoHash, bytes 536 if not stream.isNil: 537 await stream.close() 538 ``` 539 540 Now, two important points. First, when the streaming happens to be interrupted the stream will be closed in the `finally` block. This in turns will be detected by the `monitorStream` in `streamTorrent` causing the `prefetch` job to be cancelled. Second, when either piece validation fails, or if any of the `getBlock` future awaiting completion fails, `prefetch` will return error, which will cause the stream to be closed: 541 542 ```nim 543 proc prefetch(): Future[void] {.async: (raises: []).} = 544 try: 545 if err =? ( 546 await self.fetchPieces(torrentManifest, codexManifest, onPieceReceived) 547 ).errorOption: 548 error "Unable to fetch blocks", err = err.msg 549 await stream.close() 550 except CancelledError: 551 trace "Prefetch cancelled" 552 ``` 553 554 Without this detection mechanism, we would either continue fetching blocks even when streaming API request has been interrupted, or we would continue streaming, even when it is already known that the piece validation phase has failed. This would result in invalid content being returned to the client. After any failure in the `prefetch` job, the pieces will no longer be validated, thus it does not make any sense to continue the streaming operation, which otherwise, would cause fetching blocks one-by-one in the streaming loop in `retrieve`: 555 556 ```nim 557 while not stream.atEof: 558 var 559 buff = newSeqUninitialized[byte](BitTorrentBlockSize.int) 560 len = await stream.readOnce(addr buff[0], buff.len) 561 562 buff.setLen(len) 563 if buff.len <= 0: 564 break 565 566 bytes += buff.len 567 568 await resp.sendChunk(addr buff[0], buff.len) 569 ``` 570 571 The `stream.readOnce` implemented in `StoreStream`, which uses the same underlying `networkStore` that is also used in `fetchPieces` proc shown above, will be calling that same `getBlock` operation, which in case the block is not already in local store (because it was already there or as a result of the prefetch operation), will request it from the block exchange engine via `BlockExcEngine.requestBlock` operation. In case there is already a pending request for the given block address, the `PendingBlocksManager` will return the existing block handle, so that `BlockExcEngine.requestBlock` operation will not cause duplicate request. It will, however potentially return an invalid block to the client, before the containing piece has been validated in the prefetch phase. 572 573 > [!danger] 574 This may occasionally cause an unlikely event in which both `fetchPieces` in the `prefetch` job and `readOnce` on the `StoreStream` will be awaiting on the `getBlock` of the very last block request, and `readOnce` will regain the control before `fetchPieces` does. In such a case, the client may complete the streaming operation successfully, even if the corresponding last piece validation fails. 575 > 576 > OR, in even more unlikely event that all the blocks belonging to the last piece are in the local store, `readOnce` may consume them before the last piece is validated. 577 > 578 > We should perhaps make sure that the very last piece can only be streamed to the client after the `prefetch` operation completes. 579 > 580 581 ```bash 582 TRC 2025-03-19 16:11:06.334+01:00 torrent requested: topics="codex restapi" tid=5050038 multihash=sha1/4249FFB943675890CF09342629CD3782D107B709 583 TRC 2025-03-19 16:11:06.334+01:00 Retrieving torrent manifest for infoHashCid topics="codex node" tid=5050038 infoHashCid=z8q*fCdDsv 584 TRC 2025-03-19 16:11:06.340+01:00 Successfully retrieved torrent manifest with given block cid topics="codex node" tid=5050038 cid=z8q*fCdDsv infoHashCid=z8q*fCdDsv 585 TRC 2025-03-19 16:11:06.340+01:00 Decoding torrent manifest topics="codex node" tid=5050038 586 TRC 2025-03-19 16:11:06.340+01:00 Decoded torrent manifest topics="codex node" tid=5050038 infoHashCid=z8q*fCdDsv torrentManifest="BitTorrentManifest(info: BitTorrentInfo(length: 10485760, pieceLength: 262144, pieces: @[sha1/5B3B77A971432C07B1C3C36A73BA5E79DBAE20EE, sha1/322890271DAE134970AABAA9CB30AF7464971599, sha1/2394F90BABA3677D44DDA0565543591AC26CA584, sha1/36EF17CDEFE621AE433A0A30EF3F6A20E0D07DB2, sha1/C21F2C1A1A6BF6E3E6CA75C13969F32EBE802867, sha1/B576B2118EC0B5B0D6AC5BB60BF12F881E5E395C, sha1/3FC4F41C6A83C8411BAE41E83AE327299FA31A98, sha1/92BD07E01D40E8BFB7C3B4597568B44F805508B5, sha1/DEC4E1492D2447436AF54AF3FC6319D409E2F48F, sha1/3E4BFDB4F970A9EDA9906E007073B0FB94A4F91A, sha1/CFB9B01285FA00BC43C5E439A65958D8CBBCD046, sha1/EAFB36F2DD5AA8EE80811946C26DFB8F95ABB18A, sha1/FEDFD7A30E09B395D8F78148CB826127070272DD, sha1/F1588A342C11776DB7DC8C5BCFA72A1701DF0442, sha1/34D4E095D53A0CA33375444A19652BD866027350, sha1/AE7FF2CA95BF751B68D2E22BB015435347CFE49C, sha1/85FCC1B3A8CE7D5C0397E8D27084C067A3067496, sha1/88C3DF9C35A23FE6B1E363F606068F15DB5EE093, sha1/98F7A4A6113A3CC9CB3EF086A2B9E36DAA92B78D, sha1/91DDF5B1F25715C17CD709019B71D2233A142CC1, sha1/1A850C78AB1CB596D8298371132A135DB96D5943, sha1/88E8F31AE70A6A81E25FF9E6D2DC824F07F1AF9A, sha1/A335A0DE3F7E1F4191D89E68F9AE9D4406CE4449, sha1/7AC5488A3B6C8A93F7DF3DE64BBCF2FA1F185F4D, sha1/7A88908AF090C1F8FC2BCF51C8BBB6F92D13AE01, sha1/BBEBBD427BEE80178C291366A570A355204E67CA, sha1/0112EDE76962115D7DDD24F7D14BE41BCCD51634, sha1/EFBEFC1DFBB0F7676447C9D0936775A8933008D2, sha1/6508DE0FD7FC8D8EE300C88F0743A52CD79CB616, sha1/F1B02EFB043CC6F37FE22C59055960AAC4ABDCC6, sha1/74FDA7F48FC089AA7A684D4E7C2C1F4BC5B08980, sha1/FAFAC020F2C1DEC260581810F8BB27BBCB211AFD, sha1/1AB425BA8DD6A2AD1C469417C7F7CB54E3226B82, sha1/08338248DAA53ADCBF65D052E93EB8BC677E14B3, sha1/BAD3B38D731F16B3F6663131EEE06E56EBFAB7C2, sha1/8651B73B110025390F4DDCE6AFD43F34ED65A0BE, sha1/486DCD80D8F25432E31BF59C8818003DFE8709F0, sha1/336038716957C64D3124EA1985D8180EE6A097E8, sha1/C7E2B6ADCAB60B57A9A47CCE8454BDAE46E6EE57, sha1/8B6DA36E26C2654B6407253A7816343F818DEDE5], name: some(\"data10M.bin\")), codexManifestCid: zDvZRwzm1GAigGvTcrXyC2cATBL9W2xNPqjKLKb835pHu4Xcm1E6)" 587 TRC 2025-03-19 16:11:06.341+01:00 Retrieving manifest for cid topics="codex node" tid=5050038 cid=zDv*Xcm1E6 588 TRC 2025-03-19 16:11:06.343+01:00 Decoding manifest for cid topics="codex node" tid=5050038 cid=zDv*Xcm1E6 589 TRC 2025-03-19 16:11:06.343+01:00 Decoded manifest topics="codex node" tid=5050038 cid=zDv*Xcm1E6 590 TRC 2025-03-19 16:11:06.344+01:00 Retrieving pieces from torrent topics="codex node" tid=5050038 591 TRC 2025-03-19 16:11:06.344+01:00 Fetching torrent pieces topics="codex node" tid=5050038 592 TRC 2025-03-19 16:11:06.344+01:00 Creating store stream for torrent manifest topics="codex node" tid=5050038 593 TRC 2025-03-19 16:11:06.344+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=0 594 TRC 2025-03-19 16:11:06.371+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 595 TRC 2025-03-19 16:11:06.372+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=0 596 TRC 2025-03-19 16:11:06.372+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=0 597 TRC 2025-03-19 16:11:06.385+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=1 598 TRC 2025-03-19 16:11:06.411+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 599 TRC 2025-03-19 16:11:06.412+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=1 600 TRC 2025-03-19 16:11:06.412+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=1 601 TRC 2025-03-19 16:11:06.419+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=2 602 TRC 2025-03-19 16:11:06.431+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 603 TRC 2025-03-19 16:11:06.431+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=2 604 TRC 2025-03-19 16:11:06.431+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=2 605 TRC 2025-03-19 16:11:06.438+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=3 606 TRC 2025-03-19 16:11:06.471+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 607 TRC 2025-03-19 16:11:06.471+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=3 608 TRC 2025-03-19 16:11:06.471+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=3 609 TRC 2025-03-19 16:11:06.480+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=4 610 TRC 2025-03-19 16:11:06.490+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 611 TRC 2025-03-19 16:11:06.491+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=4 612 TRC 2025-03-19 16:11:06.491+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=4 613 TRC 2025-03-19 16:11:06.496+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=5 614 TRC 2025-03-19 16:11:06.524+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 615 TRC 2025-03-19 16:11:06.524+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=5 616 TRC 2025-03-19 16:11:06.524+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=5 617 TRC 2025-03-19 16:11:06.531+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=6 618 TRC 2025-03-19 16:11:06.549+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 619 TRC 2025-03-19 16:11:06.549+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=6 620 TRC 2025-03-19 16:11:06.549+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=6 621 TRC 2025-03-19 16:11:06.556+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=7 622 TRC 2025-03-19 16:11:06.575+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 623 TRC 2025-03-19 16:11:06.575+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=7 624 TRC 2025-03-19 16:11:06.575+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=7 625 TRC 2025-03-19 16:11:06.583+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=8 626 TRC 2025-03-19 16:11:06.599+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 627 TRC 2025-03-19 16:11:06.600+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=8 628 TRC 2025-03-19 16:11:06.600+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=8 629 TRC 2025-03-19 16:11:06.605+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=9 630 TRC 2025-03-19 16:11:06.624+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 631 TRC 2025-03-19 16:11:06.625+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=9 632 TRC 2025-03-19 16:11:06.625+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=9 633 TRC 2025-03-19 16:11:06.631+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=10 634 TRC 2025-03-19 16:11:06.643+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 635 TRC 2025-03-19 16:11:06.643+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=10 636 TRC 2025-03-19 16:11:06.643+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=10 637 TRC 2025-03-19 16:11:06.648+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=11 638 TRC 2025-03-19 16:11:06.675+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 639 TRC 2025-03-19 16:11:06.675+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=11 640 TRC 2025-03-19 16:11:06.675+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=11 641 TRC 2025-03-19 16:11:06.682+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=12 642 TRC 2025-03-19 16:11:06.693+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 643 TRC 2025-03-19 16:11:06.694+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=12 644 TRC 2025-03-19 16:11:06.694+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=12 645 TRC 2025-03-19 16:11:06.699+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=13 646 TRC 2025-03-19 16:11:06.725+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 647 TRC 2025-03-19 16:11:06.725+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=13 648 TRC 2025-03-19 16:11:06.726+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=13 649 TRC 2025-03-19 16:11:06.732+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=14 650 TRC 2025-03-19 16:11:06.744+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 651 TRC 2025-03-19 16:11:06.744+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=14 652 TRC 2025-03-19 16:11:06.744+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=14 653 TRC 2025-03-19 16:11:06.749+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=15 654 TRC 2025-03-19 16:11:06.775+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 655 TRC 2025-03-19 16:11:06.776+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=15 656 TRC 2025-03-19 16:11:06.776+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=15 657 TRC 2025-03-19 16:11:06.783+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=16 658 TRC 2025-03-19 16:11:06.794+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 659 TRC 2025-03-19 16:11:06.794+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=16 660 TRC 2025-03-19 16:11:06.794+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=16 661 TRC 2025-03-19 16:11:06.800+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=17 662 TRC 2025-03-19 16:11:06.825+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 663 TRC 2025-03-19 16:11:06.826+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=17 664 TRC 2025-03-19 16:11:06.826+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=17 665 TRC 2025-03-19 16:11:06.832+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=18 666 TRC 2025-03-19 16:11:06.844+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 667 TRC 2025-03-19 16:11:06.844+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=18 668 TRC 2025-03-19 16:11:06.844+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=18 669 TRC 2025-03-19 16:11:06.850+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=19 670 TRC 2025-03-19 16:11:06.877+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 671 TRC 2025-03-19 16:11:06.877+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=19 672 TRC 2025-03-19 16:11:06.877+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=19 673 TRC 2025-03-19 16:11:06.884+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=20 674 TRC 2025-03-19 16:11:06.902+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 675 TRC 2025-03-19 16:11:06.902+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=20 676 TRC 2025-03-19 16:11:06.902+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=20 677 TRC 2025-03-19 16:11:06.907+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=21 678 TRC 2025-03-19 16:11:06.926+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 679 TRC 2025-03-19 16:11:06.927+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=21 680 TRC 2025-03-19 16:11:06.927+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=21 681 TRC 2025-03-19 16:11:06.933+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=22 682 TRC 2025-03-19 16:11:06.945+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 683 TRC 2025-03-19 16:11:06.945+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=22 684 TRC 2025-03-19 16:11:06.945+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=22 685 TRC 2025-03-19 16:11:06.951+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=23 686 TRC 2025-03-19 16:11:06.978+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 687 TRC 2025-03-19 16:11:06.979+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=23 688 TRC 2025-03-19 16:11:06.980+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=23 689 TRC 2025-03-19 16:11:06.994+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=24 690 TRC 2025-03-19 16:11:07.023+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 691 TRC 2025-03-19 16:11:07.023+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=24 692 TRC 2025-03-19 16:11:07.023+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=24 693 TRC 2025-03-19 16:11:07.033+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=25 694 TRC 2025-03-19 16:11:07.049+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 695 TRC 2025-03-19 16:11:07.049+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=25 696 TRC 2025-03-19 16:11:07.049+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=25 697 TRC 2025-03-19 16:11:07.055+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=26 698 TRC 2025-03-19 16:11:07.084+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 699 TRC 2025-03-19 16:11:07.084+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=26 700 TRC 2025-03-19 16:11:07.084+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=26 701 TRC 2025-03-19 16:11:07.091+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=27 702 TRC 2025-03-19 16:11:07.121+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 703 TRC 2025-03-19 16:11:07.121+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=27 704 TRC 2025-03-19 16:11:07.121+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=27 705 TRC 2025-03-19 16:11:07.128+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=28 706 TRC 2025-03-19 16:11:07.140+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 707 TRC 2025-03-19 16:11:07.140+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=28 708 TRC 2025-03-19 16:11:07.141+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=28 709 TRC 2025-03-19 16:11:07.148+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=29 710 TRC 2025-03-19 16:11:07.401+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 711 TRC 2025-03-19 16:11:07.401+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=29 712 TRC 2025-03-19 16:11:07.401+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=29 713 TRC 2025-03-19 16:11:07.409+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=30 714 TRC 2025-03-19 16:11:07.442+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 715 TRC 2025-03-19 16:11:07.442+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=30 716 TRC 2025-03-19 16:11:07.442+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=30 717 TRC 2025-03-19 16:11:07.449+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=31 718 TRC 2025-03-19 16:11:07.481+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 719 TRC 2025-03-19 16:11:07.481+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=31 720 TRC 2025-03-19 16:11:07.481+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=31 721 TRC 2025-03-19 16:11:07.488+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=32 722 TRC 2025-03-19 16:11:07.499+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 723 TRC 2025-03-19 16:11:07.500+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=32 724 TRC 2025-03-19 16:11:07.500+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=32 725 TRC 2025-03-19 16:11:07.505+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=33 726 TRC 2025-03-19 16:11:07.536+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 727 TRC 2025-03-19 16:11:07.536+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=33 728 TRC 2025-03-19 16:11:07.536+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=33 729 TRC 2025-03-19 16:11:07.544+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=34 730 TRC 2025-03-19 16:11:07.555+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 731 TRC 2025-03-19 16:11:07.556+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=34 732 TRC 2025-03-19 16:11:07.556+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=34 733 TRC 2025-03-19 16:11:07.561+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=35 734 TRC 2025-03-19 16:11:07.587+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 735 TRC 2025-03-19 16:11:07.588+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=35 736 TRC 2025-03-19 16:11:07.588+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=35 737 TRC 2025-03-19 16:11:07.594+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=36 738 TRC 2025-03-19 16:11:07.606+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 739 TRC 2025-03-19 16:11:07.606+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=36 740 TRC 2025-03-19 16:11:07.606+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=36 741 TRC 2025-03-19 16:11:07.612+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=37 742 TRC 2025-03-19 16:11:07.637+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 743 TRC 2025-03-19 16:11:07.638+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=37 744 TRC 2025-03-19 16:11:07.638+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=37 745 TRC 2025-03-19 16:11:07.646+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=38 746 TRC 2025-03-19 16:11:07.663+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 747 TRC 2025-03-19 16:11:07.663+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=38 748 TRC 2025-03-19 16:11:07.663+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=38 749 TRC 2025-03-19 16:11:07.669+01:00 Waiting for piece topics="codex restapi" tid=5050038 pieceIndex=39 750 TRC 2025-03-19 16:11:07.691+01:00 Fetched torrent piece - verifying... topics="codex node" tid=5050038 751 TRC 2025-03-19 16:11:07.692+01:00 Piece verified topics="codex node" tid=5050038 pieceIndex=39 752 TRC 2025-03-19 16:11:07.692+01:00 Got piece topics="codex restapi" tid=5050038 pieceIndex=39 753 INF 2025-03-19 16:11:07.696+01:00 Sent bytes for torrent topics="codex restapi" tid=5050038 infoHash=sha1/4249FFB943675890CF09342629CD3782D107B709 bytes=10485760 754 755 ```