1  # Build Codex
  2  
  3  ## Table of Contents
  4  
  5  - [Install developer tools](#prerequisites)
  6    - [Linux](#linux)
  7    - [macOS](#macos)
  8    - [Windows + MSYS2](#windows-msys2)
  9    - [Other](#other)
 10  - [Clone and prepare the Git repository](#repository)
 11  - [Build the executable](#executable)
 12  - [Run the example](#example-usage)
 13  
 14  **Optional**
 15  - [Run the tests](#tests)
 16  
 17  ## Prerequisites
 18  
 19  To build nim-codex, developer tools need to be installed and accessible in the OS.
 20  
 21  Instructions below correspond roughly to environmental setups in nim-codex's [CI workflow](https://github.com/codex-storage/nim-codex/blob/master/.github/workflows/ci.yml) and are known to work.
 22  
 23  Other approaches may be viable. On macOS, some users may prefer [MacPorts](https://www.macports.org/) to [Homebrew](https://brew.sh/). On Windows, rather than use MSYS2, some users may prefer to install developer tools with [winget](https://docs.microsoft.com/en-us/windows/package-manager/winget/), [Scoop](https://scoop.sh/), or [Chocolatey](https://chocolatey.org/), or download installers for e.g. Make and CMake while otherwise relying on official Windows developer tools. Community contributions to these docs and our build system are welcome!
 24  
 25  ### Rust
 26  
 27  The current implementation of Codex's zero-knowledge proving circuit requires the installation of rust v1.79.0 or greater. Be sure to install it for your OS and add it to your terminal's path such that the command `cargo --version` gives a compatible version.
 28  
 29  ### Linux
 30  
 31  > [!WARNING]
 32  > Linux builds currently require gcc $\leq$ 13. If this is not an option in your
 33  > system, you can try [building within Docker](#building-within-docker) as a workaround.
 34  
 35  *Package manager commands may require `sudo` depending on OS setup.*
 36  
 37  On a bare bones installation of Debian (or a distribution derived from Debian, such as Ubuntu), run
 38  
 39  ```shell
 40  apt-get update && apt-get install build-essential cmake curl git rustc cargo
 41  ```
 42  
 43  Non-Debian distributions have different package managers: `apk`, `dnf`, `pacman`, `rpm`, `yum`, etc.
 44  
 45  For example, on a bare bones installation of Fedora, run
 46  
 47  ```shell
 48  dnf install @development-tools cmake gcc-c++ rust cargo
 49  ```
 50  
 51  In case your distribution does not provide required Rust version, we may install it using [rustup](https://www.rust-lang.org/tools/install)
 52  ```shell
 53  curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs/ | sh -s -- --default-toolchain=1.79.0 -y
 54  
 55  . "$HOME/.cargo/env"
 56  ```
 57  
 58  Note that you will currently not be able to build Codex with gcc 14. To verify that 
 59  you have a supported version, run:
 60  
 61  ```shell
 62  gcc --version
 63  ```
 64  
 65  If you get a number that starts with 14 (e.g. `14.2.0`), then you need to either
 66  downgrade, or try a workaround like [building within Docker](#building-within-docker).
 67  
 68  ### macOS
 69  
 70  Install the [Xcode Command Line Tools](https://mac.install.guide/commandlinetools/index.html) by opening a terminal and running
 71  ```shell
 72  xcode-select --install
 73  ```
 74  
 75  Install [Homebrew (`brew`)](https://brew.sh/) and in a new terminal run
 76  ```shell
 77  brew install bash cmake rust
 78  ```
 79  
 80  Check that `PATH` is setup correctly
 81  ```shell
 82  which bash cmake
 83  
 84  # /usr/local/bin/bash
 85  # /usr/local/bin/cmake
 86  ```
 87  
 88  ### Windows + MSYS2
 89  
 90  *Instructions below assume the OS is 64-bit Windows and that the hardware or VM is [x86-64](https://en.wikipedia.org/wiki/X86-64) compatible.*
 91  
 92  Download and run the installer from [msys2.org](https://www.msys2.org/).
 93  
 94  Launch an MSYS2 [environment](https://www.msys2.org/docs/environments/). UCRT64 is generally recommended: from the Windows *Start menu* select `MSYS2 MinGW UCRT x64`.
 95  
 96  Assuming a UCRT64 environment, in Bash run
 97  ```shell
 98  pacman -Suy
 99  pacman -S base-devel git unzip mingw-w64-ucrt-x86_64-toolchain mingw-w64-ucrt-x86_64-cmake mingw-w64-ucrt-x86_64-rust
100  ```
101  
102  We should downgrade GCC to version 13 [^gcc-14]
103  ```shell
104  pacman -U --noconfirm \
105    https://repo.msys2.org/mingw/ucrt64/mingw-w64-ucrt-x86_64-gcc-13.2.0-6-any.pkg.tar.zst \
106    https://repo.msys2.org/mingw/ucrt64/mingw-w64-ucrt-x86_64-gcc-libs-13.2.0-6-any.pkg.tar.zst
107  ```
108  
109  <!-- #### Headless Windows container -->
110  <!-- add instructions re: getting setup with MSYS2 in a Windows container -->
111  <!-- https://github.com/StefanScherer/windows-docker-machine -->
112  
113  #### Optional: VSCode Terminal integration
114  
115  You can link the MSYS2-UCRT64 terminal into VSCode by modifying the configuration file as shown below.
116  File: `C:/Users/<username>/AppData/Roaming/Code/User/settings.json`
117  ```json
118  {
119      ...
120      "terminal.integrated.profiles.windows": {
121        ...
122        "MSYS2-UCRT64": {
123          "path": "C:\\msys64\\usr\\bin\\bash.exe",
124          "args": [
125            "--login",
126            "-i"
127          ],
128          "env": {
129            "MSYSTEM": "UCRT64",
130            "CHERE_INVOKING": "1",
131            "MSYS2_PATH_TYPE": "inherit"
132          }
133        }
134      }
135  }
136  ```
137  
138  ### Other
139  
140  It is possible that nim-codex can be built and run on other platforms supported by the [Nim](https://nim-lang.org/) language: BSD family, older versions of Windows, etc. There has not been sufficient experimentation with nim-codex on such platforms, so instructions are not provided. Community contributions to these docs and our build system are welcome!
141  
142  ## Repository
143  
144  In Bash run
145  ```shell
146  git clone https://github.com/codex-storage/nim-codex.git repos/nim-codex && cd repos/nim-codex
147  ```
148  
149  nim-codex uses the [nimbus-build-system](https://github.com/status-im/nimbus-build-system), so next run
150  ```shell
151  make update
152  ```
153  
154  This step can take a while to complete because by default it builds the [Nim compiler](https://nim-lang.org/docs/nimc.html).
155  
156  To see more output from `make` pass `V=1`. This works for all `make` targets in projects using the nimbus-build-system
157  ```shell
158  make V=1 update
159  ```
160  
161  ## Executable
162  
163  In Bash run
164  ```shell
165  make
166  ```
167  
168  The default `make` target creates the `build/codex` executable.
169  
170  ## Tools
171  
172  ### Circuit download tool
173  
174  To build the circuit download tool located in `tools/cirdl` run:
175  
176  ```shell
177  make cirdl
178  ```
179  
180  ## Example usage
181  
182  See the instructions in the [Quick Start](/learn/quick-start).
183  
184  ## Tests
185  
186  In Bash run
187  ```shell
188  make test
189  ```
190  
191  ### testAll
192  
193  #### Prerequisites
194  
195  To run the integration tests, an Ethereum test node is required. Follow these instructions to set it up.
196  
197  ##### Windows (do this before 'All platforms')
198  
199  1. Download and install Visual Studio 2017 or newer. (Not VSCode!) In the Workloads overview, enable `Desktop development with C++`. ( https://visualstudio.microsoft.com )
200  
201  ##### All platforms
202  
203  1. Install NodeJS (tested with v18.14.0), consider using NVM as a version manager. [Node Version Manager (`nvm`)](https://github.com/nvm-sh/nvm#readme)
204  1. Open a terminal
205  1. Go to the vendor/codex-contracts-eth folder: `cd /<git-root>/vendor/codex-contracts-eth/`
206  1. `npm install` -> Should complete with the number of packages added and an overview of known vulnerabilities.
207  1. `npm test` -> Should output test results. May take a minute.
208  
209  Before the integration tests are started, you must start the Ethereum test node manually.
210  1. Open a terminal
211  1. Go to the vendor/codex-contracts-eth folder: `cd /<git-root>/vendor/codex-contracts-eth/`
212  1. `npm start` -> This should launch Hardhat, and output a number of keys and a warning message.
213  
214  #### Run
215  
216  The `testAll` target runs the same tests as `make test` and also runs tests for nim-codex's Ethereum contracts, as well a basic suite of integration tests.
217  
218  To run `make testAll`.
219  
220  Use a new terminal to run:
221  ```shell
222  make testAll
223  ```
224  
225  ## Building Within Docker
226  
227  For the specific case of Linux distributions which ship with gcc 14
228  and a downgrade to 13 is not possible/desirable, building within a Docker
229  container and pulling the binaries out by copying or mounting remains an
230  option; e.g.:
231  
232  ```bash=
233  # Clone original repo.
234  git clone https://github.com/codex-storage/nim-codex
235  
236  # Build inside docker
237  docker build -t codexstorage/nim-codex:latest -f nim-codex/docker/codex.Dockerfile nim-codex
238  
239  # Extract executable
240  docker create --name=codex-build codexstorage/nim-codex:latest
241  docker cp codex-build:/usr/local/bin/codex ./codex
242  docker cp codex-build:/usr/local/bin/cirdl ./cirdl
243  ```
244  
245  and voilà, you should have the binaries available in the current folder.