docker_s3_tutorial.ipynb
1 { 2 "cells": [ 3 { 4 "cell_type": "markdown", 5 "metadata": {}, 6 "source": [ 7 "# Using Evidently AI with a MinIO S3 Workspace\n", 8 "\n", 9 "This notebook is a complete guide to setting up and running the Evidently UI service with a remote S3-compatible workspace (AWS S3, Google Cloud Storage, MinIO, etc.). We will use **MinIO** as our self-hosted object storage solution, which is fully compatible with the S3 API.\n", 10 "\n", 11 "By the end of this tutorial, you will have:\n", 12 "1. A running MinIO instance managed by Docker.\n", 13 "2. A custom Evidently Docker image with the necessary S3 libraries.\n", 14 "3. The Evidently UI service running and connected to your MinIO workspace.\n", 15 "\n", 16 "This guide is structured into four parts:\n", 17 "- **Part 1:** Setting up MinIO with Docker Compose.\n", 18 "- **Part 2:** Building a Custom Evidently Docker Image with S3 Support.\n", 19 "- **Part 3:** Running the Evidently Service.\n", 20 "- **Bonus Tip:** Enhancing Security with an Unprivileged User.\n", 21 "\n", 22 "> This tutorial also works if you're using Google Cloud Storage directly, just ignore the MinIO parts." 23 ] 24 }, 25 { 26 "cell_type": "markdown", 27 "metadata": {}, 28 "source": [ 29 "--- \n", 30 "## Part 1: Setting Up MinIO with Docker Compose\n", 31 "\n", 32 "First, we need an S3-compatible object storage server. MinIO is a perfect choice for local development. The easiest way to run it is with `docker-compose`, which allows us to define and manage the service cleanly.\n", 33 "\n", 34 "> **Skippable Step:** If you already have an S3 bucket (on AWS S3, GCS, etc.), you can skip this part and go to Part 2 directly. Just make sure you have your Access Key, Secret Key, and the correct Endpoint URL ready." 35 ] 36 }, 37 { 38 "cell_type": "markdown", 39 "metadata": {}, 40 "source": [ 41 "#### **Step 1.1: Create a `docker-compose.yml` file**\n", 42 "Create a new file named `docker-compose.yml` and paste the following content into it. This configuration will start a MinIO server." 43 ] 44 }, 45 { 46 "cell_type": "markdown", 47 "metadata": {}, 48 "source": [ 49 "```yaml\n", 50 "# docker-compose.yml\n", 51 "version: '3.8'\n", 52 "\n", 53 "services:\n", 54 " minio:\n", 55 " image: minio/minio:latest\n", 56 " container_name: minio\n", 57 " command: server /data --console-address \":9001\"\n", 58 " ports:\n", 59 " - \"9000:9000\" # API Port\n", 60 " - \"9001:9001\" # Console Port\n", 61 " environment:\n", 62 " - MINIO_ROOT_USER=minioadmin\n", 63 " - MINIO_ROOT_PASSWORD=minioadmin\n", 64 " volumes:\n", 65 " - minio_data:/data\n", 66 "\n", 67 "volumes:\n", 68 " minio_data:\n", 69 " driver: local\n", 70 "```" 71 ] 72 }, 73 { 74 "cell_type": "markdown", 75 "metadata": {}, 76 "source": [ 77 "#### **Step 1.2: Start the MinIO Service**\n", 78 "\n", 79 "Open a terminal in the same directory where you saved the `docker-compose.yml` file and run:\n", 80 "\n", 81 "```bash\n", 82 "docker compose up -d\n", 83 "```\n", 84 "\n", 85 "This command will download the MinIO image and start the service in the background.\n", 86 "\n", 87 "> When you're finished with this tutorial, you can stop and remove the MinIO instance with the following command (ran from the same directory): `docker compose down`" 88 ] 89 }, 90 { 91 "cell_type": "markdown", 92 "metadata": {}, 93 "source": [ 94 "#### **Step 1.3: Create a Bucket**\n", 95 "\n", 96 "Now that MinIO is running, you need to create a \"bucket\" to store your Evidently projects.\n", 97 "\n", 98 "1. Open your web browser and navigate to **`http://localhost:9001`**.\n", 99 "2. Log in with the credentials you set in the `docker-compose.yml` file:\n", 100 " - **Username:** `minioadmin`\n", 101 " - **Password:** `minioadmin`\n", 102 "3. Click the **Create Bucket** button on the left sidebar.\n", 103 "4. Name your bucket `evidently-ai` and click **Create Bucket**." 104 ] 105 }, 106 { 107 "cell_type": "markdown", 108 "metadata": {}, 109 "source": [ 110 "#### **Step 1.4 (Optional but Recommended): Create a Dedicated Access Key**\n", 111 "\n", 112 "For better security, it's best to avoid using the root user (`minioadmin`) for applications. Let's create a new, dedicated user with its own Access Key and Secret Key.\n", 113 "\n", 114 "> You can skip this step for dev purposes and use the `minioadmin` username and password as Access Key and Secret Key later.\n", 115 "\n", 116 "1. **Enter the MinIO container's shell:**\n", 117 " ```bash\n", 118 " docker exec -it minio /bin/sh\n", 119 " ```\n", 120 "2. **Configure the MinIO Client (`mc`) alias inside the container:**\n", 121 " ```sh\n", 122 " # Format: mc alias set <ALIAS_NAME> <SERVER_URL> <ROOT_USER> <ROOT_PASSWORD>\n", 123 " mc alias set local http://localhost:9000 minioadmin minioadmin\n", 124 " ```\n", 125 "3. **Create the new user (this defines your new Access Key and Secret Key):**\n", 126 " ```sh\n", 127 " # Format: mc admin user add <ALIAS> <NEW_ACCESS_KEY> <NEW_SECRET_KEY>\n", 128 " mc admin user add local evidently-user a-very-strong-secret-key\n", 129 " ```\n", 130 "4. **Assign permissions to the new user:**\n", 131 " ```sh\n", 132 " # Format: mc admin policy attach <ALIAS> <POLICY> --user <USER_ACCESS_KEY>\n", 133 " mc admin policy attach local readwrite --user evidently-user\n", 134 " ```\n", 135 "5. **Exit the container:**\n", 136 " ```sh\n", 137 " exit\n", 138 " ```\n", 139 "Now you have a new set of credentials! **Remember to use these in Part 3 instead of the `minioadmin` user.**" 140 ] 141 }, 142 { 143 "cell_type": "markdown", 144 "metadata": {}, 145 "source": [ 146 "--- \n", 147 "## Part 2: Building a Custom Evidently Docker Image\n", 148 "\n", 149 "The official `evidently/evidently-service` image is minimal and does not include the `s3fs` or `gcsfs` library needed to communicate with remote data storage backends. We need to build a custom image that adds this dependency.\n", 150 "\n", 151 "Here are two ways to do it." 152 ] 153 }, 154 { 155 "cell_type": "markdown", 156 "metadata": {}, 157 "source": [ 158 "### Method A: Build on Top of the Official Image (Recommended for Simplicity)\n", 159 "This is the easiest method. We create a new `Dockerfile` that uses the official image as a base and simply installs the required packages.\n", 160 "\n", 161 "#### **Step 2.1.A: Create a Dockerfile**\n", 162 "Create a file named `Dockerfile.s3` and add the following content:" 163 ] 164 }, 165 { 166 "cell_type": "markdown", 167 "metadata": {}, 168 "source": [ 169 "```Dockerfile\n", 170 "FROM evidently/evidently-service:latest\n", 171 "\n", 172 "# Install the necessary libraries for S3 (s3fs) and GCS (gcsfs) support.\n", 173 "# You can remove the one you don't need.\n", 174 "RUN pip install s3fs gcsfs\n", 175 "```" 176 ] 177 }, 178 { 179 "cell_type": "markdown", 180 "metadata": {}, 181 "source": [ 182 "#### **Step 2.2.A: Build the Image**\n", 183 "In your terminal, run the following command to build your custom image. We'll tag it as `evidently/evidently-service:s3`.\n", 184 "\n", 185 "```bash\n", 186 "docker build -t evidently/evidently-service:s3 -f Dockerfile.s3 .\n", 187 "```" 188 ] 189 }, 190 { 191 "cell_type": "markdown", 192 "metadata": {}, 193 "source": [ 194 "### Method B: Build from the Evidently Source Code\n", 195 "This method is for users who want to build directly from the official Evidently GitHub repository.\n", 196 "\n", 197 "#### **Step 2.1.B: Clone the Repository**\n", 198 "```bash\n", 199 "git clone [https://github.com/evidentlyai/evidently.git](https://github.com/evidentlyai/evidently.git)\n", 200 "cd evidently\n", 201 "```\n", 202 "\n", 203 "#### **Step 2.2.B: Build the Image with a Build Argument**\n", 204 "You can build the image by passing the `[fsspec]` extra to install all filesystem dependencies. It's also possible to install just `[s3]` or `[gcs]`.\n", 205 "\n", 206 "```bash\n", 207 "# This command should be run from the root of the cloned 'evidently' repository\n", 208 "docker build \\\n", 209 " --build-arg INSTALL_EXTRAS=\"[fsspec]\" \\\n", 210 " -f docker/Dockerfile.service \\\n", 211 " -t evidently/evidently-service:s3 .\n", 212 "```" 213 ] 214 }, 215 { 216 "cell_type": "markdown", 217 "metadata": {}, 218 "source": [ 219 "--- \n", 220 "## Part 3: Running the Custom Evidently Service\n", 221 "\n", 222 "Now we have our MinIO instance running and our custom Docker image ready. Let's run the Evidently container and connect everything.\n", 223 "\n", 224 "The command below will:\n", 225 "- Run our custom image (`evidently/evidently-service:s3`).\n", 226 "- Pass the MinIO credentials and endpoint URL as environment variables.\n", 227 "- Tell the Evidently UI to use our MinIO bucket as its workspace.\n", 228 "\n", 229 "> For GCS, the workspace should be `gs://evidently-ai/workspace`, *assuming your bucket is also named **evidently-ai***.\n", 230 "\n", 231 "Execute the following command in your terminal. **Remember to use the dedicated credentials you created in Step 1.4 if you completed that step.** If you didn't, don't worry! You can use the \"minioadmin\"." 232 ] 233 }, 234 { 235 "cell_type": "code", 236 "execution_count": null, 237 "metadata": { 238 "vscode": { 239 "languageId": "shellscript" 240 } 241 }, 242 "outputs": [], 243 "source": [ 244 "docker run -d -p 8000:8000 \\\n", 245 " -e FSSPEC_S3_KEY=\"minioadmin\" \\\n", 246 " -e FSSPEC_S3_SECRET=\"minioadmin\" \\\n", 247 " -e FSSPEC_S3_ENDPOINT_URL=\"http://host.docker.internal:9000\" \\\n", 248 " --name evidently-ui \\\n", 249 " evidently/evidently-service:s3 --workspace s3://evidently-ai/workspace" 250 ] 251 }, 252 { 253 "cell_type": "markdown", 254 "metadata": {}, 255 "source": [ 256 "That works because the default `ENTRYPOINT` already starts the Evidently UI, we're just appending the workspace to it.\n", 257 "\n", 258 "> **Important Note on `ENDPOINT_URL`:**\n", 259 "> - **Never** include a trailing slash (`/`) in the endpoint URL.\n", 260 "\n", 261 "#### **Verify the Setup**\n", 262 "\n", 263 "Open your browser and go to [http://localhost:8000](http://localhost:8000). You should now see the Evidently UI. You can create a new project, and it will be saved directly into your MinIO bucket! Check it from the [MinIO Console](http://localhost:9001)." 264 ] 265 }, 266 { 267 "cell_type": "markdown", 268 "metadata": {}, 269 "source": [ 270 "--- \n", 271 "## Bonus Tip: Enhancing Security with an Unprivileged User\n", 272 "\n", 273 "Running Docker containers as the `root` user is a security risk. It's a best practice to create and switch to a non-root user inside your image. Here’s how you can modify the simple Dockerfile from **Method A** to add an unprivileged user.\n", 274 "\n", 275 "#### **Create a `Dockerfile.s3.secure` file:**" 276 ] 277 }, 278 { 279 "cell_type": "markdown", 280 "metadata": {}, 281 "source": [ 282 "```Dockerfile\n", 283 "FROM evidently/evidently-service:latest\n", 284 "\n", 285 "# `s3fs` for S3-compatible storages\n", 286 "# `gcsfs` if you're using Google Cloud Storage \n", 287 "RUN pip install s3fs gcsfs\n", 288 "\n", 289 "# The `adduser` command is a user-friendly utility available in Debian-based images.\n", 290 "RUN adduser --disabled-password --uid 1000 evidentlyuser\n", 291 "\n", 292 "USER evidentlyuser\n", 293 "```" 294 ] 295 }, 296 { 297 "cell_type": "markdown", 298 "metadata": {}, 299 "source": [ 300 "You would then build and run this image just as before. This simple change significantly improves the security posture of your container." 301 ] 302 } 303 ], 304 "metadata": { 305 "kernelspec": { 306 "display_name": "Python 3", 307 "language": "python", 308 "name": "python3" 309 }, 310 "language_info": { 311 "codemirror_mode": { 312 "name": "ipython", 313 "version": 3 314 }, 315 "file_extension": ".py", 316 "mimetype": "text/x-python", 317 "name": "python", 318 "nbconvert_exporter": "python", 319 "pygments_lexer": "ipython3", 320 "version": "3.8.8" 321 } 322 }, 323 "nbformat": 4, 324 "nbformat_minor": 4 325 }