retriever-evaluation-tutorial.ipynb
1 { 2 "cells": [ 3 { 4 "cell_type": "markdown", 5 "metadata": { 6 "application/vnd.databricks.v1+cell": { 7 "cellMetadata": {}, 8 "inputWidgets": {}, 9 "nuid": "f8938de9-7fae-41cd-ad6b-7ee26c288eab", 10 "showTitle": false, 11 "title": "" 12 } 13 }, 14 "source": [ 15 "# Retriever Evaluation Tutorial\n", 16 "\n", 17 "In MLflow 2.8.0, we introduced a new model type \"retriever\" to the `mlflow.evaluate()` API. It helps you to evaluate the retriever in a RAG application. It contains two built-in metrics `precision_at_k` and `recall_at_k`. In MLflow 2.9.0, `ndcg_at_k` is available.\n", 18 "\n", 19 "This notebook illustrates how to use `mlflow.evaluate()` to evaluate the retriever in a RAG application. It has the following steps:\n", 20 "\n", 21 "* Step 1: Install and Load Packages\n", 22 "* Step 2: Evaluation Dataset Preparation\n", 23 "* Step 3: Calling `mlflow.evaluate()`\n", 24 "* Step 4: Result Analysis and Visualization" 25 ] 26 }, 27 { 28 "cell_type": "markdown", 29 "metadata": {}, 30 "source": [ 31 "## Step 1: Install and Load Packages" 32 ] 33 }, 34 { 35 "cell_type": "code", 36 "execution_count": null, 37 "metadata": { 38 "application/vnd.databricks.v1+cell": { 39 "cellMetadata": { 40 "byteLimit": 2048000, 41 "rowLimit": 10000 42 }, 43 "inputWidgets": {}, 44 "nuid": "5bf12edb-2498-4edd-aeff-b4844451850f", 45 "showTitle": false, 46 "title": "" 47 } 48 }, 49 "outputs": [], 50 "source": [ 51 "%pip install mlflow==2.9.0 langchain==0.0.339 openai faiss-cpu gensim nltk pyLDAvis tiktoken" 52 ] 53 }, 54 { 55 "cell_type": "code", 56 "execution_count": 1, 57 "metadata": { 58 "application/vnd.databricks.v1+cell": { 59 "cellMetadata": { 60 "byteLimit": 2048000, 61 "rowLimit": 10000 62 }, 63 "inputWidgets": {}, 64 "nuid": "414eb948-7f7a-411b-8308-facadb0bdde8", 65 "showTitle": false, 66 "title": "" 67 } 68 }, 69 "outputs": [], 70 "source": [ 71 "import ast\n", 72 "import os\n", 73 "import pprint\n", 74 "\n", 75 "import pandas as pd\n", 76 "from langchain.docstore.document import Document\n", 77 "from langchain.embeddings.openai import OpenAIEmbeddings\n", 78 "from langchain.text_splitter import CharacterTextSplitter\n", 79 "from langchain.vectorstores import FAISS\n", 80 "\n", 81 "import mlflow\n", 82 "\n", 83 "os.environ[\"OPENAI_API_KEY\"] = \"<redacted>\"\n", 84 "\n", 85 "CHUNK_SIZE = 1000\n", 86 "\n", 87 "# Assume running from https://github.com/mlflow/mlflow/blob/master/examples/llms/rag\n", 88 "OUTPUT_DF_PATH = \"question_answer_source.csv\"\n", 89 "SCRAPPED_DOCS_PATH = \"mlflow_docs_scraped.csv\"\n", 90 "EVALUATION_DATASET_PATH = \"static_evaluation_dataset.csv\"\n", 91 "DB_PERSIST_DIR = \"faiss_index\"" 92 ] 93 }, 94 { 95 "cell_type": "markdown", 96 "metadata": { 97 "application/vnd.databricks.v1+cell": { 98 "cellMetadata": {}, 99 "inputWidgets": {}, 100 "nuid": "eebcf0d9-6634-47d9-808d-e79c5a50fbbf", 101 "showTitle": false, 102 "title": "" 103 } 104 }, 105 "source": [ 106 "## Step 2: Evaluation Dataset Preparation\n", 107 "The evaluation dataset should contain three columns: questions, ground truth doc IDs, retrieved relevant doc IDs. A \"doc ID\" is a unique string identifier of the documents in you RAG application. For example, it could be the URL of a documentation web page, or the file path of a PDF document.\n", 108 "\n", 109 "If you have a list of questions that you would like to evaluate, please see 1.1 Manual Preparation. If you do not have a question list yet, please see 1.2 Generate the Evaluation Dataset.\n" 110 ] 111 }, 112 { 113 "cell_type": "markdown", 114 "metadata": { 115 "application/vnd.databricks.v1+cell": { 116 "cellMetadata": {}, 117 "inputWidgets": {}, 118 "nuid": "f8a690cc-7672-4f24-8518-8faabfc9afea", 119 "showTitle": false, 120 "title": "" 121 } 122 }, 123 "source": [ 124 "### Manual Preparation\n", 125 "\n", 126 "When evaluating a retriever, it's recommended to save the retrieved document IDs into a static dataset represented by a Pandas Dataframe or an MLflow Pandas Dataset containing the input queries, retrieved relevant document IDs, and the ground-truth document IDs for the evaluation.\n", 127 "\n", 128 "#### Concepts\n", 129 "\n", 130 "A \"document ID\" is a string that identifies a document.\n", 131 "\n", 132 "A list of \"retrieved relevant document IDs\" are the output of the retriever for a specific input query and a `k` value.\n", 133 "\n", 134 "A list of \"ground-truth document IDs\" are the labeled relevant documents for a specific input query.\n", 135 "\n", 136 "#### Expected Data Format\n", 137 "\n", 138 "For each row, the retrieved relevant document IDs and the ground-truth relevant document IDs should be provided as a tuple of document ID strings.\n", 139 "\n", 140 "The column name of the retrieved relevant document IDs should be specified by the `predictions` parameter, and the column name of the ground-truth relevant document IDs should be specified by the `targets` parameter.\n", 141 "\n", 142 "Here is a simple example dataset that illustrates the expected data format. The doc IDs are the paths of the documentation pages." 143 ] 144 }, 145 { 146 "cell_type": "code", 147 "execution_count": null, 148 "metadata": { 149 "application/vnd.databricks.v1+cell": { 150 "cellMetadata": { 151 "byteLimit": 2048000, 152 "rowLimit": 10000 153 }, 154 "inputWidgets": {}, 155 "nuid": "1a61b1b2-582e-49d5-864d-b58d2b6c3392", 156 "showTitle": false, 157 "title": "" 158 } 159 }, 160 "outputs": [], 161 "source": [ 162 "data = pd.DataFrame({\n", 163 " \"questions\": [\n", 164 " \"What is MLflow?\",\n", 165 " \"What is Databricks?\",\n", 166 " \"How to serve a model on Databricks?\",\n", 167 " \"How to enable MLflow Autologging for my workspace by default?\",\n", 168 " ],\n", 169 " \"retrieved_context\": [\n", 170 " [\n", 171 " \"mlflow/index.html\",\n", 172 " \"mlflow/quick-start.html\",\n", 173 " ],\n", 174 " [\n", 175 " \"introduction/index.html\",\n", 176 " \"getting-started/overview.html\",\n", 177 " ],\n", 178 " [\n", 179 " \"machine-learning/model-serving/index.html\",\n", 180 " \"machine-learning/model-serving/model-serving-intro.html\",\n", 181 " ],\n", 182 " [],\n", 183 " ],\n", 184 " \"ground_truth_context\": [\n", 185 " [\"mlflow/index.html\"],\n", 186 " [\"introduction/index.html\"],\n", 187 " [\n", 188 " \"machine-learning/model-serving/index.html\",\n", 189 " \"machine-learning/model-serving/llm-optimized-model-serving.html\",\n", 190 " ],\n", 191 " [\"mlflow/databricks-autologging.html\"],\n", 192 " ],\n", 193 "})" 194 ] 195 }, 196 { 197 "cell_type": "markdown", 198 "metadata": { 199 "application/vnd.databricks.v1+cell": { 200 "cellMetadata": {}, 201 "inputWidgets": {}, 202 "nuid": "f740b47c-71ee-4633-944c-172887ff5081", 203 "showTitle": false, 204 "title": "" 205 } 206 }, 207 "source": [ 208 "### Generate the Evaluation Dataset\n", 209 "There are two steps to generate the evaluation dataset: generate questions with ground truth doc IDs and retrieve relevant doc IDs. " 210 ] 211 }, 212 { 213 "cell_type": "markdown", 214 "metadata": { 215 "application/vnd.databricks.v1+cell": { 216 "cellMetadata": {}, 217 "inputWidgets": {}, 218 "nuid": "f6beddae-85e2-44e7-8ec6-7ca2f02bc16b", 219 "showTitle": false, 220 "title": "" 221 } 222 }, 223 "source": [ 224 "\n", 225 "#### Generate Questions with Ground Truth Doc IDs\n", 226 "If you don't have a list of questions to evaluate, you can generate them using LLMs. The [Question Generation Notebook](https://mlflow.org/docs/latest/llms/rag/notebooks/question-generation-retrieval-evaluation.html) provides an example way to do it. Here is the result of running that notebook." 227 ] 228 }, 229 { 230 "cell_type": "code", 231 "execution_count": 5, 232 "metadata": { 233 "application/vnd.databricks.v1+cell": { 234 "cellMetadata": { 235 "byteLimit": 2048000, 236 "rowLimit": 10000 237 }, 238 "inputWidgets": {}, 239 "nuid": "98bf55c7-3e58-4fff-bc0e-1af58d64839f", 240 "showTitle": false, 241 "title": "" 242 } 243 }, 244 "outputs": [], 245 "source": [ 246 "generated_df = pd.read_csv(OUTPUT_DF_PATH)" 247 ] 248 }, 249 { 250 "cell_type": "code", 251 "execution_count": 7, 252 "metadata": { 253 "application/vnd.databricks.v1+cell": { 254 "cellMetadata": { 255 "byteLimit": 2048000, 256 "rowLimit": 10000 257 }, 258 "inputWidgets": {}, 259 "nuid": "17baa097-457f-46df-9e25-56061972785f", 260 "showTitle": false, 261 "title": "" 262 } 263 }, 264 "outputs": [ 265 { 266 "data": { 267 "text/html": [ 268 "<div>\n", 269 "<style scoped>\n", 270 " .dataframe tbody tr th:only-of-type {\n", 271 " vertical-align: middle;\n", 272 " }\n", 273 "\n", 274 " .dataframe tbody tr th {\n", 275 " vertical-align: top;\n", 276 " }\n", 277 "\n", 278 " .dataframe thead th {\n", 279 " text-align: right;\n", 280 " }\n", 281 "</style>\n", 282 "<table border=\"1\" class=\"dataframe\">\n", 283 " <thead>\n", 284 " <tr style=\"text-align: right;\">\n", 285 " <th></th>\n", 286 " <th>question</th>\n", 287 " <th>answer</th>\n", 288 " <th>chunk</th>\n", 289 " <th>chunk_id</th>\n", 290 " <th>source</th>\n", 291 " </tr>\n", 292 " </thead>\n", 293 " <tbody>\n", 294 " <tr>\n", 295 " <th>0</th>\n", 296 " <td>What is the purpose of the MLflow Model Registry?</td>\n", 297 " <td>The purpose of the MLflow Model Registry is to...</td>\n", 298 " <td>Documentation MLflow Model Registry MLflow Mod...</td>\n", 299 " <td>0</td>\n", 300 " <td>model-registry.html</td>\n", 301 " </tr>\n", 302 " <tr>\n", 303 " <th>1</th>\n", 304 " <td>What is the purpose of registering a model wit...</td>\n", 305 " <td>The purpose of registering a model with the Mo...</td>\n", 306 " <td>logged, this model can then be registered with...</td>\n", 307 " <td>1</td>\n", 308 " <td>model-registry.html</td>\n", 309 " </tr>\n", 310 " <tr>\n", 311 " <th>2</th>\n", 312 " <td>What can you do with registered models and mod...</td>\n", 313 " <td>With registered models and model versions, you...</td>\n", 314 " <td>associate with registered models and model ver...</td>\n", 315 " <td>2</td>\n", 316 " <td>model-registry.html</td>\n", 317 " </tr>\n", 318 " </tbody>\n", 319 "</table>\n", 320 "</div>" 321 ], 322 "text/plain": [ 323 " question \\\n", 324 "0 What is the purpose of the MLflow Model Registry? \n", 325 "1 What is the purpose of registering a model wit... \n", 326 "2 What can you do with registered models and mod... \n", 327 "\n", 328 " answer \\\n", 329 "0 The purpose of the MLflow Model Registry is to... \n", 330 "1 The purpose of registering a model with the Mo... \n", 331 "2 With registered models and model versions, you... \n", 332 "\n", 333 " chunk chunk_id \\\n", 334 "0 Documentation MLflow Model Registry MLflow Mod... 0 \n", 335 "1 logged, this model can then be registered with... 1 \n", 336 "2 associate with registered models and model ver... 2 \n", 337 "\n", 338 " source \n", 339 "0 model-registry.html \n", 340 "1 model-registry.html \n", 341 "2 model-registry.html " 342 ] 343 }, 344 "execution_count": 7, 345 "metadata": {}, 346 "output_type": "execute_result" 347 } 348 ], 349 "source": [ 350 "generated_df.head(3)" 351 ] 352 }, 353 { 354 "cell_type": "code", 355 "execution_count": 8, 356 "metadata": { 357 "application/vnd.databricks.v1+cell": { 358 "cellMetadata": { 359 "byteLimit": 2048000, 360 "rowLimit": 10000 361 }, 362 "inputWidgets": {}, 363 "nuid": "93165dc5-aff9-46f9-83ab-e6dbfcbbc32b", 364 "showTitle": false, 365 "title": "" 366 } 367 }, 368 "outputs": [ 369 { 370 "data": { 371 "text/html": [ 372 "<div>\n", 373 "<style scoped>\n", 374 " .dataframe tbody tr th:only-of-type {\n", 375 " vertical-align: middle;\n", 376 " }\n", 377 "\n", 378 " .dataframe tbody tr th {\n", 379 " vertical-align: top;\n", 380 " }\n", 381 "\n", 382 " .dataframe thead th {\n", 383 " text-align: right;\n", 384 " }\n", 385 "</style>\n", 386 "<table border=\"1\" class=\"dataframe\">\n", 387 " <thead>\n", 388 " <tr style=\"text-align: right;\">\n", 389 " <th></th>\n", 390 " <th>question</th>\n", 391 " <th>source</th>\n", 392 " </tr>\n", 393 " </thead>\n", 394 " <tbody>\n", 395 " <tr>\n", 396 " <th>0</th>\n", 397 " <td>What is the purpose of the MLflow Model Registry?</td>\n", 398 " <td>[model-registry.html]</td>\n", 399 " </tr>\n", 400 " <tr>\n", 401 " <th>1</th>\n", 402 " <td>What is the purpose of registering a model wit...</td>\n", 403 " <td>[model-registry.html]</td>\n", 404 " </tr>\n", 405 " <tr>\n", 406 " <th>2</th>\n", 407 " <td>What can you do with registered models and mod...</td>\n", 408 " <td>[model-registry.html]</td>\n", 409 " </tr>\n", 410 " </tbody>\n", 411 "</table>\n", 412 "</div>" 413 ], 414 "text/plain": [ 415 " question source\n", 416 "0 What is the purpose of the MLflow Model Registry? [model-registry.html]\n", 417 "1 What is the purpose of registering a model wit... [model-registry.html]\n", 418 "2 What can you do with registered models and mod... [model-registry.html]" 419 ] 420 }, 421 "execution_count": 8, 422 "metadata": {}, 423 "output_type": "execute_result" 424 } 425 ], 426 "source": [ 427 "# Prepare dataframe `data` with the required format\n", 428 "data = pd.DataFrame({})\n", 429 "data[\"question\"] = generated_df[\"question\"].copy(deep=True)\n", 430 "data[\"source\"] = generated_df[\"source\"].apply(lambda x: [x])\n", 431 "data.head(3)" 432 ] 433 }, 434 { 435 "cell_type": "markdown", 436 "metadata": { 437 "application/vnd.databricks.v1+cell": { 438 "cellMetadata": {}, 439 "inputWidgets": {}, 440 "nuid": "3eabe651-28be-45bb-94ad-58e6bc582137", 441 "showTitle": false, 442 "title": "" 443 } 444 }, 445 "source": [ 446 "#### Retrieve Relevant Doc IDs\n", 447 "\n", 448 "Once we have a list of questions with ground truth doc IDs from 1.1, we can collect the retrieved relevant doc IDs. In this tutorial, we use a LangChain retriever. You can plug in your own retriever as needed." 449 ] 450 }, 451 { 452 "cell_type": "markdown", 453 "metadata": { 454 "application/vnd.databricks.v1+cell": { 455 "cellMetadata": {}, 456 "inputWidgets": {}, 457 "nuid": "9817f671-f2fd-4b2e-abe9-3bc9afd9ce3c", 458 "showTitle": false, 459 "title": "" 460 } 461 }, 462 "source": [ 463 "First, we build a FAISS retriever from the docs saved at https://github.com/mlflow/mlflow/blob/master/examples/llms/question_generation/mlflow_docs_scraped.csv. See the [Question Generation Notebook](https://mlflow.org/docs/latest/llms/rag/notebooks/question-generation-retrieval-evaluation.html) for how to create this csv file." 464 ] 465 }, 466 { 467 "cell_type": "code", 468 "execution_count": 10, 469 "metadata": { 470 "application/vnd.databricks.v1+cell": { 471 "cellMetadata": { 472 "byteLimit": 2048000, 473 "rowLimit": 10000 474 }, 475 "inputWidgets": {}, 476 "nuid": "178b45b4-11f9-47ca-9564-c8caa32d2504", 477 "showTitle": false, 478 "title": "" 479 } 480 }, 481 "outputs": [], 482 "source": [ 483 "embeddings = OpenAIEmbeddings()" 484 ] 485 }, 486 { 487 "cell_type": "code", 488 "execution_count": null, 489 "metadata": { 490 "application/vnd.databricks.v1+cell": { 491 "cellMetadata": { 492 "byteLimit": 2048000, 493 "rowLimit": 10000 494 }, 495 "inputWidgets": {}, 496 "nuid": "e5a113bb-11b8-4d1a-a21b-b59b523f3525", 497 "showTitle": false, 498 "title": "" 499 } 500 }, 501 "outputs": [], 502 "source": [ 503 "scrapped_df = pd.read_csv(SCRAPPED_DOCS_PATH)\n", 504 "list_of_documents = [\n", 505 " Document(page_content=row[\"text\"], metadata={\"source\": row[\"source\"]})\n", 506 " for i, row in scrapped_df.iterrows()\n", 507 "]\n", 508 "text_splitter = CharacterTextSplitter(chunk_size=CHUNK_SIZE, chunk_overlap=0)\n", 509 "docs = text_splitter.split_documents(list_of_documents)\n", 510 "db = FAISS.from_documents(docs, embeddings)\n", 511 "\n", 512 "# Save the db to local disk\n", 513 "db.save_local(DB_PERSIST_DIR)" 514 ] 515 }, 516 { 517 "cell_type": "code", 518 "execution_count": 11, 519 "metadata": { 520 "application/vnd.databricks.v1+cell": { 521 "cellMetadata": { 522 "byteLimit": 2048000, 523 "rowLimit": 10000 524 }, 525 "inputWidgets": {}, 526 "nuid": "bace7c63-e3d5-42f3-bf6a-00ef1842baae", 527 "showTitle": false, 528 "title": "" 529 } 530 }, 531 "outputs": [], 532 "source": [ 533 "# Load the db from local disk\n", 534 "db = FAISS.load_local(DB_PERSIST_DIR, embeddings)\n", 535 "retriever = db.as_retriever()" 536 ] 537 }, 538 { 539 "cell_type": "code", 540 "execution_count": 13, 541 "metadata": { 542 "application/vnd.databricks.v1+cell": { 543 "cellMetadata": { 544 "byteLimit": 2048000, 545 "rowLimit": 10000 546 }, 547 "inputWidgets": {}, 548 "nuid": "c06bcb3c-58c8-454c-bf5b-e29ec227991f", 549 "showTitle": false, 550 "title": "" 551 } 552 }, 553 "outputs": [ 554 { 555 "data": { 556 "text/plain": [ 557 "4" 558 ] 559 }, 560 "execution_count": 13, 561 "metadata": {}, 562 "output_type": "execute_result" 563 } 564 ], 565 "source": [ 566 "# Test the retriever with a query\n", 567 "retrieved_docs = retriever.get_relevant_documents(\n", 568 " \"What is the purpose of the MLflow Model Registry?\"\n", 569 ")\n", 570 "len(retrieved_docs)" 571 ] 572 }, 573 { 574 "cell_type": "markdown", 575 "metadata": { 576 "application/vnd.databricks.v1+cell": { 577 "cellMetadata": {}, 578 "inputWidgets": {}, 579 "nuid": "2ec7b458-c248-4ec0-9d85-0e447d6b4ecd", 580 "showTitle": false, 581 "title": "" 582 } 583 }, 584 "source": [ 585 "After building a retriever, we define a function that takes a question string as input and returns a list of relevant doc ID strings." 586 ] 587 }, 588 { 589 "cell_type": "code", 590 "execution_count": 14, 591 "metadata": { 592 "application/vnd.databricks.v1+cell": { 593 "cellMetadata": { 594 "byteLimit": 2048000, 595 "rowLimit": 10000 596 }, 597 "inputWidgets": {}, 598 "nuid": "bc688e4b-3389-4804-b7bf-159bce4f9db8", 599 "showTitle": false, 600 "title": "" 601 } 602 }, 603 "outputs": [], 604 "source": [ 605 "# Define a function to return a list of retrieved doc ids\n", 606 "def retrieve_doc_ids(question: str) -> list[str]:\n", 607 " docs = retriever.get_relevant_documents(question)\n", 608 " return [doc.metadata[\"source\"] for doc in docs]" 609 ] 610 }, 611 { 612 "cell_type": "markdown", 613 "metadata": { 614 "application/vnd.databricks.v1+cell": { 615 "cellMetadata": {}, 616 "inputWidgets": {}, 617 "nuid": "330a3336-6ca2-455f-a1ae-5bd842a4d2bb", 618 "showTitle": false, 619 "title": "" 620 } 621 }, 622 "source": [ 623 "We can store the retrieved doc IDs in the dataframe as a column \"retrieved_doc_ids\"." 624 ] 625 }, 626 { 627 "cell_type": "code", 628 "execution_count": 17, 629 "metadata": { 630 "application/vnd.databricks.v1+cell": { 631 "cellMetadata": { 632 "byteLimit": 2048000, 633 "rowLimit": 10000 634 }, 635 "inputWidgets": {}, 636 "nuid": "f96ec69b-bea3-4023-8cd3-6bee1e327ff0", 637 "showTitle": false, 638 "title": "" 639 } 640 }, 641 "outputs": [ 642 { 643 "data": { 644 "text/html": [ 645 "<div>\n", 646 "<style scoped>\n", 647 " .dataframe tbody tr th:only-of-type {\n", 648 " vertical-align: middle;\n", 649 " }\n", 650 "\n", 651 " .dataframe tbody tr th {\n", 652 " vertical-align: top;\n", 653 " }\n", 654 "\n", 655 " .dataframe thead th {\n", 656 " text-align: right;\n", 657 " }\n", 658 "</style>\n", 659 "<table border=\"1\" class=\"dataframe\">\n", 660 " <thead>\n", 661 " <tr style=\"text-align: right;\">\n", 662 " <th></th>\n", 663 " <th>question</th>\n", 664 " <th>source</th>\n", 665 " <th>retrieved_doc_ids</th>\n", 666 " </tr>\n", 667 " </thead>\n", 668 " <tbody>\n", 669 " <tr>\n", 670 " <th>0</th>\n", 671 " <td>What is the purpose of the MLflow Model Registry?</td>\n", 672 " <td>[model-registry.html]</td>\n", 673 " <td>[model-registry.html, introduction/index.html,...</td>\n", 674 " </tr>\n", 675 " <tr>\n", 676 " <th>1</th>\n", 677 " <td>What is the purpose of registering a model wit...</td>\n", 678 " <td>[model-registry.html]</td>\n", 679 " <td>[model-registry.html, models.html, introductio...</td>\n", 680 " </tr>\n", 681 " <tr>\n", 682 " <th>2</th>\n", 683 " <td>What can you do with registered models and mod...</td>\n", 684 " <td>[model-registry.html]</td>\n", 685 " <td>[model-registry.html, models.html, deployment/...</td>\n", 686 " </tr>\n", 687 " </tbody>\n", 688 "</table>\n", 689 "</div>" 690 ], 691 "text/plain": [ 692 " question source \\\n", 693 "0 What is the purpose of the MLflow Model Registry? [model-registry.html] \n", 694 "1 What is the purpose of registering a model wit... [model-registry.html] \n", 695 "2 What can you do with registered models and mod... [model-registry.html] \n", 696 "\n", 697 " retrieved_doc_ids \n", 698 "0 [model-registry.html, introduction/index.html,... \n", 699 "1 [model-registry.html, models.html, introductio... \n", 700 "2 [model-registry.html, models.html, deployment/... " 701 ] 702 }, 703 "execution_count": 17, 704 "metadata": {}, 705 "output_type": "execute_result" 706 } 707 ], 708 "source": [ 709 "data[\"retrieved_doc_ids\"] = data[\"question\"].apply(retrieve_doc_ids)\n", 710 "data.head(3)" 711 ] 712 }, 713 { 714 "cell_type": "code", 715 "execution_count": null, 716 "metadata": { 717 "application/vnd.databricks.v1+cell": { 718 "cellMetadata": { 719 "byteLimit": 2048000, 720 "rowLimit": 10000 721 }, 722 "inputWidgets": {}, 723 "nuid": "5e5c4cd1-38c3-4709-8d41-6e319fb8a924", 724 "showTitle": false, 725 "title": "" 726 } 727 }, 728 "outputs": [], 729 "source": [ 730 "# Persist the static evaluation dataset to disk\n", 731 "data.to_csv(EVALUATION_DATASET_PATH, index=False)" 732 ] 733 }, 734 { 735 "cell_type": "code", 736 "execution_count": 16, 737 "metadata": { 738 "application/vnd.databricks.v1+cell": { 739 "cellMetadata": { 740 "byteLimit": 2048000, 741 "rowLimit": 10000 742 }, 743 "inputWidgets": {}, 744 "nuid": "deabd8f0-44cf-409f-a27b-e82dd4d99940", 745 "showTitle": false, 746 "title": "" 747 } 748 }, 749 "outputs": [ 750 { 751 "data": { 752 "text/html": [ 753 "<div>\n", 754 "<style scoped>\n", 755 " .dataframe tbody tr th:only-of-type {\n", 756 " vertical-align: middle;\n", 757 " }\n", 758 "\n", 759 " .dataframe tbody tr th {\n", 760 " vertical-align: top;\n", 761 " }\n", 762 "\n", 763 " .dataframe thead th {\n", 764 " text-align: right;\n", 765 " }\n", 766 "</style>\n", 767 "<table border=\"1\" class=\"dataframe\">\n", 768 " <thead>\n", 769 " <tr style=\"text-align: right;\">\n", 770 " <th></th>\n", 771 " <th>question</th>\n", 772 " <th>source</th>\n", 773 " <th>retrieved_doc_ids</th>\n", 774 " </tr>\n", 775 " </thead>\n", 776 " <tbody>\n", 777 " <tr>\n", 778 " <th>0</th>\n", 779 " <td>What is the purpose of the MLflow Model Registry?</td>\n", 780 " <td>[model-registry.html]</td>\n", 781 " <td>[model-registry.html, introduction/index.html,...</td>\n", 782 " </tr>\n", 783 " <tr>\n", 784 " <th>1</th>\n", 785 " <td>What is the purpose of registering a model wit...</td>\n", 786 " <td>[model-registry.html]</td>\n", 787 " <td>[model-registry.html, models.html, introductio...</td>\n", 788 " </tr>\n", 789 " <tr>\n", 790 " <th>2</th>\n", 791 " <td>What can you do with registered models and mod...</td>\n", 792 " <td>[model-registry.html]</td>\n", 793 " <td>[model-registry.html, models.html, deployment/...</td>\n", 794 " </tr>\n", 795 " </tbody>\n", 796 "</table>\n", 797 "</div>" 798 ], 799 "text/plain": [ 800 " question source \\\n", 801 "0 What is the purpose of the MLflow Model Registry? [model-registry.html] \n", 802 "1 What is the purpose of registering a model wit... [model-registry.html] \n", 803 "2 What can you do with registered models and mod... [model-registry.html] \n", 804 "\n", 805 " retrieved_doc_ids \n", 806 "0 [model-registry.html, introduction/index.html,... \n", 807 "1 [model-registry.html, models.html, introductio... \n", 808 "2 [model-registry.html, models.html, deployment/... " 809 ] 810 }, 811 "execution_count": 16, 812 "metadata": {}, 813 "output_type": "execute_result" 814 } 815 ], 816 "source": [ 817 "# Load the static evaluation dataset from disk and deserialize the source and retrieved doc ids\n", 818 "data = pd.read_csv(EVALUATION_DATASET_PATH)\n", 819 "data[\"source\"] = data[\"source\"].apply(ast.literal_eval)\n", 820 "data[\"retrieved_doc_ids\"] = data[\"retrieved_doc_ids\"].apply(ast.literal_eval)\n", 821 "data.head(3)" 822 ] 823 }, 824 { 825 "cell_type": "markdown", 826 "metadata": { 827 "application/vnd.databricks.v1+cell": { 828 "cellMetadata": {}, 829 "inputWidgets": {}, 830 "nuid": "c62b5306-9c0d-4ce8-8c4a-23cb1ecc7f66", 831 "showTitle": false, 832 "title": "" 833 } 834 }, 835 "source": [ 836 "## Step 3: Calling `mlflow.evaluate()`" 837 ] 838 }, 839 { 840 "cell_type": "markdown", 841 "metadata": { 842 "application/vnd.databricks.v1+cell": { 843 "cellMetadata": {}, 844 "inputWidgets": {}, 845 "nuid": "bdeebdcc-b4e7-4f9d-8fdc-366f9c13ed20", 846 "showTitle": false, 847 "title": "" 848 } 849 }, 850 "source": [ 851 "### Metrics Definition\n", 852 "\n", 853 "There are three built-in metrics provided for the retriever model type. Click the metric name below to see the metrics definitions.\n", 854 "\n", 855 "1. [mlflow.metrics.precision_at_k(k)](https://mlflow.org/docs/latest/python_api/mlflow.metrics.html#mlflow.metrics.precision_at_k)\n", 856 "1. [mlflow.metrics.recall_at_k(k)](https://mlflow.org/docs/latest/python_api/mlflow.metrics.html#mlflow.metrics.recall_at_k)\n", 857 "1. [mlflow.metrics.ndcg_at_k(k)](https://mlflow.org/docs/latest/python_api/mlflow.metrics.html#mlflow.metrics.ndcg_at_k) \n", 858 "\n", 859 "All metrics compute a score between 0 and 1 for each row representing the corresponding metric of the retriever model at the given `k` value.\n", 860 "\n", 861 "The `k` parameter should be a positive integer representing the number of retrieved documents\n", 862 "to evaluate for each row. `k` defaults to 3.\n", 863 "\n", 864 "When the model type is `\"retriever\"`, these metrics will be calculated automatically with the\n", 865 "default `k` value of 3.\n" 866 ] 867 }, 868 { 869 "cell_type": "markdown", 870 "metadata": { 871 "application/vnd.databricks.v1+cell": { 872 "cellMetadata": {}, 873 "inputWidgets": {}, 874 "nuid": "25a32237-b2ef-4f4a-9e5a-4537e7e43012", 875 "showTitle": false, 876 "title": "" 877 } 878 }, 879 "source": [ 880 "### Basic usage\n", 881 "\n", 882 "There are two supported ways to specify the retriever's output:\n", 883 "\n", 884 "* Case 1: Save the retriever's output to a static evaluation dataset\n", 885 "* Case 2: Wrap the retriever in a function" 886 ] 887 }, 888 { 889 "cell_type": "code", 890 "execution_count": null, 891 "metadata": { 892 "application/vnd.databricks.v1+cell": { 893 "cellMetadata": { 894 "byteLimit": 2048000, 895 "rowLimit": 10000 896 }, 897 "inputWidgets": {}, 898 "nuid": "0390728a-a6cf-4c84-867a-0c6832114471", 899 "showTitle": false, 900 "title": "" 901 } 902 }, 903 "outputs": [ 904 { 905 "name": "stderr", 906 "output_type": "stream", 907 "text": [ 908 "2023/11/22 14:39:59 WARNING mlflow.data.pandas_dataset: Failed to infer schema for Pandas dataset. Exception: Unable to map 'object' type to MLflow DataType. object can be mapped iff all values have identical data type which is one of (string, (bytes or byterray), int, float).\n", 909 "2023/11/22 14:39:59 INFO mlflow.models.evaluation.base: Evaluating the model with the default evaluator.\n", 910 "2023/11/22 14:39:59 INFO mlflow.models.evaluation.default_evaluator: Testing metrics on first row...\n", 911 "2023/11/22 14:39:59 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: precision_at_3\n", 912 "2023/11/22 14:39:59 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: recall_at_3\n", 913 "2023/11/22 14:39:59 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: ndcg_at_3\n" 914 ] 915 } 916 ], 917 "source": [ 918 "# Case 1: Evaluating a static evaluation dataset\n", 919 "with mlflow.start_run() as run:\n", 920 " evaluate_results = mlflow.evaluate(\n", 921 " data=data,\n", 922 " model_type=\"retriever\",\n", 923 " targets=\"source\",\n", 924 " predictions=\"retrieved_doc_ids\",\n", 925 " evaluators=\"default\",\n", 926 " )" 927 ] 928 }, 929 { 930 "cell_type": "code", 931 "execution_count": 18, 932 "metadata": { 933 "application/vnd.databricks.v1+cell": { 934 "cellMetadata": { 935 "byteLimit": 2048000, 936 "rowLimit": 10000 937 }, 938 "inputWidgets": {}, 939 "nuid": "70aa6719-f69d-4fda-8a67-ac4e0d8ea6d8", 940 "showTitle": false, 941 "title": "" 942 } 943 }, 944 "outputs": [ 945 { 946 "data": { 947 "text/html": [ 948 "<div>\n", 949 "<style scoped>\n", 950 " .dataframe tbody tr th:only-of-type {\n", 951 " vertical-align: middle;\n", 952 " }\n", 953 "\n", 954 " .dataframe tbody tr th {\n", 955 " vertical-align: top;\n", 956 " }\n", 957 "\n", 958 " .dataframe thead th {\n", 959 " text-align: right;\n", 960 " }\n", 961 "</style>\n", 962 "<table border=\"1\" class=\"dataframe\">\n", 963 " <thead>\n", 964 " <tr style=\"text-align: right;\">\n", 965 " <th></th>\n", 966 " <th>question</th>\n", 967 " <th>source</th>\n", 968 " </tr>\n", 969 " </thead>\n", 970 " <tbody>\n", 971 " <tr>\n", 972 " <th>0</th>\n", 973 " <td>What is the purpose of the MLflow Model Registry?</td>\n", 974 " <td>[model-registry.html]</td>\n", 975 " </tr>\n", 976 " <tr>\n", 977 " <th>1</th>\n", 978 " <td>What is the purpose of registering a model wit...</td>\n", 979 " <td>[model-registry.html]</td>\n", 980 " </tr>\n", 981 " <tr>\n", 982 " <th>2</th>\n", 983 " <td>What can you do with registered models and mod...</td>\n", 984 " <td>[model-registry.html]</td>\n", 985 " </tr>\n", 986 " </tbody>\n", 987 "</table>\n", 988 "</div>" 989 ], 990 "text/plain": [ 991 " question source\n", 992 "0 What is the purpose of the MLflow Model Registry? [model-registry.html]\n", 993 "1 What is the purpose of registering a model wit... [model-registry.html]\n", 994 "2 What can you do with registered models and mod... [model-registry.html]" 995 ] 996 }, 997 "execution_count": 18, 998 "metadata": {}, 999 "output_type": "execute_result" 1000 } 1001 ], 1002 "source": [ 1003 "question_source_df = data[[\"question\", \"source\"]]\n", 1004 "question_source_df.head(3)" 1005 ] 1006 }, 1007 { 1008 "cell_type": "code", 1009 "execution_count": null, 1010 "metadata": { 1011 "application/vnd.databricks.v1+cell": { 1012 "cellMetadata": { 1013 "byteLimit": 2048000, 1014 "rowLimit": 10000 1015 }, 1016 "inputWidgets": {}, 1017 "nuid": "00672280-3dfc-4c00-9ae2-bea50732ef8b", 1018 "showTitle": false, 1019 "title": "" 1020 } 1021 }, 1022 "outputs": [ 1023 { 1024 "name": "stderr", 1025 "output_type": "stream", 1026 "text": [ 1027 "2023/11/22 14:09:12 WARNING mlflow.data.pandas_dataset: Failed to infer schema for Pandas dataset. Exception: Unable to map 'object' type to MLflow DataType. object can be mapped iff all values have identical data type which is one of (string, (bytes or byterray), int, float).\n", 1028 "2023/11/22 14:09:12 INFO mlflow.models.evaluation.base: Evaluating the model with the default evaluator.\n", 1029 "2023/11/22 14:09:12 INFO mlflow.models.evaluation.default_evaluator: Computing model predictions.\n", 1030 "2023/11/22 14:09:24 INFO mlflow.models.evaluation.default_evaluator: Testing metrics on first row...\n", 1031 "2023/11/22 14:09:24 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: precision_at_3\n", 1032 "2023/11/22 14:09:24 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: recall_at_3\n", 1033 "2023/11/22 14:09:24 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: ndcg_at_3\n" 1034 ] 1035 } 1036 ], 1037 "source": [ 1038 "# Case 2: Evaluating a function\n", 1039 "def retriever_model_function(question_df: pd.DataFrame) -> pd.Series:\n", 1040 " return question_df[\"question\"].apply(retrieve_doc_ids)\n", 1041 "\n", 1042 "\n", 1043 "with mlflow.start_run() as run:\n", 1044 " evaluate_results = mlflow.evaluate(\n", 1045 " model=retriever_model_function,\n", 1046 " data=question_source_df,\n", 1047 " model_type=\"retriever\",\n", 1048 " targets=\"source\",\n", 1049 " evaluators=\"default\",\n", 1050 " )" 1051 ] 1052 }, 1053 { 1054 "cell_type": "code", 1055 "execution_count": null, 1056 "metadata": { 1057 "application/vnd.databricks.v1+cell": { 1058 "cellMetadata": { 1059 "byteLimit": 2048000, 1060 "rowLimit": 10000 1061 }, 1062 "inputWidgets": {}, 1063 "nuid": "cb24318b-6149-4703-ad06-731c8a75866f", 1064 "showTitle": false, 1065 "title": "" 1066 } 1067 }, 1068 "outputs": [ 1069 { 1070 "name": "stdout", 1071 "output_type": "stream", 1072 "text": [ 1073 "{ 'ndcg_at_3/mean': 0.7530888125490431,\n", 1074 " 'ndcg_at_3/p90': 1.0,\n", 1075 " 'ndcg_at_3/variance': 0.1209151911325433,\n", 1076 " 'precision_at_3/mean': 0.26785714285714285,\n", 1077 " 'precision_at_3/p90': 0.3333333333333333,\n", 1078 " 'precision_at_3/variance': 0.017538265306122448,\n", 1079 " 'recall_at_3/mean': 0.8035714285714286,\n", 1080 " 'recall_at_3/p90': 1.0,\n", 1081 " 'recall_at_3/variance': 0.15784438775510204}\n" 1082 ] 1083 } 1084 ], 1085 "source": [ 1086 "pp = pprint.PrettyPrinter(indent=4)\n", 1087 "pp.pprint(evaluate_results.metrics)" 1088 ] 1089 }, 1090 { 1091 "cell_type": "markdown", 1092 "metadata": { 1093 "application/vnd.databricks.v1+cell": { 1094 "cellMetadata": {}, 1095 "inputWidgets": {}, 1096 "nuid": "7a9b83c5-1544-4b0e-81f6-7abc1fafa258", 1097 "showTitle": false, 1098 "title": "" 1099 } 1100 }, 1101 "source": [ 1102 "### Try different k values\n", 1103 "To use another `k` value, use the `evaluator_config` parameter\n", 1104 "in the `mlflow.evaluate()` API as follows: `evaluator_config={\"retriever_k\": <k_value>}`.\n", 1105 "\n", 1106 "\n", 1107 "```python\n", 1108 "# Case 1: Specifying the model type\n", 1109 "evaluate_results = mlflow.evaluate(\n", 1110 " data=data,\n", 1111 " model_type=\"retriever\",\n", 1112 " targets=\"ground_truth_context\",\n", 1113 " predictions=\"retrieved_context\",\n", 1114 " evaluators=\"default\",\n", 1115 " evaluator_config={\"retriever_k\": 5}\n", 1116 " )\n", 1117 "```\n", 1118 "\n", 1119 "Alternatively, you can directly specify the desired metrics\n", 1120 "in the `extra_metrics` parameter of the `mlflow.evaluate()` API without specifying a model\n", 1121 "type. In this case, the `k` value specified in the `evaluator_config` parameter will be\n", 1122 "ignored.\n", 1123 "\n", 1124 "\n", 1125 "```python\n", 1126 "# Case 2: Specifying the extra_metrics\n", 1127 "evaluate_results = mlflow.evaluate(\n", 1128 " data=data,\n", 1129 " targets=\"ground_truth_context\",\n", 1130 " predictions=\"retrieved_context\",\n", 1131 " extra_matrics=[\n", 1132 " mlflow.metrics.precision_at_k(4),\n", 1133 " mlflow.metrics.precision_at_k(5)\n", 1134 " ],\n", 1135 " )\n", 1136 "```" 1137 ] 1138 }, 1139 { 1140 "cell_type": "code", 1141 "execution_count": null, 1142 "metadata": { 1143 "application/vnd.databricks.v1+cell": { 1144 "cellMetadata": { 1145 "byteLimit": 2048000, 1146 "rowLimit": 10000 1147 }, 1148 "inputWidgets": {}, 1149 "nuid": "4b7174aa-0aa2-497d-aaa5-842121fcf270", 1150 "showTitle": false, 1151 "title": "" 1152 } 1153 }, 1154 "outputs": [ 1155 { 1156 "name": "stderr", 1157 "output_type": "stream", 1158 "text": [ 1159 "2023/11/22 14:40:22 WARNING mlflow.data.pandas_dataset: Failed to infer schema for Pandas dataset. Exception: Unable to map 'object' type to MLflow DataType. object can be mapped iff all values have identical data type which is one of (string, (bytes or byterray), int, float).\n", 1160 "2023/11/22 14:40:22 INFO mlflow.models.evaluation.base: Evaluating the model with the default evaluator.\n", 1161 "2023/11/22 14:40:22 INFO mlflow.models.evaluation.default_evaluator: Testing metrics on first row...\n", 1162 "2023/11/22 14:40:22 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: precision_at_1\n", 1163 "2023/11/22 14:40:22 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: precision_at_2\n", 1164 "2023/11/22 14:40:22 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: precision_at_3\n", 1165 "2023/11/22 14:40:22 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: recall_at_1\n", 1166 "2023/11/22 14:40:22 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: recall_at_2\n", 1167 "2023/11/22 14:40:22 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: recall_at_3\n", 1168 "2023/11/22 14:40:22 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: ndcg_at_1\n", 1169 "2023/11/22 14:40:22 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: ndcg_at_2\n", 1170 "2023/11/22 14:40:22 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: ndcg_at_3\n" 1171 ] 1172 } 1173 ], 1174 "source": [ 1175 "with mlflow.start_run() as run:\n", 1176 " evaluate_results = mlflow.evaluate(\n", 1177 " data=data,\n", 1178 " targets=\"source\",\n", 1179 " predictions=\"retrieved_doc_ids\",\n", 1180 " evaluators=\"default\",\n", 1181 " extra_metrics=[\n", 1182 " mlflow.metrics.precision_at_k(1),\n", 1183 " mlflow.metrics.precision_at_k(2),\n", 1184 " mlflow.metrics.precision_at_k(3),\n", 1185 " mlflow.metrics.recall_at_k(1),\n", 1186 " mlflow.metrics.recall_at_k(2),\n", 1187 " mlflow.metrics.recall_at_k(3),\n", 1188 " mlflow.metrics.ndcg_at_k(1),\n", 1189 " mlflow.metrics.ndcg_at_k(2),\n", 1190 " mlflow.metrics.ndcg_at_k(3),\n", 1191 " ],\n", 1192 " )" 1193 ] 1194 }, 1195 { 1196 "cell_type": "code", 1197 "execution_count": null, 1198 "metadata": { 1199 "application/vnd.databricks.v1+cell": { 1200 "cellMetadata": { 1201 "byteLimit": 2048000, 1202 "rowLimit": 10000 1203 }, 1204 "inputWidgets": {}, 1205 "nuid": "d57c201b-3718-43af-b8c2-ef22bfa2c15b", 1206 "showTitle": false, 1207 "title": "" 1208 } 1209 }, 1210 "outputs": [ 1211 { 1212 "data": { 1213 "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAHHCAYAAABDUnkqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB4f0lEQVR4nO3dd1zV9f4H8NfZ7D1FBBRwM1VUXJkjc6Td0hy5NXf+zDIzLbX0lpnmvm4zV64sNbVIU5wloLhFUFxM2Ztzvr8/jhw5MgQFDhxez8eDe/N7vuf7fZ8jcl58pkgQBAFEREREekKs6wKIiIiIKhLDDREREekVhhsiIiLSKww3REREpFcYboiIiEivMNwQERGRXmG4ISIiIr3CcENERER6heGGiIiI9ArDDVEZbd68GSKRCHfv3tV1KbXGiRMnIBKJcOLECV2XUq2V9D5t3boVjRo1gkwmg4WFheb4okWLUL9+fUgkEvj4+FRprURVgeGGqp2CECESiRAcHFzkcUEQ4OzsDJFIhF69er3UPVatWoXNmze/YqVV48SJE3j77bfh4OAAuVwOOzs79O7dG/v27dN1aVRGCxYswC+//FKmc+/evav5/heJRJDJZLCxsUHbtm3x2WefITo6ukzXuXHjBoYPH44GDRpg3bp1WLt2LQDg2LFj+OSTTxAYGIhNmzZhwYIFL/uyKt2ZM2fw5ZdfIjk5uUznDx8+HCYmJkWOX758GTY2NnB1deUvJ7WFQFTNbNq0SQAgGBgYCOPHjy/y+PHjxwUAgkKhEHr27PlS92jatKnQsWPHcj0nPz9fyMrKElQq1Uvd82XMmTNHACB4eHgIc+bMETZs2CB8++23QqdOnQQAwrZt26qsFl1QKpVCVlaWoFQqdV3KKzE2NhaGDRtWpnOjoqIEAMLAgQOFrVu3Clu2bBGWLl0qDB48WDA0NBSMjIyEHTt2aD2nuPdp9erVAgDh9u3bWufOmDFDEIvFQk5Oziu/rsq2aNEiAYAQFRVVpvOHDRsmGBsbax0LDw8XbGxshHr16gmRkZGVUCVVR1Id5iqiUr355pvYvXs3li1bBqn02bfq9u3b4e/vj4SEhCqpIyMjA8bGxpBIJJBIJFVyTwDYs2cP5s2bh3feeQfbt2+HTCbTPPbxxx/j6NGjyMvLq7J6qlJ2djbkcjnEYjEMDAx0XY5O+Pn5YciQIVrH7t27h27dumHYsGFo3LgxvL29AaDY9ykuLg4AtLqjCo4bGhpCLpdXWK2ZmZkwMjKqsOtVlKtXr6Jz584wNDTE8ePH4ebmpuuSqKroOl0RPa+g5Wb37t2CSCQSDh8+rHksJydHsLS0FBYvXiy4uLgUablRKpXCkiVLhCZNmggKhUKws7MTxo4dKzx58kRzjouLiwBA66ugFafg3idOnBDGjx8v2NraChYWFlqPPf9b5OHDh4UOHToIJiYmgqmpqdCiRQutFpVbt24Jb7/9tmBvby8oFArByclJGDBggJCcnFzq+9CoUSPByspKSE1NLdP7FhsbK4wcOVKws7MTFAqF4OXlJWzevFnrnIJWgUWLFgkrVqwQ3NzcBENDQ6Fr165CdHS0oFKphHnz5glOTk6CgYGB0KdPHyExMVHrGgXv+9GjRwVvb29BoVAIjRs3Fvbu3at1XmJiovDRRx8JzZo1E4yNjQVTU1PhjTfeEMLCwrTOK2iJ27FjhzBr1iyhTp06gkgkEpKSkjSPHT9+vFzvZ15enjBv3jyhfv36glwuF1xcXISZM2cK2dnZxb6WU6dOCS1bthQUCoXg5uYmbNmypUzv+aJFi4Q2bdoIVlZWgoGBgeDn5yfs3r1b65znv9cAlNqKU/jvqDhnzpwRAAiDBg0q8h4WvE/FfY9/8cUXxdayadMmzXW2bt0q+Pn5CQYGBoKlpaUwYMAAITo6Wuv+HTt2FJo2bSr8+++/Qvv27QVDQ0Phww8/FARBELKzs4U5c+YIDRo0EORyuVC3bl3h448/LvK+AxAmTpwo7N+/X2jatKkgl8uFJk2aCL///rvmnJLqLa0Vp3DLzbVr1wR7e3uhbt26QkRERJFz//nnH6Fbt26CtbW1YGBgILi6ugojRowo8dpUs7DlhqotV1dXtGnTBjt27ECPHj0AAL///jtSUlLw3nvvYdmyZUWe88EHH2Dz5s0YMWIEpkyZgqioKKxYsQKhoaE4ffo0ZDIZli5dismTJ8PExASzZs0CANjb22tdZ8KECbC1tcWcOXOQkZFRYo2bN2/GyJEj0bRpU8ycORMWFhYIDQ3FkSNHMGjQIOTm5qJ79+7IycnB5MmT4eDggIcPH+LgwYNITk6Gubl5sde9ffs2bty4gZEjR8LU1PSF71VWVhY6deqEiIgITJo0CW5ubti9ezeGDx+O5ORkfPjhh1rnb9u2Dbm5uZg8eTKePHmCb7/9Fv3790fnzp1x4sQJzJgxAxEREVi+fDmmT5+OjRs3FqlvwIABGDduHIYNG4ZNmzbh3XffxZEjR9C1a1cAQGRkJH755Re8++67cHNzQ2xsLP73v/+hY8eOuHbtGurUqaN1zfnz50Mul2P69OnIyckptmWhrO/n6NGjsWXLFrzzzjv46KOPcP78eSxcuBDXr1/H/v37ta4ZERGBd955B6NGjcKwYcOwceNGDB8+HP7+/mjatGmp7/sPP/yAPn36YPDgwcjNzcXOnTvx7rvv4uDBg+jZsycA9aDe0aNHo1WrVhg7diwAoEGDBi/6Ky1RmzZt0KBBA/zxxx8lnrN06VL8+OOP2L9/P1avXg0TExN4eXnB3d0da9euxYULF7B+/XoAQNu2bQEAX3/9NWbPno3+/ftj9OjRiI+Px/Lly9GhQweEhoZqtQAlJiaiR48eeO+99zBkyBDY29tDpVKhT58+CA4OxtixY9G4cWOEh4djyZIluHXrVpExR8HBwdi3bx8mTJgAU1NTLFu2DP/5z38QHR0Na2trvP3227h16xZ27NiBJUuWwMbGBgBga2v7wvfo5s2b6Ny5M6RSKY4fP17k/Y6Li0O3bt1ga2uLTz/9FBYWFrh79y7HsekTXacroucVtJD8888/wooVKwRTU1MhMzNTEARBePfdd4XXXntNEAShSMvNqVOnih2HcuTIkSLHSxpzU3Dvdu3aCfn5+cU+VvCbY3JysmBqaioEBAQIWVlZWucWjMsJDQ3VtEKVx4EDBwQAwpIlS8p0/tKlSwUAwk8//aQ5lpubK7Rp00YwMTHRtP4UtArY2tpqtXTMnDlTACB4e3sLeXl5muMDBw4U5HK51m/eBa0ChVtqUlJSBEdHR8HX11dzLDs7u8hYmaioKEGhUAjz5s3THCtodahfv77m7/n5xwpaJMryfoaFhQkAhNGjR2sdnz59ugBA+Ouvv4q8lpMnT2qOxcXFCQqFQvjoo49KvEeB5+vNzc0VmjVrJnTu3Fnr+MuMuSmp5UYQBOGtt94SAAgpKSmCIBR9nwThWctHfHy81nOLG5dy9+5dQSKRCF9//bXW8fDwcEEqlWod79ixowBAWLNmjda5W7duFcRisXDq1Cmt42vWrBEACKdPn9YcAyDI5XKtFpVLly4JAITly5drjr3MmBuZTCY4OjoKderUEW7dulXsefv379f8jCH9xNlSVK31798fWVlZOHjwINLS0nDw4EEMGjSo2HN3794Nc3NzdO3aFQkJCZovf39/mJiY4Pjx42W+75gxY144vuaPP/5AWloaPv300yLjHUQiEQBoWhKOHj2KzMzMMt8/NTUVAMrUagMAhw8fhoODAwYOHKg5JpPJMGXKFKSnp+Pvv//WOv/dd9/VajUKCAgAAAwZMkRrfFNAQAByc3Px8OFDrefXqVMH/fr10/zZzMwMQ4cORWhoKGJiYgAACoUCYrH6R4xSqURiYiJMTEzQsGFDhISEFHkNw4YNg6GhYamvsyzv5+HDhwEA06ZN0zr+0UcfAQAOHTqkdbxJkyZo37695s+2trZo2LAhIiMjS60FgFa9SUlJSElJQfv27Yt9fRWpYEZQWlpahVxv3759UKlU6N+/v9a/HQcHB3h4eBT5t6NQKDBixAitY7t370bjxo3RqFEjrWt07twZAIpco0uXLlotKl5eXjAzMyvT+14apVKJhIQEWFlZaVp7nlfQCnXw4EG9HbdW2zHcULVma2uLLl26YPv27di3bx+USiXeeeedYs+9ffs2UlJSYGdnB1tbW62v9PR0zQDLsijLwMM7d+4AAJo1a1bqdaZNm4b169fDxsYG3bt3x8qVK5GSklLqtc3MzACU/cPr3r178PDw0ISJAo0bN9Y8Xli9evW0/lwQGpydnYs9npSUpHXc3d1dE+AKeHp6AoBmqq1KpcKSJUvg4eEBhUIBGxsb2Nra4vLly8W+/rK852V5P+/duwexWAx3d3et5zo4OMDCwuKF7wUAWFpaFnnNxTl48CBat24NAwMDWFlZwdbWFqtXr37h3++rSk9PB1D28Psit2/fhiAI8PDwKPJv5/r160X+7Tg5ORXpNrx9+zauXr1a5PkF3xfPX+NV3vfSGBoa4scff8S1a9fQs2fPYruVO3bsiP/85z+YO3cubGxs8NZbb2HTpk3Iycl5pXtT9cExN1TtDRo0CGPGjEFMTAx69OhRZPZHAZVKBTs7O2zbtq3Yx8vSV1/gRS0I5bF48WIMHz4cBw4cwLFjxzBlyhQsXLgQ586dQ926dYt9TqNGjQAA4eHhFVZHYSW1SpV0XBCEct9jwYIFmD17NkaOHIn58+fDysoKYrEYU6dOhUqlKnJ+Wd/zsr6fz4evkrzsaz516hT69OmDDh06YNWqVXB0dIRMJsOmTZuwffv2Mt37ZV25cgV2dnaaEPyqVCoVRCIRfv/992Lfj+fXjinu70qlUqF58+b4/vvvi73H88G5Ir/Xnvfee+8hKSkJEyZMwNtvv43ffvtNK4yJRCLs2bMH586dw2+//YajR49i5MiRWLx4Mc6dO1fsWjlUszDcULXXr18/fPDBBzh37hx27dpV4nkNGjTAn3/+icDAwBd+UJb1g680BU3qV65cKdJK8LzmzZujefPm+Pzzz3HmzBkEBgZizZo1+Oqrr4o939PTEw0bNsSBAwfwww8/vPCHrYuLCy5fvgyVSqXVenPjxg3N4xUpIiICgiBovY+3bt0CoB4IDqinsr/22mvYsGGD1nOTk5NL7C4oq9LeTxcXF6hUKty+fVvTcgUAsbGxSE5OrrD3Yu/evTAwMMDRo0ehUCg0xzdt2lTk3Ir4fitw9uxZ3Llzp8g08VfRoEEDCIIANzc3TUvLy1zj0qVLeP311yvs9b7KdcaPH48nT57g888/x5AhQ7Bz584iLZutW7dG69at8fXXX2P79u0YPHgwdu7cidGjR79q6aRj7Jaias/ExASrV6/Gl19+id69e5d4Xv/+/aFUKjF//vwij+Xn52utcmpsbFzmVU9L0q1bN5iammLhwoXIzs7Weqzgt8/U1FTk5+drPda8eXOIxeIXNoHPnTsXiYmJGD16dJFrAOqVZg8ePAhAvSZQTEyMVvjLz8/H8uXLYWJigo4dO77UayzJo0ePtGYdpaam4scff4SPjw8cHBwAqH8zf/638N27dxcZv1MeZXk/33zzTQDqGUOFFbQoFMxielUSiQQikQhKpVJz7O7du8WuRFwR32+Austt+PDhkMvl+Pjjj1/5egXefvttSCQSzJ07t8jfmSAISExMfOE1+vfvj4cPH2LdunVFHsvKyip11mFJjI2NAeCl37tZs2bh//7v/7B792588MEHmuNJSUlFXmfBNhTsmtIPbLmhGmHYsGEvPKdjx4744IMPsHDhQoSFhaFbt26QyWS4ffs2du/ejR9++EEzXsff3x+rV6/GV199BXd3d9jZ2WkGPpaVmZkZlixZgtGjR6Nly5YYNGgQLC0tcenSJWRmZmLLli3466+/MGnSJLz77rvw9PREfn4+tm7dColEgv/85z+lXn/AgAEIDw/H119/jdDQUAwcOBAuLi5ITEzEkSNHEBQUpOn+GDt2LP73v/9h+PDhuHjxIlxdXbFnzx6cPn0aS5curbCxGQU8PT0xatQo/PPPP7C3t8fGjRsRGxur1WrRq1cvzJs3DyNGjEDbtm0RHh6Obdu2oX79+i9937K8n97e3hg2bBjWrl2L5ORkdOzYERcuXMCWLVvQt29fvPbaa6/8+gF1SPr+++/xxhtvYNCgQYiLi8PKlSvh7u6Oy5cva53r7++PP//8E99//z3q1KkDNzc3zSDukoSEhOCnn36CSqVCcnIy/vnnH+zduxcikQhbt26Fl5dXhbwOQN3q8tVXX2HmzJm4e/cu+vbtC1NTU0RFRWH//v0YO3Yspk+fXuo13n//ffz8888YN24cjh8/jsDAQCiVSty4cQM///wzjh49ihYtWpSrLn9/fwDqkPLee+9BJpOhd+/emtBTFosXL0ZSUhLWr18PKysrfPPNN9iyZQtWrVqFfv36oUGDBkhLS8O6detgZmamCcdUw+lolhZRiQpPBS9NcYv4CYIgrF27VvD39xcMDQ0FU1NToXnz5sInn3wiPHr0SHNOTEyM0LNnT8HU1LTYRfyKu3dJi/j9+uuvQtu2bQVDQ0PBzMxMaNWqlWZ5/MjISGHkyJFCgwYNBAMDA8HKykp47bXXhD///LPM70dQUJDw1ltvCXZ2doJUKhVsbW2F3r17CwcOHNA6LzY2VhgxYoRgY2MjyOVyoXnz5loLtAlCydOMC6YSPz/Furj3o/Aifl5eXoJCoRAaNWpU5LnZ2dnCRx99JDg6OgqGhoZCYGCgcPbsWaFjx45a0/BLunfhxwqmOJf1/czLyxPmzp0ruLm5CTKZTHB2di51Eb/nPV9jSTZs2CB4eHho3oNNmzZppmAXduPGDaFDhw6CoaFhmRfxK/iSSqWClZWVEBAQIMycOVO4d+/eC98nQSjfVPACe/fuFdq1aycYGxsLxsbGQqNGjYSJEycKN2/e1HpvmjZtWuzzc3NzhW+++UZo2rSpoFAoBEtLS8Hf31+YO3euZtq6IDxbxO95Li4uRd6b+fPnC05OToJYLC7XIn6F5efnC3379hUACAsXLhRCQkKEgQMHCvXq1dMs9tmrVy/h33//LfHaVLOIBKECRm8RUa3h6uqKZs2aabrEiIiqG465ISIiIr3CcENERER6heGGiIiI9ArH3BAREZFeYcsNERER6RWGGyIiItIrtW4RP5VKhUePHsHU1LRCl0QnIiKiyiMIAtLS0lCnTp0iW2k8r9aFm0ePHhXZwI2IiIhqhvv375e46XCBWhduCpahv3//foXtqEtERESVKzU1Fc7OzmXaTqbWhZuCrigzMzOGGyIiohqmLENKOKCYiIiI9ArDDREREekVhhsiIiLSK7VuzE1ZKZVK5OXl6boMqgZkMhkkEomuyyAiojJiuHmOIAiIiYlBcnKyrkuhasTCwgIODg5cG4mIqAZguHlOQbCxs7ODkZERP8xqOUEQkJmZibi4OACAo6OjjisiIqIXYbgpRKlUaoKNtbW1rsuhasLQ0BAAEBcXBzs7O3ZRERFVcxxQXEjBGBsjIyMdV0LVTcH3BMdhERFVfww3xWBXFD2P3xNERDUHww0RERHpFYYbemknTpyASCQq08yy8pxb2apTLUREVPEYbuiltW3bFo8fP4a5uXmFnvsyoqOjMX36dHh7e8PGxgb169fHO++8gyNHjlTK/YiIqPrSebhZuXIlXF1dYWBggICAAFy4cKHU85cuXYqGDRvC0NAQzs7O+L//+z9kZ2dXUbX6Izc395WvIZfLy7z2S3nOLa+tW7eiWbNmePjwIb788ksEBQVhx44daN26NcaOHYuhQ4dCqVRW+H2JiKgYaTFAQoROS9BpuNm1axemTZuGL774AiEhIfD29kb37t01a4o8b/v27fj000/xxRdf4Pr169iwYQN27dqFzz77rIorr346deqESZMmYdKkSTA3N4eNjQ1mz54NQRAAAK6urpg/fz6GDh0KMzMzjB07FgAQHByM9u3ba8LilClTkJGRobluTk4OZsyYAWdnZygUCri7u2PDhg0Ainbv3Lt3D71794alpSWMjY3RtGlTHD58uNhzAWDv3r1o2rQpFAoFXF1dsXjxYq3X5OrqigULFmDkyJEwNTVFvXr1sHbtWq1zfvvtN3z88cc4duwYduzYgX79+sHb2xsBAQGYPn06rl+/jri4OEydOrXE9y4zMxM9evRAYGAgu6qIiMoj9TFw83fg+EJg+wDgu4bA4obAkU91WpZO17n5/vvvMWbMGIwYMQIAsGbNGhw6dAgbN27Ep58WfWPOnDmDwMBADBo0CID6w2/gwIE4f/58pdUoCAKy8nTzW7+hTFKulo4tW7Zg1KhRuHDhAv7991+MHTsW9erVw5gxYwAA3333HebMmYMvvvgCAHDnzh288cYb+Oqrr7Bx40bEx8drAtKmTZsAAEOHDsXZs2exbNkyeHt7IyoqCgkJCcXef+LEicjNzcXJkydhbGyMa9euwcTEpNhzL168iP79++PLL7/EgAEDcObMGUyYMAHW1tYYPny45rzFixdj/vz5+Oyzz7Bnzx6MHz8eHTt2RMOGDZGbm4tJkyZh8+bNaN26NYKDgzF16lTcv38f/fr1Q2ZmJrp3745t27bB09MTU6dORYMGDbTqSE5ORs+ePWFiYoI//viDywAQERVHEIDUR8DjMOBRGPD4kvq/02OLnisSA/m67VHRWbjJzc3FxYsXMXPmTM0xsViMLl264OzZs8U+p23btvjpp59w4cIFtGrVCpGRkTh8+DDef//9Eu+Tk5ODnJwczZ9TU1PLVWdWnhJN5hwt13MqyrV53WEkL/tfkbOzM5YsWQKRSISGDRsiPDwcS5Ys0YSbzp0746OPPtKcP3r0aAwePFjTquHh4YFly5ahY8eOWL16NaKjo/Hzzz/jjz/+QJcuXQAA9evXL/H+0dHR+M9//oPmzZu/8Nzvv/8er7/+OmbPng0A8PT0xLVr17Bo0SKtcPPmm29iwoQJAIAZM2ZgyZIlOH78OBo2bIi///4btra2eOONN5CcnIy33noLkyZNQr9+/bBnzx7897//RefOnWFtbY0333wTf/zxh1a4iYmJwYABA+Dh4YHt27dDLpeX+b0mItJbggCkPHgWYB6Fqf8/I77ouSIxYNsIcPQGHH2AOj6AQ3NAblylJT9PZ+EmISEBSqUS9vb2Wsft7e1x48aNYp8zaNAgJCQkoF27dhAEAfn5+Rg3blyp3VILFy7E3LlzK7T26qp169ZaLT1t2rTB4sWLNeNNWrRooXX+pUuXcPnyZWzbtk1zTBAEqFQqREVFITw8HBKJBB07dizT/adMmYLx48fj2LFj6NKlC/7zn//Ay8ur2HOvX7+Ot956S+tYYGAgli5dCqVSqVkFuPDzRSIRHBwcNN2W4eHhaNu2LQB1q561tbXm79rHxwe7du3SPNfR0RFJSUla9+vatStatWqFXbt2cdVhIqqdBAFIuf8swBS0ymQW00IvkqiDTB2fZ0HGvhkgr34t3jVq+4UTJ05gwYIFWLVqFQICAhAREYEPP/wQ8+fP17QAPG/mzJmYNm2a5s+pqalwdnYu8z0NZRJcm9f9lWt/GYayiv3ANTbWTtLp6en44IMPMGXKlCLn1qtXDxER5RsQNnr0aHTv3h2HDh3CsWPHsHDhQixevBiTJ09+6ZplMpnWn0UiEVQqFQAgPz9fszVCbm5ukddXuEssJCQEH3zwgdbjPXv2xN69e3Ht2jVNaxMRkd4SBCD53rMg8/iS+r+znhQ9VywFbBsDdZ62yDj6AA7NAJlhlZb8snQWbmxsbCCRSBAbq91fFxsbCwcHh2KfM3v2bLz//vsYPXo0AKB58+bIyMjA2LFjMWvWLIjFRcdHKxQKKBSKl65TJBKVq2tIl54fe3Tu3Dl4eHiU2Crh5+eHa9euwd3dvdjHmzdvDpVKhb///lvTLfUizs7OGDduHMaNG4eZM2di3bp1xYabxo0b4/Tp01rHTp8+DU9PzzK3ori7u+PEiRMAgJYtW+LGjRs4cOAAevfujd9++w2XLl1CVlYWFi1ahPv376NPnz5az//vf/8LExMTvP766zhx4gSaNGlSpvsSEVV7ggAkRT0LMAVhJiup6LliKWDXRN21VMcHcPQF7JsCMoMqLrri6OxTWy6Xw9/fH0FBQejbty8AQKVSISgoCJMmTSr2OZmZmUUCTMEHYcGsoNosOjoa06ZNwwcffICQkBAsX768yAykwmbMmIHWrVtj0qRJGD16tGYQ8B9//IEVK1bA1dUVw4YNw8iRIzUDiu/du4e4uDj079+/yPWmTp2KHj16wNPTE0lJSTh+/DgaN25c7L0/+ugjtGzZEvPnz8eAAQNw9uxZrFixAqtWrSrz6+3SpQvGjBmDW7duwdPTEytXrsTAgQORm5uLli1bonv37vjwww/Ro0cPBAUFFRtyv/vuOyiVSnTu3BknTpxAo0aNynx/IqJqQRCAJ5Ha42MeXwKyU4qeK5YB9k2edSs5+qiDjPTlGwGqI502SUybNg3Dhg1DixYt0KpVKyxduhQZGRma2VNDhw6Fk5MTFi5cCADo3bs3vv/+e/j6+mq6pWbPno3evXtzzATU71dWVhZatWoFiUSCDz/8UDPluzheXl74+++/MWvWLLRv3x6CIKBBgwYYMGCA5pzVq1fjs88+w4QJE5CYmIh69eqVOMZJqVRi4sSJePDgAczMzPDGG29gyZIlxZ7r5+eHn3/+GXPmzMH8+fPh6OiIefPmaQ0mfhEzMzPMmDED/fv3R1BQEEaOHIkhQ4YgMTERjo6OSExMhJGRkabrqiRLlizRCjienp5lroGIqEqpVM+CjCbMXAZyigkyErk6uBQOMnaN9S7IFEck6LjJY8WKFVi0aBFiYmLg4+ODZcuWISAgAIB67RZXV1ds3rwZgHqMxddff42tW7fi4cOHsLW1Re/evfH111/DwsKiTPdLTU2Fubk5UlJSYGZmpvVYdnY2oqKi4ObmBgODmtUc16lTJ/j4+GDp0qW6LqVKCYKACRMm4ODBg5gzZw769u0LW1tbZGRk4MiRI5g/fz7Wr19fZDB1edXk7w0iqqFUKuDJHe3BvjGXgZxiZv1KFOoxMYVnLdk2BqT6Mwu0tM/v5+k83FQ1hhv99Ouvv+Lbb7/F2bNnIZVKkZ+fjxYtWuDjjz/GO++888rXr8nfG0RUA6iUQGJE0SCTm170XKmBepZS4VlLto0AiazouXqkPOGmZoyUJXqBPn36oE+fPsjKykJCQgIsLCxgamqq67KIiIpSKYGEW9qL4T2+DORlFD1XaqheN6ZwkLFpCEj48V0avjt6omDWUG1XsI0EEVG1oMxXB5nCg31jwoG8zKLnyozUQabwGBkbTwaZl8B3jIiIqCIo84H4G88FmStAflbRc2XGgKPXc0HGAxBzckxFYLghIiIqL2UeEHddezG82CvF76kkN9Ee6OvoA1g3YJCpRAw3REREpcnPBeKvaw/2jb0KKHOKnqswexpkCoUZqwZAMYvMUuVhuCEiIiqQnwPEXdMOMnHXAGVu0XMV5uquJc1gX1/A0o1BphpguCEiotopP0fdlVR4i4LYa4Aqr+i5Buba3UqO3oBVfaDQZsVUfTDcEBGR/svLVnclPQ59FmTirgOq/KLnGlo+CzAFYcbSlUGmBmG4oZf25Zdf4pdffkFYWBgAYPjw4UhOTsYvv/zyStd1dXXF1KlTMXXq1FeukYhqobws9SylwrOW4q4DgrLouYZW2mvIOPoAFvUYZGo4hhuqdHl5edi0aRN+/vlnXL9+HUqlEvXr18fbb7+NCRMmwMjISNclElFNlZupXjemYDG8R2Hq6djFBRkjm+eCjDdg7swgo4cYbvRUbm4u5HLd7ykSGRmJt956C2KxGOPHj4eXlxdMTExw48YNbNq0CStXrsTRo0e5WSURvVhuhjrIFB7sm3ATEFRFzzW2exZgCsKMmRODTC3BcKMnOnXqhGbNmkEqleKnn35C8+bNsXz5cnz88cc4deoUjI2N0a1bNyxZsgQ2NjYAAJVKhe+++w5r167F/fv3YW9vjw8++ACzZs0CAMyYMQP79+/HgwcP4ODggMGDB2POnDmQycq2f0lKSgq6d++OgQMHYu7cuRAV+qHi5eWF/v37Y926dejWrRtCQ0NhaWlZ7HXWr1+P6dOnY+/evXj99ddf8Z0iohohJ129t5JWkLkFoJjtEE3stbuV6vgApo4MMrUYw82LCELxy2RXBZlRuf5xbtmyBePHj8fp06eRnJyMzp07Y/To0ViyZAmysrIwY8YM9O/fH3/99RcAYObMmVi3bh2WLFmCdu3a4fHjx7hx44bmeqampti8eTPq1KmD8PBwjBkzBqampvjkk0/KVM9///tf+Pv7Y968eUhOTsbEiRMRFBSE+vXr47333sPvv/+O33//HSdPnsTSpUsxd+7cItf49ttv8e233+LYsWNo1apVmd8LIqpBslPVQabwrKWE2yg2yJg6Fp21ZOZYldVSDcBw8yJ5mcCCOrq592ePALlxmU/38PDAt99+CwD46quv4OvriwULFmge37hxI5ydnXHr1i04Ojrihx9+wIoVKzBs2DAAQIMGDdCuXTvN+Z9//rnmv11dXTF9+nTs3LmzzOFm69atOHLkCADgo48+QlRUFA4cOIC4uDiMHTsWDRs2BKAeiDxr1qwi4WbGjBnYunUr/v77bzRt2rTM7wMRVWPZKepNIgsP9k28g2KDjJlT0VlLpvZVWCzVVAw3esTf31/z35cuXcLx48dhYmJS5Lw7d+4gOTkZOTk5pXbz7Nq1C8uWLcOdO3eQnp6O/Pz8F24zX+DJkydIS0tDs2bNAAC//fYbfvnlFwQEBAAAJk2ahD/++AMA4OjoiKSkJK3nL168GBkZGfj3339Rv379Mt2TiKqZrGTtgb6PLwFP7hR/rlndooN9TeyqqlLSMww3LyIzUreg6Ore5WBs/KyVJz09Hb1798Y333xT5DxHR0dERkaWeq2zZ89i8ODBmDt3Lrp37w5zc3Ps3LkTixcvLlMt+fn5MDAw0Pw5NzdXq77CoSskJATu7u5az2/fvj0OHTqEn3/+GZ9++mmZ7klEOpSV9CzAFISZpKjizzWvB9R5bq8lY5uqqpRqAYabFxGJytU1VF34+flh7969cHV1hVRa9K/Zw8MDhoaGCAoKwujRo4s8fubMGbi4uGgGFwPAvXv3ynx/Gxsb5ObmIjY2Fvb29mjXrh2+/fZbrF+/Hk+ePMG6detgY2ODM2fOYNasWdi4caPW81u1aoVJkybhjTfegFQqxfTp08vx6omoUmU+0e5WehQGJJfw88HCRXvWkqMPYGxdRYVSbcVwo6cmTpyIdevWYeDAgfjkk09gZWWFiIgI7Ny5E+vXr4eBgQFmzJiBTz75BHK5HIGBgYiPj8fVq1cxatQoeHh4IDo6Gjt37kTLli1x6NAh7N+/v8z3F4vF6NOnD1atWoW5c+fihx9+QO/evWFiYgJzc3MMGzYMS5cuxciRI/HDDz8U2z3Wtm1bHD58GD169IBUKuWifkS6kJGovarv40tAcnTx51q6Fh3sa2RVVZUSaTDc6Kk6derg9OnTmDFjBrp164acnBy4uLjgjTfegPjppm6zZ8+GVCrFnDlz8OjRIzg6OmLcuHEAgD59+uD//u//MGnSJOTk5KBnz56YPXs2vvzyyzLXMGfOHLRq1QqtW7dGjx49cO3aNcTExMDS0hIqlQqzZs3STEsvSbt27XDo0CG8+eabkEgkmDx58ku/J0T0AunxT7uVQp91MaXcL/5cq/rPBRkv9bYFRNWASBCEYoao66/U1FSYm5sjJSWlyODY7OxsREVFwc3NTWu8CL28Y8eO4b333sOQIUMwZswYzayn8PBwfPfdd7C1tcX333+v4ypfjN8bpHfS47S7lR6HAakPiz/X2l17MTwHL8DQoooKJVIr7fP7eWy5oUrVrVs3XLx4EfPmzUP79u2Rnp4OALCzs8OwYcMwc+ZMHVdIVAukxWh3Kz0KA9KKmyghUgeZwrOWHLwAg7LNkiSqLhhuqNK5ublh06ZN2LBhA2JjYyEWi2Fvz7UqiCqcIABpj7UXw3sUBqTHFHOyCLDxfC7INAcUplVYMNVk+ap8pOamIjknGak56v9PzklGSk4KrAys0LtBb53VxnBDVUYsFsPRkSuJElUIQQBSHxWdtZQRV/RckRiwaai9GJ5Dc0BRdB0sqn0EQUB6XrommKTkpGiCilZoyU1BSnaK5ry0vLQSr+lj68NwQ0REpRAEIOWB9mJ4j8OAjPii54rEgG0j7cG+Ds1q5JIWVH7Z+dnFh5TcVCRnP2tZSclN0TpPWdwu6mVkKjOFucIcFgoLmCvMYa4wRwOLBhX4qsqP4YaIqDoRBPVUa62VfcOAzMSi54okgF1j7SBj3xSQl28BUKp+8lX5muCRkptSajAp3MqSrcx+6XsaSAyKhJSC/37+/wv+20xuBqm4+kWJ6lcREVFtIQjqxe+0Zi1dArKeFD1XLH0WZBy9gTq+6iAjM6zamqlcBEFAWl7as+6cUoJJ4TEr6XnpL31PqUgKM4VZkSBSJLTItQOMgVR/ZoIy3BARVQVBUG9HULhb6fEl9bYFzxPL1EGm8GBfu6aATH8+fGqirPwsTSAp/P+FQ0rhP6fkpCA1N/XVunzkps+CiEGhkCIvuWXFWGYMkUhUga+85mG4ISKqaCqVOsgU7lZ6fEm9I/bzJHLArslzQaYJIFVUZcW1Sp4q71mXTzFhpUj3T7b6v3OUOS99T0OpYYktJiWFFFO5abXs8qkJ+K4REb0KlQp4Evk0yIQ+bZW5DOQUF2QU6q6kwnst2TUBpPIqLlo/qAQV0nLTSg4m2UXHqCTnJCMjL+Ol7ykVSUsMJiV2/yjMoZAwrFYlhptaTCQSYf/+/ejbt69O6+jUqRN8fHywdOlSndZB9EIqFZAYob0YXsxlICe16LkShXqWUuHBvnaNAYmsSkuuCQRB0O7yea7FpKRuoNTcVKgE1UvdUwTRsy6f54JJwXiV57uDzOXm7PKpIRhuqMKpVCrs3r0bP/30Ey5duoSsrCy4uLigV69emDx5MqytuSMw1QAqpTrIFB7sG3MZyC1moKfUQL1uTOEgY9uwVgaZPGXes1aTUoLJ88dyVbkvfU9DqWHxIUX+NKQYaHcFFXT5SMSSCnzlVJ0w3FCFSkhIwDvvvIP79+9j4sSJ+Pjjj2FlZYXIyEhs374dTZo0wf79+9G2bVtdl0r0jEoJJNx6LsiEA8V1X8iMngWZgkXxbBoCEv36cVrQ5VPSQNmSjmXmZ770PaViaZmCiVbLisIccgm79Uibfv1rrMU6deoELy8vGBgYYP369ZDL5Rg3bpxmF+/bt29j1KhRuHDhAurXr48ffvihyDUePHiAjz/+GEePHkVOTg4aN26MlStXIiAgAADw1VdfYdmyZcjKysKAAQNgY2ODI0eOICwsDACQn5+PPn36oHHjxvjjjz8gkz37rbVZs2bo06cPDh06hH79+uHs2bOoX79+sa/l0KFDGDRoEFatWoXBgwdX7BtFpMwHEm5qz1qKCQfyivlQlhmp91YqPNjXxhOoQb/xF3T5vCikaFagLejyyUmFgJfbV1kE0bOpyCUMni0upBhJjdjlQxWC4eYFCn4w6IKh1LBc/9C3bNmCadOm4fz58zh79iyGDx+OwMBAvP7663j77bdhb2+P8+fPIyUlBVOnTtV6bnp6Ojp27AgnJyf8+uuvcHBwQEhICFQqdX/2tm3b8PXXX2PVqlUIDAzEzp07sXjxYri5uWmusWHDBohEIqxduxb5+fmYPHky9u3bB1tbW0yZMgWLFy/G1atXMXbsWMydOxdbtmwp8hq2b9+OcePGYfv27ejVq9fLvXFEBZR5QPxN7VlLMVeA4v5Ny02KBhlr92oVZPKUecWGFK1gkq0dUlJyUpCnynvpexpJjco1eNZCYQETmQm7fEinGG5eICs/CwHbA3Ry7/ODzsNIVvaVRr28vPDFF18AADw8PLBixQoEBQVBEATcuHEDR48eRZ06dQAACxYsQI8ePTTP3b59O+Lj4/HPP//AysoKAODu7q55fPny5Rg1ahRGjBgBAJgzZw6OHTum2eUbAH788UfMnDkTEokECxYswLFjx7Bt2zYIgoAJEyYgK0v9gVIQup63cuVKzJo1C7/99hs6duxY5tdNBEC9jkzCLeD++WdBJvYqkF/Miq1y00L7LD2dtWTtDojFVVKqUqXU6vIp2Hzw+WDy/OJur/KLlkwsK9vg2YLHDNRdQ+zyoZqI4UaPeHl5af3Z0dERcXFxuH79OpydnTXBBgDatGmjdW5YWBh8fX01weZ5N2/exIQJE7SOtWrVCn/99Zfmz+Hh4ZqxNL/99hu++OILdOrUCQDw+eefY9asWZq6kpK0Fy7bs2cP4uLicPr0abRs2bIcr5pqLUFQt8rcPQXcDQbunS5+ryWF2dMA83RVX0cfwKp+hQQZQRCQmZ9ZYpdPSSElLTftpbt8xCKxZjxK4WCiGaOisIC5gTnM5dotK+VtCSaqyRhuXsBQaojzg87r7N7lUXiMC6Ce6l3QrfTCexm++hLu+fn5muvk5ubC2PjZRn0mJs92Hw4JCdFqFQIAX19fhISEYOPGjWjRogV/CFNRKhUQf+NpkAkG7p4GMhO0z5EaAHVbqkNMQfeSpVuZgkyuMrfELp/nl8YvPEYlX5X/0i/JWGZcJJg8Pw7l+e4fU7kpxKKqaWEiqqkYbl5AJBKVq2uoOmrcuDHu37+Px48fw9HREQBw7tw5rXO8vLywfv16PHnypNjWm4YNG+Kff/7B0KFDNcf++ecfrXPc3d0RHh6OVq1aoV27dvjhhx/QoUMHANAMYL569SrGjx+Pjz/+WOu5DRo0wOLFi9GpUydIJBKsWLHi1V841WwqFRB3Td0ic/eUOsw8v+eS1BCoFwC4tANc2wFOflCKpZpunpScFKQ8PFVyy0qhqcqv0uUjF8tfGEyKPCY3h6wWThUnqgoMN7VAly5d4OnpiWHDhmHRokVITU3VdBEVGDhwIBYsWIC+ffti4cKFcHR0RGhoKOrUqYM2bdpg8uTJGDNmDFq0aIG2bdti165duHz5staMp379+mHlypVo1aoVvvzyS/Tt2xfW1tYwNDTElClTcPz4cbzxxhuYPXs2hg8fXqROT09PHD9+HJ06dYJUKuWifrWNSgXEXVW3zBR0Mz2/75LMCHAOUAcZ1/YQHH3wIDsOoXGhCHl4FGGh3yAyJfKVunwKxp2UZYZPQasLu3yIqheGm1pALBZj//79GDVqFFq1agVXV1csW7YMb7zxhuYcuVyOY8eO4aOPPsKbb76J/Px8NGnSBCtXrgQADB48GJGRkZg+fTqys7PRv39/DB8+HBcuXNBcY+rUqWjevDnWr1+P0aNHIzg4GPHx8TA2NoZcLsfUqVNhb29faq0NGzbEX3/9pWnBWbx4ceW8KaR7KiUQe+VpmDmtDjPZydrnyIyBeq2fhpl2yHdojpupkQiNDUVo5M8IPTcT8VnFjLMBYCIzKRpS5OawMNBuPSkcWtjlQ6QfRIIgvNyvODVUamoqzM3NkZKSAjMzM63HsrOzERUVBTc3NxgYcPfdF+natSscHBywdetWzbHQ0FD07NkTnTt3xpQpU+Dn5wepVIrbt29j+fLliImJwc8//6zDql8OvzcqgEqpXuH37umnLTNniu6/JDcB6rUBXAMB1/bItPHApSfXEBYXhpC4EFyKv1Sk+0gqlqKJdRP42fnB184XTa2bwsrQCjIxu3yI9Elpn9/PY8sNlUlmZibWrFmD7t27QyKRYMeOHfjzzz/xxx9/aJ3n6+uLsLAwfP311+jduzcSEhIgFothZmaGAQMGYNmyZTp6BVTllPlPw8zTbqbos0X3YJKbAi5tNC0z8WZ1EJoYru5mCluEm09uQikotZ5iKjOFt523Jsw0s2kGAykDJxE9w5abQvjbecmysrLQu3dvhIaGIjs7Gw0bNsTnn3+Ot99+u8TnCIKA+Ph45Ofnw8HBAeIqWkOkMvB7owyU+eoVf++eUncx3TsL5KZpn6MwA1zaAq7toKrXBneNzBCScFkdZmJD8CD9QZHLOho7wtfOF352fvCx84G7hTsXiCOqhdhyQxXO0NAQf/75Z7meIxKJYGdnV0kVkc4p89SL5d0raJk5V3RTSQNzwCUQcAlEbr0AXJOKEZJwST1mJngXUp7rlhJBBE9LT/ja+Wq+HE0cq+41EZFeYLghorJR5gGPQp8tmhd9vujGkgYW6jDj2g4pTr64hGyExF9CaNw/uHJ8U5Gdnw0kBmhu2xw+tj7ws/eDt603TOWmVfeaiEgvMdwUo5b11FEZ1Mrvifxc4FHIszVm7p8vurmkoSXgEgjBpR0eOTREiDIdofFhCI37AxG3Vhe5pJWBlSbI+Nr5orFVY671QkQVjuGmkIIVfjMzMytkxV7SH5mZ6g/151eB1iv5OcDDi09nM50C7l8ousGkkTXgEgilS1vcsnZGSF6yOsw82IO4W3FFLuli5qIZL+Nr5wsXMxeuB0NElY7hphCJRAILCwvExal/SBsZGfEHcS0nCAIyMzMRFxcHCwsLSCR6NJA1Lxt4+O+zMPPgn6KbTBrZAK6ByKzXGuHmtgjNSUBofBguRW5Bxk3tLimpSIrG1o01Ycbbzhs2hjZV+IKIiNQYbp7j4OAAAJqAQwQAFhYWmu+NGisvWx1gClb/vX8BUOZon2NsC7i2Q0JdX4QZWyAkOxahcaG4fmttkSnZxjJj+Nj6qMOMvR+a2TQr935oRESVgeHmOSKRCI6OjrCzs0NeXp6uy6FqQCaT1cwWm7wsdYC593TRvAf/Fg0zJvYQXAJx17EJQo2MEZL5CKFxoYi+tb7I5eyM7OBv5w9fe3XLDKdkE1F1xXBTAolEUjM/0Kj2ys1UD/otCDMPLwJK7dlJMHFAnktbXLN3R6hCjpCM+wiLC0PSnX+1ThNBBHdLd81YGV87XzgaO7KblohqBIYbopoqN0MdZgr2Znp4EVA919poWgepLq1xyaYeQmVihKRF4UpCOHKitMOMXCxHc9vmmoXyvG29Ya4wr8IXQ0RUcapFuFm5ciUWLVqEmJgYeHt7Y/ny5WjVqlWx53bq1Al///13keNvvvkmDh06VNmlEulOTjpw/9yzvZkehQCqfO1zzJzwuF4rhFg5IFSsQmjqHdxOugghQzvMWCgstBbKa2LdBHKJvApfDBFR5dF5uNm1axemTZuGNWvWICAgAEuXLkX37t1x8+bNYle33bdvH3JznzW1JyYmwtvbG++++25Vlk1U+XLS1AvlFSya9ygUeG5Qr9LcGRHOvggxt0WoKAehSbcQk3EReG5tvXqm9eBj56PuZrL3hZuZG7uYiEhv6XxvqYCAALRs2RIrVqwAAKhUKjg7O2Py5Mn49NNPX/j8pUuXYs6cOXj8+DGMjY1feH559qYgqlLZqeotDAr2ZnoUViTMZFnUw5W6zRFqYokQIQOXkm4iPU97ywOJSIJGVo00s5h87Xw5JZuIarwas7dUbm4uLl68iJkzZ2qOicVidOnSBWfPni3TNTZs2ID33nuvxGCTk5ODnJxnM0RSU1OLPY+oymWnqDeXLNib6fElQFBpnfLEygWhjo0RamyK0PxkXEu+g/yMcK2WGSOpEbxtvTWzmJrbNIeRzKiKXwwRUfWh03CTkJAApVIJe3t7reP29va4cePGC59/4cIFXLlyBRs2bCjxnIULF2Lu3LmvXCvRK8tKBqLPPh0AfAqICdcKMwKAezZuCLVzR6ihAUJzE3A3/SGQeQ0otOuBnaEdfO19NYvleVh6QCrWeQ8zEVG1UaN/Im7YsAHNmzcvcfAxAMycORPTpk3T/Dk1NRXOzs5VUR7VdplPCoWZYHWYwbNe4DwAN2zdEGLrilC5FKFZMXiSmwJk3QQK7XrgbuGuNfjXycSJ42WIiEqh03BjY2MDiUSC2NhYreOxsbEvXA02IyMDO3fuxLx580o9T6FQQKFQvHKtRC+U+eTpGjNPZzPFXkHhMJMuEuGSbQOEWDshVAqEZz1GtjIHyLqjCTMysQzNbZprgoyPnQ+nZBMRlZNOw41cLoe/vz+CgoLQt29fAOoBxUFBQZg0aVKpz929ezdycnIwZMiQKqiUqBgZic8WzLsbDMRd1Xo4RiJBqK0rQizsESbOx62sOKiQB2Tf1ZxjrjCHr62vppupiXUTKCQM40REr0Ln3VLTpk3DsGHD0KJFC7Rq1QpLly5FRkYGRowYAQAYOnQonJycsHDhQq3nbdiwAX379oW1tbUuyqbaKD3+WZi5dxqIu6Z5SAUgQiZDqE09hJrbIFTIwqPcZAB5QM4DzXl1TerCz95PMy3bzdwNYpG4yl8KEZE+03m4GTBgAOLj4zFnzhzExMTAx8cHR44c0Qwyjo6Ohlis/cP/5s2bCA4OxrFjx3RRMtUW6XHPgszdYCD+2SD3bJEIVwwUCLN2RoixGcJU6UhTZkMdZh4DAMQisWZKdsGXnVHRtZuIiKhi6Xydm6rGdW6oRGmxz6Zl3w0GEm5pHkoSixFmoECoZR2EGBnjqjIN+c+tQWMoNYSXrZdmPyYvWy8Yy1689hIREb1YjVnnhkinUh8/bZU5pR4EnHgbgHoI8AOpFCEmxgi1dECIQo4oZcHCMnlAfjIAwMbQRjMd29feFw0tG3JKNhFRNcCfxFR7pDzUHgD85A4AIB/ATbkcIWamCLWwQ4hMjERVwcKPeYBSvRllffP6Wqv+1jWpyynZRETVEMMN6a+UB0+nZT/dmykpCgCQIRLhkkKBUEsLhJpa4bJUQJamiykPUAFSsRTNrJtpVv31sfWBhYGFzl4KERGVHcMN6Y/k6GdrzNwLBpLuAgDiJBKEGCgQam2JUBML3BQr8Wxd4HxAAEzlploDf5taN4WB1EBHL4SIiF4Fww3VXEn3Cs1mOgUkR0MFIFImQ4iBAmG2NggxNsFDUeH9mtQtNE4mTlphpoFFA07JJiLSEww3VDMIgrolpvCYmZT7yBEB1+RydcuMvS1CDY2QKio8AVAFsUiMhpYNNWvL+Nj5wMG49BWwiYio5mK4oepJENRjZAqCzN3TQOoDpIjFCFM8DTOO9riiUCBPa0yvAEOpoWYLAz87P3jZesFEbqKrV0JERFWM4YaqB0EAnkQ+m5Z9NxhC2iM8lEoQaqBAiEKBUCdH3JHLijzVysBKs7aMn70fGlo1hExc9DwiIqodGG5INwQBSIwo1DITjPz0GNySy9RhxkCBUAsnxEslRZ7qauaqmY7tZ+cHZ1NnTskmIiINhhuqGoIAJNx+Ni373mlkZsThskKuDjMmCly2rovM57bakIqlaGLdRNMy42PnAysDKx29CCIiqgkYbqhyCAIQf1MdZu6dBu6eRnx2AkIVCoQaKBBqpsANm7pQPtfiYiozhbedtybMNLNpxinZRERULgw3VDFUKvXGkk+nZQt3TyMqL1ndvaRQINRSgfuyukWe5mjsqOle8rHzgbuFOyTiol1RREREZcVwQy9HpQLirz8dL3MKuffO4JoyTTP4N8xGgWRJHa2niCCCp6Wn1voyjiaOOnoBRESkrxhuqGxUKiDuqmbwb0r0aVwSsjSDf6/YGiFXrL0DtoFEgea2XvCx9YGfvR+8bb1hKjfV0QsgIqLaguGGiqdSArFXgLunIUSdwuOHZxGCHE2YibA3BaAdVKwUlvAptLFkY6vGkEk4JZuIiKoWww2pqZRATDhwNxjKqFO4/fg8QkR5mjATZ1c0zLiY1oOvvZ9m8K+LmQunZBMRkc4x3NRWynwg5jJwNxiZd0/hSsy/CJHkI1ShwCUDBTJstYOMVCRBY6vGmjDjbecNG0MbHRVPRERUMoab2kKZDzy+BNwLRkLUCYTFhSFUqkKoQoHrCjnybbS3JzCWGMDHzk8dZuz90MymGQylhjoqnoiIqOwYbvSVMg94fAlC1EncvXscYYlXECIFQg0UuCeTAdbaYcZOYQl/xwBNywynZBMRUU3FcKMvlHnAo1DkRZ3A9bsnEJp0AyEyEcIMFHgikQCWz8KMCIC7iTP86rSGr72/ekq2sSPHyxARkV5guKmp8nOBRyFIi/wLl+6dQEjqHYTKxAhXyJEjFgMWz6Zly0USNLfwgJ9TIHyeTsk2V5jrsHgiIqLKw3BTU+TnAA8vIibiGEIenERIWjRC5RLclssgiESAmZHmVAuJAXytm8G3bnv42vuhiXUTyCVyHRZPRERUdRhuqqv8HCjvX0DE7UMIfXwWIRmPEKqQIkb69K/M9Nng3noyc/VCefU6wdfBD25mbuxiIiKiWovhprrIy0b2vdMIjziI0Jh/EZIdi8tyGdIkT3fJNlZvHikB0MjADr4OLeDn8jp87f04JZuIiKgQhhtdycvCkztBCL1zGKHxYQjNTcQ1uQz5IpF6xK+hAgBgBDG8jevCt04b+Ll2QXNbLxjJjEq/NhERUS3GcFNFhJwMREccQsidIwhLvIoQZSruyp6+/SIACvWYGFuRDH6mburxMm5d4WnVEFIx/5qIiIjKip+alSQvOwU3buxD6N0ghCbdRIiQoZ6SDQBiAE8Di7vYEL7mHvCt9xp863eHk2ldjpchIiJ6BQw3FSQzIx7rfv8f8pWhuJZ5F+HIRpb46XgZMQBIIBMENJeawceyMfzcusKn/hswN7DQYdVERET6h+GmguwIWon1GXvVfxCr/8dMJcBXZgVfWy/4NeiBJq6vQyE10GWZREREeo/hpoK09x2IXUf3wDbLBMosVzSv3wcfv/Uu5DKZrksjIiKqVcS6LkBfeDo3xKERl9HIbSPOPXkf6/41x9BN/yIuLVvXpREREdUqDDcVSCYRY3avJlg5yA/GcgnORT5Br2XBuBD1RNelERER1RoMN5Wgp5cjfp3cDp72JohLy8HAdeew9uQdCIKg69KIiIj0HsNNJWlga4JfJgain68TlCoBCw7fwLifLiI1O0/XpREREek1hptKZCSX4vv+3viqbzPIJWIcvRqLPsuDcf1xqq5LIyIi0lsMN5VMJBJhSGsX7B7XBk4WhribmIl+q05jz8UHui6NiIhILzHcVBFvZwscnNwOnRraIjtPhem7L2HmvsvIzlPqujQiIiK9wnBThSyN5dg4rCWmdfWESATsuHAf76w5g/tPMnVdGhERkd5guKliYrEIU173wI8jW8HKWI4rD1PRc9kpBF2P1XVpREREeoHhRkfae9ji4OR28K1ngdTsfIza8i++PXID+UqVrksjIiKq0RhudKiOhSF2jW2D4W1dAQCrTtzB+xsuID4tR7eFERER1WAMNzoml4rxZZ+mWD7QF0ZyCc5GJqLX8lP45y5XNSYiInoZDDfVRG/vOvh1UiDc7UwQm5qD99aew/pTkVzVmIiIqJwYbqoRdztTHJgYiD7edaBUCfjq0HVM3B6CNK5qTEREVGYMN9WMsUKKH97zwby3mkImEeFweAzeWnEaN2K4qjEREVFZMNxUQyKRCEPbuOLnD9qgjrkBIhMy0HflaewL4arGREREL8JwU4351rPEwSnt0cFTvarxtJ8v4bP94VzVmIiIqBQMN9WclbEcm4a3xNQuHhCJgO3no/HumrNc1ZiIiKgEDDc1gEQswtQuntg8ohUsjWQIf5iCXsuDcfxGnK5LIyIiqnYYbmqQjp62ODilPbydLZCSlYcRm//B4mM3oVRxujgREVEBhpsaxsnCED9/0BpD27gAAJb/FYFhGy8gMZ2rGhMREQHVINysXLkSrq6uMDAwQEBAAC5cuFDq+cnJyZg4cSIcHR2hUCjg6emJw4cPV1G11YNCKsG8t5rhh/d8YCiTIDgiAT2XBePivSRdl0ZERKRzOg03u3btwrRp0/DFF18gJCQE3t7e6N69O+Liih9Lkpubi65du+Lu3bvYs2cPbt68iXXr1sHJyamKK68e3vJxwq+TAtHA1hgxqdkY8L+z2BgcxVWNiYioVhMJOvwkDAgIQMuWLbFixQoAgEqlgrOzMyZPnoxPP/20yPlr1qzBokWLcOPGDchkspe6Z2pqKszNzZGSkgIzM7NXqr+6SM/Jx6d7L+Pg5ccAgJ5ejvjmP14wUUh1XBkREVHFKM/n9yu13GRnZ7/0c3Nzc3Hx4kV06dLlWTFiMbp06YKzZ88W+5xff/0Vbdq0wcSJE2Fvb49mzZphwYIFUCpr97ovJgoplg/0xZe9m0AqFuHQ5cfosyIYt2LTdF0aERFRlSt3uFGpVJg/fz6cnJxgYmKCyMhIAMDs2bOxYcOGMl8nISEBSqUS9vb2Wsft7e0RExNT7HMiIyOxZ88eKJVKHD58GLNnz8bixYvx1VdflXifnJwcpKaman3pI5FIhOGBbtj1QRs4mhsgMj4Db604jV9CH+q6NCIioipV7nDz1VdfYfPmzfj2228hl8s1x5s1a4b169dXaHHPU6lUsLOzw9q1a+Hv748BAwZg1qxZWLNmTYnPWbhwIczNzTVfzs7OlVqjrvm7WOLg5HZo526DrDwlpu4Kw+xfriAnv3a3bhERUe1R7nDz448/Yu3atRg8eDAkEonmuLe3N27cuFHm69jY2EAikSA2NlbreGxsLBwcHIp9jqOjIzw9PbXu27hxY8TExCA3N7fY58ycORMpKSmar/v375e5xprK2kSBLSNbYUpndwDA1nP30H/NWTxI4qrGRESk/8odbh4+fAh3d/cix1UqFfLy8sp8HblcDn9/fwQFBWldIygoCG3atCn2OYGBgYiIiIBKpdIcu3XrFhwdHbVakQpTKBQwMzPT+qoNJGIRpnVriE0jWsLCSIZLD9SrGp+4yVWNiYhIv5U73DRp0gSnTp0qcnzPnj3w9fUt17WmTZuGdevWYcuWLbh+/TrGjx+PjIwMjBgxAgAwdOhQzJw5U3P++PHj8eTJE3z44Ye4desWDh06hAULFmDixInlfRm1xmsN7XBwcjt41TVHcqZ6VePv/7jFVY2JiEhvlXuu8Jw5czBs2DA8fPgQKpUK+/btw82bN/Hjjz/i4MGD5brWgAEDEB8fjzlz5iAmJgY+Pj44cuSIZpBxdHQ0xOJn+cvZ2RlHjx7F//3f/8HLywtOTk748MMPMWPGjPK+jFqlrqURdo9rg/kHr+Gnc9FYFnQbodFJ+OE9X1gZF9/iRUREVFO91Do3p06dwrx583Dp0iWkp6fDz88Pc+bMQbdu3Sqjxgqlj+vclMf+0Af4bN8VZOUpUcfcACsH+8G3nqWuyyIiIipVeT6/dbqIny7U9nADADdj0jD+p4uITMiATCLC5z2bYGgbF4hEIl2XRkREVKwqW8SPaqaGDqY4MCkQbzZ3QJ5SwBe/XsWUnWHIyMnXdWlERESvrNzhRiwWQyKRlPhFNYOpgQwrB/lhdi/1qsa/XXqEt1aeRkQcVzUmIqKardwDivfv36/157y8PISGhmLLli2YO3duhRVGlU8kEmFUOzd41zXHxO0hiIhLR58Vp/Hf/3ihj3cdXZdHRET0UipszM327duxa9cuHDhwoCIuV2k45qZ4Cek5mLIjFGfuJAIAhrVxwayeTSCXsueSiIh0Tydjblq3bq21IB/VLDYmCmwdFYBJr6kXaNxy9h76/+8sHiVn6bgyIiKi8qmQcJOVlYVly5bBycmpIi5HOiIRizC9e0NsGNYCZgZShN1PRs9lp3DyVryuSyMiIiqzco+5sbS01JoyLAgC0tLSYGRkhJ9++qlCiyPdeL2xPQ5NaY/x2y7iysNUDNt0AVNf98Tkzu4QizldnIiIqrdyj7nZvHmzVrgRi8WwtbVFQEAALC2r/2JwHHNTdtl5Ssz97Rp2XIgGAHT0tMXSAT6w5KrGRERUxbiIXykYbspv78UHmPVLOLLzVHCyMMTKwX7wcbbQdVlERFSLVHi4uXz5cplv7uXlVeZzdYHh5uVcf5yKCdtCEPV0VeM5vZpgSGuuakxERFWjwsONWCyGSCTCi04ViURQKpXlq7aKMdy8vNTsPHyy+zKOXI0BAPT1qYMFbzeHkbzcQ7eIiIjKpTyf32X6VIqKiqqQwqhmMzOQYfUQP6w/FYX/HrmBX8Ie4eqjVKwe4g93OxNdl0dERASAY250XU6NdSHqCSZtD0FcWg6M5RJ8+443eno56rosIiLSU1UyoPjatWuIjo5Gbm6u1vE+ffq8zOWqDMNNxYlLy8aUHaE4F/kEADAi0BUzezTmqsZERFThKjXcREZGol+/fggPD9cah1MwsJRjbmqXfKUKi/+4hdUn7gAA/OpZYOVgPziaG+q4MiIi0ieVuv3Chx9+CDc3N8TFxcHIyAhXr17FyZMn0aJFC5w4ceJla6YaSioRY8YbjbBuaAuYGkgREp2MXsuCEXw7QdelERFRLVXucHP27FnMmzcPNjY2EIvFEIvFaNeuHRYuXIgpU6ZURo1UA3RtYo+Dk9uhiaMZEjNy8f7G81jx122oVLVqSBcREVUD5Q43SqUSpqamAAAbGxs8evQIAODi4oKbN29WbHVUo7hYG2PfhLYY0MIZggB8d+wWRm35B8mZuS9+MhERUQUpd7hp1qwZLl26BAAICAjAt99+i9OnT2PevHmoX79+hRdINYuBTIJv3vHCt+94QSEV4/jNePRcFozLD5J1XRoREdUS5Q43n3/+OVQqFQBg3rx5iIqKQvv27XH48GEsW7aswgukmql/C2fsm9AWLtZGeJichXdWn8W28/deuBAkERHRqyrzbKkWLVpg9OjRGDRoUJFRyk+ePCmyW3h1xdlSVSslKw8f776EY9diAQBv+zrh637NYSiX6LgyIiKqSSpltpS3tzc++eQTODo6YujQoVozo6ysrGpEsKGqZ24ow//e98fMHo0gEYuwL/Qh+q48jcj4dF2XRkREeqrM4WbDhg2IiYnBypUrER0djddffx3u7u5YsGABHj58WJk1Ug0nEonwQccG2DY6ALamCtyMTUOfFafxe/hjXZdGRER66KVXKL5z5w42bdqErVu34tGjR+jWrRtGjRqFt99+u6JrrFDsltKtuNRsTNoRigtR6lWNR7Vzw6c9GkEm4arGRERUsirZfqGAIAjYu3cvPvjgAyQnJ3OFYnqhfKUKi47exP9ORgIAWrhYYsUgPziYG+i4MiIiqq4qdYXiwk6cOIHhw4dj+PDhUCqVGDNmzKtcjmoJqUSMmW82xv/e94epQop/7yWh1/JTOHOHqxoTEdGrK3e4efDgAb766iu4u7ujc+fOuHv3LlatWoXHjx9jzZo1lVEj6anuTR3w2+R2aORgioT0XAxZfx4rj0dwVWMiInolZe6W+vnnn7Fx40YEBQXBzs4Ow4YNw8iRI+Hu7l7ZNVYodktVP9l5Ssz+5Qp2X3wAAHi9kR2+7+8DcyOZjisjIqLqolLG3MjlcvTs2ROjRo3Cm2++CbG4Zg4AZbipvnb9E43ZB64iN18FZytDrB7sj2ZO5roui4iIqoFKCTdxcXGws7OrkAJ1ieGmervyMAXjt13E/SdZkEvFmNenKQa0dOY6SkREtVylDCjWh2BD1V8zJ3McnNQeXRrbITdfhU/3hePjPZeRlVu9Z+EREVH1UTP7lkivmRvJsPb9FvjkjYYQi4A9Fx+g36rTuJuQoevSiIioBmC4oWpJLBZhQid3/DQ6ADYmctyISUPv5cE4ciVG16UREVE1x3BD1VrbBjY4NKU9WrpaIi0nH+N+uogFh68jX6nSdWlERFRNlTvc/PPPPzh//nyR4+fPn8e///5bIUURFWZvZoDtY1pjTHs3AMDak5EYtO484lKzdVwZERFVR+UONxMnTsT9+/eLHH/48CEmTpxYIUURPU8mEWNWzyZYPdgPJgopLtx9gjeXBeNcZKKuSyMiomqm3OHm2rVr8PPzK3Lc19cX165dq5CiiErSo7kjfp0U+HRV4xwMWncOq0/cwStukUZERHqk3OFGoVAgNja2yPHHjx9DKpVWSFFEpalva4L9EwLxtp8TVALwzZEbGLv1IlKy8nRdGhERVQPlDjfdunXDzJkzkZKSojmWnJyMzz77DF27dq3Q4ohKYiiXYPG73ljQrznkEjH+uBaLPiuCcfVRyoufTEREeq3MKxQXePjwITp06IDExET4+voCAMLCwmBvb48//vgDzs7OlVJoReEKxfon/IF6VeMHSVlQSMWY/1Yz9G9Zvb8PiYiofCpl+4XCMjIysG3bNly6dAmGhobw8vLCwIEDIZNV/40OGW70U3JmLqb9fAl/3YgDAPRvURfz3moGA5lEx5UREVFFqPRwU5Mx3OgvlUrA6r/vYPGxm1AJQBNHM6we4gcXa2Ndl0ZERK+owsPNr7/+ih49ekAmk+HXX38t9dw+ffqUr9oqxnCj/05HJGDKjlAkZuTC1ECKxe96o1tTB12XRUREr6DCw41YLEZMTAzs7OwgFpc8BlkkEkGprN4bHDLc1A4xKdmYuD0EF+8lAQA+6FgfH3drCKmEi3ITEdVEFb4ruEql0uwKrlKpSvyq7sGGag8HcwPsHNsaIwPVqxr/7+9IDNlwHnFpXNWYiEjflevX2Ly8PLz++uu4fft2ZdVDVGFkEjHm9G6ClYP8YCyX4FzkE/RaFowLUU90XRoREVWicoUbmUyGy5cvV1YtRJWip5cjfp3cDp72JohLy8HAdeew9iRXNSYi0lflHoAwZMgQbNiwoTJqIao0DWxN8MvEQPTzdYJSJWDB4RsY99NFpGZzVWMiIn1T7v0S8vPzsXHjRvz555/w9/eHsbH2NNvvv/++woojqkhGcim+7+8NPxdLzP/tGo5ejcXNmGCsHuKPxo4cXE5EpC/Kvc7Na6+9Vurjx48ff6WCKhtnSxEAXLqfjAnbQvAwOQsGMjG+6tsc7/jX1XVZRERUAi7iVwqGGyqQlJGLqbvC8PeteADAwFbO+KJ3U65qTERUDVX4VPDCRo4cibS0tCLHMzIyMHLkyPJeDgCwcuVKuLq6wsDAAAEBAbhw4UKJ527evBkikUjry8DA4KXuS7WbpbEcm4a3xLSunhCJgB0X7uOdNWdw/0mmrksjIqJXUO5ws2XLFmRlZRU5npWVhR9//LHcBezatQvTpk3DF198gZCQEHh7e6N79+6Ii4sr8TlmZmZ4/Pix5uvevXvlvi8RAIjFIkx53QM/jmwFSyMZrjxMRc9lpxB0PVbXpRER0Usqc7hJTU1FSkoKBEFAWloaUlNTNV9JSUk4fPiwZqG/8vj+++8xZswYjBgxAk2aNMGaNWtgZGSEjRs3lvgckUgEBwcHzZe9vX2570tUWHsPWxya0h6+9SyQmp2PUVv+xbdHbiBfqdJ1aUREVE5lDjcWFhawsrKCSCSCp6cnLC0tNV82NjYYOXIkJk6cWK6b5+bm4uLFi+jSpcuzgsRidOnSBWfPni3xeenp6XBxcYGzszPeeustXL16tcRzc3JytIJYampquWqk2qOOhSF2jW2D4W1dAQCrTtzB+xsuID4tR7eFERFRuZR5Kvjx48chCAI6d+6MvXv3wsrKSvOYXC6Hi4sL6tSpU66bJyQkQKlUFml5sbe3x40bN4p9TsOGDbFx40Z4eXkhJSUF3333Hdq2bYurV6+ibt2is10WLlyIuXPnlqsuqr3kUjG+7NMUfi6W+HTvZZyNTESv5aewYpAfWrpavfgCRESkc+WeLXXv3j3Uq1cPIpHolW/+6NEjODk54cyZM2jTpo3m+CeffIK///4b58+ff+E18vLy0LhxYwwcOBDz588v8nhOTg5ycp795p2amgpnZ2fOlqIXiohLw7ifQhARlw6JWISZPRphVDu3CvneJyKi8qnU2VIuLi4IDg7GkCFD0LZtWzx8+BAAsHXrVgQHB5frWjY2NpBIJIiN1R68GRsbCwcHhzJdQyaTwdfXFxEREcU+rlAoYGZmpvVFVBbudqY4MDEQfbzrQKkS8NWh65iwLQRpXNWYiKhaK3e42bt3L7p37w5DQ0OEhIRoWkVSUlKwYMGCcl1LLpfD398fQUFBmmMqlQpBQUFaLTmlUSqVCA8Ph6OjY7nuTVQWxgopfnjPB/PeagqZRITfr8Sgz4rTuBHDsVtERNVVucPNV199hTVr1mDdunWQyWSa44GBgQgJCSl3AdOmTcO6deuwZcsWXL9+HePHj0dGRgZGjBgBABg6dChmzpypOX/evHk4duwYIiMjERISgiFDhuDevXsYPXp0ue9NVBYikQhD27ji5w/aoI65AaISMtB35WnsC3mg69KIiKgY5d5b6ubNm+jQoUOR4+bm5khOTi53AQMGDEB8fDzmzJmDmJgY+Pj44MiRI5pBxtHR0RCLn2WwpKQkjBkzBjExMbC0tIS/vz/OnDmDJk2alPveROXhW88SB6e0x4c7Q3HqdgKm/XwJ/95LwpxeTbiqMRFRNVLuAcX169fH2rVr0aVLF5iamuLSpUuoX78+fvzxR/z3v//FtWvXKqvWCsHtF+hVKVUClgXdxrK/bkMQgOZO5lg12A/OVka6Lo2ISG9V6oDiMWPG4MMPP8T58+chEonw6NEjbNu2DdOnT8f48eNfumiimkIiFuH/unpi0/CWsDCSIfxhCnotD8bxGyWvqk1ERFWn3C03giBgwYIFWLhwITIz1XvwKBQKTJ8+vdip2NUNW26oIj1MzsKEbSG4dD8ZADC5szumdvGERMzp4kREFalKdgXPzc1FREQE0tPT0aRJE5iYmLxUsVWN4YYqWk6+El8fuo4fz6r3OGvnboMf3vOBtYlCx5UREemPKgk3NRXDDVWWA2EP8enecGTlKeFgZoCVg/3g72Kp67KIiPRCpYSbkSNHlunmpW14WR0w3FBluhWbhnE/XURkfAakYhE+e7MxRgS6clVjIqJXVCnhRiwWw8XFBb6+vijtKfv37y9ftVWM4YYqW3pOPmbsvYxDlx8DAHp6OeKb/3jBRFHulReIiOip8nx+l/mn7fjx47Fjxw5ERUVhxIgRGDJkiNbmmUSkZqKQYsVAX7RwscTXh67j0OXHuP44FWuG+MPT3lTX5RER6b0yTwVfuXIlHj9+jE8++QS//fYbnJ2d0b9/fxw9erTUlhyi2kgkEmFEoBt2fdAGDmYGiIzPwFsrTuOX0Ie6Lo2ISO+99IDie/fuYfPmzfjxxx+Rn5+Pq1ev1ogZU+yWoqqWmJ6DD3eGITgiAQDwfmsXfN6rMRRSrmpMRFRWlbqIn+aJYjFEIhEEQYBSqXzZyxDpPWsTBbaMbIUpnd0BAFvP3UP/NWfxIClTx5UREemncoWbnJwc7NixA127doWnpyfCw8OxYsUKREdH14hWGyJdkYhFmNatITYNbwlzQxkuPVCvanziJlc1JiKqaGXulpowYQJ27twJZ2dnjBw5EoMHD4aNjU1l11fh2C1Funb/SSYmbg/B5QcpEImAyZ098OHrHlzVmIioFJU2FbxevXrw9fUtdc2Offv2la/aKsZwQ9VBTr4S8367hm3nowEA7T1s8MN7vrAyluu4MiKi6qlSpoIPHTqUC5ERVRCFVIKv+zWHv4slPtsfjlO3E9Bz2SmsHOwHv3pc1ZiI6FVw+wUiHbsZk4bxP11EZEIGZBIRZr3ZGMPaclVjIqLCqmS2FBFVjIYOpjgwKRBvNndAnlLAl79dw5SdYcjIydd1aURENRLDDVE1YGogw8pBfpjdqwmkYhF+u/QIb608jYi4NF2XRkRU4zDcEFUTIpEIo9q5YefY1rA3UyAiLh19VpzGr5ce6bo0IqIaheGGqJpp4WqFQ1Pao20Da2TmKjFlRyi+OHAFufkqXZdGRFQjMNwQVUM2JgpsHRWAia81AABsOXsP/f93Fo+Ss3RcGRFR9cdwQ1RNScQifNy9ETYMawEzAynC7iej57JTOHkrXtelERFVaww3RNXc643tcWhKezRzMkNSZh6GbbqAH/68DZWqVq3iQERUZgw3RDWAs5UR9oxri4GtnCEIwJI/b2HE5n+QlJGr69KIiKodhhuiGsJAJsHCt73w3bveUEjF+PtWPHotD0bY/WRdl0ZEVK0w3BDVMO/418UvEwPham2Eh8lZeHfNGWw9exe1bLFxIqISMdwQ1UCNHc3w6+R26N7UHnlKAbMPXMX/7QpDZi5XNSYiYrghqqHMDGRYM8Qfs95sDIlYhF/CHuGtFacREZeu69KIiHSK4YaoBhOJRBjToT52jGkNO1MFbsel460VwTh4masaE1HtxXBDpAdauVnh4JR2aF3fChm5SkzaHoq5v13lqsZEVCsx3BDpCTtTA/w0KgDjOqpXNd50+i7eW3sWj1O4qjER1S4MN0R6RCoR49MejbBuaAuYGkgREp2MnsuCEXw7QdelERFVGYYbIj3UtYk9Dk5uhyaOZniSkYv3N57Hir+4qjER1Q4MN0R6ysXaGPsmtMWAFupVjb87dgujtvyD5EyuakxE+o3hhkiPGcgk+OYdL3z7jhcUUjGO34xHz2XBuPwgWdelERFVGoYbolqgfwtn7JvQFvWs1Ksav7P6LLadv8dVjYlILzHcENUSTeuY47fJ7dC1iT1ylSrM2n8FH/18CVm5Sl2XRkRUoRhuiGoRc0MZ1r7vj097NIJYBOwLfYi+K08jMp6rGhOR/mC4IaplRCIRxnVsgO1jWsPGRIGbsWnos+I0fg9/rOvSiIgqBMMNUS3Vur41Dk9ph1auVkjPycf4bSGYf/Aa8pRc1ZiIajaGG6JazM7MANvHBOCDDvUBABuCozBw7TnEpGTruDIiopfHcENUy0klYsx8szHWDPGHqUKKf+8lodfyUzhzh6saE1HNxHBDRACAN5o54NfJ7dDIwRQJ6bkYsv48Vh6P4KrGRFTjMNwQkYabjTH2TwjEO/51oRKARUdvYsyP/yIlM0/XpRERlRnDDRFpMZRLsOgdL/z37eaQS8UIuhGHXitO4crDFF2XRkRUJgw3RFSESCTCe63qYd/4tnC2MsT9J1l4e/UZ7LgQzVWNiajaY7ghohI1czLHwUnt0aWxHXLzVZi5LxzTd1/mqsZEVK0x3BBRqcyNZFj7fgt83L0hxCJgb8gD9Ft1GlEJGboujYioWAw3RPRCYrEIE19zx0+jA2BjIseNmDT0WR6MI1didF0aEVERDDdEVGZtG9jg4OT2aOFiibScfIz76SIWHL6OfK5qTETVCMMNEZWLg7kBdoxtjdHt3AAAa09GYtC684hL5arGRFQ9MNwQUbnJJGJ83qsJVg/2g4lCigt3n+DNZcE4F5mo69KIiBhuiOjl9WjuiF8nBaKhvSkS0nMwaN05rD5xh9PFiUinqkW4WblyJVxdXWFgYICAgABcuHChTM/buXMnRCIR+vbtW7kFElGJ6tuaYP/Etnjb1wkqAfjmyA2M3XoRKVlc1ZiIdEPn4WbXrl2YNm0avvjiC4SEhMDb2xvdu3dHXFxcqc+7e/cupk+fjvbt21dRpURUEiO5FIv7e2NBv+aQS8T441os+qwIxtVHXNWYiKqezsPN999/jzFjxmDEiBFo0qQJ1qxZAyMjI2zcuLHE5yiVSgwePBhz585F/fr1q7BaIiqJSCTCoIB62DO+DZwsDHEvMRNvrzqDn/+5r+vSiKiW0Wm4yc3NxcWLF9GlSxfNMbFYjC5duuDs2bMlPm/evHmws7PDqFGjXniPnJwcpKaman0RUeXxqmuBQ1Pa4bWGtsjJV+GTvZfxyZ5LyM7jqsZEVDV0Gm4SEhKgVCphb2+vddze3h4xMcUvDhYcHIwNGzZg3bp1ZbrHwoULYW5urvlydnZ+5bqJqHQWRnJsGNYS07t5QiwCfv73Ad5edQb3ErmqMRFVPp13S5VHWloa3n//faxbtw42NjZles7MmTORkpKi+bp/n03kRFVBLBZhUmcPbB0VAGtjOa49TkWv5cE4dpWrGhNR5ZLq8uY2NjaQSCSIjY3VOh4bGwsHB4ci59+5cwd3795F7969NcdUKvXKqFKpFDdv3kSDBg20nqNQKKBQKCqheiIqi0B3Gxyc0g4Tt4UgJDoZY7dexAcd6+Pjbg0hldSo36+IqIbQ6U8WuVwOf39/BAUFaY6pVCoEBQWhTZs2Rc5v1KgRwsPDERYWpvnq06cPXnvtNYSFhbHLiaiacjQ3xM6xbTAi0BUA8L+/IzF4/XnEpXFVYyKqeDptuQGAadOmYdiwYWjRogVatWqFpUuXIiMjAyNGjAAADB06FE5OTli4cCEMDAzQrFkzredbWFgAQJHjRFS9yKVifNG7KfxdLDFjz2Wcj3qCnsuCsXKQH1q5Wem6PCLSIzoPNwMGDEB8fDzmzJmDmJgY+Pj44MiRI5pBxtHR0RCL2XRNpC96edVBIwczTNh2Ebdi0zFw3TnMeKMhxrSvD5FIpOvyiEgPiIRatk56amoqzM3NkZKSAjMzM12XQ1RrZebm47N94fgl7BEAoHtTeyx61xtmBjIdV0ZE1VF5Pr/ZJEJEOmEkl2LJAB/M79sMcokYR6/Gos/yYFx/zLWoiOjVMNwQkc6IRCK839oFu8epVzW+m5iJfqtOY8/FB7oujYhqMIYbItI5b2cLHJzcDh09bZGdp8L03Zcwc99lrmpMRC+F4YaIqgVLYzk2DW+JaV09IRIBOy7cxztrzuD+k0xdl0ZENQzDDRFVG2KxCFNe98CWEa1gaSTDlYep6LnsFIKux774yURETzHcEFG108HTFoemtIePswVSs/Mxasu/+PbIDeQrVboujYhqAIYbIqqW6lgY4ucP2mB4W1cAwKoTd/D+hguIT8vRbWFEVO0x3BBRtSWXivFln6ZYNtAXRnIJzkYmotfyU/jn7hNdl0ZE1RjDDRFVe3286+DXSYFwtzNBbGoO3lt7DutPRaKWrUFKRGXEcENENYK7nSkOTAxEb+86UKoEfHXoOiZsC0Fadp6uSyOiaobhhohqDGOFFMve88HcPk0hk4jw+5UY9FlxGjdiuKoxET3DcENENYpIJMKwtq7Y9UEb1DE3QFRCBvquPI19IVzVmIjUGG6IqEbyq2eJg1Pao72HDbLzVJj28yV8tj+cqxoTEcMNEdVcVsZybB7RCh++7gGRCNh+Phr9Vp3B5tNRuBOfzgHHRLWUSKhl//rLs2U6EdUcJ27GYequMCRnPhtg7GRhiPYeNmjvYYtAd2tYGMl1WCERvYryfH4z3BCR3ohNzca+kIcIjojHP1FJyC20orFIBHjVtUCHp2HHt54FZBI2XhPVFAw3pWC4IaodsnKVOB+ViFO3E3DqdjxuxaZrPW4sl6BNA2u097BFew8buNkYQyQS6ahaInoRhptSMNwQ1U4xKdk4dTsep24nIDgiAU8ycrUed7IwRAfPp11YDWxgbiTTUaVEVByGm1Iw3BCRSiXg2uNUnLwdj1O3EvDvvSfIUz77USgu3IXlaQsfZ3ZhEekaw00pGG6I6HmZufk4H/kEJ2/HI/h2Am7HaXdhmSikaNPAWjNex8XaiF1YRFWM4aYUDDdE9CKPU7KejtVJQPDteCRlam/x4GxliPYetujgYYM2DWxgbsguLKLKxnBTCoYbIioPlUrA1UdPu7Bux+PivaQiXVg+zhbqsONpA++6FpCyC4uowjHclILhhoheRUZOPs5HJeLkLfUsrDvxGVqPmz7twmrvqW7ZcbE21lGlRPqF4aYUDDdEVJEeJmch+HY8Tt5OwOmIBK1FBAGgnpWRZiHBtu7WMDNgFxbRy2C4KQXDDRFVFqVKwJWHKTj1NOyE3EtCvurZj1iJWPS0C0sddrzrmrMLi6iMGG5KwXBDRFUlPScf5+4kqtfXiUhA5PNdWAZSBDawQXtPG3TwsIWzlZGOKiWq/hhuSsFwQ0S68iApE8EFs7AiEpCSpd2F5WptpFkxuU0Da5iyC4tIg+GmFAw3RFQdKFUCwh+m4NQt9arJIdFFu7B8n87Cau9pAy8ndmFR7cZwUwqGGyKqjtKy83Au8olmi4ioBO0uLDMDKQLdbTQtO+zCotqG4aYUDDdEVBPcf5Kp2fTzdEQCUrPztR53szHWDExuXd+KXVik9xhuSsFwQ0Q1Tb5ShcsPU56O14lHSHQylIW6sKRiEfzqWarDjqctmjuZQyLm9hCkXxhuSsFwQ0Q1XWp23tNZWOqwczcxU+txc0MZ2rnboJ2HDdp72KCuJbuwqOZjuCkFww0R6ZvoxEycilDvcH76TgLSnuvCql+4C6uBNUwUUh1VSvTyGG5KwXBDRPosX6nCpQcpmoHJYfeL6cJysdTscN6MXVhUQzDclILhhohqk5SsPJwtWEjwdgKin2h3YVkYyRDoboMOHjZo52ELJwtDHVVKVDqGm1Iw3BBRbXYvMUMzVudMRCLScrS7sBrYGmt2OA9ws4Yxu7CommC4KQXDDRGRmroLK1mzw3nY/WQU6sGCTKKehdXBU722TrM65hCzC4t0hOGmFAw3RETFU3dhJeDk7QScvBWPB0lZWo9barqwbNHOwwZ12IVFVYjhphQMN0RELyYIAu4lZmp2OD97JxHpz3VhuduZoL2HOuwE1LeCkZxdWFR5GG5KwXBDRFR+eUoVwu4n49Qtddi5/KBoF1YLFyvNDudNHM3YhUUViuGmFAw3RESvLiUzD2cKdWE9TNbuwrIylqOdu41mfR0HcwMdVUr6guGmFAw3REQVSxAERCVkIDgiASdvJeDsnQRk5Cq1zvGwM9HscB7gxi4sKj+Gm1Iw3BARVa48pQqh0cma8TqXHySj8CeNXCJGC1dLzQ7n7MKismC4KQXDDRFR1UrKyMWZpwsJnrwVj0cp2VqPWxvLn+6DpQ479mbswqKiGG5KwXBDRKQ7giAgMiEDp26pV0w+G5mIzOe6sBram2p2OG/lagVDuURH1VJ1wnBTCoYbIqLqIzdfhZDoJM32EOEPU7S7sKRitHK10gxMbuRgyi6sWorhphQMN0RE1VdSRi5O30nAqVsJOHk7Ho+f68KyMVE8DTo2aOduAzt2YdUaDDelYLghIqoZBEHAnfgMTavO2TuJyMrT7sJq5GCqadVp5WYFAxm7sPQVw00pGG6IiGqmnHwlQu4la8LOlUdFu7AC3LS7sEQidmHpC4abUjDcEBHph8T0HJy+k6gZnByTqt2FZWuqQHt3G7T3tEGguw3sTNmFVZMx3JSC4YaISP8IgoCIuHScvJ2A4NvxOBf5pEgXVmNHM3R42qrTwtWSXVg1TI0LNytXrsSiRYsQExMDb29vLF++HK1atSr23H379mHBggWIiIhAXl4ePDw88NFHH+H9998v070YboiI9F9OvhIX7yXh1O0EnLodjysPU7UeV0jFCKhvjQ4eNmjnYYOG9uzCqu5qVLjZtWsXhg4dijVr1iAgIABLly7F7t27cfPmTdjZ2RU5/8SJE0hKSkKjRo0gl8tx8OBBfPTRRzh06BC6d+/+wvsx3BAR1T4J6Tk4HZGgCTuxqTlaj9uZKtDu6Q7nge42sDVV6KhSKkmNCjcBAQFo2bIlVqxYAQBQqVRwdnbG5MmT8emnn5bpGn5+fujZsyfmz5//wnMZboiIajdBEHA7Lh0nn47VOR+ViOw8ldY5TRzNNDuc+7uwC6s6KM/nt053LsvNzcXFixcxc+ZMzTGxWIwuXbrg7NmzL3y+IAj466+/cPPmTXzzzTeVWSoREekJkUgET3tTeNqbYnT7+sjOU3dhnbwdj1O3EnDtcarm639/R8JAJkaAmzXae9igg6ctPOxM2IVVzek03CQkJECpVMLe3l7ruL29PW7cuFHi81JSUuDk5IScnBxIJBKsWrUKXbt2LfbcnJwc5OQ8a35MTU0t9jwiIqqdDGQSBLqrZ1TN7AHEp6m7sE4+nXIen5aDv2/F4+9b8cCh67A3U2j2wWrnbgNrE3ZhVTc1cs95U1NThIWFIT09HUFBQZg2bRrq16+PTp06FTl34cKFmDt3btUXSURENZKtqQJ9fZ3Q19cJgiDgZmwaTt1KwKmIBJyPTERsag72XHyAPRcfAACaOZmpw467DfxdLaGQsgtL13Q65iY3NxdGRkbYs2cP+vbtqzk+bNgwJCcn48CBA2W6zujRo3H//n0cPXq0yGPFtdw4OztzzA0REZVbdp4S/95V74V18nYCrj/W7g0wlEkQUN8K7T1s0cHDBu7swqowNWbMjVwuh7+/P4KCgjThRqVSISgoCJMmTSrzdVQqlVaAKUyhUEChYJMhERG9OgOZBO2eTh+fCSAuLVs9C+tWAk7eTkBCeg5O3IzHiZvxAAAHMwPNDuft3G1gZSzX7QuoJXTeLTVt2jQMGzYMLVq0QKtWrbB06VJkZGRgxIgRAIChQ4fCyckJCxcuBKDuZmrRogUaNGiAnJwcHD58GFu3bsXq1at1+TKIiKgWsjM1QD/fuujnWxeCIOBGTJpme4jzUU8Qk5qN3RcfYPfFBxCJgGZ1zDXbQ/i7WEIuFev6JeglnYebAQMGID4+HnPmzEFMTAx8fHxw5MgRzSDj6OhoiMXP/vIzMjIwYcIEPHjwAIaGhmjUqBF++uknDBgwQFcvgYiICCKRCI0dzdDY0QxjOzRAdp4SF6KeaMLOjZg0hD9MQfjDFKw6cQdGcgla17fWhJ0GtsbswqogOl/npqpxnRsiItKFuNRsnLqdgOAI9UKCCem5Wo/XMTdAu6dBp527DSzZhaWlRi3iV9UYboiISNdUKu0urAt3nyA3/9lCgiIR0NzpWReWXz12YTHclILhhoiIqpusXCUu3H2i2eH8Zmya1uNGcgnaFHRhedqivk3t68JiuCkFww0REVV3sU+7sE7djkfw7QQkZmh3YTlZGGpadQLdrWFhpP9dWAw3pWC4ISKimkSlEnDtcaom7Px7Nwm5Su0uLK+6FujwNOz41rOATKJ/XVgMN6VguCEioposMzf/6Swsddi5FZuu9bixXII2Daw1W0S46UkXFsNNKRhuiIhIn8SkZGsGJgdHJOBJMV1YHTyfdmE1sIG5kUxHlb4ahptSMNwQEZG+KujCKtjh/N97T5CnfPYxLy7cheVpCx/nmtOFxXBTCoYbIiKqLTJz83E+8olmh/OIOO0uLBOFFG0aWGvG67hYG1XbLiyGm1Iw3BARUW31KDkLwbcTcPJ2PE5HJCApM0/rcWcrQ82mn20a2MDcsPp0YTHclILhhoiISN2FdfWRugvr5K14hEQnFenC8nG2UIcdTxt417WAVIddWAw3pWC4ISIiKiojJx/noxJx8pZ6Ftad+Aytx02fdmG191S37LhYG1dpfQw3pWC4ISIierGHyVkIvh2Pk7cTcDoiAcnPdWHVszLSLCTY1t0aZgaV24XFcFMKhhsiIqLyUaoEXHmYglNPw07IvSTkq57FB4lY9LQLSx12vOuaV3gXFsNNKRhuiIiIXk16Tj7O3UnUrK8TmaDdheVqbYTj0ztV6Myr8nx+SyvsrkRERFQrmCik6NLEHl2a2AMA7j/JRHDEs72wmte10OmUcrbcEBERUYVRqgSkZedV+Gae5fn8rhnLEhIREVGNIBGLdL5LOcMNERER6RWGGyIiItIrDDdERESkVxhuiIiISK8w3BAREZFeYbghIiIivcJwQ0RERHqF4YaIiIj0CsMNERER6RWGGyIiItIrDDdERESkVxhuiIiISK8w3BAREZFekeq6gKomCAIA9dbpREREVDMUfG4XfI6XptaFm7S0NACAs7OzjishIiKi8kpLS4O5uXmp54iEskQgPaJSqfDo0SOYmppCJBJV6LVTU1Ph7OyM+/fvw8zMrEKvTUQvxn+DRLpXWf8OBUFAWloa6tSpA7G49FE1ta7lRiwWo27dupV6DzMzM/5gJdIh/hsk0r3K+Hf4ohabAhxQTERERHqF4YaIiIj0CsNNBVIoFPjiiy+gUCh0XQpRrcR/g0S6Vx3+Hda6AcVERESk39hyQ0RERHqF4YaIiIj0CsMNERER6RWGGyIiItIrDDcV4OTJk+jduzfq1KkDkUiEX375RdclEdUqCxcuRMuWLWFqago7Ozv07dsXN2/e1HVZRLXG6tWr4eXlpVm4r02bNvj99991Vg/DTQXIyMiAt7c3Vq5cqetSiGqlv//+GxMnTsS5c+fwxx9/IC8vD926dUNGRoauSyOqFerWrYv//ve/uHjxIv7991907twZb731Fq5evaqTejgVvIKJRCLs378fffv21XUpRLVWfHw87Ozs8Pfff6NDhw66LoeoVrKyssKiRYswatSoKr93rdtbioj0X0pKCgD1D1ciqlpKpRK7d+9GRkYG2rRpo5MaGG6ISK+oVCpMnToVgYGBaNasma7LIao1wsPD0aZNG2RnZ8PExAT79+9HkyZNdFILww0R6ZWJEyfiypUrCA4O1nUpRLVKw4YNERYWhpSUFOzZswfDhg3D33//rZOAw3BDRHpj0qRJOHjwIE6ePIm6devquhyiWkUul8Pd3R0A4O/vj3/++Qc//PAD/ve//1V5LQw3RFTjCYKAyZMnY//+/Thx4gTc3Nx0XRJRradSqZCTk6OTezPcVID09HRERERo/hwVFYWwsDBYWVmhXr16OqyMqHaYOHEitm/fjgMHDsDU1BQxMTEAAHNzcxgaGuq4OiL9N3PmTPTo0QP16tVDWloatm/fjhMnTuDo0aM6qYdTwSvAiRMn8NprrxU5PmzYMGzevLnqCyKqZUQiUbHHN23ahOHDh1dtMUS10KhRoxAUFITHjx/D3NwcXl5emDFjBrp27aqTehhuiIiISK9whWIiIiLSKww3REREpFcYboiIiEivMNwQERGRXmG4ISIiIr3CcENERER6heGGiIiI9ArDDRHVeJ06dcLUqVN1XQYRVRMMN0RERKRXGG6IiIhIrzDcEJHeOXToEMzNzbFt2zZdl0JEOsBdwYlIr2zfvh3jxo3D9u3b0atXL12XQ0Q6wJYbItIbK1euxIQJE/Dbb78x2BDVYmy5ISK9sGfPHsTFxeH06dNo2bKlrsshIh1iyw0R6QVfX1/Y2tpi48aNEARB1+UQkQ4x3BCRXmjQoAGOHz+OAwcOYPLkybouh4h0iN1SRKQ3PD09cfz4cXTq1AlSqRRLly7VdUlEpAMMN0SkVxo2bIi//voLnTp1gkQiweLFi3VdEhFVMZHAzmkiIiLSIxxzQ0RERHqF4YaIiIj0CsMNERER6RWGGyIiItIrDDdERESkVxhuiIiISK8w3BAREZFeYbghIiIivcJwQ0RERHqF4YaIiIj0CsMNERER6RWGGyIiItIr/w8fdGIz5LM/lAAAAABJRU5ErkJggg==", 1214 "text/plain": [ 1215 "<Figure size 640x480 with 1 Axes>" 1216 ] 1217 }, 1218 "metadata": {}, 1219 "output_type": "display_data" 1220 } 1221 ], 1222 "source": [ 1223 "import matplotlib.pyplot as plt\n", 1224 "\n", 1225 "# Plotting each metric\n", 1226 "for metric_name in [\"precision\", \"recall\", \"ndcg\"]:\n", 1227 " y = [evaluate_results.metrics[f\"{metric_name}_at_{k}/mean\"] for k in range(1, 4)]\n", 1228 " plt.plot([1, 2, 3], y, label=f\"{metric_name}@k\")\n", 1229 "\n", 1230 "# Adding labels and title\n", 1231 "plt.xlabel(\"k\")\n", 1232 "plt.ylabel(\"Metric Value\")\n", 1233 "plt.title(\"Metrics Comparison at Different Ks\")\n", 1234 "# Setting x-axis ticks\n", 1235 "plt.xticks([1, 2, 3])\n", 1236 "plt.legend()\n", 1237 "\n", 1238 "# Display the plot\n", 1239 "plt.show()" 1240 ] 1241 }, 1242 { 1243 "cell_type": "markdown", 1244 "metadata": { 1245 "application/vnd.databricks.v1+cell": { 1246 "cellMetadata": {}, 1247 "inputWidgets": {}, 1248 "nuid": "cac23d4b-bece-4274-836f-9ca2b7c3860d", 1249 "showTitle": false, 1250 "title": "" 1251 } 1252 }, 1253 "source": [ 1254 "### Corner case handling\n", 1255 "\n", 1256 "There are a few corner cases handle specially for each built-in metric." 1257 ] 1258 }, 1259 { 1260 "cell_type": "markdown", 1261 "metadata": { 1262 "application/vnd.databricks.v1+cell": { 1263 "cellMetadata": {}, 1264 "inputWidgets": {}, 1265 "nuid": "e05a4ede-db44-46d2-bce8-752b0ce5d807", 1266 "showTitle": false, 1267 "title": "" 1268 } 1269 }, 1270 "source": [ 1271 "#### Empty retrieved document IDs\n", 1272 "\n", 1273 "When no relevant docs are retrieved:\n", 1274 "\n", 1275 "- `mlflow.metrics.precision_at_k(k)` is defined as:\n", 1276 " * 0 if the ground-truth doc IDs is non-empty\n", 1277 " * 1 if the ground-truth doc IDs is also empty\n", 1278 "\n", 1279 "- `mlflow.metrics.ndcg_at_k(k)` is defined as:\n", 1280 " * 0 if the ground-truth doc IDs is non-empty\n", 1281 " * 1 if the ground-truth doc IDs is also empty" 1282 ] 1283 }, 1284 { 1285 "cell_type": "markdown", 1286 "metadata": { 1287 "application/vnd.databricks.v1+cell": { 1288 "cellMetadata": {}, 1289 "inputWidgets": {}, 1290 "nuid": "931a32e7-29cb-4a22-b94e-ea2bf4f0b1a7", 1291 "showTitle": false, 1292 "title": "" 1293 } 1294 }, 1295 "source": [ 1296 "#### Empty ground-truth document IDs\n", 1297 "\n", 1298 "When no ground-truth document IDs are provided:\n", 1299 "\n", 1300 "- `mlflow.metrics.recall_at_k(k)` is defined as:\n", 1301 " * 0 if the retrieved doc IDs is non-empty\n", 1302 " * 1 if the retrieved doc IDs is also empty\n", 1303 "\n", 1304 "- `mlflow.metrics.ndcg_at_k(k)` is defined as:\n", 1305 " * 0 if the retrieved doc IDs is non-empty\n", 1306 " * 1 if the retrieved doc IDs is also empty" 1307 ] 1308 }, 1309 { 1310 "cell_type": "markdown", 1311 "metadata": { 1312 "application/vnd.databricks.v1+cell": { 1313 "cellMetadata": {}, 1314 "inputWidgets": {}, 1315 "nuid": "5a1453f6-a62d-43da-b230-955841c66651", 1316 "showTitle": false, 1317 "title": "" 1318 } 1319 }, 1320 "source": [ 1321 "#### Duplicate retreived document IDs\n", 1322 "\n", 1323 "It is a common case for the retriever in a RAG system to retrieve multiple chunks in the same document for a given query. In this case, `mlflow.metrics.ndcg_at_k(k)` is calculated as follows:\n", 1324 "\n", 1325 "If the duplicate doc IDs are in the ground truth,\n", 1326 " they will be treated as different docs. For example, if the ground truth doc IDs are\n", 1327 " [1, 2] and the retrieved doc IDs are [1, 1, 1, 3], the score will be equavalent to\n", 1328 " ground truth doc IDs [10, 11, 12, 2] and retrieved doc IDs [10, 11, 12, 3].\n", 1329 "\n", 1330 "If the duplicate doc IDs are not in the ground truth, the ndcg score is calculated as normal." 1331 ] 1332 }, 1333 { 1334 "cell_type": "markdown", 1335 "metadata": { 1336 "application/vnd.databricks.v1+cell": { 1337 "cellMetadata": {}, 1338 "inputWidgets": {}, 1339 "nuid": "525ccc10-3a60-4dc9-804e-083cfa313349", 1340 "showTitle": false, 1341 "title": "" 1342 } 1343 }, 1344 "source": [ 1345 "## Step 4: Result Analysis and Visualization\n", 1346 "\n", 1347 "You can view the per-row scores in the logged \"eval_results_table.json\" in artifacts by either loading it to a pandas dataframe (shown below) or visiting the MLflow run comparison UI." 1348 ] 1349 }, 1350 { 1351 "cell_type": "code", 1352 "execution_count": null, 1353 "metadata": { 1354 "application/vnd.databricks.v1+cell": { 1355 "cellMetadata": { 1356 "byteLimit": 2048000, 1357 "rowLimit": 10000 1358 }, 1359 "inputWidgets": {}, 1360 "nuid": "32f3d5b3-245c-46b7-87ce-d85e261eac28", 1361 "showTitle": true, 1362 "title": "" 1363 } 1364 }, 1365 "outputs": [ 1366 { 1367 "data": { 1368 "application/vnd.jupyter.widget-view+json": { 1369 "model_id": "ee4bfb1998174c558e537ebb1dd737d9", 1370 "version_major": 2, 1371 "version_minor": 0 1372 }, 1373 "text/plain": [ 1374 "Downloading artifacts: 0%| | 0/1 [00:00<?, ?it/s]" 1375 ] 1376 }, 1377 "metadata": {}, 1378 "output_type": "display_data" 1379 }, 1380 { 1381 "data": { 1382 "text/html": [ 1383 "<div>\n", 1384 "<style scoped>\n", 1385 " .dataframe tbody tr th:only-of-type {\n", 1386 " vertical-align: middle;\n", 1387 " }\n", 1388 "\n", 1389 " .dataframe tbody tr th {\n", 1390 " vertical-align: top;\n", 1391 " }\n", 1392 "\n", 1393 " .dataframe thead th {\n", 1394 " text-align: right;\n", 1395 " }\n", 1396 "</style>\n", 1397 "<table border=\"1\" class=\"dataframe\">\n", 1398 " <thead>\n", 1399 " <tr style=\"text-align: right;\">\n", 1400 " <th></th>\n", 1401 " <th>question</th>\n", 1402 " <th>source</th>\n", 1403 " <th>retrieved_doc_ids</th>\n", 1404 " <th>precision_at_1/score</th>\n", 1405 " <th>precision_at_2/score</th>\n", 1406 " <th>precision_at_3/score</th>\n", 1407 " <th>recall_at_1/score</th>\n", 1408 " <th>recall_at_2/score</th>\n", 1409 " <th>recall_at_3/score</th>\n", 1410 " <th>ndcg_at_1/score</th>\n", 1411 " <th>ndcg_at_2/score</th>\n", 1412 " <th>ndcg_at_3/score</th>\n", 1413 " </tr>\n", 1414 " </thead>\n", 1415 " <tbody>\n", 1416 " <tr>\n", 1417 " <th>0</th>\n", 1418 " <td>What is the purpose of the MLflow Model Registry?</td>\n", 1419 " <td>[model-registry.html]</td>\n", 1420 " <td>[model-registry.html, introduction/index.html,...</td>\n", 1421 " <td>1</td>\n", 1422 " <td>0.5</td>\n", 1423 " <td>0.333333</td>\n", 1424 " <td>1</td>\n", 1425 " <td>1</td>\n", 1426 " <td>1</td>\n", 1427 " <td>1</td>\n", 1428 " <td>1.0</td>\n", 1429 " <td>0.919721</td>\n", 1430 " </tr>\n", 1431 " <tr>\n", 1432 " <th>1</th>\n", 1433 " <td>What is the purpose of registering a model wit...</td>\n", 1434 " <td>[model-registry.html]</td>\n", 1435 " <td>[model-registry.html, models.html, introductio...</td>\n", 1436 " <td>1</td>\n", 1437 " <td>0.5</td>\n", 1438 " <td>0.333333</td>\n", 1439 " <td>1</td>\n", 1440 " <td>1</td>\n", 1441 " <td>1</td>\n", 1442 " <td>1</td>\n", 1443 " <td>1.0</td>\n", 1444 " <td>1.000000</td>\n", 1445 " </tr>\n", 1446 " <tr>\n", 1447 " <th>2</th>\n", 1448 " <td>What can you do with registered models and mod...</td>\n", 1449 " <td>[model-registry.html]</td>\n", 1450 " <td>[model-registry.html, models.html, deployment/...</td>\n", 1451 " <td>1</td>\n", 1452 " <td>0.5</td>\n", 1453 " <td>0.333333</td>\n", 1454 " <td>1</td>\n", 1455 " <td>1</td>\n", 1456 " <td>1</td>\n", 1457 " <td>1</td>\n", 1458 " <td>1.0</td>\n", 1459 " <td>1.000000</td>\n", 1460 " </tr>\n", 1461 " <tr>\n", 1462 " <th>3</th>\n", 1463 " <td>How can you add, modify, update, or delete a m...</td>\n", 1464 " <td>[model-registry.html]</td>\n", 1465 " <td>[model-registry.html, models.html, deployment/...</td>\n", 1466 " <td>1</td>\n", 1467 " <td>0.5</td>\n", 1468 " <td>0.333333</td>\n", 1469 " <td>1</td>\n", 1470 " <td>1</td>\n", 1471 " <td>1</td>\n", 1472 " <td>1</td>\n", 1473 " <td>1.0</td>\n", 1474 " <td>1.000000</td>\n", 1475 " </tr>\n", 1476 " <tr>\n", 1477 " <th>4</th>\n", 1478 " <td>How can you deploy and organize models in the ...</td>\n", 1479 " <td>[model-registry.html]</td>\n", 1480 " <td>[model-registry.html, deployment/index.html, d...</td>\n", 1481 " <td>1</td>\n", 1482 " <td>0.5</td>\n", 1483 " <td>0.333333</td>\n", 1484 " <td>1</td>\n", 1485 " <td>1</td>\n", 1486 " <td>1</td>\n", 1487 " <td>1</td>\n", 1488 " <td>1.0</td>\n", 1489 " <td>0.919721</td>\n", 1490 " </tr>\n", 1491 " </tbody>\n", 1492 "</table>\n", 1493 "</div>" 1494 ], 1495 "text/plain": [ 1496 " question source \\\n", 1497 "0 What is the purpose of the MLflow Model Registry? [model-registry.html] \n", 1498 "1 What is the purpose of registering a model wit... [model-registry.html] \n", 1499 "2 What can you do with registered models and mod... [model-registry.html] \n", 1500 "3 How can you add, modify, update, or delete a m... [model-registry.html] \n", 1501 "4 How can you deploy and organize models in the ... [model-registry.html] \n", 1502 "\n", 1503 " retrieved_doc_ids precision_at_1/score \\\n", 1504 "0 [model-registry.html, introduction/index.html,... 1 \n", 1505 "1 [model-registry.html, models.html, introductio... 1 \n", 1506 "2 [model-registry.html, models.html, deployment/... 1 \n", 1507 "3 [model-registry.html, models.html, deployment/... 1 \n", 1508 "4 [model-registry.html, deployment/index.html, d... 1 \n", 1509 "\n", 1510 " precision_at_2/score precision_at_3/score recall_at_1/score \\\n", 1511 "0 0.5 0.333333 1 \n", 1512 "1 0.5 0.333333 1 \n", 1513 "2 0.5 0.333333 1 \n", 1514 "3 0.5 0.333333 1 \n", 1515 "4 0.5 0.333333 1 \n", 1516 "\n", 1517 " recall_at_2/score recall_at_3/score ndcg_at_1/score ndcg_at_2/score \\\n", 1518 "0 1 1 1 1.0 \n", 1519 "1 1 1 1 1.0 \n", 1520 "2 1 1 1 1.0 \n", 1521 "3 1 1 1 1.0 \n", 1522 "4 1 1 1 1.0 \n", 1523 "\n", 1524 " ndcg_at_3/score \n", 1525 "0 0.919721 \n", 1526 "1 1.000000 \n", 1527 "2 1.000000 \n", 1528 "3 1.000000 \n", 1529 "4 0.919721 " 1530 ] 1531 }, 1532 "metadata": {}, 1533 "output_type": "display_data" 1534 } 1535 ], 1536 "source": [ 1537 "eval_results_table = evaluate_results.tables[\"eval_results_table\"]\n", 1538 "eval_results_table.head(5)" 1539 ] 1540 }, 1541 { 1542 "cell_type": "markdown", 1543 "metadata": { 1544 "application/vnd.databricks.v1+cell": { 1545 "cellMetadata": {}, 1546 "inputWidgets": {}, 1547 "nuid": "cf18dd29-1017-4245-9f3b-923dbd46f742", 1548 "showTitle": false, 1549 "title": "" 1550 } 1551 }, 1552 "source": [ 1553 "With the evaluate results table, you can further visualize the well-answered questions and poorly-answered questions using topical analysis techniques." 1554 ] 1555 }, 1556 { 1557 "cell_type": "code", 1558 "execution_count": null, 1559 "metadata": { 1560 "application/vnd.databricks.v1+cell": { 1561 "cellMetadata": { 1562 "byteLimit": 2048000, 1563 "rowLimit": 10000 1564 }, 1565 "inputWidgets": {}, 1566 "nuid": "b1d9e40a-ccf6-4d6a-b24c-8cf41bbfa005", 1567 "showTitle": true, 1568 "title": "Utilitity functions" 1569 } 1570 }, 1571 "outputs": [ 1572 { 1573 "name": "stderr", 1574 "output_type": "stream", 1575 "text": [ 1576 "[nltk_data] Downloading package punkt to\n", 1577 "[nltk_data] /Users/liang.zhang/nltk_data...\n", 1578 "[nltk_data] Package punkt is already up-to-date!\n", 1579 "[nltk_data] Downloading package stopwords to\n", 1580 "[nltk_data] /Users/liang.zhang/nltk_data...\n", 1581 "[nltk_data] Package stopwords is already up-to-date!\n" 1582 ] 1583 } 1584 ], 1585 "source": [ 1586 "import nltk\n", 1587 "import pyLDAvis.gensim_models as gensimvis\n", 1588 "from gensim import corpora, models\n", 1589 "from nltk.corpus import stopwords\n", 1590 "from nltk.tokenize import word_tokenize\n", 1591 "\n", 1592 "# Initialize NLTK resources\n", 1593 "nltk.download(\"punkt\")\n", 1594 "nltk.download(\"stopwords\")\n", 1595 "\n", 1596 "\n", 1597 "def topical_analysis(questions: list[str]):\n", 1598 " stop_words = set(stopwords.words(\"english\"))\n", 1599 "\n", 1600 " # Tokenize and remove stop words\n", 1601 " tokenized_data = []\n", 1602 " for question in questions:\n", 1603 " tokens = word_tokenize(question.lower())\n", 1604 " filtered_tokens = [word for word in tokens if word not in stop_words and word.isalpha()]\n", 1605 " tokenized_data.append(filtered_tokens)\n", 1606 "\n", 1607 " # Create a dictionary and corpus\n", 1608 " dictionary = corpora.Dictionary(tokenized_data)\n", 1609 " corpus = [dictionary.doc2bow(text) for text in tokenized_data]\n", 1610 "\n", 1611 " # Apply LDA model\n", 1612 " lda_model = models.LdaModel(corpus, num_topics=5, id2word=dictionary, passes=15)\n", 1613 "\n", 1614 " # Get topic distribution for each question\n", 1615 " topic_distribution = []\n", 1616 " for i, ques in enumerate(questions):\n", 1617 " bow = dictionary.doc2bow(tokenized_data[i])\n", 1618 " topics = lda_model.get_document_topics(bow)\n", 1619 " topic_distribution.append(topics)\n", 1620 " print(f\"Question: {ques}\\nTopic: {topics}\")\n", 1621 "\n", 1622 " # Print all topics\n", 1623 " print(\"\\nTopics found are:\")\n", 1624 " for idx, topic in lda_model.print_topics(-1):\n", 1625 " print(f\"Topic: {idx} \\nWords: {topic}\\n\")\n", 1626 " return lda_model, corpus, dictionary" 1627 ] 1628 }, 1629 { 1630 "cell_type": "code", 1631 "execution_count": null, 1632 "metadata": { 1633 "application/vnd.databricks.v1+cell": { 1634 "cellMetadata": { 1635 "byteLimit": 2048000, 1636 "rowLimit": 10000 1637 }, 1638 "inputWidgets": {}, 1639 "nuid": "e892d804-a4d8-468c-93e2-acc4a5fbcf2c", 1640 "showTitle": false, 1641 "title": "" 1642 } 1643 }, 1644 "outputs": [], 1645 "source": [ 1646 "filtered_df = eval_results_table[eval_results_table[\"precision_at_1/score\"] == 1]\n", 1647 "hit_questions = filtered_df[\"question\"].tolist()\n", 1648 "filtered_df = eval_results_table[eval_results_table[\"precision_at_1/score\"] == 0]\n", 1649 "miss_questions = filtered_df[\"question\"].tolist()" 1650 ] 1651 }, 1652 { 1653 "cell_type": "code", 1654 "execution_count": null, 1655 "metadata": { 1656 "application/vnd.databricks.v1+cell": { 1657 "cellMetadata": { 1658 "byteLimit": 2048000, 1659 "rowLimit": 10000 1660 }, 1661 "inputWidgets": {}, 1662 "nuid": "7c178b69-37d4-4a6b-9737-b93e7f3d75c5", 1663 "showTitle": false, 1664 "title": "" 1665 } 1666 }, 1667 "outputs": [ 1668 { 1669 "name": "stdout", 1670 "output_type": "stream", 1671 "text": [ 1672 "Question: What is the purpose of the MLflow Model Registry?\n", 1673 "Topic: [(0, 0.0400703), (1, 0.040002838), (2, 0.040673085), (3, 0.04075462), (4, 0.8384991)]\n", 1674 "Question: What is the purpose of registering a model with the Model Registry?\n", 1675 "Topic: [(0, 0.0334267), (1, 0.033337697), (2, 0.033401005), (3, 0.033786207), (4, 0.8660484)]\n", 1676 "Question: What can you do with registered models and model versions?\n", 1677 "Topic: [(0, 0.04019648), (1, 0.04000775), (2, 0.040166058), (3, 0.8391777), (4, 0.040452003)]\n", 1678 "Question: How can you add, modify, update, or delete a model in the Model Registry?\n", 1679 "Topic: [(0, 0.025052568), (1, 0.025006149), (2, 0.025024023), (3, 0.025236268), (4, 0.899681)]\n", 1680 "Question: How can you deploy and organize models in the Model Registry?\n", 1681 "Topic: [(0, 0.033460867), (1, 0.033337582), (2, 0.033362914), (3, 0.8659808), (4, 0.033857808)]\n", 1682 "Question: What method do you use to create a new registered model?\n", 1683 "Topic: [(0, 0.028867528), (1, 0.028582651), (2, 0.882546), (3, 0.030021703), (4, 0.029982116)]\n", 1684 "Question: How can you deploy and organize models in the Model Registry?\n", 1685 "Topic: [(0, 0.033460878), (1, 0.033337586), (2, 0.033362918), (3, 0.8659798), (4, 0.03385884)]\n", 1686 "Question: How can you fetch a list of registered models in the MLflow registry?\n", 1687 "Topic: [(0, 0.0286206), (1, 0.028577656), (2, 0.02894385), (3, 0.88495284), (4, 0.028905064)]\n", 1688 "Question: What is the default channel logged for models using MLflow v1.18 and above?\n", 1689 "Topic: [(0, 0.02862059), (1, 0.028577654), (2, 0.028883327), (3, 0.8851736), (4, 0.028744776)]\n", 1690 "Question: What information is stored in the conda.yaml file?\n", 1691 "Topic: [(0, 0.050020963), (1, 0.051287953), (2, 0.051250603), (3, 0.7968765), (4, 0.05056402)]\n", 1692 "Question: How can you save a model with a manually specified conda environment?\n", 1693 "Topic: [(0, 0.02862434), (1, 0.02858204), (2, 0.02886313), (3, 0.8851747), (4, 0.028755778)]\n", 1694 "Question: What are inference params and how are they used during model inference?\n", 1695 "Topic: [(0, 0.86457103), (1, 0.03353862), (2, 0.033417325), (3, 0.034004394), (4, 0.034468662)]\n", 1696 "Question: What is the purpose of model signatures in MLflow?\n", 1697 "Topic: [(0, 0.040070876), (1, 0.04000346), (2, 0.040688124), (3, 0.040469088), (4, 0.8387685)]\n", 1698 "Question: What is the API used to set signatures on models?\n", 1699 "Topic: [(0, 0.033873636), (1, 0.033508822), (2, 0.033337757), (3, 0.035357967), (4, 0.8639218)]\n", 1700 "Question: What components are used to generate the final time series?\n", 1701 "Topic: [(0, 0.028693806), (1, 0.8853218), (2, 0.028573763), (3, 0.02862714), (4, 0.0287835)]\n", 1702 "Question: What functionality does the configuration DataFrame submitted to the pyfunc flavor provide?\n", 1703 "Topic: [(0, 0.02519801), (1, 0.025009492), (2, 0.025004204), (3, 0.025004204), (4, 0.8997841)]\n", 1704 "Question: What is a common configuration for lowering the total memory pressure for pytorch models within transformers pipelines?\n", 1705 "Topic: [(0, 0.93316424), (1, 0.016669936), (2, 0.016668117), (3, 0.016788227), (4, 0.016709473)]\n", 1706 "Question: What does the save_model() function do?\n", 1707 "Topic: [(0, 0.10002145), (1, 0.59994656), (2, 0.10001026), (3, 0.10001026), (4, 0.10001151)]\n", 1708 "Question: What is an MLflow Project?\n", 1709 "Topic: [(0, 0.06667001), (1, 0.06667029), (2, 0.7321751), (3, 0.06711196), (4, 0.06737265)]\n", 1710 "Question: What are the entry points in a MLproject file and how can you specify parameters for them?\n", 1711 "Topic: [(0, 0.02857626), (1, 0.88541776), (2, 0.02868285), (3, 0.028626908), (4, 0.02869626)]\n", 1712 "Question: What are the project environments supported by MLflow?\n", 1713 "Topic: [(0, 0.040009078), (1, 0.040009864), (2, 0.839655), (3, 0.040126894), (4, 0.040199146)]\n", 1714 "Question: What is the purpose of specifying a Conda environment in an MLflow project?\n", 1715 "Topic: [(0, 0.028579442), (1, 0.028580135), (2, 0.8841217), (3, 0.028901232), (4, 0.029817443)]\n", 1716 "Question: What is the purpose of the MLproject file?\n", 1717 "Topic: [(0, 0.05001335), (1, 0.052611485), (2, 0.050071735), (3, 0.05043289), (4, 0.7968705)]\n", 1718 "Question: How can you pass runtime parameters to the entry point of an MLflow Project?\n", 1719 "Topic: [(0, 0.025007373), (1, 0.025498485), (2, 0.8993807), (3, 0.02504522), (4, 0.025068246)]\n", 1720 "Question: How does MLflow run a Project on Kubernetes?\n", 1721 "Topic: [(0, 0.04000677), (1, 0.040007353), (2, 0.83931196), (3, 0.04012452), (4, 0.04054937)]\n", 1722 "Question: What fields are replaced when MLflow creates a Kubernetes Job for an MLflow Project?\n", 1723 "Topic: [(0, 0.022228329), (1, 0.022228856), (2, 0.023192631), (3, 0.02235802), (4, 0.90999216)]\n", 1724 "Question: What is the syntax for searching runs using the MLflow UI and API?\n", 1725 "Topic: [(0, 0.025003674), (1, 0.02500399), (2, 0.02527212), (3, 0.89956146), (4, 0.025158761)]\n", 1726 "Question: What is the syntax for searching runs using the MLflow UI and API?\n", 1727 "Topic: [(0, 0.025003672), (1, 0.025003988), (2, 0.025272164), (3, 0.8995614), (4, 0.025158769)]\n", 1728 "Question: What are the key parts of a search expression in MLflow?\n", 1729 "Topic: [(0, 0.03334423), (1, 0.03334517), (2, 0.8662702), (3, 0.033611353), (4, 0.033429127)]\n", 1730 "Question: What are the key attributes for the model with the run_id 'a1b2c3d4' and run_name 'my-run'?\n", 1731 "Topic: [(0, 0.05017508), (1, 0.05001634), (2, 0.05058142), (3, 0.7985237), (4, 0.050703418)]\n", 1732 "Question: What information does each run record in MLflow Tracking?\n", 1733 "Topic: [(0, 0.03333968), (1, 0.033340227), (2, 0.86639804), (3, 0.03349555), (4, 0.033426523)]\n", 1734 "Question: What are the two components used by MLflow for storage?\n", 1735 "Topic: [(0, 0.0334928), (1, 0.033938777), (2, 0.033719826), (3, 0.03357158), (4, 0.86527705)]\n", 1736 "Question: What interfaces does the MLflow client use to record MLflow entities and artifacts when running MLflow on a local machine with a SQLAlchemy-compatible database?\n", 1737 "Topic: [(0, 0.014289577), (1, 0.014289909), (2, 0.94276434), (3, 0.014325481), (4, 0.014330726)]\n", 1738 "Question: What is the default backend store used by MLflow?\n", 1739 "Topic: [(0, 0.033753525), (1, 0.03379533), (2, 0.033777602), (3, 0.86454684), (4, 0.0341267)]\n", 1740 "Question: What information does autologging capture when launching short-lived MLflow runs?\n", 1741 "Topic: [(0, 0.028579954), (1, 0.02858069), (2, 0.8851724), (3, 0.029027484), (4, 0.028639426)]\n", 1742 "Question: What is the purpose of the --serve-artifacts flag?\n", 1743 "Topic: [(0, 0.06670548), (1, 0.066708855), (2, 0.067003354), (3, 0.3969311), (4, 0.40265122)]\n", 1744 "\n", 1745 "Topics found are:\n", 1746 "Topic: 0 \n", 1747 "Words: 0.059*\"inference\" + 0.032*\"models\" + 0.032*\"used\" + 0.032*\"configuration\" + 0.032*\"common\" + 0.032*\"transformers\" + 0.032*\"total\" + 0.032*\"within\" + 0.032*\"pytorch\" + 0.032*\"pipelines\"\n", 1748 "\n", 1749 "Topic: 1 \n", 1750 "Words: 0.036*\"file\" + 0.035*\"mlproject\" + 0.035*\"used\" + 0.035*\"components\" + 0.035*\"entry\" + 0.035*\"parameters\" + 0.035*\"specify\" + 0.035*\"final\" + 0.035*\"points\" + 0.035*\"time\"\n", 1751 "\n", 1752 "Topic: 2 \n", 1753 "Words: 0.142*\"mlflow\" + 0.066*\"project\" + 0.028*\"information\" + 0.028*\"use\" + 0.028*\"record\" + 0.028*\"run\" + 0.015*\"key\" + 0.015*\"running\" + 0.015*\"artifacts\" + 0.015*\"client\"\n", 1754 "\n", 1755 "Topic: 3 \n", 1756 "Words: 0.066*\"models\" + 0.066*\"model\" + 0.066*\"mlflow\" + 0.041*\"using\" + 0.041*\"registry\" + 0.028*\"api\" + 0.028*\"registered\" + 0.028*\"runs\" + 0.028*\"syntax\" + 0.028*\"searching\"\n", 1757 "\n", 1758 "Topic: 4 \n", 1759 "Words: 0.089*\"model\" + 0.074*\"purpose\" + 0.074*\"mlflow\" + 0.046*\"registry\" + 0.031*\"used\" + 0.031*\"signatures\" + 0.017*\"kubernetes\" + 0.017*\"fields\" + 0.017*\"job\" + 0.017*\"replaced\"\n", 1760 "\n" 1761 ] 1762 } 1763 ], 1764 "source": [ 1765 "lda_model, corpus, dictionary = topical_analysis(hit_questions)\n", 1766 "vis_data = gensimvis.prepare(lda_model, corpus, dictionary)" 1767 ] 1768 }, 1769 { 1770 "cell_type": "code", 1771 "execution_count": 3, 1772 "metadata": { 1773 "application/vnd.databricks.v1+cell": { 1774 "cellMetadata": {}, 1775 "inputWidgets": {}, 1776 "nuid": "a0587a0f-b35d-488d-9054-55435a9585bf", 1777 "showTitle": false, 1778 "title": "" 1779 } 1780 }, 1781 "outputs": [], 1782 "source": [ 1783 "# Uncomment the following line to render the interactive widget\n", 1784 "# pyLDAvis.display(vis_data)" 1785 ] 1786 }, 1787 { 1788 "cell_type": "code", 1789 "execution_count": null, 1790 "metadata": { 1791 "application/vnd.databricks.v1+cell": { 1792 "cellMetadata": { 1793 "byteLimit": 2048000, 1794 "rowLimit": 10000 1795 }, 1796 "inputWidgets": {}, 1797 "nuid": "1375250d-9818-4503-87ec-f14020d87c81", 1798 "showTitle": false, 1799 "title": "" 1800 } 1801 }, 1802 "outputs": [ 1803 { 1804 "name": "stdout", 1805 "output_type": "stream", 1806 "text": [ 1807 "Question: What is the purpose of the mlflow.sklearn.log_model() method?\n", 1808 "Topic: [(0, 0.0669118), (1, 0.06701085), (2, 0.06667974), (3, 0.73235476), (4, 0.06704286)]\n", 1809 "Question: How can you fetch a specific model version?\n", 1810 "Topic: [(0, 0.83980393), (1, 0.040003464), (2, 0.04000601), (3, 0.040101767), (4, 0.040084846)]\n", 1811 "Question: How can you fetch the latest model version in a specific stage?\n", 1812 "Topic: [(0, 0.88561153), (1, 0.028575428), (2, 0.028578365), (3, 0.0286214), (4, 0.028613236)]\n", 1813 "Question: What can you do to promote MLflow Models across environments?\n", 1814 "Topic: [(0, 0.8661927), (1, 0.0333396), (2, 0.03362743), (3, 0.033428304), (4, 0.033411972)]\n", 1815 "Question: What is the name of the model and its version details?\n", 1816 "Topic: [(0, 0.83978903), (1, 0.04000637), (2, 0.04001106), (3, 0.040105395), (4, 0.040088095)]\n", 1817 "Question: What is the purpose of saving the model in pickled format?\n", 1818 "Topic: [(0, 0.033948876), (1, 0.03339717), (2, 0.033340737), (3, 0.86575514), (4, 0.033558063)]\n", 1819 "Question: What is an MLflow Model and what is its purpose?\n", 1820 "Topic: [(0, 0.7940762), (1, 0.05068333), (2, 0.050770763), (3, 0.053328265), (4, 0.05114142)]\n", 1821 "Question: What are the flavors defined in the MLmodel file for the mlflow.sklearn library?\n", 1822 "Topic: [(0, 0.86628276), (1, 0.033341788), (2, 0.03334801), (3, 0.03368498), (4, 0.033342462)]\n", 1823 "Question: What command can be used to package and deploy models to AWS SageMaker?\n", 1824 "Topic: [(0, 0.89991224), (1, 0.025005225), (2, 0.025009066), (3, 0.025006713), (4, 0.025066752)]\n", 1825 "Question: What is the purpose of the --build-image flag when running mlflow run?\n", 1826 "Topic: [(0, 0.033957016), (1, 0.033506736), (2, 0.034095332), (3, 0.034164555), (4, 0.86427635)]\n", 1827 "Question: What is the relative path to the python_env YAML file within the MLflow project's directory?\n", 1828 "Topic: [(0, 0.02243), (1, 0.02222536), (2, 0.022470985), (3, 0.9105873), (4, 0.02228631)]\n", 1829 "Question: What are the additional local volume mounted and environment variables in the docker container?\n", 1830 "Topic: [(0, 0.022225259), (1, 0.9110914), (2, 0.02222932), (3, 0.022227468), (4, 0.022226628)]\n", 1831 "Question: What are some examples of entity names that contain special characters?\n", 1832 "Topic: [(0, 0.028575381), (1, 0.88568854), (2, 0.02858065), (3, 0.028578246), (4, 0.028577149)]\n", 1833 "Question: What type of constant does the RHS need to be if LHS is a metric?\n", 1834 "Topic: [(0, 0.028575381), (1, 0.8856886), (2, 0.028580645), (3, 0.028578239), (4, 0.028577147)]\n", 1835 "Question: How can you get all active runs from experiments IDs 3, 4, and 17 that used a CNN model with 10 layers and had a prediction accuracy of 94.5% or higher?\n", 1836 "Topic: [(0, 0.015563371), (1, 0.015387185), (2, 0.015389071), (3, 0.015427767), (4, 0.9382326)]\n", 1837 "Question: What is the purpose of the 'experimentIds' variable in the given paragraph?\n", 1838 "Topic: [(0, 0.040206533), (1, 0.8384999), (2, 0.040013183), (3, 0.040967643), (4, 0.040312726)]\n", 1839 "Question: What is the MLflow Tracking component used for?\n", 1840 "Topic: [(0, 0.8390845), (1, 0.04000697), (2, 0.040462855), (3, 0.04014182), (4, 0.040303845)]\n", 1841 "Question: How can you create an experiment in MLflow?\n", 1842 "Topic: [(0, 0.050333958), (1, 0.0500024), (2, 0.7993825), (3, 0.050153885), (4, 0.05012722)]\n", 1843 "Question: How can you create an experiment using MLflow?\n", 1844 "Topic: [(0, 0.04019285), (1, 0.04000254), (2, 0.8396381), (3, 0.040091105), (4, 0.04007539)]\n", 1845 "Question: What is the architecture depicted in this example scenario?\n", 1846 "Topic: [(0, 0.04000523), (1, 0.040007014), (2, 0.040012203), (3, 0.04000902), (4, 0.83996654)]\n", 1847 "\n", 1848 "Topics found are:\n", 1849 "Topic: 0 \n", 1850 "Words: 0.078*\"model\" + 0.059*\"mlflow\" + 0.059*\"version\" + 0.041*\"models\" + 0.041*\"fetch\" + 0.041*\"specific\" + 0.041*\"used\" + 0.022*\"command\" + 0.022*\"deploy\" + 0.022*\"sagemaker\"\n", 1851 "\n", 1852 "Topic: 1 \n", 1853 "Words: 0.030*\"local\" + 0.030*\"container\" + 0.030*\"variables\" + 0.030*\"docker\" + 0.030*\"mounted\" + 0.030*\"environment\" + 0.030*\"volume\" + 0.030*\"additional\" + 0.030*\"special\" + 0.030*\"names\"\n", 1854 "\n", 1855 "Topic: 2 \n", 1856 "Words: 0.096*\"experiment\" + 0.096*\"create\" + 0.096*\"mlflow\" + 0.051*\"using\" + 0.009*\"purpose\" + 0.009*\"model\" + 0.009*\"method\" + 0.009*\"file\" + 0.009*\"version\" + 0.009*\"used\"\n", 1857 "\n", 1858 "Topic: 3 \n", 1859 "Words: 0.071*\"purpose\" + 0.039*\"file\" + 0.039*\"mlflow\" + 0.039*\"yaml\" + 0.039*\"directory\" + 0.039*\"relative\" + 0.039*\"within\" + 0.039*\"path\" + 0.039*\"project\" + 0.039*\"format\"\n", 1860 "\n", 1861 "Topic: 4 \n", 1862 "Words: 0.032*\"purpose\" + 0.032*\"used\" + 0.032*\"model\" + 0.032*\"prediction\" + 0.032*\"get\" + 0.032*\"accuracy\" + 0.032*\"active\" + 0.032*\"layers\" + 0.032*\"higher\" + 0.032*\"experiments\"\n", 1863 "\n" 1864 ] 1865 } 1866 ], 1867 "source": [ 1868 "lda_model, corpus, dictionary = topical_analysis(miss_questions)\n", 1869 "vis_data = gensimvis.prepare(lda_model, corpus, dictionary)" 1870 ] 1871 }, 1872 { 1873 "cell_type": "code", 1874 "execution_count": 4, 1875 "metadata": { 1876 "application/vnd.databricks.v1+cell": { 1877 "cellMetadata": {}, 1878 "inputWidgets": {}, 1879 "nuid": "724db985-5382-43a6-ada5-0ac1c2d49c18", 1880 "showTitle": false, 1881 "title": "" 1882 } 1883 }, 1884 "outputs": [], 1885 "source": [ 1886 "# Uncomment the following line to render the interactive widget\n", 1887 "# pyLDAvis.display(vis_data)" 1888 ] 1889 } 1890 ], 1891 "metadata": { 1892 "application/vnd.databricks.v1+notebook": { 1893 "dashboards": [], 1894 "language": "python", 1895 "notebookMetadata": { 1896 "pythonIndentUnit": 4 1897 }, 1898 "notebookName": "retriever-evaluation-tutorial", 1899 "widgets": {} 1900 }, 1901 "kernelspec": { 1902 "display_name": "Python 3 (ipykernel)", 1903 "language": "python", 1904 "name": "python3" 1905 }, 1906 "language_info": { 1907 "codemirror_mode": { 1908 "name": "ipython", 1909 "version": 3 1910 }, 1911 "file_extension": ".py", 1912 "mimetype": "text/x-python", 1913 "name": "python", 1914 "nbconvert_exporter": "python", 1915 "pygments_lexer": "ipython3", 1916 "version": "3.8.17" 1917 } 1918 }, 1919 "nbformat": 4, 1920 "nbformat_minor": 1 1921 }