1  # Haystack Documentation Website
  2  
  3  This directory contains the Docusaurus-powered documentation website for [Haystack](https://github.com/deepset-ai/haystack), an open-source framework for building production-ready applications with Large Language Models (LLMs).
  4  
  5  - **Website URL:** https://docs.haystack.deepset.ai
  6  
  7  **Table of Contents**
  8  
  9  - [About](#about)
 10  - [Prerequisites](#prerequisites)
 11  - [Quick Start](#quick-start)
 12  - [Common tasks](#common-tasks)
 13  - [Project Structure](#project-structure)
 14  - [Technology Stack](#technology-stack)
 15  - [Available Scripts](#available-scripts)
 16  - [Contributing](#contributing)
 17  - [CI/CD and Automation](#cicd-and-automation)
 18    - [Versioning](#versioning)
 19  - [Deployment](#deployment)
 20  - [llms.txt for AI tools](#llms.txt-for-ai-tools)
 21  
 22  ## About
 23  
 24  This documentation site is built with Docusaurus 3 and provides comprehensive guides, tutorials, API references, and best practices for using Haystack. The site supports multiple versions and automated API reference generation.
 25  
 26  ## Prerequisites
 27  
 28  - **Node.js** 18 or higher
 29  - **npm** (included with Node.js) or Yarn
 30  
 31  ## Quick Start
 32  
 33  > [!NOTE]
 34  > All commands must be run from the `haystack/docs-website` directory.
 35  
 36  ```bash
 37  # Clone the repository and navigate to docs-website
 38  git clone https://github.com/deepset-ai/haystack.git
 39  cd haystack/docs-website
 40  
 41  # Install dependencies
 42  npm install
 43  
 44  # Start the development server
 45  npm start
 46  
 47  # The site opens at http://localhost:3000 with live reload
 48  ```
 49  
 50  ## Common tasks
 51  
 52  - Edit a page: update files under `docs/` or `versioned_docs/` and preview at http://localhost:3000
 53  - Add to sidebar: update `sidebars.js` with your doc ID
 54  - Production check: `npm run build && npm run serve`
 55  - Full guidance: see `CONTRIBUTING.md`
 56  
 57  ## Project Structure
 58  
 59  ```
 60  docs-website/
 61  ├── docs/                          # Main documentation (guides, tutorials, concepts)
 62  │   ├── _templates/               # Authoring templates (excluded from build)
 63  │   ├── concepts/                 # Core Haystack concepts
 64  │   ├── pipeline-components/      # Component documentation
 65  │   └── ...
 66  ├── reference/                     # API reference (auto-generated, do not edit manually)
 67  ├── versioned_docs/               # Versioned copies of docs/
 68  ├── reference_versioned_docs/     # Versioned copies of reference/
 69  ├── src/                          # React components and custom code
 70  │   ├── components/              # Custom React components
 71  │   ├── css/                     # Global styles
 72  │   ├── pages/                   # Custom pages
 73  │   ├── remark/                  # Remark plugins
 74  │   └── theme/                   # Docusaurus theme customizations
 75  ├── static/                       # Static assets (images, files)
 76  ├── scripts/                      # Build and test scripts
 77  │   ├── generate_requirements.py # Generates Python dependencies
 78  │   ├── setup-dev.sh             # Development environment setup
 79  │   └── test_python_snippets.py  # Tests Python code in docs
 80  ├── sidebars.js                   # Navigation for docs/
 81  ├── reference-sidebars.js         # Navigation for reference/
 82  ├── docusaurus.config.js          # Main Docusaurus configuration
 83  ├── versions.json                 # Available docs versions
 84  ├── reference_versions.json       # Available API reference versions
 85  └── package.json                  # Node.js dependencies and scripts
 86  ```
 87  
 88  ## Technology Stack
 89  
 90  | Technology | Version | Purpose |
 91  |------------|---------|---------|
 92  | [Docusaurus](https://docusaurus.io/) | 3.8.1 | Static site generator |
 93  | [React](https://react.dev/) | 19.0.0 | UI framework |
 94  | [MDX](https://mdxjs.com/) | 3.0.0 | Markdown with JSX |
 95  | [Node.js](https://nodejs.org/) | ≥18.0 | Runtime environment |
 96  
 97  **Key Docusaurus Plugins:**
 98  - `@docusaurus/plugin-content-docs` — Two separate instances of this plugin run simultaneously:
 99    1. **Main docs instance** (via the `classic` preset): serves `docs/` at `/docs/`
100    2. **Reference instance** (explicit plugin): serves `reference/` at `/reference/`
101  
102    Each instance has its own sidebar, versioning config (`versions.json` vs `reference_versions.json`), and versioned content folders. This allows the API reference and guides to version independently and maintain separate navigation.
103  
104  - **Custom remark plugin** (`src/remark/versionedReferenceLinks.js`) — Automatically rewrites cross-links between docs and reference to include the correct version prefix. For example, if you're viewing docs version 2.19 and click a link to `/reference/some-api`, the plugin rewrites it to `/reference/2.19/some-api` so readers stay in the same version context.
105  
106  **When one might need these plugins:**
107  - **Broken cross-links after a release:** If links between docs and API reference pages break (404s), the remark plugin may need adjustment—especially if version naming conventions change.
108  - **Version dropdown issues:** If the version selector shows wrong versions or doesn't switch correctly between docs/reference, check the dual `plugin-content-docs` configs in `docusaurus.config.js`.
109  - **Sidebar mismatches:** If API reference navigation breaks separately from main docs, remember they use different sidebar files (`sidebars.js` vs `reference-sidebars.js`).
110  
111  ## Available Scripts
112  
113  **Important:** Run these commands from the `haystack/docs-website` directory:
114  
115  | Command | Description |
116  |---------|-------------|
117  | `npm install` | Install all dependencies |
118  | `npm start` | Start development server with live reload (http://localhost:3000) |
119  | `npm run build` | Build production-ready static files to `build/` |
120  | `npm run serve` | Preview production build locally |
121  | `npm run clear` | Clear Docusaurus cache (use if encountering build issues) |
122  | `npm run docusaurus` | Run Docusaurus CLI commands directly |
123  | `npm run swizzle` | Eject and customize Docusaurus theme components |
124  
125  ## Contributing
126  
127  We welcome contributions to improve the documentation! See [CONTRIBUTING.md](./CONTRIBUTING.md) for:
128  
129  - Writing and style guidelines
130  - How to author new documentation pages
131  - Setting up your development environment
132  - Testing requirements
133  - Pull request process
134  
135  For code contributions to Haystack itself, see the [main repository's contribution guide](https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md).
136  
137  ## CI/CD and Automation
138  
139  This site uses automated workflows for API reference sync and preview deployments. See [CONTRIBUTING.md](./CONTRIBUTING.md) for details.
140  
141  ### Versioning
142  
143  Documentation versions are released alongside Haystack releases and are fully automated through GitHub workflows. Contributors do not need to manually create or manage versions.
144  
145  **Automated Workflows:**
146  - `promote_unstable_docs.yml` - Automatically triggered during Haystack releases
147  - `minor_version_release.yml` - Creates new version directories and updates version configuration
148  
149  These workflows automatically create versioned documentation snapshots and pull requests during the release process.
150  
151  ## Deployment
152  
153  The documentation site is automatically deployed to **https://docs.haystack.deepset.ai** when changes are merged to the `main` branch.
154  
155  ## llms.txt for AI tools
156  
157  This docs site exposes a concatenated view of the documentation for AI tools with an `llms.txt` file, generated by the [`docusaurus-plugin-generate-llms-txt`](https://github.com/din0s/docusaurus-plugin-llms-txt) plugin.
158  
159  - **What it is**: A single, generated text file that concatenates the docs content to make it easier for LLMs and other tools to consume.
160  - **Where to find it (deployed)**: At the site root `https://docs.haystack.deepset.ai/llms.txt`.
161  - **How it’s generated**:
162    - Automatically when you run:
163      - `npm run start`
164      - `npm run build`
165    - Manually with:
166  
167      ```bash
168      npm run generate-llms-txt
169      ```
170  
171  - **Configuration**:
172    - The plugin is wired in `docusaurus.config.js` under the `plugins` array as `'docusaurus-plugin-generate-llms-txt'` with `outputFile: 'llms.txt'`.
173    - A local plugin (`plugins/txtLoaderPlugin.js`) configures Webpack to treat `.txt` files (including `llms.txt`) as text assets so they don’t cause build-time parse errors.*