/ README.md
README.md
1 # Fold-Stack: A Sovereign Development Environment 2 3 **Fold-Stack** is a self-hosted, sovereign development environment designed for fieldcraft, collaboration, and resilient data management. It integrates a suite of tools for blogging, version control, note-taking, document editing, file storage, and replication across cloud services. This stack is built with Docker Compose for easy deployment and management. 4 5 ## π Project Overview 6 7 Fold-Stack provides a modular, self-contained environment for: 8 9 - **Blogging**: Ghost for publishing content. 10 - **Version Control**: Forgejo for Git repository management. 11 - **Decentralized Collaboration**: Radicle for peer-to-peer version control. 12 - **Document Conversion**: Pandoc for converting documents (e.g., Markdown to PDF). 13 - **Email Testing**: MailHog for capturing and testing emails. 14 - **Note-Taking**: Trilium for hierarchical note management. 15 - **Collaborative Editing**: HedgeDoc for real-time Markdown collaboration. 16 - **File Storage**: Nextcloud for file storage and sharing. 17 - **Data Replication**: Rclone for syncing data to Google Drive, Internet Archive, and Web3.storage. 18 - **Document Compilation**: Typst for fast, modern document creation. 19 - **LaTeX Collaboration**: Overleaf CE for collaborative LaTeX editing. 20 - **Git Mirroring**: Git-Sync Mirror Agent for syncing Git repositories to multiple remotes. 21 22 The stack is designed to be lightweight by default, with resource-heavy services (like Overleaf CE) toggleable to optimize performance. 23 24 --- 25 26 ## π οΈ Prerequisites 27 28 Before setting up Fold-Stack, ensure you have the following: 29 30 - **Docker** and **Docker Compose** installed on your system. 31 - Install Docker: [Official Docker Installation Guide](https://docs.docker.com/get-docker/) 32 - Install Docker Compose: [Official Docker Compose Installation Guide](https://docs.docker.com/compose/install/) 33 - **Git** installed for cloning the repository. 34 - Install Git: [Official Git Installation Guide](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) 35 - A machine with at least 4GB of RAM (8GB recommended for Overleaf CE). 36 - Internet access for pulling Docker images and configuring Rclone remotes. 37 - Accounts for replication services (optional but recommended): 38 - Google Drive (for `gdrive` remote). 39 - Internet Archive (for `ia` remote). 40 - Web3.storage (for `web3` remote, requires an API token). 41 - SSH keys for GitHub and Forgejo (for Git-Sync). 42 43 --- 44 45 ## π Setup Instructions 46 47 ### 1. Clone the Repository 48 49 Clone the Fold-Stack repository to your local machine: 50 51 ```bash 52 git clone https://github.com/mrhavens/fold-stack.git 53 cd fold-stack 54 ``` 55 56 ### 2. Configure Environment Variables 57 58 Copy the example environment file and adjust as needed: 59 60 ```bash 61 cp .env.dev.example .env.dev 62 ``` 63 64 The default `.env.dev` contains: 65 66 ``` 67 USER_UID=1000 68 USER_GID=1000 69 ``` 70 71 - `USER_UID` and `USER_GID` should match your local userβs UID and GID to avoid permission issues. Check your UID/GID with: 72 ```bash 73 id -u 74 id -g 75 ``` 76 77 ### 3. Configure Rclone for Data Replication 78 79 Fold-Stack uses Rclone to replicate data to Google Drive, Internet Archive, and Web3.storage. You need to configure the remotes: 80 81 1. Run the Rclone configuration wizard: 82 ```bash 83 rclone config 84 ``` 85 86 2. Add the following remotes: 87 - **Google Drive (`gdrive`)**: 88 - Choose `n` (new remote). 89 - Name: `gdrive` 90 - Type: `drive` (Google Drive). 91 - Client ID/Secret: Leave blank. 92 - Scope: `drive` (full access, option 1). 93 - Root Folder ID: Leave blank. 94 - Service Account File: Leave blank. 95 - Edit Advanced Config: `n`. 96 - Auto Config: `y` (follow the browser prompt to authenticate). 97 - Configure as a Shared Drive: `n`. 98 - **Internet Archive (`ia`)**: 99 - Choose `n` (new remote). 100 - Name: `ia` 101 - Type: `internetarchive`. 102 - Access Key ID: Your Internet Archive access key (get from archive.org account settings). 103 - Secret Access Key: Your Internet Archive secret key. 104 - Edit Advanced Config: `n`. 105 - **Web3.storage (`web3`)**: 106 - Choose `n` (new remote). 107 - Name: `web3` 108 - Type: `ipfs`. 109 - Host: `api.web3.storage`. 110 - API Token: Your Web3.storage API token (get from web3.storage). 111 - Edit Advanced Config: `n`. 112 113 3. Copy the Rclone configuration to the project: 114 ```bash 115 mkdir -p ./config/rclone 116 cp ~/.config/rclone/rclone.conf ./config/rclone/rclone.conf 117 chmod 600 ./config/rclone/rclone.conf 118 ``` 119 120 4. Verify the remotes: 121 ```bash 122 rclone listremotes --config ./config/rclone/rclone.conf 123 ``` 124 You should see: `gdrive:`, `ia:`, `nextcloud:`, `web3:`. 125 126 ### 4. Configure Git-Sync Mirror Agent 127 128 The Git-Sync Mirror Agent syncs a local Git repository to multiple remotes (GitHub, Forgejo, Radicle, Internet Archive, and optionally Web3.storage). 129 130 1. **Initialize a Local Repository**: 131 ```bash 132 mkdir -p volumes/repos 133 cd volumes/repos 134 git init 135 echo "# Test Repo" > README.md 136 git add . 137 git commit -m "Initial commit" 138 git branch -M main 139 ``` 140 141 2. **Set Up SSH Keys for GitHub and Forgejo**: 142 - Generate SSH keys if you donβt already have them: 143 ```bash 144 ssh-keygen -t ed25519 -C "your_email@example.com" -f ~/.ssh/github_key 145 ssh-keygen -t ed25519 -C "your_email@example.com" -f ~/.ssh/forgejo_key 146 ``` 147 - Add the public keys to GitHub and Forgejo: 148 - GitHub: Add `~/.ssh/github_key.pub` to your GitHub account (Settings > SSH and GPG keys). 149 - Forgejo: Add `~/.ssh/forgejo_key.pub` to your Forgejo account (http://localhost:3000/user/settings/keys). 150 - Copy the private keys to the `git-sync` secrets directory: 151 ```bash 152 cp ~/.ssh/github_key config/git-sync/secrets/github.key 153 cp ~/.ssh/forgejo_key config/git-sync/secrets/forgejo.key 154 chmod 600 config/git-sync/secrets/github.key config/git-sync/secrets/forgejo.key 155 ``` 156 157 3. **Configure Remotes**: 158 Edit `config/git-sync/remotes.conf` to match your repository URLs: 159 ``` 160 github|git|git@github.com:mrhavens/mirror-repo.git|1 161 forgejo|git|git@localhost:2222/mrhavens/mirror-repo.git|1 162 radicle|radicle|radicle://mrhavens/mirror-repo|1 163 ia|rclone|ia:fold-stack-git-mirror|1 164 web3|rclone|web3:fold-stack-git-mirror|0 165 ``` 166 167 ### 5. Start the Stack 168 169 Fold-Stack uses Docker Compose to manage services. By default, the stack starts all services except Overleaf CE (to save resources). 170 171 1. Start the core services: 172 ```bash 173 ./scripts/up-dev.sh 174 ``` 175 176 2. (Optional) Enable Overleaf CE if needed: 177 ```bash 178 ./scripts/enable-overleaf.sh 179 ``` 180 181 3. (Optional) Enable Typst if not already running: 182 ```bash 183 ./scripts/enable-typst.sh 184 ``` 185 186 ### 6. Verify Services are Running 187 188 Check the status of all containers: 189 ```bash 190 docker ps 191 ``` 192 193 Run the diagnostic script to identify any issues: 194 ```bash 195 ./scripts/diagnose-stack.sh 196 ``` 197 198 If any service fails to start, check its logs: 199 ```bash 200 docker logs <container_name> 201 ``` 202 For example: `docker logs overleaf_dev`. 203 204 ### 7. Stop the Stack 205 206 To stop all services: 207 ```bash 208 ./scripts/down-dev.sh 209 ``` 210 211 --- 212 213 ## π Accessing Services 214 215 Below are the URLs to access each service running in Fold-Stack. All services are accessible on `localhost` with the specified ports. 216 217 | Service | URL | Description | Default Credentials (if applicable) | 218 |---------------|----------------------------|------------------------------------------|-------------------------------------| 219 | **Ghost** | [http://localhost:2368](http://localhost:2368) | Blogging platform for publishing content. | First user registration is admin. | 220 | **Forgejo** | [http://localhost:3000](http://localhost:3000) | Git repository management (Gitea fork). | First user registration is admin. | 221 | **Radicle** | N/A (CLI-based) | Decentralized version control (CLI). | N/A | 222 | **Pandoc** | N/A (CLI-based) | Document conversion tool (CLI). | N/A | 223 | **MailHog** | [http://localhost:8025](http://localhost:8025) | Email testing and capture tool. | N/A | 224 | **Trilium** | [http://localhost:8080](http://localhost:8080) | Hierarchical note-taking application. | First user registration is admin. | 225 | **HedgeDoc** | [http://localhost:3030](http://localhost:3030) | Collaborative Markdown editor. | N/A (optional login) | 226 | **Nextcloud** | [http://localhost:8081](http://localhost:8081) | File storage and sharing platform. | Username: `admin`, Password: `admin_password` | 227 | **Typst** | N/A (CLI-based) | Fast document compilation tool (CLI). | N/A | 228 | **Overleaf CE** | [http://localhost:8090](http://localhost:8090) | Collaborative LaTeX editor (run `./scripts/enable-overleaf.sh` to start). | First user registration is admin (email: `admin@example.com`). | 229 | **Git-Sync** | N/A (CLI-based) | Git repository mirroring agent. | N/A | 230 231 --- 232 233 ## π How-To Guides 234 235 ### 1. **Ghost: Publish a Blog Post** 236 237 1. Access Ghost at [http://localhost:2368](http://localhost:2368). 238 2. Register as the first user (this user will be the admin). 239 3. Log in and navigate to the admin panel at [http://localhost:2368/ghost](http://localhost:2368/ghost). 240 4. Create a new post: 241 - Click "Posts" > "New Post". 242 - Add a title and content. 243 - Click "Publish" to make it live. 244 5. View your post on the blog homepage. 245 246 **Note**: Ghost uses MailHog for email sending (e.g., for user invites). Check emails at [http://localhost:8025](http://localhost:8025). 247 248 --- 249 250 ### 2. **Forgejo: Create a Git Repository** 251 252 1. Access Forgejo at [http://localhost:3000](http://localhost:3000). 253 2. Register as the first user (this user will be the admin). 254 3. Log in and create a new repository: 255 - Click "+" > "New Repository". 256 - Name your repository and click "Create Repository". 257 4. Clone the repository locally: 258 ```bash 259 git clone http://localhost:3000/<username>/<repo-name>.git 260 ``` 261 5. Add files, commit, and push: 262 ```bash 263 cd <repo-name> 264 echo "# My Project" > README.md 265 git add . 266 git commit -m "Initial commit" 267 git push origin main 268 ``` 269 270 --- 271 272 ### 3. **Radicle: Initialize a Peer-to-Peer Repository** 273 274 1. Access the `radicle_dev` container: 275 ```bash 276 docker exec -it radicle_dev bash 277 ``` 278 2. Initialize a Radicle project: 279 ```bash 280 rad init --name my-project --description "My Radicle project" --default-branch main 281 ``` 282 3. Share your project with peers using the Radicle CLI (refer to [Radicle Documentation](https://radicle.xyz/guides)). 283 284 --- 285 286 ### 4. **Pandoc: Convert a Markdown File to PDF** 287 288 1. Access the `pandoc_dev` container: 289 ```bash 290 docker exec -it pandoc_dev /bin/sh 291 ``` 292 2. Convert a Markdown file in the `scrolls` volume to PDF: 293 ```bash 294 echo "# Hello, Pandoc" > /workspace/test.md 295 pandoc /workspace/test.md -o /workspace/test.pdf 296 ``` 297 3. On your host, check the output: 298 ```bash 299 ls ./volumes/scrolls/test.pdf 300 ``` 301 302 --- 303 304 ### 5. **MailHog: Test Email Sending** 305 306 1. Access MailHog at [http://localhost:8025](http://localhost:8025). 307 2. Trigger an email from another service (e.g., Ghost user invite): 308 - In Ghost, invite a new user via the admin panel. 309 3. Check MailHog for the captured email and view its contents. 310 311 --- 312 313 ### 6. **Trilium: Organize Your Notes** 314 315 1. Access Trilium at [http://localhost:8080](http://localhost:8080). 316 2. Register as the first user (this user will be the admin). 317 3. Create a new note: 318 - Click "Create Note" and add a title and content. 319 - Organize notes in a hierarchy using drag-and-drop. 320 4. Trilium backups are automatically synced to Web3.storage via Rclone (see `./volumes/trilium-backup`). 321 322 --- 323 324 ### 7. **HedgeDoc: Collaborate on Markdown Documents** 325 326 1. Access HedgeDoc at [http://localhost:3030](http://localhost:3030). 327 2. Create a new note: 328 - Click "New Guest Note" or log in (optional). 329 - Write Markdown content in the editor. 330 3. Share the note URL with collaborators for real-time editing. 331 332 --- 333 334 ### 8. **Nextcloud: Store and Share Files** 335 336 1. Access Nextcloud at [http://localhost:8081](http://localhost:8081). 337 2. Log in with: 338 - Username: `admin` 339 - Password: `admin_password` 340 3. Upload a file: 341 - Click "+" > "Upload File". 342 - Select a file from your local machine. 343 4. Share the file by generating a share link. 344 345 **Note**: Nextcloud mounts the `scrolls`, `ghost`, `trilium`, and `hedgedoc_uploads` directories under `/admin/files`. 346 347 --- 348 349 ### 9. **Rclone: Verify Data Replication** 350 351 Rclone automatically syncs data from the `volumes` directory to remote services: 352 353 - **Google Drive**: Syncs `./volumes/scrolls` and `./volumes/hedgedoc/uploads` to `fold-stack/scrolls` and `fold-stack/hedgedoc_uploads`. 354 - **Internet Archive**: Syncs `.scroll`, `.seal`, `.typ`, and `.tex` files from `./volumes/scrolls` to `fold-stack-scrolls`. 355 - **Web3.storage**: Syncs `./volumes/trilium-backup` to `fold-stack-trilium`. 356 357 1. Add a test file to trigger a sync: 358 ```bash 359 echo "Test file" > ./volumes/scrolls/test-rclone.scroll 360 ``` 361 2. Monitor Rclone logs: 362 ```bash 363 docker logs rclone_dev --follow 364 ``` 365 3. Verify the file appears on: 366 - Google Drive: Check `fold-stack/scrolls`. 367 - Internet Archive: Check `fold-stack-scrolls`. 368 369 --- 370 371 ### 10. **Typst: Compile a Document** 372 373 1. Access the `typst_dev` container: 374 ```bash 375 docker exec -it typst_dev /bin/sh 376 ``` 377 2. Create a sample Typst document: 378 ```bash 379 echo "#set page(width: 10cm, height: 10cm)" > /workspace/sample.typ 380 echo "#set text(size: 16pt)" >> /workspace/sample.typ 381 echo "Hello, Typst!" >> /workspace/sample.typ 382 ``` 383 3. Compile the document to PDF: 384 ```bash 385 typst compile /workspace/sample.typ /workspace/sample.pdf 386 ``` 387 4. On your host, check the output: 388 ```bash 389 ls ./volumes/scrolls/sample.pdf 390 ``` 391 392 **Note**: Typst files (`.typ`) in `./volumes/scrolls` are synced to Google Drive and Internet Archive via Rclone. 393 394 --- 395 396 ### 11. **Overleaf CE: Collaborative LaTeX Editing** 397 398 1. Ensure Overleaf CE is running: 399 ```bash 400 /scripts/enable-overleaf.sh 401 ``` 402 2. Access Overleaf CE at [http://localhost:8090](http://localhost:8090). 403 3. Register as the first user (email: `admin@example.com`, this user will be the admin). 404 4. Create a new project: 405 - Click "New Project" > "Blank Project". 406 - Add a simple LaTeX document: 407 ```latex 408 \documentclass{article} 409 \begin{document} 410 Hello, Overleaf CE! 411 This is a test document. 412 \end{document} 413 ``` 414 5. Compile the document to generate a PDF. 415 6. Access files from the `scrolls` volume: 416 - Add a file from the `scrolls` volume (e.g., `test.tex`) to your project and compile it. 417 418 **Note**: Overleaf CE is resource-heavy. Stop it when not in use: 419 ```bash 420 docker compose -f docker-compose.dev.yml stop overleaf overleaf-mongo overleaf-redis 421 ``` 422 423 --- 424 425 ### 12. **Git-Sync: Mirror a Git Repository** 426 427 The Git-Sync Mirror Agent watches the local repository at `./volumes/repos` and syncs changes to GitHub, Forgejo, Radicle, Internet Archive, and optionally Web3.storage. 428 429 1. **Add a Commit to the Local Repository**: 430 ```bash 431 cd volumes/repos 432 echo "Change 1" >> README.md 433 git add . 434 git commit -m "Change 1" 435 ``` 436 437 2. **Manually Push to All Remotes**: 438 To trigger a manual sync to all configured remotes, run: 439 ```bash 440 ./scripts/manual-push-git-sync.sh 441 ``` 442 This script pushes the latest changes to all enabled remotes immediately, bypassing the automated sync interval. 443 444 3. **Monitor Git-Sync Logs**: 445 ```bash 446 docker logs git_sync_dev --follow 447 ``` 448 You should see the sync process for each configured remote. 449 450 4. **Verify Sync**: 451 - **GitHub**: Check your GitHub repository (\`mrhavens/mirror-repo\`). 452 - **Forgejo**: Check \`http://localhost:3000/mrhavens/mirror-repo\`. 453 - **Internet Archive**: Check \`fold-stack-git-mirror\` for Git bundles. 454 - **Web3.storage**: Enable in \`remotes.conf\` and check \`fold-stack-git-mirror\`. 455 456 **Configuration**: 457 - Edit \`config/git-sync/.env\` to adjust settings: 458 ``` 459 SYNC_INTERVAL=300 # Sync check interval in seconds 460 PUSH_MODE=push # "push" for git push, "bundle" for git bundle 461 SIGN_COMMITS=false # Set to true to enable commit signing (requires GPG) 462 LOG_LEVEL=INFO # Log verbosity (INFO, ERROR) 463 RETRY_MAX=3 # Max retry attempts for failed syncs 464 RETRY_BACKOFF=5 # Base backoff time in seconds for retries 465 ``` 466 467 **Logs**: 468 - Logs are stored in \`./volumes/logs\` with filenames like \`sync-<timestamp>.log\` or \`manual-push-<timestamp>.log\`. 469 470 **Diagnostics**: 471 - Run the diagnostic script to troubleshoot issues: 472 ```bash 473 ./scripts/diagnose-git-sync.sh 474 ``` 475 This script checks the container status, configuration files, SSH keys, remote connectivity, logs, and volumes, providing detailed error messages and fixes. 476 477 **Sync Report**: 478 - Generate a report to see the latest sync activity for each remote: 479 ```bash 480 ./scripts/report-git-sync.sh 481 ``` 482 This report shows the latest commit in the local repository, the last successful sync for each remote (with timestamp and commit/bundle details), and any recent failed sync attempts. 483 484 --- 485 486 ### 13. **Unified Sovereign Dashboard: Control Plane** 487 488 The Fold Stack includes a unified dashboard powered by [Flame](https://github.com/pawelmalak/flame), running on port 80. This control plane provides visual access to all services and exposes internal scripts for diagnostics, syncing, and integrity checks. 489 490 #### Enable the Dashboard 491 492 1. Ensure `FLAME_PASSWORD` is set in `.env.dev`: 493 ```bash 494 echo "FLAME_PASSWORD=securepassword123" >> .env.dev 495 ``` 496 Replace `securepassword123` with a strong password. 497 498 2. Run the enable script: 499 ```bash 500 ./scripts/enable-dashboard.sh 501 ``` 502 503 3. Access the dashboard at [http://localhost](http://localhost). 504 - Log in using the password set in `FLAME_PASSWORD`. 505 506 #### Dashboard Features 507 508 - **Services**: Links to Ghost, Forgejo, Trilium, HedgeDoc, Nextcloud, MailHog, and a placeholder for Scroll Renderer (Typst/Pandoc). 509 - **Scripts**: Placeholders for running `diagnose-stack.sh`, `sync-stage-to-prod.sh`, `seal-foldstate.sh`, and `watch-fold-integrity.sh` (requires future implementation for UI triggering). 510 - **Customization**: Configurable via `volumes/flame/bookmarks.yml`. 511 512 #### Notes 513 514 - The dashboard runs on port 80 in development. For production, consider rebinding to port 8080 to avoid conflicts. 515 - The container is hardened with read-only filesystem and limited capabilities for security. 516 517 --- 518 519 ## π οΈ Troubleshooting 520 521 ### General Issues 522 - **Container Not Running**: Check logs for the specific container: 523 ```bash 524 docker logs <container_name> 525 ``` 526 Restart the stack: 527 ```bash 528 ./scripts/down-dev.sh && ./scripts/up-dev.sh 529 ``` 530 - **Port Conflicts**: Check for port conflicts: 531 ```bash 532 netstat -tuln | grep <port> 533 ``` 534 Stop conflicting processes or change the port in \`docker-compose.dev.yml\`. 535 536 ### Rclone Issues 537 - **Sync Not Working**: Verify Rclone remotes: 538 ```bash 539 rclone listremotes --config ./config/rclone/rclone.conf 540 ``` 541 Reconfigure if needed: 542 ```bash 543 rclone config 544 ``` 545 - **Permission Issues**: Fix volume permissions: 546 ```bash 547 chmod -R 775 ./volumes 548 chown -R 1000:1000 ./volumes 549 ``` 550 551 ### Overleaf CE Issues 552 - **Startup Errors**: Check logs for Overleaf, MongoDB, and Redis: 553 ```bash 554 docker logs overleaf_dev 555 docker logs overleaf_mongo_dev 556 docker logs overleaf_redis_dev 557 ``` 558 Ensure MongoDB and Redis are healthy before Overleaf starts (handled by \`depends_on\` in \`docker-compose.dev.yml\`). 559 560 ### Git-Sync Issues 561 - **Sync Fails**: Run diagnostics: 562 ```bash 563 ./scripts/diagnose-git-sync.sh 564 ``` 565 Ensure SSH keys are correctly set up and remotes are accessible. 566 - **Radicle Not Syncing**: Radicle sync is a placeholder. Implement the \`rad\` CLI in \`entrypoint.sh\` if needed. 567 - **Check Sync Status**: Generate a sync report: 568 ```bash 569 ./scripts/report-git-sync.sh 570 ``` 571 572 --- 573 574 ## π Additional Resources 575 576 - **Ghost Documentation**: [https://ghost.org/docs/](https://ghost.org/docs/) 577 - **Forgejo Documentation**: [https://forgejo.org/docs/](https://forgejo.org/docs/) 578 - **Radicle Documentation**: [https://radicle.xyz/guides](https://radicle.xyz/guides) 579 - **Pandoc Documentation**: [https://pandoc.org/](https://pandoc.org/) 580 - **MailHog Documentation**: [https://github.com/mailhog/MailHog](https://github.com/mailhog/MailHog) 581 - **Trilium Documentation**: [https://github.com/zadam/trilium/wiki](https://github.com/zadam/trilium/wiki) 582 - **HedgeDoc Documentation**: [https://docs.hedgedoc.org/](https://docs.hedgedoc.org/) 583 - **Nextcloud Documentation**: [https://docs.nextcloud.com/](https://docs.nextcloud.com/) 584 - **Rclone Documentation**: [https://rclone.org/docs/](https://rclone.org/docs/) 585 - **Typst Documentation**: [https://typst.app/docs/](https://typst.app/docs/) 586 - **Overleaf CE Documentation**: [https://github.com/overleaf/overleaf/wiki](https://github.com/overleaf/overleaf/wiki) 587 588 --- 589 590 ## π€ Contributing 591 592 Contributions are welcome! To contribute: 593 594 1. Fork the repository. 595 2. Create a new branch: \`git checkout -b feature/your-feature\`. 596 3. Make your changes and commit: \`git commit -m "Add your feature"\`. 597 4. Push to your branch: \`git push origin feature/your-feature\`. 598 5. Open a pull request. 599 600 --- 601 602 ## π Last Updated 603 604 This README was last updated on **May 26, 2025, at 09:55 PM CDT**.