1  **Git-Sync Mirror Agent**
  2  
  3  **Git-Sync** is a containerized mirroring agent designed to watch a local Git repository and propagate changes to multiple remote repositories with high reliability, auditability, and fault recovery. It supports mirroring to platforms like GitHub, Forgejo, Radicle, Internet Archive, and Web3.storage, ensuring your repository is resiliently backed up across diverse infrastructure. Git-Sync is a key component of the fold-stack project, emphasizing sovereignty and fieldcraft-resilient design.
  4  
  5  ---
  6  
  7  📜 **Overview**
  8  
  9  Git-Sync operates as a Docker container that:
 10  
 11  * Watches a local Git repository for changes.  
 12  * Syncs changes to multiple configured remotes (GitHub, Forgejo, Radicle, Internet Archive, Web3.storage).  
 13  * Ensures atomicity using lockfiles.  
 14  * Logs all operations for auditability.  
 15  * Implements failure recovery with retries and exponential backoff.  
 16  * Operates independently of commercial SaaS infrastructure.
 17  
 18  **Supported Remotes**
 19  
 20  * **GitHub**: Push via SSH.  
 21  * **Forgejo**: Push via SSH to a self-hosted instance.  
 22  * **Radicle**: Peer-to-peer Git (placeholder, requires rad CLI implementation).  
 23  * **Internet Archive**: Git bundles uploaded via Rclone.  
 24  * **Web3.storage**: Git bundles uploaded via Rclone (optional).  
 25  * **Extendable**: Can be extended to support S3, IPFS, Sia, etc.
 26  
 27  ---
 28  
 29  🛠️ **Prerequisites**
 30  
 31  Before setting up Git-Sync, ensure you have the following:
 32  
 33  * **Docker** and **Docker Compose** installed.  
 34    * Install Docker: [Official Docker Installation Guide](https://docs.docker.com/get-docker/)  
 35    * Install Docker Compose: [Official Docker Compose Installation Guide](https://docs.docker.com/compose/install/)  
 36  * **Git** installed for managing the local repository.  
 37    * Install Git: [Official Git Installation Guide](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)  
 38  * A local Git repository to mirror (e.g., volumes/repos in fold-stack).  
 39  * SSH keys for GitHub and Forgejo.  
 40  * Rclone configured for Internet Archive and Web3.storage (if using these remotes).  
 41  * A machine with at least 2GB of RAM (Git-Sync is lightweight but depends on fold-stack requirements).
 42  
 43  ---
 44  
 45  🚀 **Setup Instructions**
 46  
 47  **1\. Clone the** fold-stack **Repository**
 48  
 49  Git-Sync is part of the fold-stack project. Clone the repository if you haven’t already:
 50  
 51  bash
 52  
 53  git clone https://github.com/mrhavens/fold-stack.git  
 54  cd fold-stack
 55  
 56  **2\. Initialize a Local Git Repository**
 57  
 58  Git-Sync watches a local repository at volumes/repos. Initialize it if it doesn’t exist:
 59  
 60  bash
 61  
 62  mkdir \-p volumes/repos  
 63  cd volumes/repos  
 64  git init  
 65  echo "\# Test Repo" \> README.md  
 66  git add .  
 67  git commit \-m "Initial commit"  
 68  git branch \-M main  
 69  cd ../..
 70  
 71  **3\. Generate SSH Keys for GitHub and Forgejo**
 72  
 73  Git-Sync uses SSH keys to authenticate with GitHub and Forgejo.
 74  
 75  **3.1 Generate SSH Key for GitHub**  
 76  bash
 77  
 78  ssh-keygen \-t ed25519 \-C "your\_email@example.com" \-f \~/.ssh/github\_key
 79  
 80  **3.2 Generate SSH Key for Forgejo**  
 81  bash
 82  
 83  ssh-keygen \-t ed25519 \-C "your\_email@example.com" \-f \~/.ssh/forgejo\_key
 84  
 85  * Press Enter to skip setting a passphrase, or set one for added security.  
 86  * This creates \~/.ssh/github\_key and \~/.ssh/forgejo\_key (private keys) and their .pub counterparts (public keys).
 87  
 88  **4\. Configure SSH Keys on GitHub and Forgejo**
 89  
 90  **4.1 Add SSH Key to GitHub**
 91  
 92  * Copy the public key:  
 93  * bash  
 94  * cat \~/.ssh/github\_key.pub  
 95  * Go to [GitHub](https://github.com/) \> **Settings** \> **SSH and GPG keys** \> **New SSH key**.  
 96  * Title: fold-stack-git-sync.  
 97  * Key type: **Authentication Key**.  
 98  * Key: Paste the public key.  
 99  * Click **Add SSH key**.  
100  * Test the connection:  
101  * bash  
102  * ssh \-i \~/.ssh/github\_key \-T git@github.com  
103  * You should see: Hi mrhavens\! You’ve successfully authenticated....
104  
105  **4.2 Add SSH Key to Forgejo**
106  
107  * Copy the public key:  
108  * bash  
109  * cat \~/.ssh/forgejo\_key.pub  
110  * Access Forgejo at http://localhost:3000 (ensure Forgejo is running via fold-stack).  
111  * Go to **Settings** \> **SSH / GPG Keys** \> **Add Key**.  
112  * Name: fold-stack-git-sync.  
113  * Content: Paste the public key.  
114  * Click **Add Key**.  
115  * Test the connection:  
116  * bash  
117  * ssh \-i \~/.ssh/forgejo\_key \-p 2222 \-T git@localhost  
118  * You should see a success message.
119  
120  **5\. Configure Rclone for Internet Archive and Web3.storage**
121  
122  Git-Sync uses Rclone to sync Git bundles to Internet Archive and Web3.storage.
123  
124  * Run the Rclone configuration wizard:  
125  * bash  
126  * rclone config  
127  * Add the following remotes:  
128    * **Internet Archive (**ia**)**:  
129      * Choose n (new remote).  
130      * Name: ia  
131      * Type: internetarchive.  
132      * Access Key ID: Your Internet Archive access key (from archive.org account settings).  
133      * Secret Access Key: Your Internet Archive secret key.  
134      * Edit Advanced Config: n.  
135    * **Web3.storage (**web3**)**:  
136      * Choose n (new remote).  
137      * Name: web3  
138      * Type: ipfs.  
139      * Host: api.web3.storage.  
140      * API Token: Your Web3.storage API token (from web3.storage).  
141      * Edit Advanced Config: n.  
142  * Copy the Rclone configuration to the project:  
143  * bash
144  
145  mkdir \-p config/rclone  
146  cp \~/.config/rclone/rclone.conf config/rclone/rclone.conf
147  
148  * chmod 600 config/rclone/rclone.conf  
149  * Copy to git-sync config:  
150  * bash
151  
152  mkdir \-p config/git-sync
153  
154  * cp config/rclone/rclone.conf config/git-sync/rclone.conf
155  
156  **6\. Copy SSH Keys to** git-sync **Configuration**
157  
158  Copy the private keys to the git-sync secrets directory:
159  
160  bash
161  
162  mkdir \-p config/git-sync/secrets  
163  cp \~/.ssh/github\_key config/git-sync/secrets/github.key  
164  cp \~/.ssh/forgejo\_key config/git-sync/secrets/forgejo.key  
165  chmod 600 config/git-sync/secrets/github.key config/git-sync/secrets/forgejo.key
166  
167  **7\. Configure** remotes.conf
168  
169  Edit config/git-sync/remotes.conf to specify the remotes to sync to. Example:
170  
171  github|git|git@github.com:mrhavens/mirror-repo.git|1  
172  forgejo|git|git@localhost:2222/mrhavens/mirror-repo.git|1  
173  radicle|radicle|radicle://mrhavens/mirror-repo|1  
174  ia|rclone|ia:fold-stack-git-mirror|1  
175  web3|rclone|web3:fold-stack-git-mirror|0
176  
177  * **Format**: remote\_name|type|url|enabled (1 for enabled, 0 for disabled).  
178  * **Example Explanation**:  
179    * github: Syncs to mrhavens/mirror-repo on GitHub via SSH.  
180    * forgejo: Syncs to mrhavens/mirror-repo on your local Forgejo instance (port 2222).  
181    * radicle: Placeholder for Radicle (not implemented).  
182    * ia: Syncs Git bundles to fold-stack-git-mirror on Internet Archive via Rclone.  
183    * web3: Syncs Git bundles to fold-stack-git-mirror on Web3.storage (disabled by default).
184  
185  **8\. Configure Push Rules (**rules.json**)**
186  
187  Edit config/git-sync/rules.json to define which branches to sync:
188  
189  json
190  
191  {  
192    "branches": \["main", "dev"\],  
193    "exclude\_tags": \["v\*"\]  
194  }
195  
196  * **Example Explanation**:  
197    * Syncs only the main and dev branches.  
198    * Excludes tags starting with v (e.g., v1.0).
199  
200  **Note**: The current implementation syncs all branches (branches: \["\*"\]) and doesn’t exclude tags. Update entrypoint.sh to enforce these rules if needed.
201  
202  **9\. Configure Environment Variables (**.env**)**
203  
204  Edit config/git-sync/.env to set runtime options:
205  
206  SYNC\_INTERVAL=300  
207  PUSH\_MODE=push  
208  SIGN\_COMMITS=false  
209  LOG\_LEVEL=INFO  
210  RETRY\_MAX=3  
211  RETRY\_BACKOFF=5
212  
213  * **Example Explanation**:  
214    * SYNC\_INTERVAL=300: Check for changes every 300 seconds (5 minutes).  
215    * PUSH\_MODE=push: Use git push for Git remotes (alternative: bundle for Git bundles).  
216    * SIGN\_COMMITS=false: Disable GPG commit signing (placeholder).  
217    * LOG\_LEVEL=INFO: Log all messages (alternative: ERROR for errors only).  
218    * RETRY\_MAX=3: Retry failed syncs up to 3 times.  
219    * RETRY\_BACKOFF=5: Wait 5 seconds (exponential increase) between retries.
220  
221  **10\. Start the** git-sync **Service**
222  
223  Git-Sync is integrated into fold-stack’s docker-compose.dev.yml. Start the service:
224  
225  bash
226  
227  cd fold-stack  
228  ./scripts/up-dev.sh
229  
230  This starts all fold-stack services, including git-sync.
231  
232  ---
233  
234  🌐 **Usage**
235  
236  **1\. Add a Commit to the Local Repository**
237  
238  Make changes to the local repository and commit them:
239  
240  bash
241  
242  cd volumes/repos  
243  echo "New feature" \>\> README.md  
244  git add .  
245  git commit \-m "Added new feature"
246  
247  **2\. Wait for Automated Sync**
248  
249  Git-Sync will detect changes within SYNC\_INTERVAL (default: 300 seconds) and sync to all enabled remotes. Monitor the logs:
250  
251  bash
252  
253  docker logs git\_sync\_dev \--follow
254  
255  **3\. Manually Trigger a Sync**
256  
257  To sync immediately, use the manual push script:
258  
259  bash
260  
261  ./scripts/manual-push-git-sync.sh
262  
263  **4\. Verify Sync**
264  
265  * **GitHub**: Check https://github.com/mrhavens/mirror-repo.  
266  * **Forgejo**: Check http://localhost:3000/mrhavens/mirror-repo.  
267  * **Internet Archive**: Check fold-stack-git-mirror on archive.org.  
268  * **Web3.storage**: Enable in remotes.conf and check fold-stack-git-mirror.
269  
270  **5\. Generate a Sync Report**
271  
272  View the latest sync activity for each remote:
273  
274  bash
275  
276  ./scripts/report-git-sync.sh
277  
278  **Example Output: \`\`\`**
279  
280  **📊 GIT-SYNC SYNC REPORT**
281  
282  **📅** Date: Mon May 26 22:41:00 CDT 2025
283  
284  ---
285  
286  📌 **Local Repository Latest Commit**
287  
288  Commit: a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6 Message: Added new feature Time: Mon May 26 22:40:00 CDT 2025
289  
290  ---
291  
292  📌 **Latest Sync Activity by Remote**
293  
294  Remote: github (git, git@github.com:mrhavens/mirror-repo.git) Last Synced Commit: a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6 Commit Message: Added new feature Timestamp: \[Mon May 26 22:41:00 CDT 2025\] ✅ Status: Successfully synced
295  
296  Remote: forgejo (git, git@localhost:2222/mrhavens/mirror-repo.git) Last Synced Commit: a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6 Commit Message: Added new feature Timestamp: \[Mon May 26 22:41:01 CDT 2025\] ✅ Status: Successfully synced
297  
298  Remote: ia (rclone, ia:fold-stack-git-mirror) Last Synced Bundle: repo-1716777660.bundle Timestamp: \[Mon May 26 22:41:02 CDT 2025\] ✅ Status: Successfully synced
299  
300  \---
301  
302  \#\# 🛠️ Troubleshooting
303  
304  \#\#\# 1\. Run Diagnostics
305  
306  If sync fails, run the diagnostic script:
307  
308  \`\`\`bash  
309  ./scripts/diagnose-git-sync.sh
310  
311  **Example Output: \`\`\`**
312  
313  **🩺 GIT-SYNC COMPREHENSIVE DIAGNOSTICS**
314  
315  **📅** Date: Mon May 26 22:41:00 CDT 2025
316  
317  ---
318  
319  📌 **SSH Keys Check**
320  
321  **✅** /config/git-sync/secrets/github.key exists. ✅ /config/git-sync/secrets/github.key has correct permissions (600). ✅ /config/git-sync/secrets/forgejo.key exists. ✅ /config/git-sync/secrets/forgejo.key has correct permissions (600).
322  
323  ---
324  
325  📌 **Remote Connectivity Test**
326  
327  Testing github (git)... ✅ github connectivity test passed. Testing forgejo (git)... ✅ forgejo connectivity test passed. Testing ia (rclone)... ❌ ia connectivity test failed. Check rclone.conf or credentials.
328  
329  \#\#\# 2\. Common Issues and Fixes
330  
331  \- \*\*Container Not Running\*\*:  
332    \`\`\`bash  
333    ./scripts/down-dev.sh && ./scripts/up-dev.sh
334  
335  * **SSH Key Issues**:  
336    * Verify permissions: chmod 600 config/git-sync/secrets/\*.  
337    * Test connectivity: ssh \-i \~/.ssh/github\_key \-T git@github.com.  
338  * **Rclone Remote Fails**:  
339    * Reconfigure Rclone: rclone config.  
340    * Verify remotes: rclone listremotes \--config config/git-sync/rclone.conf.  
341  * **Logs Missing**:  
342    * Check volume permissions: chmod \-R 775 volumes/logs && chown \-R 1000:1000 volumes/logs.
343  
344  ---
345  
346  📚 **Advanced Configuration**
347  
348  **1\. Enable Web3.storage Sync**
349  
350  Edit config/git-sync/remotes.conf to enable Web3.storage:
351  
352  web3|rclone|web3:fold-stack-git-mirror|1
353  
354  **2\. Add a New Remote (e.g., S3)**
355  
356  Add a new remote to remotes.conf using Rclone:
357  
358  s3|rclone|s3:fold-stack-git-mirror|1
359  
360  Configure the s3 remote in config/git-sync/rclone.conf using rclone config.
361  
362  **3\. Adjust Sync Interval**
363  
364  Edit config/git-sync/.env to change the sync interval:
365  
366  SYNC\_INTERVAL=60  \# Check every 60 seconds
367  
368  Restart the service:
369  
370  bash
371  
372  docker compose \-f docker-compose.dev.yml stop git-sync  
373  docker compose \-f docker-compose.dev.yml up \-d git-sync
374  
375  **4\. Enable Commit Signing (Future Feature)**
376  
377  To enable GPG commit signing (not yet implemented), set:
378  
379  SIGN\_COMMITS=true
380  
381  You’ll need to:
382  
383  * Add GPG keys to the container.  
384  * Update entrypoint.sh to sign commits using git.
385  
386  ---
387  
388  📅 **Last Updated**
389  
390  This README was last updated on **May 26, 2025, at 10:54 PM CDT**.
391  
392  ---
393