qdrant-auto-indexing.cspec
1 name: Qdrant Automatic Code Indexing 2 version: 1.0.0 3 implemented: 2026-01-27 4 status: operational 5 server: ci.ac-dc.network (10.106.0.2) 6 7 purpose: | 8 Automatic code embedding updates for Qdrant vector database. 9 Git post-receive hooks trigger re-indexing on every push to main branch. 10 11 Benefits: 12 - Always up-to-date semantic search 13 - No manual indexing needed 14 - Minimal push latency (background processing) 15 - Automatic working directory management 16 17 architecture: 18 components: 19 qdrant_server: 20 host: ci.ac-dc.network 21 port: 6333 22 collections: 23 alpha_delta_code: 24 vectors: 298611 25 status: green 26 indexed: 98.7% 27 28 embedder: 29 script: /home/devops/working-repos/alpha-delta-context/tooling/shadow-evaluator/embedder.py 30 model: all-MiniLM-L6-v2 31 vector_size: 384 32 distance: Cosine 33 34 git_hooks: 35 type: post-receive 36 location: /var/lib/forgejo/repositories/alpha-delta-network/{repo}.git/hooks/post-receive.d/qdrant-indexer 37 repos: 38 - alphavm 39 - deltavm 40 - alphaos 41 - deltaos 42 - adnet 43 - acdc-core 44 - ac-dc 45 - adl 46 47 working_directories: /tmp/qdrant-indexing/{repo}/ 48 log_file: /var/log/qdrant-indexing.log 49 50 hook_workflow: 51 trigger: git push to main branch 52 steps: 53 1_receive_push: Forgejo receives push to main 54 2_dispatcher: Main post-receive hook executes 55 3_hook_execution: Calls qdrant-indexer in post-receive.d/ 56 4_checkout: Updates working directory from bare repo 57 5_background_index: Launches embedder in background (non-blocking) 58 6_completion: Push completes immediately, indexing continues async 59 60 hook_script_logic: | 61 #!/bin/bash 62 # 1. Read git stdin (oldrev newrev refname) 63 # 2. Check if push is to main branch 64 # 3. Clone/update working directory at /tmp/qdrant-indexing/{repo} 65 # 4. Launch embedder in background with nohup 66 # 5. Log to /var/log/qdrant-indexing.log 67 # 6. Exit immediately (non-blocking) 68 69 operations: 70 manual_reindex: 71 single_repo: | 72 python3 /home/devops/working-repos/alpha-delta-context/tooling/shadow-evaluator/embedder.py \ 73 embed --repo alphavm --path /tmp/qdrant-indexing/alphavm 74 75 all_repos: | 76 python3 /home/devops/working-repos/alpha-delta-context/tooling/shadow-evaluator/embedder.py \ 77 embed-all --base /tmp/qdrant-indexing 78 79 check_status: 80 collection_stats: | 81 curl -s http://ci.ac-dc.network:6333/collections/alpha_delta_code | jq '{points: .result.points_count, indexed: .result.indexed_vectors_count, status: .result.status}' 82 83 recent_logs: | 84 ssh -p 2584 devops@10.106.0.2 "tail -50 /var/log/qdrant-indexing.log" 85 86 active_indexing: | 87 ssh -p 2584 devops@10.106.0.2 "ps aux | grep embedder.py | grep -v grep" 88 89 test_hook: 90 trigger_manually: | 91 cd /home/devops/working-repos/alphavm 92 git commit --allow-empty -m "test: trigger indexing" 93 git push 94 95 verify_execution: | 96 ssh -p 2584 devops@10.106.0.2 "tail /var/log/qdrant-indexing.log" 97 98 maintenance: 99 clean_working_dirs: 100 command: ssh -p 2584 devops@10.106.0.2 "sudo rm -rf /tmp/qdrant-indexing/*" 101 note: Working directories will be recreated on next push 102 103 rotate_logs: 104 command: ssh -p 2584 devops@10.106.0.2 "sudo truncate -s 0 /var/log/qdrant-indexing.log" 105 schedule: manual (or add logrotate config) 106 107 update_hook: 108 steps: 109 - Edit hook script locally 110 - scp -P 2584 /tmp/qdrant-index-hook.sh devops@10.106.0.2:/tmp/ 111 - ssh -p 2584 devops@10.106.0.2 "sudo cp /tmp/qdrant-index-hook.sh /var/lib/forgejo/repositories/alpha-delta-network/*/hooks/post-receive.d/qdrant-indexer" 112 - ssh -p 2584 devops@10.106.0.2 "sudo chmod +x /var/lib/forgejo/repositories/alpha-delta-network/*/hooks/post-receive.d/qdrant-indexer" 113 114 troubleshooting: 115 hook_not_executing: 116 check: ssh -p 2584 devops@10.106.0.2 "cat /var/log/qdrant-indexing.log" 117 causes: 118 - Hook not executable (chmod +x) 119 - Wrong ownership (should be git:git) 120 - Push to non-main branch 121 - Hook script syntax error 122 fix: Verify hook permissions and test with empty commit 123 124 indexing_failing: 125 check: ssh -p 2584 devops@10.106.0.2 "grep ERROR /var/log/qdrant-indexing.log" 126 causes: 127 - Embedder script not found 128 - Python dependencies missing 129 - Qdrant server down 130 - Working directory permissions 131 fix: | 132 # Test embedder manually 133 python3 /home/devops/working-repos/alpha-delta-context/tooling/shadow-evaluator/embedder.py stats 134 135 slow_pushes: 136 check: Monitor push times in git client 137 note: Indexing runs in background, should not affect push speed 138 issue: If pushes are slow, hook may be running synchronously 139 fix: Ensure nohup and background (&) are in hook script 140 141 integration: 142 forgejo_hooks: 143 dispatcher: /var/lib/forgejo/repositories/alpha-delta-network/{repo}.git/hooks/post-receive 144 hook_directory: post-receive.d/ 145 execution_order: 146 1: gitea (Forgejo internal) 147 2: qdrant-indexer (our hook) 148 149 note: Forgejo dispatcher runs all hooks in post-receive.d/ sequentially 150 151 embedder_features: 152 incremental_updates: true 153 file_hashing: Detects changed files only 154 chunk_extraction: Respects function boundaries 155 symbol_indexing: Extracts function/type names 156 skip_tests: Configured in config.yaml 157 158 monitoring: 159 metrics: 160 hook_executions: grep "Processing push" /var/log/qdrant-indexing.log | wc -l 161 successful_indexes: grep "Index update queued" /var/log/qdrant-indexing.log | wc -l 162 errors: grep -E "ERROR|FAIL" /var/log/qdrant-indexing.log 163 164 collection_growth: 165 query: curl -s http://ci.ac-dc.network:6333/collections/alpha_delta_code 166 expected_growth: ~100-500 vectors per repo update 167 168 performance: 169 indexing_time: 170 small_change: 5-30 seconds (1-10 files) 171 medium_change: 30-120 seconds (10-50 files) 172 full_repo: 5-15 minutes (initial index) 173 174 push_latency: 175 added: <100ms (background execution) 176 note: Hook spawns process and exits immediately 177 178 security: 179 permissions: 180 hook_owner: git:git 181 log_owner: devops:devops 182 work_dir_owner: devops:devops 183 184 execution_context: 185 user: git (Forgejo's user) 186 python_env: system Python 3.12 187 network_access: yes (connects to Qdrant on localhost:6333) 188 189 deployment: 190 date: 2026-01-27 191 method: manual_installation 192 193 installation_commands: | 194 # 1. Create hook script 195 # 2. Copy to CI server 196 scp -P 2584 /tmp/qdrant-index-hook.sh devops@10.106.0.2:/tmp/ 197 198 # 3. Setup directories 199 ssh -p 2584 devops@10.106.0.2 "sudo mkdir -p /tmp/qdrant-indexing" 200 ssh -p 2584 devops@10.106.0.2 "sudo touch /var/log/qdrant-indexing.log" 201 ssh -p 2584 devops@10.106.0.2 "sudo chown devops:devops /tmp/qdrant-indexing /var/log/qdrant-indexing.log" 202 203 # 4. Install in all repos 204 for repo in alphavm deltavm alphaos deltaos adnet acdc-core ac-dc adl; do 205 sudo cp /tmp/qdrant-index-hook.sh /var/lib/forgejo/repositories/alpha-delta-network/${repo}.git/hooks/post-receive.d/qdrant-indexer 206 sudo chmod +x /var/lib/forgejo/repositories/alpha-delta-network/${repo}.git/hooks/post-receive.d/qdrant-indexer 207 sudo chown git:git /var/lib/forgejo/repositories/alpha-delta-network/${repo}.git/hooks/post-receive.d/qdrant-indexer 208 done 209 210 related_documentation: 211 forgejo_actions: /home/devops/working-repos/alpha-delta-context/infra/machine/forgejo-actions-runners.cspec 212 embedder_config: /home/devops/working-repos/alpha-delta-context/tooling/shadow-evaluator/config.yaml 213 qdrant_mcp: /home/devops/working-repos/alpha-delta-context/.claude/mcp/qdrant.py