/ AGENTS.md
AGENTS.md
1 # ShanClaw — Project Guide 2 3 ## What This Is 4 5 Go CLI tool (`shan`) — the runtime for Shannon AI agents. The primary production stack is **daemon + ShanClaw Desktop + Shannon Cloud**: the daemon connects to Cloud via WebSocket, receives channel messages, runs the agent loop locally with full tool access, and streams results back. ShanClaw also supports interactive TUI, one-shot CLI, MCP server mode, and local scheduled tasks. 6 7 ## Tech Stack 8 9 - **Go 1.25.7** — `go.mod` is source of truth 10 - **Cobra** — CLI framework 11 - **gorilla/websocket** — daemon WebSocket client (primary production path) 12 - **Bubbletea v1.3.10 + Bubbles v1.0.0** — TUI 13 - **modernc.org/sqlite** — pure-Go SQLite for session FTS5 search index 14 - **adhocore/gronx** — cron expression validation 15 - **chromedp** — browser automation (isolated Chrome profile) 16 - **mcp-go** — MCP client/server 17 - **adrg/frontmatter** — YAML frontmatter parsing for SKILL.md files 18 19 ## Project Structure 20 21 ``` 22 cmd/ 23 root.go # entry, one-shot mode, MCP serve 24 daemon.go # shan daemon start/stop/status 25 schedule.go # scheduled task management 26 update.go # self-update command 27 28 internal/ 29 daemon/ # PRIMARY PRODUCTION PATH 30 server.go # HTTP API server 31 runner.go # agent run orchestration, session routing, output format profiles 32 client.go # WebSocket client with reconnect, bounded concurrency 33 router.go # SessionCache, route locking 34 approval.go # interactive tool approval over WS 35 types.go # daemon request/response types, disconnect, approval messages 36 events.go # EventBus ring buffer for daemon/SSE subscribers 37 session_cwd.go # cloud-source scratch CWD allocator (ephemeral per-session tmp dir) 38 agent/ 39 loop.go # AgentLoop.Run() — core agentic loop 40 tools.go # Tool interface, ToolRegistry, filtering, schemas 41 partition.go # read-only batching, executeBatches 42 spill.go # large tool result spill-to-disk 43 deferred.go # deferred tool loading (tool_search) 44 statecache.go # state-aware tool result cache keyed by read/write state 45 resultshape.go # tree result shaping and stable change summaries 46 microcompact.go # Tier 2 semantic compaction for large native tool results 47 delta.go # DeltaProvider interface, TemporalDelta (date rollover) 48 loopdetect.go # stuck-loop detectors 49 readtracker.go # read-before-edit enforcement 50 approval_cache.go # per-turn approval caching 51 normalize.go # response normalization 52 skill_discovery.go # Per-turn small-model skill matching (discoverRelevantSkills) 53 agents/ 54 loader.go # LoadAgent, ListAgents, ParseAgentMention 55 api.go # daemon-side agent CRUD 56 validate.go # agent validation and builtin commands 57 embed.go # EnsureBuiltins, MaterializeBuiltin, bundled agents 58 builtin/ # Bundled agent definitions (explorer, reviewer) 59 client/ 60 gateway.go # GatewayClient: Complete, CompleteStream, ListTools 61 sse.go # SSE event parsing 62 ollama.go # Ollama provider via OpenAI-compatible chat/tool APIs 63 config/ 64 config.go # multi-level config loading and merge 65 settings.go # UI settings 66 setup.go # setup wizard 67 cwdctx/ 68 cwdctx.go # session-scoped CWD: context propagation, path resolution 69 context/ 70 window.go # token estimation, compaction shaping 71 summarize.go # two-phase conversation summary generation 72 persist.go # write-before-compact memory extraction 73 session/ 74 store.go # session JSON persistence 75 manager.go # session lifecycle, search, OnClose callbacks 76 index.go # SQLite FTS5 search index 77 title.go # session title generation 78 prompt/ 79 builder.go # static/stable/volatile prompt assembly, output profiles 80 instructions/ 81 loader.go # instructions, memory, custom commands 82 tools/ 83 register.go # local + MCP + gateway tool registration 84 schedule.go # schedule_create/list/update/remove tools 85 session_search.go # session_search tool 86 mcp_tool.go # MCPTool adapter 87 server.go # ServerTool adapter (gateway tools) 88 skills/ 89 registry.go # skill metadata 90 loader.go # skill loading 91 validate.go # skill name validation 92 memory/ 93 types.go # Wire schemas mirroring the Kocoro Cloud memory sidecar HTTP contract 94 errclass.go # ErrorClass enum + ClassifyHTTP (sub_code → class) 95 config.go # LoadConfig, ResolveAPIKey, ResolveEndpoint 96 tenant.go # sha256 fingerprint + tenant-switch detection 97 audit.go # AuditLogger interface (boolean-only key/endpoint state) 98 client.go # UDS HTTP client (Query/Reload/Health, X-Request-ID, ctx-cancel dial) 99 sidecar.go # Spawn/WaitReady/Shutdown + AttachPolicy + Supervisor 100 bundle.go # Puller: manifest fetch, tenant check, sandboxed stage, atomic install, retention 101 service.go # Orchestrator: status FSM, supervisor + puller goroutines, NewServiceAttached 102 mcp/ 103 client.go # MCP client manager 104 server.go # MCP server 105 chrome.go # Playwright Chrome profile/CDP lifecycle management 106 runstatus/ 107 runstatus.go # user-facing run state/error classification 108 schedule/ 109 schedule.go # schedule CRUD, atomic writes, validation 110 launchd_darwin.go # plist generation, launchctl 111 launchd_stub.go # non-darwin stub 112 permissions/ 113 permissions.go # hard-block > denied > split compounds > allowed > default safe > ask 114 audit/ 115 audit.go # JSON-lines logger, redaction 116 hooks/ 117 hooks.go # PreToolUse/PostToolUse/SessionStart/Stop 118 tui/ 119 app.go # Bubbletea app 120 doctor.go # TUI diagnostic checks 121 compact.go # TUI /compact command 122 update/ 123 selfupdate.go # GitHub release auto-update 124 sync/ 125 sync.go # Run(ctx, deps) — flock, marker read, scan, batch, upload, write, audit 126 marker.go # Marker + FailedEntry; atomic read/write; sidecar on unknown version 127 config.go # Typed view of sync.* config keys 128 scanner.go # Multi-dir candidate discovery; failed-retry union; exclusions 129 batcher.go # Marshal-once; dual-cap (sessions+bytes) packing; oversized + load-error rejection 130 uploader.go # Uploader interface; CloudUploader + DryRunUploader; response anomaly normalization 131 backoff.go # Reason classification + transient backoff math 132 ``` 133 134 ## Key Conventions 135 136 ### Kocoro Skill Co-Maintenance 137 The `kocoro` bundled skill (`internal/skills/bundled/skills/kocoro/`) is a platform configuration assistant. Its SKILL.md and reference files (`references/*.md`) describe daemon API endpoints, config fields, and workflows. Kocoro is the AI's only source of truth for ShanClaw's HTTP surface — missing docs cause it to hallucinate workarounds. **Adding a new endpoint or feature counts as a trigger, not only modifying existing ones**; any `mux.HandleFunc(...)` in `internal/daemon/server.go` must have a matching reference entry. See CLAUDE.md for the full mapping. 138 139 ### Agent Names 140 141 Must match `^[a-z0-9][a-z0-9_-]{0,63}$`. Validate before any path concatenation to prevent traversal. 142 143 ### Tool Priority 144 145 Local tools > MCP tools > Gateway tools. Deduplicate by name in the registry. 146 147 ### Skill Discovery 148 149 Three-layer system for triggering `use_skill`: 150 1. **Skill listing** — full descriptions embedded in the scaffolded user message on first turn. 151 2. **Semantic discovery** — blocking `model_tier: "small"` call on iteration 0 (5s timeout). Gated by `agent.skill_discovery` config (default `true`). 152 3. **Fallback catalog** — `use_skill` tool description includes all loaded skill names. 153 154 **Skill allowed-tools** uses execution-time denial, not schema filtering, to keep the tools array byte-stable for prompt cache. 155 156 ### Permission Model 157 158 ``` 159 hard-block constants → denied_commands → compound-command splitting → allowed_commands → default safe → RequiresApproval + SafeChecker 160 ``` 161 162 Unknown tools are denied by default. 163 164 ### Daemon Architecture 165 166 - Daemon connects to Shannon Cloud via WebSocket, receives channel messages, and runs the agent loop locally. 167 - Route keys are computed as: 168 - `agent:<name>` for agent-scoped sessions 169 - `session:<id>` for explicit session resume 170 - `default:<source>:<channel>` for routed channel sessions 171 - Routed managers are long-lived. Ephemeral runs (for example bypass/heartbeat paths) use short-lived managers. 172 - Output formatting uses profiles, not per-channel syntax: 173 - `markdown` is the default 174 - `plain` is used for cloud-distributed channels where Shannon Cloud owns final rendering 175 - Tool status `running` is emitted at actual execution start, not during approval/permission checks. 176 - Large tool results spill to `~/.shannon/tmp/` and are cleaned up: 177 - per-run in daemon and TUI 178 - on manager close in one-shot mode 179 - **Session sync** (`internal/sync/`): uploads local session JSON to Shannon Cloud once per day (opt-in via `sync.enabled`). Single entry point `sync.Run`; called from the daemon ticker and the `shan sessions sync` CLI; flock + atomic marker write serialize concurrent callers. Per-session ACK with persistent `marker.failed` bookkeeping; permanent reasons (`size_limit_exceeded`, `load_error`) stay forever and self-heal on session edit. 180 - **Memory client** (`internal/memory/`, Phase 2.3): daemon owns sidecar lifecycle (spawn / health / restart / shutdown) and the 24h bundle pull loop. Tool `memory_recall` (`internal/tools/memory.go`) delegates to `memory.Service.Query` via UDS; falls back to `session_search` + MEMORY.md whenever `Service.Status() != Ready`. CLI/TUI use `memory.AttachPolicy` (probe-only, never spawn) and connect via `memory.NewServiceAttached`. Privacy invariant: the resolved API key bytes never reach disk or audit logs (only `sha256[:16]` fingerprint in `<bundle_root>/.tenant_fingerprint`). 181 182 ### Turn Lifecycle 183 184 The agent loop declares an explicit phase state machine (`internal/agent/phase.go`) that external observers can reason about: 185 186 - **Phases**: `PhaseAwaitingLLM`, `PhaseExecutingTools`, `PhaseCompacting`, `PhaseAwaitingApproval`, `PhaseRetryingLLM`, `PhaseForceStop`, `PhaseInjectingMessage`, etc. Only `PhaseAwaitingLLM` and `PhaseForceStop` count as idle for the watchdog. 187 - **Idle watchdog**: with `agent.idle_soft_timeout_secs` > 0 the daemon fires an `EventRunStatus` event (code `idle_soft`) after that long in an idle-counted phase. With `agent.idle_hard_timeout_secs` > 0 the run is cancelled with `ErrHardIdleTimeout` — the partial transcript is still persisted (soft error). Defaults: soft=90, hard=0 (visibility-only). 188 - **Mid-turn checkpoint**: after each tool batch, after successful reactive compaction, and before a force-stop, the daemon rebuilds the on-disk session from a baseline + current loop snapshot. The same rebuild runs at final save so a turn is never persisted twice. `session.Session.InProgress=true` on reload indicates a crash-recovered session with a partial transcript. 189 - **Event types**: `EventRunStatus` (watchdog soft/hard, LLM retries) joins the existing `EventAgentReply`, `EventToolStatus`, `EventApprovalRequest` stream. 190 191 ### Browser Preview Bridge 192 193 For daemon runs with Playwright, `browser_navigate(file://…)` is transparently rewritten to a short-lived `http://127.0.0.1/<token>/<name>` URL served by a loopback HTTP server bound per-session: 194 - Fail-closed allowlist populated from the effective session CWD + user-attached paths. Browser reach never exceeds `permissions.CheckFilePath`. 195 - Symlinks resolved on both sides of the allowlist check; escapes rejected. 196 - Random 16-byte hex token per file; no directory listing; teardown on session close. 197 198 ### Config Merge Order 199 200 1. `~/.shannon/config.yaml` (global) 201 2. `.shannon/config.yaml` (project) 202 3. `.shannon/config.local.yaml` (local, gitignored) 203 204 Scalars override, lists merge+dedup, structs merge field-by-field. MCP server env var casing is preserved via direct YAML re-read. 205 206 ### File Paths 207 208 - Agent definitions: `~/.shannon/agents/<name>/AGENT.md`, `MEMORY.md`, `config.yaml`, `commands/*.md`, `_attached.yaml` 209 - Global skills: `~/.shannon/skills/<skill-name>/SKILL.md` 210 - Sessions: `~/.shannon/sessions/` or `~/.shannon/agents/<name>/sessions/` 211 - Session index: `<sessions-dir>/sessions.db` 212 - Spill files: `~/.shannon/tmp/tool_result_<session>_<call_id>.txt` 213 - Schedule index: `~/.shannon/schedules.json` 214 - Schedule plists: `~/Library/LaunchAgents/com.shannon.schedule.<id>.plist` 215 - Sync marker: `~/.shannon/sync_marker.json` 216 - Sync lock (flock): `~/.shannon/sync.lock` (never delete) 217 - Sync dry-run outbox: `~/.shannon/sync_outbox/` (only when `sync.dry_run=true`) 218 - Audit log: `~/.shannon/logs/audit.log` 219 - Schedule logs: `~/.shannon/logs/schedule-<id>.log` 220 - Memory sidecar socket: `~/.shannon/memory.sock` 221 - Memory bundle root: `~/.shannon/memory/` (with `bundles/<ts>/`, `current` symlink, `.tenant_fingerprint`, `bundle.lock`) 222 223 ### Prompt Cache 224 225 Source-routed TTL: channels/TUI get 1h, one-shot/subagent paths get 5m (fail-cheap default). `cache_source` tags every LLM call and propagates on the wire. `normalizeToolInput` canonicalizes nested JSON for byte stability. See `docs/cache-strategy.md` for breakpoint layout, source table, env-var overrides, and maintenance playbook. 226 227 ### Context Management 228 229 - **Proactive compaction**: persist learnings, then generate a two-phase summary, then shape history when nearing the context window. 230 - **Reactive compaction**: on context-length error, emergency compact once and retry once. `reactiveCompacted` prevents loops. 231 - **Tiered result compression**: 232 - Tier 1: old results collapse to metadata only 233 - Tier 2: mid-age results use head+tail truncation, with micro-compact for large native tool results when a small-model completer is available 234 - Tier 3: recent results stay full 235 - **Deferred tool loading**: when the toolset is large and includes deferred sources, MCP/gateway tools are exposed as summaries until the model loads schemas through `tool_search`. 236 - **Memory staleness**: dated memory headings get freshness annotations like `[today]` and `[N days ago]`. 237 - **System reminders**: short reminder blocks are appended to high-signal tool results (`file_read`, `file_write`, `file_edit`, `bash`) to reinforce key instructions in long sessions. 238 - **Disk spill**: very large tool outputs are written to disk and replaced in context with a short preview plus a readable path. 239 240 ### Anti-Hallucination 241 242 - XML `<tool_exec>` delimiters use random hex call IDs. 243 - Preamble text is suppressed when the response contains tool calls. 244 - Fabricated tool calls are detected and stripped. 245 246 ## Testing 247 248 ```bash 249 go test ./... # all tests 250 go test ./internal/agent/ -v # agent loop, batching, compaction, spill, deferred 251 go test ./internal/daemon/ -v # daemon WS client, router, runner 252 go test ./internal/agents/ -v # agent loader 253 go test ./internal/schedule/ -v # schedule CRUD + plist tests 254 go test ./test/ -v # E2E coverage 255 go test ./test/e2e/ -v # E2E offline: agents, schedule, session, MCP, cache 256 SHANNON_E2E_LIVE=1 go test ./test/e2e/ -v # E2E live: one-shot, bundled agents (daemon skipped until isolated) 257 go build ./... # build check 258 ``` 259 260 Schedule tests use temp directories and never write to the real LaunchAgents directory. 261 262 E2E tests in `test/e2e/` split into offline (no API) and live (`SHANNON_E2E_LIVE=1`). Run live tests before releases. 263 264 ## Building & Releasing 265 266 - GoReleaser: `.goreleaser.yaml` 267 - npm package: `@kocoro/shanclaw` 268 - Versioning is PATCH-only (`0.0.x`) unless explicitly directed otherwise 269 - Release flow: tag → push tag → CI builds and publishes 270 - `docs/` is gitignored — documentation lives locally only 271 272 ## Tool Inventory 273 274 ### Core Local Tools 275 276 - File ops: `file_read`, `file_write`, `file_edit`, `glob`, `grep`, `directory_list` 277 - Shell/system: `bash`, `system_info`, `process`, `http`, `think` 278 - macOS GUI: `accessibility`, `applescript`, `screenshot`, `computer`, `clipboard`, `notify`, `browser`, `wait_for`, `ghostty` 279 - Schedule: `schedule_create`, `schedule_list`, `schedule_update`, `schedule_remove` 280 - Memory: `memory_append` 281 - Skills: `use_skill` 282 283 ### Runtime-Conditional Tools 284 285 - Session: `session_search` when a session manager is present 286 - Cloud: `cloud_delegate` when gateway/cloud access is enabled 287 - Meta: `tool_search` in deferred mode only