claude.md
1 # test/ 2 3 **Context:** Integration & End-to-End Testing for ECHO 4 5 This directory contains integration tests, end-to-end tests, and test fixtures for the ECHO multi-agent system. 6 7 ## Purpose 8 9 Testing ensures: 10 - **Integration Testing** - Multi-agent workflows and interactions 11 - **End-to-End Testing** - Complete system scenarios 12 - **Regression Testing** - Prevent breaking changes 13 - **Performance Testing** - Response times and throughput 14 - **Reliability Testing** - Error handling and recovery 15 16 ## Directory Structure 17 18 ``` 19 test/ 20 ├── claude.md # This file 21 ├── integration/ # Multi-agent integration tests 22 │ ├── autonomous_mode_test.exs 23 │ ├── collaborative_mode_test.exs 24 │ ├── hierarchical_mode_test.exs 25 │ └── human_loop_test.exs 26 ├── e2e/ # End-to-end system tests 27 │ ├── full_workflow_test.exs 28 │ ├── decision_flow_test.exs 29 │ └── escalation_test.exs 30 └── fixtures/ # Test data and helpers 31 ├── decisions.json 32 ├── messages.json 33 └── test_helpers.ex 34 ``` 35 36 ## Test Organization 37 38 ### Unit Tests 39 Located in each component directory: 40 - `apps/echo_shared/test/` - Shared library tests 41 - `apps/ceo/test/` - CEO agent tests 42 - `apps/*/test/` - Individual agent tests 43 44 **Run unit tests:** 45 ```bash 46 # Test shared library 47 cd apps/echo_shared && mix test 48 49 # Test specific agent 50 cd apps/ceo && mix test 51 52 # Test all agents 53 ./scripts/testing/test_all_agents.sh 54 ``` 55 56 ### Integration Tests 57 Located in `test/integration/` - Multi-agent scenarios 58 59 **Purpose:** Test agent-to-agent communication, decision flows, and workflows 60 61 **Run integration tests:** 62 ```bash 63 cd test/integration 64 mix test 65 66 # Run specific test 67 mix test test/integration/collaborative_mode_test.exs 68 69 # Run with specific tag 70 mix test --only integration 71 ``` 72 73 ### End-to-End Tests 74 Located in `test/e2e/` - Complete system scenarios 75 76 **Purpose:** Test full workflows from initiation to completion 77 78 **Run E2E tests:** 79 ```bash 80 cd test/e2e 81 mix test 82 83 # Run with coverage 84 mix test --cover 85 86 # Run with trace 87 mix test --trace 88 ``` 89 90 ## Integration Test Patterns 91 92 ### Pattern 1: Autonomous Decision Test 93 94 ```elixir 95 defmodule ECHO.Integration.AutonomousModeTest do 96 use ExUnit.Case 97 alias EchoShared.{DecisionEngine, MessageBus, Repo} 98 alias EchoShared.Schemas.Decision 99 100 setup do 101 # Clean database before each test 102 Repo.delete_all(Decision) 103 :ok 104 end 105 106 test "CEO approves budget decision autonomously within authority limit" do 107 # 1. Create decision within CEO's authority ($1M) 108 {:ok, decision} = DecisionEngine.initiate_decision( 109 initiator_role: :ceo, 110 mode: :autonomous, 111 decision_type: "budget_approval", 112 description: "Approve Q1 marketing budget", 113 context: %{"amount" => 500_000, "department" => "marketing"} 114 ) 115 116 # 2. Wait for decision processing 117 :timer.sleep(1000) 118 119 # 3. Verify decision was approved autonomously 120 decision = Repo.get!(Decision, decision.id) 121 assert decision.status == "approved" 122 assert decision.mode == :autonomous 123 assert decision.approver_role == :ceo 124 end 125 126 test "CEO escalates budget decision exceeding authority limit" do 127 # 1. Create decision exceeding CEO's authority ($1M) 128 {:ok, decision} = DecisionEngine.initiate_decision( 129 initiator_role: :ceo, 130 mode: :autonomous, 131 decision_type: "budget_approval", 132 description: "Approve major acquisition", 133 context: %{"amount" => 5_000_000, "department" => "corporate"} 134 ) 135 136 # 2. Wait for escalation processing 137 :timer.sleep(1000) 138 139 # 3. Verify decision was escalated to human 140 decision = Repo.get!(Decision, decision.id) 141 assert decision.status == "pending_human_approval" 142 assert decision.mode == :human 143 assert decision.escalated_to == :human 144 end 145 end 146 ``` 147 148 ### Pattern 2: Collaborative Decision Test 149 150 ```elixir 151 defmodule ECHO.Integration.CollaborativeModeTest do 152 use ExUnit.Case 153 alias EchoShared.{DecisionEngine, Repo} 154 alias EchoShared.Schemas.{Decision, DecisionVote} 155 156 test "Architecture decision requires consensus from CTO, Architect, and Dev" do 157 # 1. Initiate collaborative decision 158 {:ok, decision} = DecisionEngine.initiate_decision( 159 initiator_role: :cto, 160 mode: :collaborative, 161 decision_type: "architecture_change", 162 description: "Migrate to microservices", 163 context: %{"current": "monolith", "proposed": "microservices"}, 164 participants: [:cto, :senior_architect, :senior_developer] 165 ) 166 167 # 2. Simulate votes from participants 168 DecisionEngine.vote(decision.id, :cto, true, "Supports scalability") 169 DecisionEngine.vote(decision.id, :senior_architect, true, "Good design") 170 DecisionEngine.vote(decision.id, :senior_developer, false, "Complexity concerns") 171 172 # 3. Wait for consensus calculation 173 :timer.sleep(500) 174 175 # 4. Verify decision outcome 176 decision = Repo.get!(Decision, decision.id) |> Repo.preload(:votes) 177 assert length(decision.votes) == 3 178 assert decision.consensus_score == 0.67 # 2/3 approved 179 assert decision.status == "approved" # Threshold met 180 end 181 end 182 ``` 183 184 ### Pattern 3: Hierarchical Escalation Test 185 186 ```elixir 187 defmodule ECHO.Integration.HierarchicalModeTest do 188 use ExUnit.Case 189 alias EchoShared.{DecisionEngine, Repo} 190 alias EchoShared.Schemas.Decision 191 192 test "Developer decision escalates through hierarchy: Dev -> Architect -> CTO" do 193 # 1. Developer initiates decision 194 {:ok, decision} = DecisionEngine.initiate_decision( 195 initiator_role: :senior_developer, 196 mode: :hierarchical, 197 decision_type: "implementation_approach", 198 description: "Use experimental library", 199 context: %{"library": "experimental-lib-v0.1.0"} 200 ) 201 202 # 2. Developer cannot approve (requires escalation) 203 :timer.sleep(500) 204 decision = Repo.get!(Decision, decision.id) 205 assert decision.escalated_to == :senior_architect 206 207 # 3. Architect escalates to CTO (high risk) 208 DecisionEngine.escalate(decision.id, :senior_architect, :cto, "Requires CTO approval") 209 :timer.sleep(500) 210 decision = Repo.get!(Decision, decision.id) 211 assert decision.escalated_to == :cto 212 213 # 4. CTO approves 214 DecisionEngine.approve(decision.id, :cto, "Approved for experimentation") 215 :timer.sleep(500) 216 decision = Repo.get!(Decision, decision.id) 217 assert decision.status == "approved" 218 assert decision.approver_role == :cto 219 end 220 end 221 ``` 222 223 ### Pattern 4: Message Bus Integration Test 224 225 ```elixir 226 defmodule ECHO.Integration.MessageBusTest do 227 use ExUnit.Case 228 alias EchoShared.MessageBus 229 230 test "CEO sends message to CTO, CTO receives and responds" do 231 # 1. Subscribe to channels 232 MessageBus.subscribe("messages:cto") 233 MessageBus.subscribe("messages:ceo") 234 235 # 2. CEO sends message to CTO 236 {:ok, message_id} = MessageBus.send_message( 237 from_role: :ceo, 238 to_role: :cto, 239 type: :request, 240 subject: "Architecture review needed", 241 content: %{"document_url" => "https://example.com/design.pdf"} 242 ) 243 244 # 3. CTO receives message (via pub/sub) 245 assert_receive {:message, "messages:cto", payload}, 1000 246 assert payload.from_role == "ceo" 247 assert payload.subject == "Architecture review needed" 248 249 # 4. CTO responds 250 {:ok, response_id} = MessageBus.send_message( 251 from_role: :cto, 252 to_role: :ceo, 253 type: :response, 254 subject: "Re: Architecture review needed", 255 content: %{"status" => "reviewed", "feedback" => "Looks good"}, 256 in_reply_to: message_id 257 ) 258 259 # 5. CEO receives response 260 assert_receive {:message, "messages:ceo", response_payload}, 1000 261 assert response_payload.in_reply_to == message_id 262 assert response_payload.content["status"] == "reviewed" 263 end 264 end 265 ``` 266 267 ## End-to-End Test Patterns 268 269 ### E2E Pattern: Full Decision Workflow 270 271 ```elixir 272 defmodule ECHO.E2E.FullWorkflowTest do 273 use ExUnit.Case 274 alias EchoShared.{DecisionEngine, MessageBus, Repo} 275 276 @moduletag :e2e 277 278 test "Complete workflow: Product Manager proposes feature -> Architecture -> Implementation -> Testing" do 279 # 1. PM proposes new feature 280 {:ok, decision} = DecisionEngine.initiate_decision( 281 initiator_role: :product_manager, 282 mode: :collaborative, 283 decision_type: "feature_proposal", 284 description: "Add user authentication", 285 participants: [:product_manager, :senior_architect, :senior_developer, :test_lead] 286 ) 287 288 # 2. All participants vote 289 DecisionEngine.vote(decision.id, :product_manager, true, "High priority") 290 DecisionEngine.vote(decision.id, :senior_architect, true, "Will design") 291 DecisionEngine.vote(decision.id, :senior_developer, true, "Can implement") 292 DecisionEngine.vote(decision.id, :test_lead, true, "Will test") 293 294 # 3. Wait for approval 295 :timer.sleep(1000) 296 decision = Repo.get!(Decision, decision.id) 297 assert decision.status == "approved" 298 299 # 4. Architect creates design decision 300 {:ok, design_decision} = DecisionEngine.initiate_decision( 301 initiator_role: :senior_architect, 302 mode: :collaborative, 303 decision_type: "architecture_design", 304 description: "OAuth 2.0 + JWT tokens", 305 participants: [:senior_architect, :senior_developer, :cto], 306 parent_decision_id: decision.id 307 ) 308 309 # 5. Design gets approved 310 DecisionEngine.vote(design_decision.id, :senior_architect, true, "Designed") 311 DecisionEngine.vote(design_decision.id, :senior_developer, true, "Agree") 312 DecisionEngine.vote(design_decision.id, :cto, true, "Approved") 313 :timer.sleep(1000) 314 315 # 6. Developer implements 316 {:ok, impl_decision} = DecisionEngine.initiate_decision( 317 initiator_role: :senior_developer, 318 mode: :autonomous, 319 decision_type: "implementation", 320 description: "Implement OAuth 2.0", 321 parent_decision_id: design_decision.id 322 ) 323 324 # 7. Test Lead verifies 325 {:ok, test_decision} = DecisionEngine.initiate_decision( 326 initiator_role: :test_lead, 327 mode: :autonomous, 328 decision_type: "test_results", 329 description: "All auth tests pass", 330 parent_decision_id: impl_decision.id 331 ) 332 333 # 8. Verify complete workflow 334 :timer.sleep(1000) 335 assert Repo.get!(Decision, decision.id).status == "approved" 336 assert Repo.get!(Decision, design_decision.id).status == "approved" 337 assert Repo.get!(Decision, impl_decision.id).status == "approved" 338 assert Repo.get!(Decision, test_decision.id).status == "approved" 339 end 340 end 341 ``` 342 343 ## Test Fixtures 344 345 ### Fixtures Pattern 346 347 ```elixir 348 # test/fixtures/test_helpers.ex 349 defmodule ECHO.TestHelpers do 350 alias EchoShared.Repo 351 alias EchoShared.Schemas.{Decision, Message, Memory} 352 353 def create_decision(attrs \\ %{}) do 354 defaults = %{ 355 initiator_role: :ceo, 356 mode: :autonomous, 357 decision_type: "test_decision", 358 description: "Test decision", 359 status: "pending", 360 context: %{} 361 } 362 363 attrs = Map.merge(defaults, attrs) 364 %Decision{} 365 |> Decision.changeset(attrs) 366 |> Repo.insert!() 367 end 368 369 def create_message(attrs \\ %{}) do 370 defaults = %{ 371 from_role: :ceo, 372 to_role: :cto, 373 type: :info, 374 subject: "Test message", 375 content: %{}, 376 read: false 377 } 378 379 attrs = Map.merge(defaults, attrs) 380 %Message{} 381 |> Message.changeset(attrs) 382 |> Repo.insert!() 383 end 384 385 def create_memory(attrs \\ %{}) do 386 defaults = %{ 387 role: :ceo, 388 key: "test_key", 389 value: %{"data" => "test"}, 390 tags: ["test"] 391 } 392 393 attrs = Map.merge(defaults, attrs) 394 %Memory{} 395 |> Memory.changeset(attrs) 396 |> Repo.insert!() 397 end 398 399 def clean_database() do 400 Repo.delete_all(Decision) 401 Repo.delete_all(Message) 402 Repo.delete_all(Memory) 403 end 404 end 405 ``` 406 407 ### Using Fixtures in Tests 408 409 ```elixir 410 defmodule ECHO.Integration.SomeTest do 411 use ExUnit.Case 412 import ECHO.TestHelpers 413 414 setup do 415 clean_database() 416 :ok 417 end 418 419 test "using fixtures" do 420 decision = create_decision(%{ 421 initiator_role: :ceo, 422 decision_type: "budget_approval" 423 }) 424 425 message = create_message(%{ 426 from_role: :ceo, 427 to_role: :cto, 428 subject: "Review decision #{decision.id}" 429 }) 430 431 assert decision.initiator_role == :ceo 432 assert message.subject =~ "Review decision" 433 end 434 end 435 ``` 436 437 ## Running Tests 438 439 ### All Tests 440 ```bash 441 # From project root 442 mix test 443 444 # With coverage 445 mix test --cover 446 447 # Parallel execution 448 mix test --max-cases 4 449 ``` 450 451 ### Specific Tests 452 ```bash 453 # Integration tests only 454 mix test test/integration/ 455 456 # E2E tests only 457 mix test test/e2e/ 458 459 # Specific test file 460 mix test test/integration/autonomous_mode_test.exs 461 462 # Specific test by line number 463 mix test test/integration/autonomous_mode_test.exs:15 464 ``` 465 466 ### Tagged Tests 467 ```bash 468 # Run only integration tests 469 mix test --only integration 470 471 # Run only E2E tests 472 mix test --only e2e 473 474 # Exclude slow tests 475 mix test --exclude slow 476 477 # Run only fast tests 478 mix test --only fast 479 ``` 480 481 ## Test Configuration 482 483 ### test/test_helper.exs 484 ```elixir 485 ExUnit.start() 486 487 # Set test environment 488 Application.put_env(:echo_shared, :env, :test) 489 490 # Start test database 491 Application.ensure_all_started(:postgrex) 492 Application.ensure_all_started(:ecto) 493 494 # Configure test database 495 Application.put_env(:echo_shared, EchoShared.Repo, 496 database: "echo_org_test", 497 pool: Ecto.Adapters.SQL.Sandbox 498 ) 499 500 # Start Repo 501 {:ok, _} = EchoShared.Repo.start_link() 502 503 # Configure Ecto sandbox 504 Ecto.Adapters.SQL.Sandbox.mode(EchoShared.Repo, :manual) 505 ``` 506 507 ### Test Database Setup 508 ```bash 509 # Create test database 510 PGPASSWORD=postgres psql -h localhost -p 5433 -U echo_org -c "CREATE DATABASE echo_org_test" 511 512 # Run migrations 513 cd apps/echo_shared 514 MIX_ENV=test mix ecto.migrate 515 516 # Reset test database 517 MIX_ENV=test mix ecto.reset 518 ``` 519 520 ## Performance Testing 521 522 ### Load Test Pattern 523 ```elixir 524 defmodule ECHO.Performance.LoadTest do 525 use ExUnit.Case 526 527 @moduletag :performance 528 @moduletag timeout: :infinity 529 530 test "system handles 1000 concurrent decisions" do 531 # Start timer 532 start_time = System.monotonic_time(:millisecond) 533 534 # Create 1000 decisions in parallel 535 tasks = for i <- 1..1000 do 536 Task.async(fn -> 537 DecisionEngine.initiate_decision( 538 initiator_role: :ceo, 539 mode: :autonomous, 540 decision_type: "load_test", 541 description: "Load test decision #{i}" 542 ) 543 end) 544 end 545 546 # Wait for all to complete 547 results = Task.await_many(tasks, 30_000) 548 549 # Calculate metrics 550 end_time = System.monotonic_time(:millisecond) 551 duration = end_time - start_time 552 throughput = 1000 / (duration / 1000) 553 554 # Verify all succeeded 555 assert length(results) == 1000 556 assert Enum.all?(results, fn {:ok, _} -> true; _ -> false end) 557 558 # Log metrics 559 IO.puts("Duration: #{duration}ms") 560 IO.puts("Throughput: #{throughput} decisions/sec") 561 assert throughput > 50 # Minimum 50 decisions/sec 562 end 563 end 564 ``` 565 566 ## Troubleshooting Tests 567 568 ### Database Issues 569 ```bash 570 # Test database not found 571 PGPASSWORD=postgres psql -h localhost -p 5433 -U echo_org -c "CREATE DATABASE echo_org_test" 572 573 # Stale connections 574 MIX_ENV=test mix ecto.reset 575 576 # Permission issues 577 GRANT ALL PRIVILEGES ON DATABASE echo_org_test TO echo_org; 578 ``` 579 580 ### Timeout Issues 581 ```bash 582 # Increase test timeout 583 mix test --timeout 60000 584 585 # Or in specific test 586 @moduletag timeout: 60_000 587 ``` 588 589 ### Flaky Tests 590 ```elixir 591 # Add retries for flaky tests 592 test "flaky test" do 593 Enum.find_value(1..3, fn attempt -> 594 try do 595 # Test code here 596 :ok 597 rescue 598 _ -> if attempt == 3, do: reraise, else: nil 599 end 600 end) 601 end 602 ``` 603 604 ## Best Practices 605 606 1. **Clean State** - Always clean database in `setup` 607 2. **Use Fixtures** - DRY principle with test helpers 608 3. **Tag Tests** - Use `@moduletag` for organization 609 4. **Async When Possible** - Use `use ExUnit.Case, async: true` for unit tests 610 5. **Meaningful Assertions** - Use descriptive assertion messages 611 6. **Test Edge Cases** - Error conditions, boundaries, edge cases 612 7. **Mock External Services** - Don't depend on Ollama/external APIs in tests 613 8. **Fast Tests** - Unit tests should be <1s, integration <5s, E2E <30s 614 9. **Document Test Intent** - Clear test names and comments 615 616 ## Related Documentation 617 618 - **Parent:** [../CLAUDE.md](../CLAUDE.md) - Project overview 619 - **Shared Library:** [../apps/echo_shared/claude.md](../apps/echo_shared/claude.md) - Testing shared components 620 - **Agents:** [../apps/claude.md](../apps/claude.md) - Agent testing patterns 621 622 --- 623 624 **Remember:** Tests are documentation. Write them clearly and keep them fast.