twilio-integration.md
1 --- 2 title: 'Twilio Integration' 3 category: 'other' 4 last_verified: '2026-02-15' 5 related_files: 6 - 'src/outreach/sms.js' 7 - 'tests/outreach-sms.integration.test.js' 8 - 'tests/inbound-sms.integration.test.js' 9 tags: ['twilio', 'integration', 'testing', 'database', 'api', 'ai', 'llm', 'email'] 10 status: 'current' 11 --- 12 13 # Twilio Integration Testing Guide 14 15 This document explains the comprehensive integration testing setup for Twilio SMS functionality in the 333 Method Automation project. 16 17 ## Overview 18 19 The Twilio integration tests use **four complementary testing approaches** to ensure complete coverage without incurring SMS costs: 20 21 1. **Magic Test Phone Numbers** - Twilio-provided special numbers that validate API calls without sending real SMS 22 2. **Simulated Webhook Payloads** - Mock Twilio webhook structures for testing inbound message handling 23 3. **Database-Isolated Tests** - Separate test database for each test run to ensure test isolation 24 4. **Real API Validation** - Actual Twilio API calls with test credentials to verify integration 25 26 ## Test Files 27 28 ### Outbound SMS Tests (`tests/outreach-sms.integration.test.js`) 29 30 **Coverage: 14 tests** 31 32 Tests the SMS sending functionality including: 33 34 - Sending SMS to valid test numbers 35 - Including tracking URLs in messages 36 - Handling E.164 phone number formatting (US, Australian, international) 37 - Error handling for invalid outreach IDs 38 - Rejecting non-SMS contact methods 39 - Handling pending contact extraction 40 - Truncating long messages (>1600 characters) 41 - Updating delivery status on errors 42 - Bulk SMS sending with rate limiting (1 second between messages) 43 - Respecting limit parameters 44 - Skipping non-approved outreaches 45 - Continuing on individual failures 46 - Twilio client connection validation 47 48 ### Inbound SMS Tests (`tests/inbound-sms.integration.test.js`) 49 50 **Coverage: 18 tests** 51 52 Tests the SMS receiving functionality including: 53 54 - Validating Twilio webhook payload structure 55 - Finding outreaches by phone number (exact match) 56 - Storing inbound SMS in conversations table 57 - Handling opt-out keywords (STOP, UNSUBSCRIBE, etc.) 58 - Handling messages with special characters and emoji 59 - Handling multi-segment long messages 60 - Matching various phone number formats (E.164, without +, last 10 digits) 61 - Matching Australian mobile formats (0412345678, 61412345678, +61412345678) 62 - Matching US 10-digit formats 63 - Returning undefined for non-existent numbers 64 - Storing multiple messages from same sender 65 - Enforcing foreign key constraints 66 - Setting received_at timestamps automatically 67 - Twilio client initialization 68 - Webhook signature validation availability 69 70 ## Test Credentials 71 72 ### Twilio Test Credentials 73 74 Set these environment variables for testing without using production credentials: 75 76 ```bash 77 TWILIO_TEST_ACCOUNT_SID=your_test_account_sid 78 TWILIO_TEST_AUTH_TOKEN=your_test_auth_token 79 ``` 80 81 Get test credentials from: https://www.twilio.com/console/project/settings 82 83 **Important:** Test credentials validate API requests but don't send real SMS or incur charges. 84 85 ### Magic Test Phone Numbers 86 87 These special numbers are provided by Twilio for testing: 88 89 - `+15005550006` - Valid test number (API calls succeed, SMS not actually sent) 90 - `+15005550001` - Invalid test number (API calls fail with validation error) 91 - `+15005550007` - Number that can't receive SMS 92 - `+15005550009` - Number not owned by your account 93 94 **Official Documentation:** https://www.twilio.com/docs/iam/test-credentials 95 96 ## Running the Tests 97 98 ```bash 99 # Run all integration tests (requires Twilio credentials) 100 npm run test:integration 101 102 # Run only Twilio outbound SMS tests 103 node --test tests/outreach-sms.integration.test.js 104 105 # Run only Twilio inbound SMS tests 106 node --test tests/inbound-sms.integration.test.js 107 108 # Run both Twilio tests together 109 node --test tests/outreach-sms.integration.test.js tests/inbound-sms.integration.test.js 110 ``` 111 112 ## Test Architecture 113 114 ### Dynamic Import Pattern 115 116 The tests use **dynamic imports** to ensure environment variables are set before modules initialize: 117 118 ```javascript 119 // Set environment variables BEFORE importing modules 120 process.env.DATABASE_PATH = testDbPath; 121 process.env.TWILIO_ACCOUNT_SID = twilioAccountSid; 122 process.env.TWILIO_AUTH_TOKEN = twilioAuthToken; 123 process.env.TWILIO_PHONE_NUMBER = testPhone; 124 125 // Dynamic import AFTER env vars are set 126 const { sendSMS, sendBulkSMS } = await import('../src/outreach/sms.js'); 127 ``` 128 129 **Why this matters:** The SMS modules create database connections and Twilio clients when they're imported. If environment variables aren't set first, the modules will use incorrect paths or credentials. 130 131 ### Database Isolation 132 133 Each test run creates a fresh SQLite database: 134 135 ```javascript 136 beforeEach(() => { 137 // Clean up any previous test database 138 if (existsSync(testDbPath)) { 139 rmSync(testDbPath); 140 } 141 142 // Create fresh test database with full schema 143 db = new Database(testDbPath); 144 db.exec(/* CREATE TABLES */); 145 db.close(); 146 }); 147 148 afterEach(() => { 149 // Clean up test database after each test 150 if (existsSync(testDbPath)) { 151 rmSync(testDbPath); 152 } 153 }); 154 ``` 155 156 This ensures: 157 158 - Tests don't interfere with each other 159 - No test data pollution 160 - Production database is never touched 161 - Each test starts with known state 162 163 ## What's Tested 164 165 ### Outbound SMS Functionality 166 167 ✅ **API Integration:** 168 169 - Real Twilio API calls with test credentials 170 - Message sending with tracking URLs 171 - Rate limiting enforcement (1 second between messages) 172 - Bulk sending operations 173 174 ✅ **Database Operations:** 175 176 - Outreach lookup and validation 177 - Status updates (pending → sent → failed) 178 - Tracking URL storage 179 - Foreign key constraint enforcement 180 181 ✅ **Phone Number Handling:** 182 183 - E.164 format validation (+15005550006) 184 - Australian mobile formats (0412345678, +61412345678) 185 - US formats (5551234567, +15551234567) 186 - Invalid number rejection 187 188 ✅ **Error Handling:** 189 190 - Invalid outreach IDs 191 - Non-SMS contact methods 192 - Pending contact extraction 193 - Invalid phone numbers 194 - Message truncation (>1600 chars) 195 196 ✅ **Business Logic:** 197 198 - Only approved outreaches sent 199 - Limit parameter respected 200 - Individual failure handling in bulk sends 201 - Rate limiting between messages 202 203 ### Inbound SMS Functionality 204 205 ✅ **Webhook Handling:** 206 207 - Payload structure validation 208 - MessageSid format validation 209 - Header authentication structure 210 211 ✅ **Phone Number Matching:** 212 213 - Exact E.164 format matching 214 - Normalized matching (no spaces, dashes) 215 - Country code conversion (Australian 0 → +61) 216 - Last 10 digits matching 217 - Multiple format support per number 218 219 ✅ **Conversation Storage:** 220 221 - Foreign key constraint enforcement 222 - Automatic timestamp generation 223 - Special character and emoji support 224 - Multi-segment long messages (500+ chars) 225 - Multiple messages from same sender 226 - Opt-out keyword handling (STOP, UNSUBSCRIBE, CANCEL, END, QUIT) 227 228 ✅ **Client Initialization:** 229 230 - Twilio client creation with credentials 231 - Messages resource availability 232 - Phone number format validation 233 234 ## Test Limitations 235 236 ### What Test Credentials DON'T Support 237 238 Some Twilio API operations don't work with test credentials: 239 240 ❌ `account.fetch()` - Account details retrieval 241 ❌ `messages.list()` - Listing sent/received messages 242 ❌ `messages(sid).fetch()` - Retrieving specific messages 243 244 These operations are only available with production credentials. Our tests work around this by: 245 246 - Using simulated webhook payloads instead of polling 247 - Testing phone matching with local database lookups 248 - Validating client initialization without fetching data 249 250 ### Rate Limiting 251 252 Bulk SMS tests include 1-second delays between messages to respect rate limits. This means: 253 254 - Tests take longer to run (~15-20 seconds total) 255 - Tests validate rate limiting is working 256 - No actual rate limit errors occur 257 258 ## Credential Fallback 259 260 Tests use a fallback pattern for credentials: 261 262 ```javascript 263 const twilioAccountSid = process.env.TWILIO_TEST_ACCOUNT_SID || process.env.TWILIO_ACCOUNT_SID; 264 const twilioAuthToken = process.env.TWILIO_TEST_AUTH_TOKEN || process.env.TWILIO_AUTH_TOKEN; 265 ``` 266 267 This means: 268 269 1. Try test credentials first (`TWILIO_TEST_*`) 270 2. Fall back to production credentials if test credentials not available 271 3. Skip tests entirely if no credentials found 272 273 ## Test Results 274 275 Current status: **32 tests, 100% passing** 276 277 ``` 278 ✅ Twilio Inbound SMS Integration Tests (18 tests) 279 ✅ Twilio SMS Integration Tests (14 tests) 280 281 # tests 32 282 # pass 32 283 # fail 0 284 ``` 285 286 ## Troubleshooting 287 288 ### Tests Skip with "No credentials found" 289 290 **Problem:** Tests exit immediately with warning message. 291 292 **Solution:** Set `TWILIO_ACCOUNT_SID` and `TWILIO_AUTH_TOKEN` in `.env`: 293 294 ```bash 295 # Option 1: Use production credentials (test numbers prevent real SMS) 296 TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 297 TWILIO_AUTH_TOKEN=your_auth_token 298 299 # Option 2: Use dedicated test credentials 300 TWILIO_TEST_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 301 TWILIO_TEST_AUTH_TOKEN=your_test_token 302 ``` 303 304 ### Tests Fail with "Resource not accessible with Test Account Credentials" 305 306 **Problem:** Test is trying to use an API operation that doesn't work with test credentials. 307 308 **Solution:** This is expected for operations like `account.fetch()` or `messages.list()`. The current test suite already excludes these operations. If you see this error, the test may need to be updated to use simulated data instead. 309 310 ### Database Errors 311 312 **Problem:** `FOREIGN KEY constraint failed` or `database locked` errors. 313 314 **Solution:** Ensure: 315 316 1. Test database path is set before importing modules 317 2. Database connections are closed in `afterEach()` 318 3. No other processes accessing test database 319 320 ### Module Import Issues 321 322 **Problem:** Tests use wrong database or credentials. 323 324 **Solution:** Use dynamic imports with env vars set first: 325 326 ```javascript 327 // ❌ Wrong - imports before env vars 328 import { sendSMS } from '../src/outreach/sms.js'; 329 process.env.DATABASE_PATH = testDbPath; 330 331 // ✅ Correct - env vars before import 332 process.env.DATABASE_PATH = testDbPath; 333 const { sendSMS } = await import('../src/outreach/sms.js'); 334 ``` 335 336 ## Future Enhancements 337 338 Potential additions to the test suite: 339 340 1. **Webhook Signature Validation** - Test actual Twilio signature verification 341 2. **SMS Polling Integration** - Test `pollInboundSMS()` with mocked API responses 342 3. **Reply Processing** - Test `processPendingReplies()` with mocked sending 343 4. **Error Recovery** - Test retry logic and backoff strategies 344 5. **Concurrent Sending** - Test parallel SMS operations with rate limiting 345 6. **MMS Support** - If/when MMS is added, test media attachments 346 347 ## Related Documentation 348 349 - [Resend Email Integration Testing](TESTING-RESEND.md) 350 - [Best Practices - SMS](BEST-PRACTICES-SMS.md) 351 - [Architecture Overview](ARCHITECTURE.md) 352 - [Twilio Documentation](https://www.twilio.com/docs/sms)