inbound-processing.md
1 --- 2 title: 'Inbound Processing' 3 category: 'integrations' 4 last_verified: '2026-02-15' 5 related_files: 6 - 'src/inbound/email.js' 7 - 'src/inbound/processor.js' 8 - 'src/utils/sync-email-events.js' 9 - 'tests/inbound-email.test.js' 10 - 'tests/inbound-processor.test.js' 11 tags: ['inbound', 'processing', 'cron', 'scheduling', 'testing', 'database', 'api', 'ai'] 12 status: 'current' 13 --- 14 15 # Enhanced Inbound Handling Implementation 16 17 **Date:** 2026-01-27 18 **Status:** ✅ Complete 19 20 ## Overview 21 22 Implemented comprehensive inbound handling for both email and SMS channels, with unified processing, sentiment detection, and operator reply management. 23 24 ## What Was Implemented 25 26 ### 1. Cloudflare Worker Updates 27 28 - **File:** `cloudflare-worker/resend-webhook-worker.js` 29 - **Changes:** Added support for `email.received` events 30 - The worker now captures all Resend webhook events including inbound email replies 31 32 ### 2. Inbound Email Handler 33 34 - **File:** `src/inbound/email.js` 35 - **Features:** 36 - Poll Cloudflare R2 for `email.received` events 37 - Fetch full email content via Resend Received Emails API 38 - Parse email body to remove quoted text using common delimiters: 39 - `On ... wrote:` 40 - `-----Original Message-----` 41 - `> ` quoted lines 42 - `From:` headers 43 - Detect sentiment using keyword matching: 44 - **Positive:** interested, yes, sounds good, please call, let's talk, schedule, when can, available, great, perfect, thank you 45 - **Objection:** not interested, no thanks, remove, unsubscribe, stop, don't contact, already have, too expensive, not now, busy 46 - **Neutral:** default for all other messages 47 - Map sender email to outreach (case-insensitive matching) 48 - Store in conversations table with direction='inbound' 49 - Process pending operator replies from conversations table 50 - Last poll time tracking via config table 51 52 **CLI Commands:** 53 54 ```bash 55 node src/inbound/email.js poll 56 node src/inbound/email.js process-replies 57 npm run inbound:email 58 ``` 59 60 ### 3. Unified Inbound Processor 61 62 - **File:** `src/inbound/processor.js` 63 - **Features:** 64 - Poll all inbound channels (SMS + Email) in one command 65 - Process all pending operator replies across channels 66 - Get unread conversations with limit support 67 - Get conversation thread for specific outreach (all messages chronologically) 68 - Mark individual conversations as read 69 - Mark entire threads as read 70 - Get inbound statistics (unread count, by channel, by sentiment) 71 72 **CLI Commands:** 73 74 ```bash 75 node src/inbound/processor.js poll # Poll all channels 76 node src/inbound/processor.js process-replies # Process all pending replies 77 node src/inbound/processor.js inbox [limit] # View unread conversations 78 node src/inbound/processor.js thread <outreach_id> # View conversation thread 79 node src/inbound/processor.js stats # View statistics 80 81 # npm scripts 82 npm run inbound:poll # Poll all channels 83 npm run inbound:process-replies # Process all pending replies 84 npm run inbound:inbox # View unread conversations 85 npm run inbound:stats # View statistics 86 ``` 87 88 ### 4. Email Events Sync Updates 89 90 - **File:** `src/utils/sync-email-events.js` 91 - **Changes:** Added handler for `email.received` events 92 - The sync script now acknowledges inbound emails and directs users to use the dedicated email poller 93 94 ### 5. Comprehensive Test Suite 95 96 - **Files:** 97 - `tests/inbound-email.test.js` - 26 tests covering email functions 98 - `tests/inbound-processor.test.js` - 18 tests covering processor functions 99 - **Coverage:** 100 - Email address matching (case-insensitive) 101 - Email body parsing (quoted text removal) 102 - Sentiment detection (positive, objection, neutral) 103 - Conversation storage 104 - Thread management 105 - Statistics generation 106 - **Test Results:** ✅ 44/44 tests passing 107 108 ### 6. Package.json Updates 109 110 - **New npm scripts:** 111 - `inbound:poll` - Poll all inbound channels 112 - `inbound:process-replies` - Process pending operator replies 113 - `inbound:inbox` - View unread conversations 114 - `inbound:thread` - View conversation thread 115 - `inbound:stats` - View inbound statistics 116 - `inbound:email` - Poll email specifically 117 - `inbound:sms` - Poll SMS specifically 118 119 ### 7. Documentation Updates 120 121 - **File:** `../TODO.md` 122 - Marked all Enhanced Inbound Handling tasks as complete 123 - Updated status and added implementation details 124 125 ## Architecture 126 127 ### Data Flow 128 129 ``` 130 ┌─────────────────────────────────────────────────────────┐ 131 │ Inbound Messages │ 132 ├─────────────────────┬───────────────────────────────────┤ 133 │ Email │ SMS │ 134 │ (via Resend) │ (via Twilio) │ 135 └──────────┬──────────┴───────────────┬───────────────────┘ 136 │ │ 137 ▼ ▼ 138 ┌──────────────┐ ┌──────────────┐ 139 │ Cloudflare │ │ Twilio │ 140 │ Worker │ │ API │ 141 │ (R2 Storage)│ │ │ 142 └──────┬───────┘ └──────┬───────┘ 143 │ │ 144 │ Poll every 5 min │ Poll every 5 min 145 ▼ ▼ 146 ┌──────────────────────────────────────┐ 147 │ src/inbound/processor.js │ 148 │ (Unified Inbound Processor) │ 149 └──────┬───────────────────────────────┘ 150 │ 151 │ 1. Find matching outreach 152 │ 2. Parse & detect sentiment 153 │ 3. Store in conversations 154 │ 155 ▼ 156 ┌──────────────────────────────────────┐ 157 │ conversations table │ 158 │ (SQLite - Threaded by outreach_id) │ 159 └──────┬───────────────────────────────┘ 160 │ 161 │ Operator reviews & replies 162 │ 163 ▼ 164 ┌──────────────────────────────────────┐ 165 │ process-replies command │ 166 │ (Send outbound via SMS/Email) │ 167 └──────────────────────────────────────┘ 168 ``` 169 170 ### Database Schema 171 172 The implementation uses the existing `conversations` table: 173 174 ```sql 175 CREATE TABLE conversations ( 176 id INTEGER PRIMARY KEY, 177 outreach_id INTEGER NOT NULL REFERENCES outreaches(id), 178 direction TEXT CHECK(direction IN ('inbound', 'outbound')), 179 channel TEXT CHECK(channel IN ('sms', 'email', 'form')), 180 sender_identifier TEXT, 181 message_body TEXT NOT NULL, 182 subject_line TEXT, 183 raw_payload TEXT, 184 sentiment TEXT CHECK(sentiment IN ('positive', 'neutral', 'negative', 'objection')), 185 received_at DATETIME DEFAULT CURRENT_TIMESTAMP, 186 read_at DATETIME, 187 replied_at DATETIME 188 ); 189 ``` 190 191 ## Usage Examples 192 193 ### Poll for new inbound messages 194 195 ```bash 196 # Poll all channels (email + SMS) 197 npm run inbound:poll 198 199 # Poll just email 200 npm run inbound:email 201 202 # Poll just SMS 203 npm run inbound:sms 204 ``` 205 206 ### View inbox 207 208 ```bash 209 # Show all unread conversations 210 npm run inbound:inbox 211 212 # Show 20 most recent unread 213 node src/inbound/processor.js inbox 20 214 ``` 215 216 ### View conversation thread 217 218 ```bash 219 # View full conversation for outreach #123 220 node src/inbound/processor.js thread 123 221 ``` 222 223 ### View statistics 224 225 ```bash 226 npm run inbound:stats 227 ``` 228 229 ### Process operator replies 230 231 ```bash 232 # Send all pending operator replies 233 npm run inbound:process-replies 234 ``` 235 236 ## Cron Setup 237 238 Add to crontab for automated polling (every 5 minutes): 239 240 ```bash 241 # Poll inbound messages 242 */5 * * * * cd /path/to/project && npm run inbound:poll >> /var/log/inbound-poll.log 2>&1 243 244 # Process pending replies 245 */5 * * * * cd /path/to/project && npm run inbound:process-replies >> /var/log/inbound-replies.log 2>&1 246 247 # Sync email events (tracking) 248 */5 * * * * cd /path/to/project && npm run sync-email-events >> /var/log/email-sync.log 2>&1 249 ``` 250 251 ## Key Features 252 253 ### Email Parsing 254 255 - Removes quoted text from replies 256 - Preserves original message in `raw_payload` 257 - Stores clean message body in `message_body` 258 259 ### Sentiment Detection 260 261 - Simple keyword-based matching 262 - Prioritizes negative keywords (more specific) 263 - Returns: positive, objection, or neutral 264 - Used for prioritizing operator review 265 266 ### Threading 267 268 - All messages grouped by `outreach_id` 269 - Supports both inbound and outbound messages 270 - Chronological ordering by `received_at` 271 272 ### Operator Workflow 273 274 1. Poll for new messages: `npm run inbound:poll` 275 2. View inbox: `npm run inbound:inbox` 276 3. View specific thread: `node src/inbound/processor.js thread <id>` 277 4. Operator inserts reply into conversations table (direction='outbound') 278 5. Process replies: `npm run inbound:process-replies` 279 280 ## Testing 281 282 Run tests: 283 284 ```bash 285 # All tests 286 npm test 287 288 # Just inbound tests 289 node --test tests/inbound-email.test.js tests/inbound-processor.test.js 290 ``` 291 292 ## Next Steps (Optional Enhancements) 293 294 1. **Streamlit Dashboard Integration** 295 - Display unread conversations in UI 296 - Allow operators to reply directly from dashboard 297 - Show sentiment statistics 298 299 2. **Smart Reply Templates** 300 - Suggest replies based on sentiment 301 - Quick responses for common objections 302 303 3. **Auto-classification** 304 - ML-based sentiment detection 305 - Intent classification (pricing, scheduling, objection) 306 307 4. **Escalation Rules** 308 - Auto-notify on positive sentiment 309 - Flag high-value leads 310 311 ## Files Created/Modified 312 313 ### Created 314 315 - `src/inbound/email.js` (475 lines) 316 - `src/inbound/processor.js` (443 lines) 317 - `tests/inbound-email.test.js` (274 lines) 318 - `tests/inbound-processor.test.js` (245 lines) 319 - `inbound-processing.md` (this file) 320 321 ### Modified 322 323 - `cloudflare-worker/resend-webhook-worker.js` (documentation) 324 - `src/utils/sync-email-events.js` (added email.received handler) 325 - `package.json` (added 7 new npm scripts) 326 - `../TODO.md` (marked tasks complete) 327 328 **Total:** 5 new files, 4 modified files, ~1,437 lines of new code