1  ---
  2  title: 'Dashboard'
  3  category: 'integrations'
  4  last_verified: '2026-02-15'
  5  tags: ['dashboard', 'cron', 'scheduling', 'testing', 'database', 'api', 'ai', 'llm']
  6  status: 'current'
  7  ---
  8  
  9  # 333 Method Analytics Dashboard
 10  
 11  Real-time analytics dashboard for monitoring pipeline health, outreach effectiveness, and system performance.
 12  
 13  ## Quick Start
 14  
 15  ```bash
 16  # Start dashboard (dependencies auto-installed in nix-shell)
 17  npm run dashboard
 18  
 19  # Access at http://localhost:8501
 20  
 21  # Manual install (only if not using nix-shell or need to reinstall)
 22  npm run dashboard:install
 23  ```
 24  
 25  ## Overview
 26  
 27  The dashboard provides comprehensive visibility into the 333 Method automation pipeline with 5 specialized pages:
 28  
 29  1. **Pipeline Health** - Error analysis and bottleneck detection
 30  2. **Outreach Effectiveness** - Response rates, sales tracking, and LLM costs
 31  3. **Conversations** - Reply sentiment analysis and action items
 32  4. **Compliance** - Rate limit tracking for all platforms
 33  5. **System Health** - Cron jobs, code coverage, IP burnout, E2E tests
 34  
 35  ## Pages
 36  
 37  ### Homepage
 38  
 39  Executive summary dashboard with key metrics:
 40  
 41  - **Key Metrics**: Total sites, outreach sent, response rate, sales
 42  - **Pipeline Funnel**: Visualize sites at each stage
 43  - **Response Rates**: By channel (SMS, Email, Form, X, LinkedIn)
 44  - **Top Errors**: Most common pipeline errors
 45  - **Sales Distribution**: By channel
 46  
 47  **Refresh Options**:
 48  
 49  - Auto-refresh: 30s, 60s, 5min, 10min, or manual
 50  - Manual refresh button clears cache
 51  - Last updated timestamp in sidebar
 52  
 53  ### Page 1: Pipeline Health & Error Analysis
 54  
 55  Monitor pipeline progress and identify bottlenecks.
 56  
 57  **Key Metrics**:
 58  
 59  - Total sites in pipeline
 60  - Active errors count
 61  - Stuck sites (>1 day at same status)
 62  
 63  **Visualizations**:
 64  
 65  - Pipeline status funnel
 66  - Most common errors (bar chart)
 67  - Sites stuck by error type (table with days stuck)
 68  - Daily throughput (last 30 days)
 69  
 70  **Use Cases**:
 71  
 72  - Identify which error types affect most sites
 73  - Find sites stuck in processing
 74  - Monitor daily processing capacity
 75  - Track error trends over time
 76  
 77  ### Page 2: Outreach Effectiveness (PRIORITY)
 78  
 79  Track outreach performance, ROI, and value for money.
 80  
 81  **Key Metrics**:
 82  
 83  - Total outreach sent
 84  - Overall response rate (%)
 85  - Total sales count
 86  - Total revenue ($)
 87  
 88  **Visualizations**:
 89  
 90  - Response rate by channel (bar chart)
 91  - Response distribution (pie chart)
 92  - Sales tracking dashboard:
 93    - Sales count, total revenue, avg sale value
 94    - Conversion rate (sales / outreach sent)
 95    - Revenue by channel
 96  - Delivery funnel by channel (stacked bars)
 97  - LLM token & cost analysis (coming soon)
 98  
 99  **Use Cases**:
100  
101  - Compare channel effectiveness (which channels get best response?)
102  - Track sales and revenue
103  - Calculate cost per response (when LLM costs added)
104  - Identify best-performing channels for future focus
105  
106  **TODO Enhancements**:
107  
108  - LLM token tracking by stage (scoring, rescoring, proposals)
109  - Cost breakdown by provider
110  - Value for money: response rate vs. cost per response
111  
112  ### Page 3: Conversations
113  
114  Monitor inbound replies and sentiment analysis.
115  
116  **Key Metrics**:
117  
118  - Total conversations
119  - Inbound replies count
120  - Unread messages
121  
122  **Visualizations**:
123  
124  - Sentiment distribution (pie chart)
125  - Sentiment breakdown table
126  - Action items:
127    - Unread positive responses (respond quickly!)
128    - Unread objections (address promptly)
129    - Unread negative (review messaging)
130  - Recent conversation threads (expandable cards)
131  
132  **Filters**:
133  
134  - Channel (SMS, Email, Form, X, LinkedIn)
135  - Sentiment (positive, neutral, negative, objection)
136  - Unread only checkbox
137  
138  **Use Cases**:
139  
140  - Prioritize responses (positive first)
141  - Identify common objections
142  - Track sentiment trends
143  - Find conversations needing attention
144  
145  **TODO Enhancement**:
146  
147  - Analyze negative sentiment to identify common complaints
148  - Use insights to refine proposal templates
149  
150  ### Page 4: Compliance & Rate Limits
151  
152  Monitor platform health and avoid hitting rate limits.
153  
154  **Key Metrics**:
155  
156  - Email opt-outs
157  - SMS opt-outs
158  - Total opt-outs
159  
160  **Platform Health**:
161  
162  **Resend.com (Email)**:
163  
164  - Bounce rate monitoring (Alert >5%, Critical >10%)
165  - Complaint rate tracking (Alert >0.1%, Critical >0.5%)
166  - Recommendations when thresholds exceeded
167  - Delivery rate tracking
168  
169  **Twilio (SMS)**:
170  
171  - Message delivery rate
172  - Failed rate
173  - Recommendations for carrier filtering issues
174  
175  **Other Platforms**:
176  
177  - ZenRows: API call tracking (coming soon)
178  - OpenRouter: Token usage vs. budget (coming soon)
179  - Playwright: Concurrent instances (coming soon)
180  
181  **Compliance Checklist**:
182  
183  - CAN-SPAM requirements (Email)
184  - TCPA requirements (SMS)
185  
186  **Use Cases**:
187  
188  - Avoid sender reputation damage
189  - Prevent platform bans/throttling
190  - Ensure legal compliance
191  - Identify deliverability issues early
192  
193  ### Page 5: System Health
194  
195  Monitor cron jobs, code quality, and infrastructure health.
196  
197  **Cron Job Status**:
198  
199  - Job name, last run, description
200  - Visual status indicators
201  - Quick identification of failed/stale jobs
202  
203  **Code Coverage**:
204  
205  - Overall coverage percentage (gauge chart)
206  - Lines, statements, functions, branches breakdown
207  - Alert if <70% (target: 80%+)
208  
209  **E2E Integration Tests**:
210  
211  - Last test run status (pass/fail)
212  - Total tests, passed, failed counts
213  - Test duration
214  - Categories: Email, SMS, Inbound, Full Pipeline
215  
216  **IP Burnout Tracking** (NEW):
217  
218  - HTTP error code history (last 30 days)
219  - Focus on: 403 (Forbidden), 429 (Rate Limited), 503 (Service Unavailable)
220  - Error codes over time (line chart)
221  - Alert recommendations:
222    - High 403/429 rates → Consider residential proxies
223    - Monitor VPN effectiveness
224  
225  **API Rate Limiting**:
226  
227  - Recent rate limit errors by API
228  - Stage where rate limiting occurred
229  - Error frequency and last occurrence
230  
231  **Database Health**:
232  
233  - Database size (MB)
234  - Integrity check status
235  - Table row counts (sites, outreaches, conversations, keywords)
236  
237  **Use Cases**:
238  
239  - Monitor cron job execution
240  - Ensure code quality standards maintained
241  - Detect IP blocking early
242  - Track database growth
243  - Verify system components healthy
244  
245  ## Configuration
246  
247  Edit `.env` to configure dashboard settings:
248  
249  ```bash
250  # Dashboard port (default: 8501)
251  DASHBOARD_PORT=8501
252  
253  # Auto-refresh interval in seconds (0 = manual)
254  DASHBOARD_REFRESH_INTERVAL=60
255  
256  # Query cache TTL in seconds
257  DASHBOARD_CACHE_TTL=60
258  
259  # Pagination size for tables
260  DASHBOARD_PAGE_SIZE=50
261  ```
262  
263  ## Development
264  
265  ### File Structure
266  
267  ```
268  dashboard/
269  ├── app.py                      # Homepage
270  ├── config.py                   # Configuration
271  ├── requirements.txt            # Python dependencies
272  ├── pages/
273  │   ├── 1_Pipeline_Health.py   # Error analysis
274  │   ├── 2_Outreach.py          # Response rates, sales
275  │   ├── 3_Conversations.py     # Reply sentiment
276  │   ├── 4_Compliance.py        # Rate limits
277  │   └── 5_System_Health.py     # Cron, coverage, IP burnout
278  ├── components/
279  │   ├── metrics.py             # Metric cards
280  │   ├── charts.py              # Chart components
281  │   └── filters.py             # Filter widgets
282  ├── utils/
283  │   ├── database.py            # Database queries
284  │   └── file_readers.py        # Coverage/quality reports
285  └── tests/
286      ├── test_database.py       # Database tests
287      └── test_file_readers.py   # File reader tests
288  ```
289  
290  ### Adding a New Page
291  
292  1. Create `dashboard/pages/N_PageName.py`
293  2. Use components from `dashboard/components/`
294  3. Query data using `dashboard/utils/database.py`
295  4. Follow existing page structure for consistency
296  
297  ### Adding a New Query
298  
299  1. Add function to `dashboard/utils/database.py`
300  2. Use `@st.cache_data(ttl=config.CACHE_TTL)` decorator
301  3. Return pandas DataFrame
302  4. Write test in `dashboard/tests/test_database.py`
303  
304  ### Running Tests
305  
306  ```bash
307  npm run dashboard:test
308  
309  # Or directly with pytest
310  cd dashboard
311  pytest tests/ -v
312  ```
313  
314  ## Troubleshooting
315  
316  ### Dashboard won't start
317  
318  **Check Python environment:**
319  
320  ```bash
321  which python3
322  python3 --version  # Should be 3.8+
323  ```
324  
325  **Install dependencies:**
326  
327  ```bash
328  npm run dashboard:install
329  ```
330  
331  **Verify database exists:**
332  
333  ```bash
334  ls -l db/sites.db
335  ```
336  
337  ### Data not updating
338  
339  1. Click "Refresh Now" button in sidebar
340  2. Check auto-refresh interval setting
341  3. Verify database is being written to by pipeline
342  
343  ### Permission errors
344  
345  **Ensure read access to database:**
346  
347  ```bash
348  chmod 644 db/sites.db
349  ```
350  
351  **Run as correct user** (not root):
352  
353  ```bash
354  npm run dashboard
355  ```
356  
357  ### Module import errors
358  
359  **Ensure Python path includes project root:**
360  
361  ```bash
362  cd dashboard
363  python3 -c "import sys; print(sys.path)"
364  ```
365  
366  If imports fail, the path structure in test files handles this automatically.
367  
368  ## Performance
369  
370  ### Query Optimization
371  
372  - All queries cached for 60s (configurable via `DASHBOARD_CACHE_TTL`)
373  - Read-only database connection (no writes possible)
374  - Pagination for large tables (default: 50 rows)
375  
376  ### Cache Management
377  
378  - Manual refresh clears all cached data
379  - Auto-refresh invalidates cache at interval
380  - File readers cached for 5 minutes (coverage reports don't change often)
381  
382  ### Concurrent Access
383  
384  - Multiple users can view dashboard simultaneously
385  - SQLite handles concurrent reads well (WAL mode)
386  - Dashboard never writes to database (read-only enforced)
387  
388  ## Future Enhancements
389  
390  See `../TODO.md` for planned features:
391  
392  - **Sales Tracking**: Revenue attribution by keyword, time-to-conversion, sales funnel visualization
393  - **Negative Sentiment Analysis**: Identify common complaints, categorize objections, refine messaging
394  - **LLM Token Tracking**: Track usage by stage, cost breakdown, value for money analysis
395  - **Platform Rate Limits**: ZenRows API calls, OpenRouter budget tracking
396  - **Advanced Filters**: Date range filtering across all pages, multi-select filters
397  
398  ## Notes
399  
400  - Dashboard runs independently from the pipeline
401  - No impact on pipeline performance
402  - Read-only access ensures data integrity
403  - Python environment managed separately from Node.js
404  - All sensitive data masked in display
405  - No API keys or credentials shown in dashboard