1  # Provider Selection Optimization
  2  
  3  ## Problem Statement
  4  
  5  Based on the logs analysis, the current provider selection mechanism in DaokoTube has several inefficiencies:
  6  
  7  1. Multiple providers are being tested repeatedly, even when they consistently fail
  8  2. Provider performance data isn't being effectively utilized across sessions
  9  3. CID-specific provider performance isn't tracked or prioritized
 10  4. The system doesn't adapt quickly enough to changing network conditions
 11  
 12  These issues lead to slower video loading times, more failed requests, and a poorer user experience.
 13  
 14  ## Proposed Solution
 15  
 16  Implement a more sophisticated provider caching and selection strategy that:
 17  
 18  1. Tracks global provider performance across all videos
 19  2. Maintains CID-specific performance data
 20  3. Prioritizes providers that have worked well for specific CIDs
 21  4. Adapts quickly to changing network conditions
 22  5. Persists performance data across sessions
 23  
 24  ## Implementation Plan
 25  
 26  ### 1. Create a Provider Performance Manager
 27  
 28  ```javascript
 29  // app/services/providers/provider-performance-manager.js
 30  
 31  import { LRUCache } from '../../utils/caching/lru-cache.js';
 32  import { createLogger } from '../../utils/debug/logger.js';
 33  import { LocalStorage } from '../../utils/storage/local-storage.js';
 34  
 35  const logger = createLogger('ProviderPerformanceManager');
 36  
 37  // Constants
 38  const GLOBAL_PERFORMANCE_STORAGE_KEY = 'daokotube_provider_performance';
 39  const PROVIDER_CACHE_DURATION = 30 * 60 * 1000; // 30 minutes
 40  const MAX_CID_ENTRIES = 100;
 41  const PERFORMANCE_DECAY_FACTOR = 0.9; // Weight for old performance data
 42  
 43  export class ProviderPerformanceManager {
 44    constructor() {
 45      this.globalPerformance = new Map();
 46      this.cidSpecificPerformance = new LRUCache(MAX_CID_ENTRIES);
 47      this._loadPerformanceData();
 48    }
 49  
 50    /**
 51     * Update provider performance data
 52     * @param {string} provider - Provider identifier
 53     * @param {string} cid - Content identifier
 54     * @param {number} responseTime - Response time in milliseconds
 55     * @param {boolean} success - Whether the request was successful
 56     */
 57    updateProviderPerformance(provider, cid, responseTime, success) {
 58      // Update global performance
 59      this._updateGlobalPerformance(provider, responseTime, success);
 60      
 61      // Update CID-specific performance
 62      if (cid) {
 63        this._updateCidPerformance(provider, cid, responseTime, success);
 64      }
 65      
 66      // Save performance data periodically
 67      this._savePerformanceData();
 68    }
 69  
 70    /**
 71     * Get optimal providers for a specific CID
 72     * @param {string} cid - Content identifier
 73     * @param {string[]} allProviders - All available providers
 74     * @returns {string[]} - Sorted list of providers
 75     */
 76    getOptimalProviders(cid, allProviders) {
 77      return [...allProviders].sort((a, b) => {
 78        // Get performance data
 79        const aGlobal = this.globalPerformance.get(a) || { successRate: 0, avgResponseTime: Infinity };
 80        const bGlobal = this.globalPerformance.get(b) || { successRate: 0, avgResponseTime: Infinity };
 81        
 82        // Get CID-specific data
 83        const cidPerformance = this.cidSpecificPerformance.get(cid) || new Map();
 84        const aCid = cidPerformance.get(a) || { successCount: 0, failCount: 0 };
 85        const bCid = cidPerformance.get(b) || { successCount: 0, failCount: 0 };
 86        
 87        // Calculate CID-specific success rates
 88        const aCidSuccessRate = aCid.successCount / (aCid.successCount + aCid.failCount || 1);
 89        const bCidSuccessRate = bCid.successCount / (bCid.successCount + bCid.failCount || 1);
 90        
 91        // If we have CID-specific data, prioritize it
 92        if (aCid.successCount > 0 || bCid.successCount > 0) {
 93          // If one has successes and the other doesn't, prioritize the one with successes
 94          if (aCid.successCount > 0 && bCid.successCount === 0) return -1;
 95          if (aCid.successCount === 0 && bCid.successCount > 0) return 1;
 96          
 97          // If both have successes, compare success rates
 98          if (Math.abs(aCidSuccessRate - bCidSuccessRate) > 0.1) {
 99            return bCidSuccessRate - aCidSuccessRate;
100          }
101        }
102        
103        // If no CID-specific data or they're similar, use global data
104        if (Math.abs(aGlobal.successRate - bGlobal.successRate) > 0.2) {
105          return bGlobal.successRate - aGlobal.successRate;
106        }
107        
108        // If success rates are similar, use response time
109        return aGlobal.avgResponseTime - bGlobal.avgResponseTime;
110      });
111    }
112  
113    /**
114     * Update global performance data for a provider
115     * @private
116     */
117    _updateGlobalPerformance(provider, responseTime, success) {
118      if (!this.globalPerformance.has(provider)) {
119        this.globalPerformance.set(provider, {
120          successCount: 0,
121          failCount: 0,
122          avgResponseTime: 0,
123          successRate: 0,
124          lastUpdated: Date.now()
125        });
126      }
127      
128      const data = this.globalPerformance.get(provider);
129      
130      // Update counts
131      if (success) {
132        data.successCount++;
133        
134        // Update average response time with decay factor
135        if (Number.isFinite(responseTime)) {
136          if (data.successCount === 1) {
137            data.avgResponseTime = responseTime;
138          } else {
139            data.avgResponseTime = (data.avgResponseTime * PERFORMANCE_DECAY_FACTOR) + 
140                                  (responseTime * (1 - PERFORMANCE_DECAY_FACTOR));
141          }
142        }
143      } else {
144        data.failCount++;
145      }
146      
147      // Update success rate
148      data.successRate = data.successCount / (data.successCount + data.failCount);
149      data.lastUpdated = Date.now();
150    }
151  
152    /**
153     * Update CID-specific performance data
154     * @private
155     */
156    _updateCidPerformance(provider, cid, responseTime, success) {
157      if (!this.cidSpecificPerformance.has(cid)) {
158        this.cidSpecificPerformance.set(cid, new Map());
159      }
160      
161      const cidPerformance = this.cidSpecificPerformance.get(cid);
162      
163      if (!cidPerformance.has(provider)) {
164        cidPerformance.set(provider, {
165          successCount: 0,
166          failCount: 0,
167          lastUpdated: Date.now()
168        });
169      }
170      
171      const data = cidPerformance.get(provider);
172      
173      // Update counts
174      if (success) {
175        data.successCount++;
176      } else {
177        data.failCount++;
178      }
179      
180      data.lastUpdated = Date.now();
181    }
182  
183    /**
184     * Load performance data from storage
185     * @private
186     */
187    _loadPerformanceData() {
188      try {
189        const storedData = LocalStorage.getItem(GLOBAL_PERFORMANCE_STORAGE_KEY);
190        
191        if (storedData) {
192          const parsedData = JSON.parse(storedData);
193          
194          // Load global performance
195          if (parsedData.global) {
196            this.globalPerformance = new Map(Object.entries(parsedData.global));
197          }
198          
199          // Load CID-specific performance (limited to MAX_CID_ENTRIES)
200          if (parsedData.cidSpecific) {
201            Object.entries(parsedData.cidSpecific).slice(0, MAX_CID_ENTRIES).forEach(([cid, providers]) => {
202              this.cidSpecificPerformance.set(cid, new Map(Object.entries(providers)));
203            });
204          }
205          
206          logger.info(`Loaded performance data for ${this.globalPerformance.size} providers and ${this.cidSpecificPerformance.size} CIDs`);
207        }
208      } catch (error) {
209        logger.warn('Failed to load provider performance data:', error);
210      }
211    }
212  
213    /**
214     * Save performance data to storage
215     * @private
216     */
217    _savePerformanceData() {
218      try {
219        // Convert Maps to objects for storage
220        const globalObj = {};
221        this.globalPerformance.forEach((value, key) => {
222          globalObj[key] = value;
223        });
224        
225        const cidObj = {};
226        this.cidSpecificPerformance.forEach((providerMap, cid) => {
227          cidObj[cid] = {};
228          providerMap.forEach((value, provider) => {
229            cidObj[cid][provider] = value;
230          });
231        });
232        
233        const dataToStore = {
234          global: globalObj,
235          cidSpecific: cidObj,
236          lastUpdated: Date.now()
237        };
238        
239        LocalStorage.setItem(GLOBAL_PERFORMANCE_STORAGE_KEY, JSON.stringify(dataToStore));
240      } catch (error) {
241        logger.warn('Failed to save provider performance data:', error);
242      }
243    }
244  }
245  
246  export default new ProviderPerformanceManager();
247  ```
248  
249  ### 2. Integrate with VideoRouter
250  
251  Update the VideoRouter to use the ProviderPerformanceManager:
252  
253  ```javascript
254  // app/services/video/router.js
255  
256  import providerPerformanceManager from '../providers/provider-performance-manager.js';
257  
258  // In the getMultipleVideoSources method
259  export function getMultipleVideoSources(cid, maxSources = 0, altcid = '', livestreamInfo = null) {
260    // ... existing code ...
261    
262    // Get optimal providers for this CID
263    const optimalProviders = providerPerformanceManager.getOptimalProviders(cid, eligibleProviders);
264    
265    // Use these providers first
266    const prioritizedProviders = optimalProviders.slice(0, maxSources || optimalProviders.length);
267    
268    // ... create sources using prioritizedProviders ...
269  }
270  
271  // In the updateProviderHealth method
272  export function updateProviderHealth(provider, isHealthy, cid = null) {
273    // ... existing code ...
274    
275    // Update provider performance
276    providerPerformanceManager.updateProviderPerformance(
277      provider,
278      cid,
279      isHealthy ? 1000 : Infinity, // Use a default response time
280      isHealthy
281    );
282  }
283  ```
284  
285  ### 3. Integrate with current-video-pretest.js
286  
287  Update the current-video-pretest.js to use and update the ProviderPerformanceManager:
288  
289  ```javascript
290  // app/utils/video/current-video-pretest.js
291  
292  import providerPerformanceManager from '../../services/providers/provider-performance-manager.js';
293  
294  // In the _testSingleProvider function
295  async function _testSingleProvider(providerKey, cid, altcid, now) {
296    // ... existing code ...
297    
298    try {
299      const response = await fetch(url, { /* ... */ });
300      
301      // ... existing code ...
302      
303      // Update provider performance
304      providerPerformanceManager.updateProviderPerformance(
305        providerKey,
306        cid,
307        responseTime,
308        success
309      );
310      
311      // ... existing code ...
312    } catch (error) {
313      // ... existing code ...
314      
315      // Update provider performance
316      providerPerformanceManager.updateProviderPerformance(
317        providerKey,
318        cid,
319        errorType === 'timeout' ? timeout : responseTime,
320        false
321      );
322      
323      // ... existing code ...
324    }
325  }
326  ```
327  
328  ## Expected Benefits
329  
330  1. **Faster Video Loading**: By prioritizing providers that have worked well in the past, videos will load faster.
331  2. **Reduced Failed Requests**: By avoiding providers that consistently fail, we'll reduce the number of failed requests.
332  3. **Better Adaptation**: The system will adapt to changing network conditions and provider availability.
333  4. **Improved User Experience**: Users will experience fewer stalls and faster video playback.
334  5. **Reduced Network Usage**: By making smarter provider choices, we'll reduce unnecessary network requests.
335  
336  ## Testing Plan
337  
338  1. **Unit Tests**: Create unit tests for the ProviderPerformanceManager class.
339  2. **Integration Tests**: Test the integration with VideoRouter and current-video-pretest.js.
340  3. **Performance Tests**: Measure video loading times before and after the changes.
341  4. **User Testing**: Have users test the application and provide feedback on video loading performance.
342  
343  ## Rollout Plan
344  
345  1. **Development**: Implement the changes in a development environment.
346  2. **Testing**: Test the changes thoroughly.
347  3. **Staging**: Deploy to a staging environment for further testing.
348  4. **Production**: Roll out to production in phases, monitoring performance metrics.
349  5. **Monitoring**: Monitor performance metrics and user feedback.