/ docs / docs / documentation / enterprise / secure-user-delegated-access.md
secure-user-delegated-access.md
  1  ---
  2  title: Secure User Delegated Access
  3  sidebar_position: 15
  4  ---
  5  
  6  This guide walks you through configuring Secure User Delegated Access for Agent Mesh Enterprise. You will learn how to enable users to authenticate with remote MCP tools using their own credentials through OAuth2, providing enhanced security and user-specific access control.
  7  
  8  ## Table of Contents
  9  
 10  - [Overview](#overview)
 11  - [Understanding Secure User Delegated Access](#understanding-secure-user-delegated-access)
 12  - [Prerequisites](#prerequisites)
 13  - [Configuration Steps](#configuration-steps)
 14  - [Security Considerations](#security-considerations)
 15  - [Best Practices](#best-practices)
 16  - [Troubleshooting](#troubleshooting)
 17  
 18  ## Overview
 19  
 20  Secure User Delegated Access allows users to interact with remote MCP (Model Context Protocol) tools using their own credentials rather than shared service credentials. When a user needs to access a remote MCP tool configured in an agent, they complete an OAuth2 authentication flow with the MCP provider's authorization server. This provides user-specific credentials that are securely stored and managed by Agent Mesh Enterprise.
 21  
 22  ### Why Use Secure User Delegated Access
 23  
 24  This feature provides several important benefits for enterprise deployments:
 25  
 26  **Enhanced Security**: Each user authenticates with their own credentials, eliminating shared service accounts and reducing the blast radius of compromised credentials. All API calls to remote services are made in the context of the authenticated user.
 27  
 28  **Audit and Compliance**: User-specific credentials create clear audit trails showing exactly which user performed which actions. This is essential for compliance requirements and security investigations.
 29  
 30  **Fine-Grained Access Control**: MCP providers can enforce their own access policies based on the authenticated user's permissions. Users only access resources they are authorized to use within the remote service.
 31  
 32  **Credential Lifecycle Management**: User credentials can be individually revoked at the MCP provider level without affecting other users. This simplifies offboarding and incident response.
 33  
 34  ### Supported MCP Providers
 35  
 36  Agent Mesh Enterprise supports remote MCP servers that implement the OAuth2.1 authentication flow. The following providers have been tested and validated:
 37  
 38  - **Atlassian MCP Server**: Access Jira, Confluence, and other Atlassian services
 39  - **Stripe MCP Server**: Interact with Stripe payment and billing APIs
 40  - **Cloudflare MCP Server**: Manage Cloudflare resources and configurations
 41  - **Canva MCP Server**: Access Canva design and content APIs
 42  
 43  The feature works with both SSE (Server-Sent Events) and HTTP streaming remote MCP server types.
 44  
 45  ## Understanding Secure User Delegated Access
 46  
 47  Before you configure this feature, you need to understand how it works and how credentials flow through the system.
 48  
 49  ### How the OAuth2 Flow Works
 50  
 51  When a user attempts to use an MCP tool that requires OAuth2 authentication, Agent Mesh Enterprise initiates the following flow:
 52  
 53  1. **Authentication Challenge**: The user's request triggers an authentication check. If no valid credentials exist for this user and MCP tool combination, the system prompts the user to authenticate.
 54  
 55  2. **OAuth2 Authorization**: The user is redirected to the MCP provider's authorization server (for example, Atlassian or Stripe). The user logs in using their credentials for that service and grants permission for Agent Mesh Enterprise to access their resources.
 56  
 57  3. **Authorization Code Exchange**: After successful authentication, the provider redirects back to Agent Mesh Enterprise with an authorization code. The system exchanges this code for access tokens at the provider's token endpoint.
 58  
 59  4. **Credential Storage**: Agent Mesh Enterprise stores the access token (and refresh token if provided) in the credential service. Credentials are encrypted at rest (when using a database) and isolated per agent, user, and MCP tool.
 60  
 61  5. **Authenticated Requests**: Subsequent requests to the MCP tool use the stored credentials automatically. The user does not need to re-authenticate unless credentials expire or are revoked.
 62  
 63  6. **Token Refresh**: If the MCP provider supports refresh tokens and an access token expires, Agent Mesh Enterprise automatically obtains a new access token without requiring user interaction.
 64  
 65  From an administrator's perspective, this flow is transparent once configured. Users experience a one-time authentication prompt per MCP tool, after which their access works seamlessly.
 66  
 67  ### Credential Storage
 68  
 69  Agent Mesh Enterprise manages user credentials through a dedicated credential service with several important characteristics:
 70  
 71  **Encryption at Rest**: When using database persistence, all stored credentials are automatically encrypted using the agent's unique identifier as the encryption key. This ensures that credentials cannot be used if extracted from database storage. Memory-based storage does not persist credentials to disk.
 72  
 73  **Multi-Tenant Isolation**: Credentials are scoped to the combination of agent ID, user identity, and MCP tool. Credentials for one agent cannot be accessed by another agent, even for the same user and tool. This provides strong isolation in multi-tenant deployments.
 74  
 75  **Configurable Expiration**: You can configure a time-to-live (TTL) for stored credentials. After the TTL expires, Agent Mesh Enterprise removes credentials from storage, requiring users to re-authenticate. This reduces the risk of long-lived credential compromise.
 76  
 77  **Persistence Options**: Credentials can be stored in memory (for development or ephemeral deployments) or in a database (for production deployments where credentials should survive agent restarts).
 78  
 79  ### Credential Lifecycle
 80  
 81  Understanding the credential lifecycle helps you plan operational procedures:
 82  
 83  1. **Acquisition**: Credentials are acquired when a user first authenticates with an MCP provider through the OAuth2 flow.
 84  
 85  2. **Active Use**: Stored credentials are used automatically for all subsequent requests to that MCP tool by that user.
 86  
 87  3. **Expiration**: Credentials expire either through TTL timeout (configured by you) or token expiration (set by the MCP provider). Agent Mesh Enterprise attempts to refresh expired tokens if refresh tokens are available.
 88  
 89  4. **Revocation**: Users or administrators can revoke credentials at the MCP provider level. When Agent Mesh Enterprise attempts to use revoked credentials, the request fails and the user must re-authenticate.
 90  
 91  5. **Deletion**: Credentials are marked as deleted when they expire (TTL timeout), but are retained in the persistence layer for audit purposes. With memory storage, credentials are removed from storage when the agent is restarted.
 92  
 93  ## Prerequisites
 94  
 95  Before you configure Secure User Delegated Access, ensure you have the following in place:
 96  
 97  ### MCP Provider Access
 98  
 99  Some MCP providers require administrative access to authorize Agent Mesh Enterprise before users can authenticate. The requirements vary by provider:
100  
101  **Example providers requiring domain authorization** (Atlassian, Stripe, Canva):
102  
103  - Administrative access to the MCP provider's admin console
104  - Ability to add your Agent Mesh Enterprise domain to the provider's authorized domains list
105  
106  **Example providers without administrative requirements** (Cloudflare):
107  
108  - No administrative setup required
109  - Users can authenticate directly through the OAuth2 flow
110  
111  Check the specific requirements for your chosen MCP provider.
112  
113  ### Callback URI Configuration
114  
115  Your Agent Mesh Enterprise deployment must be accessible via a stable URL for OAuth2 callbacks. During the OAuth2 flow, users are redirected to the MCP provider for authentication, then redirected back to this callback URI.
116  
117  You will configure this callback URI as an environment variable (see [Configure OAuth2 Callback URI](#configure-oauth2-callback-uri)). For providers requiring domain authorization (for example: Atlassian, Stripe, Canva), you will also register this domain in the provider's admin console.
118  
119  ### Database Setup (For Production)
120  
121  For production deployments, you should use SQL persistence to ensure credentials survive agent restarts. You need:
122  
123  - A supported SQL database (SQLite, PostgreSQL, MySQL, or other SQL database supported by SQLAlchemy)
124  - Appropriate database credentials and connection information
125  - Sufficient storage for credential data
126  
127  See [Configure Session Service Type](#configure-session-service-type) for details on setting up persistence. For development or testing, you can use memory storage, but credentials will be lost when the agent restarts.
128  
129  ## Configuration Steps
130  
131  Configuring Secure User Delegated Access involves several steps: configuring credential storage and lifecycle, optionally configuring the trust manager for enhanced security, and configuring your MCP tools with OAuth2 authentication.
132  
133  ### Step 1: Configure Credential Storage and Lifecycle
134  
135  The credential service is automatically created and manages storage, retrieval, and lifecycle of user credentials. You configure it through environment variables and your session service configuration.
136  
137  #### Configure Credential Time-to-Live
138  
139  Set the credential TTL to control how long credentials remain valid in storage:
140  
141  ```bash
142  export SECRETS_TTL_SECONDS=86400  # 24 hours
143  ```
144  
145  The TTL value is specified in seconds. Common values include:
146  
147  - `3600` - 1 hour (high security, frequent re-authentication)
148  - `86400` - 24 hours (balance of security and convenience)
149  - `604800` - 7 days (low security, infrequent re-authentication)
150  - `2592000` - 30 days (default if not specified)
151  
152  Choose a TTL based on your security requirements and user experience considerations. Shorter TTLs require users to re-authenticate more frequently but reduce the window of exposure for compromised credentials.
153  
154  #### Configure OAuth2 Callback URI
155  
156  Set the callback URI where MCP providers will redirect users after authentication:
157  
158  ```bash
159  export OAUTH_TOOL_REDIRECT_URI=https://my.domain.com/api/v1/auth/tool/callback
160  ```
161  
162  Replace `https://my.domain.com` with your actual Agent Mesh Enterprise domain.
163  
164  **Important**: The path `/api/v1/auth/tool/callback` must not be changed. This is the required callback endpoint path.
165  
166  This URL must:
167  
168  - Match exactly what you register with your MCP provider (for providers requiring domain authorization)
169  - Use HTTPS in production (HTTP is only acceptable for local development)
170  - Be accessible from users' browsers
171  
172  #### Configure Session Service Type
173  
174  User credentials are stored in the same database configured for session storage. For details on configuring session storage, see the [Session Storage documentation](../installing-and-configuring/session-storage.md).
175  
176  Memory storage does not persist credentials across agent restarts. Use this only for development and testing.
177  
178  **Important**: Ensure each agent has its own database to maintain proper credential isolation between agents.
179  
180  ### Step 2: Configure Trust Manager (Recommended)
181  
182  The trust manager provides critical security for Secure User Delegated Access by cryptographically verifying user identities throughout the system. It ensures that credentials can only be accessed by the user who created them, preventing unauthorized access even if agent communication is compromised.
183  
184  **Enabling the trust manager is strongly recommended for production deployments.**
185  
186  #### Understanding Trust Manager
187  
188  The trust manager uses public/private key cryptography to verify user identity:
189  
190  1. **Gateway Authentication**: The WebUI Gateway verifies the user's identity during login (via SSO or other authentication)
191  2. **Cryptographic Signing**: The gateway cryptographically signs each user's identity using its private key
192  3. **Agent Verification**: When an agent receives a request, it uses the gateway's public key to verify the signed identity
193  4. **Credential Access Control**: The agent only grants access to credentials if the verified identity matches the credential owner
194  
195  This cryptographic verification ensures that:
196  
197  - User identities cannot be forged or tampered with in transit
198  - Each user can only access their own credentials
199  - Compromised agents cannot access other users' credentials
200  - All credential operations have verifiable audit trails
201  
202  **The trust manager must be enabled on both the WebUI Gateway and all agents** to function correctly.
203  
204  #### Enable Trust Manager on SSE Gateway
205  
206  Add the trust manager configuration to your SSE Gateway configuration:
207  
208  ```yaml
209  # In your gateway configuration
210  trust_manager:
211    enabled: true
212  ```
213  
214  #### Enable Trust Manager on Agents
215  
216  Add the same trust manager configuration to each agent's configuration:
217  
218  ```yaml
219  # In each agent's configuration YAML
220  trust_manager:
221    enabled: true
222  ```
223  
224  #### Example Configuration
225  
226  Here is an example showing the trust manager full configuration (default values) on the gateway:
227  
228  ```yaml
229  # webui_gateway.yaml
230  apps:
231    - name: a2a_webui_app
232      app_config:
233        # ... other configuration ...
234  
235        trust_manager:
236          enabled: true
237          card_publish_interval_seconds: 10
238          card_expiration_days: 7
239          verification_mode: "permissive"  # or "strict" for production
240          clock_skew_tolerance_seconds: 300
241          enable_time_validation: true
242          jwt_default_ttl_seconds: 3600
243          jwt_max_ttl_seconds: 86400
244  ```
245  
246  #### Secure Solace Broker Provisioning for Trust Manager
247  
248  When using the trust manager, you should configure the Solace broker with proper credentials and ACLs to ensure secure trust card publishing. This prevents components from impersonating each other.
249  
250  **Distinct Credentials Per Component**
251  
252  Each gateway and agent instance should have its own unique broker credentials:
253  
254  - Each gateway instance requires unique `client_username` and `client_password`
255  - Each agent instance requires unique `client_username` and `client_password`
256  - Never share credentials between different component instances
257  
258  **ACL Configuration for Trust Card Publishing**
259  
260  Configure Access Control Lists (ACLs) on your Solace broker to restrict which topics each component can publish to:
261  
262  **For Gateway instances:**
263  
264  Only the specific gateway instance is allowed to publish on its trust card topic:
265  
266  ```
267  Topic: {namespace}/a2a/v1/trust/gateway/{gateway_broker_client_username}
268  Permission: Publish (allow only for this gateway's client username)
269  ```
270  
271  **For Agent instances:**
272  
273  Only the specific agent instance is allowed to publish on its trust card topic:
274  
275  ```
276  Topic: {namespace}/a2a/v1/trust/agent/{agent_broker_client_username}
277  Permission: Publish (allow only for this agent's client username)
278  ```
279  
280  Where:
281  - `{namespace}` is your configured namespace (e.g., `a2a/dev` or `a2a/prod`)
282  - `{gateway_broker_client_username}` is the unique broker username for the gateway
283  - `{agent_broker_client_username}` is the unique broker username for each agent
284  
285  **Example ACL Configuration:**
286  
287  If your namespace is `a2a/prod`, gateway username is `webui-gateway-01`, and agent username is `employee-agent-01`:
288  
289  ```
290  # Gateway ACL
291  Topic: a2a/prod/a2a/v1/trust/gateway/webui-gateway-01
292  Client Username: webui-gateway-01
293  Permission: Publish
294  
295  # Agent ACL
296  Topic: a2a/prod/a2a/v1/trust/agent/employee-agent-01
297  Client Username: employee-agent-01
298  Permission: Publish
299  ```
300  
301  These ACLs ensure that:
302  - Components cannot publish trust cards pretending to be other components
303  - Trust card verification remains cryptographically secure
304  - Compromised credentials for one component cannot affect other components
305  
306  ### Step 3: Configure MCP Tools with OAuth2 Authentication
307  
308  To use Secure User Delegated Access, you must configure your MCP tools to use OAuth2 authentication and provide a manifest of available tools.
309  
310  #### Basic MCP Tool Structure
311  
312  An MCP tool configuration with OAuth2 authentication follows this structure:
313  
314  ```yaml
315  tools:
316    - tool_type: mcp
317      connection_params:
318        type: sse  # or streamable-http
319        url: "https://mcp.example.com/v1/sse"
320      auth:
321        type: oauth2
322      manifest:
323        - name: exampleTool
324          description: Example tool description
325          inputSchema:
326            type: object
327            properties: {}
328            additionalProperties: false
329            $schema: http://json-schema.org/draft-07/schema#
330  ```
331  
332  The key components are:
333  
334  **`tool_type: mcp`**: Identifies this as an MCP tool configuration.
335  
336  **`connection_params`**: Specifies how to connect to the remote MCP server:
337  - `type`: Either `sse` (Server-Sent Events) or `streamable-http`
338  - `url`: The endpoint URL for the MCP server
339  
340  **`auth`**: Specifies the authentication type:
341  - `type: oauth2`: Enables OAuth2 user delegated access for this tool
342  
343  **`manifest`**: Defines the tools available from this MCP server (explained below).
344  
345  #### Understanding the Manifest Requirement
346  
347  Due to limitations in the MCP protocol, Agent Mesh Enterprise cannot automatically discover available tools from OAuth2-protected MCP servers. The OAuth2 flow requires user interaction, which prevents the automatic tool discovery process from working.
348  
349  To work around this limitation, you must provide a manifest that lists the tools available from the MCP server. This manifest is identical to what the MCP server would return from its tools list command.
350  
351  #### Obtaining the Manifest
352  
353  You can obtain the manifest in several ways:
354  
355  **Method 1: Use MCP Provider Documentation**
356  
357  Many MCP providers document their available tools. Check the provider's documentation for a list of tools and their schemas.
358  
359  **Method 2: Use MCP Inspector**
360  
361  Use a tool like [MCP Inspector](https://github.com/modelcontextprotocol/inspector) to connect to the MCP server and retrieve the output of the tools list command. You can then use this output directly as your manifest configuration.
362  
363  #### Manifest Format
364  
365  Each tool in the manifest follows this format:
366  
367  ```yaml
368  - name: toolName
369    description: Tool description explaining what it does
370    inputSchema:
371      type: object
372      properties:
373        parameterName:
374          type: string
375          description: Parameter description
376      required:
377        - parameterName
378      additionalProperties: false
379      $schema: http://json-schema.org/draft-07/schema#
380  ```
381  
382  The `inputSchema` is a standard JSON Schema (draft-07) that defines what parameters the tool accepts. This schema is used for validation and to help the AI model understand how to call the tool.
383  
384  **Note**: Complete sample MCP configurations are available in the [examples/agents/remote-mcp directory](https://github.com/SolaceLabs/solace-agent-mesh/tree/main/examples/agents/remote-mcp) for each of the tested remote MCP providers (Atlassian, Stripe, Cloudflare, Canva) to simplify setup.
385  
386  ### Step 4: Deploy Configuration
387  
388  After configuring credential storage, trust manager, and MCP tools, deploy these configurations to your Agent Mesh Enterprise installation. Ensure that:
389  
390  - Configuration files with MCP tool definitions are accessible to the agents
391  - Environment variables (`SECRETS_TTL_SECONDS`, `OAUTH_TOOL_REDIRECT_URI`) are set
392  - Database URLs are configured if using SQL persistence
393  - The installation is restarted to load the new configuration
394  
395  After deployment, test the OAuth2 flow by attempting to use an OAuth2-enabled MCP tool. Users should be prompted to authenticate with the MCP provider.
396  
397  ## Security Considerations
398  
399  Secure User Delegated Access involves storing and managing user credentials, which requires careful attention to security. This section outlines the security features built into the system and additional measures you should implement.
400  
401  ### Encryption at Rest
402  
403  When using SQL persistence, all stored credentials are automatically encrypted before being written to the database. Agent Mesh Enterprise uses the agent's unique identifier as the encryption key, ensuring that credentials cannot be used if extracted from storage.
404  
405  This encryption is automatic and requires no configuration. However, it means that:
406  
407  - Credentials encrypted by one agent cannot be decrypted by another agent
408  - If an agent's unique identifier changes, existing credentials become inaccessible
409  - Database backups contain encrypted credentials that are useless without the agent
410  
411  For additional security, ensure your database is stored on encrypted storage volumes provided by your infrastructure (for example, encrypted EBS volumes in AWS, encrypted persistent disks in GCP).
412  
413  **Note**: Memory-based storage does not persist credentials to disk, so encryption at rest does not apply. Memory storage should only be used for development and testing.
414  
415  ### Trust Manager for Identity Verification
416  
417  The trust manager provides defense-in-depth by verifying user identity on every credential operation. This prevents several attack scenarios:
418  
419  **Credential Theft via Agent Compromise**: If an attacker compromises an agent and attempts to retrieve another user's credentials, the trust manager validates the user's identity token and denies access.
420  
421  **Man-in-the-Middle Attacks**: The trust manager validates that identity tokens are signed by a trusted issuer, preventing token forgery.
422  
423  **Replay Attacks**: Identity tokens include expiration times, limiting the window for replay attacks.
424  
425  To maximize trust manager security:
426  
427  - Always enable trust manager (`enabled: true`) in production
428  - Monitor trust manager logs for denied access attempts
429  
430  ### Credential Aging and Expiration
431  
432  The `SECRETS_TTL_SECONDS` configuration provides defense against long-lived credential compromise. By setting an appropriate TTL, you ensure that:
433  
434  - Stolen credentials have a limited lifetime
435  - Terminated users' credentials expire automatically
436  - Users periodically re-authenticate, allowing detection of account compromise
437  
438  The default TTL is 30 days (2592000 seconds) if not specified.
439  
440  ### Provider-Side Revocation
441  
442  MCP providers can revoke user credentials at any time through their own admin consoles. This provides an additional security control:
443  
444  **User Offboarding**: When a user leaves the organization, revoke their access at the MCP provider level (Atlassian, Stripe, etc.) in addition to removing their access to Agent Mesh Enterprise.
445  
446  **Incident Response**: If you suspect credential compromise, immediately revoke access at the MCP provider level. This blocks credential use even before they expire in Agent Mesh Enterprise.
447  
448  **Audit Compliance**: Provider-side revocation creates audit logs in the provider's system, which may be required for compliance purposes.
449  
450  Document your incident response procedures to include MCP provider credential revocation as a standard step.
451  
452  ### Network Security
453  
454  Secure User Delegated Access involves network communication between users, Agent Mesh Enterprise, and MCP providers. Implement these network security measures:
455  
456  **Use HTTPS Everywhere**: All communication must use HTTPS in production:
457  - User to Agent Mesh Enterprise: HTTPS
458  - Agent Mesh Enterprise to MCP providers: HTTPS
459  - MCP provider redirects back to Agent Mesh Enterprise: HTTPS
460  
461  **Internal Communication Security**: If using the trust manager, ensure communication between agents and the SSE gateway is encrypted and authenticated.
462  
463  ### Logging and Monitoring
464  
465  Enable comprehensive logging to detect and respond to security issues.
466  
467  **Anomalous Patterns**: Monitor for unusual patterns such as:
468  - Multiple failed authentication attempts from a single user
469  - Credential access from unusual IP addresses or locations
470  - Rapid credential creation/deletion cycles
471  
472  Integrate these logs with your SIEM (Security Information and Event Management) system for centralized monitoring and alerting.
473  
474  ## Best Practices
475  
476  Following these best practices helps you deploy Secure User Delegated Access securely and reliably.
477  
478  ### Always Enable Trust Manager in Production
479  
480  The trust manager provides critical defense-in-depth security. Always enable it in production deployments:
481  
482  ```yaml
483  trust_manager:
484    enabled: true
485  ```
486  
487  Only disable the trust manager in development or testing environments where security is not a concern.
488  
489  ### Use SQL Persistence in Production
490  
491  Memory persistence is only suitable for development and testing. Production deployments must use SQL persistence:
492  
493  ```yaml
494  session_service:
495    type: "sql"
496    database_url: ${AGENT_DATABASE_URL}
497    default_behavior: "PERSISTENT"
498  ```
499  
500  ### Set Appropriate TTL Values
501  
502  Choose TTL values based on your security requirements and user experience considerations.
503  
504  Never exceed 30 days (2592000 seconds). Document your TTL policy and rationale in your security documentation.
505  
506  ### Configure Separate Session Storage Per Agent
507  
508  Each agent should have its own session storage database, which will automatically be used for credential storage as well.
509  
510  This isolation provides defense-in-depth and simplifies multi-tenant deployments. See the [Session Storage documentation](../installing-and-configuring/session-storage.md) for more details on configuring session storage.
511  
512  ### Use HTTPS in Production
513  
514  Configure HTTPS for all endpoints in production:
515  
516  - Set `OAUTH_TOOL_REDIRECT_URI` to an HTTPS URL
517  - Configure SSL certificates for Agent Mesh Enterprise
518  - Verify that MCP providers use HTTPS endpoints
519  - Never use HTTP in production (only acceptable for local development)
520  
521  ## Troubleshooting
522  
523  This section addresses common issues you may encounter when configuring or operating Secure User Delegated Access.
524  
525  ### OAuth2 Authentication Flow Fails
526  
527  **Symptoms**: Users report that they are redirected to the MCP provider but never redirected back to Agent Mesh Enterprise, or they see an error after attempting to authenticate.
528  
529  **Possible Causes and Solutions**:
530  
531  1. **Incorrect Callback URI**:
532     - **Verify**: Check that `OAUTH_TOOL_REDIRECT_URI` exactly matches the redirect URI registered with the MCP provider.
533     - **Common mistake**: Mismatched protocols (HTTP vs HTTPS), missing ports, trailing slashes.
534     - **Solution**: Update either the environment variable or the MCP provider configuration to match exactly.
535  
536  2. **Domain Not Authorized** (For providers requiring domain authorization):
537     - **Verify**: Check that your domain is listed in the MCP provider's authorized domains for MCP access.
538     - **Solution**: Add your domain to the provider's authorized domain list following their documentation.
539  
540  ### Credentials Not Persisting Across Restarts
541  
542  **Symptoms**: Users must re-authenticate every time the agent restarts, even though SQL persistence is configured.
543  
544  **Possible Causes and Solutions**:
545  
546  1. **Memory Storage Configured Instead of SQL**:
547     - **Verify**: Check the agent configuration to ensure `session_service.type` is set to `"sql"`, not `"memory"`.
548     - **Solution**: Update the configuration and restart the agent.
549  
550  ### Trust Manager Denying Access
551  
552  **Symptoms**: Users receive "Access Denied" errors when attempting to use MCP tools, or logs show trust manager access denials.
553  
554  **Possible Causes and Solutions**:
555  
556  1. **Trust Manager Not Configured on All Components**:
557     - **Verify**: Ensure trust manager is enabled on both the SSE Gateway and all agents.
558     - **Solution**: Add trust manager configuration to all component configurations and restart.
559  
560  ### Credentials Expire Too Quickly or Too Slowly
561  
562  **Symptoms**: Users report that they need to re-authenticate too frequently, or credentials remain valid longer than expected.
563  
564  **Possible Causes and Solutions**:
565  
566  1. **TTL Not Set Correctly**:
567     - **Verify**: Check the `SECRETS_TTL_SECONDS` environment variable value.
568     - **Solution**: Update the environment variable to the desired TTL and restart the agent.
569  
570  2. **MCP Provider Token Expiration**:
571     - **Note**: Even if your TTL is long, MCP providers may issue short-lived tokens.
572     - **Verify**: Check the MCP provider's token expiration policy.
573     - **Solution**: This is controlled by the MCP provider and cannot be changed. Ensure refresh tokens are working correctly to automatically renew expired tokens.
574  
575  3. **Refresh Tokens Not Working**:
576     - **Verify**: Check that the MCP provider issues refresh tokens and that Agent Mesh Enterprise is configured to use them.
577     - **Diagnostic**: Look for token refresh attempts in the logs.
578     - **Solution**: Ensure the OAuth2 scope includes `offline_access` or equivalent for the provider.
579  
580  ### Manifest Configuration Errors
581  
582  **Symptoms**: MCP tools are not available, or the AI model reports that tools are missing or incorrectly defined.
583  
584  **Possible Causes and Solutions**:
585  
586  1. **Manifest Missing or Empty**:
587     - **Verify**: Check that the `manifest` section in your MCP tool configuration is populated.
588     - **Solution**: Add the manifest with tool definitions as described in the Configuration Steps section.
589  
590  2. **Manifest Format Errors**:
591     - **Verify**: Ensure the manifest follows the correct YAML structure and JSON Schema format.
592     - **Common mistakes**: Missing required fields (`name`, `description`, `inputSchema`), invalid JSON Schema.
593     - **Solution**: Validate your manifest structure against the examples in this document.
594  
595  3. **JSON Schema Version Issues**:
596     - **Note**: Manifests typically use the `http://json-schema.org/draft-07/schema#` standard. However, some LLM providers may require the newer `https://json-schema.org/specification-links#2020-12` specification, which can lead to errors at inference time.
597     - **Solution**: Check your LLM provider's requirements and adjust the `$schema` field in your manifest accordingly.
598  
599  If you encounter issues, check the latest Agent Mesh Enterprise documentation or enable debug logging to review detailed error information:
600  
601  ```yaml
602  log:
603    stdout_log_level: DEBUG
604    log_file_level: DEBUG
605  ```
606  
607  ## Conclusion
608  
609  Secure User Delegated Access provides enhanced security and user-specific access control for Agent Mesh Enterprise deployments using remote MCP tools. By following the configuration steps, security considerations, and best practices in this guide, you can deploy this feature securely and reliably.
610  
611  Key takeaways:
612  
613  - Always enable the trust manager in production for defense-in-depth security
614  - Use SQL persistence to ensure credentials survive agent restarts
615  - Set appropriate TTL values balancing security and user experience
616  - Configure database-per-agent isolation for multi-tenant security
617  - Follow provider-specific setup instructions carefully
618  - Implement comprehensive monitoring and logging
619  - Maintain regular security reviews of your configuration
620  
621  Remember to keep your configuration updated as MCP providers evolve their APIs and authentication requirements, and regularly review your security posture to ensure continued protection of user credentials.