Installation Validation: Outlook Mail#
System Criticality: Business-Critical
Validation Approach: Risk-based
Related Design: DESIGN:OutlookMail
Related URS: @URS:OutlookMail
Related Risks: RISK:OutlookMail
Test Strategy#
| Level | Purpose | Owner | When |
|---|---|---|---|
| Unit | Individual functions (mail clients, auth) | Developers | During development |
| Integration | OBO flow + Graph API + tool wrappers | QA | After unit tests pass |
| System | End-to-end MCP protocol + email retrieval | QA | Before UAT |
| Security | Token handling, cross-user isolation, read-only enforcement | Security Team | Before UAT |
| UAT | Business validation with real mailboxes | End Users | Before production go-live |
Validation Scope: Risk-based testing focused on HIGH/CRITICAL risks identified in RISK:OutlookMail (R1, R2, R6).
Traceability Matrix#
| Requirement | Summary | Risk | Test Case | Type | Status |
|---|---|---|---|---|---|
| URS — ListEmails | List emails from folder | R6, R7 | TEST-001 | Integration | Not Started |
| URS — SearchEmails (FreeText) | Free-text search | R6, R7 | TEST-002 | Integration | Not Started |
| URS — SearchEmails (Sender) | Search by sender | R6, R7 | TEST-003 | Integration | Not Started |
| URS — SearchEmails (Subject) | Search by subject | R6, R7 | TEST-004 | Integration | Not Started |
| URS — SearchEmails (Attachments) | Filter emails with attachments | R5, R6 | TEST-005 | Integration | Not Started |
| URS — SearchEmails (DateRange) | Search by date range | R6, R7 | TEST-006 | Integration | Not Started |
| URS — ReadEmail | Read full email details | R6, R2, R7 | TEST-007 | Integration | Not Started |
| URS — Attachments | Retrieve attachment metadata | R5 | TEST-008 | Integration | Not Started |
| URS — Folders | List mail folders | R6 | TEST-009 | Integration | Not Started |
| URS — MailboxSettings | Retrieve mailbox settings | R6 | TEST-010 | Integration | Not Started |
| URS — UserScoped | Enforce user-scoped access | R6 | TEST-SEC-001 | Security | Not Started |
| URS — ReadOnly | Maintain read-only access | R4 | TEST-SEC-002 | Security | Not Started |
| URS — PrivacyPreservation | Preserve email privacy | R2, R3 | TEST-SEC-003 | Security | Not Started |
Coverage: 13/13 requirements = 100%
Test Cases#
TEST-001: List Emails from Inbox#
Related: URS — ListEmails | RISK-OutlookMail-R6, RISK-OutlookMail-R7 Risk Level: HIGH
Preconditions:
- Outlook MCP server deployed to dev environment (outlook.dev.connectors.novo-genai.com)
- Test user testuser1@novonordisk.com authenticated with valid Azure AD token
- Test user's inbox contains at least 10 emails
Steps:
1. MCP client sends list_emails tool call with folder_id="inbox", count=10
2. Server performs OBO token exchange for testuser1@novonordisk.com
3. Server calls GET /v1.0/me/mailFolders/inbox/messages?$top=10
4. Response returned to MCP client
Expected:
- HTTP 200 response from Graph API
- Up to 10 email objects returned
- Each email includes: id, subject, from, receivedDateTime, hasAttachments, isRead
- All emails belong to testuser1@novonordisk.com mailbox (verified by cross-checking sender/recipient fields against known test data)
- CloudWatch log entry: {"event": "email_list_requested", "user_id": "testuser1@novonordisk.com", "folder": "inbox", "count": 10}
- No email subject or body content in CloudWatch logs
Pass Criteria: - All expected results met - Zero emails from other users' mailboxes returned - Audit log entry created with correct user identity
TEST-002: Search Emails with Free-Text Query#
Related: URS — SearchEmails (FreeText) | RISK-OutlookMail-R6, RISK-OutlookMail-R7 Risk Level: HIGH
Preconditions: - Test user authenticated - Test mailbox contains emails with "quarterly report" in subject or body
Steps:
1. MCP client sends search_emails tool call with query="quarterly report", folder_id="inbox", count=10
2. Server performs OBO token exchange
3. Server calls GET /v1.0/me/mailFolders/inbox/messages?$search="quarterly report"&$top=10
4. If $search fails (HTTP 400), fallback to $filter with contains(subject, 'quarterly report')
5. Response returned to MCP client
Expected:
- HTTP 200 from Graph API (or HTTP 200 after fallback)
- Up to 10 emails matching query returned
- Progressive fallback strategy logged: {"event": "search_strategy", "strategy": "search_fallback", "original_query": "quarterly report"}
- All emails belong to authenticated user's mailbox
- No email body content in CloudWatch logs
Pass Criteria:
- All expected results met
- Fallback strategy executes correctly if $search unsupported
- User-scoped access enforced (no cross-user results)
TEST-003: Search Emails by Sender#
Related: URS — SearchEmails (Sender) | RISK-OutlookMail-R6, RISK-OutlookMail-R7 Risk Level: MEDIUM
Preconditions:
- Test user authenticated
- Test mailbox contains emails from colleague@novonordisk.com
Steps:
1. MCP client sends search_emails tool call with sender="colleague@novonordisk.com", folder_id="inbox", count=25
2. Server calls GET /v1.0/me/mailFolders/inbox/messages?$filter=from/emailAddress/address eq 'colleague@novonordisk.com'&$top=25
Expected:
- HTTP 200 from Graph API
- Only emails from colleague@novonordisk.com returned
- Results ordered by receivedDateTime desc (most recent first)
- CloudWatch log: {"event": "email_search_requested", "user_id": "testuser1@novonordisk.com", "filter_strategy": "sender_filter"}
Pass Criteria:
- All returned emails have from.emailAddress.address == "colleague@novonordisk.com"
- No false positives (emails from other senders)
- Audit trail includes sender filter criteria (without email content)
TEST-004: Search Emails by Subject Keywords#
Related: URS — SearchEmails (Subject) | RISK-OutlookMail-R6, RISK-OutlookMail-R7 Risk Level: MEDIUM
Preconditions: - Test user authenticated - Test mailbox contains emails with "budget approval" in subject
Steps:
1. MCP client sends search_emails tool call with subject="budget approval", folder_id="inbox", count=10
2. Server calls GET /v1.0/me/mailFolders/inbox/messages?$filter=contains(subject, 'budget approval')&$top=10
Expected:
- HTTP 200 from Graph API
- Only emails with "budget approval" substring in subject returned
- CloudWatch log: {"event": "email_search_requested", "user_id": "testuser1@novonordisk.com", "filter_strategy": "subject_filter"}
- No actual subject text logged (only metadata)
Pass Criteria: - All returned emails contain keywords in subject - Subject text NOT present in CloudWatch logs (privacy check) - Case-insensitive matching works correctly
TEST-005: Filter Emails with Attachments#
Related: URS — SearchEmails (Attachments) | RISK-OutlookMail-R5, RISK-OutlookMail-R6 Risk Level: MEDIUM
Preconditions: - Test user authenticated - Test mailbox contains emails with and without attachments
Steps:
1. MCP client sends search_emails tool call with has_attachments=true, folder_id="inbox", count=10
2. Server calls GET /v1.0/me/mailFolders/inbox/messages?$filter=hasAttachments eq true&$top=10
Expected:
- HTTP 200 from Graph API
- Only emails with hasAttachments=true returned
- Each result includes hasAttachments field
- Attachment content NOT automatically downloaded
- CloudWatch log: {"event": "email_search_requested", "user_id": "testuser1@novonordisk.com", "filter_strategy": "attachment_filter"}
Pass Criteria:
- All returned emails have hasAttachments == true
- No attachment content in response (metadata-only as per R5 mitigation)
- Zero emails without attachments returned
TEST-006: Search Emails by Date Range#
Related: URS — SearchEmails (DateRange) | RISK-OutlookMail-R6, RISK-OutlookMail-R7 Risk Level: MEDIUM
Preconditions: - Test user authenticated - Test mailbox contains emails from January 2024
Steps:
1. MCP client sends search_emails tool call with received_after="2024-01-01T00:00:00Z", received_before="2024-01-31T23:59:59Z", folder_id="inbox", count=50
2. Server calls GET /v1.0/me/mailFolders/inbox/messages?$filter=receivedDateTime ge 2024-01-01T00:00:00Z and receivedDateTime le 2024-01-31T23:59:59Z&$top=50
Expected:
- HTTP 200 from Graph API
- Only emails with receivedDateTime within date range returned
- CloudWatch log includes date range filter (but not email timestamps)
Pass Criteria: - All returned emails within date range - Zero emails outside date range - ISO 8601 datetime parsing correct
TEST-007: Read Full Email Details#
Related: URS — ReadEmail | RISK-OutlookMail-R6, RISK-OutlookMail-R2, RISK-OutlookMail-R7 Risk Level: HIGH
Preconditions:
- Test user authenticated
- Test email ID: AAMkAGI2THVSAAA= exists in test user's mailbox
Steps:
1. MCP client sends read_email tool call with email_id="AAMkAGI2THVSAAA="
2. Server performs OBO token exchange
3. Server calls GET /v1.0/me/messages/AAMkAGI2THVSAAA=
4. If email has hasAttachments=true, server calls GET /v1.0/me/messages/AAMkAGI2THVSAAA=/attachments
5. Response returned to MCP client
Expected:
- HTTP 200 from Graph API
- Full email details returned: subject, from, to, body, receivedDateTime, importance, hasAttachments
- If attachments exist, metadata returned: [{ id, name, size, contentType }]
- CloudWatch log: {"event": "email_read_requested", "user_id": "testuser1@novonordisk.com", "message_id": "AAMkAGI2THVSAAA="}
- Email body content NOT in CloudWatch logs
- Email subject NOT in CloudWatch logs
Pass Criteria: - All expected fields present in response - Email body content NOT persisted to DynamoDB (verify via table scan) - Email content NOT in CloudWatch logs (manual log inspection) - Attachment metadata returned without content (R5 mitigation)
TEST-008: Retrieve Attachment Metadata#
Related: URS — Attachments | RISK-OutlookMail-R5 Risk Level: MEDIUM
Preconditions:
- Test user authenticated
- Test email AAMkAGI2THVSAAA= has attachments: document.pdf (500KB), image.png (150KB)
Steps:
1. MCP client sends read_email tool call with email_id="AAMkAGI2THVSAAA="
2. Server detects hasAttachments=true
3. Server calls GET /v1.0/me/messages/AAMkAGI2THVSAAA=/attachments
4. Response includes attachment metadata only
Expected:
- Attachment list returned: [{ id: "AAA", name: "document.pdf", size: 512000, contentType: "application/pdf" }, { id: "BBB", name: "image.png", size: 153600, contentType: "image/png" }]
- Attachment content NOT included in response (binary data not downloaded)
- CloudWatch log: {"event": "attachment_metadata_requested", "user_id": "testuser1@novonordisk.com", "message_id": "AAMkAGI2THVSAAA=", "attachment_count": 2}
- Attachment names NOT in logs (R2 privacy mitigation)
Pass Criteria: - Metadata-only response (no binary content) - Content download requires separate explicit operation (future phase) - Zero risk of malware exposure (R5 mitigation verified)
TEST-009: List Mail Folders#
Related: URS — Folders | RISK-OutlookMail-R6 Risk Level: LOW
Preconditions: - Test user authenticated - Test mailbox has standard folders: Inbox, Sent Items, Drafts, Deleted Items
Steps:
1. MCP client sends list_folders tool call
2. Server calls GET /v1.0/me/mailFolders
Expected:
- HTTP 200 from Graph API
- All user's mail folders returned
- Each folder includes: id, displayName, parentFolderId, childFolderCount, unreadItemCount, totalItemCount
- CloudWatch log: {"event": "folder_list_requested", "user_id": "testuser1@novonordisk.com"}
Pass Criteria: - Standard folders present in response - User-scoped access enforced (no other users' folders visible)
TEST-010: Retrieve Mailbox Settings#
Related: URS — MailboxSettings | RISK-OutlookMail-R6 Risk Level: LOW
Preconditions: - Test user authenticated - Test user's mailbox settings configured in Outlook
Steps:
1. MCP client sends get_mailbox_settings tool call
2. Server calls GET /v1.0/me/mailboxSettings
Expected:
- HTTP 200 from Graph API
- Mailbox settings returned: timeZone, language, workingHours, automaticRepliesSetting
- Settings reflect test user's Outlook configuration
- CloudWatch log: {"event": "mailbox_settings_requested", "user_id": "testuser1@novonordisk.com"}
Pass Criteria: - All expected settings fields present - No settings from other users returned
Security Tests#
TEST-SEC-001: Enforce User-Scoped Access (Cross-User Isolation)#
Related: URS — UserScoped | RISK-OutlookMail-R6 Risk Level: CRITICAL
Preconditions:
- Two test users authenticated: testuser1@novonordisk.com, testuser2@novonordisk.com
- testuser1 has email with ID AAMkAGI2THVSAAA= in their mailbox
- testuser2 does NOT have access to testuser1 mailbox
Steps:
1. testuser2 MCP client sends read_email tool call with email_id="AAMkAGI2THVSAAA=" (testuser1's email)
2. Server performs OBO token exchange for testuser2
3. Server calls GET /v1.0/me/messages/AAMkAGI2THVSAAA= with testuser2 token
Expected:
- HTTP 403 Forbidden or HTTP 404 Not Found from Graph API (message ID does not exist in testuser2 mailbox)
- Error returned to MCP client: "Email not found or access denied"
- CloudWatch log: {"event": "email_read_failed", "user_id": "testuser2@novonordisk.com", "error": "403_forbidden"}
- Zero data from testuser1 mailbox exposed
Pass Criteria: - Graph API rejects cross-user access (defense in depth) - OBO token exchange preserves user identity - No data leakage between users (R6 mitigation verified)
TEST-SEC-002: Enforce Read-Only Access (Write Operation Rejection)#
Related: URS — ReadOnly | RISK-OutlookMail-R4 Risk Level: HIGH
Preconditions:
- Test user authenticated with Mail.Read scope only
- Test email ID AAMkAGI2THVSAAA= exists in user's mailbox
Steps:
1. Attempt to send email via Graph API: POST /v1.0/me/sendMail with test user's OBO token
2. Attempt to delete email: DELETE /v1.0/me/messages/AAMkAGI2THVSAAA= with test user's OBO token
3. Attempt to update email: PATCH /v1.0/me/messages/AAMkAGI2THVSAAA= (e.g., mark as read) with test user's OBO token
Expected:
- All write operations return HTTP 403 Forbidden from Graph API
- Error message: "Insufficient privileges to complete the operation" or similar
- Zero emails sent, deleted, or modified
- CloudWatch logs show rejected operations (but NOT the attempted content)
Pass Criteria:
- Graph API enforces Mail.Read scope (no write permissions granted)
- MCP server does NOT implement write tool wrappers (code inspection confirms no POST/PATCH/DELETE methods in tools/mail.py)
- R4 mitigation verified (least privilege principle enforced)
TEST-SEC-003: Preserve Email Privacy (No Content in Logs or Storage)#
Related: URS — PrivacyPreservation | RISK-OutlookMail-R2, RISK-OutlookMail-R3 Risk Level: HIGH
Preconditions:
- Test user authenticated
- Test email with subject "CONFIDENTIAL: Q4 Strategy" and body containing PII/sensitive data
- CloudWatch Logs Insights query prepared: fields @message | filter @message like /CONFIDENTIAL/
Steps:
1. MCP client sends read_email tool call for test email
2. Email content returned to MCP client
3. Query CloudWatch logs for test session
4. Query DynamoDB table mcp-oauth-storage-outlook-mcp-dev for test user's records
Expected:
- Email subject "CONFIDENTIAL: Q4 Strategy" NOT present in CloudWatch logs
- Email body content NOT present in CloudWatch logs
- Only metadata logged: {"event": "email_read_requested", "user_id": "testuser1@novonordisk.com", "message_id": "AAMkAGI2THVSAAA="}
- DynamoDB table contains only OAuth tokens (access_token, refresh_token, expires_at) — NO email content, subject, sender, or recipient fields
- ECS task ephemeral storage inspected post-session: zero email files cached
Pass Criteria:
- CloudWatch logs query returns zero matches for email subject or body text
- DynamoDB table scan confirms zero email content fields (R3 mitigation verified)
- LOG_LEVEL=INFO enforced in production task definition (no DEBUG logs)
- GDPR Article 5 compliance (data minimization) verified
- R2 mitigation verified (structured logging with field exclusion)
TEST-SEC-004: OBO Token Tracing (User Identity Preservation)#
Related: URS — UserScoped | RISK-OutlookMail-R1, RISK-OutlookMail-R6 Risk Level: HIGH
Preconditions:
- Test user testuser1@novonordisk.com authenticated with Azure AD
- Initial MCP client Bearer token: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Ik... (contains oid claim for testuser1)
Steps:
1. MCP client sends list_emails tool call with initial Bearer token
2. Server performs OBO token exchange via auth.py → MSAL acquire_token_on_behalf_of()
3. Server extracts oid (object ID) claim from exchanged Graph API token
4. Server uses Graph API token to call GET /v1.0/me/messages
5. CloudWatch log includes user identity from token
Expected:
- OBO-exchanged token contains same oid claim as initial token (user identity preserved)
- Graph API /me/ endpoint resolves to testuser1@novonordisk.com mailbox
- CloudWatch log: {"event": "obo_exchange_success", "user_id": "testuser1@novonordisk.com", "oid": "<testuser1_oid>"}
- Zero emails from other users' mailboxes returned
Pass Criteria:
- OBO flow preserves user identity through token chain (R1, R6 mitigation)
- oid claim matches between initial and exchanged tokens (verify via JWT decode)
- No shared service account used (each session user-scoped)
TEST-SEC-005: Authentication Enforcement (Reject Unauthenticated Requests)#
Related: URS — UserScoped | RISK-OutlookMail-R1 Risk Level: HIGH
Preconditions: - Outlook MCP server running - No Bearer token provided in request
Steps:
1. MCP client sends list_emails tool call WITHOUT Authorization: Bearer <token> header
2. Server detects missing authentication
Expected:
- HTTP 401 Unauthorized returned by FastMCP server
- Error message: "Missing or invalid authentication token"
- Zero emails returned
- CloudWatch log: {"event": "authentication_failed", "error": "missing_token"}
- No Graph API call made (authentication check at MCP server layer first)
Pass Criteria: - Unauthenticated requests rejected before any data access - Graph API never called with missing token - R1 mitigation verified (authentication required for all operations)
TEST-SEC-006: Expired/Tampered Token Rejection#
Related: URS — UserScoped | RISK-OutlookMail-R1 Risk Level: HIGH
Preconditions: - Test user's OAuth token expired (TTL > 60 minutes ago) - Tampered token: valid JWT structure but invalid signature
Steps:
1. MCP client sends list_emails tool call with expired token
2. MCP client sends list_emails tool call with tampered token
Expected:
- Expired token: OBO token exchange fails with HTTP 401 from Azure AD
- MCP server returns error: "Token expired — re-authentication required"
- Tampered token: OBO token exchange fails with HTTP 401 from Azure AD
- MCP server returns error: "Invalid token signature"
- Zero emails returned in both cases
- CloudWatch logs: {"event": "obo_exchange_failed", "error": "token_expired"}, {"event": "obo_exchange_failed", "error": "invalid_signature"}
Pass Criteria: - Azure AD rejects expired/tampered tokens (upstream validation) - MCP server propagates authentication errors to client - No Graph API call made with invalid tokens - R1 mitigation verified (token validation enforced)
TEST-SEC-007: TLS Enforcement (Transport Encryption)#
Related: URS — PrivacyPreservation | RISK-OutlookMail-R1 Risk Level: MEDIUM
Preconditions:
- Outlook MCP server URL: https://outlook.dev.connectors.novo-genai.com
- TLS 1.3 enforced at ALB layer
Steps:
1. Attempt HTTP request: http://outlook.dev.connectors.novo-genai.com/mcp (unencrypted)
2. Attempt TLS 1.0 connection (outdated protocol)
3. Valid HTTPS request: https://outlook.dev.connectors.novo-genai.com/mcp with TLS 1.3
Expected:
- HTTP request: Redirect to HTTPS (HTTP 301 or 302) or rejected (HTTP 403)
- TLS 1.0 connection: Rejected by ALB (connection refused)
- TLS 1.3 connection: Succeeds (HTTP 200)
- CloudWatch log for failed attempts: {"event": "tls_rejected", "protocol": "TLS1.0"}
Pass Criteria: - All traffic encrypted with TLS 1.3 (R1 mitigation — token protection in transit) - Legacy TLS versions rejected - HTTP-to-HTTPS redirect enforced (or HTTP blocked entirely)
Quality Gates#
Before production go-live, all of the following MUST pass:
- All HIGH/CRITICAL test cases passed (TEST-001, TEST-002, TEST-007, TEST-SEC-001, TEST-SEC-002, TEST-SEC-003, TEST-SEC-004, TEST-SEC-005, TEST-SEC-006)
- Zero critical defects open
- Zero HIGH-risk tests in "failed" state
- < 3 medium defects open (triaged and accepted by product owner)
- Security test suite passed (TEST-SEC-001 through TEST-SEC-007)
- CloudWatch log audit completed (TEST-SEC-003 verification)
- DynamoDB table schema verified (no email content fields)
- UAT sign-off obtained from at least 2 end users
- Azure AD app registration scope audit passed (
Mail.Readonly, noMail.ReadWrite) - Production ECS task definition reviewed:
LOG_LEVEL=INFO, no DEBUG logging - TLS 1.3 enforcement verified at ALB layer (TEST-SEC-007)
- OBO token flow end-to-end test passed with production Azure AD tenant
- CloudWatch log retention policy set to 90 days (GDPR Article 30 compliance)
- Runbook documented: incident response for R1 (token compromise), R2 (log exposure), R6 (cross-user access)
Test Environment#
Dev Environment:
- URL: https://outlook.dev.connectors.novo-genai.com/mcp
- ECS Cluster: aiconnectors-dev
- ECS Service: outlook-mcp-dev
- CloudWatch Log Group: /ecs/outlook-mcp-dev
- DynamoDB Table: mcp-oauth-storage-outlook-mcp-dev
- Token Storage: Memory (development only)
- Test Accounts:
- testuser1@novonordisk.com (primary test account, mailbox seeded with test data)
- testuser2@novonordisk.com (secondary account for cross-user isolation tests)
- testuser3@novonordisk.com (UAT account)
Staging/UAT Environment:
- URL: https://outlook.staging.connectors.novo-genai.com/mcp (if separate staging deployed)
- Test Data: Production-like mailbox data (anonymized/synthetic emails)
- Token Storage: DynamoDB (same as production)
Production Environment:
- URL: https://outlook.connectors.novo-genai.com/mcp
- ECS Cluster: aiconnectors-prod
- ECS Service: outlook-mcp-prod
- CloudWatch Log Group: /ecs/outlook-mcp-prod
- DynamoDB Table: mcp-oauth-storage-outlook-mcp-prod
- Token Storage: DynamoDB (encrypted at rest with KMS)
Test Tools:
- MCP client: Claude Desktop or custom MCP test harness
- HTTP client: httpx (Python) or curl for manual API tests
- CloudWatch Logs Insights: structured log queries
- AWS CLI: DynamoDB table scans, ECS task introspection
- JWT decoder: jwt.io for token claims inspection
- TLS test: nmap --script ssl-enum-ciphers or openssl s_client
Version: 1.0 Date: 2026-04-22 Author: Claude (AILab) Approved by: Pending QA Review