Ramya Chitrakar and Scott Woodgate present Microsoft’s latest efforts to make email security performance more transparent, including dashboards and benchmarking reports for Defender for Office 365 and comparisons with other security vendors.

Microsoft Defender for Office 365: Transparent Benchmarks on Email Security Effectiveness

Authors: Ramya Chitrakar and Scott Woodgate

Overview

Cyber threats targeting email systems are evolving and increasingly sophisticated. To help organizations make informed decisions about their email security, Microsoft is sharing performance data for Microsoft Defender for Office 365 and other ecosystem providers. These transparency initiatives are designed to help security leaders achieve better defense-in-depth by understanding the real-world effectiveness of layered email security solutions.

Key Announcements

  • New Transparency Initiatives:
    • Launch of a customer-facing dashboard for Defender for Office 365 showcasing threat vector efficacy and security metrics.
    • Release of comparative benchmarking reports that measure Defender for Office 365’s performance versus Secure Email Gateways (SEGs) and Integrated Cloud Email Security (ICES) vendors.
  • Data-Driven Metrics:
    • Benchmarking based on real-world threat data, not just synthetic tests.
    • Transparent methodology for comparing product performance, including clearly defined standards for “missed threats”.

Defender for Office 365 Dashboard

  • Provides actionable insights to security teams:
    • Visualizes threats blocked before email delivery and post-delivery mitigation.
    • Reports on “missed” threats using stringent detection criteria.
    • Includes breakdowns of key features like Safe Links, Safe Attachments, and Zero-hour Auto Purge.
    • Helps answer: “How are my users being protected from malicious email content and attacks?”

Benchmarking Reports

Secure Email Gateway (SEG) Benchmarking

  • Seven SEG vendors and Defender for Office 365 were compared.
  • SEG threats defined as “missed” if not blocked pre-delivery or not removed promptly after delivery; Defender applied even stricter review.
  • Results: Defender for Office 365 had the lowest rate of missed threats under these criteria.
  • Benchmarking used aggregated and anonymized data, aligning with Microsoft’s privacy commitments.

Integrated Cloud Email Security (ICES) Vendor Benchmarking

  • Evaluates vendors that act as a secondary filter after Defender for Office 365.
  • ICES products leverage the Microsoft Graph API to move or delete emails.
  • Findings:
    • The biggest ICES value came in detection of marketing and bulk email (average 20% improvement), reducing user inbox clutter.
    • Smaller, but measurable gains for spam (0.51%) and malicious threats (0.30%).
  • More details are available on the benchmarking site.

Third-Party Validation

  • Benchmark methodologies reviewed by SE Labs, an independent authority on email security testing.
  • SE Labs’ CEO highlighted the value of real-world testing alongside synthetic benchmarks for holistic product evaluation.

Microsoft’s Ongoing Commitment

  • Microsoft will provide quarterly benchmark updates and welcomes community feedback.
  • The company remains focused on empowering customers with actionable, transparent data to guide cybersecurity choices.

Additional Resources

For ongoing updates and expert coverage, follow Microsoft Security on LinkedIn and X.

This post appeared first on “Microsoft Security Blog”. Read the entire article here