The Identity & Access Management Bake-off Criteria Template

This template is part of our full evaluation guide — a step-by-step playbook for running structured bake-offs.

IAM evaluations are uniquely complex — you're not evaluating one product, you're evaluating an entire discipline. Here's how to bring structure to it.

Identity and access management evaluations are harder than most security tool evaluations because IAM isn't a single product category. It's a discipline that encompasses workforce identity, customer identity, privileged access, secrets management, Zero Trust network access and a growing category called Identity Threat Detection and Response (ITDR). A single "IAM evaluation" could be looking at any combination of these. Before you define criteria, you need to be specific about what you're actually evaluating. This template covers the most common evaluation: workforce IAM, meaning the platform that manages how your employees authenticate, what they can access, and how that access is governed. It covers SSO, MFA, lifecycle management, directory integration and basic privileged access. If you're specifically evaluating PAM for privileged users, or CIEM for cloud entitlements, those are separate templates with different criteria.

The question that determines your evaluation before it starts Are you replacing something, or building something new? Replacement evaluations have a different primary criterion: migration complexity. If 2,000 employees are currently authenticating through Okta and you're considering Microsoft Entra ID as a replacement, the criteria most likely to determine the outcome is how painful the migration is — not which platform has better MFA options. Both platforms have excellent MFA. Only one of them will migrate your existing application integrations without 6 months of re-configuration work. Greenfield evaluations — where there's no existing platform to replace — have more freedom to weight criteria purely on capability. But they also have a different risk: the platform you choose will be very difficult to replace in 3 years. Weight vendor maturity, support quality and ecosystem breadth more heavily than you would in a replacement scenario.

The 12 criteria Core Identity

1. Directory integration and user sync What you're measuring: how cleanly the platform integrates with your existing directory (Active Directory, LDAP, HR system) and how reliably it keeps user data in sync. Suggested weight: Critical (×5) Score 9–10: Native, bidirectional sync with your directory; real-time provisioning and deprovisioning; no manual steps required for joiners, movers or leavers; sync errors are visible and self-healing Score 7–8: Good sync with minor manual steps for edge cases Score 5–6: Sync works but requires scheduled jobs or manual reconciliation for some scenarios Score 3–4: Directory sync is functional but fragile; deprovisioning has known gaps Score 1–2: Directory integration requires significant custom development
2. SSO application coverage What you're measuring: how many of your applications are available as pre-built connectors vs how many require custom SAML/OIDC configuration. Suggested weight: High (×4) Score 9–10: Pre-built connectors for all your critical applications; wizard-based configuration for any app that supports SAML/OIDC; your team can add a new app integration in under 30 minutes Score 7–8: Good connector library; a few critical apps require custom configuration Score 5–6: Major apps covered; significant custom configuration required for your specific stack Score 3–4: Limited connector library; most apps require manual SAML configuration Score 1–2: No meaningful pre-built connector library
3. MFA options and phishing resistance What you're measuring: which MFA methods are available and whether the platform supports phishing-resistant authentication (FIDO2/WebAuthn/passkeys) — which is increasingly required for compliance. Suggested weight: High (×4) Score 9–10: Full FIDO2/WebAuthn/passkey support; hardware key compatibility; step-up authentication for sensitive resources; push fatigue protection; contextual MFA policies that reduce friction for low-risk sessions Score 7–8: FIDO2 available; push MFA with number matching or additional context; good policy flexibility Score 5–6: Standard MFA options; FIDO2 on roadmap or limited Score 3–4: Push and TOTP only; limited phishing-resistant options Score 1–2: SMS-dependent or limited MFA options

Lifecycle Management 4. Automated provisioning and deprovisioning What you're measuring: how reliably the platform provisions access when someone joins and removes it when they leave — and how quickly. Suggested weight: Critical (×5) Score 9–10: SCIM-based provisioning with your HR system; automated deprovisioning within minutes of termination in HR; access removal confirmed with audit trail; no manual steps in the critical path Score 7–8: Automated provisioning with minor manual steps for complex scenarios Score 5–6: Provisioning is automated for major apps; some apps require manual deprovisioning Score 3–4: Provisioning workflows are manual or semi-manual; deprovisioning has documented gaps Score 1–2: Lifecycle management is a manual process with minimal automation 5. Access request and approval workflows What you're measuring: can employees self-serve access requests through the platform, and can managers approve or deny them — without raising a helpdesk ticket? Suggested weight: Medium (×3) Score 9–10: Self-service access request portal for employees; manager approval workflows with delegation; automatic access removal after a set period; full audit trail; integrates with your ticketing system Score 7–8: Self-service available; approval workflows functional with minor gaps Score 5–6: Basic access request capability; approval workflows require significant configuration Score 3–4: Self-service is limited; most access requests still go through helpdesk Score 1–2: No meaningful self-service or approval workflow capability 6. Access review and certification What you're measuring: does the platform support periodic access reviews — where managers confirm or revoke access for their team — and does it make those reviews actionable rather than ceremonial? Suggested weight: Medium (×3) Score 9–10: Scheduled access certification campaigns with manager self-service; automated reminders and escalation; revocation actions directly from the review interface; compliance evidence generated automatically Score 7–8: Access reviews available; automated reminders; some manual steps in the revocation workflow Score 5–6: Basic certification capability; high manual overhead for both managers and IT Score 3–4: Access reviews are a report, not a workflow Score 1–2: No meaningful access certification capability

Security & Governance 7. Adaptive and risk-based authentication What you're measuring: does the platform adjust authentication requirements based on risk signals — location, device, behaviour, threat intelligence — rather than applying the same policy to every login? Suggested weight: High (×4) Score 9–10: Real-time risk scoring on every authentication; step-up MFA triggered by anomalous signals; impossible travel detection; device trust integration; threat intelligence feeds; reduced friction for low-risk, trusted sessions Score 7–8: Good adaptive authentication; some risk signals require additional configuration Score 5–6: Basic conditional access policies; limited real-time risk scoring Score 3–4: Location and device-based policies; no meaningful behavioural risk assessment Score 1–2: Policy-based authentication with no adaptive capability 8. Privileged access and admin governance What you're measuring: how the platform controls and audits access to sensitive administrative functions — including the IAM platform itself. Suggested weight: High (×4) Score 9–10: Just-in-time privileged access for administrative functions; session recording for privileged operations; break-glass access procedures with alerts; admin access to the IAM platform itself is governed and audited Score 7–8: Good privileged access controls; session recording available for key scenarios Score 5–6: Basic admin role separation; privileged access controls are limited Score 3–4: Admin access is governed by role assignment; minimal JIT or session control Score 1–2: Administrative access is managed outside the platform 9. Identity threat detection (ITDR) What you're measuring: does the platform detect identity-based attacks — credential stuffing, account takeover, insider threats, anomalous access patterns — and alert or respond automatically? Suggested weight: Medium (×3) to High (×4) depending on your threat model Score 9–10: Continuous behavioural monitoring across all identities; automated detection of credential stuffing, impossible travel, anomalous access patterns; automated step-up or lockout responses; SIEM integration for correlation Score 7–8: Good threat detection; some manual investigation required for complex scenarios Score 5–6: Basic anomaly detection; limited automated response Score 3–4: Alerts on known-bad indicators; limited behavioural analysis Score 1–2: No meaningful identity threat detection

Integration & Operations 10. SIEM and security tool integration What you're measuring: does the platform send rich, structured authentication and access events to your SIEM — and can your SIEM trigger responses through the IAM platform? Suggested weight: High (×4) Score 9–10: Native integration with your SIEM; structured event data with full authentication context; bidirectional integration enabling your SIEM to trigger account lockout, MFA step-up or session revocation; API fully documented Score 7–8: Good SIEM integration; most events available; some manual enrichment required Score 5–6: Syslog-based integration; events are available but require parsing Score 3–4: Basic event export; limited integration depth Score 1–2: No meaningful SIEM integration 11. Developer and API experience What you're measuring: can your development teams integrate their applications with the IAM platform using standard protocols, without significant friction? Suggested weight: Medium (×3) to High (×4) for engineering-led organisations Score 9–10: First-class OIDC and SAML support with excellent documentation; SDKs for your development languages; sandbox environment for testing; clear developer portal with working examples Score 7–8: Good developer experience; some SDK gaps for specific languages Score 5–6: Standard protocols supported; documentation is adequate but not excellent Score 3–4: Developer integration requires significant vendor support Score 1–2: Integration is painful without dedicated vendor professional services

Vendor Factors 12. Pricing model and seat complexity What you're measuring: do you understand what you'll actually pay at your scale — including growth, and including all the features you've identified as requirements? Suggested weight: High (×4) Score 9–10: Per-user pricing clearly stated; features included in base vs premium clearly documented; total cost at current and projected scale provided upfront; no feature gating that makes the base product significantly less useful Score 7–8: Pricing clear; some premium features affect total cost Score 5–6: Pricing model requires modelling; some feature gating surprises Score 3–4: True cost only visible late in the commercial process Score 1–2: Opaque pricing that requires full scoping before any number is shared

The 5 POC tasks Task 1: New employee provisioning Simulate a new employee joining your company. Create their record in your HR system or directory. Measure how long it takes for the IAM platform to provision their accounts across your top 5 applications. Is anything manual? Does it work for a remote employee with a new device? Success looks like: Account provisioned across all 5 applications within 15 minutes of HR record creation; no manual steps required. Task 2: Employee offboarding Terminate an employee record in your HR system. Measure how quickly the IAM platform revokes access across all connected applications. Are any applications missed? Is there an audit trail? Success looks like: Access revoked across all connected applications within 15 minutes of termination; audit trail confirms which applications were updated and when. Task 3: Phishing-resistant MFA test Enrol a test user with FIDO2/WebAuthn. Attempt a phishing-style attack (AiTM proxy if you have the capability, or simply verify that push notification can be blocked without FIDO2). Does the platform's MFA hold up? Success looks like: FIDO2 authentication cannot be bypassed by a phishing proxy; push MFA has number matching or similar phishing-resistant mechanism. Task 4: Suspicious login simulation Simulate a suspicious login — impossible travel (authenticate from London, then from New York 10 minutes later), or a login from a new device in an unusual country. Does the platform detect it? What happens — step-up MFA, block, alert? Success looks like: Platform detects the anomalous login within 5 minutes; appropriate response is triggered automatically or flagged clearly for manual review. Task 5: Access review campaign Run a small access certification campaign — ask 5 managers to review and certify the access of their direct reports across 3 applications. Measure: how long does it take them? How much guidance do they need? Are revocations carried out automatically? Success looks like: Managers complete the review in under 10 minutes each with minimal guidance; revocations are automatic; audit trail is complete.

The IAM-specific negotiating point most teams miss IAM pricing is typically per-user, but the definition of "user" varies significantly between vendors. Some charge per named user. Some charge per monthly active user. Some charge per authentication. At 500 employees with contractors, partners and service accounts, these three models can produce dramatically different annual costs. Ask this specifically: "What counts as a billable user in your pricing model — and does that include contractors, service accounts and API integrations?" The answer will change your total cost calculation significantly. In some cases it will change which vendor wins the commercial comparison entirely.

Running an IAM bake-off on pmpa — criteria locked before vendors enter, same POC tasks for every contender — is free for security teams. → picari.io May the best contender win.

---

Related reading: Qualify vendors before your IAM POC, learn why vendors win demos but fail POCs, and once you have your verdict, read how to negotiate once you have a verdict.