The CNAPP Bake-off Criteria Template

This template is part of our full evaluation guide — a step-by-step playbook for running structured bake-offs.

Cloud security is the fastest-growing evaluation category in security right now. Here's the criteria framework to run one properly — without getting misled by vendor demos.

CNAPP — Cloud-Native Application Protection Platform — is the category that replaced five separate tools with one platform that theoretically does all of them. Cloud Security Posture Management. Cloud Workload Protection. Cloud Infrastructure Entitlement Management. Container security. Runtime protection. The pitch is consolidation: one platform, one data model, one place to see everything. The reality of evaluating CNAPPs is that every vendor in the category sounds identical. "Unified visibility." "Agentless deployment." "AI-powered risk prioritisation." "Single pane of glass." The vocabulary is so homogenised that it's genuinely difficult to tell the products apart until you're six weeks into a POC and something important fails. This template is designed to surface those failures before you commit — with twelve weighted criteria, five POC tasks, and the specific questions that expose the gap between what vendors claim and what they actually deliver.

Before you open the template: the three questions that determine everything

1. Agentless or agent-based — and what does that actually mean for you? Every CNAPP vendor claims agentless deployment. What they mean by agentless varies significantly. Some are fully agentless for posture management but require agents for runtime enforcement. Some are agentless for scanning but blind to runtime behaviour without kernel-level access. Some give you a choice — agentless for discovery, agents for depth. The right answer for your environment depends on whether you need to see threats or stop them. Agentless deployment sees everything. Agent-based or kernel-level enforcement (via eBPF or LSM) can block at runtime. If your requirement is containment, not just detection, agentless-only won't get you there. Ask every vendor before they demo: "What specifically requires an agent or kernel module to work, and what works without one?" The answer immediately separates honest vendors from evasive ones.
2. Detect-only or enforce? This is the single most important question. The most common CNAPP disappointment is discovering — after the POC — that the platform detects threats but can't block them. "Runtime enforcement" is a heavily marketed term that can mean anything from "we alert on suspicious behaviour" to "we block the process at kernel level before damage occurs." Ask specifically: "What happens when your platform detects a crypto-miner running in a production container? Does it alert? Isolate? Block the process? How?" A vendor who says "we alert and you take action" is a detect-only platform. A vendor who says "we can block at the kernel level via eBPF enforcement" is a different product category. Your requirement determines which one you need.
3. What's your cloud footprint reality? A CNAPP that works beautifully on AWS might have gaps on GCP. A platform with excellent Kubernetes security might have limited serverless visibility. Before you write a single criterion, map your actual cloud footprint: which providers, how much Kubernetes, what serverless usage, any hybrid or on-premise workloads, and how much IaC you're running. Your criteria weights should reflect this reality, not a theoretical enterprise cloud environment.

The 12 criteria Visibility & Coverage

1. Cloud asset discovery and inventory What you're measuring: does the platform see everything in your environment — every account, every service, every workload — automatically and continuously? Suggested weight: Critical (×5) Score 9–10: Full asset inventory across all your cloud providers within 24 hours of deployment; continuous discovery as new resources are created; no manual configuration per service Score 7–8: Good coverage with minor gaps in specific services or regions; auto-discovery for major services Score 5–6: Solid coverage for compute and storage; gaps in serverless, managed databases or newer cloud services Score 3–4: Good AWS coverage but significant gaps on secondary providers Score 1–2: Requires significant manual configuration to achieve reasonable coverage
2. Kubernetes and container security What you're measuring: depth of visibility and control across your Kubernetes environments, including image scanning, runtime behaviour, and network policy. Suggested weight: Set based on your K8s footprint — Critical (×5) if K8s-heavy, High (×4) if mixed, Medium (×3) if limited Score 9–10: Full K8s visibility including cluster configuration, workload behaviour, network policy analysis, and admission controller integration; runtime alerts on anomalous container behaviour Score 7–8: Good K8s coverage with documented gaps in specific configurations Score 5–6: Image scanning and basic posture; limited runtime visibility inside containers Score 3–4: K8s treated as compute nodes; container-specific security is limited Score 1–2: Minimal K8s awareness
3. Serverless and managed service coverage What you're measuring: visibility into Lambda functions, managed databases, SaaS integrations and other services that don't have traditional compute surfaces. Suggested weight: High (×4) to Critical (×5) depending on your serverless footprint Score 9–10: Full visibility into serverless functions including configuration, permissions, data access and execution behaviour Score 7–8: Good serverless coverage; some managed services require manual configuration Score 5–6: Basic serverless inventory; limited behavioural visibility Score 3–4: Serverless is a known gap; vendor is working on it Score 1–2: No meaningful serverless coverage

Risk Prioritisation 4. Attack path analysis and exploitability context What you're measuring: does the platform understand which vulnerabilities are actually exploitable given your specific cloud configuration — not just which CVEs exist? Suggested weight: Critical (×5) Score 9–10: Context-aware risk prioritisation that combines vulnerability severity, internet exposure, identity permissions and lateral movement paths; can show the full attack path from external entry to crown jewel Score 7–8: Good risk contextualisation; attack path analysis available for major risk scenarios Score 5–6: Vulnerability severity + exposure context; limited attack path depth Score 3–4: Primarily CVSS-based prioritisation with limited cloud context Score 1–2: No meaningful prioritisation beyond raw vulnerability counts 5. Identity and entitlement risk (CIEM) What you're measuring: does the platform understand who can access what — and flag where permissions are excessive, misconfigured or create lateral movement risk? Suggested weight: High (×4) Score 9–10: Full least-privilege analysis across all cloud identities; can identify unused permissions, privilege escalation paths and cross-account risks; specific remediation suggestions Score 7–8: Good CIEM coverage; some complex cross-account or federation scenarios require manual review Score 5–6: Basic IAM misconfiguration detection; limited entitlement analysis depth Score 3–4: Flags obvious IAM misconfigurations; doesn't understand entitlement relationships Score 1–2: No meaningful CIEM capability 6. Noise management — signal-to-noise ratio What you're measuring: how many of the platform's alerts require real investigation vs how many are safe to ignore. This is the metric that determines whether your team uses the product. Suggested weight: High (×4) Score 9–10: Alert volume is manageable; demonstrated low false positive rate in your environment; suppression and tuning tools are intuitive; your team can action the alert queue without constant triage overhead Score 7–8: Acceptable signal-to-noise; tuning required but straightforward Score 5–6: Alert fatigue is a risk; significant tuning required before the product is operational Score 3–4: High noise without meaningful tuning options; team is likely to ignore the platform Score 1–2: Volume of alerts makes the product operationally unusable without a dedicated full-time analyst

Runtime Security 7. Runtime enforcement depth What you're measuring: can the platform block threats at runtime — not just detect and alert — and what is the mechanism? Suggested weight: Set based on your requirements — Critical (×5) if you need blocking, Medium (×3) if detection is sufficient Score 9–10: Kernel-level enforcement via eBPF or LSM; can block specific process behaviours, file access or network connections without killing the workload; demonstrated in your environment Score 7–8: Runtime enforcement available; some limitations in specific environments or container runtimes Score 5–6: Runtime detection with manual response actions; blocking requires human trigger Score 3–4: Runtime monitoring only; enforcement is roadmap Score 1–2: Detect-only with no meaningful runtime enforcement path 8. DevSecOps and CI/CD integration What you're measuring: does the platform shift security left — catching misconfigurations and vulnerabilities in code and IaC before they reach production? Suggested weight: High (×4) to Critical (×5) for DevSecOps-mature teams Score 9–10: Native integrations with your CI/CD toolchain; IaC scanning that works without manual configuration; developer-facing feedback in existing workflows (GitHub, GitLab, Jira); minimal friction Score 7–8: Good CI/CD integration; some manual configuration required; developer feedback available Score 5–6: IaC scanning available but not integrated into CI/CD pipeline; developer friction is significant Score 3–4: Post-deployment scanning only; limited shift-left capability Score 1–2: No meaningful DevSecOps integration

Compliance 9. Compliance framework coverage and evidence What you're measuring: does the platform support your required compliance frameworks continuously — and does it generate evidence your auditors will actually accept? Suggested weight: High (×4) if compliance-driven, Medium (×3) if not a primary driver Score 9–10: Continuous monitoring against your specific frameworks (SOC 2, ISO 27001, PCI DSS, GDPR, etc.); live compliance posture dashboard; audit-ready evidence export that your auditors have accepted; custom framework support Score 7–8: Good framework coverage; evidence export available; some manual work required for specific audit requirements Score 5–6: Framework coverage adequate; evidence quality requires supplementing with manual documentation Score 3–4: Compliance reports available but not audit-ready; significant manual work to translate findings into evidence Score 1–2: Compliance coverage is limited or generic

Operations 10. Deployment time and operational overhead What you're measuring: how long from decision to operational visibility, and how much ongoing maintenance the platform requires from your team. Suggested weight: Medium (×3) Score 9–10: Agentless deployment achieving broad visibility within hours; no infrastructure to manage; platform self-updates; ongoing maintenance is configuration, not operations Score 7–8: Deployment measured in days, not weeks; low ongoing overhead Score 5–6: Deployment requires professional services or significant engineering time; ongoing maintenance is non-trivial Score 3–4: Deployment is a project; operational overhead is a recurring cost Score 1–2: Deployment timeline measured in months; ongoing operations require dedicated resource 11. Multi-cloud parity What you're measuring: does the platform provide equivalent coverage and depth across all your cloud providers — or is one provider clearly the primary focus? Suggested weight: High (×4) if multi-cloud, Medium (×3) if primarily single-cloud Score 9–10: Genuine feature parity across AWS, Azure and GCP; equivalent depth of coverage, risk scoring and remediation across all providers; no provider-specific gaps that affect your environment Score 7–8: Good multi-cloud coverage; some secondary provider gaps are documented and acceptable Score 5–6: AWS-first with adequate but lighter Azure/GCP coverage Score 3–4: One provider is clearly primary; secondary providers feel bolted on Score 1–2: Effective single-cloud platform with multi-cloud marketing

Vendor Factors 12. Pricing model and total cost transparency What you're measuring: do you understand the true cost of operating this platform at your scale — before you sign? Suggested weight: High (×4) Score 9–10: Asset-based or workload-based pricing clearly stated; total cost at your scale provided upfront; no surprise modules; multi-year discount structure transparent; data egress costs explained Score 7–8: Pricing model clear; total cost requires some modelling but vendor is transparent Score 5–6: Pricing requires commercial negotiation before true cost is visible Score 3–4: Opaque until late in the process Score 1–2: Pricing only available under NDA after extensive scoping

The 5 POC tasks Task 1: Asset discovery completeness Deploy the platform across a representative test account (or invite read-only access to a production-equivalent account). Measure: how many assets are discovered automatically vs how many require manual configuration. How long does it take to achieve broad inventory? Are there services in your stack that the platform doesn't cover? Success looks like: 90%+ of your cloud assets discovered automatically within 24 hours of deployment. Task 2: Attack path demonstration Ask the vendor to demonstrate a realistic attack path in your environment — not a lab. Specifically: identify a path from an internet-facing workload to a sensitive resource (database, secrets manager, S3 bucket with sensitive data). Does the platform show the full path? Does it prioritise this risk above lower-severity findings? Success looks like: Platform identifies at least one real attack path in your environment within 48 hours of deployment, with clear prioritisation and specific remediation guidance. Task 3: Runtime enforcement test If runtime enforcement is a requirement: deploy a test workload and simulate a known-bad behaviour (reverse shell, crypto-miner, privilege escalation). Does the platform detect it? Alert on it? Block it? How quickly? Success looks like: Detection within 60 seconds; blocking (if required) without workload disruption; full audit trail available. Task 4: CI/CD integration test Introduce a known misconfiguration into an IaC template (an overly permissive IAM role, a public S3 bucket, an unencrypted database) and push it through your CI/CD pipeline. Does the platform catch it before deployment? How does the developer receive the feedback? Success looks like: Misconfiguration caught at PR/merge request stage; developer receives actionable feedback without leaving their existing workflow. Task 5: Compliance evidence export Generate a compliance report for one of your required frameworks. Export the evidence. Show it to the person who will present it to your auditors. Does it meet their requirements? What's missing? Success looks like: Auditor or compliance lead confirms the evidence is usable; no significant gaps requiring manual supplementation.

The one question that separates real platforms from marketing Ask this of every CNAPP contender at the start of your first demo: "If a crypto-miner starts running in one of our production containers right now, what exactly happens — and can you show me that in your platform?" The answer tells you everything. A detect-only platform says it alerts your team. A platform with runtime enforcement says it blocks the process at kernel level and shows you exactly how. A platform still on the roadmap for this capability deflects to a future release date. You've just discovered the most important fact about the product in under two minutes. The rest of the evaluation can proceed from there.

Running a CNAPP bake-off on pmpa — criteria set before any vendor enters, all three contenders working from the same brief, scoring based on evidence not impressions — is free for security teams. → picari.io May the best contender win.

---

Related reading: Learn why vendors win demos but fail POCs — and once you have your verdict, read how to negotiate once you have a verdict.