{"id":801,"date":"2026-04-16T05:11:40","date_gmt":"2026-04-16T05:11:40","guid":{"rendered":"https:\/\/www.devopsschool.com\/tutorials\/google-cloud-external-key-manager-tutorial-architecture-pricing-use-cases-and-hands-on-guide-for-security\/"},"modified":"2026-04-16T05:11:40","modified_gmt":"2026-04-16T05:11:40","slug":"google-cloud-external-key-manager-tutorial-architecture-pricing-use-cases-and-hands-on-guide-for-security","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/tutorials\/google-cloud-external-key-manager-tutorial-architecture-pricing-use-cases-and-hands-on-guide-for-security\/","title":{"rendered":"Google Cloud External Key Manager Tutorial: Architecture, Pricing, Use Cases, and Hands-On Guide for Security"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Category<\/h2>\n\n\n\n<p>Security<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">1. Introduction<\/h2>\n\n\n\n<p>Cloud External Key Manager is a Google Cloud Security capability that lets Google Cloud services use encryption keys that are stored and controlled outside of Google Cloud\u2014typically in a third-party key manager (for example, an HSM-backed enterprise key management system) that you operate or buy from a supported partner.<\/p>\n\n\n\n<p>In simple terms: Google Cloud can encrypt and decrypt your data using a key that never lives in Google\u2019s key storage. Google services ask your external key system to perform cryptographic operations, and your external system enforces your policies (availability, approvals, geography, access controls, rotation rules, and auditing) while Google Cloud keeps running your workloads.<\/p>\n\n\n\n<p>Technically, Cloud External Key Manager is implemented as <strong>Cloud EKM<\/strong> within <strong>Cloud Key Management Service (Cloud KMS)<\/strong>. You create Cloud KMS resources (key rings, keys, and versions) with an <strong>external protection level<\/strong>, and you configure an <strong>EKM connection<\/strong> so Cloud KMS can securely reach your external key manager endpoint (typically using mTLS and private networking patterns). When a Google Cloud service needs to encrypt\/decrypt with a Customer-Managed Encryption Key (CMEK), it calls Cloud KMS; for external keys, Cloud KMS then brokers the request to your external key manager.<\/p>\n\n\n\n<p>The problem it solves: organizations that require <strong>key custody, sovereignty, separation of duties, externalized governance, or \u201chold your own keys\u201d<\/strong> controls can still adopt Google Cloud services while keeping encryption keys under their control\u2014often to satisfy internal security requirements and regulatory expectations.<\/p>\n\n\n\n<blockquote>\n<p>Naming note (important): In current Google Cloud documentation, this capability is frequently referred to as <strong>Cloud EKM<\/strong> (External Key Manager) and is part of <strong>Cloud KMS<\/strong>. This tutorial uses the exact service name <strong>Cloud External Key Manager<\/strong> throughout, and maps it to the same capability described in official docs as \u201cCloud EKM\u201d. Verify the latest naming and UI labels in official documentation if you see differences.<\/p>\n<\/blockquote>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">2. What is Cloud External Key Manager?<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Official purpose<\/h3>\n\n\n\n<p>Cloud External Key Manager enables you to use <strong>externally managed encryption keys<\/strong> with Google Cloud services that integrate with <strong>Cloud KMS<\/strong>. The key material and\/or cryptographic operations are handled by a key manager outside Google Cloud, while Google Cloud services consume keys via Cloud KMS APIs and CMEK integrations.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Core capabilities<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>External key references in Cloud KMS<\/strong>: Create and manage Cloud KMS key resources that represent keys hosted externally.<\/li>\n<li><strong>Secure connectivity (EKM connection)<\/strong>: Configure how Cloud KMS reaches your external key manager, including identity and TLS controls.<\/li>\n<li><strong>CMEK-like integration model<\/strong>: Use externally managed keys in supported Google Cloud services through the familiar CMEK pattern (service \u2192 Cloud KMS \u2192 key operation).<\/li>\n<li><strong>Central policy and auditing<\/strong>: Keep key policy enforcement and audit trails in your external key manager, while also retaining Google Cloud audit logs for Cloud KMS access.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Major components (conceptual)<\/h3>\n\n\n\n<p>While exact resource names and fields may evolve, the Cloud External Key Manager model generally includes:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Cloud KMS key ring (location-scoped)<\/strong>: Container for keys in a specific Google Cloud location.<\/li>\n<li><strong>Cloud KMS crypto key (external protection level)<\/strong>: A logical key object representing an external key.<\/li>\n<li><strong>Cloud KMS key version<\/strong>: A version that points to an external key reference (for example, an external key URI\/identifier).<\/li>\n<li><strong>EKM connection<\/strong>: Configuration object that tells Cloud KMS how to reach and authenticate to the external key manager endpoint.<\/li>\n<li><strong>External key manager endpoint<\/strong>: Your external KMS\/HSM system (self-hosted or partner-hosted) that performs cryptographic operations or otherwise controls key usage.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Service type<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Managed Google Cloud control plane capability<\/strong>, implemented within <strong>Cloud KMS<\/strong>.<\/li>\n<li>You still manage your external key infrastructure separately (partner service or self-managed).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Scope: regional\/global\/zonal and project\/org boundaries<\/h3>\n\n\n\n<p>Cloud External Key Manager resources are primarily <strong>Cloud KMS resources<\/strong>, and Cloud KMS is <strong>location-based<\/strong>:\n&#8211; <strong>Key rings and keys are created in a specific Cloud KMS location<\/strong> (regional or multi-region, depending on Cloud KMS location options).\n&#8211; <strong>EKM connection resources are also location-based<\/strong> and must align with your Cloud KMS usage and supported integrations.\n&#8211; IAM is enforced at the <strong>project<\/strong> level (and can be managed via org\/folder hierarchy), with roles applied to Cloud KMS resources.<\/p>\n\n\n\n<p>Because exact supported locations, connection models, and partner availability can change, <strong>verify current location support<\/strong> in the official documentation for Cloud EKM \/ external keys.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How it fits into the Google Cloud ecosystem<\/h3>\n\n\n\n<p>Cloud External Key Manager sits at the intersection of:\n&#8211; <strong>Cloud KMS (Security)<\/strong>: primary API and resource model for keys and crypto operations.\n&#8211; <strong>Google Cloud services using CMEK<\/strong>: Cloud Storage, BigQuery, Compute Engine, and many others use Cloud KMS for encryption key operations (support for external keys varies by service\u2014verify per service).\n&#8211; <strong>IAM &amp; Audit Logs<\/strong>: Access control and auditability for key usage.\n&#8211; <strong>Networking &amp; connectivity services<\/strong>: Often needed to reach external endpoints in a private, controlled way (for example, Service Directory, private connectivity patterns, and restricted egress\u2014verify exact patterns for your deployment).<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">3. Why use Cloud External Key Manager?<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Business reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Regulatory alignment and audits<\/strong>: Some organizations must demonstrate that encryption keys are controlled outside the cloud provider\u2019s key store.<\/li>\n<li><strong>Vendor risk management<\/strong>: Reduces reliance on a single cloud provider for key custody.<\/li>\n<li><strong>Standardization<\/strong>: Reuse an existing enterprise key platform across multiple clouds and on-prem systems.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Technical reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>External key custody<\/strong>: Keys remain in an HSM-backed key manager you operate or contract.<\/li>\n<li><strong>Centralized cryptographic policy enforcement<\/strong>: Approval flows, geo-fencing, time-based access, quorum controls, and key usage limits can be enforced outside Google Cloud (depending on your external platform).<\/li>\n<li><strong>Consistent key lifecycle<\/strong>: Rotation and decommission processes stay consistent across environments.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Operational reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Separation of duties<\/strong>: Platform teams can operate workloads in Google Cloud while a security team controls the external key platform.<\/li>\n<li><strong>Unified auditing<\/strong>: External KMS audit trails plus Google Cloud audit logs provide layered evidence of access and use.<\/li>\n<li><strong>Central incident response<\/strong>: In some models, disabling or restricting a key externally can effectively stop access to encrypted data (use carefully; can cause outages).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Security\/compliance reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Key sovereignty<\/strong>: Keep keys in specific jurisdictions or under specific operational controls.<\/li>\n<li><strong>\u201cHold Your Own Key\u201d posture<\/strong>: Support internal policies that require keys not be stored in cloud-native KMS.<\/li>\n<li><strong>Access Transparency\/Justification patterns<\/strong>: In some organizations, external key control complements transparency programs. (For advanced features like Key Access Justifications, verify current requirements and availability in official docs.)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Scalability\/performance reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Scale Google Cloud services while keeping key control external<\/strong>: You can adopt managed services without importing all crypto into the app layer.<\/li>\n<li><strong>Operational offload<\/strong>: Google Cloud services still handle encryption integration; you don\u2019t have to rebuild envelope encryption in every application.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">When teams should choose Cloud External Key Manager<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>You have a strong requirement that <strong>key material and\/or cryptographic operations must remain outside Google Cloud<\/strong>.<\/li>\n<li>You already operate (or are willing to operate) a <strong>highly available external key manager<\/strong>, often backed by HSMs.<\/li>\n<li>You need <strong>separation of duties<\/strong>: cloud operators must not control keys.<\/li>\n<li>You have a compliance\/audit mandate that explicitly calls for <strong>external key control<\/strong>.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">When teams should not choose it<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>You need the simplest approach: <strong>Cloud KMS software keys<\/strong> or <strong>Cloud HSM<\/strong> are usually simpler to deploy and operate.<\/li>\n<li>You cannot operate an external key manager with <strong>high availability, low latency, and strong operational discipline<\/strong>.<\/li>\n<li>Your workloads are extremely latency-sensitive and would be impacted by additional network hops to an external endpoint.<\/li>\n<li>Your target Google Cloud services <strong>do not support external keys<\/strong> (support differs by product\u2014verify first).<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">4. Where is Cloud External Key Manager used?<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Industries<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Financial services (banking, payments, insurance)<\/li>\n<li>Healthcare and life sciences<\/li>\n<li>Government and regulated public sector<\/li>\n<li>Telecommunications<\/li>\n<li>Energy and critical infrastructure<\/li>\n<li>Large enterprises with strict internal security governance<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Team types<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Security engineering \/ cryptography teams<\/li>\n<li>Platform engineering (internal cloud platforms)<\/li>\n<li>SRE and operations teams<\/li>\n<li>Compliance and risk teams (policy definition and evidence)<\/li>\n<li>DevOps teams integrating CMEK into pipelines<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Workloads<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Managed storage encryption (object storage, data warehouses, databases) where CMEK is supported<\/li>\n<li>Compute workloads requiring disk encryption with enterprise key control<\/li>\n<li>Multi-cloud workloads seeking consistent key governance across providers<\/li>\n<li>Workloads that must demonstrate key custody outside cloud provider boundaries<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Architectures<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Centralized enterprise key platform + many cloud projects (hub-and-spoke)<\/li>\n<li>Shared services project hosting Cloud KMS resources, consumed by service projects<\/li>\n<li>Regulated landing zones with restricted egress and tight IAM controls<\/li>\n<li>Hybrid designs where external key manager is on-prem or partner-hosted<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Real-world deployment contexts<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Production<\/strong>: The most common and most justified use. Requires HA, monitoring, incident response, and tested failover.<\/li>\n<li><strong>Dev\/test<\/strong>: Useful for integration testing of CMEK\/external-key flows, but can be costly\/complex if it requires real external HSMs. Many teams use dev\/test to validate IAM, auditing, and failure handling.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">5. Top Use Cases and Scenarios<\/h2>\n\n\n\n<p>Below are realistic use cases for Cloud External Key Manager. Each includes the problem, why Cloud External Key Manager fits, and a short scenario.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">1) \u201cHold Your Own Key\u201d for regulated storage encryption<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Regulators or internal policy require encryption keys to be controlled outside the cloud provider.<\/li>\n<li><strong>Why it fits<\/strong>: Cloud External Key Manager lets Google Cloud services use external keys without storing key material in Google Cloud KMS.<\/li>\n<li><strong>Scenario<\/strong>: A bank stores sensitive reports in a Google Cloud storage service using CMEK with external keys hosted in an HSM cluster controlled by the bank\u2019s security team.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">2) Separation of duties between cloud operators and key custodians<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Cloud administrators should not be able to decrypt regulated data, even if they have broad access to cloud resources.<\/li>\n<li><strong>Why it fits<\/strong>: Key usage can be governed externally, with distinct admin teams and approval workflows.<\/li>\n<li><strong>Scenario<\/strong>: A platform team deploys workloads, but a security team must approve key usage for decryption operations in the external key manager.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">3) Central enterprise key governance across multiple clouds<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Different cloud-native KMS systems create inconsistent policies and auditing.<\/li>\n<li><strong>Why it fits<\/strong>: External key manager becomes the central point of control, while Google Cloud uses it through Cloud KMS integration.<\/li>\n<li><strong>Scenario<\/strong>: A global enterprise uses one external KMS platform for AWS, Azure, and Google Cloud and standardizes rotation and access controls.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">4) Data residency and key residency constraints<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Data may be processed in Google Cloud, but keys must remain under jurisdiction-specific controls.<\/li>\n<li><strong>Why it fits<\/strong>: External key custody can align with jurisdiction and operational constraints (subject to your external deployment).<\/li>\n<li><strong>Scenario<\/strong>: A healthcare provider processes data in Google Cloud but keeps keys in a dedicated external key manager controlled by a country-specific security team.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">5) Rapid key disablement (\u201ccryptographic kill switch\u201d) for incident response<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: During an incident, you need a decisive way to prevent decryption.<\/li>\n<li><strong>Why it fits<\/strong>: External policy can block key usage; Google services relying on the key will fail to decrypt.<\/li>\n<li><strong>Scenario<\/strong>: A breach is suspected; the security team disables a key externally to prevent access until investigation completes (with careful planning to avoid prolonged outages).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">6) Strong external auditing requirements<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Audit teams require evidence from an independent key system, not only cloud provider logs.<\/li>\n<li><strong>Why it fits<\/strong>: External key manager provides its own audit trails; Google Cloud provides Cloud KMS audit logs.<\/li>\n<li><strong>Scenario<\/strong>: A fintech must provide auditors with external HSM audit logs proving when keys were used and by what policy.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">7) Controlled key rotation and lifecycle processes<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Key rotation must follow a strict enterprise process with approvals and scheduled windows.<\/li>\n<li><strong>Why it fits<\/strong>: External KMS can implement enterprise rotation workflows while Google Cloud continues to reference the key.<\/li>\n<li><strong>Scenario<\/strong>: Rotation is performed in the external platform quarterly; Cloud KMS key versions map to external identifiers.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">8) Cloud migration where legacy key infrastructure must remain<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Existing apps and governance depend on an on-prem key manager; migrating keys is risky or disallowed.<\/li>\n<li><strong>Why it fits<\/strong>: You can migrate workloads to Google Cloud while keeping the key system unchanged.<\/li>\n<li><strong>Scenario<\/strong>: A company migrates analytics to Google Cloud but keeps encryption keys in its existing HSM estate due to compliance controls.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">9) Partner-managed HSM platform integration<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: You want external key custody but don\u2019t want to operate the HSMs yourself.<\/li>\n<li><strong>Why it fits<\/strong>: Many organizations use supported partners to host and operate external key systems (subject to partner offering and contracts).<\/li>\n<li><strong>Scenario<\/strong>: A SaaS vendor uses a partner-managed key service to satisfy customer contracts requiring externalized key custody.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">10) Granular policy enforcement based on context (time, app, environment)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Need to restrict key usage by environment or time windows.<\/li>\n<li><strong>Why it fits<\/strong>: External key manager can enforce policy conditions that are independent from cloud IAM (capabilities vary by platform).<\/li>\n<li><strong>Scenario<\/strong>: Production decryption is allowed only during business hours with on-call approval; dev environments use separate keys.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">11) Multi-project landing zone with centralized external keys<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Many projects need encryption but centralized governance.<\/li>\n<li><strong>Why it fits<\/strong>: Cloud KMS resources can be centralized (with IAM and shared services patterns), while key operations remain external.<\/li>\n<li><strong>Scenario<\/strong>: A platform team provides external keys from a security project; application projects reference them through supported CMEK patterns.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">12) Customer-facing \u201cbring your own key custody\u201d for SaaS<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: Enterprise customers demand encryption keys controlled outside the SaaS provider.<\/li>\n<li><strong>Why it fits<\/strong>: SaaS can run on Google Cloud while integrating external keys controlled by a separate key custodian function.<\/li>\n<li><strong>Scenario<\/strong>: A SaaS platform offers a premium tier where encryption keys are held in a dedicated external key manager under strict controls.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">6. Core Features<\/h2>\n\n\n\n<blockquote>\n<p>Note: Some features depend on your external key manager implementation or partner integration. Always verify exact compatibility and requirements in the official Cloud EKM \/ Cloud External Key Manager documentation and your external provider documentation.<\/p>\n<\/blockquote>\n\n\n\n<h3 class=\"wp-block-heading\">1) External key representation in Cloud KMS<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Lets you create Cloud KMS keys that represent externally managed keys.<\/li>\n<li><strong>Why it matters<\/strong>: You keep the standard Google Cloud integration model (CMEK via Cloud KMS) while shifting key custody externally.<\/li>\n<li><strong>Practical benefit<\/strong>: Minimal changes to Google Cloud services that already support CMEK; you manage external custody without rewriting every app.<\/li>\n<li><strong>Caveats<\/strong>: Not all Google Cloud services that support Cloud KMS necessarily support external keys\u2014verify per service.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">2) EKM connection configuration<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Defines how Cloud KMS connects to your external key manager (endpoint discovery and secure transport).<\/li>\n<li><strong>Why it matters<\/strong>: This is the trust boundary. Misconfiguration can cause outages or security gaps.<\/li>\n<li><strong>Practical benefit<\/strong>: Centralized, reusable connectivity and identity configuration for external key calls.<\/li>\n<li><strong>Caveats<\/strong>: You must design for HA and network reliability; Cloud KMS will depend on this connection at runtime.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">3) mTLS and endpoint authentication (typical pattern)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Uses mutual TLS to authenticate Cloud KMS to the external key manager and secure the channel.<\/li>\n<li><strong>Why it matters<\/strong>: Prevents unauthorized endpoints and reduces man-in-the-middle risk.<\/li>\n<li><strong>Practical benefit<\/strong>: Strong identity for Cloud KMS clients; can be integrated with certificate rotation.<\/li>\n<li><strong>Caveats<\/strong>: Certificate lifecycle management is operationally sensitive; expired certs can cause widespread encryption failures.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">4) IAM-based access control (Google Cloud side)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Controls which principals can administer connections, create external keys, and request cryptographic operations via Cloud KMS.<\/li>\n<li><strong>Why it matters<\/strong>: Prevents unauthorized key usage from within Google Cloud.<\/li>\n<li><strong>Practical benefit<\/strong>: Standard IAM, audit logs, and policy hierarchy (org\/folder\/project).<\/li>\n<li><strong>Caveats<\/strong>: IAM alone is not sufficient if you also need external approvals\u2014use your external key manager policies too.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">5) Audit logging in Google Cloud<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Records administrative actions and key usage requests handled by Cloud KMS in Cloud Audit Logs (subject to logging configuration).<\/li>\n<li><strong>Why it matters<\/strong>: Enables forensic investigation and compliance evidence.<\/li>\n<li><strong>Practical benefit<\/strong>: Central log routing to SIEM (Chronicle, Splunk, etc.).<\/li>\n<li><strong>Caveats<\/strong>: External key manager logs are separate; you need correlation strategy.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">6) Centralized key lifecycle management (metadata + references)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Lets you manage key names, versions, labels, and IAM in Cloud KMS even though key material is external.<\/li>\n<li><strong>Why it matters<\/strong>: Cloud services integrate with Cloud KMS resource names.<\/li>\n<li><strong>Practical benefit<\/strong>: Consistent IaC patterns and inventory within Google Cloud.<\/li>\n<li><strong>Caveats<\/strong>: True lifecycle (creation\/rotation\/destruction) depends on the external key manager and how you map external IDs to KMS key versions.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">7) Integration with CMEK-enabled Google Cloud services (where supported)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: Enables encryption control for managed services without custom application encryption.<\/li>\n<li><strong>Why it matters<\/strong>: You can keep managed service benefits while maintaining external key control.<\/li>\n<li><strong>Practical benefit<\/strong>: Encrypt data at rest with externally controlled keys.<\/li>\n<li><strong>Caveats<\/strong>: Support varies by product and sometimes by region; verify before designing.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">8) Reliability patterns via multiple endpoints \/ HA external platforms<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does<\/strong>: In enterprise setups, the external key manager is deployed in HA; Cloud KMS should be configured to reach resilient endpoints.<\/li>\n<li><strong>Why it matters<\/strong>: If external key manager is down, encryption\/decryption may fail across dependent services.<\/li>\n<li><strong>Practical benefit<\/strong>: Reduced downtime risk if designed correctly.<\/li>\n<li><strong>Caveats<\/strong>: Your external platform\u2019s HA design is your responsibility (or your partner\u2019s).<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">7. Architecture and How It Works<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">High-level architecture<\/h3>\n\n\n\n<p>Cloud External Key Manager works as a <strong>brokered cryptographic operation flow<\/strong>:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>A Google Cloud service (for example, a storage or data analytics service) needs to encrypt or decrypt using a CMEK.<\/li>\n<li>The service calls <strong>Cloud KMS<\/strong> with the key resource name.<\/li>\n<li>If the key is an <strong>external key<\/strong>, Cloud KMS uses the configured <strong>EKM connection<\/strong> to contact the external key manager endpoint.<\/li>\n<li>The external key manager evaluates policy (and may require approvals or enforce usage constraints) and performs or authorizes the cryptographic operation.<\/li>\n<li>Cloud KMS returns the result to the Google Cloud service, which completes the operation.<\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\">Control plane vs data plane<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Control plane<\/strong>: Creating keys, configuring EKM connections, IAM, policies, rotation procedures.<\/li>\n<li><strong>Data plane<\/strong>: Runtime cryptographic operations (encrypt\/decrypt\/sign\/unwrap operations depending on integration) that may be latency-sensitive and availability-critical.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Integrations with related services<\/h3>\n\n\n\n<p>Commonly involved Google Cloud services and features:\n&#8211; <strong>Cloud KMS<\/strong> (primary integration point)\n&#8211; <strong>Cloud IAM<\/strong> (authorization)\n&#8211; <strong>Cloud Audit Logs<\/strong> (auditing)\n&#8211; <strong>Cloud Monitoring\/Logging<\/strong> (operational visibility)\n&#8211; <strong>Service Directory \/ networking components<\/strong> (often used for endpoint discovery and private connectivity patterns\u2014verify the current recommended setup in official docs)\n&#8211; <strong>CMEK integrations<\/strong> across Google Cloud services (verify per service)<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Dependency services (typical)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Cloud KMS API enabled in the project(s)<\/li>\n<li>Networking services required to reach the external endpoint<\/li>\n<li>External key manager service (self-hosted or partner-hosted), plus its HA, monitoring, and audit pipeline<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Security\/authentication model (typical)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Google Cloud IAM authorizes who can use keys and administer connections.<\/li>\n<li>Cloud KMS authenticates to the external endpoint using <strong>TLS-based identity (often mTLS)<\/strong> and relies on endpoint trust configuration.<\/li>\n<li>External key manager authenticates\/authorizes the request based on its own access control and policy engine (implementation-specific).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Networking model (typical)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Connectivity is designed to ensure Cloud KMS can reach the external key manager endpoint reliably and securely.<\/li>\n<li>Many enterprises prefer <strong>private connectivity<\/strong> patterns rather than exposing external key endpoints to the public internet.<\/li>\n<li>Exact supported patterns (for example, how Service Directory is used, whether endpoints are within Google Cloud VPC, on-prem, or partner networks) should be confirmed in official docs and partner guidance.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Monitoring\/logging\/governance considerations<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Monitor availability and latency<\/strong> of external key operations end-to-end.<\/li>\n<li><strong>Alert on errors<\/strong> (timeouts, permission denials, TLS failures).<\/li>\n<li><strong>Track key usage<\/strong> in both Cloud Audit Logs and external key manager logs.<\/li>\n<li><strong>Change management<\/strong> for certificates, endpoints, and key policies is critical\u2014small changes can impact many dependent services.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Simple architecture diagram<\/h3>\n\n\n\n<pre><code class=\"language-mermaid\">flowchart LR\n  A[Google Cloud Service\\n(CMEK-enabled)] --&gt;|Encrypt\/Decrypt request| B[Cloud KMS]\n  B --&gt;|External key operation| C[Cloud External Key Manager\\n(EKM Connection)]\n  C --&gt;|mTLS \/ secure channel| D[External Key Manager\\n(Partner or Self-hosted)]\n  D --&gt;|Result| B\n  B --&gt;|Result| A\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Production-style architecture diagram<\/h3>\n\n\n\n<pre><code class=\"language-mermaid\">flowchart TB\n  subgraph GC[Google Cloud Project(s)]\n    SVC1[Workload \/ Managed Service\\nCMEK-enabled] --&gt; KMS[Cloud KMS\\nExternal Key]\n    IAM[IAM Policies\\nLeast privilege] --&gt; KMS\n    LOGS[Cloud Audit Logs\\n+ Log Router] --&gt; SIEM[(SIEM \/ SOC)]\n    KMS --&gt; LOGS\n  end\n\n  subgraph NET[Connectivity \/ Discovery Layer]\n    SD[Service Directory \/ Endpoint Discovery\\n(verify requirement in docs)]\n    KMS --&gt; SD\n  end\n\n  subgraph EXT[External Environment]\n    LB[HA Endpoint \/ Load Balancer]\n    EKM1[External Key Manager Node A]\n    EKM2[External Key Manager Node B]\n    HSM[(HSMs \/ Key Store)]\n    AUD[External KMS Audit Logs]\n    LB --&gt; EKM1\n    LB --&gt; EKM2\n    EKM1 --&gt; HSM\n    EKM2 --&gt; HSM\n    AUD --&gt; SIEM\n  end\n\n  SD --&gt; LB\n  KMS --&gt;|mTLS| LB\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">8. Prerequisites<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Google Cloud account and project<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>A Google Cloud billing account and at least one Google Cloud project.<\/li>\n<li>Ability to enable APIs and create IAM policies.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Permissions \/ IAM roles (minimum guidance)<\/h3>\n\n\n\n<p>You typically need:\n&#8211; Permissions to enable APIs (Project Owner or Editor, or a custom role)\n&#8211; Cloud KMS admin capabilities to create key rings\/keys and configure EKM connection resources<br\/>\n  (Often <code>roles\/cloudkms.admin<\/code> for setup; for day-to-day usage prefer narrower roles.)\n&#8211; IAM admin permissions if you will bind roles (<code>roles\/resourcemanager.projectIamAdmin<\/code> or appropriate delegated admin roles)<\/p>\n\n\n\n<p>For runtime usage by services:\n&#8211; Service agents \/ workload identities must be granted permissions to use the key (for example, crypto operation permissions in Cloud KMS). The exact role depends on the service and operation.<\/p>\n\n\n\n<blockquote>\n<p>Verify current recommended roles in Cloud KMS and Cloud EKM documentation. Roles and permissions can evolve.<\/p>\n<\/blockquote>\n\n\n\n<h3 class=\"wp-block-heading\">Billing requirements<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Cloud KMS usage is billable.<\/li>\n<li>External key manager costs are separate (partner contract or self-managed infrastructure).<\/li>\n<li>Network egress to external endpoints can be billable depending on topology.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">CLI \/ SDK \/ tools<\/h3>\n\n\n\n<p>Optional but recommended:\n&#8211; <a href=\"https:\/\/cloud.google.com\/sdk\/docs\/install\">Google Cloud SDK (<code>gcloud<\/code>)<\/a>\n&#8211; Access to Google Cloud Console\n&#8211; Your organization\u2019s certificate tooling (OpenSSL \/ internal PKI) if you manage mTLS certificates for EKM<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Region availability<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Cloud KMS is location-based.<\/li>\n<li>Cloud External Key Manager support can vary by location and by integration.<\/li>\n<li><strong>Verify<\/strong> supported locations and product availability in the official docs.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Quotas \/ limits<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Cloud KMS and related APIs enforce quotas (requests per minute, etc.).<\/li>\n<li>External key operations can introduce new bottlenecks (external endpoint throughput, TLS handshake costs, etc.).<\/li>\n<li><strong>Verify<\/strong> current quotas\/limits in Cloud KMS documentation and test load patterns.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Prerequisite services<\/h3>\n\n\n\n<p>Depending on architecture:\n&#8211; Cloud KMS API\n&#8211; Service Directory API and networking components if required by your connection model (verify in docs)\n&#8211; Cloud Logging \/ Monitoring (recommended)<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">9. Pricing \/ Cost<\/h2>\n\n\n\n<p>Cloud External Key Manager costs are a combination of <strong>Google Cloud costs<\/strong> and <strong>external provider costs<\/strong>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Pricing dimensions (Google Cloud side)<\/h3>\n\n\n\n<p>Cloud External Key Manager is implemented through Cloud KMS resources and operations, so cost commonly tracks:\n&#8211; <strong>Key\/version management<\/strong> (creating and storing key metadata in Cloud KMS)\n&#8211; <strong>Cryptographic operations<\/strong> (requests routed through Cloud KMS)\n&#8211; Potential charges related to <strong>external key operations<\/strong> (if priced differently from standard Cloud KMS operations\u2014verify the pricing SKU details)<\/p>\n\n\n\n<p>Because SKUs and pricing can change, use official sources:\n&#8211; Cloud KMS pricing: https:\/\/cloud.google.com\/kms\/pricing<br\/>\n&#8211; Google Cloud Pricing Calculator: https:\/\/cloud.google.com\/products\/calculator<\/p>\n\n\n\n<blockquote>\n<p>Do not rely on blog-post numbers. Always confirm current SKUs and rates in the pricing page for your currency and region.<\/p>\n<\/blockquote>\n\n\n\n<h3 class=\"wp-block-heading\">Free tier<\/h3>\n\n\n\n<p>Cloud KMS sometimes has limited free usage tiers for certain operations in some contexts, but this varies. For Cloud External Key Manager scenarios, <strong>assume billable usage<\/strong> and confirm free tier applicability (if any) on the official pricing page.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Major cost drivers<\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li>\n<p><strong>Number of cryptographic operations<\/strong>\n   &#8211; Encrypt\/decrypt requests (and any rewrap, sign, or unwrap operations depending on service)\n   &#8211; High-throughput services can generate very large numbers of KMS operations.<\/p>\n<\/li>\n<li>\n<p><strong>External key manager costs<\/strong>\n   &#8211; Partner subscription, HSM costs, licensing\n   &#8211; If self-hosted: compute, storage, HSM appliances, data center costs, support<\/p>\n<\/li>\n<li>\n<p><strong>Network and connectivity<\/strong>\n   &#8211; Latency and egress charges if traffic crosses regions or leaves Google Cloud\n   &#8211; Private connectivity services (if used) can have their own costs\n   &#8211; Load balancers, interconnects, VPN, or partner connectivity<\/p>\n<\/li>\n<li>\n<p><strong>Operational overhead<\/strong>\n   &#8211; Monitoring, logging export, SIEM ingestion costs\n   &#8211; Certificate management and rotation operations\n   &#8211; SRE time and incident response readiness<\/p>\n<\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\">Hidden or indirect costs<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Outage blast radius<\/strong>: If your external key manager is unavailable, multiple Google Cloud services may fail encryption\/decryption. Designing HA reduces risk but adds cost.<\/li>\n<li><strong>Performance headroom<\/strong>: External crypto adds network hops; you may need larger external clusters.<\/li>\n<li><strong>Audit retention<\/strong>: Storing high-volume audit logs can be expensive at scale.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Network\/data transfer implications<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Calls from Cloud KMS to an external endpoint can incur:<\/li>\n<li><strong>Egress charges<\/strong> if traffic leaves Google Cloud or crosses billing boundaries<\/li>\n<li><strong>Inter-region charges<\/strong> if KMS location and endpoint location are mismatched<\/li>\n<li>Minimize cross-region dependency where possible; align Cloud KMS locations with external endpoint placement.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">How to optimize cost<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Right-size KMS operations<\/strong>: Avoid excessive per-object KMS calls; use service-native encryption patterns efficiently (many services cache data keys; verify for each service).<\/li>\n<li><strong>Limit key usage scope<\/strong>: Separate dev\/test keys and reduce production-scale testing traffic on external keys.<\/li>\n<li><strong>Design for locality<\/strong>: Place external endpoints to minimize latency and network costs.<\/li>\n<li><strong>Use logging filters and routing<\/strong>: Export only necessary KMS logs to expensive destinations; keep full fidelity where required for compliance.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Example low-cost starter estimate (conceptual)<\/h3>\n\n\n\n<p>A realistic starter test often includes:\n&#8211; A small number of Cloud KMS external key resources\n&#8211; Low volume of cryptographic operations (integration testing only)\n&#8211; Minimal log export and no heavy SIEM ingestion<\/p>\n\n\n\n<p>Because prices vary by region and SKU, <strong>build an estimate in the Pricing Calculator<\/strong> using:\n&#8211; Expected monthly crypto operations\n&#8211; Any additional KMS resource costs shown in pricing\n&#8211; Logging export volume (GB\/month)<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Example production cost considerations (conceptual)<\/h3>\n\n\n\n<p>In production, you should model:\n&#8211; Peak operations per second and total monthly operations\n&#8211; External key manager HA cluster sizing and licensing\n&#8211; Network egress and connectivity services\n&#8211; Monitoring + log retention + SIEM ingestion\n&#8211; Incident response readiness (on-call, runbooks, testing)<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">10. Step-by-Step Hands-On Tutorial<\/h2>\n\n\n\n<p>This lab focuses on the <strong>Google Cloud side setup<\/strong> for Cloud External Key Manager: enabling APIs, creating a Cloud KMS key ring in a chosen location, and creating the foundational objects you need before integrating a real external key manager endpoint.<\/p>\n\n\n\n<p>Because a fully functional Cloud External Key Manager deployment requires an <strong>EKM-compatible external key manager<\/strong> (partner or self-managed) and secure connectivity, this lab is designed to be <strong>safe, low-cost, and executable<\/strong> without requiring you to procure HSMs or partner services. Where external endpoint details are required, you will configure placeholders and validate resource creation and IAM\u2014then you\u2019ll learn how to proceed for real integration.<\/p>\n\n\n\n<blockquote>\n<p>If your organization already has an EKM-compatible key manager, you can adapt the \u201cplaceholder\u201d values to your real endpoints and certificates by following the official documentation and your vendor\u2019s integration guide.<\/p>\n<\/blockquote>\n\n\n\n<h3 class=\"wp-block-heading\">Objective<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Enable required APIs.<\/li>\n<li>Create a Cloud KMS key ring in a chosen location.<\/li>\n<li>Create the base configuration objects typically required for Cloud External Key Manager integration.<\/li>\n<li>Apply least-privilege IAM for admins vs users.<\/li>\n<li>Validate that the configuration objects exist and that audit logs capture admin activity.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Lab Overview<\/h3>\n\n\n\n<p>You will:\n1. Create\/select a Google Cloud project and set environment variables.\n2. Enable Cloud KMS (and related APIs if needed).\n3. Create a key ring in a location you choose.\n4. Create an external-key placeholder configuration (metadata-side).\n5. Configure IAM bindings for admin and crypto usage.\n6. Validate resources and check audit logs.\n7. Clean up.<\/p>\n\n\n\n<blockquote>\n<p>The exact Console screens and <code>gcloud<\/code> commands for Cloud EKM resources can change. Where there is ambiguity, this lab uses <strong>Console-first<\/strong> steps and points you to the authoritative docs to confirm the latest CLI syntax.<\/p>\n<\/blockquote>\n\n\n\n<h3 class=\"wp-block-heading\">Step 1: Create\/select a project and set up <code>gcloud<\/code><\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li>In the Google Cloud Console, select or create a project:\n   &#8211; <strong>Console<\/strong> \u2192 project selector \u2192 <strong>New Project<\/strong><\/li>\n<li>Ensure billing is enabled for the project.<\/li>\n<\/ol>\n\n\n\n<p>Set your CLI context (optional but recommended):<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud auth login\ngcloud config set project YOUR_PROJECT_ID\ngcloud config set compute\/region us-central1\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome<\/strong>\n&#8211; Your project is selected in Console.\n&#8211; <code>gcloud config get-value project<\/code> returns your project ID.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 2: Enable APIs<\/h3>\n\n\n\n<p>Enable Cloud KMS:<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud services enable cloudkms.googleapis.com\n<\/code><\/pre>\n\n\n\n<p>Depending on your intended architecture, you may also need Service Directory and other APIs for endpoint discovery and connectivity. Only enable what you need:<\/p>\n\n\n\n<pre><code class=\"language-bash\"># Enable if your chosen Cloud EKM setup requires it (verify in official docs)\ngcloud services enable servicedirectory.googleapis.com\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome<\/strong>\n&#8211; APIs show as enabled in <strong>APIs &amp; Services<\/strong> \u2192 <strong>Enabled APIs &amp; services<\/strong>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 3: Choose a Cloud KMS location and create a key ring<\/h3>\n\n\n\n<p>Pick a Cloud KMS location that aligns with:\n&#8211; Where your workloads run\n&#8211; Where your external key endpoint is reachable with acceptable latency\n&#8211; Your compliance requirements<\/p>\n\n\n\n<p>Create a key ring:<\/p>\n\n\n\n<pre><code class=\"language-bash\">export KMS_LOCATION=\"us-central1\"\nexport KEYRING_NAME=\"ekm-lab-ring\"\n\ngcloud kms keyrings create \"${KEYRING_NAME}\" \\\n  --location=\"${KMS_LOCATION}\"\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome<\/strong>\n&#8211; Key ring is created successfully.\n&#8211; You can see it in <strong>Security<\/strong> \u2192 <strong>Key Management<\/strong> (Cloud KMS) in the Console under the selected location.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 4: Create an external key placeholder (Cloud KMS metadata)<\/h3>\n\n\n\n<p>In Cloud KMS, external keys are represented as keys with an <strong>external protection level<\/strong>. The exact CLI flags and steps can differ over time, so use the Console to avoid syntax drift.<\/p>\n\n\n\n<p><strong>Console steps<\/strong>\n1. Go to <strong>Security<\/strong> \u2192 <strong>Key Management<\/strong>.\n2. Select your location (for example, <code>us-central1<\/code>) and your key ring (<code>ekm-lab-ring<\/code>).\n3. Click <strong>Create key<\/strong>.\n4. Set:\n   &#8211; <strong>Name<\/strong>: <code>ekm-lab-key<\/code>\n   &#8211; <strong>Protection level<\/strong>: choose <strong>External<\/strong> (or <strong>External key manager<\/strong>, depending on UI wording)\n5. Proceed to create the key.<\/p>\n\n\n\n<p>If the Console asks you to configure:\n&#8211; <strong>EKM connection<\/strong>: you may need to create one first (see next step), or you may be allowed to create a key without binding immediately (depends on current UI and requirements).\n&#8211; <strong>External key URI \/ identifier<\/strong>: enter a placeholder only if the UI allows it; otherwise, create the connection first and follow the official workflow.<\/p>\n\n\n\n<p><strong>Expected outcome<\/strong>\n&#8211; A Cloud KMS key resource exists as an external key placeholder (metadata in Cloud KMS).\n&#8211; You can list keys in the key ring and see <code>ekm-lab-key<\/code>.<\/p>\n\n\n\n<p>Optional validation (list keys):<\/p>\n\n\n\n<pre><code class=\"language-bash\">gcloud kms keys list \\\n  --location=\"${KMS_LOCATION}\" \\\n  --keyring=\"${KEYRING_NAME}\"\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Step 5: Create an EKM connection (foundation for real integrations)<\/h3>\n\n\n\n<p>A functioning Cloud External Key Manager setup requires an <strong>EKM connection<\/strong> that tells Cloud KMS where and how to reach your external key manager.<\/p>\n\n\n\n<p>Because EKM connection configuration is sensitive to:\n&#8211; Endpoint discovery model\n&#8211; TLS\/mTLS configuration\n&#8211; Vendor compatibility<\/p>\n\n\n\n<p>\u2026use official docs and vendor guidance for the exact required fields.<\/p>\n\n\n\n<p><strong>Console-first approach (recommended)<\/strong>\n1. In <strong>Security<\/strong> \u2192 <strong>Key Management<\/strong>, find <strong>External key manager connections<\/strong> (wording may vary).\n2. Click <strong>Create connection<\/strong>.\n3. Select the same location as your key ring (for example, <code>us-central1<\/code>).\n4. For endpoint\/discovery configuration:\n   &#8211; If the UI requires Service Directory service, create\/select one.\n   &#8211; If the UI asks for certificates, follow your PKI process.<\/p>\n\n\n\n<p>For a low-cost lab without a real external endpoint, you may be able to create the connection object without a reachable endpoint (behavior varies). If the UI blocks creation without a reachable endpoint, stop here and proceed with the rest of the lab (IAM + logging), then plan integration once you have an EKM-compatible endpoint.<\/p>\n\n\n\n<p><strong>Expected outcome<\/strong>\n&#8211; An EKM connection object exists in the chosen location, or you have identified the exact missing requirements (endpoint, certs, Service Directory) needed to proceed.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 6: Apply IAM for admins vs users (least privilege)<\/h3>\n\n\n\n<p>A practical Cloud External Key Manager design separates:\n&#8211; <strong>Key administrators<\/strong> (create keys, set IAM, configure connections)\n&#8211; <strong>Key users<\/strong> (perform crypto operations \/ use CMEK from services)<\/p>\n\n\n\n<p>Create two Google Groups (recommended in enterprises), or use two test users if available:\n&#8211; <code>gcp-ekm-admins@YOUR_DOMAIN<\/code>\n&#8211; <code>gcp-ekm-users@YOUR_DOMAIN<\/code><\/p>\n\n\n\n<p>Then bind roles at the <strong>key ring<\/strong> or <strong>key<\/strong> level where possible.<\/p>\n\n\n\n<p>Examples (project-level shown for simplicity; prefer resource-level bindings in production):<\/p>\n\n\n\n<pre><code class=\"language-bash\"># Replace with your group or user principal\nexport EKM_ADMINS_PRINCIPAL=\"group:gcp-ekm-admins@YOUR_DOMAIN\"\nexport EKM_USERS_PRINCIPAL=\"group:gcp-ekm-users@YOUR_DOMAIN\"\n\n# Admins: manage Cloud KMS resources (broad)\ngcloud projects add-iam-policy-binding \"$(gcloud config get-value project)\" \\\n  --member=\"${EKM_ADMINS_PRINCIPAL}\" \\\n  --role=\"roles\/cloudkms.admin\"\n\n# Users: crypto operations (narrower)\ngcloud projects add-iam-policy-binding \"$(gcloud config get-value project)\" \\\n  --member=\"${EKM_USERS_PRINCIPAL}\" \\\n  --role=\"roles\/cloudkms.cryptoKeyEncrypterDecrypter\"\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome<\/strong>\n&#8211; Admins can create\/configure KMS resources.\n&#8211; Users can use permitted keys for encryption\/decryption (subject to key-level IAM and external policy).<\/p>\n\n\n\n<blockquote>\n<p>In production, scope <code>roles\/cloudkms.cryptoKeyEncrypterDecrypter<\/code> to specific keys, not entire projects.<\/p>\n<\/blockquote>\n\n\n\n<h3 class=\"wp-block-heading\">Step 7: Validate using audit logs<\/h3>\n\n\n\n<p>Cloud KMS admin actions are logged in Cloud Audit Logs.<\/p>\n\n\n\n<p><strong>Console<\/strong>\n1. Go to <strong>Logging<\/strong> \u2192 <strong>Logs Explorer<\/strong>\n2. Use a query like:<\/p>\n\n\n\n<pre><code class=\"language-text\">resource.type=\"cloudkms_cryptokey\"\nOR resource.type=\"cloudkms_keyring\"\n<\/code><\/pre>\n\n\n\n<p>You can also filter by method names if visible in your environment. (Method name fields can vary; adjust based on observed logs.)<\/p>\n\n\n\n<p><strong>Expected outcome<\/strong>\n&#8211; You can find log entries showing key ring creation, key creation, IAM policy changes, or connection attempts.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Validation<\/h3>\n\n\n\n<p>At the end of the lab, you should be able to confirm:\n&#8211; Cloud KMS API is enabled.\n&#8211; A key ring exists in your chosen location.\n&#8211; An external key placeholder exists (or you documented why it couldn\u2019t be created yet).\n&#8211; IAM bindings reflect admin vs user separation.\n&#8211; Audit logs show the administrative activities.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Troubleshooting<\/h3>\n\n\n\n<p><strong>Issue: \u201cAPI not enabled\u201d<\/strong>\n&#8211; Fix: Enable <code>cloudkms.googleapis.com<\/code> and retry.<\/p>\n\n\n\n<p><strong>Issue: \u201cPermission denied\u201d creating key ring or key<\/strong>\n&#8211; Fix: Ensure your principal has <code>roles\/cloudkms.admin<\/code> (or specific permissions) and that Org Policies do not block KMS usage.<\/p>\n\n\n\n<p><strong>Issue: Location mismatch<\/strong>\n&#8211; Fix: Ensure key ring, external key, and EKM connection are created in compatible locations as required by Cloud KMS.<\/p>\n\n\n\n<p><strong>Issue: EKM connection creation requires endpoint\/certificates<\/strong>\n&#8211; Fix: This is expected for real integrations. Follow official Cloud EKM docs and your vendor guide for:\n  &#8211; Endpoint discovery (often via Service Directory or a supported pattern)\n  &#8211; mTLS certificate requirements\n  &#8211; Network path and firewall rules<\/p>\n\n\n\n<p><strong>Issue: CMEK-enabled service can\u2019t use the key<\/strong>\n&#8211; Fix: Confirm:\n  &#8211; The service supports <strong>external keys<\/strong> (not just Cloud KMS keys).\n  &#8211; The service agent has crypto permissions on the key.\n  &#8211; The external key manager policy allows the operation.\n  &#8211; The EKM connection is healthy and reachable.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Cleanup<\/h3>\n\n\n\n<p>To avoid ongoing costs and reduce clutter:<\/p>\n\n\n\n<pre><code class=\"language-bash\"># Delete keys inside the key ring first (if created via CLI; via Console is also fine)\n# Listing keys:\ngcloud kms keys list --location=\"${KMS_LOCATION}\" --keyring=\"${KEYRING_NAME}\"\n\n# Delete key ring (will require keys to be removed\/disabled per KMS rules)\ngcloud kms keyrings delete \"${KEYRING_NAME}\" --location=\"${KMS_LOCATION}\"\n<\/code><\/pre>\n\n\n\n<p>Also consider:\n&#8211; Remove IAM bindings you added for test groups\/users.\n&#8211; Disable APIs if the project is purely for lab work.<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">11. Best Practices<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Architecture best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Design for external dependency failure<\/strong>: Treat the external key manager as a critical dependency. Model failure modes: timeouts, partial outages, certificate expiry, network partition.<\/li>\n<li><strong>Align locations<\/strong>: Place Cloud KMS keys and external endpoints to minimize cross-region calls and reduce latency and egress costs.<\/li>\n<li><strong>Use dedicated projects<\/strong>: Centralize Cloud KMS resources and EKM connections in a security or shared services project when appropriate, and grant access via IAM to service projects.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">IAM\/security best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Least privilege<\/strong>: Use narrow roles (crypto user roles) scoped at the key level wherever possible.<\/li>\n<li><strong>Separate admin duties<\/strong>:<\/li>\n<li>External key manager admins (external platform)<\/li>\n<li>Google Cloud KMS admins (resource configuration)<\/li>\n<li>Workload identities (crypto operations)<\/li>\n<li><strong>Protect IAM policy changes<\/strong>: Use organization policy, approvals, and monitoring for IAM changes affecting key access.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Cost best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Estimate crypto operation volume<\/strong> early. KMS operations can scale fast in high-throughput systems.<\/li>\n<li><strong>Avoid dev\/test traffic using production external keys<\/strong>.<\/li>\n<li><strong>Tune log routing<\/strong>: Keep necessary audit logs, but avoid exporting massive volumes unnecessarily to expensive destinations.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Performance best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Measure latency end-to-end<\/strong>: Service \u2192 Cloud KMS \u2192 external endpoint \u2192 back.<\/li>\n<li><strong>Scale external endpoints<\/strong>: Ensure your external key manager can handle peak request rates with headroom.<\/li>\n<li><strong>Reduce handshake overhead<\/strong>: Use stable TLS configurations and avoid frequent certificate disruptions.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Reliability best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>HA external key manager<\/strong>: Multi-node, redundant networking, tested failover.<\/li>\n<li><strong>Certificate rotation runbooks<\/strong>: Automate and test rotation; monitor expiry.<\/li>\n<li><strong>Change control<\/strong>: Treat connection and cert changes like production changes with rollback plans.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Operations best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Monitoring and alerting<\/strong>:<\/li>\n<li>Error rates for crypto operations<\/li>\n<li>Latency and timeouts<\/li>\n<li>External endpoint health<\/li>\n<li><strong>Correlate logs<\/strong>:<\/li>\n<li>Cloud Audit Logs (KMS request events)<\/li>\n<li>External KMS audit logs<\/li>\n<li>Service logs for CMEK failures<\/li>\n<li><strong>Runbooks<\/strong>:<\/li>\n<li>\u201cExternal key manager down\u201d<\/li>\n<li>\u201cCertificate expired\u201d<\/li>\n<li>\u201cPermission denied due to external policy\u201d<\/li>\n<li>\u201cLatency spike causing application timeouts\u201d<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Governance\/tagging\/naming best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Use labels consistently<\/strong> on KMS resources (environment, data classification, owner, cost center).<\/li>\n<li><strong>Standardize naming<\/strong>: Include environment and domain (for example, <code>ekm-prod-bq-pii-key<\/code>).<\/li>\n<li><strong>Document ownership<\/strong>: Each key should have a clear business owner and operational owner.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">12. Security Considerations<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Identity and access model<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Google Cloud IAM<\/strong> controls who can:<\/li>\n<li>Administer KMS resources (key rings, keys, IAM policies)<\/li>\n<li>Use keys for crypto operations<\/li>\n<li><strong>External key manager policies<\/strong> control:<\/li>\n<li>Whether the operation is allowed<\/li>\n<li>Any approvals or constraints<\/li>\n<li>External auditing and enforcement<\/li>\n<\/ul>\n\n\n\n<p>A secure design assumes <strong>both<\/strong> layers are necessary:\n&#8211; IAM prevents unauthorized use from Google Cloud.\n&#8211; External policy prevents unauthorized use even if cloud permissions are misconfigured.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Encryption<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Data is encrypted by Google Cloud services using envelope encryption patterns; Cloud KMS (and external key manager) are used for key operations rather than bulk data encryption.<\/li>\n<li>With Cloud External Key Manager, keys are externally controlled; actual encryption semantics depend on the Google Cloud service and integration pattern.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Network exposure<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Prefer private, controlled network paths to external endpoints.<\/li>\n<li>Avoid exposing external key manager endpoints publicly unless there is a strong justification and compensating controls.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Secrets handling<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Treat TLS private keys, client certificates, and any external platform credentials as secrets.<\/li>\n<li>Store secrets in a dedicated secret management system and restrict access.<\/li>\n<li>Rotate and revoke promptly on incident.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Audit\/logging<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Enable and retain:<\/li>\n<li>Cloud KMS audit logs<\/li>\n<li>External key manager audit logs<\/li>\n<li>Correlate requests using timestamps, key IDs, and request metadata where available.<\/li>\n<li>Protect logs from tampering and ensure retention meets compliance requirements.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Compliance considerations<\/h3>\n\n\n\n<p>Cloud External Key Manager is often used for compliance, but it does not automatically make a workload compliant. You still need:\n&#8211; Documented controls and evidence\n&#8211; Key management procedures\n&#8211; Access reviews\n&#8211; Incident response procedures\n&#8211; Vendor risk management for partner-hosted key systems<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Common security mistakes<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Granting <code>roles\/cloudkms.admin<\/code> too broadly.<\/li>\n<li>Allowing workloads to use overly broad crypto permissions across many keys.<\/li>\n<li>Ignoring certificate expiry and rotation planning.<\/li>\n<li>Designing a single external endpoint with no redundancy.<\/li>\n<li>Failing to test outage scenarios (resulting in unexpected data unavailability).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Secure deployment recommendations<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Start with a threat model: identify who must not be able to decrypt data.<\/li>\n<li>Implement separation of duties:<\/li>\n<li>Cloud platform team cannot change external key policy.<\/li>\n<li>Security key custodians cannot deploy workloads.<\/li>\n<li>Use policy-as-code and approvals for IAM and connection changes.<\/li>\n<li>Establish SLOs for key operations and external endpoint availability.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">13. Limitations and Gotchas<\/h2>\n\n\n\n<blockquote>\n<p>Some items below vary by Google Cloud service integration and external key manager vendor. Verify the latest constraints in official documentation and vendor guides.<\/p>\n<\/blockquote>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Service support varies<\/strong>: Not every CMEK-enabled Google Cloud service supports external keys.<\/li>\n<li><strong>External dependency risk<\/strong>: If the external key manager is unavailable, encryption\/decryption operations may fail, causing application or service disruption.<\/li>\n<li><strong>Latency sensitivity<\/strong>: External calls add network hops; performance may be impacted.<\/li>\n<li><strong>Certificate lifecycle complexity<\/strong>: mTLS certificate expiry or mis-rotation can cause widespread failures.<\/li>\n<li><strong>Network design complexity<\/strong>: Private connectivity and endpoint discovery can be non-trivial in hybrid designs.<\/li>\n<li><strong>Operational burden<\/strong>: You now operate and monitor two systems: Google Cloud KMS metadata + external key platform.<\/li>\n<li><strong>Quota\/throughput constraints<\/strong>: Cloud KMS quotas and external platform throughput both matter; bottlenecks can appear in either place.<\/li>\n<li><strong>Change management<\/strong>: External key policy changes can immediately impact Google Cloud workloads.<\/li>\n<li><strong>Migration complexity<\/strong>: Moving from Cloud KMS native keys to external keys (or vice versa) often requires careful planning and may not be seamless for all services.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">14. Comparison with Alternatives<\/h2>\n\n\n\n<p>Cloud External Key Manager is one option in a broader key management decision space.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Comparison table<\/h3>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>Option<\/th>\n<th>Best For<\/th>\n<th>Strengths<\/th>\n<th>Weaknesses<\/th>\n<th>When to Choose<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><strong>Cloud External Key Manager (Cloud EKM)<\/strong><\/td>\n<td>Strict key custody\/separation-of-duties requirements<\/td>\n<td>Keys controlled outside Google Cloud; external governance\/audit; integrates with supported CMEK services<\/td>\n<td>More complex; external dependency; potential latency; requires external platform<\/td>\n<td>You must keep keys external for compliance or governance<\/td>\n<\/tr>\n<tr>\n<td><strong>Cloud KMS (software keys)<\/strong><\/td>\n<td>General-purpose CMEK and app crypto needs<\/td>\n<td>Simple; fully managed; strong IAM and audit logs; low operational overhead<\/td>\n<td>Key custody is within Google Cloud<\/td>\n<td>Default choice for most teams<\/td>\n<\/tr>\n<tr>\n<td><strong>Cloud HSM (via Cloud KMS)<\/strong><\/td>\n<td>HSM-backed keys managed by Google<\/td>\n<td>Managed HSM-backed keys; reduces operational burden<\/td>\n<td>Still hosted\/controlled within Google Cloud boundary<\/td>\n<td>You need HSM assurance but not external custody<\/td>\n<\/tr>\n<tr>\n<td><strong>Customer-supplied encryption keys (CSEK)<\/strong><\/td>\n<td>Niche scenarios where you supply raw keys to services<\/td>\n<td>You supply key material for certain services<\/td>\n<td>Operationally risky; limited support; can be complex and has sharp edges<\/td>\n<td>Only when you fully understand constraints and need this specific pattern<\/td>\n<\/tr>\n<tr>\n<td><strong>Self-managed application-layer encryption (e.g., envelope encryption in app using external KMS)<\/strong><\/td>\n<td>Custom crypto workflows; portable across platforms<\/td>\n<td>Maximum control; not tied to CMEK support<\/td>\n<td>High engineering complexity; risk of mistakes; operational overhead<\/td>\n<td>When managed service CMEK support is insufficient or you need custom crypto logic<\/td>\n<\/tr>\n<tr>\n<td><strong>AWS KMS External Key Store (XKS)<\/strong><\/td>\n<td>Multi-cloud strategy; AWS workloads requiring external custody<\/td>\n<td>Similar external key custody model in AWS<\/td>\n<td>Not Google Cloud; separate ecosystem<\/td>\n<td>Choose for AWS workloads needing external custody<\/td>\n<\/tr>\n<tr>\n<td><strong>Azure Key Vault Managed HSM \/ external key custody patterns<\/strong><\/td>\n<td>Azure-centric organizations<\/td>\n<td>Tight integration with Azure<\/td>\n<td>Not Google Cloud; different constraints<\/td>\n<td>Choose for Azure workloads<\/td>\n<\/tr>\n<tr>\n<td><strong>HashiCorp Vault (Transit) \/ self-managed KMS<\/strong><\/td>\n<td>App crypto and centralized secrets\/key ops<\/td>\n<td>Flexible; multi-cloud; strong policy<\/td>\n<td>Not natively CMEK for Google managed services; requires app integration<\/td>\n<td>When you need app-level crypto portability more than CMEK integration<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">15. Real-World Example<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Enterprise example: Regulated analytics platform with external key custody<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: A financial institution wants to use Google Cloud analytics services but must keep encryption keys under control of its internal key management team using an HSM-backed enterprise KMS.<\/li>\n<li><strong>Proposed architecture<\/strong>:<\/li>\n<li>Data platform runs in Google Cloud projects segmented by environment.<\/li>\n<li>Cloud KMS contains external key resources in a security project.<\/li>\n<li>Cloud External Key Manager is configured to reach the bank\u2019s external key manager through a private, controlled network path.<\/li>\n<li>CMEK-enabled services reference the external keys for encryption at rest.<\/li>\n<li>Logs from Cloud Audit Logs and external KMS audit logs are exported to a SIEM for correlation.<\/li>\n<li><strong>Why this service was chosen<\/strong>:<\/li>\n<li>Satisfies key custody and separation-of-duties requirements without rewriting every workload for application-layer encryption.<\/li>\n<li><strong>Expected outcomes<\/strong>:<\/li>\n<li>Compliance evidence for key custody.<\/li>\n<li>Centralized audit trail across cloud and external key platform.<\/li>\n<li>Consistent key governance across environments.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Startup\/small-team example: Customer-required external key control for a premium tier<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem<\/strong>: A SaaS startup sells to enterprises that require externally controlled encryption keys. The startup wants to stay on Google Cloud managed services but needs an external key custody option for a premium plan.<\/li>\n<li><strong>Proposed architecture<\/strong>:<\/li>\n<li>Default tier uses Cloud KMS software keys for simplicity.<\/li>\n<li>Premium tier uses Cloud External Key Manager with a partner-managed external key service (contracted), limiting operational burden.<\/li>\n<li>Premium tenant data is stored in dedicated projects\/buckets\/datasets with separate keys and strict IAM.<\/li>\n<li><strong>Why this service was chosen<\/strong>:<\/li>\n<li>Meets enterprise customer contract requirements while keeping the product on managed Google Cloud services.<\/li>\n<li><strong>Expected outcomes<\/strong>:<\/li>\n<li>Higher enterprise adoption.<\/li>\n<li>Clear separation between standard and premium security controls.<\/li>\n<li>Reduced need for custom application cryptography.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">16. FAQ<\/h2>\n\n\n\n<p>1) <strong>Is Cloud External Key Manager a separate product from Cloud KMS?<\/strong><br\/>\nCloud External Key Manager is commonly implemented as <strong>Cloud EKM<\/strong> within <strong>Cloud KMS<\/strong>. You manage external-key resources using Cloud KMS concepts (key rings\/keys) plus EKM connection configuration.<\/p>\n\n\n\n<p>2) <strong>Does Google Cloud ever store my external key material?<\/strong><br\/>\nThe goal of Cloud External Key Manager is to keep key material external. Exact behaviors depend on the integration model. Verify details in official Cloud EKM documentation and your external provider architecture.<\/p>\n\n\n\n<p>3) <strong>Do all CMEK-supported services support external keys?<\/strong><br\/>\nNo. Support varies by Google Cloud service, and sometimes by region and feature. Always verify external key support for the specific product you want to encrypt.<\/p>\n\n\n\n<p>4) <strong>What happens if my external key manager goes down?<\/strong><br\/>\nServices that rely on that key may fail encryption\/decryption operations, potentially causing outages. You must design HA, monitoring, and incident procedures for the external key manager.<\/p>\n\n\n\n<p>5) <strong>Will using external keys increase latency?<\/strong><br\/>\nOften yes, because operations involve an additional network hop to an external endpoint. Measure end-to-end latency and plan capacity accordingly.<\/p>\n\n\n\n<p>6) <strong>Can I use external keys for disk encryption on compute workloads?<\/strong><br\/>\nSome compute and storage services support CMEK and may support external keys, but not universally. Verify per product and per region.<\/p>\n\n\n\n<p>7) <strong>How do I rotate external keys?<\/strong><br\/>\nRotation is primarily performed in the external key manager. In Cloud KMS, you may map new external key identifiers to new key versions depending on the supported workflow. Verify recommended rotation patterns in official docs.<\/p>\n\n\n\n<p>8) <strong>How do IAM permissions work with external keys?<\/strong><br\/>\nIAM controls which principals can request operations through Cloud KMS. The external key manager enforces its own policies too. Both must allow the operation.<\/p>\n\n\n\n<p>9) <strong>Can I restrict who can change the EKM connection?<\/strong><br\/>\nYes\u2014use IAM with least privilege and consider organization policies and change approval processes to control admin actions.<\/p>\n\n\n\n<p>10) <strong>Do I still get Cloud Audit Logs when using external keys?<\/strong><br\/>\nYou should still see Cloud KMS-related audit logs for requests and admin actions, subject to logging configuration. You also need external key manager audit logs for full visibility.<\/p>\n\n\n\n<p>11) <strong>Is Cloud External Key Manager the same as client-side encryption?<\/strong><br\/>\nNo. Client-side encryption means your application encrypts data before sending it to Google Cloud. Cloud External Key Manager is still server-side encryption integrated with Cloud KMS, but the key is external.<\/p>\n\n\n\n<p>12) <strong>Can I use Cloud External Key Manager for application signing keys?<\/strong><br\/>\nPossibly, depending on the external key manager capabilities and Cloud KMS integration features. Verify supported algorithms and operations in the official docs and vendor documentation.<\/p>\n\n\n\n<p>13) <strong>Do I need a partner to use Cloud External Key Manager?<\/strong><br\/>\nNot necessarily. Some organizations use supported partners; others deploy their own EKM-compatible key manager. Compatibility requirements are strict\u2014verify the supported models.<\/p>\n\n\n\n<p>14) <strong>How do I monitor failures?<\/strong><br\/>\nMonitor:\n&#8211; Cloud KMS error rates and latency (Cloud Monitoring metrics, if available)\n&#8211; External endpoint health\n&#8211; CMEK failure logs in the dependent Google Cloud service\n&#8211; External KMS audit and operational logs<\/p>\n\n\n\n<p>15) <strong>What\u2019s the simplest way to start?<\/strong><br\/>\nStart by:\n&#8211; Inventorying which Google Cloud services you need to encrypt\n&#8211; Verifying external key support for each\n&#8211; Building a proof of concept with one service and one key\n&#8211; Adding HA, monitoring, and rotation only after basic functionality is proven<\/p>\n\n\n\n<p>16) <strong>Can I combine Cloud HSM and Cloud External Key Manager?<\/strong><br\/>\nThey solve different problems. Cloud HSM keeps keys in Google-managed HSMs; Cloud External Key Manager keeps keys external. Choose based on custody and governance requirements.<\/p>\n\n\n\n<p>17) <strong>Does disabling a key externally instantly block access to data?<\/strong><br\/>\nIt can block future decrypt operations, which may prevent access. This can be useful for incident response but can also cause outages. Plan and test carefully.<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">17. Top Online Resources to Learn Cloud External Key Manager<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>Resource Type<\/th>\n<th>Name<\/th>\n<th>Why It Is Useful<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Official documentation<\/td>\n<td>Cloud KMS documentation<\/td>\n<td>Primary reference for key rings, keys, IAM, audit logs, and CMEK integration concepts: https:\/\/cloud.google.com\/kms\/docs<\/td>\n<\/tr>\n<tr>\n<td>Official documentation<\/td>\n<td>Cloud EKM \/ external keys documentation<\/td>\n<td>Authoritative setup and concepts for Cloud External Key Manager (Cloud EKM). Verify the latest entry point from Cloud KMS docs or search \u201cCloud EKM Google Cloud\u201d. Start here: https:\/\/cloud.google.com\/kms\/docs\/ekm (Verify URL in case it changes)<\/td>\n<\/tr>\n<tr>\n<td>Official pricing page<\/td>\n<td>Cloud KMS pricing<\/td>\n<td>Official SKUs and pricing model: https:\/\/cloud.google.com\/kms\/pricing<\/td>\n<\/tr>\n<tr>\n<td>Official tool<\/td>\n<td>Google Cloud Pricing Calculator<\/td>\n<td>Build estimates for crypto operations and related services: https:\/\/cloud.google.com\/products\/calculator<\/td>\n<\/tr>\n<tr>\n<td>Official security docs<\/td>\n<td>Cloud Audit Logs<\/td>\n<td>Learn how KMS actions are logged and exported: https:\/\/cloud.google.com\/logging\/docs\/audit<\/td>\n<\/tr>\n<tr>\n<td>Official docs<\/td>\n<td>CMEK overview<\/td>\n<td>Understand how CMEK works across Google Cloud services: https:\/\/cloud.google.com\/kms\/docs\/cmek<\/td>\n<\/tr>\n<tr>\n<td>Official docs<\/td>\n<td>Service Directory (if used)<\/td>\n<td>If your EKM connection model uses Service Directory, this is foundational: https:\/\/cloud.google.com\/service-directory\/docs<\/td>\n<\/tr>\n<tr>\n<td>Official architecture center<\/td>\n<td>Google Cloud Architecture Center<\/td>\n<td>Reference architectures and security patterns (search for KMS\/CMEK\/EKM): https:\/\/cloud.google.com\/architecture<\/td>\n<\/tr>\n<tr>\n<td>Official videos<\/td>\n<td>Google Cloud Tech YouTube channel<\/td>\n<td>Product overviews and talks; search for \u201cCloud KMS EKM\u201d: https:\/\/www.youtube.com\/@googlecloudtech<\/td>\n<\/tr>\n<tr>\n<td>Samples \/ SDK docs<\/td>\n<td>Cloud KMS API reference<\/td>\n<td>Understand programmatic access and request patterns: https:\/\/cloud.google.com\/kms\/docs\/reference\/rest<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">18. Training and Certification Providers<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>Institute<\/th>\n<th>Suitable Audience<\/th>\n<th>Likely Learning Focus<\/th>\n<th>Mode<\/th>\n<th>Website URL<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>DevOpsSchool.com<\/td>\n<td>DevOps engineers, SREs, platform teams<\/td>\n<td>Google Cloud operations, DevOps practices, security integration concepts<\/td>\n<td>Check website<\/td>\n<td>https:\/\/www.devopsschool.com\/<\/td>\n<\/tr>\n<tr>\n<td>ScmGalaxy.com<\/td>\n<td>Beginners to intermediate engineers<\/td>\n<td>DevOps fundamentals, tooling, process-oriented learning<\/td>\n<td>Check website<\/td>\n<td>https:\/\/www.scmgalaxy.com\/<\/td>\n<\/tr>\n<tr>\n<td>CLoudOpsNow.in<\/td>\n<td>Cloud engineers and ops teams<\/td>\n<td>Cloud operations, monitoring, reliability, cloud administration<\/td>\n<td>Check website<\/td>\n<td>https:\/\/cloudopsnow.in\/<\/td>\n<\/tr>\n<tr>\n<td>SreSchool.com<\/td>\n<td>SREs and reliability-focused engineers<\/td>\n<td>SRE principles, incident response, monitoring and SLIs\/SLOs<\/td>\n<td>Check website<\/td>\n<td>https:\/\/sreschool.com\/<\/td>\n<\/tr>\n<tr>\n<td>AiOpsSchool.com<\/td>\n<td>Ops teams exploring AIOps<\/td>\n<td>Operational analytics, automation concepts, monitoring practices<\/td>\n<td>Check website<\/td>\n<td>https:\/\/aiopsschool.com\/<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">19. Top Trainers<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>Platform\/Site<\/th>\n<th>Likely Specialization<\/th>\n<th>Suitable Audience<\/th>\n<th>Website URL<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>RajeshKumar.xyz<\/td>\n<td>DevOps\/cloud training and guidance (verify offerings)<\/td>\n<td>Beginners to working professionals<\/td>\n<td>https:\/\/rajeshkumar.xyz\/<\/td>\n<\/tr>\n<tr>\n<td>devopstrainer.in<\/td>\n<td>DevOps training (verify cloud\/security modules)<\/td>\n<td>DevOps engineers, SREs<\/td>\n<td>https:\/\/www.devopstrainer.in\/<\/td>\n<\/tr>\n<tr>\n<td>devopsfreelancer.com<\/td>\n<td>Freelance DevOps consulting\/training platform (verify services)<\/td>\n<td>Teams needing practical implementation help<\/td>\n<td>https:\/\/devopsfreelancer.com\/<\/td>\n<\/tr>\n<tr>\n<td>devopssupport.in<\/td>\n<td>DevOps support\/training (verify scope)<\/td>\n<td>Ops teams and engineers<\/td>\n<td>https:\/\/www.devopssupport.in\/<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">20. Top Consulting Companies<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>Company<\/th>\n<th>Likely Service Area<\/th>\n<th>Where They May Help<\/th>\n<th>Consulting Use Case Examples<\/th>\n<th>Website URL<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>cotocus.com<\/td>\n<td>Cloud\/DevOps engineering (verify service catalog)<\/td>\n<td>Architecture, implementation, automation, operations<\/td>\n<td>Designing CMEK\/EKM rollout plan; building monitoring\/runbooks; IaC for KMS policies<\/td>\n<td>https:\/\/cotocus.com\/<\/td>\n<\/tr>\n<tr>\n<td>DevOpsSchool.com<\/td>\n<td>DevOps and cloud consulting\/training (verify consulting offerings)<\/td>\n<td>Enablement, best practices, implementation support<\/td>\n<td>Security landing zone planning; IAM hardening; operational readiness for external key dependencies<\/td>\n<td>https:\/\/www.devopsschool.com\/<\/td>\n<\/tr>\n<tr>\n<td>DEVOPSCONSULTING.IN<\/td>\n<td>DevOps consulting (verify service catalog)<\/td>\n<td>CI\/CD, cloud operations, reliability practices<\/td>\n<td>Implementing policy-as-code for IAM\/KMS; improving incident response and observability<\/td>\n<td>https:\/\/devopsconsulting.in\/<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">21. Career and Learning Roadmap<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">What to learn before Cloud External Key Manager<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Google Cloud IAM fundamentals<\/strong>: roles, service accounts, policy inheritance, least privilege<\/li>\n<li><strong>Cloud KMS basics<\/strong>: key rings, keys, versions, rotation concepts, audit logs<\/li>\n<li><strong>CMEK concepts<\/strong>: how Google Cloud services use Cloud KMS<\/li>\n<li><strong>Networking fundamentals<\/strong>: private connectivity, DNS, TLS\/mTLS basics<\/li>\n<li><strong>PKI\/certificates<\/strong>: certificate issuance, rotation, trust chains, secure storage<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">What to learn after<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Advanced compliance patterns<\/strong>: audit evidence pipelines, retention, and access reviews<\/li>\n<li><strong>SRE for security dependencies<\/strong>: SLOs, error budgets, incident response for external KMS dependency<\/li>\n<li><strong>Landing zone architecture<\/strong>: hub-and-spoke, shared services projects, org policies<\/li>\n<li><strong>Key governance automation<\/strong>: policy-as-code, approvals, CI\/CD for IAM\/KMS changes<\/li>\n<li><strong>Vendor-specific integration<\/strong>: your external key manager\u2019s HA, policy engine, and audit exports<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Job roles that use it<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Cloud Security Engineer<\/li>\n<li>Security Architect<\/li>\n<li>Platform Engineer (security-focused)<\/li>\n<li>SRE (platform reliability with security dependencies)<\/li>\n<li>Compliance-focused Cloud Engineer<\/li>\n<li>DevOps Engineer implementing CMEK standards<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Certification path (Google Cloud)<\/h3>\n\n\n\n<p>Google Cloud certifications change over time, but relevant tracks often include:\n&#8211; Google Cloud security-focused certifications (verify current names and availability)\n&#8211; Cloud architect and professional-level certifications that cover IAM, KMS, and security architecture<\/p>\n\n\n\n<p>Always verify current Google Cloud certification catalog: https:\/\/cloud.google.com\/learn\/certification<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Project ideas for practice<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Build a CMEK standard for one Google Cloud service and enforce it with policy checks in CI.<\/li>\n<li>Create an IAM least-privilege model for key usage across multiple projects.<\/li>\n<li>Build an audit correlation dashboard: Cloud Audit Logs + external KMS logs (simulated if needed).<\/li>\n<li>Run a game day: external key manager outage simulation and recovery.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">22. Glossary<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Cloud KMS<\/strong>: Google Cloud Key Management Service; manages encryption keys and performs cryptographic operations.<\/li>\n<li><strong>Cloud External Key Manager<\/strong>: Google Cloud capability (Cloud EKM) that allows use of externally managed keys via Cloud KMS.<\/li>\n<li><strong>Cloud EKM<\/strong>: Common shorthand for External Key Manager in Google Cloud documentation.<\/li>\n<li><strong>CMEK<\/strong>: Customer-Managed Encryption Key; encryption key managed by the customer and referenced by Google Cloud services.<\/li>\n<li><strong>External key manager<\/strong>: A key management system outside Google Cloud, often backed by HSMs, that stores\/controls keys.<\/li>\n<li><strong>HSM<\/strong>: Hardware Security Module; specialized hardware for secure key storage and cryptographic operations.<\/li>\n<li><strong>Key ring<\/strong>: Cloud KMS resource container for keys in a specific location.<\/li>\n<li><strong>Crypto key<\/strong>: Cloud KMS logical key resource; may have multiple versions.<\/li>\n<li><strong>Key version<\/strong>: Specific version of a crypto key; used for rotation and lifecycle management.<\/li>\n<li><strong>IAM<\/strong>: Identity and Access Management; authorization system in Google Cloud.<\/li>\n<li><strong>mTLS<\/strong>: Mutual TLS; both client and server authenticate with certificates.<\/li>\n<li><strong>Audit Logs<\/strong>: Logs of administrative and data access events for Google Cloud services.<\/li>\n<li><strong>Separation of duties<\/strong>: Security principle that splits responsibilities across different roles\/teams to reduce insider risk.<\/li>\n<li><strong>Envelope encryption<\/strong>: Pattern where a data encryption key encrypts data, and a key encryption key (KMS-managed) encrypts the data key.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">23. Summary<\/h2>\n\n\n\n<p>Cloud External Key Manager (Cloud EKM) is the Google Cloud Security capability that lets Google Cloud services use encryption keys that remain controlled outside Google Cloud, while still integrating through Cloud KMS and (where supported) CMEK-enabled services.<\/p>\n\n\n\n<p>It matters most when you have strong requirements for <strong>key custody, separation of duties, sovereign control, and independent auditing<\/strong>. Architecturally, it introduces a critical external dependency: your external key manager\u2019s availability, latency, certificates, and network path directly affect workloads that rely on the keys.<\/p>\n\n\n\n<p>Cost and operations require careful planning:\n&#8211; Model Cloud KMS crypto operation volume and any external-key SKUs (verify pricing).\n&#8211; Include external key manager platform costs and potential network egress\/connectivity costs.\n&#8211; Build monitoring, alerting, and runbooks for external key failures and certificate rotation.<\/p>\n\n\n\n<p>Use Cloud External Key Manager when external key control is a hard requirement and you can operate (or contract) a robust external key platform. Otherwise, prefer Cloud KMS (software keys) or Cloud HSM for simpler operations.<\/p>\n\n\n\n<p>Next learning step: read the official Cloud KMS and Cloud EKM documentation, then run a proof of concept for <strong>one<\/strong> Google Cloud service you use most\u2014validate service compatibility, latency, failure behavior, IAM, and audit evidence end-to-end.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Security<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[51,10],"tags":[],"class_list":["post-801","post","type-post","status-publish","format-standard","hentry","category-google-cloud","category-security"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/801","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/comments?post=801"}],"version-history":[{"count":0,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/801\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/media?parent=801"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/categories?post=801"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/tags?post=801"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}