Google Cloud Logging Tutorial: Architecture, Pricing, Use Cases, and Hands-On Guide for Observability and monitoring

Category

Observability and monitoring

1. Introduction

Cloud Logging is Google Cloud’s managed service for collecting, storing, searching, analyzing, and routing logs from Google Cloud services and your applications.

In simple terms: Cloud Logging is where your Google Cloud logs go. It captures logs from services like Cloud Run, Compute Engine, GKE, Cloud Functions, and Google Cloud APIs (audit logs), then lets you explore them, filter them, set retention, and export them to other destinations.

Technically: Cloud Logging provides a centralized logging pipeline made of log ingestion endpoints, the Log Router (for filtering, routing, and exporting), log storage (log buckets with retention policies), and user access controls (views and IAM). It integrates with Cloud Monitoring for metrics/alerting, with BigQuery/Cloud Storage/Pub/Sub for exports, and with security services for auditing and forensics.

The main problem it solves is operational visibility: without centralized logs you can’t reliably debug incidents, prove compliance, investigate security events, or understand application behavior across distributed systems.

2. What is Cloud Logging?

Official purpose (scope): Cloud Logging is Google Cloud’s logging platform for collecting and managing logs from Google Cloud resources, applications, and on-prem/hybrid environments. It supports log exploration, retention management, and exporting (routing) logs to other systems.

Core capabilities – Collect logs from Google Cloud services automatically (for example, Cloud Run request logs; Google Kubernetes Engine system/workload logs; Cloud Audit Logs). – Ingest custom application logs (structured or unstructured). – Search and analyze logs with advanced filters and (where enabled) analytics-style querying. – Route logs to destinations using the Log Router (for example, BigQuery, Pub/Sub, Cloud Storage, or another log bucket). – Control retention and access with log buckets and log views. – Reduce noise and cost using exclusions. – Create log-based metrics (counts or extracted numeric values) that can be used for dashboards and alerting (via Cloud Monitoring).

Major components (mental model) – Log entries: The individual records (timestamp, severity, textPayload/jsonPayload/protoPayload, labels, and monitored resource). – Monitored resource: The thing that produced the log (for example, cloud_run_revision, gce_instance, k8s_container). – Log buckets: Storage containers with retention settings and (optionally) specialized capabilities. Projects have default buckets; you can create additional buckets. – Log views: Filtered windows into a bucket that you can grant access to (least privilege for logs). – Log Router: The routing layer that evaluates log entries against sinks and exclusions. – Sinks: Routing rules that export logs to supported destinations (BigQuery, Pub/Sub, Cloud Storage, or other log buckets). – Log Explorer / Logs Query UI: Primary console interface to find and analyze logs. – Logging API / gcloud logging: Programmatic interfaces to write, read, and manage logging configuration.

Service type – Fully managed Google Cloud service (part of the Google Cloud “Operations”/observability family). – API-driven with Console and CLI tooling.

Scope (project/folder/org + location) – Log ingestion and storage are configured at the project level, but you can also create aggregated exports at the folder or organization level (via sinks). – Log buckets have a location (often global or multi-region/region options depending on current product capabilities). Choose location based on compliance and latency needs. Verify current bucket location options in official docs because availability can evolve.

How it fits into the Google Cloud ecosystem – Cloud Logging is the foundational “event record” layer in Observability and monitoring on Google Cloud. – It integrates tightly with: – Cloud Monitoring (dashboards, alerting, SLOs; log-based metrics feed Monitoring). – Cloud Trace / Profiler / Error Reporting (correlating application telemetry). – Cloud IAM and Cloud Audit Logs (who did what, when). – BigQuery (long-term analytics), Pub/Sub (streaming), Cloud Storage (archival), and SIEM tooling (via exports).

3. Why use Cloud Logging?

Business reasons

• Faster incident resolution reduces downtime and support costs.
• Auditability supports compliance programs (SOC 2, ISO 27001, PCI, HIPAA—depending on your workload and controls).
• Centralization reduces operational overhead versus managing log servers.

Technical reasons

• Native integration with Google Cloud services: logs appear without installing agents for many services.
• Powerful filtering: query by resource type, labels, severity, trace correlation fields, JSON payload keys, and more.
• Programmable routing: export different log subsets to different destinations.

Operational reasons

• Retention controls: keep logs for the time you need (and no longer).
• Noise reduction: exclusions can drop low-value logs before storage/export.
• Least-privilege access: views allow narrowing what different teams can see.

Security/compliance reasons

• Cloud Audit Logs provide immutable evidence of API activity (within platform constraints).
• Central log exports support security operations (SIEM ingestion, threat hunting).
• Access control via IAM, and advanced protections (for example, organization policies and VPC Service Controls where applicable—verify applicability to your environment).

Scalability/performance reasons

• Designed for high-volume ingestion from managed services and distributed workloads.
• Avoids scaling and operating your own ingestion pipeline for baseline needs.

When teams should choose Cloud Logging

• You run workloads on Google Cloud and want a managed logging system.
• You need consistent access controls and retention policy management.
• You want flexible exports (archive, analytics, or streaming) with minimal plumbing.

When teams should not choose it (or should augment it)

• You require a single cross-cloud logging system with identical semantics across all clouds and you already standardized on a third-party SIEM/observability platform (you may still export from Cloud Logging).
• You need full-text indexing and extremely customized query semantics at very large scale and already operate a mature Elastic/OpenSearch/Splunk platform (Cloud Logging can still be the ingestion source).
• You must keep logs entirely off-cloud for regulatory reasons (Cloud Logging may still be used transiently, but you’d prioritize immediate export—verify compliance constraints with your legal/security team).

4. Where is Cloud Logging used?

Industries

• SaaS and technology companies (incident response, debugging, SRE)
• Financial services (audit trails, fraud investigation, compliance)
• Healthcare (auditability; protected data handling policies)
• Retail and e-commerce (transaction monitoring, performance troubleshooting)
• Media and gaming (high-scale workloads, real-time ops)
• Public sector (governance, compliance, retention requirements)

Team types

• SRE and platform engineering teams (reliability, runbooks, incident response)
• DevOps teams (CI/CD observability, deployment debugging)
• Application developers (request tracing, error diagnostics)
• Security engineering / SOC (audit logs, threat detection via exports)
• Data engineering (log analytics via BigQuery exports)

Workloads

• Microservices (Cloud Run, GKE)
• VM-based legacy apps (Compute Engine)
• Event-driven systems (Pub/Sub, Cloud Functions)
• Data platforms (Dataproc, Dataflow—service logs)
• API-driven integrations (audit logs for governance)

Architectures

• Single-project startups (simple default logging)
• Multi-project enterprises (centralized logging with aggregated sinks)
• Hybrid (on-prem apps sending logs via agents to Cloud Logging)
• Regulated environments (separate buckets with controlled retention and views)

Production vs dev/test usage

• Dev/test: shorter retention, more verbose logs, aggressive exclusions for cost control.
• Production: structured logging, strict IAM/view separation, longer retention for audit/security, dedicated export pipelines for SIEM and analytics.

5. Top Use Cases and Scenarios

Below are realistic scenarios where Cloud Logging is commonly used in Google Cloud.

1) Centralized troubleshooting for Cloud Run services

• Problem: Errors occur intermittently across multiple services; developers can’t reproduce locally.
• Why Cloud Logging fits: Automatic capture of request and application logs; filter by service name/revision/trace.
• Example: Filter resource.type="cloud_run_revision" and severity>=ERROR during an incident to find failing endpoints after a rollout.

2) GKE workload and platform logging

• Problem: Pods restart and nodes show unusual behavior; you need container logs plus cluster system signals.
• Why it fits: Integrates with GKE logging pipelines and monitored resources like k8s_container.
• Example: Query logs for a namespace/label to isolate one microservice causing elevated 5xx.

3) Governance and audit trails with Cloud Audit Logs

• Problem: Need to know who changed IAM policies or deleted resources.
• Why it fits: Cloud Audit Logs are delivered into Cloud Logging (Admin Activity, Data Access, System Event, Policy Denied—availability depends on service and configuration).
• Example: Investigate project IAM changes by filtering audit logs for SetIamPolicy.

4) Compliance-driven retention and access separation

• Problem: Compliance requires 1-year retention for security logs, but only 30 days for app debug logs.
• Why it fits: Buckets can have different retention; views and IAM can restrict access.
• Example: Store audit/security logs in a long-retention bucket while dropping debug logs via exclusions.

5) Cost control by dropping noisy logs

• Problem: Health checks and debug logs generate high volume and cost.
• Why it fits: Exclusions can drop logs before storage/export.
• Example: Exclude severity=DEBUG from high-traffic services in production while keeping INFO+.

6) Streaming logs to SIEM via Pub/Sub

• Problem: Security team needs near-real-time log ingestion into a SIEM.
• Why it fits: Log Router supports Pub/Sub sinks; downstream consumers process/forward.
• Example: Export audit logs to Pub/Sub, then a Dataflow pipeline normalizes and forwards to SIEM.

7) Long-term archival to Cloud Storage

• Problem: Need cheap, long-term archive for rare investigations.
• Why it fits: Export to Cloud Storage; apply lifecycle rules (Nearline/Coldline/Archive).
• Example: Export all logs older than operational window to GCS with object lifecycle policies.

8) Analytics and reporting in BigQuery

• Problem: Need weekly reports on error rates per endpoint/customer and ad-hoc forensics.
• Why it fits: BigQuery sinks enable SQL analysis at scale; join logs to business data.
• Example: Export structured logs to BigQuery, then build Looker dashboards for error trends.

9) Release validation and canary analysis

• Problem: After deployment, need to compare error spikes between revisions.
• Why it fits: Logs contain revision labels and can be filtered rapidly.
• Example: Compare logs between revision_name values and correlate with trace IDs.

10) Creating alertable metrics from logs (log-based metrics)

• Problem: You need alerts on specific log patterns (e.g., “payment failed”) without code changes.
• Why it fits: Log-based metrics count matching logs; Cloud Monitoring alerts trigger on metric thresholds.
• Example: Metric increments when jsonPayload.event="PAYMENT_FAILED" appears; alert if > X/min.

11) Cross-project centralization for platform teams

• Problem: Enterprise has dozens of projects; incident response needs a central place to search.
• Why it fits: Aggregated sinks at folder/org export logs into a central project.
• Example: Export all Admin Activity audit logs from all projects into a single “security-logging” project.

12) Forensics after a security incident

• Problem: Need timeline reconstruction: API calls, workload errors, suspicious access.
• Why it fits: Cloud Logging + Audit Logs provide time-indexed record; exports preserve evidence.
• Example: Use saved queries and exports to isolate activity for a compromised service account.

6. Core Features

1) Automatic ingestion for Google Cloud services

• What it does: Many services write logs to Cloud Logging by default.
• Why it matters: You get immediate observability without deploying log infrastructure.
• Practical benefit: Faster time-to-debug and consistent format for platform logs.
• Caveats: Some log types (notably certain Data Access audit logs) may require explicit enablement and can increase cost. Verify per-service audit logging behavior.

2) Structured logging support (JSON payloads)

• What it does: Lets apps emit JSON fields (jsonPayload) rather than only plain text.
• Why it matters: Structured fields are easier to filter, aggregate, and export reliably.
• Practical benefit: Query jsonPayload.userId="123" or jsonPayload.latency_ms>500.
• Caveats: Ensure your app logs valid JSON and avoids putting sensitive data in logs.

3) Log Explorer (search, filter, saved queries)

• What it does: UI for querying logs with advanced filters; supports saving queries.
• Why it matters: Incident response depends on fast, accurate log searches.
• Practical benefit: Quickly narrow down by resource.type, severity, labels, and text.
• Caveats: Very broad queries over large time ranges can be slower; use precise filters.

4) Live Tail (near real-time log viewing)

• What it does: Streams recent logs matching a filter for quick debugging.
• Why it matters: Useful while reproducing issues or validating a fix.
• Practical benefit: Watch errors appear immediately after a deploy.
• Caveats: Not a substitute for durable retention/exports; it’s a troubleshooting tool.

5) Log buckets (storage + retention)

• What it does: Store logs in buckets with configurable retention.
• Why it matters: Retention and separation are core governance controls.
• Practical benefit: Short retention for dev logs, longer for security/audit logs.
• Caveats: Longer retention increases storage cost. Bucket location choices may affect compliance.

6) Log views (least-privilege access)

• What it does: Creates filtered views into a bucket; grant access to the view.
• Why it matters: Logs often contain sensitive operational details.
• Practical benefit: Give app teams access only to their service logs, not security logs.
• Caveats: Requires careful IAM design; test access boundaries before production rollout.

7) Log Router (routing + processing point)

• What it does: Evaluates every incoming log entry against sinks and exclusions.
• Why it matters: Central place to implement “what do we keep, and where do we send it?”
• Practical benefit: Export audit logs to SIEM, app logs to BigQuery, archive to GCS.
• Caveats: Exclusions drop data—treat them as data-loss controls; document decisions.

8) Sinks (exports to other services)

• What it does: Exports matching logs to supported destinations:
• BigQuery (analytics)
• Pub/Sub (streaming pipelines)
• Cloud Storage (archive)
• Log bucket (re-bucket/reroute within Logging)
• Why it matters: Enables downstream analytics and security workflows.
• Practical benefit: Centralize logs across projects, build SIEM pipelines.
• Caveats: Destination services have their own costs and permissions. Exports can create high downstream spend if filters are broad.

9) Exclusions (drop noisy logs)

• What it does: Prevents certain logs from being stored/exported.
• Why it matters: Cost control and noise reduction.
• Practical benefit: Drop health check spam or debug logs in prod.
• Caveats: Excluded logs cannot be recovered. Use carefully and version-control your filters.

10) Log-based metrics

• What it does: Creates metrics from logs (counter or distribution, depending on metric type).
• Why it matters: Turns log patterns into measurable signals for alerting and SLOs.
• Practical benefit: Alert on “authentication failures” derived from logs.
• Caveats: Metric cardinality and volume can affect Monitoring costs/limits. Prefer structured fields.

11) Cloud Audit Logs integration

• What it does: Stores Admin Activity and other audit logs in Cloud Logging.
• Why it matters: Security and governance foundation.
• Practical benefit: Investigate changes to IAM, firewall rules, storage access.
• Caveats: Data Access logs can be high volume; scope them deliberately.

12) Programmatic access (Logging API, CLI, client libraries)

• What it does: Read logs, write logs, manage sinks/buckets/views via APIs and gcloud.
• Why it matters: Enables IaC and automated governance.
• Practical benefit: CI pipeline validates logging sinks and retention policies.
• Caveats: Protect admin capabilities; changes to sinks/exclusions impact evidence collection.

13) (Optional/advanced) Log analytics-style querying

• What it does: Some environments support enhanced analytics on logs (for example, SQL-like querying against logs stored in certain bucket configurations).
• Why it matters: Enables deeper analysis beyond basic filtering.
• Practical benefit: Aggregations, joins (often via exports to BigQuery).
• Caveats: Capabilities and billing can vary. Verify in official docs for current behavior and cost.

7. Architecture and How It Works

High-level architecture

Cloud Logging is best understood as a pipeline:

1. Log producers emit logs: – Google Cloud services automatically (Cloud Run, GKE, Load Balancers, etc.) – Your applications (stdout/stderr, client libraries, or agents) – Audit logs from Google Cloud APIs

2. Ingestion accepts log entries and normalizes them into a consistent schema.

3. Log Router processes each log entry: – Applies exclusions (drop) – Evaluates sinks (export copies to destinations) – Routes remaining logs to log buckets for storage

4. Storage stores logs in buckets with retention policies.

5. Access and analysis: – Users query logs via Log Explorer or APIs (subject to IAM). – Log-based metrics can be generated and sent to Cloud Monitoring. – Exports deliver logs to BigQuery/GCS/Pub/Sub for other use cases.

Data flow vs control flow

• Data flow: log entries → ingestion → Log Router → bucket storage and/or sink destinations.
• Control flow: admins configure buckets, views, sinks, exclusions, and IAM policies.

Integrations with related services

• Cloud Monitoring: dashboards/alerts; uses metrics derived from logs.
• BigQuery: long-term analytics via sinks; often used for reporting and SIEM-like queries.
• Pub/Sub + Dataflow: real-time log processing pipelines.
• Cloud Storage: archival with lifecycle management.
• Cloud IAM: authorization for reading logs and administering routing.
• Security Command Center / SIEM tooling: commonly consumes exports (architecture varies).

Dependency services (typical)

• IAM (for roles and permissions)
• Destination services for exports (BigQuery/GCS/Pub/Sub)
• Workload runtime (Cloud Run/GKE/Compute Engine)

Security/authentication model

• User access is controlled via IAM roles (viewer vs admin vs view access).
• Sink exports use a writer identity (a service account identity managed for the sink). You must grant that identity permissions on the destination (e.g., write to GCS or publish to Pub/Sub).

Networking model

• Cloud Logging is a Google-managed service; you access it via Google APIs.
• Exports to BigQuery/GCS/Pub/Sub are internal Google Cloud service-to-service operations, but you still need to consider:
• Organization policies / VPC Service Controls (if used)
• Cross-project permissions
• Any egress to external SIEM typically happens from your pipeline (e.g., Dataflow) rather than directly from Cloud Logging

Governance considerations

• Establish a bucket strategy (by environment, by sensitivity, by compliance domain).
• Use views for least-privilege access.
• Centralize audit logs into a dedicated security project via aggregated sinks.
• Document and review exclusions and sink filters regularly.

Simple architecture diagram (Mermaid)

flowchart LR
  A[Google Cloud services<br/>Cloud Run / GKE / GCE] --> B[Cloud Logging Ingestion]
  C[Cloud Audit Logs] --> B
  B --> D[Log Router]
  D -->|Store| E[Log Bucket(s)<br/>Retention policy]
  D -->|Export via Sink| F[BigQuery / Pub/Sub / Cloud Storage]
  E --> G[Log Explorer / Logging API]
  E --> H[Log-based Metrics]
  H --> I[Cloud Monitoring<br/>Dashboards & Alerts]

Production-style architecture diagram (Mermaid)

flowchart TB
  subgraph Org["Google Cloud Organization"]
    subgraph FolderA["Folder: Prod"]
      P1["Project: app-prod-1"] --> LR1["Log Router"]
      P2["Project: app-prod-2"] --> LR2["Log Router"]
    end

    subgraph SecProj["Project: security-logging (central)"]
      CentralBucket["Central Log Bucket(s)<br/>- Audit (long retention)<br/>- Security (long retention)"]
      SIEMTopic["Pub/Sub Topic: siem-stream"]
      ArchiveBucket["Cloud Storage Bucket: log-archive<br/>Lifecycle rules"]
      BQ["BigQuery Dataset: log_analytics"]
    end

    LR1 -->|Aggregated sink filter: Audit logs| CentralBucket
    LR2 -->|Aggregated sink filter: Audit logs| CentralBucket

    CentralBucket -->|Sink: security stream| SIEMTopic
    CentralBucket -->|Sink: archive| ArchiveBucket
    CentralBucket -->|Sink: analytics| BQ
  end

  SIEMTopic --> DF["Dataflow pipeline<br/>normalize/enrich"] --> ExtSIEM["External SIEM<br/>(vendor managed)"]

8. Prerequisites

Account/project/billing

• A Google Cloud account with an active billing account attached to your project.
• A Google Cloud project where you can create resources (Cloud Run, Cloud Storage, Logging configuration).

Permissions (IAM roles)

At minimum, for the hands-on lab you typically need: – Cloud Logging: – roles/logging.viewer (to view logs) – roles/logging.configWriter or roles/logging.admin (to create sinks/exclusions/buckets/metrics) – Cloud Run: – roles/run.admin (to deploy a service) – roles/iam.serviceAccountUser (to allow Cloud Run to use a service account, if you specify one) – Cloud Storage: – roles/storage.admin (or at least permission to create a bucket and set IAM on it)

In production, avoid broad roles and prefer least privilege (covered later).

Tools

• Google Cloud Console access
• gcloud CLI (Cloud SDK) and authentication:
• Cloud Shell includes gcloud preinstalled.
• gsutil (included with Cloud SDK) for Cloud Storage operations.
• Optional: curl for sending test requests.

APIs to enable

• Cloud Logging API: logging.googleapis.com
• Cloud Run Admin API: run.googleapis.com
• Cloud Build API (for source-based deploys): cloudbuild.googleapis.com
• Cloud Storage API: storage.googleapis.com

Region availability

• Cloud Logging is a global Google Cloud service with location-based storage options (log bucket location).
• Cloud Run is regional; pick a supported region (e.g., us-central1). Verify in the Cloud Run locations doc if needed.

Quotas/limits to be aware of (high level)

• Logging ingestion, read API usage, export volume, and sink counts are quota-managed.
• Maximum log entry size and label limits exist.
• Always check the official quotas/limits docs before large production rollouts, especially for high-volume workloads or centralized exports.

Prerequisite services

• Cloud Run (to generate logs in the lab)
• Cloud Storage (as an export destination in the lab)

9. Pricing / Cost

Cloud Logging pricing is usage-based. The exact SKUs and free-tier amounts can change, so confirm using official sources: – Pricing page: https://cloud.google.com/logging/pricing – Cloud Pricing Calculator: https://cloud.google.com/products/calculator

Pricing dimensions (what you pay for)

Common cost components include: 1. Log ingestion volume
– Typically charged by the amount of log data ingested (bytes/GiB). 2. Log storage / retention
– Default retention is provided for a baseline period; keeping logs longer or in additional buckets can incur storage charges (GiB-month). 3. Log routing / exports (indirect costs)
– Cloud Logging itself routes logs, but the destination services charge you: – BigQuery: storage + query processing – Cloud Storage: object storage + operations – Pub/Sub: message volume + delivery/storage (depending on configuration) – Dataflow (if you process exports): compute + streaming charges 4. Log-based metrics (indirect costs)
– The metrics created from logs are used in Cloud Monitoring and may contribute to Monitoring metric volume and alerting costs depending on your usage and Google Cloud’s current pricing model. Verify current Monitoring pricing.

Free tier (important)

Google Cloud typically provides some no-cost baseline for logging (often a monthly ingestion allowance and/or default retention). The exact amounts and what’s included can change. Verify the current free tier details on the official pricing page.

Key cost drivers

• High-volume workloads (HTTP access logs, verbose debug logs, chatty services)
• Multi-tenant platforms producing large structured logs per request
• Enabling high-volume audit logs (especially Data Access logs) without scoping
• Broad export sinks sending “everything” to BigQuery (query costs can become significant)
• Long retention on large buckets

Hidden/indirect costs to plan for

• BigQuery query costs from analysts running wide scans over exported logs.
• Storage growth when you extend retention for large log streams.
• Egress if you forward logs to an external SIEM from a pipeline running in Google Cloud (network costs depend on routing and destination).
• Operational overhead: managing sink filters, schema normalization, and access controls.

Network/data transfer implications

• Intra-Google service routing (Logging → BigQuery/GCS/Pub/Sub) is typically not treated like public internet egress, but costs and policies can vary by service, region, and configuration. Verify in the pricing docs for the specific destination service and your network architecture.

How to optimize cost (high impact)

• Use structured logging and log only what you will query.
• Implement exclusions for noisy low-value logs (health checks, debug).
• Separate buckets by retention needs (short for dev/debug, longer for audit/security).
• Export only what you need:
• Use sink filters to export subsets (e.g., severity>=ERROR to BigQuery).
• For BigQuery exports:
• Partition/cluster appropriately (if applicable)
• Restrict who can run expensive queries
• Use scheduled queries/materialized summaries rather than repeated ad-hoc scans

Example low-cost starter estimate (qualitative)

A small Cloud Run service with moderate traffic: – Keeps default retention – Excludes DEBUG logs – No BigQuery export (or exports only ERROR logs) Often stays near the low end of Logging costs; your spend is primarily driven by ingestion volume. Check the pricing calculator with your expected GiB/month.

Example production cost considerations (what to model)

For a platform with dozens of services and security requirements: – Centralized audit log exports across projects – Long retention for audit/security buckets (months/years) – Streaming export to SIEM via Pub/Sub + Dataflow You should model: – Total ingestion GiB/month (by environment) – Retention duration by bucket and expected growth – BigQuery storage/query costs (if used) – Dataflow and Pub/Sub throughput – Access controls that prevent “runaway queries” and unnecessary exports

10. Step-by-Step Hands-On Tutorial

Objective

Deploy a small Cloud Run service that emits structured logs, explore those logs in Cloud Logging, create a log-based metric, and export logs to Cloud Storage using a sink.

Lab Overview

You will: 1. Set up variables and enable required APIs. 2. Deploy a Cloud Run service that generates INFO and ERROR logs. 3. Query logs using Log Explorer and gcloud logging read. 4. Create a log-based metric for ERROR logs. 5. Create a Cloud Storage sink and verify exported log objects. 6. Clean up all resources to avoid ongoing charges.

This lab is designed to be low-cost, but you should still use a billing-enabled project and clean up afterward.

---

Step 1: Select a project, region, and enable APIs

1) Open Cloud Shell in the Google Cloud Console.

2) Set environment variables (adjust as needed):

export PROJECT_ID="YOUR_PROJECT_ID"
export REGION="us-central1"
gcloud config set project "${PROJECT_ID}"
gcloud config set run/region "${REGION}"

3) Enable the required APIs:

gcloud services enable \
  logging.googleapis.com \
  run.googleapis.com \
  cloudbuild.googleapis.com \
  storage.googleapis.com

Expected outcome: APIs enable successfully (this may take a minute).

---

Step 2: Create and deploy a Cloud Run service that writes structured logs

1) Create a working directory:

mkdir -p ~/cloud-logging-lab && cd ~/cloud-logging-lab

2) Create a simple Python service.

Create main.py:

import json
import logging
import os
from flask import Flask, request

app = Flask(__name__)

# Cloud Run captures stdout/stderr into Cloud Logging automatically.
logger = logging.getLogger("log-demo")
logger.setLevel(logging.INFO)

@app.get("/")
def hello():
    payload = {
        "event": "hello",
        "path": request.path,
        "user_agent": request.headers.get("User-Agent"),
    }
    logger.info(json.dumps(payload))
    return ("OK\n", 200)

@app.get("/error")
def error():
    payload = {
        "event": "forced_error",
        "path": request.path,
        "reason": "demo",
    }
    logger.error(json.dumps(payload))
    return ("ERROR\n", 500)

if __name__ == "__main__":
    port = int(os.environ.get("PORT", "8080"))
    app.run(host="0.0.0.0", port=port)

Create requirements.txt:

Flask==3.0.3
gunicorn==22.0.0

Create Procfile (for buildpacks):

web: gunicorn -b :$PORT main:app

3) Deploy to Cloud Run:

gcloud run deploy log-demo \
  --source . \
  --allow-unauthenticated

Expected outcome: Deployment completes and prints a service URL, such as: https://log-demo-<hash>-uc.a.run.app

Save it:

export SERVICE_URL="$(gcloud run services describe log-demo --format='value(status.url)')"
echo "${SERVICE_URL}"

---

Step 3: Generate logs and confirm they appear in Cloud Logging

1) Send a few requests:

curl -s "${SERVICE_URL}/" >/dev/null
curl -s "${SERVICE_URL}/" >/dev/null
curl -s -o /dev/null -w "%{http_code}\n" "${SERVICE_URL}/error"

You should see 500 for the last request.

2) In the Console: – Go to Logging → Log Explorer – Use a query like:

resource.type="cloud_run_revision"
resource.labels.service_name="log-demo"

Optionally narrow to errors:

resource.type="cloud_run_revision"
resource.labels.service_name="log-demo"
severity>=ERROR

Expected outcome: You see recent log entries from Cloud Run, including your JSON message content.

3) Verify via CLI (fast and scriptable):

gcloud logging read \
  'resource.type="cloud_run_revision" AND resource.labels.service_name="log-demo"' \
  --limit 10 \
  --format "value(timestamp,severity,textPayload)"

Expected outcome: You see timestamps and log messages. Depending on how Cloud Run captured the output, your JSON may appear in textPayload (as a JSON string) or as structured fields. Either is fine for this lab.

---

Step 4: Create a log-based metric for ERROR logs

Create a counter metric that increments for each ERROR (or higher) log entry from this service:

gcloud logging metrics create log_demo_error_count \
  --description="Count of ERROR logs from Cloud Run service log-demo" \
  --log-filter='resource.type="cloud_run_revision"
AND resource.labels.service_name="log-demo"
AND severity>=ERROR'

List and describe it:

gcloud logging metrics list --format="table(name,description)"
gcloud logging metrics describe log_demo_error_count

Expected outcome: The metric exists in Cloud Logging configuration.

To generate more error logs:

for i in {1..5}; do
  curl -s -o /dev/null -w "%{http_code}\n" "${SERVICE_URL}/error"
done

Verification in Console (optional): – Go to Monitoring → Metrics Explorer – Look for a user-defined metric related to Logging (often under a namespace like logging.googleapis.com/user/...). – It can take a few minutes for new time series to appear.

Note: Alerting is configured in Cloud Monitoring, not directly in Cloud Logging. If you build alerts, ensure you understand Monitoring pricing and metric cardinality.

---

Step 5: Export Cloud Run logs to Cloud Storage using a sink

This step demonstrates the Log Router + Sink workflow.

1) Create a Cloud Storage bucket (choose a globally-unique name):

export EXPORT_BUCKET="log-demo-export-${PROJECT_ID}-${RANDOM}"
gsutil mb -l "${REGION}" "gs://${EXPORT_BUCKET}"

2) Create a sink that exports only log-demo service logs:

gcloud logging sinks create log-demo-to-gcs \
  "storage.googleapis.com/${EXPORT_BUCKET}" \
  --log-filter='resource.type="cloud_run_revision"
AND resource.labels.service_name="log-demo"'

3) Grant the sink’s writer identity permission to write objects into the bucket:

export WRITER_IDENTITY="$(gcloud logging sinks describe log-demo-to-gcs --format='value(writerIdentity)')"
echo "${WRITER_IDENTITY}"

gsutil iam ch "${WRITER_IDENTITY}:objectCreator" "gs://${EXPORT_BUCKET}"

4) Generate more logs:

curl -s "${SERVICE_URL}/" >/dev/null
curl -s -o /dev/null -w "%{http_code}\n" "${SERVICE_URL}/error"

5) Wait a bit (exports can be delayed), then list exported objects:

gsutil ls "gs://${EXPORT_BUCKET}/**" || true

Expected outcome: You should eventually see objects written by the export pipeline. The exact directory structure and file naming are managed by Cloud Logging.

---

Step 6: (Optional) Create an exclusion to drop DEBUG logs

If your app were emitting debug logs, you could exclude them to reduce noise/cost.

Create an exclusion (example filter; adjust as needed):

gcloud logging exclusions create drop-debug-cloudrun \
  --description="Drop DEBUG logs from Cloud Run to reduce noise" \
  --filter='resource.type="cloud_run_revision" AND severity=DEBUG'

Expected outcome: Exclusion is created. Future matching logs will not be stored/exported.

Important: Exclusions permanently drop matching logs. Use carefully.

---

Validation

Use this checklist:

• Cloud Run service is reachable:
• curl ${SERVICE_URL}/ returns OK
• curl ${SERVICE_URL}/error returns HTTP 500
• Logs appear in Cloud Logging:
• Log Explorer shows entries for service_name="log-demo"
• gcloud logging read ... returns recent entries
• Log-based metric exists:
• gcloud logging metrics describe log_demo_error_count succeeds
• Sink exports are working:
• gsutil ls gs://${EXPORT_BUCKET}/** shows objects (after some delay)

---

Troubleshooting

Issue: No logs appear in Log Explorer – Wait 1–3 minutes; ingestion and UI indexing aren’t always instant. – Confirm you used the correct resource filter: – resource.type="cloud_run_revision" – resource.labels.service_name="log-demo" – Confirm you’re in the correct project in the Console and Cloud Shell: – gcloud config get-value project

Issue: Sink created but nothing shows up in Cloud Storage – Exports can be delayed; wait several minutes and try again. – Confirm bucket IAM was granted to the sink writer identity: – Re-run gcloud logging sinks describe ... and gsutil iam get ... – Confirm the sink filter matches the log entries you are generating. – Ensure the destination bucket exists and you used the correct bucket name.

Issue: Permission denied when creating sinks/exclusions/metrics – You likely need roles/logging.configWriter or roles/logging.admin. – In locked-down orgs, organization policies might restrict sink destinations or storage creation.

Issue: Cloud Run deploy fails – Confirm Cloud Build API is enabled. – Confirm you have roles/run.admin and permissions to build. – Try gcloud run deploy ... --verbosity=debug for details.

---

Cleanup

To avoid ongoing charges and clutter, delete everything you created.

1) Delete the Cloud Run service:

gcloud run services delete log-demo --quiet

2) Delete the log sink:

gcloud logging sinks delete log-demo-to-gcs --quiet

3) Delete the log-based metric:

gcloud logging metrics delete log_demo_error_count --quiet

4) Delete the exclusion (if created):

gcloud logging exclusions delete drop-debug-cloudrun --quiet

5) Delete exported objects and the Cloud Storage bucket:

gsutil -m rm -r "gs://${EXPORT_BUCKET}/**" || true
gsutil rb "gs://${EXPORT_BUCKET}"

Expected outcome: All lab resources are removed.

11. Best Practices

Architecture best practices

• Design bucket strategy intentionally
• Separate by environment (dev/test/prod) and by sensitivity (audit/security/app).
• Assign retention based on real requirements.
• Centralize audit logs
• Use aggregated sinks at folder/org to route audit logs into a dedicated security project.
• Use structured logging
• Prefer JSON fields and consistent keys (event, request_id, user_id, tenant_id, latency_ms).
• Correlate logs with traces
• Where supported, propagate trace context and include trace identifiers to pivot between logs and traces.

IAM/security best practices

• Least privilege
• Use views to restrict access to subsets of logs.
• Separate “read logs” from “admin routing configuration” roles.
• Separate duties
• Security team owns audit/security buckets and export sinks.
• App teams get access to their views only.
• Protect sink configuration
• Changing sinks/exclusions can break compliance evidence pipelines. Restrict who can edit them.

Cost best practices

• Drop what you don’t need
• Exclude overly verbose logs (health checks, debug in production).
• Scope exports
• Export only relevant logs to BigQuery/SIEM. Broad exports increase downstream cost.
• Short retention for noisy logs
• Use shorter retention buckets for high-volume access logs unless required longer.

Performance best practices (operational efficiency)

• Use specific filters (resource type, labels, severity) rather than broad text searches.
• Save common queries and document them in runbooks.
• For analytics at scale, export to BigQuery and design a dataset strategy for query performance.

Reliability best practices

• Treat exports as production pipelines:
• Validate sink filters after changes.
• Monitor destination write errors (where available).
• Ensure permissions remain correct (especially if buckets/datasets are recreated).

Operations best practices

• Standardize log formats across teams.
• Use consistent severity levels.
• Include request IDs and tenant identifiers for multi-tenant systems (but avoid sensitive data).
• Create an “incident mode” checklist: what queries to run, what time windows, what buckets.

Governance/tagging/naming best practices

• Naming conventions for sinks and buckets:
• env-domain-destination (e.g., prod-audit-to-siem-pubsub)
• Document ownership:
• Who owns each sink/bucket, why it exists, and what filter it uses.
• Version-control filters:
• Store sink and exclusion filters in IaC (Terraform or similar) to reduce drift.

12. Security Considerations

Identity and access model

Cloud Logging uses Cloud IAM: – Viewer roles can read logs (subject to restrictions). – Admin/config roles can change sinks, exclusions, buckets, and views. – Sink exports use a writer identity that must be granted destination permissions.

Key recommendations: – Use views to limit exposure instead of giving broad project-level log viewer access. – Separate roles for: – Reading logs (operators, developers) – Administering routing/retention (platform/security admins)

Encryption

• Logs are encrypted at rest by Google by default.
• For stricter requirements, Cloud Logging supports customer-managed encryption keys (CMEK) for some storage configurations (such as log buckets). Verify current CMEK support and limitations in official docs and validate with your compliance team.

Network exposure

• Cloud Logging is accessed via Google APIs.
• If your organization uses perimeter controls (e.g., VPC Service Controls), verify how Logging access and exports behave within your perimeter.

Secrets handling

Common mistake: logging secrets accidentally. – Never log raw credentials, API keys, OAuth tokens, session cookies, or private keys. – Implement application-level log scrubbing/redaction. – Restrict who can view sensitive logs; use separate buckets/views.

Audit/logging of logging changes

• Changes to sinks, exclusions, buckets, and IAM should be monitored.
• Use Cloud Audit Logs to track admin actions on logging configuration.
• Consider exporting logging configuration audit events to a security-owned destination for independent oversight.

Compliance considerations

• Define retention that meets regulatory requirements and legal holds.
• Ensure log bucket location aligns with data residency needs.
• Ensure access controls align with privacy policies (logs can contain personal data).

Common security mistakes

• Granting roles/logging.admin to too many users.
• Exporting “all logs” to external systems without data classification.
• Using exclusions without review, dropping critical audit/security events.
• Storing logs long-term without access review (stale permissions).

Secure deployment recommendations

• Create separate security log buckets with strict IAM.
• Use aggregated sinks for organization-level audit coverage.
• Limit who can disable/alter sinks and exclusions.
• Implement periodic access reviews and automated policy checks (IaC + CI).

13. Limitations and Gotchas

The following are common pitfalls. Exact limits evolve; verify quotas/limits in official documentation for your environment.

Known limitations / quotas (examples to check)

• Maximum log entry size and payload constraints.
• Rate limits for writing logs (API) and reading logs (queries).
• Limits on number of sinks, exclusions, and metrics per project.
• Export delivery is not necessarily instantaneous; sinks can have delays.

Regional constraints

• Log bucket location options may not match every region.
• If you need strict residency, validate bucket location support and export destinations.

Pricing surprises

• Enabling high-volume audit logs (especially Data Access) can increase ingestion significantly.
• BigQuery exports can cause large query costs if analysts scan huge time ranges.
• Verbose application logs (INFO per request with large payloads) can dominate ingestion.

Compatibility issues

• Some workloads output logs in formats that are hard to query (giant unstructured text).
• If you mix languages/frameworks, ensure a consistent structured schema.

Operational gotchas

• Exclusions permanently drop data—treat them as high-risk changes.
• Sink filters that are too broad can overwhelm downstream systems.
• Cross-project exports require careful IAM and can fail silently if permissions drift.
• Developers may rely on logs that are excluded in prod—ensure a policy and communication.

Migration challenges

• Moving from self-managed stacks (ELK/Splunk) requires:
• Schema mapping decisions
• Query translation
• Retention and access model redesign
• Export pipeline design for existing SIEM

Vendor-specific nuances

• Cloud Logging uses Google’s monitored resource model; learning resource types/labels is essential for reliable filters.

14. Comparison with Alternatives

Cloud Logging is the native choice for Google Cloud logs, but it’s not the only option.

Comparison table

Option	Best For	Strengths	Weaknesses	When to Choose
Cloud Logging (Google Cloud)	Google Cloud-first workloads	Native integration, Log Router exports, IAM/views, audit logs	Costs scale with volume; analytics may require exports for deep SQL	You run on Google Cloud and want managed logging with strong routing/governance
Cloud Monitoring (Google Cloud)	Metrics, dashboards, alerting	Excellent for time-series; integrates with log-based metrics	Not a log store; limited for raw event forensics	Use alongside Cloud Logging; don’t replace logs with metrics
BigQuery (via Logging sink)	Large-scale log analytics	SQL, joins with business data, long-term analysis	Separate cost model; requires schema/partitioning strategy	When you need analytics/reporting beyond interactive log search
Pub/Sub + Dataflow pipeline	Real-time processing/forwarding	Streaming enrichment, SIEM forwarders	More moving parts; operational overhead	When you need near-real-time detection or cross-tool distribution
Cloud Storage (via sink)	Low-cost archival	Cheap long-term storage + lifecycle classes	Not interactive for search; retrieval/processing needed	When logs are rarely accessed but must be retained
AWS CloudWatch Logs	AWS workloads	Tight AWS integration	Not native to Google Cloud; cross-cloud overhead	Only if primary workloads are on AWS
Azure Monitor Logs (Log Analytics)	Azure workloads	Azure-native querying and integrations	Not native to Google Cloud	Only if primary workloads are on Azure
Elastic Stack / OpenSearch (self-managed or managed)	Custom search + dashboards	Powerful search/indexing; flexible	Requires ops or managed service costs; ingestion pipelines	If you need custom full-text search and already standardize on Elastic/OpenSearch
Splunk / Datadog / other SaaS observability	Multi-cloud standardization	Unified UI across clouds; advanced analytics	Vendor costs; still need exports and governance	If org standardizes on a third-party platform; export from Cloud Logging

15. Real-World Example

Enterprise example: Centralized audit logging for a regulated organization

Problem A financial services company operates 100+ Google Cloud projects. Auditors require: – Centralized access to Admin Activity audit logs – Long retention for specific audit/security logs – Strict separation: developers must not access security logs

Proposed architecture – Folder/org-level aggregated sinks export: – Admin Activity audit logs – Policy Denied events – Selected Data Access logs (scoped and approved) – Destination: a central “security-logging” project with: – Dedicated long-retention log buckets – Views for different security personas (SOC analysts vs compliance auditors) – Additional sinks export subsets to: – Pub/Sub for SIEM streaming – Cloud Storage for long-term archive with lifecycle rules

Why Cloud Logging was chosen – Native audit log delivery and consistent schema – Built-in routing (Log Router) and permission model (views + IAM) – Works across projects with organization-level configuration patterns

Expected outcomes – Faster audit response and forensics – Reduced risk of missing evidence due to project-by-project variation – Controlled cost via scoped exports and retention policies

---

Startup/small-team example: Cloud Run microservices debugging and alerting

Problem A startup runs microservices on Cloud Run. They need: – Quick debugging during feature releases – Alerts when error rate spikes – Low operational overhead

Proposed architecture – Use Cloud Logging as the primary log store (default buckets) – Standardize structured JSON logs (event, severity, request_id) – Create a few log-based metrics (e.g., payment_failed, auth_error) – Configure Cloud Monitoring alerts on those metrics – Export only ERROR logs to Cloud Storage for lightweight archiving

Why Cloud Logging was chosen – No agents required for Cloud Run – Fast search during incidents – Easy to add routing later as the company grows

Expected outcomes – Faster debugging and fewer “blind” incidents – Predictable baseline costs by excluding DEBUG logs – A path to mature exports (BigQuery/SIEM) later without redesign

16. FAQ

1) Is “Cloud Logging” the same as Stackdriver Logging?

Cloud Logging is the current Google Cloud product name. “Stackdriver Logging” is an older name you may still see in older posts and legacy references.

2) Do I need to install an agent to get logs from Google Cloud services?

Many Google Cloud services send logs automatically. For VM-based workloads or custom apps, you may use an agent (for example, Google Cloud Ops Agent) or write logs via stdout/client libraries depending on the environment. Verify the recommended collection method for your compute platform.

3) What’s the difference between a log bucket and a sink?

A log bucket stores logs in Cloud Logging with retention. A sink exports (routes) matching logs to a destination (BigQuery, Pub/Sub, Cloud Storage, or another bucket).

4) What are log views used for?

Views provide least-privilege access to subsets of logs (via a filter), so teams can see only what they need without broad access to all project logs.

5) How do exclusions affect my logs?

Exclusions drop matching logs before they are stored or exported. This can reduce cost and noise, but it is irreversible data loss for excluded entries.

6) Can I export logs from multiple projects into one central project?

Yes. Common patterns use aggregated sinks at the folder or organization level to route logs into a central logging project.

7) Are Cloud Audit Logs stored in Cloud Logging automatically?

Admin Activity logs are typically enabled by default for many services. Other audit log types (like Data Access) may require explicit configuration and can be high volume. Verify per-service audit logging behavior.

8) Can I send logs to BigQuery for SQL analysis?

Yes, using a BigQuery sink. Be sure to model BigQuery storage and query costs and apply dataset governance.

9) How do I alert on log messages?

Cloud Logging itself is for log storage/search/routing. Alerts are typically implemented in Cloud Monitoring, often using log-based metrics created from filters.

10) What’s the best way to reduce Logging costs?

Focus on ingestion volume: – Exclude low-value logs – Reduce verbosity in production – Export only the subset you need to expensive destinations – Use shorter retention where appropriate

11) Can logs contain sensitive data?

Yes, and that’s a common risk. Apply strict logging hygiene (redaction), access controls (views/IAM), and retention policies.

12) How quickly do logs appear after an event?

Usually within seconds, but it can vary. Exports via sinks can also be delayed. Design operational processes with that in mind.

13) What’s the difference between textPayload and jsonPayload?

• textPayload: unstructured text logs
• jsonPayload: structured key/value data (preferred for consistent filtering and analytics)

14) Can I use Terraform to manage Cloud Logging sinks and buckets?

Yes, Cloud Logging resources are commonly managed with IaC. Confirm the current Terraform Google provider supports the specific resources you need (buckets/views/exclusions) and test changes carefully.

15) How do I share logs with a vendor or external auditor safely?

Prefer exporting the required subset to a controlled destination (BigQuery dataset or Cloud Storage bucket) and granting time-bound access, or create a log view with minimal scope. Avoid broad project-level log access.

16) What are the most important fields for filtering logs?

Typically: – resource.type – resource.labels.* – severity – logName – jsonPayload.* (for structured logging) – time range

17) Should I export everything to my SIEM?

Usually no. Start with audit/security-critical logs and high-value application signals (ERROR/auth failures). Broad exports increase cost and noise and can overwhelm downstream systems.

17. Top Online Resources to Learn Cloud Logging

Resource Type	Name	Why It Is Useful
Official documentation	Cloud Logging docs — https://cloud.google.com/logging/docs	Canonical reference for concepts (buckets, sinks, views), API usage, and best practices
Official pricing	Cloud Logging pricing — https://cloud.google.com/logging/pricing	Current SKUs, free tier, ingestion/storage pricing model
Pricing calculator	Google Cloud Pricing Calculator — https://cloud.google.com/products/calculator	Estimate costs for Logging and export destinations (BigQuery, Pub/Sub, Storage)
Getting started	Cloud Logging quickstart — https://cloud.google.com/logging/docs/quickstart	Step-by-step introduction to viewing and using logs
API reference	Cloud Logging API — https://cloud.google.com/logging/docs/reference/v2/rest	Programmatic management and log entry operations
CLI reference	gcloud logging command group — https://cloud.google.com/sdk/gcloud/reference/logging	Practical CLI commands for reading logs and managing sinks/metrics
Architecture guidance	Google Cloud Architecture Center — https://cloud.google.com/architecture	Reference architectures (use search for logging/export/SIEM patterns)
Sinks/export docs	Routing and exporting logs overview — https://cloud.google.com/logging/docs/routing/overview	Authoritative guide to Log Router, sinks, exclusions, and destinations
Audit logs	Cloud Audit Logs overview — https://cloud.google.com/logging/docs/audit	How audit logs work, types, and configuration considerations
Video (official)	Google Cloud Tech YouTube — https://www.youtube.com/googlecloudtech	Official videos often cover Logging/Operations; search within the channel for “Cloud Logging”
Samples	GoogleCloudPlatform GitHub — https://github.com/GoogleCloudPlatform	Official samples across products; search repos for logging client libraries and patterns
Community (reputable)	Google Cloud Skills Boost — https://www.cloudskillsboost.google	Hands-on labs often include Logging/Operations scenarios (availability varies by catalog)

18. Training and Certification Providers

1. DevOpsSchool.com – Suitable audience: DevOps engineers, SREs, cloud engineers, platform teams – Likely learning focus: Google Cloud operations, observability fundamentals, logging/monitoring practices, DevOps toolchains – Mode: check website – Website: https://www.devopsschool.com/

2. ScmGalaxy.com – Suitable audience: Students, early-career engineers, DevOps practitioners – Likely learning focus: DevOps fundamentals, CI/CD, cloud basics, operations practices – Mode: check website – Website: https://www.scmgalaxy.com/

3. CLoudOpsNow.in – Suitable audience: Cloud operations teams, administrators, DevOps engineers – Likely learning focus: Cloud operations and runbooks, monitoring/logging fundamentals, operational readiness – Mode: check website – Website: https://cloudopsnow.in/

4. SreSchool.com – Suitable audience: SREs, reliability engineers, platform engineering teams – Likely learning focus: SRE practices, incident response, SLOs, observability (including logging) – Mode: check website – Website: https://sreschool.com/

5. AiOpsSchool.com – Suitable audience: Operations teams exploring AIOps, monitoring/observability engineers – Likely learning focus: AIOps concepts, event correlation, automation around observability signals (logs/metrics/traces) – Mode: check website – Website: https://aiopsschool.com/

19. Top Trainers

1. RajeshKumar.xyz – Likely specialization: DevOps and cloud training content (verify offerings on site) – Suitable audience: Beginners to intermediate engineers seeking practical guidance – Website URL: https://rajeshkumar.xyz/

2. devopstrainer.in – Likely specialization: DevOps tooling and practices (verify current course catalog) – Suitable audience: DevOps engineers, SREs, CI/CD practitioners – Website URL: https://devopstrainer.in/

3. devopsfreelancer.com – Likely specialization: Freelance DevOps consulting/training resources (verify services offered) – Suitable audience: Teams seeking hands-on support or short-term enablement – Website URL: https://devopsfreelancer.com/

4. devopssupport.in – Likely specialization: DevOps support and operational assistance (verify service scope) – Suitable audience: Ops teams needing troubleshooting support and guidance – Website URL: https://devopssupport.in/

20. Top Consulting Companies

1. cotocus.com – Likely service area: Cloud/DevOps consulting (verify exact offerings) – Where they may help: Cloud adoption, platform engineering, operational readiness, observability implementations – Consulting use case examples:

   • Designing centralized logging exports and retention strategy
   • Implementing least-privilege access via log views and IAM
   • Building Pub/Sub + Dataflow pipelines for SIEM forwarding
   • Website URL: https://cotocus.com/

2. DevOpsSchool.com – Likely service area: DevOps and cloud consulting/training (verify exact offerings) – Where they may help: DevOps transformations, CI/CD, SRE practices, cloud operations enablement – Consulting use case examples:

   • Establishing logging standards (structured logging schema, severity conventions)
   • Implementing IaC for sinks, buckets, and exclusions
   • Cost optimization reviews for logging ingestion and exports
   • Website URL: https://www.devopsschool.com/

3. DEVOPSCONSULTING.IN – Likely service area: DevOps consulting services (verify exact offerings) – Where they may help: Cloud operations maturity, automation, monitoring/logging practices – Consulting use case examples:

   • Logging governance model (ownership, access reviews, compliance retention)
   • Export architecture to BigQuery and Storage with cost controls
   • Incident response runbook development using Cloud Logging queries
   • Website URL: https://devopsconsulting.in/

21. Career and Learning Roadmap

What to learn before Cloud Logging

• Google Cloud fundamentals:
• Projects, IAM, service accounts
• Regions vs global services
• Basic Linux and application logging concepts:
• stdout/stderr, log levels, JSON logs
• Core Google Cloud runtimes:
• Cloud Run or GKE basics (so you understand resource labels and log sources)

What to learn after Cloud Logging

• Cloud Monitoring (dashboards, alerting policies, SLOs)
• Export destinations and pipelines:
• BigQuery (partitioning, cost controls)
• Pub/Sub + Dataflow (streaming processing patterns)
• Cloud Storage lifecycle and archival strategy
• Security operations on Google Cloud:
• Cloud Audit Logs deep dive
• IAM hardening and org policies
• SIEM integration patterns

Job roles that use Cloud Logging

• Site Reliability Engineer (SRE)
• Cloud/Platform Engineer
• DevOps Engineer
• Security Engineer / SOC Analyst (via audit logs and exports)
• Cloud Architect
• Operations Engineer / NOC Engineer
• Application Developer (production debugging)

Certification path (Google Cloud)

Google Cloud certifications that commonly benefit from strong logging knowledge include: – Associate Cloud Engineer – Professional Cloud DevOps Engineer – Professional Cloud Security Engineer – Professional Cloud Architect

(Exact exam objectives change; verify current guides on Google Cloud’s certification site.)

Project ideas for practice

• Build a “logging standard” for a microservice system:
• JSON schema, correlation IDs, severity rules
• Create a centralized logging project:
• Folder-level sink → central bucket
• Views per team
• Implement a SIEM pipeline:
• Logging sink → Pub/Sub → Dataflow → external endpoint
• Cost optimization exercise:
• Identify top log sources by volume
• Add exclusions and validate impact
• Compliance simulation:
• Long-retention bucket for audit logs
• Restricted access via views and IAM conditions

22. Glossary

• Cloud Logging: Google Cloud service for log ingestion, storage, search, analysis, and routing.
• Log entry: A single log record with timestamp, severity, payload, and resource metadata.
• Severity: Log importance level (e.g., DEBUG, INFO, WARNING, ERROR).
• Monitored resource: The resource model describing where the log came from (e.g., gce_instance, cloud_run_revision).
• Log bucket: Storage container in Cloud Logging with retention and location settings.
• Retention: How long logs are kept before automatic deletion.
• Log view: A filtered view into a bucket used for least-privilege access control.
• Log Router: The routing layer that applies exclusions and exports logs using sinks.
• Sink: A routing/export rule that sends matching logs to a destination.
• Exclusion: A rule that drops matching logs (not stored/exported).
• Cloud Audit Logs: Logs produced by Google Cloud services for governance/audit, including admin actions and data access (depending on configuration).
• Log-based metric: A metric derived from log entries, used in Cloud Monitoring.
• BigQuery sink: Export path from Cloud Logging into BigQuery for SQL analytics.
• Pub/Sub sink: Export path from Cloud Logging into Pub/Sub for streaming pipelines.
• Cloud Storage sink: Export path from Cloud Logging into Cloud Storage for archival.
• Writer identity: The identity (service account) used by a sink to write to its destination.
• Structured logging: Logging using a consistent JSON schema rather than unstructured text.
• Cardinality: Number of unique label/value combinations in metrics; high cardinality increases cost/limits.
• Observability: Ability to understand system behavior using logs, metrics, and traces.

23. Summary

Cloud Logging is Google Cloud’s managed logging service in the Observability and monitoring category. It collects logs from Google Cloud services and your applications, stores them in configurable buckets, and lets you search, analyze, and route logs using the Log Router.

It matters because logs are the fastest path to incident resolution, auditability, and security investigations in modern distributed systems. Cloud Logging fits best when you want a Google Cloud-native logging foundation with strong routing (sinks), governance (buckets/views), and integrations (Monitoring, BigQuery, Pub/Sub, Cloud Storage).

Cost is primarily driven by ingestion volume, retention, and downstream export costs (especially BigQuery queries and streaming pipelines). Security hinges on least-privilege IAM, careful use of views, and disciplined handling of sensitive data in logs.

If you run workloads on Google Cloud and want a managed, integrated logging platform, Cloud Logging is the default choice. Next, deepen your skills by pairing it with Cloud Monitoring (alerting/SLOs) and by practicing export architectures (BigQuery for analytics, Pub/Sub for SIEM pipelines, Cloud Storage for archival).