{"id":118,"date":"2026-04-12T21:25:16","date_gmt":"2026-04-12T21:25:16","guid":{"rendered":"https:\/\/www.devopsschool.com\/tutorials\/aws-data-pipeline-tutorial-architecture-pricing-use-cases-and-hands-on-guide-for-analytics\/"},"modified":"2026-04-12T21:25:16","modified_gmt":"2026-04-12T21:25:16","slug":"aws-data-pipeline-tutorial-architecture-pricing-use-cases-and-hands-on-guide-for-analytics","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/tutorials\/aws-data-pipeline-tutorial-architecture-pricing-use-cases-and-hands-on-guide-for-analytics\/","title":{"rendered":"AWS Data Pipeline Tutorial: Architecture, Pricing, Use Cases, and Hands-On Guide for Analytics"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Category<\/h2>\n\n\n\n<p>Analytics<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">1. Introduction<\/h2>\n\n\n\n<p>AWS Data Pipeline is an AWS Analytics service for defining, scheduling, and orchestrating batch data workflows\u2014moving data between AWS services and running compute steps (for example on Amazon EC2 or Amazon EMR) on a schedule with retries, dependencies, and basic operational visibility.<\/p>\n\n\n\n<p>In simple terms: you define a \u201cpipeline\u201d that says <em>what data to move or process, where it starts and ends, what steps to run, and when to run them<\/em>. AWS Data Pipeline then coordinates the workflow and (optionally) provisions the required compute resources to execute it.<\/p>\n\n\n\n<p>Technically, AWS Data Pipeline is a managed workflow orchestration service that uses pipeline definitions composed of typed objects (data nodes, activities, schedules, resources, and preconditions). A \u201cTask Runner\u201d process, running on an EC2 instance or other compute environment, polls the AWS Data Pipeline service for tasks and executes them with the permissions you configure through IAM roles.<\/p>\n\n\n\n<p>What problem it solves: reliable batch data movement and batch orchestration. It helps replace fragile cron jobs and ad-hoc scripts with repeatable workflows that include scheduling, retries, dependency management, centralized definitions, and basic operational controls.<\/p>\n\n\n\n<p><strong>Important service status note (read first):<\/strong> AWS Data Pipeline is an older AWS orchestration service and is commonly considered <em>legacy for new designs<\/em> compared to newer AWS options such as AWS Glue, AWS Step Functions, and Amazon Managed Workflows for Apache Airflow (MWAA). AWS Data Pipeline is still present in AWS, but for new projects you should validate current AWS guidance and long-term suitability in the official documentation before committing to it in greenfield architectures.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">2. What is AWS Data Pipeline?<\/h2>\n\n\n\n<p>AWS Data Pipeline\u2019s official purpose is to help you <strong>process and move data reliably<\/strong> between different AWS compute and storage services, on a schedule or based on dependencies, with retries and tracking.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Core capabilities<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Define workflows (pipelines)<\/strong> that include data locations, activities (work to run), schedules, and dependencies.<\/li>\n<li><strong>Move data<\/strong> between supported AWS data stores (commonly Amazon S3, Amazon RDS, Amazon DynamoDB, Amazon Redshift) using built-in activity types.<\/li>\n<li><strong>Run compute<\/strong> steps on Amazon EC2 or Amazon EMR (for example, Hive\/Pig jobs on EMR, or shell commands on EC2).<\/li>\n<li><strong>Automate scheduling<\/strong> (periodic runs) and support <strong>retries<\/strong> and <strong>failure handling<\/strong>.<\/li>\n<li><strong>Centralize operational metadata<\/strong> and provide a console view into pipeline execution states.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Major components (conceptual model)<\/h3>\n\n\n\n<p>AWS Data Pipeline uses a set of object types in a <strong>pipeline definition<\/strong>:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Pipeline<\/strong>: the top-level container (name, description, schedule type, logging, roles).<\/li>\n<li><strong>Data nodes<\/strong>: define data sources\/targets (for example, an S3 prefix, a DynamoDB table, a database table).<\/li>\n<li><strong>Activities<\/strong>: define the work to perform (for example, copy data, run SQL, run a shell command, run an EMR job).<\/li>\n<li><strong>Resources<\/strong>: compute environments where activities execute (for example, <code>Ec2Resource<\/code>, <code>EmrCluster<\/code>).<\/li>\n<li><strong>Schedules<\/strong>: when the pipeline runs (time-based scheduling).<\/li>\n<li><strong>Preconditions\/Dependencies<\/strong>: conditions that must be met before an activity runs (for example, data exists).<\/li>\n<li><strong>Task Runner<\/strong>: the execution agent that polls the service for tasks and runs them.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Service type<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Managed workflow orchestration<\/strong> for batch data movement and batch processing.<\/li>\n<li>Not a general-purpose streaming platform; not an interactive query engine; not a modern DAG orchestration UI like Airflow.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Scope (regional\/global\/account)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>AWS Data Pipeline is generally <strong>regional<\/strong>: pipelines are created in an AWS Region and coordinate resources in that Region.<\/li>\n<li>It is <strong>account-scoped<\/strong> within a Region (pipelines live within your AWS account).<\/li>\n<li>Some connected services (like Amazon S3) are global namespaces but backed by regional data locations; plan for region alignment.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">How it fits into the AWS ecosystem<\/h3>\n\n\n\n<p>AWS Data Pipeline often sits \u201cbetween\u201d storage and compute:\n&#8211; Storage\/data services: <strong>Amazon S3, Amazon RDS, Amazon DynamoDB, Amazon Redshift<\/strong>\n&#8211; Compute for batch processing: <strong>Amazon EC2, Amazon EMR<\/strong>\n&#8211; Governance\/operations: <strong>AWS IAM<\/strong> for permissions, <strong>AWS CloudTrail<\/strong> for API auditing, and S3-based logging configured in the pipeline.<\/p>\n\n\n\n<p>For new architectures, teams frequently compare AWS Data Pipeline to:\n&#8211; <strong>AWS Glue<\/strong> (serverless ETL and orchestration features)\n&#8211; <strong>AWS Step Functions<\/strong> (state-machine orchestration)\n&#8211; <strong>Amazon MWAA<\/strong> (managed Apache Airflow)<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">3. Why use AWS Data Pipeline?<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Business reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Automate repeatable batch workflows<\/strong> without maintaining a dedicated orchestration server.<\/li>\n<li><strong>Reduce manual operations<\/strong> for routine data loads (daily exports, nightly transformations).<\/li>\n<li><strong>Standardize data movement<\/strong> between AWS data stores with consistent run history and failure semantics.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Technical reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Declarative pipeline definitions<\/strong>: a consistent model for sources, outputs, compute resources, and schedules.<\/li>\n<li><strong>Managed provisioning option<\/strong>: AWS Data Pipeline can create and terminate EC2\/EMR resources for each run when configured to do so, reducing \u201calways-on\u201d costs (but see pricing and limitations).<\/li>\n<li><strong>Retry and dependency support<\/strong>: helps avoid partial failures and brittle scripts.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Operational reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Central console to view pipeline status and task execution states.<\/li>\n<li>Built-in mechanisms for <strong>logging to S3<\/strong> (commonly via <code>pipelineLogUri<\/code> in the pipeline definition).<\/li>\n<li>Ability to re-run and troubleshoot with a known workflow definition.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Security\/compliance reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Uses <strong>IAM roles<\/strong> for authorization and separation of duties (service role vs. resource role).<\/li>\n<li>Integrates with <strong>CloudTrail<\/strong> for auditing API calls.<\/li>\n<li>Can run compute inside your VPC (for EC2\/EMR resources), enabling private networking patterns.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Scalability\/performance reasons<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Scales primarily by scaling the underlying compute (EC2\/EMR) used for activities.<\/li>\n<li>Supports large data movement patterns (for example, S3-based staging and distributed processing on EMR).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">When teams should choose it<\/h3>\n\n\n\n<p>AWS Data Pipeline can be reasonable when:\n&#8211; You already have existing AWS Data Pipeline workloads to maintain.\n&#8211; You need a relatively simple batch orchestration layer for periodic transfers and jobs.\n&#8211; You rely on specific legacy templates or object types in AWS Data Pipeline.\n&#8211; You want AWS-managed provisioning of EMR\/EC2 resources as part of a defined workflow (and you accept the service\u2019s older orchestration model).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">When teams should not choose it<\/h3>\n\n\n\n<p>Consider alternatives when:\n&#8211; You need <strong>modern DAG orchestration<\/strong>, rich UI, plugins, flexible scheduling, and advanced operational controls (often MWAA\/Airflow).\n&#8211; You need <strong>serverless ETL<\/strong> with schema discovery, Spark jobs, and native data catalog integration (often AWS Glue).\n&#8211; You need event-driven workflows, microservice orchestration, and fine-grained state management (often Step Functions + Lambda\/ECS).\n&#8211; You are starting a new platform and want long-term strategic alignment with AWS\u2019s newer services.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">4. Where is AWS Data Pipeline used?<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Industries<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Finance and insurance: nightly batch reconciliations, compliance exports, warehousing loads.<\/li>\n<li>Retail\/e-commerce: daily sales ingestion into a warehouse, product catalog updates.<\/li>\n<li>Media\/adtech: periodic log processing, batch reporting.<\/li>\n<li>Healthcare\/life sciences: scheduled ETL for analytics (with careful compliance controls).<\/li>\n<li>SaaS: customer usage aggregation, periodic billing exports.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Team types<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Data engineering teams maintaining batch pipelines.<\/li>\n<li>Platform teams running shared data ingestion frameworks.<\/li>\n<li>Analytics engineering teams coordinating periodic data loads.<\/li>\n<li>DevOps\/SRE teams supporting legacy data orchestration.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Workloads<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Scheduled batch data movement (S3 \u2194 database\/warehouse).<\/li>\n<li>Batch transformations on EMR (Hive\/Pig) or EC2 shell steps.<\/li>\n<li>Incremental loads (where the \u201cincrement\u201d is managed by partitioned paths and run windows).<\/li>\n<li>Data export\/import tasks for downstream analytics.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Architectures<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>S3 data lake ingestion pipelines that land raw data and trigger batch transforms.<\/li>\n<li>Warehouse loading patterns (S3 staging \u2192 Redshift COPY, or RDS extracts \u2192 S3).<\/li>\n<li>Hybrid patterns where an EC2 task runner can access data sources reachable via network connectivity.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Production vs dev\/test usage<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Dev\/test<\/strong>: validate pipeline definitions, permissions, logging locations, and schedules; test with small datasets.<\/li>\n<li><strong>Production<\/strong>: focus on IAM least privilege, predictable runtime windows, cost controls around EC2\/EMR, reliable logging, and clear ownership of pipeline definitions and changes.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">5. Top Use Cases and Scenarios<\/h2>\n\n\n\n<p>Below are realistic scenarios where AWS Data Pipeline is commonly used. Each includes the problem, why AWS Data Pipeline fits, and an example.<\/p>\n\n\n\n<p>1) <strong>Nightly S3-to-S3 partitioned copy<\/strong>\n&#8211; <strong>Problem:<\/strong> You receive daily drops into <code>s3:\/\/raw-bucket\/app-logs\/dt=YYYY-MM-DD\/<\/code> and need to copy them into a curated bucket with a standardized prefix.\n&#8211; <strong>Why it fits:<\/strong> AWS Data Pipeline can schedule and run a copy activity with logging and retries.\n&#8211; <strong>Example scenario:<\/strong> Each night at 01:00, copy yesterday\u2019s partition to <code>s3:\/\/curated-bucket\/logs\/<\/code>.<\/p>\n\n\n\n<p>2) <strong>S3 staging to Amazon Redshift load (batch)<\/strong>\n&#8211; <strong>Problem:<\/strong> Load CSV\/Parquet exports into Amazon Redshift for reporting.\n&#8211; <strong>Why it fits:<\/strong> AWS Data Pipeline has patterns and activities that coordinate Redshift loads (often using S3 staging).\n&#8211; <strong>Example scenario:<\/strong> Export daily transactions to S3, then run a Redshift load step; retry on transient failures.<\/p>\n\n\n\n<p>3) <strong>Amazon RDS extract to S3 for analytics<\/strong>\n&#8211; <strong>Problem:<\/strong> Operational data is in RDS, but analytics wants periodic snapshots\/exports to S3.\n&#8211; <strong>Why it fits:<\/strong> AWS Data Pipeline can run SQL-based export patterns via supported activities (depending on engine\/type) and land files to S3.\n&#8211; <strong>Example scenario:<\/strong> Nightly export of a reporting table to S3 as CSV for downstream ingestion.<\/p>\n\n\n\n<p>4) <strong>DynamoDB table export to S3 (batch archival)<\/strong>\n&#8211; <strong>Problem:<\/strong> Archive key-value data periodically to S3 for long-term retention and offline analysis.\n&#8211; <strong>Why it fits:<\/strong> AWS Data Pipeline has data node types for DynamoDB and can orchestrate export-like workflows (capabilities vary; confirm in docs).\n&#8211; <strong>Example scenario:<\/strong> Weekly archive of a DynamoDB table into an S3 prefix with a date partition.<\/p>\n\n\n\n<p>5) <strong>Run a shell-based ETL step on EC2<\/strong>\n&#8211; <strong>Problem:<\/strong> You have a trusted script that cleans data, generates aggregates, or calls an internal API.\n&#8211; <strong>Why it fits:<\/strong> <code>ShellCommandActivity<\/code> can run commands on a managed EC2 resource.\n&#8211; <strong>Example scenario:<\/strong> Launch a short-lived EC2 instance nightly, run a Python script to normalize files in S3, terminate instance.<\/p>\n\n\n\n<p>6) <strong>EMR Hive batch transformation<\/strong>\n&#8211; <strong>Problem:<\/strong> You need distributed SQL-style transformations on large S3 datasets.\n&#8211; <strong>Why it fits:<\/strong> AWS Data Pipeline can orchestrate EMR clusters and submit Hive activities.\n&#8211; <strong>Example scenario:<\/strong> Create an EMR cluster for nightly processing, run Hive queries to build curated datasets, then terminate the cluster.<\/p>\n\n\n\n<p>7) <strong>EMR Pig processing for legacy jobs<\/strong>\n&#8211; <strong>Problem:<\/strong> Existing Pig scripts still run business-critical transformations.\n&#8211; <strong>Why it fits:<\/strong> AWS Data Pipeline supports Pig activities on EMR, enabling scheduling and controlled retries.\n&#8211; <strong>Example scenario:<\/strong> Run Pig-based enrichment on clickstream data each night.<\/p>\n\n\n\n<p>8) <strong>Cross-account data movement via assumed roles (governed)<\/strong>\n&#8211; <strong>Problem:<\/strong> Central analytics account needs to pull data from a producer account on a schedule.\n&#8211; <strong>Why it fits:<\/strong> With IAM role design and S3 access patterns, a pipeline can coordinate cross-account copies (architecture must be reviewed carefully).\n&#8211; <strong>Example scenario:<\/strong> Daily copy from producer S3 bucket to central lake bucket using scoped bucket policies and roles.<\/p>\n\n\n\n<p>9) <strong>Precondition-based processing (\u201conly run if data exists\u201d)<\/strong>\n&#8211; <strong>Problem:<\/strong> Upstream system sometimes fails to deliver a daily file; you don\u2019t want downstream jobs to run and produce empty outputs.\n&#8211; <strong>Why it fits:<\/strong> Preconditions can gate execution based on data presence or other checks (verify the exact precondition types you plan to use).\n&#8211; <strong>Example scenario:<\/strong> Only run transformation if <code>s3:\/\/raw\/...\/dt=...\/<\/code> exists.<\/p>\n\n\n\n<p>10) <strong>Backfill and reprocessing control<\/strong>\n&#8211; <strong>Problem:<\/strong> You need to re-run a pipeline for a historical date range after a bug fix.\n&#8211; <strong>Why it fits:<\/strong> Pipeline scheduling and run windows can support backfills (implementation details vary; verify in docs and test carefully).\n&#8211; <strong>Example scenario:<\/strong> Recompute aggregates for the last 14 days, storing outputs partitioned by date.<\/p>\n\n\n\n<p>11) <strong>Data quality checks with controlled failure<\/strong>\n&#8211; <strong>Problem:<\/strong> You need to stop the pipeline when a validation fails (row counts, schema mismatch).\n&#8211; <strong>Why it fits:<\/strong> A shell\/SQL activity can run checks and fail the run, with logs captured for investigation.\n&#8211; <strong>Example scenario:<\/strong> Run a SQL count check; if below threshold, fail the pipeline and alert.<\/p>\n\n\n\n<p>12) <strong>Legacy batch orchestration consolidation<\/strong>\n&#8211; <strong>Problem:<\/strong> Multiple cron jobs on an EC2 instance are hard to audit and maintain.\n&#8211; <strong>Why it fits:<\/strong> AWS Data Pipeline provides a centralized definition, scheduling, and IAM role separation.\n&#8211; <strong>Example scenario:<\/strong> Move \u201cnightly copy + transform + load\u201d into a single pipeline definition with clear ownership.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">6. Core Features<\/h2>\n\n\n\n<p>AWS Data Pipeline\u2019s feature set is oriented around defining and executing batch workflows.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">6.1 Pipeline definitions (declarative workflow objects)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does:<\/strong> Lets you define pipelines as a set of objects (data nodes, activities, schedules, resources).<\/li>\n<li><strong>Why it matters:<\/strong> Your workflow becomes a versionable artifact (even if stored outside AWS), rather than a collection of server-side cron jobs.<\/li>\n<li><strong>Practical benefit:<\/strong> Easier to review changes, reason about dependencies, and standardize patterns.<\/li>\n<li><strong>Caveats:<\/strong> The object model is older and can feel verbose compared to modern DAG tools.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">6.2 Scheduling and time-based runs<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does:<\/strong> Runs activities on schedules and within time windows.<\/li>\n<li><strong>Why it matters:<\/strong> Most batch analytics depends on predictable schedules (nightly, hourly).<\/li>\n<li><strong>Practical benefit:<\/strong> Fewer missed runs and fewer manual triggers.<\/li>\n<li><strong>Caveats:<\/strong> Scheduling semantics and backfills can be less flexible than Airflow\/Step Functions; validate behavior for your exact requirements.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">6.3 Dependency management and preconditions<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does:<\/strong> Allows gating an activity on conditions (such as data availability).<\/li>\n<li><strong>Why it matters:<\/strong> Prevents downstream steps from running when prerequisites aren\u2019t met.<\/li>\n<li><strong>Practical benefit:<\/strong> More reliable pipelines and fewer \u201cempty output\u201d incidents.<\/li>\n<li><strong>Caveats:<\/strong> Precondition types are limited; confirm that the checks you need are supported.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">6.4 Managed compute resources (EC2\/EMR) for execution<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does:<\/strong> Can provision EC2 instances or EMR clusters to run activities, then terminate them.<\/li>\n<li><strong>Why it matters:<\/strong> You don\u2019t necessarily need a permanent worker fleet.<\/li>\n<li><strong>Practical benefit:<\/strong> Cost control for periodic workloads; consistent environments per run.<\/li>\n<li><strong>Caveats:<\/strong> You still pay for EC2\/EMR and associated data transfer; startup time may impact SLAs.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">6.5 Activity types for common data and compute operations<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does:<\/strong> Provides activity types for copying data, running shell commands, running EMR jobs, and running SQL operations (capabilities vary by activity).<\/li>\n<li><strong>Why it matters:<\/strong> Reduces the amount of custom glue code needed for standard tasks.<\/li>\n<li><strong>Practical benefit:<\/strong> Faster implementation for common batch patterns.<\/li>\n<li><strong>Caveats:<\/strong> Some activity integrations are legacy; for new development, compare to Glue\/Step Functions\/MWAA.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">6.6 Retry and failure behavior<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does:<\/strong> Supports retrying failed tasks and capturing execution details.<\/li>\n<li><strong>Why it matters:<\/strong> Batch pipelines regularly fail due to transient issues (network, throttling, temporary service problems).<\/li>\n<li><strong>Practical benefit:<\/strong> Improved resilience without manual restarts.<\/li>\n<li><strong>Caveats:<\/strong> Understand idempotency\u2014retries can duplicate work if the underlying operation isn\u2019t safe to repeat.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">6.7 Logging (commonly to Amazon S3)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does:<\/strong> Captures execution logs and task runner logs to a specified S3 location (configured in the pipeline).<\/li>\n<li><strong>Why it matters:<\/strong> Central logs are essential for troubleshooting and audits.<\/li>\n<li><strong>Practical benefit:<\/strong> Postmortems and debugging are possible even after ephemeral compute terminates.<\/li>\n<li><strong>Caveats:<\/strong> Ensure the log bucket is protected (least privilege, encryption, retention). CloudWatch integration may be limited; verify in official docs.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">6.8 IAM role separation (service role vs resource role)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does:<\/strong> Uses a service role for AWS Data Pipeline control actions and a resource role\/instance profile for compute resources.<\/li>\n<li><strong>Why it matters:<\/strong> Avoids over-privileged execution environments and supports separation of duties.<\/li>\n<li><strong>Practical benefit:<\/strong> More controlled access to S3 buckets, databases, and KMS keys.<\/li>\n<li><strong>Caveats:<\/strong> Misconfigured roles are a top cause of failures (access denied, unable to create resources).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">6.9 Templates and guided creation (console)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>What it does:<\/strong> Provides templates in the AWS console for common patterns.<\/li>\n<li><strong>Why it matters:<\/strong> Helps beginners create working pipelines without writing full definitions from scratch.<\/li>\n<li><strong>Practical benefit:<\/strong> Faster time-to-first-run and fewer syntax errors.<\/li>\n<li><strong>Caveats:<\/strong> Template availability and fields can change; confirm in the console and docs.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">7. Architecture and How It Works<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">High-level service architecture<\/h3>\n\n\n\n<p>At a high level, AWS Data Pipeline has a <strong>control plane<\/strong> (the Data Pipeline service) and a <strong>data plane<\/strong> (the compute resources that run your tasks).<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Control plane (AWS Data Pipeline service):<\/strong><\/li>\n<li>Stores pipeline definitions.<\/li>\n<li>Schedules tasks and tracks state.<\/li>\n<li>\n<p>Coordinates which tasks are ready to run based on schedule and dependencies.<\/p>\n<\/li>\n<li>\n<p><strong>Data plane (Task Runner on compute):<\/strong><\/p>\n<\/li>\n<li>A Task Runner process runs on an EC2 instance or other supported compute.<\/li>\n<li>It polls the service for tasks assigned to its worker group.<\/li>\n<li>It executes activities (copy, shell commands, EMR steps) using IAM permissions provided to the compute resource.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Request\/data\/control flow (typical run)<\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li>You <strong>create a pipeline<\/strong> and <strong>upload a pipeline definition<\/strong> (via console, CLI, or SDK).<\/li>\n<li>You <strong>activate<\/strong> the pipeline.<\/li>\n<li>On schedule, AWS Data Pipeline <strong>creates task instances<\/strong> (based on your definition).<\/li>\n<li>A <strong>Task Runner<\/strong> polls for tasks and claims them.<\/li>\n<li>The Task Runner <strong>performs the activity<\/strong>, reading\/writing data to services like S3\/RDS\/Redshift.<\/li>\n<li>Logs are written (commonly to S3), and the pipeline state updates.<\/li>\n<li>If configured, ephemeral resources may terminate after completion.<\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\">Integrations with related services<\/h3>\n\n\n\n<p>Common integrations include:\n&#8211; <strong>Amazon S3<\/strong>: source\/target, staging area, and pipeline logs.\n&#8211; <strong>Amazon EC2<\/strong>: Task Runner host and shell-based processing.\n&#8211; <strong>Amazon EMR<\/strong>: managed Hadoop\/Spark (legacy Hive\/Pig patterns are common with Data Pipeline).\n&#8211; <strong>Amazon RDS \/ JDBC-accessible databases<\/strong>: extract\/load using SQL activities (capability varies).\n&#8211; <strong>Amazon DynamoDB<\/strong>: some pipeline patterns support reads\/writes (verify exact node\/activity types).\n&#8211; <strong>AWS IAM<\/strong>: roles and permissions.\n&#8211; <strong>AWS CloudTrail<\/strong>: audit of pipeline API actions.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Dependency services<\/h3>\n\n\n\n<p>AWS Data Pipeline itself orchestrates, but typically depends on:\n&#8211; IAM roles and policies\n&#8211; S3 for logs and\/or staging\n&#8211; EC2\/EMR for actual execution\n&#8211; VPC networking and security groups if running in private subnets<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Security\/authentication model<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>API access to AWS Data Pipeline is governed by <strong>IAM policies<\/strong>.<\/li>\n<li>Pipeline execution uses IAM roles:<\/li>\n<li><strong>Service role<\/strong>: lets AWS Data Pipeline create\/describe resources on your behalf.<\/li>\n<li><strong>Resource role \/ instance profile<\/strong>: grants the EC2\/EMR runtime access to S3, logs, and any data endpoints.<\/li>\n<li>For data stores that require credentials (for example, databases), avoid embedding secrets directly in definitions; prefer AWS managed secret patterns where possible (see Security Considerations).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Networking model<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Pipelines are regional; resources usually run in a specific VPC\/subnet if configured.<\/li>\n<li>Task Runner must reach the AWS Data Pipeline endpoint over HTTPS.<\/li>\n<li>If your resource is in a private subnet, you typically need controlled egress (NAT Gateway or equivalent). <strong>Verify whether an AWS PrivateLink (VPC endpoint) exists for AWS Data Pipeline in your region<\/strong>; many older services do not provide it.<\/li>\n<li>Data access to S3 can be optimized with <strong>S3 Gateway VPC endpoints<\/strong>.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Monitoring\/logging\/governance considerations<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Execution visibility<\/strong>: console pipeline status and per-activity status.<\/li>\n<li><strong>Logs<\/strong>: configure S3 log URI; enforce encryption and retention.<\/li>\n<li><strong>Audit<\/strong>: enable CloudTrail and review Data Pipeline API calls.<\/li>\n<li><strong>Tagging<\/strong>: apply tags to pipelines and downstream resources (where supported) for cost allocation and ownership.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Simple architecture diagram (Mermaid)<\/h3>\n\n\n\n<pre><code class=\"language-mermaid\">flowchart LR\n  subgraph AWS_Region[\"AWS Region\"]\n    DP[\"AWS Data Pipeline\\n(Control plane)\"]\n    S3A[\"Amazon S3\\nSource prefix\"]\n    S3B[\"Amazon S3\\nDestination prefix\"]\n    EC2[\"EC2 Resource\\n(Task Runner)\"]\n    LOGS[\"S3 Bucket\/Prefix\\nPipeline logs\"]\n  end\n\n  DP --&gt;|Schedules task| EC2\n  EC2 --&gt;|Read| S3A\n  EC2 --&gt;|Write| S3B\n  EC2 --&gt;|Write logs| LOGS\n  EC2 --&gt;|Poll\/Report status| DP\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Production-style architecture diagram (Mermaid)<\/h3>\n\n\n\n<pre><code class=\"language-mermaid\">flowchart TB\n  subgraph Account[\"AWS Account (Prod)\"]\n    subgraph VPC[\"VPC\"]\n      subgraph PrivateSubnets[\"Private subnets\"]\n        EC2TR[\"EC2 Task Runner\\n(Ephemeral or managed)\"]\n        EMR[\"Amazon EMR (optional)\\nBatch processing\"]\n      end\n      NAT[\"NAT Gateway \/ Egress\\n(if needed for HTTPS to control plane)\"]\n      VPCE_S3[\"S3 Gateway VPC Endpoint\"]\n    end\n\n    DP[\"AWS Data Pipeline\\n(Control plane)\"]\n    S3RAW[\"Amazon S3 Data Lake\\nRaw bucket\"]\n    S3CUR[\"Amazon S3 Data Lake\\nCurated bucket\"]\n    RDS[\"Amazon RDS\\n(operational DB)\"]\n    RS[\"Amazon Redshift\\n(warehouse)\"]\n    CT[\"AWS CloudTrail\"]\n    IAM[\"AWS IAM Roles\\n(service + resource roles)\"]\n    KMS[\"AWS KMS\\n(bucket\/key encryption)\"]\n  end\n\n  DP --&gt;|Create\/coordinate tasks| EC2TR\n  DP --&gt;|Optionally orchestrate| EMR\n\n  EC2TR --&gt;|Read\/Write via VPCE| S3RAW\n  EC2TR --&gt;|Write curated| S3CUR\n  EC2TR --&gt;|Extract\/Load| RDS\n  EC2TR --&gt;|Load\/Unload| RS\n\n  EC2TR --&gt; NAT\n  EC2TR --&gt; VPCE_S3\n\n  DP --&gt; CT\n  EC2TR --&gt; IAM\n  S3RAW --&gt; KMS\n  S3CUR --&gt; KMS\n<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">8. Prerequisites<\/h2>\n\n\n\n<p>Before starting, ensure you have the following.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">AWS account and billing<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>An AWS account with billing enabled.<\/li>\n<li>Permission to create S3 buckets and (during the lab) to create AWS Data Pipeline pipelines.<\/li>\n<li>Permission to create IAM roles (or permission to pass\/use existing roles), and to launch EC2 resources if using managed compute.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Permissions \/ IAM roles<\/h3>\n\n\n\n<p>At minimum for the hands-on lab, your IAM principal (user\/role) should be able to:\n&#8211; Use AWS Data Pipeline APIs (create pipeline, put definition, activate\/deactivate, delete).\n&#8211; Create or use:\n  &#8211; A <strong>service role<\/strong> for AWS Data Pipeline.\n  &#8211; A <strong>resource role<\/strong> \/ instance profile for EC2 resources.\n&#8211; Create and manage S3 buckets and objects used in the lab.<\/p>\n\n\n\n<p>In many accounts, the console may offer to create default roles such as:\n&#8211; <code>DataPipelineDefaultRole<\/code>\n&#8211; <code>DataPipelineDefaultResourceRole<\/code><\/p>\n\n\n\n<p>Names can vary; verify in your account and region.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Tools<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>AWS Management Console access<\/li>\n<li>Optional: AWS CLI for verification and cleanup (recommended)<\/li>\n<li>Install\/verify AWS CLI: https:\/\/docs.aws.amazon.com\/cli\/latest\/userguide\/getting-started-install.html<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Region availability<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>AWS Data Pipeline is not available in every region. Confirm region support in the console and official docs:<\/li>\n<li>https:\/\/docs.aws.amazon.com\/datapipeline\/latest\/DeveloperGuide\/what-is-datapipeline.html (navigate to region notes if present)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Quotas \/ limits<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>AWS Data Pipeline has service limits (for example, counts of pipelines, objects, and possibly active resources).<\/li>\n<li>Check limits in the AWS Data Pipeline documentation and\/or AWS Service Quotas (if listed for this service).<\/li>\n<li>In production, also check EC2, EMR, and S3 limits that will affect your pipeline.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Prerequisite services for the lab<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Amazon S3 (source data, destination data, and logs)<\/li>\n<li>EC2 (if the pipeline uses an EC2 resource to execute tasks)<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">9. Pricing \/ Cost<\/h2>\n\n\n\n<p>AWS Data Pipeline costs are a combination of:\n1) <strong>AWS Data Pipeline service charges<\/strong>, and<br\/>\n2) <strong>Charges for the AWS resources<\/strong> the pipeline uses (often the larger component).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Pricing dimensions (service)<\/h3>\n\n\n\n<p>AWS Data Pipeline pricing is typically based on:\n&#8211; <strong>The number of pipelines<\/strong> you run per month\n&#8211; <strong>The schedule frequency<\/strong> (for example, more frequent schedules vs. less frequent schedules)<\/p>\n\n\n\n<p>Exact rates can vary and may be updated by AWS. Use the official pricing page for current details:\n&#8211; Official pricing: https:\/\/aws.amazon.com\/datapipeline\/pricing\/\n&#8211; AWS Pricing Calculator: https:\/\/calculator.aws\/#\/<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Free tier<\/h3>\n\n\n\n<p>AWS Data Pipeline historically has not been a \u201cfree-tier-heavy\u201d service in the way some serverless services are. Check the current pricing page for any promotional free tier or special conditions.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Primary cost drivers<\/h3>\n\n\n\n<p>Even when the AWS Data Pipeline service charge is modest, pipelines commonly incur costs from:\n&#8211; <strong>EC2 instances<\/strong> launched for Task Runner \/ ShellCommandActivity \/ Copy activities\n&#8211; <strong>EMR clusters<\/strong> (if used) and EMR steps\n&#8211; <strong>S3 storage<\/strong> (raw data, curated data, logs) and requests\n&#8211; <strong>Data transfer<\/strong>:\n  &#8211; Cross-AZ\/cross-region transfers if you design across boundaries\n  &#8211; NAT Gateway data processing charges if private-subnet resources need internet egress (common hidden cost)\n&#8211; <strong>KMS requests<\/strong> (if using SSE-KMS for buckets and writing lots of small objects)<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Hidden or indirect costs to watch<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>NAT Gateway<\/strong>: If your EC2 Task Runner is in a private subnet and must reach AWS endpoints over the internet, NAT can add hourly + per-GB charges (verify NAT pricing for your region).<\/li>\n<li><strong>Logging volume<\/strong>: verbose task logs to S3 can generate storage and request costs.<\/li>\n<li><strong>Retries<\/strong>: repeated execution can multiply EC2 runtime and data transfer.<\/li>\n<li><strong>Cross-region S3 copies<\/strong>: egress and inter-region costs can dominate.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">How to optimize cost<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Use the <strong>smallest EC2 instance type<\/strong> that reliably completes the job.<\/li>\n<li>Terminate resources after completion (use ephemeral compute patterns where supported).<\/li>\n<li>Keep data and compute in the <strong>same region<\/strong> to avoid inter-region charges.<\/li>\n<li>Prefer <strong>S3 VPC endpoints<\/strong> for S3 access from private subnets (reduces NAT data usage for S3 traffic).<\/li>\n<li>Reduce log verbosity and set lifecycle policies on the log bucket\/prefix.<\/li>\n<li>If using EMR, consider:<\/li>\n<li>Short-lived clusters<\/li>\n<li>Spot Instances where appropriate (with careful failure handling)<\/li>\n<li>EMR managed scaling (if it aligns with your job pattern)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Example low-cost starter estimate (method, not fabricated numbers)<\/h3>\n\n\n\n<p>A realistic \u201cstarter\u201d lab pattern is:\n&#8211; One pipeline that runs infrequently (for example, daily or on-demand)\n&#8211; Copies a small file from one S3 prefix to another\n&#8211; Uses a short-lived EC2 instance for a few minutes\n&#8211; Writes logs to S3<\/p>\n\n\n\n<p>To estimate:\n&#8211; Use the AWS Pricing Calculator:\n  1. Add <strong>AWS Data Pipeline<\/strong> (if listed) and configure one pipeline and frequency.\n  2. Add <strong>EC2<\/strong>: a small instance type, short runtime per run, and runs per month.\n  3. Add <strong>S3<\/strong>: storage size and request counts (PUT\/GET) and log retention.\n  4. Add <strong>data transfer<\/strong> if cross-region or via NAT.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Example production cost considerations<\/h3>\n\n\n\n<p>In production, the main cost levers are:\n&#8211; Number of pipelines and frequency (service charges)\n&#8211; Total compute-hours (EC2\/EMR), especially if pipelines overlap\n&#8211; Data volume and transfer patterns (S3 request rates, cross-region movement)\n&#8211; Networking design (NAT vs endpoints)\n&#8211; Security controls that add per-request overhead (SSE-KMS at high object counts)<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">10. Step-by-Step Hands-On Tutorial<\/h2>\n\n\n\n<p>This lab builds a small but real AWS Data Pipeline workflow that copies objects from one S3 prefix to another on a single run (or on a simple schedule), with S3-based logging. It is designed to be low-cost and beginner-friendly.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Objective<\/h3>\n\n\n\n<p>Create and run an AWS Data Pipeline that copies a file from a source Amazon S3 bucket\/prefix to a destination S3 bucket\/prefix, then verify the output and clean up all created resources.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Lab Overview<\/h3>\n\n\n\n<p>You will:\n1. Create two S3 buckets (source and destination) and upload a small test file.\n2. Create an AWS Data Pipeline using a copy-based template\/pattern.\n3. Configure pipeline logging to S3.\n4. Activate the pipeline and monitor execution.\n5. Validate that the file was copied.\n6. Clean up the pipeline and S3 buckets.<\/p>\n\n\n\n<p><strong>Estimated time:<\/strong> 30\u201360 minutes<br\/>\n<strong>Cost:<\/strong> Low, but not zero. The pipeline may launch short-lived EC2 resources depending on the template\/pattern. You will also incur S3 request\/storage charges.<\/p>\n\n\n\n<blockquote>\n<p>If the AWS console experience differs in your region (templates\/UI may change), use the official developer guide to map concepts to the current UI:<br\/>\nhttps:\/\/docs.aws.amazon.com\/datapipeline\/latest\/DeveloperGuide\/what-is-datapipeline.html<\/p>\n<\/blockquote>\n\n\n\n<h3 class=\"wp-block-heading\">Step 1: Choose a region and set naming<\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Choose an AWS Region where AWS Data Pipeline is available.<\/li>\n<li>Decide a unique suffix for resources (S3 bucket names must be globally unique). Example suffix:\n   &#8211; <code>&lt;account-id&gt;-&lt;region&gt;-dp-lab<\/code><\/li>\n<\/ol>\n\n\n\n<p>You will create:\n&#8211; Source bucket: <code>dp-lab-src-&lt;suffix&gt;<\/code>\n&#8211; Destination bucket: <code>dp-lab-dst-&lt;suffix&gt;<\/code>\n&#8211; Logs prefix: <code>s3:\/\/dp-lab-src-&lt;suffix&gt;\/pipeline-logs\/<\/code><\/p>\n\n\n\n<p><strong>Expected outcome:<\/strong> You have a region selected and a naming plan that avoids collisions.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 2: Create S3 buckets and upload a test file<\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Open the Amazon S3 console.<\/li>\n<li>Create the <strong>source bucket<\/strong> (example): <code>dp-lab-src-&lt;suffix&gt;<\/code>\n   &#8211; Keep \u201cBlock Public Access\u201d enabled.\n   &#8211; Enable default encryption (SSE-S3 is fine for this lab; SSE-KMS is optional).<\/li>\n<li>Create the <strong>destination bucket<\/strong> (example): <code>dp-lab-dst-&lt;suffix&gt;<\/code>\n   &#8211; Same security defaults.<\/li>\n<li>In the source bucket, create a prefix (folder) called:\n   &#8211; <code>input\/<\/code><\/li>\n<li>Upload a small file into <code>input\/<\/code>, for example <code>hello.txt<\/code> with content:\n   &#8211; <code>hello from aws data pipeline<\/code><\/li>\n<\/ol>\n\n\n\n<p>Optional CLI to upload:<\/p>\n\n\n\n<pre><code class=\"language-bash\">aws s3 cp .\/hello.txt s3:\/\/dp-lab-src-&lt;suffix&gt;\/input\/hello.txt\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome:<\/strong><br\/>\n&#8211; <code>s3:\/\/dp-lab-src-&lt;suffix&gt;\/input\/hello.txt<\/code> exists<br\/>\n&#8211; Destination bucket is empty (for now)<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 3: Create an AWS Data Pipeline pipeline<\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Open the AWS Data Pipeline console: https:\/\/console.aws.amazon.com\/datapipeline\/<\/li>\n<li>Choose <strong>Create new pipeline<\/strong>.<\/li>\n<li>Enter:\n   &#8211; <strong>Name<\/strong>: <code>dp-lab-s3-copy<\/code>\n   &#8211; <strong>Description<\/strong>: <code>Copy a test file from S3 source to S3 destination<\/code><\/li>\n<li>For <strong>Source<\/strong> \/ <strong>Template<\/strong>:\n   &#8211; Choose a template that performs <strong>S3-to-S3 copy<\/strong> (template names can vary).\n   &#8211; If you don\u2019t see an S3-to-S3 template, choose a \u201cCopy activity\u201d template and configure both input and output as S3 locations.<\/li>\n<li>Configure the key parameters (exact field names vary by template):\n   &#8211; <strong>Input S3 path<\/strong>: <code>s3:\/\/dp-lab-src-&lt;suffix&gt;\/input\/<\/code>\n   &#8211; <strong>Output S3 path<\/strong>: <code>s3:\/\/dp-lab-dst-&lt;suffix&gt;\/output\/<\/code>\n   &#8211; <strong>Log URI<\/strong> (if available): <code>s3:\/\/dp-lab-src-&lt;suffix&gt;\/pipeline-logs\/<\/code>\n   &#8211; <strong>Schedule<\/strong>:<ul>\n<li>For a low-cost lab, prefer <strong>on-demand<\/strong> or a schedule that runs once soon.<\/li>\n<li>If the template requires a recurring schedule, set it to run once and then you will deactivate it after validation. (Behavior depends on template; verify in the console.)<\/li>\n<li><strong>Resource settings<\/strong>:<\/li>\n<li>If prompted for an EC2 instance type, select a small instance type (for example <code>t3.micro<\/code> if allowed in your region\/account).<\/li>\n<\/ul>\n<\/li>\n<li>IAM roles:\n   &#8211; If the console offers to create default roles (often named <code>DataPipelineDefaultRole<\/code> and <code>DataPipelineDefaultResourceRole<\/code>), allow it for the lab.\n   &#8211; If your organization restricts role creation, coordinate with your IAM admin and use pre-approved roles.<\/li>\n<\/ol>\n\n\n\n<p><strong>Expected outcome:<\/strong> A pipeline is created in \u201cDraft\u201d (or similar) state with a valid configuration and an S3 log location.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 4: Validate the pipeline definition and activate it<\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li>In the pipeline details, use the console option to <strong>Validate<\/strong> (or similar) the pipeline definition.<\/li>\n<li>Fix any validation errors:\n   &#8211; S3 path typos are common.\n   &#8211; Missing permissions for the log bucket\/prefix is common.<\/li>\n<li>Choose <strong>Activate<\/strong>.<\/li>\n<\/ol>\n\n\n\n<p><strong>Expected outcome:<\/strong>\n&#8211; Pipeline transitions from \u201cDraft\u201d to \u201cActive\u201d.\n&#8211; Within a few minutes, the pipeline should begin scheduling tasks.\n&#8211; If the pipeline uses managed EC2 resources, you may see related EC2 instances being created temporarily.<\/p>\n\n\n\n<p>Optional: watch EC2 instances created by the pipeline\n&#8211; Open the EC2 console and look for newly launched instances with tags indicating Data Pipeline ownership. Tag formats can vary.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 5: Monitor execution and review logs<\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li>In the AWS Data Pipeline console, open the pipeline.<\/li>\n<li>Check:\n   &#8211; <strong>Status<\/strong> of the pipeline and the most recent run\n   &#8211; Any <strong>activity\/task<\/strong> status views (names vary)<\/li>\n<li>In Amazon S3, open:\n   &#8211; <code>s3:\/\/dp-lab-src-&lt;suffix&gt;\/pipeline-logs\/<\/code><\/li>\n<li>Review log output if the run fails or is delayed.<\/li>\n<\/ol>\n\n\n\n<p><strong>Expected outcome:<\/strong>\n&#8211; The copy activity completes successfully (or provides actionable error details in logs).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 6: Validate that the file was copied<\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Open the destination bucket in S3:\n   &#8211; <code>s3:\/\/dp-lab-dst-&lt;suffix&gt;\/output\/<\/code><\/li>\n<li>Confirm <code>hello.txt<\/code> exists and contains the expected content.<\/li>\n<\/ol>\n\n\n\n<p>Optional CLI validation:<\/p>\n\n\n\n<pre><code class=\"language-bash\">aws s3 ls s3:\/\/dp-lab-dst-&lt;suffix&gt;\/output\/\naws s3 cp s3:\/\/dp-lab-dst-&lt;suffix&gt;\/output\/hello.txt -\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome:<\/strong> The destination prefix contains the copied file.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Validation<\/h3>\n\n\n\n<p>Use this checklist:\n&#8211; [ ] Source file exists: <code>s3:\/\/dp-lab-src-&lt;suffix&gt;\/input\/hello.txt<\/code>\n&#8211; [ ] Pipeline is active and shows a successful run (or completed activity)\n&#8211; [ ] Destination file exists: <code>s3:\/\/dp-lab-dst-&lt;suffix&gt;\/output\/hello.txt<\/code>\n&#8211; [ ] Pipeline logs exist in: <code>s3:\/\/dp-lab-src-&lt;suffix&gt;\/pipeline-logs\/<\/code><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Troubleshooting<\/h3>\n\n\n\n<p>Common issues and fixes:<\/p>\n\n\n\n<p>1) <strong>Pipeline stuck in \u201cWAITING_FOR_RUNNER\u201d or similar<\/strong>\n&#8211; <strong>Cause:<\/strong> No Task Runner is available, or the managed resource didn\u2019t start correctly.\n&#8211; <strong>Fix:<\/strong>\n  &#8211; Verify whether the template uses a managed EC2 resource and whether it launched.\n  &#8211; Check EC2 limits (vCPU\/instance count).\n  &#8211; Check VPC\/subnet settings if using a private subnet (needs outbound HTTPS).\n  &#8211; Review pipeline logs in S3.<\/p>\n\n\n\n<p>2) <strong>AccessDenied to S3 (input\/output\/logs)<\/strong>\n&#8211; <strong>Cause:<\/strong> Resource role doesn\u2019t have permission to read from source, write to destination, and write logs.\n&#8211; <strong>Fix:<\/strong>\n  &#8211; Ensure the resource role policy allows <code>s3:GetObject<\/code> on the source prefix and <code>s3:PutObject<\/code> on destination\/log prefixes.\n  &#8211; If SSE-KMS is enabled, ensure the role has <code>kms:Encrypt<\/code>, <code>kms:Decrypt<\/code>, and <code>kms:GenerateDataKey<\/code> for the relevant key.<\/p>\n\n\n\n<p>3) <strong>Bucket policy blocks access<\/strong>\n&#8211; <strong>Cause:<\/strong> Explicit deny in bucket policy or organization SCP.\n&#8211; <strong>Fix:<\/strong> Update bucket policy (or SCP) to allow the pipeline resource role.<\/p>\n\n\n\n<p>4) <strong>Objects copied but with unexpected key names<\/strong>\n&#8211; <strong>Cause:<\/strong> Template may preserve full paths or apply a prefix mapping.\n&#8211; <strong>Fix:<\/strong> Review template settings for input\/output folder semantics; test with a single file and check exact output.<\/p>\n\n\n\n<p>5) <strong>Unexpected recurring runs<\/strong>\n&#8211; <strong>Cause:<\/strong> You selected a recurring schedule.\n&#8211; <strong>Fix:<\/strong> Deactivate the pipeline immediately after your successful validation.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Cleanup<\/h3>\n\n\n\n<p>To avoid ongoing charges and clutter:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>\n<p><strong>Deactivate and delete the pipeline<\/strong>\n   &#8211; In the AWS Data Pipeline console:<\/p>\n<ol>\n<li>Open your pipeline <code>dp-lab-s3-copy<\/code><\/li>\n<li>Choose <strong>Deactivate<\/strong><\/li>\n<li>After it is inactive, choose <strong>Delete<\/strong><\/li>\n<\/ol>\n<\/li>\n<li>\n<p><strong>Confirm no EC2 instances remain<\/strong>\n   &#8211; In the EC2 console, ensure any instances created for this lab are terminated.\n   &#8211; If an instance is still running, terminate it.<\/p>\n<\/li>\n<li>\n<p><strong>Delete S3 objects and buckets<\/strong>\n   &#8211; Delete contents of:<\/p>\n<ul>\n<li><code>s3:\/\/dp-lab-src-&lt;suffix&gt;\/input\/<\/code><\/li>\n<li><code>s3:\/\/dp-lab-src-&lt;suffix&gt;\/pipeline-logs\/<\/code><\/li>\n<li><code>s3:\/\/dp-lab-dst-&lt;suffix&gt;\/output\/<\/code><\/li>\n<li>Then delete both buckets.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<p>Optional CLI cleanup:<\/p>\n\n\n\n<pre><code class=\"language-bash\">aws s3 rm s3:\/\/dp-lab-src-&lt;suffix&gt; --recursive\naws s3 rm s3:\/\/dp-lab-dst-&lt;suffix&gt; --recursive\naws s3api delete-bucket --bucket dp-lab-src-&lt;suffix&gt; --region &lt;region&gt;\naws s3api delete-bucket --bucket dp-lab-dst-&lt;suffix&gt; --region &lt;region&gt;\n<\/code><\/pre>\n\n\n\n<p><strong>Expected outcome:<\/strong> No active pipeline, no running EC2 instances from the lab, and no remaining S3 buckets\/objects.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">11. Best Practices<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Architecture best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Keep data and compute co-located<\/strong> in the same AWS Region to reduce latency and data transfer costs.<\/li>\n<li>Use S3 as a <strong>durable staging layer<\/strong>: land raw data, validate, then transform\/load.<\/li>\n<li>Design activities to be <strong>idempotent<\/strong> where possible (safe to retry without duplicating effects).<\/li>\n<li>Prefer <strong>small, composable pipelines<\/strong> with clear ownership over one monolithic pipeline spanning many teams.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">IAM\/security best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Enforce <strong>least privilege<\/strong> for:<\/li>\n<li>Data Pipeline service role permissions (control plane actions)<\/li>\n<li>Resource role permissions (data plane access to S3, RDS, Redshift, logs, KMS)<\/li>\n<li>Separate roles by environment (dev\/test\/prod) and by data domain.<\/li>\n<li>Prefer controlled secrets handling (see Security Considerations); do not hardcode credentials in definitions if avoidable.<\/li>\n<li>Lock down log buckets:<\/li>\n<li>Block public access<\/li>\n<li>Encryption<\/li>\n<li>Limited write permissions only from pipeline roles<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Cost best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Use <strong>ephemeral resources<\/strong> when feasible and terminate them after completion.<\/li>\n<li>Reduce NAT usage:<\/li>\n<li>Use <strong>S3 Gateway endpoints<\/strong> for S3 access from private subnets.<\/li>\n<li>Keep Task Runner networking requirements in mind (if no VPC endpoint exists for Data Pipeline, you may still need NAT for the control plane).<\/li>\n<li>Add S3 lifecycle policies to pipeline logs.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Performance best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>For large transfers, ensure the chosen activity\/resource pattern is appropriate:<\/li>\n<li>EMR-based approaches can parallelize more effectively for very large datasets.<\/li>\n<li>Avoid tiny file explosions; coalesce outputs where possible.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Reliability best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Use retries judiciously and with exponential backoff patterns where supported.<\/li>\n<li>Add preconditions to ensure upstream data exists before processing.<\/li>\n<li>Make downstream writes atomic when possible:<\/li>\n<li>Write to a temporary prefix, then move\/rename (in S3 this typically means copy+delete; consider how consumers discover \u201cready\u201d data).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Operations best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Standardize on:<\/li>\n<li>Naming conventions for pipelines and objects<\/li>\n<li>Tagging for owner\/team\/cost center\/data classification<\/li>\n<li>Track pipeline definitions in version control and use a change management process.<\/li>\n<li>Use CloudTrail for auditing, and centralize pipeline logs.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Governance\/tagging\/naming best practices<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Pipeline naming: <code>env-domain-purpose-frequency<\/code> (example: <code>prod-finance-rds-to-s3-nightly<\/code>)<\/li>\n<li>Required tags (example):<\/li>\n<li><code>Owner<\/code>, <code>Team<\/code>, <code>CostCenter<\/code>, <code>Environment<\/code>, <code>DataClassification<\/code><\/li>\n<li>Apply consistent S3 prefix structure:<\/li>\n<li><code>s3:\/\/bucket\/domain\/dataset\/dt=YYYY-MM-DD\/<\/code><\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">12. Security Considerations<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Identity and access model<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>IAM controls<\/strong> who can create\/modify\/activate pipelines.<\/li>\n<li><strong>Service role<\/strong> authorizes AWS Data Pipeline to call other AWS services as part of orchestration.<\/li>\n<li><strong>Resource role<\/strong> (instance profile for EC2\/EMR) controls what the executing compute can access (S3, KMS, databases, logs).<\/li>\n<\/ul>\n\n\n\n<p>Security guidance:\n&#8211; Treat the resource role as highly sensitive: it is the effective runtime identity that can read\/write your data.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Encryption<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>At rest<\/strong>:<\/li>\n<li>Use S3 default encryption (SSE-S3 or SSE-KMS).<\/li>\n<li>If EC2 uses EBS volumes, enable EBS encryption (account-level defaults help).<\/li>\n<li><strong>In transit<\/strong>:<\/li>\n<li>Use TLS for service endpoints and database connections.<\/li>\n<li>For RDS\/Redshift connections, enforce SSL where supported.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Network exposure<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Run pipeline resources in a <strong>VPC<\/strong> and prefer private subnets where feasible.<\/li>\n<li>Control egress carefully:<\/li>\n<li>Task Runner needs to communicate with the AWS Data Pipeline endpoint (HTTPS).<\/li>\n<li>If a VPC endpoint is not available for AWS Data Pipeline in your region, plan NAT\/proxy egress and restrict it.<\/li>\n<li>Use <strong>security groups<\/strong> with least privilege:<\/li>\n<li>Only required outbound\/DB ports.<\/li>\n<li>Avoid broad inbound rules; many pipeline patterns require no inbound access.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Secrets handling<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Avoid embedding plaintext credentials in pipeline definitions.<\/li>\n<li>Prefer:<\/li>\n<li>IAM authentication\/roles where possible<\/li>\n<li>AWS Secrets Manager for database credentials (integration depends on your activity pattern; you may need a custom step that fetches secrets at runtime)<\/li>\n<li>If you must provide credentials for a legacy integration, scope them tightly and rotate them; verify the safest supported pattern in the official docs.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Audit\/logging<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Enable <strong>AWS CloudTrail<\/strong> in all regions (or at least the region you use) and send logs to a central, immutable logging account if available.<\/li>\n<li>Configure pipeline execution logs to a dedicated S3 log bucket with:<\/li>\n<li>encryption<\/li>\n<li>object ownership controls<\/li>\n<li>lifecycle retention policies<\/li>\n<li>Consider S3 access logging or CloudTrail data events for sensitive buckets (cost tradeoff).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Compliance considerations<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Data classification: ensure pipeline logs do not inadvertently capture sensitive data payloads.<\/li>\n<li>If processing regulated data:<\/li>\n<li>Confirm region residency requirements<\/li>\n<li>Validate encryption and access controls<\/li>\n<li>Ensure you can demonstrate audit trails (CloudTrail + log retention)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Common security mistakes<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Over-permissive resource role (<code>s3:*<\/code> on <code>*<\/code>, broad KMS permissions)<\/li>\n<li>Writing logs to a bucket with weak policies or no encryption<\/li>\n<li>Running Task Runner in a public subnet with an overly open security group<\/li>\n<li>Uncontrolled cross-account bucket access without clear ownership and monitoring<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Secure deployment recommendations<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Use separate AWS accounts (or at minimum separate roles and buckets) per environment.<\/li>\n<li>Use permission boundaries\/SCPs to prevent accidental broad IAM policies.<\/li>\n<li>Build a pipeline \u201cbaseline\u201d module:<\/li>\n<li>standardized roles<\/li>\n<li>standardized log bucket<\/li>\n<li>encryption defaults<\/li>\n<li>tagging<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">13. Limitations and Gotchas<\/h2>\n\n\n\n<p>AWS Data Pipeline is functional, but you should design with its boundaries in mind.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Known limitations \/ practical constraints<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Legacy service posture:<\/strong> AWS Data Pipeline is older and may not match modern expectations for orchestration UX, integrations, or rapid feature evolution.<\/li>\n<li><strong>Orchestration richness:<\/strong> Complex DAGs, dynamic task mapping, and rich backfill controls are generally better served by MWAA\/Airflow or Step Functions.<\/li>\n<li><strong>Operational visibility:<\/strong> Execution visibility is more limited than modern orchestrators; you often rely on S3 logs and careful monitoring patterns.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Quotas<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Limits exist for number of pipelines, objects per pipeline definition, and\/or concurrency. Do not assume defaults.<\/li>\n<li>Check the AWS Data Pipeline docs and any Service Quotas entries for current values:<\/li>\n<li>https:\/\/docs.aws.amazon.com\/datapipeline\/latest\/DeveloperGuide\/what-is-datapipeline.html<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Regional constraints<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Not all regions support AWS Data Pipeline. Confirm region availability before designing multi-region patterns.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Pricing surprises<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>NAT Gateway charges (private subnet) can exceed compute costs for small workloads.<\/li>\n<li>EMR clusters can be expensive if left running or sized incorrectly.<\/li>\n<li>Cross-region copies incur data transfer charges.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Compatibility issues<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Some activity types are designed around older ecosystems (for example, legacy EMR\/Hadoop workflows). Validate compatibility with your data formats and security posture.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Operational gotchas<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>IAM roles: missing <code>iam:PassRole<\/code> permissions (for the caller) and missing S3\/KMS permissions (for the resource role) are frequent causes of failure.<\/li>\n<li>Logging bucket policies can silently break troubleshooting if writes are denied.<\/li>\n<li>Retries can duplicate work if your processing isn\u2019t idempotent.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Migration challenges<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Moving from AWS Data Pipeline to AWS Glue, Step Functions, or MWAA can require rethinking:<\/li>\n<li>scheduling semantics<\/li>\n<li>how you model dependencies and retries<\/li>\n<li>how you manage runtime environments (serverless vs managed workers)<\/li>\n<li>how you handle secrets and connections<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">14. Comparison with Alternatives<\/h2>\n\n\n\n<p>AWS Data Pipeline is one option among several orchestration and ETL services. The \u201cbest\u201d alternative depends on whether you need ETL transformation, orchestration, managed scheduling, or data transfer.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Comparison table<\/h3>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>Option<\/th>\n<th>Best For<\/th>\n<th>Strengths<\/th>\n<th>Weaknesses<\/th>\n<th>When to Choose<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><strong>AWS Data Pipeline<\/strong><\/td>\n<td>Legacy batch data movement and scheduled workflows<\/td>\n<td>Simple scheduling, built-in activity types, managed EC2\/EMR resource patterns<\/td>\n<td>Legacy posture, less flexible DAG features, fewer modern integrations<\/td>\n<td>Maintain existing pipelines; simple batch orchestration where it already fits<\/td>\n<\/tr>\n<tr>\n<td><strong>AWS Glue (Jobs\/Workflows)<\/strong><\/td>\n<td>Serverless ETL and data preparation<\/td>\n<td>Serverless Spark, integration with Glue Data Catalog, modern ETL patterns<\/td>\n<td>Learning curve; not always ideal for non-ETL orchestration<\/td>\n<td>New ETL pipelines, lakehouse ingestion\/transform<\/td>\n<\/tr>\n<tr>\n<td><strong>AWS Step Functions<\/strong><\/td>\n<td>Orchestrating AWS services and microservices<\/td>\n<td>Clear state machines, retries\/timeouts, service integrations<\/td>\n<td>Not an ETL engine; large data moves require additional services<\/td>\n<td>Event-driven workflows, coordination across Lambda\/ECS\/Batch\/Glue<\/td>\n<\/tr>\n<tr>\n<td><strong>Amazon MWAA (Managed Airflow)<\/strong><\/td>\n<td>Complex DAG orchestration with Airflow ecosystem<\/td>\n<td>Rich scheduling, DAG UI, plugins\/operators<\/td>\n<td>Operates an Airflow environment; cost\/ops overhead<\/td>\n<td>Standardize on Airflow; complex dependency graphs; multi-team orchestration<\/td>\n<\/tr>\n<tr>\n<td><strong>AWS DataSync<\/strong><\/td>\n<td>Managed data transfer (file\/object)<\/td>\n<td>Efficient transfers, scheduling, agents for on-prem<\/td>\n<td>Not a general workflow orchestrator<\/td>\n<td>Large-scale data movement between on-prem\/NFS\/S3\/EFS\/FSx<\/td>\n<\/tr>\n<tr>\n<td><strong>AWS Batch \/ ECS<\/strong><\/td>\n<td>Batch compute execution<\/td>\n<td>Scales compute, job queues<\/td>\n<td>Orchestration and dependencies need extra tooling<\/td>\n<td>Compute-heavy batch jobs, containerized workloads<\/td>\n<\/tr>\n<tr>\n<td><strong>Azure Data Factory<\/strong><\/td>\n<td>Managed data integration in Azure<\/td>\n<td>Broad connectors, UI-driven pipelines<\/td>\n<td>Different cloud; migration effort<\/td>\n<td>If your data platform is primarily on Azure<\/td>\n<\/tr>\n<tr>\n<td><strong>Google Cloud Dataflow<\/strong><\/td>\n<td>Stream\/batch data processing (Apache Beam)<\/td>\n<td>Unified model, scalable processing<\/td>\n<td>Different cloud; not primarily an orchestrator<\/td>\n<td>Beam-based pipelines in GCP<\/td>\n<\/tr>\n<tr>\n<td><strong>Apache Airflow (self-managed)<\/strong><\/td>\n<td>Full control over orchestration<\/td>\n<td>Flexibility, huge ecosystem<\/td>\n<td>You manage infra and upgrades<\/td>\n<td>When you need maximum customization and accept ops burden<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">15. Real-World Example<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Enterprise example (regulated industry)<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem:<\/strong> A financial services company has a legacy nightly batch process that:\n  1) extracts data from an operational database,\n  2) lands it in S3,\n  3) runs transformations,\n  4) loads a warehouse for reporting by 7 AM.<\/li>\n<li><strong>Proposed architecture:<\/strong><\/li>\n<li>AWS Data Pipeline orchestrates a scheduled workflow.<\/li>\n<li>S3 is the staging layer; logs also go to a secured S3 log prefix.<\/li>\n<li>Compute runs on short-lived EC2\/EMR resources inside a private VPC.<\/li>\n<li>IAM roles enforce least privilege; buckets are SSE-KMS encrypted; CloudTrail enabled.<\/li>\n<li><strong>Why AWS Data Pipeline was chosen:<\/strong><\/li>\n<li>Existing investment and operational familiarity.<\/li>\n<li>The workflow pattern aligns with Data Pipeline templates and activity types.<\/li>\n<li><strong>Expected outcomes:<\/strong><\/li>\n<li>Reduced operational toil vs cron scripts<\/li>\n<li>Better auditability (CloudTrail + centralized logs)<\/li>\n<li>Controlled retries and predictable scheduling<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Startup \/ small-team example<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Problem:<\/strong> A small SaaS company needs a daily export of usage events from S3 raw logs into a curated S3 prefix for BI tools.<\/li>\n<li><strong>Proposed architecture:<\/strong><\/li>\n<li>AWS Data Pipeline runs once daily and copies raw partitions into curated prefixes (or triggers a lightweight transform step).<\/li>\n<li>Uses a small, short-lived EC2 resource; pipeline logs go to S3 with lifecycle policies.<\/li>\n<li><strong>Why AWS Data Pipeline was chosen:<\/strong><\/li>\n<li>The team already has a legacy pipeline and wants minimal changes.<\/li>\n<li>Simple scheduled batch movement is the primary requirement.<\/li>\n<li><strong>Expected outcomes:<\/strong><\/li>\n<li>Automated daily dataset availability for analysts<\/li>\n<li>Low-touch operations and easy rollback (deactivate pipeline)<\/li>\n<\/ul>\n\n\n\n<blockquote>\n<p>For many startups building new platforms today, AWS Glue or Step Functions is often a better long-term fit. AWS Data Pipeline can still be workable for narrow, stable batch needs\u2014especially when maintaining existing systems.<\/p>\n<\/blockquote>\n\n\n\n<h2 class=\"wp-block-heading\">16. FAQ<\/h2>\n\n\n\n<p>1) <strong>Is AWS Data Pipeline still available?<\/strong><br\/>\nAWS Data Pipeline is still present in AWS, but it is widely treated as a legacy service for new architectures. Always confirm current AWS guidance and region availability in the official docs: https:\/\/docs.aws.amazon.com\/datapipeline\/latest\/DeveloperGuide\/what-is-datapipeline.html<\/p>\n\n\n\n<p>2) <strong>What type of workloads is AWS Data Pipeline best for?<\/strong><br\/>\nBatch workflows: scheduled data movement and batch processing steps (copy, SQL steps, EMR jobs, shell commands).<\/p>\n\n\n\n<p>3) <strong>Is AWS Data Pipeline an ETL service like AWS Glue?<\/strong><br\/>\nNot exactly. AWS Data Pipeline orchestrates and coordinates steps; ETL transformations are typically performed by the compute you run (EC2\/EMR) or by supported activities. AWS Glue is purpose-built for ETL with serverless Spark and catalog integration.<\/p>\n\n\n\n<p>4) <strong>Does AWS Data Pipeline support streaming data?<\/strong><br\/>\nAWS Data Pipeline is primarily for batch. For streaming, consider services like Amazon Kinesis, Amazon MSK, or near-real-time architectures orchestrated with Step Functions and event triggers.<\/p>\n\n\n\n<p>5) <strong>Where do pipeline logs go?<\/strong><br\/>\nCommonly to an S3 location configured on the pipeline (for example, <code>pipelineLogUri<\/code>). Confirm the current logging options in the docs.<\/p>\n\n\n\n<p>6) <strong>What IAM roles does AWS Data Pipeline use?<\/strong><br\/>\nTypically a service role (control plane) and a resource role\/instance profile (data plane). Default roles may be created by the console. Exact naming and permissions should be reviewed and minimized.<\/p>\n\n\n\n<p>7) <strong>Can I run pipeline resources in a private subnet?<\/strong><br\/>\nOften yes for EC2\/EMR resources, but the Task Runner must reach the AWS Data Pipeline endpoint over HTTPS. Private subnets may require NAT\/proxy unless a VPC endpoint is available. Verify networking requirements in your region.<\/p>\n\n\n\n<p>8) <strong>How do retries work, and can retries cause duplicates?<\/strong><br\/>\nRetries re-run failed activities. If your activity is not idempotent (for example, appending to a file or reloading a table without deduplication), retries can duplicate work. Design for idempotency.<\/p>\n\n\n\n<p>9) <strong>Can AWS Data Pipeline copy data across regions?<\/strong><br\/>\nIt can orchestrate copies that involve cross-region endpoints, but cross-region S3 copies incur transfer costs and can be slower. Prefer same-region designs unless you have a clear DR or residency requirement.<\/p>\n\n\n\n<p>10) <strong>How do I version control pipeline definitions?<\/strong><br\/>\nStore pipeline definitions (and any scripts) in a Git repository, promote changes through environments, and apply a review process. Even if you use the console, export definitions where possible and keep them in source control.<\/p>\n\n\n\n<p>11) <strong>Can AWS Data Pipeline integrate with AWS Secrets Manager?<\/strong><br\/>\nNot in the same way as newer services with native secret references. You may need a custom activity step that retrieves secrets at runtime. Validate supported patterns in current docs and follow security best practices.<\/p>\n\n\n\n<p>12) <strong>What\u2019s the best alternative for new orchestration work?<\/strong><br\/>\nOften AWS Step Functions for service orchestration, AWS Glue for ETL, or MWAA for Airflow DAG orchestration\u2014depending on the workload.<\/p>\n\n\n\n<p>13) <strong>Does AWS Data Pipeline provide SLAs and managed high availability?<\/strong><br\/>\nAWS services typically publish availability information in service terms, but you should review current AWS documentation. Your pipeline\u2019s reliability will also depend heavily on the underlying compute\/data services.<\/p>\n\n\n\n<p>14) <strong>How do I troubleshoot a pipeline that won\u2019t run?<\/strong><br\/>\nCheck:\n&#8211; Pipeline status in console\n&#8211; S3 pipeline logs\n&#8211; IAM role permissions (especially S3\/KMS)\n&#8211; Whether EC2\/EMR resources launched successfully\n&#8211; VPC egress connectivity<\/p>\n\n\n\n<p>15) <strong>How do I prevent unexpected costs?<\/strong><br\/>\n&#8211; Deactivate pipelines when not needed\n&#8211; Use small ephemeral compute\n&#8211; Avoid NAT-heavy designs\n&#8211; Apply S3 lifecycle policies to logs\n&#8211; Use cost allocation tags and budgets\/alerts<\/p>\n\n\n\n<p>16) <strong>Can I use AWS Data Pipeline for complex multi-branch DAGs?<\/strong><br\/>\nYou can model dependencies, but if you require advanced branching, dynamic tasks, or sophisticated backfills, MWAA\/Airflow or Step Functions is often a better fit.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">17. Top Online Resources to Learn AWS Data Pipeline<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>Resource Type<\/th>\n<th>Name<\/th>\n<th>Why It Is Useful<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Official Documentation<\/td>\n<td>AWS Data Pipeline Developer Guide<\/td>\n<td>Primary reference for concepts, object model, activities, and configuration: https:\/\/docs.aws.amazon.com\/datapipeline\/latest\/DeveloperGuide\/what-is-datapipeline.html<\/td>\n<\/tr>\n<tr>\n<td>Official Pricing<\/td>\n<td>AWS Data Pipeline Pricing<\/td>\n<td>Current pricing model and dimensions: https:\/\/aws.amazon.com\/datapipeline\/pricing\/<\/td>\n<\/tr>\n<tr>\n<td>Pricing Tool<\/td>\n<td>AWS Pricing Calculator<\/td>\n<td>Estimate end-to-end costs including EC2\/EMR\/S3\/NAT: https:\/\/calculator.aws\/#\/<\/td>\n<\/tr>\n<tr>\n<td>Console Entry Point<\/td>\n<td>AWS Data Pipeline Console<\/td>\n<td>Build and monitor pipelines in the UI: https:\/\/console.aws.amazon.com\/datapipeline\/<\/td>\n<\/tr>\n<tr>\n<td>Security Auditing<\/td>\n<td>AWS CloudTrail User Guide<\/td>\n<td>Audit Data Pipeline API calls and operational changes: https:\/\/docs.aws.amazon.com\/awscloudtrail\/latest\/userguide\/cloudtrail-user-guide.html<\/td>\n<\/tr>\n<tr>\n<td>IAM Reference<\/td>\n<td>IAM Documentation<\/td>\n<td>Create least-privilege policies and role separation: https:\/\/docs.aws.amazon.com\/IAM\/latest\/UserGuide\/introduction.html<\/td>\n<\/tr>\n<tr>\n<td>S3 Security<\/td>\n<td>Amazon S3 Security Best Practices<\/td>\n<td>Secure S3 buckets used for data and logs: https:\/\/docs.aws.amazon.com\/AmazonS3\/latest\/userguide\/security-best-practices.html<\/td>\n<\/tr>\n<tr>\n<td>Architecture Guidance<\/td>\n<td>AWS Architecture Center<\/td>\n<td>Broader AWS data\/analytics reference architectures: https:\/\/aws.amazon.com\/architecture\/<\/td>\n<\/tr>\n<tr>\n<td>Modern Alternatives<\/td>\n<td>AWS Glue Documentation<\/td>\n<td>If migrating or choosing a new ETL service: https:\/\/docs.aws.amazon.com\/glue\/latest\/dg\/what-is-glue.html<\/td>\n<\/tr>\n<tr>\n<td>Modern Alternatives<\/td>\n<td>AWS Step Functions Documentation<\/td>\n<td>For modern orchestration patterns: https:\/\/docs.aws.amazon.com\/step-functions\/latest\/dg\/welcome.html<\/td>\n<\/tr>\n<tr>\n<td>Modern Alternatives<\/td>\n<td>Amazon MWAA Documentation<\/td>\n<td>For managed Apache Airflow: https:\/\/docs.aws.amazon.com\/mwaa\/latest\/userguide\/what-is-mwaa.html<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">18. Training and Certification Providers<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>Institute<\/th>\n<th>Suitable Audience<\/th>\n<th>Likely Learning Focus<\/th>\n<th>Mode<\/th>\n<th>Website URL<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>DevOpsSchool.com<\/td>\n<td>DevOps engineers, cloud engineers, platform teams<\/td>\n<td>AWS operations, CI\/CD, DevOps practices that often support analytics pipelines<\/td>\n<td>Check website<\/td>\n<td>https:\/\/www.devopsschool.com\/<\/td>\n<\/tr>\n<tr>\n<td>ScmGalaxy.com<\/td>\n<td>Beginners to intermediate engineers<\/td>\n<td>DevOps fundamentals, tooling, and process-oriented training<\/td>\n<td>Check website<\/td>\n<td>https:\/\/www.scmgalaxy.com\/<\/td>\n<\/tr>\n<tr>\n<td>CLoudOpsNow.in<\/td>\n<td>Cloud operations teams, SRE\/ops<\/td>\n<td>Cloud ops practices, monitoring, operational readiness<\/td>\n<td>Check website<\/td>\n<td>https:\/\/www.cloudopsnow.in\/<\/td>\n<\/tr>\n<tr>\n<td>SreSchool.com<\/td>\n<td>SREs, reliability-focused engineers<\/td>\n<td>Reliability engineering, operations, incident response<\/td>\n<td>Check website<\/td>\n<td>https:\/\/www.sreschool.com\/<\/td>\n<\/tr>\n<tr>\n<td>AiOpsSchool.com<\/td>\n<td>Ops + automation practitioners<\/td>\n<td>AIOps concepts, automation, operational analytics<\/td>\n<td>Check website<\/td>\n<td>https:\/\/www.aiopsschool.com\/<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">19. Top Trainers<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>Platform\/Site<\/th>\n<th>Likely Specialization<\/th>\n<th>Suitable Audience<\/th>\n<th>Website URL<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>RajeshKumar.xyz<\/td>\n<td>DevOps and cloud training content (verify current offerings)<\/td>\n<td>Individuals and teams seeking practical training<\/td>\n<td>https:\/\/www.rajeshkumar.xyz\/<\/td>\n<\/tr>\n<tr>\n<td>devopstrainer.in<\/td>\n<td>DevOps\/cloud training services (verify current offerings)<\/td>\n<td>Beginners to experienced engineers<\/td>\n<td>https:\/\/www.devopstrainer.in\/<\/td>\n<\/tr>\n<tr>\n<td>devopsfreelancer.com<\/td>\n<td>Freelance DevOps\/automation expertise (verify scope)<\/td>\n<td>Teams needing short-term help or mentoring<\/td>\n<td>https:\/\/www.devopsfreelancer.com\/<\/td>\n<\/tr>\n<tr>\n<td>devopssupport.in<\/td>\n<td>DevOps support and training resources (verify scope)<\/td>\n<td>Ops\/DevOps teams needing guided support<\/td>\n<td>https:\/\/www.devopssupport.in\/<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">20. Top Consulting Companies<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>Company<\/th>\n<th>Likely Service Area<\/th>\n<th>Where They May Help<\/th>\n<th>Consulting Use Case Examples<\/th>\n<th>Website URL<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>cotocus.com<\/td>\n<td>Cloud\/DevOps consulting (verify current service catalog)<\/td>\n<td>Architecture, migrations, operations<\/td>\n<td>Data pipeline modernization, IAM reviews, cost optimization<\/td>\n<td>https:\/\/www.cotocus.com\/<\/td>\n<\/tr>\n<tr>\n<td>DevOpsSchool.com<\/td>\n<td>DevOps and cloud consulting and training<\/td>\n<td>Delivery acceleration, platform engineering<\/td>\n<td>Standardizing CI\/CD for analytics jobs, operational readiness<\/td>\n<td>https:\/\/www.devopsschool.com\/<\/td>\n<\/tr>\n<tr>\n<td>DEVOPSCONSULTING.IN<\/td>\n<td>DevOps consulting (verify current offerings)<\/td>\n<td>Automation, cloud operations<\/td>\n<td>Building deployment pipelines, monitoring and alerting setup<\/td>\n<td>https:\/\/www.devopsconsulting.in\/<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">21. Career and Learning Roadmap<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">What to learn before AWS Data Pipeline<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>AWS basics: IAM, VPC, EC2, S3, CloudWatch, CloudTrail<\/li>\n<li>Data fundamentals: batch vs streaming, partitions, file formats (CSV\/JSON\/Parquet), data lakes vs warehouses<\/li>\n<li>Linux basics and scripting (bash, Python) for shell-based activities<\/li>\n<li>Security fundamentals: least privilege, encryption, logging, key management<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">What to learn after AWS Data Pipeline<\/h3>\n\n\n\n<p>Given the industry shift toward newer services, consider:\n&#8211; <strong>AWS Glue<\/strong> for ETL and data catalog-driven pipelines\n&#8211; <strong>AWS Step Functions<\/strong> for modern orchestration\n&#8211; <strong>Amazon MWAA<\/strong> (Airflow) for DAG-based orchestration at scale\n&#8211; <strong>EventBridge<\/strong> for event-driven scheduling and triggers\n&#8211; Data warehousing\/lakehouse patterns with Redshift, Athena, Lake Formation (as applicable)<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Job roles that use it<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Data Engineer (batch pipelines, orchestration, S3-based lakes)<\/li>\n<li>Cloud Engineer (infrastructure, IAM, networking for pipelines)<\/li>\n<li>DevOps Engineer \/ Platform Engineer (automation and operations)<\/li>\n<li>Analytics Engineer (data modeling and scheduled loads)<\/li>\n<li>SRE\/Operations (monitoring, incident response for data jobs)<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Certification path (AWS)<\/h3>\n\n\n\n<p>AWS Data Pipeline is not a standalone certification topic, but it appears in broader data\/analytics and architecture contexts. Relevant AWS certifications to consider:\n&#8211; AWS Certified Solutions Architect (Associate\/Professional)\n&#8211; AWS Certified Data Engineer (if currently offered under your AWS certification track; verify current certification lineup on AWS Training and Certification)\n&#8211; AWS Certified DevOps Engineer \u2013 Professional<\/p>\n\n\n\n<p>Official certification hub:\n&#8211; https:\/\/aws.amazon.com\/certification\/<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Project ideas for practice<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Build a daily S3 ingestion pipeline with:<\/li>\n<li>input validation (preconditions)<\/li>\n<li>copy to curated prefix<\/li>\n<li>log retention policies<\/li>\n<li>Create a pipeline that triggers a short-lived compute job (EC2\/EMR) and writes output to partitioned S3<\/li>\n<li>Implement least-privilege IAM roles and verify with Access Analyzer and CloudTrail<\/li>\n<li>Cost optimization exercise:<\/li>\n<li>compare NAT vs VPC endpoints for S3 access<\/li>\n<li>compare EC2 sizing and runtime<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">22. Glossary<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Activity:<\/strong> A unit of work in AWS Data Pipeline (copy, shell command, SQL, EMR job, etc.).<\/li>\n<li><strong>Data node:<\/strong> A definition of a data location\/source\/target (for example, an S3 prefix or database table).<\/li>\n<li><strong>Pipeline:<\/strong> The overall workflow container in AWS Data Pipeline.<\/li>\n<li><strong>Pipeline definition:<\/strong> The declarative set of objects (activities, data nodes, schedules, resources) that describe the pipeline.<\/li>\n<li><strong>Resource:<\/strong> Compute environment used to run activities (for example, EC2 instance or EMR cluster).<\/li>\n<li><strong>Task Runner:<\/strong> The agent process that polls AWS Data Pipeline for tasks and executes them.<\/li>\n<li><strong>Schedule:<\/strong> Configuration that determines when activities run.<\/li>\n<li><strong>Precondition:<\/strong> A condition that must be met before an activity can run (for example, data availability).<\/li>\n<li><strong>Idempotent:<\/strong> A task is idempotent if running it multiple times produces the same result without harmful duplicates.<\/li>\n<li><strong>Least privilege:<\/strong> Security principle of granting only the minimum permissions needed.<\/li>\n<li><strong>SSE-S3 \/ SSE-KMS:<\/strong> Server-side encryption options for S3 (S3-managed keys vs AWS KMS keys).<\/li>\n<li><strong>NAT Gateway:<\/strong> A managed AWS service enabling outbound internet access for private subnet resources; can be a significant cost driver.<\/li>\n<li><strong>CloudTrail:<\/strong> AWS service that records API calls for auditing and security investigation.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">23. Summary<\/h2>\n\n\n\n<p>AWS Data Pipeline (AWS Analytics) is a managed batch workflow orchestration service used to define, schedule, and run data movement and processing steps across AWS services. It matters most for organizations running legacy batch orchestration patterns\u2014especially where templates, simple scheduling, and EC2\/EMR-based execution fit existing operational models.<\/p>\n\n\n\n<p>Cost-wise, the main drivers are typically the underlying compute (EC2\/EMR), data transfer (especially NAT and cross-region), and storage\/logging (S3 + KMS). Security-wise, success depends on well-scoped IAM roles (service vs resource role separation), protected log buckets, encryption, and auditable operations via CloudTrail.<\/p>\n\n\n\n<p>Use AWS Data Pipeline when you need straightforward batch orchestration or you are maintaining existing pipelines. For new designs, evaluate AWS Glue, AWS Step Functions, or Amazon MWAA for a more modern orchestration and data engineering experience.<\/p>\n\n\n\n<p>Next step: read the official AWS Data Pipeline Developer Guide, then compare an equivalent workflow implemented in AWS Glue or Step Functions to understand tradeoffs in operations, cost, and long-term fit:<br\/>\nhttps:\/\/docs.aws.amazon.com\/datapipeline\/latest\/DeveloperGuide\/what-is-datapipeline.html<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Analytics<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21,20],"tags":[],"class_list":["post-118","post","type-post","status-publish","format-standard","hentry","category-analytics","category-aws"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/118","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/comments?post=118"}],"version-history":[{"count":0,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/posts\/118\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/media?parent=118"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/categories?post=118"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/tutorials\/wp-json\/wp\/v2\/tags?post=118"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}