Analytics
AWS Data Pipeline
Web service for automating the movement and transformation of data through workflows in which each task depends on the successful completion of the previous one. It is in maintenance mode and is no longer available to new customers.
Exam phrases that point here
«automate the movement and transformation of data between AWS services on a schedule»
That was the purpose of AWS Data Pipeline: defining scheduled data flows in which one task depends on the previous one finishing successfully.
When not: AWS Data Pipeline is in maintenance mode and no longer accepts new customers. In a current scenario, the answer is usually AWS Glue (ETL and its orchestration), AWS Step Functions (orchestrating multiple services), or Amazon MWAA (workflows in Apache Airflow).
«wait until the source data exists before running the next task»
Data Pipeline preconditions checked, for example, that an S3 key or data in a DynamoDB table existed before launching an activity.
When not: With the current alternatives, this is solved with a check inside the Step Functions workflow or with triggers and sensors (in Glue or Airflow). If the task must start right when an object arrives in S3, an S3 event to EventBridge or Lambda is usually enough.
«run data processing tasks on on-premises servers»
In Data Pipeline, Task Runner could be installed on a server of your own, including an on-premises one, so that it ran the pipeline's activities there.
When not: For new workloads, AWS suggests Step Functions with AWS Systems Manager Run Command (with the SSM agent on the server) or hybrid workflows with Amazon MWAA. If the goal is only to copy files from on premises, think of AWS DataSync.
«periodically copy data between S3, DynamoDB, databases, and Redshift»
Data Pipeline had prebuilt activities to copy data, load into Redshift, run SQL, or launch jobs on EMR, and templates such as exporting a DynamoDB table to S3.
When not: For new customers it is not an option: the migration guide recommends AWS Glue for this kind of copy and transformation. For DynamoDB backups, DynamoDB has its own backup and export features.
Often paired with
- Amazon S3
- Amazon EMR