Arkinu
AWS service comparison

AWS Data Pipeline vs AWS Glue: differences and when to use each

12AWS Data Pipeline
AWS Glue

Often confused on the examQuestions tend to offer one as a distractor for the other: look for the phrase in the scenario that tips the balance.

Analytics

AWS Data Pipeline

Web service for automating the movement and transformation of data through workflows in which each task depends on the successful completion of the previous one. It is in maintenance mode and is no longer available to new customers.

Exam phrases that point here

  • «automate the movement and transformation of data between AWS services on a schedule»

    That was the purpose of AWS Data Pipeline: defining scheduled data flows in which one task depends on the previous one finishing successfully.

    When not: AWS Data Pipeline is in maintenance mode and no longer accepts new customers. In a current scenario, the answer is usually AWS Glue (ETL and its orchestration), AWS Step Functions (orchestrating multiple services), or Amazon MWAA (workflows in Apache Airflow).

  • «wait until the source data exists before running the next task»

    Data Pipeline preconditions checked, for example, that an S3 key or data in a DynamoDB table existed before launching an activity.

    When not: With the current alternatives, this is solved with a check inside the Step Functions workflow or with triggers and sensors (in Glue or Airflow). If the task must start right when an object arrives in S3, an S3 event to EventBridge or Lambda is usually enough.

  • «run data processing tasks on on-premises servers»

    In Data Pipeline, Task Runner could be installed on a server of your own, including an on-premises one, so that it ran the pipeline's activities there.

    When not: For new workloads, AWS suggests Step Functions with AWS Systems Manager Run Command (with the SSM agent on the server) or hybrid workflows with Amazon MWAA. If the goal is only to copy files from on premises, think of AWS DataSync.

  • «periodically copy data between S3, DynamoDB, databases, and Redshift»

    Data Pipeline had prebuilt activities to copy data, load into Redshift, run SQL, or launch jobs on EMR, and templates such as exporting a DynamoDB table to S3.

    When not: For new customers it is not an option: the migration guide recommends AWS Glue for this kind of copy and transformation. For DynamoDB backups, DynamoDB has its own backup and export features.

Often paired with

  • Amazon S3
  • Amazon EMR
Study AWS Data Pipeline →

Analytics

AWS Glue

Serverless data integration service for discovering, preparing, moving, and integrating data from many sources. It provides ETL jobs and a centralized Data Catalog whose data can be queried with Amazon Athena, Amazon EMR, and Amazon Redshift Spectrum.

Exam phrases that point here

  • «serverless ETL service to prepare data for analytics»

    AWS Glue runs extract, transform, and load jobs with no clusters to provision: you define the job and the service provides the capacity and releases it when the job finishes.

    When not: If you need direct control of the cluster or Hadoop ecosystem tools beyond Spark, Amazon EMR is a better fit. If the requirement is to migrate a database to AWS, the answer is AWS DMS, not Glue.

  • «automatically discover the schema of the data stored in S3»

    A Glue crawler scans the data store, infers the format and schema, and creates or updates the tables in the Data Catalog, including partitions.

    When not: If what needs to be discovered is sensitive data (such as personal information) inside S3 objects, the right service is usually Amazon Macie, not a crawler.

  • «a central metadata catalog for Athena, EMR, and Redshift Spectrum»

    The AWS Glue Data Catalog stores table definitions once, and Athena, EMR, Redshift Spectrum, and Glue jobs all use it. It is compatible with Hive Metastore, so it can replace it.

    When not: If you also need to control who sees which tables, columns, or rows of the data lake, the piece that solves it is AWS Lake Formation, which works on top of that same catalog.

  • «clean and normalize data without writing code»

    AWS Glue DataBrew and the Glue Studio visual interface let you prepare data with clicks: DataBrew comes with hundreds of prebuilt transformations and Glue Studio generates the Spark code.

    When not: If data preparation is part of a complete workflow to build, train, and deploy machine learning models, the scenario may point to Amazon SageMaker AI.

  • «complex transformations on streaming data before storing it»

    Glue can read streams from Amazon Kinesis Data Streams or Apache Kafka and apply Spark ETL to them, including joins and partitioned writes.

    When not: If it is enough to deliver the stream to S3, Redshift, or OpenSearch with light transformations, the Glue FAQ recommends Amazon Data Firehose (formerly Kinesis Data Firehose), which is simpler.

Often paired with

  • Amazon S3
  • Amazon Athena
  • AWS Lake Formation
Study AWS Glue →