Analytics
Amazon EMR
Managed cluster platform for running big data frameworks such as Apache Hadoop and Apache Spark. It is used to process and analyze large volumes of data and to move it into and out of data stores such as Amazon S3 and Amazon DynamoDB.
Exam phrases that point here
«process large volumes of data with Apache Spark, Hadoop, or Hive»
Amazon EMR runs open-source big data frameworks (Spark, Hive, Trino, Flink, among others) on managed clusters, on EKS, or in serverless mode.
When not: If you only need ETL without managing infrastructure, AWS Glue is usually the answer. If it's interactive SQL queries over S3, Amazon Athena.
«full control over the cluster and the installed software»
EMR on EC2 lets you choose instance types, install your own software with bootstrap actions, connect over SSH to the primary node, and tune the Hadoop and Spark configuration.
When not: If the scenario prioritizes not managing servers, the option is EMR Serverless or AWS Glue; cluster control is precisely what you give up in exchange.
«reduce costs with Spot Instances in a batch processing job that tolerates interruptions»
In EMR, task nodes only provide compute and don't store HDFS data, so they usually run on Spot Instances: if AWS reclaims them, no HDFS data is lost. Whether the job keeps going depends on its main process (the application master) not running on those nodes, which is configured depending on the EMR version.
When not: Core nodes store HDFS and removing them can lose data; putting them on Spot requires care. If the batch work is not big data but container jobs or scripts, think of AWS Batch.
«a cluster that shuts itself down when the job finishes»
With automatic termination, the cluster shuts down when all its steps are done (a transient cluster); since the input and output data are in S3, nothing is lost.
When not: If you need to run Spark every now and then without creating or shutting down clusters, EMR Serverless or AWS Glue remove that management entirely.
«run Spark applications without provisioning or managing clusters»
EMR Serverless runs Spark and Hive, allocating and releasing workers per job, with no clusters to size.
When not: For serverless ETL with an integrated catalog, AWS Glue also fits; for interactive analysis with Spark in Python notebooks without managing resources, there is also Athena for Apache Spark.
Often paired with
- Amazon S3
- Amazon DynamoDB