Analytics
Amazon Kinesis
Family of services for streaming data. Kinesis Data Streams collects and processes streams of records in real time with your own consumer applications, and Amazon Data Firehose delivers streaming data, without writing applications, to destinations such as Amazon S3, Amazon Redshift, or Amazon OpenSearch Service.
Exam phrases that point here
«collect and process streaming data in real time»
Kinesis Data Streams ingests continuous streams of records (clicks, logs, telemetry) and makes them available to consumer applications within seconds or less.
When not: If you only need to move that data to Amazon S3, Amazon Redshift, or Amazon OpenSearch Service without writing consumers, and a delay of a few seconds or minutes is acceptable, Amazon Data Firehose is a better fit. If the company already uses Apache Kafka, think of Amazon MSK; if what arrives is video, think of Amazon Kinesis Video Streams.
«multiple applications must read the same data at the same time»
In a data stream, each consumer application reads the full stream independently: one can compute metrics while another archives, without taking records away from each other.
When not: If each message must be processed by a single worker and then disappear, the pattern is a queue (Amazon SQS). If the message must be pushed to multiple subscribers as a notification, think of Amazon SNS.
«reprocess the data from the last few days»
Kinesis Data Streams keeps records for the retention period, so a consumer can read them again from an earlier point, for example after fixing a bug.
When not: The default retention is 24 hours; for longer you have to extend it, and that costs money. If the goal is to keep the history for months or years so it can be queried, it is usually archived in Amazon S3 (for example, with Firehose).
«load streaming data into Amazon S3, Amazon Redshift, or Amazon OpenSearch Service without managing applications»
Amazon Data Firehose is a managed delivery service: producers send data, and Firehose batches it, optionally transforms it, and delivers it to the configured destination.
When not: Firehose accumulates data by size or by time before delivering it, so it is near real time, not instantaneous. If you need to process each record with your own logic, with multiple consumers or with reprocessing, use Kinesis Data Streams.
«convert the data to Apache Parquet before storing it in Amazon S3»
Firehose can convert JSON records to columnar formats such as Parquet or ORC and partition the data in S3 during delivery, which makes later queries cheaper and faster.
When not: If the transformation is a complex batch ETL over data that is already stored, AWS Glue is a better fit. If it is enough to query the data as it sits in S3, Amazon Athena can read it without prior conversion.
«process each device's records in order»
The partition key decides which shard each record goes to; records with the same key go to the same shard and are read in the order they were written.
When not: If what you need is ordering plus per-message acknowledgment and deletion by a single consumer, an Amazon SQS FIFO queue with a message group ID may be a better fit.
Often paired with
- AWS Lambda
- Amazon S3
- Amazon Redshift
- Amazon OpenSearch Service