Arkinu
AWS service comparison

Amazon Kinesis vs Amazon SQS: differences and when to use each

Amazon Kinesis
Amazon SQS

Often confused on the examQuestions tend to offer one as a distractor for the other: look for the phrase in the scenario that tips the balance.

Analytics

Amazon Kinesis

Family of services for streaming data. Kinesis Data Streams collects and processes streams of records in real time with your own consumer applications, and Amazon Data Firehose delivers streaming data, without writing applications, to destinations such as Amazon S3, Amazon Redshift, or Amazon OpenSearch Service.

Exam phrases that point here

  • «collect and process streaming data in real time»

    Kinesis Data Streams ingests continuous streams of records (clicks, logs, telemetry) and makes them available to consumer applications within seconds or less.

    When not: If you only need to move that data to Amazon S3, Amazon Redshift, or Amazon OpenSearch Service without writing consumers, and a delay of a few seconds or minutes is acceptable, Amazon Data Firehose is a better fit. If the company already uses Apache Kafka, think of Amazon MSK; if what arrives is video, think of Amazon Kinesis Video Streams.

  • «multiple applications must read the same data at the same time»

    In a data stream, each consumer application reads the full stream independently: one can compute metrics while another archives, without taking records away from each other.

    When not: If each message must be processed by a single worker and then disappear, the pattern is a queue (Amazon SQS). If the message must be pushed to multiple subscribers as a notification, think of Amazon SNS.

  • «reprocess the data from the last few days»

    Kinesis Data Streams keeps records for the retention period, so a consumer can read them again from an earlier point, for example after fixing a bug.

    When not: The default retention is 24 hours; for longer you have to extend it, and that costs money. If the goal is to keep the history for months or years so it can be queried, it is usually archived in Amazon S3 (for example, with Firehose).

  • «load streaming data into Amazon S3, Amazon Redshift, or Amazon OpenSearch Service without managing applications»

    Amazon Data Firehose is a managed delivery service: producers send data, and Firehose batches it, optionally transforms it, and delivers it to the configured destination.

    When not: Firehose accumulates data by size or by time before delivering it, so it is near real time, not instantaneous. If you need to process each record with your own logic, with multiple consumers or with reprocessing, use Kinesis Data Streams.

  • «convert the data to Apache Parquet before storing it in Amazon S3»

    Firehose can convert JSON records to columnar formats such as Parquet or ORC and partition the data in S3 during delivery, which makes later queries cheaper and faster.

    When not: If the transformation is a complex batch ETL over data that is already stored, AWS Glue is a better fit. If it is enough to query the data as it sits in S3, Amazon Athena can read it without prior conversion.

  • «process each device's records in order»

    The partition key decides which shard each record goes to; records with the same key go to the same shard and are read in the order they were written.

    When not: If what you need is ordering plus per-message acknowledgment and deletion by a single consumer, an Amazon SQS FIFO queue with a message group ID may be a better fit.

Often paired with

  • AWS Lambda
  • Amazon S3
  • Amazon Redshift
  • Amazon OpenSearch Service
Study Amazon Kinesis →

Application Integration

Amazon SQS

Managed message queue that stores messages until a consumer processes and deletes them, or until their retention period expires (14 days maximum). It is used to decouple the components of a distributed system and to absorb load spikes as a request buffer.

Exam phrases that point here

  • «decouple the application components»

    A queue sits between whoever produces the work and whoever processes it: the producer leaves the message and moves on, and the consumer picks it up when it can. If either one fails or slows down, the other does not notice.

    When not: If the same message must reach several targets at once, the pattern is publish/subscribe (Amazon SNS), and if you need to route events based on their content or react to events from AWS or third-party services, Amazon EventBridge is usually a better fit.

  • «absorb traffic spikes without losing requests»

    The queue stores the messages while the consumers cannot keep up, so the spike turns into a processing delay instead of lost requests.

    When not: It only works if the work can be done asynchronously. If the user needs the response right away, the solution is to scale the tier that responds (for example, with a load balancer and Auto Scaling), not to queue.

  • «process messages in the exact order they were sent and without duplicates»

    Amazon SQS FIFO queues preserve ordering within each message group and prevent processing the same message twice.

    When not: A standard queue guarantees neither ordering nor the absence of duplicates. And if several consumers must read the same data stream or reprocess it in real time, think of Amazon Kinesis Data Streams, not a queue.

  • «isolate messages that could not be processed so they can be analyzed later»

    A dead-letter queue receives the messages that consumers failed to process after several attempts, so they do not block the queue and can be reviewed.

    When not: If the scenario asks to retry or capture failures of an asynchronously invoked AWS Lambda function, or of an AWS Step Functions workflow, the answer may lie in that service's configuration and not in an SQS queue.

  • «each task must be processed by a single worker»

    In a queue, a received message is hidden from the other consumers for the duration of the visibility timeout; the worker that processes it deletes it when done.

    When not: If several systems must be notified of the task at once, each with its own copy, the clue points to Amazon SNS (often combined with several SQS queues).

Often paired with

  • AWS Lambda
  • Amazon SNS
Study Amazon SQS →