Arkinu
AWS service comparison

Amazon Kinesis vs Amazon Managed Streaming for Apache Kafka (Amazon MSK): differences and when to use each

Amazon Kinesis
Amazon Managed Streaming for Apache Kafka (Amazon MSK)

Often confused on the examQuestions tend to offer one as a distractor for the other: look for the phrase in the scenario that tips the balance.

Analytics

Amazon Kinesis

Family of services for streaming data. Kinesis Data Streams collects and processes streams of records in real time with your own consumer applications, and Amazon Data Firehose delivers streaming data, without writing applications, to destinations such as Amazon S3, Amazon Redshift, or Amazon OpenSearch Service.

Exam phrases that point here

  • «collect and process streaming data in real time»

    Kinesis Data Streams ingests continuous streams of records (clicks, logs, telemetry) and makes them available to consumer applications within seconds or less.

    When not: If you only need to move that data to Amazon S3, Amazon Redshift, or Amazon OpenSearch Service without writing consumers, and a delay of a few seconds or minutes is acceptable, Amazon Data Firehose is a better fit. If the company already uses Apache Kafka, think of Amazon MSK; if what arrives is video, think of Amazon Kinesis Video Streams.

  • «multiple applications must read the same data at the same time»

    In a data stream, each consumer application reads the full stream independently: one can compute metrics while another archives, without taking records away from each other.

    When not: If each message must be processed by a single worker and then disappear, the pattern is a queue (Amazon SQS). If the message must be pushed to multiple subscribers as a notification, think of Amazon SNS.

  • «reprocess the data from the last few days»

    Kinesis Data Streams keeps records for the retention period, so a consumer can read them again from an earlier point, for example after fixing a bug.

    When not: The default retention is 24 hours; for longer you have to extend it, and that costs money. If the goal is to keep the history for months or years so it can be queried, it is usually archived in Amazon S3 (for example, with Firehose).

  • «load streaming data into Amazon S3, Amazon Redshift, or Amazon OpenSearch Service without managing applications»

    Amazon Data Firehose is a managed delivery service: producers send data, and Firehose batches it, optionally transforms it, and delivers it to the configured destination.

    When not: Firehose accumulates data by size or by time before delivering it, so it is near real time, not instantaneous. If you need to process each record with your own logic, with multiple consumers or with reprocessing, use Kinesis Data Streams.

  • «convert the data to Apache Parquet before storing it in Amazon S3»

    Firehose can convert JSON records to columnar formats such as Parquet or ORC and partition the data in S3 during delivery, which makes later queries cheaper and faster.

    When not: If the transformation is a complex batch ETL over data that is already stored, AWS Glue is a better fit. If it is enough to query the data as it sits in S3, Amazon Athena can read it without prior conversion.

  • «process each device's records in order»

    The partition key decides which shard each record goes to; records with the same key go to the same shard and are read in the order they were written.

    When not: If what you need is ordering plus per-message acknowledgment and deletion by a single consumer, an Amazon SQS FIFO queue with a message group ID may be a better fit.

Often paired with

  • AWS Lambda
  • Amazon S3
  • Amazon Redshift
  • Amazon OpenSearch Service
Study Amazon Kinesis →

Analytics

Amazon Managed Streaming for Apache Kafka (Amazon MSK)

Fully managed service for building and running applications that use Apache Kafka to process streaming data. It runs open source versions of Kafka, so existing applications and tools work without code changes.

Exam phrases that point here

  • «migrate applications that already use Apache Kafka to AWS without changing their code»

    Amazon MSK runs open-source Apache Kafka, so existing Kafka producers, consumers, and tools keep working: only the cluster address changes.

    When not: If Kafka is not involved and you are starting streaming from scratch, Amazon Kinesis Data Streams is usually the simplest option. If the application uses messaging protocols such as JMS or AMQP with a traditional broker, think of Amazon MQ.

  • «reduce the work of operating Apache Kafka clusters»

    MSK takes care of provisioning brokers, replacing the ones that fail, replicating data across Availability Zones, applying patches, and upgrading versions.

    When not: With MSK Provisioned you still choose the broker type and number. If the scenario asks for no sizing or scaling of capacity at all, it points to MSK Serverless.

  • «run Kafka without sizing or scaling the cluster capacity»

    MSK Serverless provisions and scales capacity on its own, distributes the partitions, and charges for the data that is streamed and retained.

    When not: MSK Serverless requires IAM authentication and does not support Kafka ACLs. If you need other authentication methods or fine-grained control of the broker configuration, MSK Provisioned fits.

  • «move data between Kafka and databases or Amazon S3 with managed connectors»

    MSK Connect runs Kafka Connect connectors with no servers to manage: source connectors bring data from external systems into topics and sink connectors send it out to other systems.

    When not: If you only need to move data from an MSK topic to S3 with no code, Amazon Data Firehose can also read from MSK and deliver to S3. If there is no Kafka and the task is to migrate or replicate a database, think of AWS DMS.

  • «replicate Kafka topics to another Region to withstand a Regional outage»

    MSK Replicator copies data, topic configuration, and consumer group offsets between MSK clusters, including across Regions, with active-active or active-passive topologies.

    When not: MSK Replicator only replicates between clusters in the same account. And if what you need is high availability within a Region, the replication across Availability Zones that the cluster already does is enough.

Study Amazon Managed Streaming for Apache Kafka (Amazon MSK) →