AWS Data Services: Choosing Between Them

Scope — the service-selection round of a senior data engineering interview on AWS: which service owns which layer, and how to choose and defend one against its look-alikes — S3 vs EBS vs EFS, RDS vs DynamoDB vs Redshift, Kinesis Data Streams vs Firehose vs MSK, SQS vs SNS vs EventBridge vs Kinesis, Glue vs EMR vs Lambda, Athena vs Redshift, Step Functions vs MWAA vs EventBridge, the Glue Data Catalog vs Lake Formation and the layers that decide data access, what to monitor and what drives cost per layer, three classic architecture scenarios, and an answer structure for “why did you choose X?”. The depth on Glue, EMR, Iceberg, Athena, RDS extraction, S3 + SQS + Lambda ingestion and S3 layout is aws_data.md’s; Kafka internals are kafka.md’s; streaming semantics are stream_pipeline.md’s. See also: ./aws_data.md — the mechanism behind every Glue, EMR, Athena, Iceberg, RDS, SQS and Lambda answer here; ./kafka.md — what MSK runs; ./stream_pipeline.md — event time, watermarks, exactly-once; ./sql_vs_nosql.md — choosing a store, engine-neutral; ../db/faq_redshift.md — the Redshift notes; ../faq_DE_interview.md — the overview and last-minute checklist.

AWS has two or three services for almost every job, and the interview question is never “what is Kinesis?” — it is “why Kinesis and not Firehose, or MSK?” So this file is organised by decision, not by service: each section is one family of look-alikes, the table that separates them, the tell that picks one, and the wrong thing people say about them. Learn the service names last; learn the trade-off each one makes first.

0) The Short Answer ★★★★★Priority 5 of 5 — Must know — expect it in almost every loop

“S3 is the lake; everything else is a choice about how data gets in, how it is transformed, how it is queried and how it is run. I pick by the requirement — latency, volume, who consumes it, how much control the team needs, and what it costs to operate — and I say what I would use instead if that requirement changed.”

When asked “why did you choose service X?”, answer in six beats:

1. Requirement — the volume, latency, throughput and SLA that constrained it.
2. Choice — what X does that the requirement needs.
3. Alternative — the look-alike, and the tell that ruled it out.
4. Trade-offs — cost, operations, performance, reliability.
5. Failure handling — retries, duplicates, data loss, replay.
6. Monitoring — the metric that says the pipeline is healthy.

“I would choose AWS Glue for a standard batch ETL workload because it gives managed Spark execution and integrates with S3 and the Glue Data Catalog, and compared with EMR it removes cluster management. If I needed control over the Spark runtime, dependencies or cluster configuration — or Spot pricing at a scale where it matters — I would move it to EMR. I would benchmark both on a representative day of data, comparing run time, resources and total cost, rather than assume either is faster.”

1) The Map: Which Layer Each Service Owns ★★★★★Priority 5 of 5 — Must know — expect it in almost every loop

 Sources      RDS / Aurora · DynamoDB · SaaS / APIs · application logs · Kafka
                 |
 Ingestion    Kinesis Data Streams · Firehose · MSK · DMS · SQS · AppFlow · Transfer Family
                 |
 Storage      S3 data lake:  raw (bronze)  ->  cleaned (silver)  ->  curated (gold)
                 |
 Processing   Glue ETL · EMR (Spark, Flink, Trino) · Lambda · Athena CTAS
                 |
 Serving      Redshift · Athena · OpenSearch · QuickSight · DynamoDB (lookups)

 Cross-cutting
   orchestration  Step Functions · MWAA (Airflow) · EventBridge (schedules, events)
   catalog        Glue Data Catalog              governance  Lake Formation
   monitoring     CloudWatch · CloudTrail        security    IAM · KMS · Secrets Manager · VPC

Every section below is one row of that picture with more than one candidate in it. The layer that has only one answer — S3 as the lake — is the one to say first, and the one aws_data.md §8 covers in depth.

2) Storage and Databases ★★★★★Priority 5 of 5 — Must know — expect it in almost every loop

2-1) S3 vs EBS vs EFS vs FSx ★★★★☆Priority 4 of 5 — High value — a gap here costs you rounds

“S3 is object storage for the lake; EBS is a disk for one instance; EFS is a shared file system; FSx is a specific file system I need for a specific reason.”

What S3 is, and is not:

• Objects are immutable. You replace an object, you do not edit it — which is why table formats like Iceberg write new files and swap a pointer (aws_data.md §4-2).
• Strongly consistent for reads after writes and for listings (since Dec 2020), so the old “S3 is eventually consistent” caveat is out of date.
• Small files are the classic failure. Every file is a request to list, open and read, so a table of a million 10 KB files costs more and runs slower than the same data in a hundred large Parquet files; and request rates are per prefix (thousands of reads and writes per second each), so a single hot prefix throttles (aws_data.md §8-2).
• Storage classes and Lifecycle move cold data to Infrequent Access and Glacier tiers; Intelligent-Tiering does it per object when the access pattern is unknown.

2-2) OLTP vs NoSQL vs OLAP ★★★★★Priority 5 of 5 — Must know — expect it in almost every loop

Name the workload first; the service follows.

2-3) RDS vs DynamoDB ★★★★★Priority 5 of 5 — Must know — expect it in almost every loop

The design answer splits the data by access pattern, rather than putting everything in the store that scales best:

• accounts and payments — RDS / Aurora, because they need transactions and joins;
• sessions and profile lookups by user id — DynamoDB, because the key is known and the latency matters;
• the full event history — S3, because it is large and append-only;
• analytics over all of it — Athena or Redshift.

Trap. “DynamoDB scales, so put everything there.” A query you did not design a key or index for becomes a full-table scan, and an ad hoc question becomes an export job. The data model and the access patterns decide, not the scaling story.

3) Streaming Ingestion: Kinesis Data Streams, Firehose or MSK ★★★★★Priority 5 of 5 — Must know — expect it in almost every loop

3-1) What each one is

• Kinesis Data Streams (KDS) — a durable, ordered stream. Producers write records with a partition key, records are ordered per shard, and consumers read at their own pace and can re-read within the retention period (24 hours by default, extendable to 365 days).
• Amazon Data Firehose — a managed delivery service. It buffers incoming records and writes them to a destination — S3, Redshift, OpenSearch, Splunk, HTTP endpoints, Iceberg tables — optionally transforming them with a Lambda and converting them to Parquet / ORC on the way. You write no consumer, and you cannot rewind it.
• Amazon MSK — managed Apache Kafka (provisioned brokers or MSK Serverless). Topics, partitions, consumer groups, Kafka Connect (MSK Connect), and the whole Kafka client ecosystem (kafka.md).

3-2) The comparison ★★★★★Priority 5 of 5 — Must know — expect it in almost every loop

3-3) How to choose ★★★★★Priority 5 of 5 — Must know — expect it in almost every loop

• Only landing the data in S3 (or Redshift, OpenSearch)? → Firehose. It batches, can write Parquet, partitions by a field in the record, and needs no code to run.
• Several consumers, your own processing, replay? → Kinesis Data Streams, read by Flink (Amazon Managed Service for Apache Flink), Spark Structured Streaming or Lambda. Add Firehose as one more consumer when you also want the raw stream in S3.
• Already on Kafka, or need Kafka Connect, Kafka’s semantics, or portability off AWS? → MSK.

The latency question: KDS consumers see records in well under a second (around 200 ms with shared-throughput polling, lower with enhanced fan-out). Firehose’s latency is its buffer: it flushes when the buffer reaches a size (1–128 MB) or an interval (0–900 s) — by default 5 MB or 300 s for S3 — so S3 objects arrive in seconds to minutes, not milliseconds.

Traps:

• “Firehose is a cheaper Kafka.” It is a delivery pipe, not an event platform: no consumer groups, no offsets, no replay. When the destination fails it retries: for S3 until the source’s retention runs out — 24 hours for Direct PUT, after which the data is lost; for OpenSearch, Splunk or HTTP for a configured duration, then backs the failed records up to an error prefix in S3; for Redshift it keeps the staged S3 objects and writes a manifest of the skipped ones under errors/, which you COPY to backfill. So alarm on delivery, not only on the error prefix.
• “Kinesis and Kafka are the same thing.” Both are partitioned logs with many consumers and replay, but the scaling unit differs (a shard has fixed throughput; a Kafka partition is bounded by its broker), and so do retention limits, consumer APIs and the ecosystem.
• Hot shard. A low-cardinality partition key (a country, a constant) puts most records on one shard, which throttles (WriteProvisionedThroughputExceeded) while the others sit idle — the streaming version of data skew.
• Duplicates. Producer retries and consumer restarts both re-deliver, on all three; the sink has to be idempotent (aws_data.md §7-5).

4) Queues and Events: SQS, SNS, EventBridge or Kinesis ★★★★☆Priority 4 of 5 — High value — a gap here costs you rounds

4-1) The four shapes ★★★★★Priority 5 of 5 — Must know — expect it in almost every loop

A queue hands each message to one worker; a topic copies it to every subscriber; an event bus routes it by content; a stream keeps it, in order, for anyone to read.

The tells: “each job processed once by one of N workers” is SQS; “every event re-readable by a new consumer next month” is a stream; “when an S3 object lands or a Glue job fails, start X” is EventBridge.

4-2) SQS details the interviewer probes ★★★★☆Priority 4 of 5 — High value — a gap here costs you rounds

• Standard queue — nearly unlimited throughput, at-least-once, best-effort ordering: duplicates and reordering happen.
• FIFO queue — ordering per message group, deduplication within a 5-minute window, and lower throughput limits. Ordering across groups is not promised; one stuck group does not block the others.
• Visibility timeout — after a consumer receives a message it is hidden for this long; not deleted in time, it reappears and runs again. Size it to the processing time (aws_data.md §7-4).
• DLQ — after maxReceiveCount receives, the message moves to a dead-letter queue, so one bad message stops blocking the rest; redrive it back once fixed (aws_data.md §7-6).
• Retention — up to 14 days; a queue is a buffer, not an archive.

4-3) The other ingestion services ★★★☆☆Priority 3 of 5 — Worth knowing — usually a variant of a must-know pattern

5) Processing: Glue, EMR or Lambda ★★★★★Priority 5 of 5 — Must know — expect it in almost every loop

5-1) The three in one table

“Glue to run Spark without a cluster, EMR to control the cluster, Lambda to react to events — not to process the data.”

The full matrix, with Athena and Redshift in it, is aws_data.md §1-2; the Glue-or-EMR decision is §3-4; why Lambda starts the Spark job rather than replacing it is §7-1.

5-2) Choosing ★★★★★Priority 5 of 5 — Must know — expect it in almost every loop

Trap: “Glue is slower” or “EMR is cheaper”, stated as fact. Speed and cost depend on the data volume, the worker or instance types, job shape, startup time and how well the job is tuned. What is true in general: EMR exposes more knobs and Spot pricing, which wins at sustained large scale if someone tunes and operates it; Glue’s per-DPU price buys away that operations work. Say what you would measure, not which one wins.

5-3) The Spark follow-ups ★★★★☆Priority 4 of 5 — High value — a gap here costs you rounds

Choosing Glue or EMR moves the conversation to the engine underneath. Be ready for: reducing shuffle, handling skew, sizing executors, avoiding small files, choosing the partition count, reading the Spark UI for the slow stage, and making a failed job safe to rerun. All of them are spark.md’s and data_performance.md’s; the rerun is batch_pipeline.md’s.

6) Querying: Athena or Redshift ★★★★★Priority 5 of 5 — Must know — expect it in almost every loop

6-1) The comparison

“Athena queries the lake where it lies and bills for what it scans; Redshift is a warehouse I load data into, so repeated heavy queries are fast and predictable.”

Choose Athena when the data is already Parquet in S3, the queries are exploratory or occasional, and nobody wants to run a warehouse. Choose Redshift when a BI workload runs the same heavy joins all day, needs predictable latency under concurrency, or needs workload management. A good answer often uses both: Athena over the lake, Redshift for the gold tables the dashboards hit — and Redshift Spectrum to join the two.

6-2) Tuning each ★★★★☆Priority 4 of 5 — High value — a gap here costs you rounds

Athena — the bill and the latency have the same cause, bytes scanned (aws_data.md §5-2):

• columnar, compressed files (Parquet / ORC);
• partition on the column queries filter by, so partitions are pruned;
• name the columns — no SELECT *;
• compact small files; use Iceberg when the table takes updates;
• a workgroup per team with a per-query scan limit.

Redshift — the layout decides how much data moves between nodes (faq_redshift.md):

• distribution style: KEY on the join column of two large tables, so matching rows sit on the same slice; ALL for small dimensions; EVEN when there is no good key; AUTO to let Redshift choose. A skewed KEY piles work on one slice;
• sort key on the column ranges are filtered by (usually a date), so blocks are skipped;
• materialized views for the aggregates dashboards repeat;
• workload management queues so an analyst’s ad hoc query cannot starve the dashboards;
• EXPLAIN and the system views to find the step that redistributes or broadcasts data.

6-3) The rest of the analytics shelf ★★★☆☆Priority 3 of 5 — Worth knowing — usually a variant of a must-know pattern

7) Orchestration: Step Functions, MWAA or EventBridge ★★★★☆Priority 4 of 5 — High value — a gap here costs you rounds

7-1) The comparison

“Step Functions for a workflow of AWS service calls; MWAA when it is a data pipeline with schedules, dependencies and backfills; EventBridge to start things when something happens.”

7-2) Choosing ★★★★☆Priority 4 of 5 — High value — a gap here costs you rounds

• A few AWS steps with branching and retries (start a Glue job, wait, run Athena, notify) → Step Functions. Standard workflows run for up to a year and execute each step once per attempt — but a Retry policy re-runs a failed task, so the task’s side effect (a write, a job start) must still be idempotent. Express workflows are for short, high-volume event processing.
• Many interdependent pipelines, schedules, backfills of past dates, cross-system tasks → MWAA. Backfilling a date range is what Airflow is built around; the price is an environment that runs, and costs, around the clock.
• “Run when the file lands” or “every day at 02:00” → EventBridge (rule or Scheduler) starting a state machine, a Glue job or a Lambda. It does not track a multi-step workflow.

Glue’s own workflows and triggers also chain Glue jobs and crawlers; they are fine when every step is Glue, and limiting the moment one is not.

8) Catalog, Governance and Access ★★★★★Priority 5 of 5 — Must know — expect it in almost every loop

8-1) The services

Catalog vs Lake Formation in one line: the Data Catalog says what the data is; Lake Formation says who may see which part of it — for the engines integrated with it (Athena, Redshift Spectrum, EMR, Glue).

8-2) Why “I gave it IAM access” is not enough ★★★★★Priority 5 of 5 — Must know — expect it in almost every loop

A read from the lake succeeds only if no layer it crosses denies it and the grants that apply allow it:

 caller's IAM policy        may this role call s3:GetObject / glue:GetTable / athena:...?
 S3 bucket policy           cross-account: must allow this principal; same account: an IAM allow
                            is enough, so the bucket policy restricts only by explicit Deny
 KMS key policy             may this role kms:Decrypt with the key the object was written with?
 Lake Formation grant       does the role hold SELECT on this table / column / row?
 Glue catalog policy        is the catalog resource shared with this account?
 VPC endpoint policy        does the endpoint the request goes through allow it?
 Organizations SCP          does the account's guardrail allow the action at all?

An explicit deny anywhere wins. In the same account, S3 grants access when either the identity policy or the bucket policy allows it; across accounts both must. Either way, a permissions boundary, a session policy or an SCP can still cap it without any deny. The usual debugging miss is KMS: the role can list and get the object, and the read still fails with an access error because it may not decrypt.

8-3) “How do you secure an S3 data lake?” ★★★★★Priority 5 of 5 — Must know — expect it in almost every loop

1. IAM roles per job and service, least privilege, scoped to their prefixes and tables.
2. Bucket policies with explicit Deny statements — for principals outside the expected set, requests without TLS, and requests not through the VPC endpoint. Scope that last deny with exceptions for AWS services that call S3 on your behalf (aws:ViaAWSService, service principals such as Firehose’s role), or it blocks Athena, Lake Formation or delivery. An allow-list alone does not restrict a same-account role whose IAM policy already grants access.
3. Block Public Access on, at the account level.
4. SSE-KMS for sensitive data, with key policies that name who may decrypt.
5. Lake Formation for table-, column- and row-level access, with LF-tags at scale.
6. CloudTrail (plus S3 data events) to audit who did what.
7. Masking or tokenisation of PII before it reaches the wide-access layers.

Encryption is not access control. Encryption protects the bytes if storage leaks; IAM, bucket policies and Lake Formation decide who may read them. An interviewer listens for whether you keep the two apart.

9) Monitoring and Reliability ★★★★☆Priority 4 of 5 — High value — a gap here costs you rounds

9-1) The services

9-2) The metric per layer ★★★★★Priority 5 of 5 — Must know — expect it in almost every loop

Alarm on the SLA, not only on errors: a pipeline that silently processes nothing raises no error. Freshness and volume checks catch it. The longer version is aws_data.md §9-2.

10) Cost per Layer ★★★★☆Priority 4 of 5 — High value — a gap here costs you rounds

The senior version of “use Spot instances.” Saying it is not the answer; the follow-ups are: which work can be interrupted, where the checkpoint is, how a lost task retries, whether reclaimed capacity stretches the run past the SLA, and whether the saving is worth that risk. Cost, reliability and performance get traded together, never one at a time. More levers are in aws_data.md §9-3.

11) Three Architecture Scenarios ★★★★★Priority 5 of 5 — Must know — expect it in almost every loop

11-1) A 1 TB daily batch ★★★★★Priority 5 of 5 — Must know — expect it in almost every loop

 RDS / APIs / CSV / logs
        |
 S3 raw (bronze)  --Glue or EMR Spark-->  S3 silver  -->  S3 gold (Iceberg, Parquet)
                                                               |
                                               Athena (ad hoc)   Redshift (dashboards)
 orchestration: Step Functions or MWAA      monitoring: CloudWatch alarms on freshness

Why: S3 stores raw and processed data cheaply and keeps raw for reprocessing; Spark handles the distributed transform; Parquet means a query reads only the columns it needs; partitions mean it reads only the dates it needs; Redshift gives dashboards a stable SLA; Athena gives everyone else the lake.

The senior follow-ups — how do you make it faster and safe?

• process incrementally (CDC, bookmarks, a watermark) instead of rescanning everything;
• partition on what queries filter by, and do not over-partition;
• compact small files;
• tune the Spark job’s shuffle, joins and partition count;
• make reruns idempotent — MERGE on a key, or overwrite whole partitions — so a retry never duplicates rows;
• data-quality checks before publishing, a retry policy, and a backfill path.

The full worked version, RDS orders to an Iceberg lakehouse, is aws_data.md §11.

11-2) Real-time application logs ★★★★☆Priority 4 of 5 — High value — a gap here costs you rounds

 application logs / CloudWatch Logs
        |
 Kinesis Data Streams
        |---- Flink / Spark Structured Streaming ----> alerts, aggregates
        |---- Firehose ----> S3 (history, Parquet)  ----> Athena
        `---- Firehose ----> OpenSearch (log search)

Why KDS: several consumers read the same records, the stream can be replayed within its retention, and the processing is real time. If the only requirement is “logs into S3”, Firehose alone is simpler — no stream, no Spark.

“How do you handle duplicate events?”

• give each event a stable event_id at the source;
• make processing idempotent, and deduplicate on event_id within a bounded window where needed;
• checkpoint the consumer’s position so a restart resumes rather than starts over;
• remember that two sinks share no distributed transaction — writing to S3 and to OpenSearch can succeed for one and fail for the other, so each must tolerate a replay (stream_pipeline.md).

“And late-arriving data?” — event-time windows with a watermark in the streaming job; in the batch layer, a lookback that reprocesses the last few partitions.

11-3) RDS into the lake ★★★★★Priority 5 of 5 — Must know — expect it in almost every loop

 Amazon RDS / Aurora  --DMS: full load, then CDC-->  S3 raw  --Glue / EMR MERGE-->  Iceberg  -->  Athena / Redshift

Why DMS: it does the initial full load and then streams changes from the database log, so the source is never rescanned. The follow-ups, each answered in aws_data.md §6:

• the initial snapshot and the moment CDC starts from, so nothing falls in the gap;
• deletes — DMS marks them with Op = 'D'; the MERGE must apply them;
• ordering — apply changes by the source’s change sequence, not by a timestamp that can tie;
• duplicates — the merge on the primary key makes a re-delivered change harmless;
• schema evolution — a new source column must not break the job;
• replication lag — alarm on it; on PostgreSQL a stalled consumer also fills the disk with WAL (aws_data.md §6-4).

12) Interview Toolkit ★★★★★Priority 5 of 5 — Must know — expect it in almost every loop

12-1) Service-selection table

12-2) The fifteen questions to rehearse

12-3) Things people say that are wrong

12-4) Last-review checklist

Storage and databases

• [ ] S3 vs EBS vs EFS
• [ ] S3 partitioning and the small-file problem
• [ ] RDS vs DynamoDB vs Redshift

Ingestion

• [ ] Kinesis Data Streams vs Firehose vs MSK
• [ ] SQS vs SNS vs EventBridge vs Kinesis
• [ ] DMS full load vs CDC

Processing

• [ ] Glue vs EMR vs Lambda
• [ ] Spark driver, executors, shuffle and skew
• [ ] Parquet, compression and partitioning

Analytics

• [ ] Athena vs Redshift
• [ ] Redshift distribution and sort keys
• [ ] EXPLAIN and query tuning

Architecture and operations

• [ ] Step Functions vs MWAA
• [ ] Glue Data Catalog vs Lake Formation
• [ ] IAM, KMS and Secrets Manager
• [ ] CloudWatch, retries, DLQs and idempotency
• [ ] cost per layer

Study order: S3, Glue, EMR, Spark, Kinesis, Redshift and Athena first — they carry most of an AWS system-design answer — then MWAA, Step Functions, DMS and Lake Formation.