ORIGINAL SYSTEM DESIGN QUESTION
Streaming data transformation
Transform events into curated analytics records with schema evolution.
Original exercise · advanced. Editorial role relevance does not imply a company has asked this question.
Requirements and constraints
- 50,000 events/second; average 2 KB; seven-day replay.
- Events can arrive late; transformations must be versioned and reproducible.
- State assumptions and justify security, cost, operational ownership and data retention.
Pressure round
A new transform version misclassifies a week of events.
Build and explain the design
Requirements & scope
Who uses it? What must it do? What is explicitly out of scope? State latency, durability and privacy goals.
Self-review guidance
Define the actors, required operations and out-of-scope behavior. Explicit requirement: Events can arrive late; transformations must be versioned and reproducible.
Capacity estimates
State assumptions, peak requests/second, data volume and retention. Show your arithmetic and units.
Self-review guidance
Use this workload: 50,000 events/second; average 2 KB; seven-day replay. Convert rates and payload sizes into peak compute/storage assumptions; show units, headroom and what remains unmeasured.
APIs & data model
Describe endpoints/events, entity keys, indexes, authorization, and idempotency.
Self-review guidance
Define source offsets, schema versions and output deduplication keys.
Architecture & data flow
Describe components and request/event paths. Use plain text arrows to sketch your design; explain each responsibility.
Self-review guidance
Use partitioned ingestion, checkpointed transforms and immutable replay inputs.
Failures & observability
What happens on overload, duplicate events, partial failures or lost dependencies? Discuss monitoring and recovery.
Self-review guidance
Pressure-test: A new transform version misclassifies a week of events. Describe bounded recovery, deduplication, observability and a user-visible failure state.
Tradeoffs & alternatives
Compare two plausible approaches. State the costs, consistency model and what would make you change the design.
Self-review guidance
Compare replay into new tables with in-place correction and consumer disruption.
Design guidance describes assumptions and tradeoffs; no architecture has been simulated or benchmarked.
Practice this question
Save your work privately, explain decisions and return to your revision queue.
Open practice workspaceBrowse the question library →