This started manageable and then just kept growing.
Start by clarifying requirements: throughput, K, accuracy, time windows, and query patterns. Then propose a sharded ingestion pipeline where each shard maintains a Count-Min Sketch (or Space-Saving) for approximate counting, and a merging layer aggregates per-shard top-K candidates. Finally, discuss time-window semantics (tumbling/sliding) and trade-offs between accuracy, memory, and latency.
Pro tip: Emphasize that exact top-K over a high-throughput stream is infeasible at scale, so approximate algorithms with bounded error are the pragmatic choice—and explain how to merge sketches across shards without losing accuracy guarantees.
Ask about event rate, K value, acceptable error, time-window definitions (tumbling vs. sliding), and query latency. This scopes the design and shows you avoid premature optimization.
Define endpoints for querying top-K (e.g., GET /topk?window=5m&k=10) and for ingesting events. Describe a partitioned message queue (Kafka) feeding stateless ingestion workers that route events to shards by key.
Propose Count-Min Sketch or Space-Saving per shard for memory-efficient frequency estimation. Explain sharding by item hash to distribute load, and how each shard maintains its own sketch.
Describe a merger service that periodically collects per-shard top-K candidates (or merges sketches) and computes the global top-K. Discuss merge frequency, consistency, and caching for low-latency queries.
Explain how to support tumbling windows (fixed buckets) and sliding windows (overlapping buckets or decay). Discuss trade-offs: sketch size vs. accuracy, merge overhead vs. freshness, and memory vs. throughput.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.