← Okta Interview Insights

Okta·Software Engineer·Onsite - System Design / Architecture·Senior

Senior
Apr 2026

Summary

System design round at Okta for a software engineering role. The whole session was one big deep-dive into designing a distributed metrics backend at scale, think Thanos or Cortex territory. Pretty intense if you haven't thought hard about time-series storage before.

Questions Asked (5)

Q1

Design a distributed backend for storing and querying time-series metrics at large scale, supporting high-volume ingestion, PromQL-style label queries, long-term retention, and multi-tenant isolation.

System DesignTechnical Trade-offsData Modeling
Author's notes

This was the whole interview, not just one question.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and scale (ingestion rate, retention, query patterns, tenant count). Then propose a layered architecture: ingestion pipeline, distributed storage with time-based partitioning, query engine with label indexing, and multi-tenant isolation via namespaces and resource quotas. Discuss trade-offs between consistency, latency, and cost, and how to handle high cardinality and long-term retention.

Pro tip: Emphasize multi-tenant isolation at every layer (ingestion, storage, query) and discuss how to prevent noisy neighbors through quotas and rate limiting. Also, mention the importance of cardinality control and downsampling for long-term retention to manage costs.

1. Clarify Requirements and Scale

Ask about expected ingestion volume (e.g., millions of samples/sec), retention period, query types (instant vs range, aggregation), and tenant count. This shapes design decisions.

2. Design Ingestion Pipeline

Propose a distributed write path: load balancers, stateless ingestion services, a message queue (e.g., Kafka) for buffering and decoupling, and batch writes to storage. Include validation, rate limiting, and tenant tagging.

3. Design Storage Layer

Use a time-series database (e.g., Prometheus TSDB, Cortex, Thanos) or custom storage with time-based partitioning (e.g., by day/week). Store recent data in hot storage (SSD) and older data in cold storage (object store). Implement downsampling and retention policies.

4. Design Query Engine and Indexing

Build a query service that parses PromQL, uses label indexes (inverted index) to find series, and executes queries in parallel across shards. Cache frequent queries and pre-aggregate data for common queries.

5. Ensure Multi-Tenant Isolation and Scalability

Isolate tenants via separate namespaces, resource quotas, and rate limits. Use sharding by tenant and time to scale horizontally. Monitor and alert on per-tenant usage to prevent noisy neighbors.

Key Points to Mention

  • Time-based partitioning and downsampling for efficient long-term storage and query performance.
  • Label indexing (inverted index) to support PromQL-style queries and handle high cardinality.
  • Use of a distributed message queue (e.g., Kafka) for reliable, high-throughput ingestion.
  • Multi-tenant isolation through namespaces, quotas, and rate limiting at ingestion and query layers.
  • Trade-offs between consistency, availability, and latency (e.g., eventual consistency for ingestion vs strong consistency for queries).
  • Cost optimization by tiered storage (hot/warm/cold) and compression techniques.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

How would you design the query path to handle fan-out across shards and deduplicate replicated samples efficiently?

System DesignTechnical Trade-offs
Author's notes

I fumbled a bit here.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the requirements and constraints of the query path, such as latency, consistency, and data volume. Then, outline a design that addresses fan-out and deduplication, discussing trade-offs between different strategies. Finally, explain how you would validate and monitor the solution.

Pro tip: Emphasize the importance of idempotency and how you would handle partial failures during fan-out to ensure consistent deduplication. Also, mention that you would consider using a bloom filter or similar probabilistic data structure to reduce memory overhead in deduplication.

1. Clarify Requirements and Constraints

Ask questions to understand the expected query latency, consistency requirements, data volume, and shard characteristics. This ensures the design aligns with business needs.

2. Design Fan-Out Mechanism

Describe how queries are distributed to shards, including parallelism, timeouts, and handling of slow or failed shards. Consider using a scatter-gather pattern with result aggregation.

3. Implement Deduplication

Explain how to deduplicate replicated samples, such as using a unique identifier (e.g., sample ID) and a hash set or bloom filter. Discuss trade-offs between exact and approximate deduplication.

4. Address Trade-Offs and Optimizations

Discuss trade-offs between latency, accuracy, and resource usage. Mention optimizations like early termination, caching, and adaptive parallelism.

5. Validation and Monitoring

Outline how to test the design (e.g., load testing, chaos engineering) and monitor key metrics (e.g., latency, deduplication rate, error rates).

Key Points to Mention

  • Scatter-gather pattern for fan-out queries
  • Unique identifiers for deduplication (e.g., sample ID)
  • Trade-offs between exact and approximate deduplication (e.g., bloom filters)
  • Handling partial failures and timeouts during fan-out
  • Idempotency and consistency guarantees
  • Monitoring and observability for query performance

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

What consistency model would you choose for this system and how would you handle failures in the ingestion path?

System DesignTechnical Trade-offs
Author's notes

Went with eventual consistency, justified it by saying metric data is append-only and losing a few samples is usually acceptable versus blocking writes.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the system's requirements and constraints, then propose a consistency model that balances correctness and availability for the ingestion path. Explain how you would handle failures using techniques like idempotency, retries, and dead-letter queues, and discuss trade-offs.

Pro tip: Tie your answer to Okta's identity and security context by emphasizing auditability and data integrity, and mention how eventual consistency can be acceptable for non-critical data but strong consistency is crucial for security events.

1. Clarify Requirements

Ask questions to understand the system's purpose, data volume, latency requirements, and consistency needs. For Okta, consider identity data and security implications.

2. Choose Consistency Model

Propose a consistency model (e.g., eventual, strong, causal) based on requirements. Justify why it fits, considering trade-offs like latency vs. correctness.

3. Design Failure Handling

Outline strategies for handling failures in the ingestion path, such as retries with exponential backoff, idempotent writes, dead-letter queues, and monitoring/alerting.

4. Address Trade-offs

Discuss the trade-offs of your choices, including impact on availability, durability, and complexity. Mention how you would validate the design.

Key Points to Mention

  • CAP theorem and its implications for distributed systems
  • Idempotency and deduplication to handle retries safely
  • Dead-letter queues for poison messages and failure isolation
  • Monitoring, alerting, and metrics for ingestion pipeline health
  • Backpressure and rate limiting to handle traffic spikes
  • Audit logging and data integrity for security compliance

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

How would you deal with cardinality explosion and noisy or high-volume tenants in a multi-tenant metrics system?

System DesignTechnical Trade-offsProduct Analytics & Metrics
Author's notes

Cardinality explosion is one of those things you either know about or you don't, and I knew just enough to not embarrass myself.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by acknowledging the two related but distinct challenges: cardinality explosion (too many unique time series) and noisy tenants (high-volume or bursty traffic). Then propose a multi-layered strategy that includes prevention (e.g., label design, cardinality limits), detection (e.g., monitoring cardinality and tenant usage), and mitigation (e.g., sampling, aggregation, rate limiting, and isolation). Emphasize trade-offs between accuracy, cost, and fairness, and how you would prioritize based on business impact.

Pro tip: Show that you understand the difference between cardinality explosion (a data model problem) and noisy tenants (a resource contention problem) and that you would address them with different tools and policies. Also, mention that you would instrument the metrics system itself to detect these issues early and set up alerts.

1. Clarify requirements and constraints

Ask about scale (number of tenants, metrics per tenant, ingestion rate), SLAs, and whether the system is push or pull based. Understand what 'noisy' means in this context (e.g., high cardinality, high volume, or both).

2. Prevent cardinality explosion at the source

Enforce label best practices (e.g., avoid high-cardinality labels like user IDs), set cardinality limits per metric, and use aggregation or rollups before ingestion. Consider using a schema or validation to reject or sanitize high-cardinality data.

3. Detect and monitor cardinality and tenant usage

Implement monitoring for cardinality growth and per-tenant ingestion rates. Set up alerts for anomalies and track which tenants contribute most to cardinality and volume.

4. Mitigate noisy tenants with isolation and controls

Apply per-tenant rate limiting, quotas, and sampling. Isolate tenants using separate pipelines, shards, or clusters to prevent one tenant from impacting others. Consider tiered service levels.

5. Optimize storage and query performance

Use techniques like downsampling, retention policies, and efficient indexing. For high-cardinality metrics, consider alternative stores (e.g., columnar or time-series databases) and query optimizations.

Key Points to Mention

  • Cardinality explosion: causes (e.g., unbounded labels, dynamic dimensions) and prevention (label design, limits, aggregation).
  • Noisy tenant: definition (high volume, bursty, or high cardinality) and impact on shared resources.
  • Rate limiting and quotas: per-tenant limits, token buckets, and fair scheduling.
  • Isolation strategies: separate pipelines, shards, or clusters; cell-based architecture.
  • Sampling and aggregation: adaptive sampling, rollups, and downsampling to reduce load.
  • Monitoring and alerting: track cardinality, ingestion rates, and query performance; set SLOs.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q5

Walk through the storage tier design: what lives in hot memory versus what gets flushed to object storage, and how does compaction work?

System DesignData Modeling
Author's notes

Recent data in memory plus WAL, older blocks compacted and shipped to S3.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the storage tier design around access patterns and performance requirements, then clearly delineate what data belongs in hot memory (e.g., recent, frequently accessed) versus what gets flushed to object storage (e.g., cold, archival). Finally, explain the compaction process, including triggers, mechanics, and trade-offs, ensuring to tie it back to scalability and cost efficiency.

Pro tip: Emphasize that compaction is not just about merging files but also about maintaining read performance and reducing write amplification; mention how you'd monitor compaction lag and adjust strategies dynamically.

1. Clarify requirements and access patterns

Ask about read/write ratios, latency SLAs, data volume, and retention policies to ground your design in concrete needs.

2. Define hot memory tier

Describe what data lives in memory (e.g., recent writes, frequently accessed records) and the data structures used (e.g., memtables, caches) for fast access.

3. Define flushing to object storage

Explain the flush policy: when and how data moves from memory to object storage (e.g., size-based, time-based), and the format (e.g., SSTables, Parquet).

4. Explain compaction mechanics

Detail the compaction process: merging smaller files into larger ones, removing duplicates/tombstones, and the strategies (e.g., size-tiered, leveled) with their trade-offs.

5. Discuss trade-offs and optimizations

Cover how compaction impacts read/write performance, storage cost, and how to tune it (e.g., compaction thresholds, background vs. foreground).

Key Points to Mention

  • Hot memory stores recent writes and frequently accessed data (e.g., memtable, cache) for low-latency reads.
  • Flushing to object storage is triggered by memory thresholds or time, often using immutable files (e.g., SSTables).
  • Compaction merges files to reduce read amplification and reclaim space, with strategies like size-tiered or leveled compaction.
  • Compaction trade-offs: write amplification vs. read performance, and impact on I/O and CPU.
  • Monitoring compaction lag and adjusting parameters (e.g., compaction interval, file size) for optimal performance.
  • Object storage characteristics: durability, cost, and higher latency, suitable for cold data.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.