← Atlassian Interview InsightsSoftware Engineer

Atlassian·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say
Jun 2026

Summary

System design round at Atlassian for a software engineer role. The whole thing was one big distributed systems question about building an async web crawler service, and it went pretty deep into territory I wasn't fully prepared for.

Questions Asked (6)

Q1

Design a REST API service where clients submit URLs and the service crawls each URL plus all its sublinks to extract image references. The API should support submitting a batch of URLs and polling for job status and results.

System DesignAPI & IntegrationsData Modeling
Author's notes

I got the basic shape of it pretty quickly: POST to create a job, GET to poll status, return discovered images once done.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining the REST API contract clearly — endpoints for batch submission, status polling, and result retrieval — then walk through the async processing architecture that handles crawling at scale. Ground your design in practical trade-offs like rate limiting, depth control, and idempotency, which signal real-world engineering maturity.

Pro tip: Atlassian values systems that are resilient and observable; proactively mention how you'd handle crawl failures, retries with exponential backoff, and how operators would monitor job health via metrics or webhooks — this shows you think beyond the happy path.

1. Define the API Contract

Specify the core endpoints: POST /jobs to submit a batch of URLs (returning a job ID), GET /jobs/{jobId} for status polling, and GET /jobs/{jobId}/results for paginated image results. Clarify request/response schemas including fields like status (PENDING, IN_PROGRESS, COMPLETED, FAILED), submitted URLs, and extracted image URLs.

2. Design the Async Processing Architecture

Explain why crawling must be asynchronous — network I/O and unknown link depth make synchronous responses impractical. Describe a job queue (e.g., SQS, Kafka, or RabbitMQ) where each URL becomes a task consumed by a pool of crawler workers.

3. Model the Data Layer

Define the data model: a Jobs table (jobId, status, createdAt, totalUrls, completedUrls) and an Images table (jobId, sourceUrl, imageUrl, discoveredAt). Discuss using a relational DB for job metadata and optionally a key-value store or blob storage for large result sets.

4. Handle Crawling Constraints and Edge Cases

Address crawl depth limits, visited URL deduplication (using a distributed set like Redis), robots.txt compliance, and per-domain rate limiting to avoid being blocked or causing harm. Mention timeouts and retry logic with exponential backoff for failed URLs.

5. Address Scalability and Observability

Discuss horizontal scaling of crawler workers, idempotent job submission to prevent duplicate processing, and optional webhook callbacks so clients aren't forced to poll. Highlight metrics to expose (queue depth, crawl latency, error rates) and how partial results can be streamed before a job fully completes.

Key Points to Mention

  • Idempotent batch submission using a client-supplied or server-generated job ID to safely handle retries
  • Distributed URL deduplication with a Redis set or Bloom filter to prevent re-crawling the same link within a job
  • Configurable crawl depth and domain scoping to prevent runaway crawls and respect resource boundaries
  • Pagination on the results endpoint (cursor-based) to handle jobs that may yield thousands of image references
  • Webhook/callback support as an alternative to polling, reducing client load and improving responsiveness
  • robots.txt compliance and per-domain rate limiting to ensure ethical and safe crawling behavior

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

How would you architect the crawler itself? Walk through the worker pool, the URL frontier/queue, and how you'd deduplicate already-visited URLs.

System DesignTechnical Trade-offsAlgorithms & Data Structures
Author's notes

This part I felt okay about.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Structure your answer by walking through each major component in logical order: the URL frontier/queue, the worker pool, and deduplication, explaining the design decisions and trade-offs at each layer. Ground your architecture in concrete data structures and distributed systems primitives rather than staying abstract. Tie your choices back to real-world constraints like scale, fault tolerance, and politeness policies.

Pro tip: Demonstrate senior-level thinking by proactively addressing the 'politeness' problem — rate-limiting crawls per domain to avoid hammering servers — as this is a nuance many candidates miss and signals you've thought beyond the happy path.

1. Define Scale & Constraints

Briefly establish the scale you're designing for (e.g., billions of URLs, distributed across many machines) to justify your architectural choices. Clarify whether this is a focused crawler or a broad web crawler, as this shapes every subsequent decision.

2. Design the URL Frontier / Queue

Describe a multi-priority, multi-queue frontier that separates URLs by domain to enforce politeness and prioritize high-value pages. Explain using a combination of a priority queue (for importance scoring) backed by persistent storage like Kafka or Redis to survive crashes and handle backpressure.

3. Architect the Worker Pool

Explain a pool of async worker threads or processes that pull URLs from the frontier, fetch content, parse links, and emit new URLs back to the queue. Discuss using async I/O (e.g., Python asyncio or Java NIO) to maximize throughput since crawling is heavily I/O-bound, not CPU-bound.

4. Implement URL Deduplication

Walk through a layered deduplication strategy: a Bloom filter for fast, memory-efficient in-memory checks as the first gate, backed by a distributed key-value store (e.g., Redis or Cassandra) storing URL hashes for authoritative deduplication. Acknowledge the Bloom filter's false-positive trade-off and explain why it's acceptable here.

5. Address Fault Tolerance & Politeness

Explain how workers should be stateless so failed jobs can be retried, and how the frontier should use acknowledgment-based dequeuing to prevent URL loss. Describe per-domain rate limiting using token buckets or delay queues to respect robots.txt and avoid IP bans.

Key Points to Mention

  • Bloom filter for probabilistic, memory-efficient deduplication with a fallback persistent hash store (e.g., Redis SET with URL hashes or SHA-256 fingerprints)
  • Multi-queue URL frontier segmented by domain to enforce politeness (per-domain crawl delay) and support priority-based scheduling
  • Async I/O worker pool (e.g., asyncio, Netty) to handle high concurrency without thread-per-request overhead, since crawling is I/O-bound
  • Persistent, fault-tolerant queue backend (e.g., Kafka or Redis Streams) to enable at-least-once delivery and crash recovery
  • Content deduplication beyond URL deduplication using SimHash or MD5 of page content to avoid re-indexing duplicate pages at different URLs
  • Robots.txt compliance and per-domain rate limiting via token bucket algorithm to implement politeness and avoid being blocked

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

How would you handle polite crawling, including rate limiting per domain and respecting robots.txt?

System DesignTechnical Trade-offs
Author's notes

Blanked for a second on robots.txt parsing and just said I'd fetch and cache it at the start of crawling each domain.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Frame your answer around the core principles of being a 'good citizen' on the web, covering both technical mechanisms (rate limiting, robots.txt parsing) and architectural decisions (distributed coordination, backoff strategies). Structure your response by first defining the problem constraints, then walking through the implementation layers from protocol compliance to infrastructure design. Demonstrate awareness of trade-offs between crawl speed and server impact.

Pro tip: Mention that robots.txt should be cached with a TTL (e.g., 24 hours) and fetched before any other page on a domain — this signals you understand real-world operational concerns, not just the happy path. Bonus points if you bring up the Crawl-Delay directive in robots.txt and how it should override your default rate limits.

1. Robots.txt Compliance

Explain how to fetch and parse robots.txt for each domain before crawling any URLs, respecting Disallow rules, Allow overrides, and the Crawl-Delay directive. Discuss caching the parsed result with an appropriate TTL to avoid redundant fetches.

2. Per-Domain Rate Limiting

Describe maintaining a per-domain request queue with configurable delays (e.g., minimum 1 second between requests to the same domain by default). Explain using token bucket or leaky bucket algorithms to smooth out request rates rather than bursty behavior.

3. Distributed Coordination

Address how rate limits must be enforced globally across multiple crawler workers using a shared store like Redis, ensuring no single domain is hammered even when crawling at scale. Discuss using atomic operations or distributed locks to safely decrement and check rate limit counters.

4. Adaptive Backoff & Error Handling

Explain implementing exponential backoff when receiving 429 (Too Many Requests) or 503 responses, and honoring the Retry-After header when provided. Discuss temporarily deprioritizing or pausing a domain's queue rather than dropping URLs entirely.

5. Monitoring & Tuning

Highlight the importance of tracking per-domain metrics like error rates, response times, and crawl throughput to dynamically adjust rate limits. Mention alerting on anomalies such as a sudden spike in 4xx/5xx responses from a specific domain as a signal to back off further.

Key Points to Mention

  • Token bucket or leaky bucket algorithm for smooth, configurable rate limiting per domain
  • Robots.txt fetching, parsing, and caching with TTL — including Crawl-Delay directive support
  • Redis or a distributed cache for globally consistent per-domain rate limit state across crawler workers
  • Exponential backoff with jitter on 429/503 responses and honoring the Retry-After header
  • Prioritizing politeness over crawl speed as a design principle, with configurable defaults per domain tier
  • Handling edge cases like robots.txt fetch failures (fail open vs. fail closed) and wildcard User-Agent rules

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

How do you know when a crawl job is fully complete? Specifically, how do you detect that all sublinks have been discovered and processed?

System DesignAlgorithms & Data Structures
Author's notes

This was the hardest part and I did not have a clean answer ready.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Frame your answer around the core termination condition problem in distributed crawling: a job is complete when the frontier (queue of URLs to visit) is empty AND no worker is actively processing a URL that could produce new links. Walk through concrete data structures and coordination mechanisms that enforce this guarantee, then address edge cases like cycles and transient failures.

Pro tip: Mention the subtle race condition where the queue appears empty but in-flight workers haven't yet enqueued their discovered links — interviewers at Atlassian love candidates who proactively identify distributed systems pitfalls rather than waiting to be prompted.

1. Define 'Complete' Precisely

Start by clarifying that completion means both the URL frontier is empty AND all in-flight crawl tasks have finished without producing new URLs. This dual condition is the crux of the problem and sets up the rest of your answer.

2. Track URL States with a Visited Set

Describe maintaining three sets or states for URLs: discovered (queued), in-progress (being fetched), and completed (processed). A distributed set like Redis or a database table prevents re-crawling and provides a global view of progress.

3. Use an Active Worker Counter

Introduce an atomic counter (or distributed semaphore) that increments when a worker picks up a URL and decrements when it finishes — including after enqueuing any new child links. Completion is signaled only when the frontier is empty AND this counter reaches zero.

4. Handle Cycles and Deduplication

Explain that without deduplication, cycles in the web graph cause infinite crawling; the visited set acts as a bloom filter or hash set to skip already-seen URLs. Mention canonicalization (normalizing URLs) to avoid treating the same page as distinct.

5. Address Failure and Timeout Scenarios

Discuss how crashed workers can leave URLs stuck in 'in-progress' state, requiring a heartbeat or lease mechanism with a timeout to reclaim and re-queue those URLs. This ensures the termination condition is eventually reachable even under partial failures.

Key Points to Mention

  • Dual termination condition: empty frontier queue AND zero active/in-flight workers
  • Distributed visited set (e.g., Redis SET or bloom filter) for deduplication and cycle detection
  • Atomic in-flight counter or lease-based task tracking to close the race condition window
  • URL canonicalization to prevent duplicate entries masquerading as new links
  • Heartbeat/lease expiry mechanism to recover URLs from crashed or stalled workers
  • Politeness and rate-limiting considerations that can affect perceived completion timing

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q5

How would you handle failures, retries, and timeouts in the crawler?

System DesignTechnical Trade-offs
Author's notes

Covered exponential backoff with a max retry cap, dead-letter queue for URLs that keep failing, and per-request timeouts to avoid a single slow page blocking a worker.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Frame your answer around building a resilient, fault-tolerant crawler by addressing failures at multiple layers: network, parsing, and infrastructure. Walk through a concrete strategy covering detection, retry policies, and graceful degradation, demonstrating that you think about both the happy path and edge cases. Tie your decisions back to trade-offs like throughput vs. reliability and resource consumption vs. completeness.

Pro tip: Mention idempotency as a first-class design concern — ensuring that retrying a URL never causes duplicate processing or data corruption signals senior-level thinking and is especially valued at Atlassian where data integrity in tools like Confluence and Jira is critical.

1. Classify Failure Types

Distinguish between transient failures (network timeouts, 503s), permanent failures (404s, malformed HTML), and systemic failures (target site is down). Each category warrants a different response strategy rather than a one-size-fits-all retry.

2. Define Retry Policies

Apply exponential backoff with jitter for transient errors to avoid thundering herd problems, and set a maximum retry count (e.g., 3-5 attempts) before marking a URL as failed. Permanent errors like 404 should be logged and skipped immediately without retries.

3. Implement Timeout Controls

Set layered timeouts — connection timeout, read timeout, and an overall request deadline — to prevent slow or unresponsive hosts from blocking crawler threads. Use circuit breakers to temporarily stop crawling domains that consistently fail, protecting system resources.

4. Maintain a Persistent Failure Queue

Store failed URLs with metadata (error type, retry count, last attempt timestamp) in a durable queue or database so the system can resume after crashes without losing state. This enables scheduled re-crawls and post-mortem analysis of failure patterns.

5. Monitor, Alert, and Tune

Instrument the crawler with metrics on failure rates, retry counts, and timeout frequency per domain to detect anomalies early. Use these signals to dynamically adjust crawl rate limits and timeout thresholds, and set up alerts for sustained failure spikes.

Key Points to Mention

  • Exponential backoff with jitter to prevent retry storms and thundering herd issues
  • Circuit breaker pattern to isolate and quarantine consistently failing domains
  • Idempotency of URL processing to ensure safe retries without duplicate data
  • Differentiated handling for HTTP status codes (e.g., 4xx vs. 5xx vs. network-level errors)
  • Layered timeout configuration (connect, read, and overall deadline) at the HTTP client level
  • Dead-letter queue or persistent failure store for auditing and scheduled re-crawl of failed URLs

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q6

How would you store intermediate crawl state so the system can recover if a worker crashes mid-job?

System DesignData Modeling
Author's notes

Said I'd persist the queue state and per-URL status to a durable store rather than keeping it only in memory.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Frame your answer around fault tolerance and idempotency: explain how you would checkpoint crawl progress in a durable external store so any worker can resume from the last known good state. Structure your response by first identifying what state needs to be persisted, then how it's stored, and finally how recovery is triggered and executed.

Pro tip: Mention that all state transitions should be idempotent — re-processing a URL that was already crawled should be a no-op — because this eliminates the need for complex exactly-once guarantees and makes recovery trivially safe. This signals production-level thinking that interviewers at Atlassian will appreciate.

1. Define the State to Persist

Identify all meaningful intermediate state: the URL frontier (pending, in-progress, completed), crawl depth, metadata fetched so far, and any pagination cursors. Being explicit about what constitutes 'state' shows you understand the problem scope.

2. Choose a Durable State Store

Select an appropriate storage backend — e.g., Redis with persistence (RDB/AOF), a relational DB, or a distributed KV store like DynamoDB — based on latency, durability, and scale requirements. Justify your choice by weighing consistency needs against performance.

3. Design the Checkpointing Mechanism

Implement periodic checkpoints where workers atomically write their progress (e.g., last processed URL, batch offset, or page token) before moving to the next unit of work. Use a heartbeat or lease mechanism so the coordinator can detect stale workers.

4. Handle Worker Failure and Recovery

When a worker crashes or its lease expires, a coordinator (or another worker) reads the last checkpoint and re-enqueues the in-progress work. Ensure all crawl operations are idempotent so re-processing doesn't cause duplicate data or side effects.

5. Address Edge Cases and Trade-offs

Discuss trade-offs such as checkpoint frequency (too frequent adds overhead; too infrequent increases re-work on failure) and how to handle partial writes using atomic operations or two-phase commits. Mention monitoring and alerting for stuck or repeatedly failing jobs.

Key Points to Mention

  • Idempotency of crawl operations to make re-processing safe and simplify recovery logic
  • Lease/heartbeat mechanism to detect crashed workers and reclaim in-progress tasks
  • Atomic checkpoint writes (e.g., using Redis transactions or conditional DB updates) to avoid partial state corruption
  • URL frontier state management — distinguishing between pending, in-flight, and completed URLs using status fields or separate queues
  • Checkpoint granularity trade-off: balancing write overhead against the amount of work lost on failure
  • Distributed coordination patterns such as using a message queue (e.g., SQS, Kafka) with visibility timeouts as a natural lease mechanism

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.